Stage 03 · reinforcement learning

DQN trading agent

Can an agent learn when to buy and sell from those features?

A deep Q-network trades AAPL one share at a time, trained on March 2020 to April 2023 and tested on 338 days it never saw. The trading desk runs this exact network in your browser, so you can watch every decision it makes.

$301best model (episode 2) vs $308 hold

$130final model, smallest drawdown

$64final max drawdown vs $163

338unseen test days

Watch the agent trade on the desk →

How it works

State. Daily changes in the seven features (squashed to 0–1) plus three position features: share of the five-share limit held, a holding flag and open profit.
Actions. Hold, buy one share (up to five) or sell the oldest share. Impossible actions are masked out.
Reward. Each day’s mark-to-market profit, net of 5 bp costs, so holding a losing position hurts as it happens.
Learning. Experience replay, a target network synced every 200 steps, and exploration that fades from 100% to 2% over 80% of training.
Fixing collapse. Early versions bought every day and never sold. Position-aware states, the daily reward and paced exploration fixed it.

Results

Final training episode trades and model loss — rl_trading_agent.ipynb · training

Training: profit swung as exploration faded

Episode	1	2	3	4	5
Training P&L	$145	$284	$31	$162	$63
Closed trades	215	149	145	103	122
Exploration ε	0.39	0.15	0.06	0.02	0.02
Mean loss	16.2	10.1	11.9	15.6	14.7

Episode 2 earned the most in training, and its saved model also did best on unseen data.

Test-period trades of the final model — rl_trading_agent.ipynb · test

Test period: trading around a core position

The final (episode 5) model made 170 buys and 167 sells in 338 days yet held shares on 95% of days, nudging its position up and down rather than going flat. It grossed $162, or $130 after costs.

Cumulative profit of the agent versus buy and hold and random, with shares held — rl_trading_agent.ipynb · test

Smallest drawdown, but buy-and-hold won

	Agent	Buy & hold	Random
Net P&L	$130	$308	$241
Closed trades	170	5	107
Win rate	56%	100%	56%
Max drawdown	$64	$163	$119
Profit factor	1.62	–	2.15
Days holding	95%	99%	89%

The agent rarely held all five shares, so it lagged in the mid-2024 rally but fell least in drawdowns.

Backtest of the saved model with its trade log — rl_trading_agent.ipynb · backtest

Backtest and trade log

Reloading the saved model and re-running the test period reproduces it exactly: $162.12 gross, 170 closed trades (167 sold by the agent, 3 at the end of the data). Every trade is logged with prices, dates and the indicators at entry.

Episode 2 model on the test period — model_ep2.keras · run through the notebook test code

The best model mostly held

Model	Test P&L	Trades
Episode 1	$49	49
Episode 2	$301	16
Episode 3	$231	62
Episode 5 (final)	$130	170
Buy & hold	$308	5

Episode 2 held all five shares on 95% of days, trimming only briefly, and nearly matched buy-and-hold with 10 of 16 trades winning.

Takeaways

The environment matters as much as the network. Position awareness, the right reward and paced exploration were what made it trade at all.
Pick models on validation, not the last epoch. The best checkpoint was episode 2, not the final one.
Stock-only agents ride the trend. With only buy and sell in a bull market, the best policy is close to holding.

In the repository

View on GitHub: damion-m/Quantitative-Strategies

rl_trading_agent.ipynbAAPL_trading_features.csvmodel_ep1–5.kerasfunc_lib.py