Iteration 3 Recap: The Bill Came Due, and I Found Two Bugs I Should Have Caught
Every portfolio lost money. Every portfolio lost to the index. This is the first iteration where that has happened, and it happened to all four at once.
Iteration 3 Final Summary (June 30 to August 31)
| Return | vs S&P 500 | Result | |
|---|---|---|---|
| S&P 500 | +3.51% | ||
| Ensemble | -10.07% | -13.58 pp | L |
| Ensemble #2 | -6.91% | -10.43 pp | L |
| Growth Evaluator | -6.81% | -10.32 pp | L |
| ML Model #2 | -16.15% | -19.67 pp | L |
The S&P 500 did fine. It just kept plodding along at +3.51% while the portfolios gave back a chunk of what they had made. That is the concentration risk I have been writing about for two iterations finally showing its other face.
Best and Worst Picks per Portfolio
| Portfolio | Top picks | Bottom picks |
|---|---|---|
| Ensemble | PLTR +60.6%, ANET +19.1%, NVDA +11.0% | AMAT -33.7%, APP -36.4% |
| Ensemble #2 | PLTR +60.6%, NEM +35.8%, ANET +19.1% | AMAT -33.7%, APP -36.4% |
| Growth Evaluator | PTC +38.0%, NEM +35.8%, NVDA +11.0% | SNDK -29.5%, APP -36.4% |
| ML Model #2 | MSFT +37.6%, AVGO -1.4%, KEYS -5.4% | INTC -31.8%, AMAT -33.7% |
Look at ML Model #2's "top picks" column for a second. Its second and third best holdings were down 1.4% and 5.4%. When your bronze medal is a loss, you did not have a bad stock, you had a bad book.
AppLovin (APP) was the single worst holding across three of the four portfolios at -36.4%, and it is the one that bothers me most. It got downgraded during the iteration and my models had nothing to say about it. Palantir was the bright spot at +60.6%, the same name that was a bottom pick last iteration, which is a decent reminder that 42 trading days is a short window to judge anything on.
Where We Stand Since Launch
| Since launch | |
|---|---|
| S&P 500 | +13.06% |
| Ensemble | +35.54% |
| Ensemble #2 (since Apr 30) | +28.28% |
| Growth Evaluator | +32.41% |
| ML Model #2 | +36.30% |
So the lead survives. Six months in, the portfolios are still 15 to 23 points ahead of the index, and I am not going to pretend a bad 42 days erases that. But the cushion got noticeably thinner, and the reason it got thinner is worth more than the scoreboard is.
Mistake One: I Did Not Actually Know What Was In My Own Database
After this iteration went sideways I stopped tuning models and went and audited the data instead. I should have done it months ago. Here is what turned up.
My nightly job was quietly undoing its own repairs. The fundamentals fetcher would write a row with only partial data when the data provider returned an incomplete set. Worse, on every re-fetch it would blank out fields that were already good. I repaired 521 damaged rows one afternoon, and the nightly job rewrote 526 of them three hours later. I only caught it because I happened to look twice.
Two of my risk guards had been doing nothing at all. The Growth Evaluator has a guard that is supposed to penalize companies posting huge earnings growth on stalled revenue, the kind of number that is usually an accounting artifact rather than a real business. It fired on zero stocks. Not a few. Zero, on every single scoring date since April. It was comparing an absolute threshold against a value that had already been converted to a percentile rank, so the comparison could never be true.
The second one is almost funny. There is a penalty for companies that just missed earnings. It divided by 100 one time too many, so a 15% earnings miss produced a 0.15% penalty instead of the intended 15%. That one had been broken since the guard was written.
And my sentiment data was measuring only the survivors. The table I had been using to test news sentiment was built from the current S&P 500 list. Of the 84 companies that dropped out of the index between 2020 and 2024, it had data for exactly zero of them. Any conclusion I drew from that table was a conclusion about companies that had already succeeded. I have since backfilled it, all 157,000 rows, and closed most of the gap.
None of these alone ruined the picks, it was death by a thousand papercuts. They were all the sort of thing a careful look would have caught, and I did not take a careful look until after a bad iteration made me.
Mistake Two: My Hold Bias Was a Lock, Not a Tiebreaker
Every rebalance, a stock I already own gets a bonus when the models re-rank the universe. The idea is sensible: do not churn the whole book over a rounding error. I had that bonus set to 10 points on a 0 to 100 scale, and in my head that was a modest thumb on the scale. A stock would need to be meaningfully worse to get sold.
That is not what it was doing. The scores at the top of the list are packed extremely tightly. Between rank 15 and rank 100, the whole range spans about 18 points, which works out to roughly 0.21 points per rank. So a 10 point bonus was not worth a few places. Measured across every rebalance date, it was worth an average of 48 ranks, ranging from 33 to 64 depending on the day.
In plain terms: a stock my models had pushed down to 60th place was getting promoted to roughly 12th, and staying in the portfolio. That is not a tiebreaker. That is a stock that basically could not be sold. AMAT, which lost 33.7% this iteration, was ranked 23rd when I held it. FIX was 35th. AppLovin was 38th.
I backtested the alternatives properly this time, 117 rebalances going back to 2007 and 1,749 individual stock picks. The old setting came 14th out of 17 options on a combined ranking of win rate, total return, and drawdown. It returned less than half what the best setting did, with a worse drawdown.
What Changes Going Forward
The data gets audited days before a rebalance, not after one goes wrong. I have written the checks that would have caught all three of the problems above, and they run against the database rather than living in my head. The rule for me is simple: I do not generate picks until the audit is clean, and I do not run the audit on rebalance morning when I am in a hurry to fill.
The hold bias is now measured in ranks, not points. Instead of "add 10 to the score", it is "move an existing holding up 5 places". Five places means five places on every date, regardless of how tightly the scores happen to be packed that morning. The old setting quietly meant something different every single rebalance, which is exactly the kind of thing you do not notice until you go looking. This is the setting that won the backtest, and it is what I filled with today.
One more small thing that fell out of the audit: I was holding two share classes of the same company. Fox and Fox Class A, Alphabet Class A and Class C. Two tickers, two positions, one bet, and in the Growth Evaluator that meant I was really holding 13 companies in 15 slots. The system now treats share classes as one company and gives the freed slot to the next name on the list.
Where That Leaves Iteration 4
The new books are filled as of this morning. The models are still heavily weighted toward memory and semiconductors. Micron, Seagate, SanDisk, Western Digital, Lam Research and Teradyne have gone up somewhere between 2x and 28x in the last twelve months. That is a lot of run to buy into, and I am aware of it.
The one thing that genuinely improved is that the four portfolios no longer look alike. Last iteration they held 29 distinct companies across 60 slots, with four names in all four books. This time it is 39 distinct companies and only one name common to all four. The Growth Evaluator in particular has rotated hard into energy and now has the lowest internal correlation of the group. The concentrated memory bet is still there, but it now sits in one portfolio instead of being replicated across all of them.
I would rather be writing about another +30% iteration. But I learned considerably more from this one, and the two things I fixed were both silently wrong for months while the portfolios were winning. That is the uncomfortable part. Good results hid them. A bad iteration is what finally made me look.
See you in 42 trading days.