Why We Moved From LSTMs to Gradient-Boosted Trees for Equity Signals
Our LSTM models looked great on backtest but degraded fast in live trading. Here is why XGBoost with hand-crafted features outperformed deep learning for our ASX signal pipeline.
When we first built trade's equity signal pipeline, LSTMs were the obvious choice. Sequence models for time-series data — it makes intuitive sense. Our backtests showed strong Sharpe ratios and the architecture was clean.
Then we went live on ASX equities. Within three weeks, signal quality had degraded measurably. The LSTM was overfitting to recent market regimes and failing to generalise when conditions shifted. Retraining helped temporarily, but the cycle repeated.
We ran a head-to-head comparison: LSTM vs. XGBoost with 23 hand-crafted features (rolling volatility ratios, order flow imbalance, cross-asset correlations). The XGBoost model was more stable across regime changes, faster to retrain, and easier to interpret — we could see exactly which features drove each signal.
The key insight: for our use case, feature engineering matters more than model complexity. A well-engineered gradient-boosted ensemble with domain-specific features outperformed a deep learning model that learned its own representations. The LSTM was learning spurious patterns in the training window that did not persist.
We still use transformer architectures for our NLP sentiment pipeline, where they genuinely excel. But for price-action signals, XGBoost with handcrafted features has been our production workhorse for over a year now.