Building an ML Pipeline, and Learning to Test It

Learning to build an ML pipeline end to end, and then to test whether the model was any good. It scored 86% accuracy, lost money in backtest, and had quietly been swapped for a mock.

1. 46-FEATURE LGBM • Basis term structure • OI velocity & skew • 86% direction acc. 2. STRATEGY SHELL • Entry quantile gate • Kelly sizing cap • Regime exit stop 3. TUI + VERCEL • Replay simulator • Live signal API • Production live
LightGBM Quantile Regression (86% Direction Accuracy across 46 Features): Predicts next-interval funding direction and magnitude quantiles while keeping entry, sizing, and exit rules modular.

A project for learning to build a machine-learning pipeline end to end, and then to tell whether the model it produced was worth anything. The model predicted funding rates: the periodic payment one side of a perpetual futures contract pays the other, which a hedged strategy can collect. An earlier phase, funding-rate-arb-backtesting, had built the backtester and found a simple baseline nothing beat: pick the best-paying venue and hold.

The pipeline covered the whole path. A 323K-row dataset across six trading venues and 20 assets, 46 engineered features, a LightGBM model that forecast a range rather than a single number so each prediction carried its own uncertainty, train, validation and holdout splits by date so the model never saw the future, and a live FastAPI + Vercel cron job serving predictions.

On paper it was excellent: 86% accuracy calling the direction of the next rate, 93% on the best venue. But that venue pays positive about 93% of the time anyway, so always guessing “positive” would have scored about the same. Past rates made up two-thirds of what the model leaned on. Its confidence scores were broken too, stuck between 0.05 and 0.11 whatever its accuracy, so the position sizer (Kelly, which bets more when confidence is high) churned 79 trades and lost more than its starting capital in a leveraged backtest.

The pipeline itself was quietly wrong in three places. The default prediction source was a mock, not the trained model. Rates were summed where they should have been averaged, inflating predictions three to ten times. And the filter meant to block low-confidence trades was switched off. None of it threw an error. Each bug just produced a plausible number. I fixed all three and added 32 tests to keep them fixed.

Two habits came out of it. Test the strategy separately from the model: the strategy shell treats entry, sizing and exit as swappable parts, and running it first on synthetic predictions showed the exit rule mattered most. Waiting for two actual negative rates beat exiting on predicted ones by 40-plus points, because prediction-based exits churned 600+ trades into fees. And hold the model to the dumb baseline. Once fixed, the best model-driven strategy returned 2.37% on the validation window against 2.17% for just holding the best venue, a thin edge on a single test period. A follow-up XGBoost model reached 96.6% holdout accuracy, a number that after all this says little on its own, and I never ran it against that baseline.