When I wrote about the
first edition of Eric Siegel’s
Predictive Analytic: The Power to Predict Who Will Click, Buy, Lie, or Die, I highlighted the work of John Elder, founder and CEO of Elder Research. He ran a small hedge fund for nearly a decade, but eventually found that “the key measure of system integrity began to decline.” At that time I asked readers to offer ideas on measuring system integrity but was met with deafening silence.
Now, three years later, the revised edition of Siegel’s book has been published (Wiley, 2016). Re-reading it (and it is an engaging, if sometimes repetitive read), I was led to Elder’s
2014 white paper “Evaluate the Validity of Your Discovery with Target Shuffling,” an excerpt from “
3 Ways to Test the Accuracy of Your Predictive Models.” Systems traders may find some useful ideas here.
Posts
In the trading context, I regress a few predictors against returns to see if the tstats and coefficients are of a constant sign and significant magnitude over time. I do this in a rolling window, and if the tstats were to get too close to 0, that would be a warning sign that the predictors were losing their edge.
ReplyDelete