Modeling—Best Practices

You may remember Scott E. Page from my two posts on his terrific book The Difference. He is also the coauthor with John H. Miller of Complex Adaptive Systems: An Introduction to Computational Models of Social Life (Princeton University Press, 2007). This book isn’t as much fun to read, but here and there it sparked my imagination. Today I’m going to explore the topic of modeling. In a later post I’ll tackle the concept of “in between.”

Researchers in complex adaptive systems inevitably rely on computational models. In an appendix the authors offer some guidance about constructing models, guidance that I think is valuable for anyone trying to construct trading models. (I very loosely include both mechanical systems and discretionary guideposts under the rubric of models.) Let me highlight some of the salient points.

First, keep the model simple. “Making sure that your model has just enough of the right elements and no more is the most fundamental practice for any kind of modeling.” (p. 246)

Second, have turnable dials to control key assumptions. The authors consider optimization a kind of turnable dial in the sense that a human being is more or less an optimizing agent. But their concerns apply as well to trading system optimization. Of those system builders whose best friend is optimization, the authors ask, “Does the behavior of the system undergo slow changes as we turn the dial, or does the system experience rapid phase transitions where small movements of the dial result in abrupt changes in the system? What effect does the very first movement of the dial away from optimality have on the system?” (pp. 248-49)

Third, construct flexible frameworks. “A well-designed framework puts very few a priori constraints on the model, and thus the outcome is rich in possibilities.” (p. 249) Flexible frameworks allow trading ideas to have multiple, unique instantiations and allow trading systems to adapt to changing market conditions.

Fourth, create multiple implementations. For instance, test a trading system on a variety of markets and time frames.

Fifth, keep a lab notebook. It’s imperative to keep a record of how your model plays out in real life.

Finally, reward the right things. “Like any branch of science, one needs to reward the right accomplishments. While it may be true that lovely graphics, advanced coding techniques, frontier hardware, and so on may enhance computational models, ultimately it is the resulting science that must be judged.” (p. 254) Amen.