Evaluating a trading system

We’ve designed a trading system, we’ve optimized it, and we’ve tested its forecasting strength. (See the two previous posts: Degrees of freedom—KISS and Optimization. Are we there yet? No. We still have to look at performance metrics to determine whether we have a system to which we are willing to commit cold hard cash. If, for instance, profits look great but drawdowns are outsized, perhaps it’s not the system for us.

There are a whole host of performance metrics available, some trying to improve on the classic Sharpe ratio which penalizes upside and downside volatility equally. To measure risk-adjusted returns we have, among others, the Sortino ratio and the omega ratio.

Jaekle and Tomasini in Trading Systems suggest some general principles at play in choosing performance metrics. First, any indicator we use must always compare return to risk. Second, it should be normalized so it can be used across asset classes or trading systems. Net profit, the authors point out, has neither of these features. So those hawkers of black box systems who claim a 244% return or the ads that say “turn $10,000 into $68,769 in only six weeks” are, at the very least, not using acceptable performance metrics. Third, a performance indicator should indicate how consistent the system is. No one wants a system that made a ton of money ten years ago and has lost money every year since.

The authors tout an indicator included in the Trade Station report known as the RINA Index. It “represents the reward-risk ratio per one unit of time and it compares the ‘select net profit’ (net profit minus the positive and negative outlier trades, that is minus the abnormal trades that overcome the three standard deviation limit away from the average) divided by the average drawdown and again divided by the percentage of time in the market indicator. This indicator should always be over 30 and the higher the better.” (p. 32) Who am I to criticize? But this is not a stand-alone indicator; we can’t simply ignore outliers, however we choose to deal with them.

Perhaps the simplest way to evaluate a trading system is to look at the equity line which should grow fairly smoothly without many deep drawdowns. “Personally,” the authors continue, “we also appreciate many ‘flat times’, that is parts of the equity line that are horizontal: it means that no trading was done in that period since a filter took the system out of the market. We believe that there is no need to trade continuously and a good system should know when there is some edge to be exploited over the markets and conversely when it is more appropriate to sit on the sidelines.” (pp. 32-33)

* * *

Over the course of three posts I’ve outlined elements involved in designing and testing a trading system. I have just skimmed the surface. There are many resources available that dig deeper. Among them,

Bandy, Howard. Quantitative Trading Systems—Practical Methods for Design Testing and Validation (using Amibroker)

Carstens, Henry. “Introduction to Testing Trading Ideas.” Available online, along with a host of other pieces

Hill, John R., George Pruit, and Lundy Hill. The Ultimate Trading Guide

Jaekle, Urban and Emilio Tomasini. Trading Systems—A New Approach to System Development and Portfolio Optimisation

Jonelis, John. “Building a System” (using MetaStock). Available online, along with several other articles

Pardo, Robert. The Evaluation and Optimization of Trading Strategies

Stridsman, Thomas. Trading Systems and Money Management

Stridsman, Thomas. Trading Systems That Work

I’m sure I’ve missed some vital literature; feel free to add to my mini-bibliography.

And if you want to see brilliant trading system design in action go to CSS Analytics and its quant links.