Silver, The Signal and the Noise

Nate Silver, the author of the popular political forecasting blog FiveThirtyEight, now part of The New York Times stable, is out with his first book, The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t (Penguin Press, 2012). Although Silver covers a broad spectrum of topics, from weather forecasting to sports, with a terrific chapter on poker from the vantage point of a Bayesian, I am going to focus on a couple of general ideas that are relevant for investors and traders and then turn to his discussion of the financial markets.

We live in an age of information glut. A few years back retail investors opened up their daily newspapers to see how their investments were doing; now they can follow their holdings in real time, tick by tick. In the past they got stock recommendations from their broker, read (or didn’t read) annual reports, and perhaps watched Wall $treet Week with Louis Rukeyser. Now they are deluged with the constant chatter of pundits, twitter feeds, webinars—you name it. But are investors better off with all this information? Silver doesn’t think so: “We face danger whenever information growth outpaces our understanding of how to process it.” (p. 7)

The challenge is to separate the signal from the noise, to construct a predictive model that rises above the fanciful level of a child finding animal patterns in clouds. “Finding patterns is easy in any kind of data-rich environment…. The key is in determining whether the patterns represent noise or signal.” (p. 240) If the NYSE has closed higher on Mondays 59% of the time over the past year (a figure I invented) but over the last five months has been up only twice and down a whopping 17 (information I gleaned a while back from Bespoke: it is not current), does this information offer a tradeable signal? Or is it just noise? Even if it has statistical significance, does it have practical significance? That is, could an investor profit from this pattern? Silver offers his own example to suggest a negative outcome: the “Manic Momentum” strategy, that over a ten-year period outperformed the market handily without transaction costs but lost almost 99% of the trader’s original capital with a 0.25% per trade transaction cost.

Technical traders have another problem: in trying to find signals amid the noise they are prone to overfitting. They devise a complex function that “chases down every outlying data point, weaving up and down implausibly as it tries to connect the dots. This moves us further away from the true relationship,” if there is in fact any true relationship in price action, “and will lead to worse predictions.” (p. 166)

Simply trying to parse data in search of a predictive signal is a fool’s errand, Silver believes. He illustrates this point when he takes ECRI to task for its September 2011 prediction of the near certainty of a double dip recession. In explaining its reasoning ECRI invoked “dozens of specialized leading indexes.” “Theirs,” Silver writes, “was a story about data—as though data itself caused recessions—and not a story about the economy. ECRI actually seems quite proud of this approach. ‘Just as you do not need to know exactly how a car engine works in order to drive safely,’ it advised its clients in a 2004 book, ‘You do not need to understand all the intricacies of the economy to accurately read those gauges.’ This kind of statement is becoming more common in the age of Big Data. Who needs theory when you have so much information? But this is categorically the wrong attitude to take toward forecasting, especially in a field like economics where the data is so noisy. Statistical inferences are much stronger when backed up by theory or at least some deeper thinking about their root causes.” (p. 197)

We make predictions every day, most of them quite mindless. But when predictions are important, mindlessness has no place. Not only should we theorize about causes and relationships, we should also couch our conclusions probabilistically. And yet “most of us—including most of us who invest for a living—are [very poor] at estimating probabilities.” The exceptions are the skilled options traders “who make bets on probabilistic assessments of how much a share price might move.” (Silver is quick to point out, lest the reader miss the qualifier ‘skilled’, that “You should not rush out and become an options trader. … [M]ost options traders receive a poor return.”) (p. 364)

The Signal and the Noise is a very rich book, one that I highly recommend. It takes a technical topic and makes it not only accessible to the statistically unwashed but engrossing. And does so with vividly portrayed illustrations. Let me close with one of my favorites: the two-track market.

“There is the signal track, the stock market of the 1950s that we read about in textbooks. This is the market that prevails in the long run, with investors making relatively few trades, and prices well tied down to fundamentals. … Then there is the fast track, the noise track, which is full of momentum trading, positive feedbacks, skewed incentives and herding behavior. Usually it is just a rock-paper-scissors game that does no real good to the broader economy—but also perhaps no real harm. It’s just a bunch of sweaty traders passing money around. However, these tracks happen to run along the same road, as though some city decided to hold a Formula 1 race but by some bureaucratic oversight forgot to close one lane to commuter traffic. Sometimes, like during the financial crisis, there is a big accident, and regular investors get run over.” (p. 368)

Aburdene, Conscious Money

Conscious Money: Living, Creating, and Investing with Your Values for a Sustainable New Prosperity (Atria/Simon & Schuster, Beyond Words Publishing, 2012) by Patricia Aburdene addresses the common conflict that people experience between making money and making a difference. It argues that a person’s values are not an impediment to wealth but a foundation for wealth creation and that conscious investors “seek to thrive financially without damaging others, themselves, or the Earth. Instead they: rely on human values to guide investment choices, invest in initiatives that further human and planetary evolution, and tap into intuition to balance and complement rational, objective financial data.” (p. 186)

As you might suspect, Aburdene draws on the mainstays of self-help literature as well as the socially responsible investing movement to make her case. In Megatrends 2010: The Rise of Conscious Capitalism, she described companies that had a set of values different from the run-of-the-mill; for one thing, they embraced a purpose beyond earning money. In Conscious Money she adds the individual and his/(mainly, it seems) her value set to the mix. Part I is entitled “The Inner Dimension of Conscious Money”; Part II, “The Conscious Marketplace.”

The result is something of a new-age intellectual mishmash. For instance, once you have “an instinctive sense of the business environments and practices that attract you, it’s time to turn inward, tap into your intuitive potential, and ‘get a feel’ for companies where you might want to do business.” (p. 105) How is this accomplished? In four steps: (1) creating a sacred space, (2) collecting questions, (3) connecting to a “felt sense,” and (4) harvesting the fruits of intuition. To accomplish step (3), “Take five deep, slow breaths to relax. Spend the next five minutes attempting to ‘feel into’ the business. Notice any place in your body that enters your awareness. If you feel constriction, for example, inquire as to what that feeling might be telling you. If your heart feels open, invite it to speak its message.” (p. 106)

For those who are open-minded and “open-hearted,” Aburdene offers some practical steps to understanding and improving one’s money mind-set. Some of these suggestions are eminently reasonable; others, such as “Make your next bill-paying session an Abundance ritual. Buy flowers or light candles. Play relaxing music,” not so much.

If you are a hard-core fear and greed investor, this book is not for you. If you believe in reason above intuition or spiritual consciousness when it comes to choosing companies in which to invest, you will probably not be persuaded otherwise (even though the power of intuition is well documented). If it takes arguments to convince you to consider a hypothesis, you will come away unconvinced. I admit to falling into the skeptical camp. And in many ways that’s too bad because our value systems underlie everything we do—how we vote, where we shop, who are friends are, what we hang onto and what we are willing to let go. Our attitudes toward wealth make it easier or harder to attain wealth, more or less desirable even to try.

Conscious Money touches on a range of important, timely topics. It just didn’t speak to me.

Wagner & Balog, Advanced Technical Analysis of ETFs

Deron Wagner, among his many other activities, is the founder of Morpheus Capital LP and Morpheus Trading Group, a trader education firm specializing in ETFs and stocks. Edward Balog is the head ETF trader for Morpheus Capital and coauthor of the Wagner Daily newsletter. Wagner and Balog have pooled their skills and experience, drawing extensively on their newsletter, to produce Advanced Technical Analysis of ETFs: Strategies and Market Psychology for Serious Traders (Bloomberg/Wiley, 2012).

The authors are swing traders who follow the trend of the broad market and who look for ETFs that have relative strength compared to the market as a whole. Their basic charts include the 20-day EMA as well as the 50- and 200-day MAs as trend guides. In addition, the authors explain their so-called advanced techniques: candlestick patterns, Fibonacci price levels and time series, and accumulation-distribution with RSI. They amply illustrate the use of these indicators with Tradestation charts.

What distinguishes this book from so many others in the field is that it walks the reader through fifteen long trades and fifteen short trades that the authors took using their own capital, including losing trades. Those who are expecting to see advanced technical analysis in action will be disappointed, however. As the authors write, “Because we have already provided specific trade examples of how to apply the advanced technical strategies, the trades in this [and the next] chapter seek to illustrate that profitable trading can be achieved simply through following the basics of our top-down ETF strategy. Application of the additional advanced strategies would only serve to enhance one’s profitability even further.” (p. 71) The skeptical reader is forced to ask why, if the strategies described in this book would add to profitability, the authors didn’t use them in the 2010 and 2011 trades they described in their newsletter. After all, these strategies aren’t exactly brand new.

Wagner and Balog round out their book with some thoughts on market psychology and a description of the most recent innovations in exchange-traded products (along with their tax consequences).

The authors use this book in part to promote the Wagner Daily newsletter, in existence now for ten years (morpheustrading.com). But the book is not merely a promotional piece. It shows how technical trading is actually done and provides some valuable lessons.

Little bets

Following up on my last post, here's an interesting piece from the blog Barking up the wrong tree.

Duhigg, The Power of Habit

“All our life, so far as it has definite form, is but a mass of habits,” William James wrote in 1892. Well, that might be a bit of an overstatement: a researcher in 2006 knocked that “mass” down to “over 40 percent.” Whatever the percentage, we are creatures of habit. In The Power of Habit: Why We Do What We Do and How to Change It (Random House, 2012) Charles Duhigg explores the work that neurologists, psychologists, sociologists, and marketers have done over the past two decades to figure out how habits work and how they change. It’s a fascinating tale.

So what is a habit anyway and why are habits so important? After we figure out a sequence of actions and practice it sufficiently (Duhigg uses the example of backing out of the driveway), our brain converts that sequence into an automatic routine, a habit, and stores it in our basal ganglia. We no longer have to think about backing out of the driveway; our brain is free to think about something else or to quiet itself. “Habits, scientists say, emerge because the brain is constantly looking for ways to save effort. Left to its own devices, the brain will try to make almost any routine into a habit, because habits allow our minds to ramp down more often.” The brain becomes more efficient; we can devote our mental energy to “inventing spears, irrigation systems, and, eventually, airplanes and video games.” (p. 28)

Most habits are innocuous enough; they don’t make a major difference in our lives. But some habits do, and not always for the better. Moreover, no matter what we do, those bad habits never really disappear; “they’re encoded into the structures of our brain.” The good news is that although old habits never die, they can be “ignored, changed, or replaced.” (pp. 29-30) How? Quite simply, at least in theory: by changing the habit loop of cue, routine, reward.

Ad men figured this out early on. Claude Hopkins, for instance, was responsible for making Pepsodent a sensation at a time when hardly any Americans brushed their teeth; “when the government started drafting men for World War I, so many recruits had rotting teeth that officials said poor dental hygiene was a national security risk.” (p. 39) A decade after the first Pepsodent campaign, more than half the American population brushed their teeth daily and flashed that Pepsodent smile. And they presumably believed that they no longer had that dingy film on their teeth that they could feel when they ran their tongue across their teeth, the cue that Hopkins devised to entice them to brush in the first place. (In fact, the toothpaste did nothing to remove the film, but then ads have never been known for their truthfulness.) So a simple habit loop was formed: cue (tooth film), routine (brushing), reward (beautiful teeth).

What if you already have a habit that you want to change? In that case, “you must keep the old cue, and deliver the old reward, but insert a new routine.” (p. 60) As one of the developers of habit reversal training said, “It seems ridiculously simple, but once you’re aware of how your habit works, once you recognize the cues and rewards, you’re halfway to changing it.” (p. 70) Some people need a support group to reinforce their belief in change, others are fine on their own.

Changing some habits makes very little impact on other parts of a person’s life; keystone habits, by contrast, have ripple effects. They “start a process that, over time, transforms everything.” (p. 87) “They help other habits to flourish by creating new structures, and they establish cultures where change becomes contagious.” (p. 94) Duhigg recalls Paul O’Neill’s fixation with safety when he became CEO of the troubled Alcoa and how “O’Neill’s plan for getting to zero injuries entailed the most radical realignment in Alcoa’s history.” (p. 91)

Keystone habits, which admittedly are difficult to identify and put into practice, create widespread changes because of the principle of small wins. “A huge body of research has shown that small wins have enormous power, an influence disproportionate to the accomplishments of the victories themselves. ‘Small wins are a steady application of a small advantage,’ one Cornell professor wrote in 1984. ‘Once a small win has been accomplished, forces are set in motion that favor another small win.’ Small wins fuel transformative changes by leveraging tiny advantages into patterns that convince people that bigger achievements are within reach.” (p. 96)

One example of what seems for many people to be a keystone habit is exercising. “Typically, people who exercise start eating better and becoming more productive at work. They smoke less and show more patience with colleagues and family. They use their credit cards less frequently and say they feel less stressed. It’s not completely clear why. But for many people, exercise is a keystone habit that triggers widespread change.” (p. 93)

Duhigg extends his analysis to willpower, how retailers predict (and manipulate) habits, how movements happen, and the neurology of free will. But let me stop here and make a couple of off-the-cuff, not especially profound observations.

Traders often repeat the same mistakes over and over, acting on triggers that have served them poorly time and time again even as they continue to expect a tidy profit as a reward. Not only is this insanity, the problem is that they’ve developed a powerfully destructive habit loop. They need to figure out a new routine—and in this case, I believe, contrary to habit reversal theory, either a new, non-monetary reward or a more probabilistic view of the reward.

Beginning traders sometimes feel compelled to swing a big line, and normally they lose big. Just think how many accounts have been blown out. Small wins are powerful levers (and don’t have the downside risk of using too much leverage). “Levers, not leverage”—it has a nice ring to it!

The Power of Habit is chock full of fascinating information—from Michael Phelps’s training regimen to how Target “targets” pregnant women as potential big-time spenders without seeming intrusive. I thoroughly enjoyed it and learned from it. Who can ask for much more? Now on to that next small win. (If you’re intrigued with this topic, you can follow it up with Peter Sims, Little Bets: How Breakthrough Ideas Emerge from Small Discoveries. I admit I haven’t read it yet.)

Buytendijk, Socrates Reloaded

A couple of years ago I wrote two posts on an earlier book by Frank Buytendijk, Dealing with Dilemmas. He is back with yet another thought-provoking book, Socrates Reloaded: The Case for Ethics in Business and Technology (Beingfrank Publications, 2012).

Buytendijk is an IT guy, not a philosopher. But he’s an IT guy with a fresh way of looking at intellectual and practical problems; as he titles his first chapter, “IT: Information Technology or Independent Thinking … Interesting Thought!”

Socrates Reloaded is wide-ranging. The author contemplates whether Marx predicted the end of the Internet giants, investigates the often stifling notion of best practices, and dispels the myth of one version of the truth.

In this post, however, I’m going to confine myself to a single topic: Can computers think?

Buytendijk starts with the famous Turing test from 1950: “Is it imaginable that a computer could fool a human being, and be taken for a human being as well?” If we focus only on the results, Buytendijk argues, “we cannot escape the conclusion that computers can think. In fact, they can think much better than we can. They can reason better, faster and deeper than human beings, with much more precision.” (p. 76) They would ace IQ tests. They can learn and evolve. Moreover, computers can be self-aware in the sense that they can run self-diagnostic software and report system malfunctions. Computers don’t seem to be all that different from us.

And yet we know they are, at least in their current state. “The killer argument is that computers do not create and invent things like we do. Computers haven’t created any true art simply because they felt like it. Computers haven’t displayed altruistic behavior. Computers don’t make weird lateral thinking steps and invent Post-it Notes when confronted with glue that doesn’t really stick, or invent penicillin by mistake.”

Buytendijk contends that ‘mistake’ is the key word here. Intellectually, “we, human beings, are special because we are deeply flawed. We make mistakes, we don’t always think rationally, our programming over many, many years of evolution is full of code that doesn’t make any sense, and so forth. We are special because we are imperfect.” (p.82) So for computers to become more like human beings they need to become more imperfect and rely more on fuzzy interpretation. Google’s search engine is, according to the author, a good example of the non-perfect computing paradigm.

Socrates Reloaded meanders through the history of philosophy looking for insights into business and IT problems. As a philosophical exercise, it’s not particularly satisfying. But for those in business or IT who want to expand their intellectual horizons it’s a great read. It’s challenging, not in the sense that it’s hard but in the sense that it shakes up preconceptions. And for those who worry about their privacy in the information age where function creep—in which data are used for purposes far different from the purposes for which they were collected—“simply happens” (p. 135), this book is a wake-up call. Reading Socrates Reloaded is definitely a worthwhile way to spend a few hours.

Schüll, Addiction by Design

Natasha Dow Schüll’s Addiction by Design: Machine Gambling in Las Vegas (Princeton University Press, 2012) is one of the most compelling books I’ve read in the past few years. Not because I was ever captivated by slot machines or video poker. In my entire life I lost a total of $5 to a slot machine and, quite frankly, even then I didn’t consider the experience worth anywhere close to $5. But the experience has changed, thanks to technology and mathematical algorithms; it has a deeper hook. Screen traders will recognize its addictive appeal.

Schüll, an associate professor at MIT, argues that addiction to machine gambling stems from the interplay between the gambler and the machine. Drawing on fifteen years of field research in Las Vegas and extensive interviews with both designers and addicts, she shows how the “duty to extract as much money” as possible from customers and the desire to play for as long as possible combine to produce a recipe for potential addiction.

Slot machines have come a long way from the coin-fed mechanical one-armed bandits. They now use video technology, which speeds up play significantly. On average, pulling a handle resulted in 300 games an hour. Video poker players can complete 900 to 1,200 hands an hour; the rate is similar on video slots. (p. 55) The financial flow in casinos has also sped up. Players no longer have to carry around heavy cups of coins or wait for payouts. Instead, casinos are “cashless.” Moreover, players who run out of money can easily tap into their checking accounts, credit cards, or debit cards—in numerous jurisdictions right from their machines—to keep on going.

Early on programmers devised techniques “not only to distort players’ perception of games’ odds but also to distort their perception of losses, by creating ‘near miss’ effects. Through a technique known as ‘clustering,’ game designers map a disproportionate number of virtual reel stops to blanks directly adjacent to winning symbols on the physical reels, so that when these blanks show up on the central payline, winning symbols appear above and below them far more often than by chance alone.” (p. 92)

Increasingly, mathematicians are designing games that match “math with markets, player types with schedule types.” (p. 109) There are two basic types of players—jackpot players who prefer “high volatility, low hit frequency” games and escape players (play-to-win-to-play players) who prefer “low volatility, high hit frequency” games. “On both machines you end up in the same place, which is zero. … It just takes longer to get there on the second one.” (p. 111)

The gamblers that Schüll interviewed were escape players. As a casino executive said, “What they really want to do ... is to play and forget and lose themselves. … [They want to] get in the zone” where “their own actions become indistinguishable from the functioning of the machine.” (pp. 170-71) This zone is not a happy place. Unlike Csikszentmihalyi’s flow, which is “life affirming, restorative, and enriching, … repeat machine gamblers … experience a flow that is depleting, entrapping, and associated with a loss of autonomy.” (p. 167) It is decidedly worse than T. S. Eliot’s melodramatic description of playing solitaire as “the nearest thing to being dead.”

Schüll’s book is masterfully crafted; it “hooks” you to keep reading until there is nothing left to read. You come away, however, enriched, not depleted. The interviews are gripping, the analysis is sophisticated, and the topic is important—not least because we are all being bombarded with technology that enables us to be “in the zone” in a host of deleterious ways. Traders who are even slightly self-reflective are bound to see something of themselves in these gambling addicts, and may perhaps take a deep breath.