agrotime blog

It sounds like something out of The Matrix: a giant, world-spanning electronic network where high-powered machines, some of them using GPUs to gain a speed advantage, run secret, rapidly-evolving software algorithms that battle it out for profits in a high-stakes game of cat-and-mouse, attack-counterattack, that yields some $21 billion a year for the winners and can spell ruin for the losers. Except that it’s not The Matrix—it’s the stock and commodities markets, and the fact that these markets mainly consist now of computers trading against one another has been brought closer to the public’s attention by last month’s alleged theft of Goldman Sachs’ proprietary trading code.

The collection of computer-automated, high-speed trading technologies and techniques that are typically lumped under the heading of "high-frequency trading" (HFT) have been around for a while, but HFT has recently become heavily identified with the banking giant, Goldman Sachs, that dominates some aspects of it on the New York Stock Exchange. And as Goldman draws more media and congressional scrutiny, so will HFT. To prepare you for the high-frequency trading media onslaught, we’ll take a look at HFT and at a stock market that really isn’t what you thought it was.

If you look under the hood of the markets in 2009, you’ll find that the trading floor has been replaced by electronic networks, the frantic, hand-signaling traders have been replaced by computer systems, and all of moves in the trader’s dance—a thousand little tricks and techniques (some legal, some questionable, and some outright illegal) for taking regular advantage of speed, location, and information to generate profits—are executed hundreds of times per second, billions of times per day. And the whole enterprise is mainly powered by the same hardware from Intel, AMD, and NVIDIA, that Ars readers use for gaming.

The world that has been pulled over your eyes to blind you from the truth

Press reports of trading days that end with big gains or losses are typically accompanied by shots of a trading floor where young traders are either euphorically throwing papers into the air (up days) or staring dejectedly at a stock ticker with hand pressed to forehead, shoulders slumped in defeat (down days). These guys, you think, are "the market," and if you looked up the New York Stock Exchange (NYSE) on Investopedia you’d find nothing in the "Stocks Basics: How Stocks Trade" entry to disabuse you of this widely hold notion. "The NYSE is the first type of exchange… where much of the trading is done face-to-face on a trading floor," Investopedia declares, and it goes on to provide a description of how a floor-centered, face-to-face NYSE that hasn’t matched reality for about five years.

The floor of the New York Stock Exchange

Only about three percent of trading volume on the NYSE is actually carried out by means of traditional "open outcry" trading, where flesh-and-blood humans used to gather to buy and sell securities. The other 97 percent of NYSE trades are executed via electronic communication networks (ECNs), which, over the past ten years, have rapidly replaced trading floors as the main global venue for buying and selling every for of asset, derivative, and contract. So the ECNs are the markets in 2009, and those pit traders who pose for the cameras are mainly there for the cameras.

"Why don’t you know BATS?," Bernard Donefer, a finance professor and HFT expert at CUNY’s Baruch College, asked me rhetorically. "Because there’s nothing to look at. It’s based in Kansas; the computers are in Jersey City."

At the time that Donefer and I spoke last week, BATS was the third largest equity market in the world, behind the NYSE and NASDAQ, and it has been all-electronic since it began life in 2005. There has never been a floor that a CNBC camera crew could report from, so it’s essentially invisible to the general public. The NYSE and a few other exchanges keep hang on to their trading floors "mainly for branding purposes," Donefer told me.

The ECNs offer the advantages of speed, anonymity, error minimization, and audit trails. They’ve also "ported" many of the problems endemic to electronic networks—security vulnerabilities, the "garbage-in, garbage out" (GIGO) problem, and the problem of technology moving too fast for lawmakers, to name just three—from the Internet to the markets. But the problems with ECNs are a topic for another day. The real issue is that when the average retail investor gets an E*Trade account and tries to play the stock market, she typically has no idea that she’s going up against the market equivalent of IBM’s chess grandmaster-thumping supercomputer, Deep Blue.

When Apple discussed the new features of the forthcoming iPhone OS 3.0, SVP of iPhone Software Engineering Scott Forstall said that the iPhone would be capable of streaming video and audio directly over HTTP. Apple also advertised HTTP streaming as a feature of QuickTime X, the update of its media architecture coming in Snow Leopard. What it failed to explain, at least publicly, is how this streaming would be accomplished. Fortunately, Apple submitted its proposed protocol last month to the Internet Engineering Task Force (IETF) in the hopes that it will become a ubiquitous standard.

Apple identified what it considers a few issues with standard streaming, which generally uses the Real Time Streaming Protocol originally developed by Netscape and Real in the late ’90s. The biggest issue with RTSP is that the protocol or its necessary ports may be blocked by routers or firewall settings, preventing a device from accessing the stream. As the standard protocol for the Web, though, HTTP is generally accessible. Furthermore, no special server is required other than a standard HTTP server, which is more widely supported in content distribution networks, and more expertise in optimizing HTTP delivery is generally available than for RTSP.

Enter HTTP Live Streaming. The basic mechanics involve using software on the server to break an MPEG-2 transport stream into small chunks saved as separate files, and an extension to the .m3u playlist specification (.m3u8) to tell the client where to get the files that make up the complete stream. The media player client merely downloads and plays the small chunks in the order specified in the playlist, and in the case of a live stream, periodically refreshes the playlist to see if there have been any new chunks added to the stream.

This is in contrast to real-time streaming, as there would necessarily be a minimum latency of whatever duration the server slices the stream into (Apple refers to 10 seconds as an example). As the server encodes the video and slices it into 10 second clips, for instance, it creates or updates a playlist for the stream with the URL of the next clip. The client begins by downloading one or more of the clips, playing them in order. As one clip plays, the client begins downloading the next specified clip until it reaches a tag in the playlist that signals the end of the stream.

The protocol offers a way to specify alternate streams by pointing to separate playlists for each alternate stream. These generally would be of different quality and bandwidth requirements, so the client can request an appropriate stream for whatever network conditions allow. The client can also change to any of the alternate streams as needed, "such as when a mobile device enters or leaves a WiFi hotspot," according to Apple. So if your iPhone moves out of WiFi range and switches to 3G, the QuickTime player could request a lower bandwidth stream and begin downloading the alternate, smaller chunks instead.

Further, the protocol allows for the individual media clips to be encrypted so that broadcasters can limit access to paid subscribers, for instance. In this case, key files for decoding the encrypted clips are referenced in the playlist, and the client uses the key files to decrypt each one before playing. There is also a flag that broadcasters can set to disallow caching of individual media files as they are downloaded.

The only requirement is that the media must be formatted as an MPEG-2 transport stream, program stream, or audio elementary stream. Apple’s current implementation uses (unsurprisingly) H.264 video with AAC audio, though audio-only streams can use AAC, MP3, or the MPEG-2 elementary stream. The version of QuickTime in iPhone OS 3.0 is compatible with these formats, as is the version of QuickTime that will ship with Snow Leopard. Apple also has a beta version of a stream segmenter—currently only available to Apple Developer Connection members—to slice up a stream into individual files, create .m3u8 playlist files, and handle encryption and key generation.

Since the entire method works with standard HTTP transport and essentially any off-the-shelf software or hardware encoder can make the necessary MPEG-2 stream, it opens up streaming to nearly anyone. What Apple doesn’t say explicitly is that its protocol can negate the need for proprietary solutions such as Adobe’s Flash or Microsoft’s Silverlight to deliver remotely hosted or encrypted content. And since neither of those are likely to appear on the iPhone anytime soon, it’s one of the few ways to stream live video reliably to Apple’s mobile platform.

Currently the standard is an Internet-Draft, and we’ve yet to see any evidence that others are ready to jump on Apple’s bandwagon. But given the issues Apple has identified with RTSP, the proprietary nature of other common solutions, and the inclusion of support for encrypted streams, it has potential to appeal to a width swath of content providers.

Further Reading:

• iPhone Reference Library: HTTP Live Streaming Overview
• The beta of Apple’s Stream Segmenter software, with documentation, can be downloaded from ADC if you have a valid ADC membership.