By Dr. Giles Nelson, Deputy Chief Technology Officer (CTO), Progress Software

Follow Giles on Twitter

There is a fine line between keeping an eye on something, and snooping.  For years there have been howls of civil liberty-related outrage about the increase in closed circuit television (CCTV) cameras in our cities, phone tapping, and email tracking.  Nosy, interfering neighbours are traditionally reviled, and become the butt of jokes and even lawsuits.

Yet curious neighbours (a T-shirt vendor) spotted a would-be bomb on Times Square in New York City on May 1st (the NYC Metropolitan Transportation Authority’s  slogan -- "if you see something, say something" -- at work!).  Police investigations that included mobile phone tracking and tracing the car's registration nailed down the suspect.  And one of those CCTVs may have identified the bomber's accomplices.  In short, nosiness and surveillance paid off.

When we shift our attention to the financial markets, monitoring and market surveillance are supposed to be done by the regulators and the exchanges. They are the nosy neighbours-cum-police.  Whenever a fraud is committed or a market error costs someone money, the howls of outrage are directed at the regulators or the exchanges for not spotting it sooner.

Thursday, May 6th was a classic case.  The Dow Jones Industrial Average crashed by nearly 1,000 points in the space of 20 minutes, then immediately recovered most of the losses. The day saw the sixth-largest daily share trading volume of all time (in US markets) and the biggest ever intraday point drop for the DJIA.   Fat fingers, algorithms, NYSE's circuit breaker rule, and market manipulation were all blamed, but days later, no one knew what actually caused the so-called 'flash crash' (after 'flash' algorithmic trading programs).

The Regulators' Uphill Battle

Regulators are frequently excoriated for lacking the tools and experience, and for not having mandated enough transparency into the markets, to find out what happened.  But when the same regulators or exchanges try to increase surveillance and monitoring, they are often given short shrift by politicians for interfering with free markets.  Regulators' budgets are stretched tighter and tighter by cost-cutting governments, making it difficult or impossible to keep up with technological advancements.

Their targets, on the other hand, have comparatively unlimited funds to spend on technology.  Banks, hedge funds and buy-side institutions collectively spend billions of dollars per year to perfect algorithmic trading platforms, and to build or tweak algorithms to get the most out of a market.  As such, the innovations on Wall Street and in the City of London have outdistanced the regulators by miles.  It is time for the regulators to play catch-up.

The financial services industry agrees.  A survey conducted by Progress Software during TradeTech 2010 in London revealed that 75% of respondents would rather see transparency created with real-time monitoring systems than with more rules and regulations.  If real-time monitoring can detect trading errors and market abuse or manipulation, increased regulation is not as critical, they reasoned.

Transparency is the key. Regulators in the US and Europe are concerned about the lack of transparency in markets where high frequency algorithmic trading takes place, as well as in dark pools.  Insider trading remains a priority, especially in light of the latest insider-dealing 'ring’ in the UK.   This case saw seven people charged with conspiring to participate in a £2.5m deal by making use of information from banks' confidential printing offices.  While the FSA deployed 143 of its own investigators in the swoop alongside collaboration with the Serious Organised Crime Agency (SOCA), the issue is the time it took to conclude.  The raids were part of an investigation that had begun in late 2007, but not brought to light until 2010.

The traditional forms of detection such as mandatory recording of phone calls, emails and messaging conversations are all useful, of course.  However, to truly address the problem and regain trust and credibility, real time market surveillance technology must be adopted to monitor and detect patterns that indicate potential market abuse, such as insider trading or market manipulation.

The US Commodities Futures Trading Commission and Securities and Exchange Commission have both admitted that they lack the financial, technological and human resources to keep up with market participants.  The events of May 6th highlighted this shortfall.

Keeping Up with High Frequency Trading

The market moves a lot faster than it used to, thanks to algorithmic trading.  What has not kept pace is monitoring high-speed trading.  Not using monitoring technology is akin to flying an airplane that has no safety devices.  If a pilot goes rogue and decides to crash the plane, or has a heart attack, the plane would go down if there were no safety devices and procedures in place.  Exchanges, ECNs, brokers, traders and regulators all must take an intelligent approach to monitoring and surveillance in order to prevent rogue trades and fat fingers.

The detection of abusive patterns must happen in real time, before any suspicious behaviour has a chance to move the market.  This approach should be taken on board not just by the regulators, but by the industry as a whole.  Only then can it be one step ahead of market abuse and trading errors that cause a meltdown (or up).

Technology can't solve all of these problems, but it can help to give much more market transparency.  To restore confidence in capital markets, organisations involved in trading need to have a much more accurate, real-time view on what's going on.  In this way, issues can be prevented or at least identified much more quickly. 

In the case of the May 6th meltdown, only the NYSE had a mandate to slow down trading - or a circuit breaker - so that participants could take a deep breath and try to figure out what was happening.  Other, fully electronic, trading destinations kept plugging away as liquidity dried up. This anomaly probably contributed to the speed at which the market dropped.

One of the first things to be agreed by almost all of the exchanges and ECNs after the so-called 'flash crash' was to coordinate market-wide circuit breakers. There is already technology available that can automate these circuit breakers, and alert clients at the same time. The question begging to be asked, is why is it not in use already?

If the initial trigger to the 'flash crash' had been a fat finger trading error, or market manipulation, the trading organization where it originated should have caught it before it hit the market.  Again, there is technology available that enables trading organisations to monitor trading behavior, whether to ensure compliance with risk limits, or to spot abusive patterns. Brokers, exchanges and regulators are particularly relevant, but buy-side organisations can also benefit from it.

Other solutions could include electronic 'tagging' of orders by source.  If the order is from an algorithm it should be tagged as such, if from a broker - also tagged.  This would enable exchanges or regulators to suspend orders from suspect sources and make forensic analysis after an event much easier.

Simple, easy to implement rules are also key. There should be no excuse for firms not to implement them.  Better visibility is also crucial. The TradeTech survey showed that more than half of respondents would support regulators having open, real-time access to information about the firm's trading activity.  That is a pretty strong sign that the industry is willing to open up, rather than be subjected to draconian new rules.