The "data bash" is finished

For the past three months I have been working on a huge data processing project in which a number of variations of our current trading strategy have been thoroughly "blind" tested against six months of old data (from around January 2010 onwards). The aim of this was to produce a final selection from the various methids we have been using with the view of then to trade just this going forward.

The trading systems are too complex to programme into a standard testing package and so each day has to be processed manually. It takes me about 20 minutes to process one day for one market and I have been able to process about 5 days per day for each market we have been looking at. But there have also been many diversions, off into other systems, or other models, where we seemed to have seen something of interest only to have the effect disappear as we process more days of data.

In the past week I have been finally coming to the end of this process. On Monday I still had 56 day to go, by Wednesday it was down to 27, then to 15 by Friday and then at around 10:00 on Sunday morning the last day was done - August 15th 2011, joining the old data from the blind test to the data we have processed in real time. Yet from this process has come the broad family of systems which we will use going forward. No longer will I get a message asking me something like "have we looked at what happends when . . . . .". Instead the focus is on the actually process of complying with the exact models. No more data bashing . . . .

I suspect that last statement will be wrong, but at least i am sure that the issues will be quite small ones to look at - for instance, our methodology for taking our first trades of the day needs looking at in more detail. And so it will go on . . .

But the broad thrust is now done. I am almost overcome with joy!


The last day processed - August 15th 2011