A huge increase in available data

Though we are fairly sure that the results of all our testing are statistically robust, there is always the slight nagging doubt in the back of your mind that your might have "data-mined" the result and that really you are tricking yourself with the results you seem to have. According to statistical theory, we do have enough data to reach valid conclusions and the chances that we have fooled ourselves are not high - but still the doubt persists.

But out current data provider only has limited amounts of intra-day data available. Ideally we would like a few years worth of data and could then select some three month periods and assess the trading result for these, in comparison with what we have seen over our current testing period. I would like to find a period when markets were really quiet. I would also like, ideally, to have a look at some difficult periods - the time in 2010 associated with the Greek problems, or better still, August 2008 to March 2009 - the peak of the credit crunch

And this weekend, it appears we found what we have been looking for. A cut-down version of esignal, the trading package we used to use at the hedge fund, has extensive intra-day back data now, going back to 2007. I can do as much testing as I like!!

So I have started with a quiet period, early 2010, and hope to cover the entire trading result for 2010. Then move back to the credit crunch period. That would have been a great time to trade as we do now, I can tell that already. I can only manage about a month of data bashing per day, so this is going to be a huge project. Jerome has already indicated that he is not keen to be involved in this (as we would expect). I am rather eager to get started.