Samstag, 26. September 2009

99% Backtest Quality with MT4 and TickData

I got a usefull text from a france forum, the author is BIRT, so I saved and extented the text here:

Aim of this topic is, to achieve a real reliable backtest, some known problems of backtests with MT4 are:
- data you get from most MT4 broker are very, very poor and "lie" (try it out!)
- so not able to use them for scalping EAs
- the problem of not able to test spread and commission is still here!

BIRT words:
As the title says, I prepared a set of tools that enable testing with 99% modeling quality. I also downloaded and processed the tick data from Dukascopy (until 02.Sept.2009) (it's about 400 MB compressed, ~4.5GB uncompressed).

If you only want the scripts and the loader, get them here:

Download - 99_percent_MT4_build225_backtesting.zip

If you also want CSV data:
see below

How to get 99% modeling quality when backtesting with metatrader 4 build 225
============================================================================

Here are the steps you must perform:

1. Get some good tick data.
2. Convert the tick data into HST & FXT.
3. Copy the HST & FXT files in the correct folders.
4. Run MT4 with a loader to allow loading your custom FXT file, then start backtesting.

I'll describe each step in detail.

1. When it comes to the tick data, you shouldn't have a big problem. Luckily for you, I downloaded all the Dukascopy tick data, parsed it and saved it as a CSV that you most likely found in the same archive as this file. Should you want to update the CSV data, you will find the scripts to do it in the "optional tools\php\Dukascopy tick data" folder. If you want to play with the Gain Capital tick data (really bad data quality!!), I also supplied some tools to help you with that. I'm afraid that I won't offer any instructions on running the scripts besides the fact that you must have PHP installed to run them. All the PHP scripts provided are meant to be run in command line (e.g. putting them on a webserver won't do much).

2. Here's what you must do to convert the above CSV data to HST & FXT:

- copy the files from the "convert" folder into your MT4 folder;
- move the tick data file (the CSV file) to experts\files;
- open up a chart for the pair that you have data for (if you have EURUSD.csv you MUST open an EURUSD chart);
- select the timeframe that you wish to generate the FXT for (if you want to backtest on H1, then select H1 as the chart timeframe);
- make sure you are connected to the server, then attach the Dukascopy2FXT script to your chart;
- in the parameters section set the ExtCsvFile to whatever your CSV filename is (e.g. EURUSD.csv; the file must be in the "experts\files" folder at this point) and if you didn't previously generate HST files for the pair, you have to set ExtCreateHst to true - this step only has to be performed once.
- go get some coffee while the system processes, this will take a very long time even on high end PCs.

For the masochists, some scripts are provided for use with Gain Capital tick data. You will not need these scripts unless you downloaded and parsed the Gain Capital tick data yourself with the scripts described at step 1 (which no sane person would have reason to do, given the availability of the much better Dukascopy tick data).

3. After your script at 2. has finished running, you will end up with a lot of files in your "experts\files" folder. At this point, exit the MT4 terminal then proceed to moving all .HST files from "experts\files" to "history\your_server_name". Pay very close attention if you have multiple server directories in your "history" folder - you will have to move them into the one that's correct for active account!
Note: unless you modify the CSV, the HST files only need to be generated and moved once.

After moving the HST files, move the generated FXT file from "experts\files" to "tester\history".

4. To use the files you created in a backtest, you will have to use the FXT loader, found in the "loader" folder. Note: this loader works only on MT4 build 225. You just have to copy it into the folder of your MT4 client and instead of running terminal.exe, run "fxt loader.exe". If you're worried about running executables and you can do a manual patch, see the bottom of this file for the offsets and changes.

What the loader does is disabling MT4's default behavior to regenerate the backtest files every time you start a new test - so, if a file is already there, a new one will not be generated. By placing the FXT file you generated with the script in the "tester\history" folder, you ensure that it's the file it is going to use for your backtest; of course, the selected backtest currency pair and timeframe must coincide with what you generated the FXT with.

How to verify it worked? Easy, if your test is running and the file in tester\history was not recreated, it means everything is ok.

Note: the GMT offset of the data appears to be 0 all around the year. Correct me if I'm wrong on this one.

Bonus: if you can run PHP scripts, I also packed my script that merges backtest results, thus simulating running an EA on multiple pairs or possibly more than an EA at a time.

All the PHP scripts and the loader were written by me (birt). Some MQL files were not originally written by me, but I modified them to work with the data & new MT4.

Credits for these:
- GainData2fxt.mq4, unknown author
- FXTHeader.mqh, stringo@codebase.mql4.com
Note: my Dukascopy2FXT.mq4 and GainData2hst.mq4 are based on the above two files.

If don't want to use the "fx loader.exe" you could manual set the following offsets with an hexeditor, terminal.exe patch offsets (hex offset -> old byte -> new byte):

1402FC -> EE -> 00
1402FD -> 02 -> 00
14041B -> 9D -> 00
14041C -> 01 -> 00
140521 -> 97 -> 00

End of BIRT words...

Some additional infos:

To run PHP scripts under Windows, use this installation tool WAMP -> http://www.wampserver.com/en/index.php
There is a preconfigured Apache server with PHP and MySQL support installation.
So you can use this either to download tickdata from Dukascopy and/or use the MySQL server to use it for logging your own tickdata from a reliable broker.

Here are the links for the tickdata from Dukascopy:

EURUSD:
Download - EURUSD-tickdata-csv-dukascopy.part01.rar
Download - EURUSD-tickdata-csv-dukascopy.part02.rar
Download - EURUSD-tickdata-csv-dukascopy.part03.rar
Download - EURUSD-tickdata-csv-dukascopy.part04.rar

EURCHF:
Download - EURCHF-tickdata-csv-dukascopy.part01.rar
Download - EURCHF-tickdata-csv-dukascopy.part02.rar
Download - EURCHF-tickdata-csv-dukascopy.part03.rar

EURGBP:
Download - EURGBP-tickdata-csv-dukascopy.part01.rar
Download - EURGBP-tickdata-csv-dukascopy.part02.rar
Download - EURGBP-tickdata-csv-dukascopy.part03.rar

AUDNZD:
Download - AUDNZD.rar

EURNOK:
Download - EURNOK-tickdata-csv-dukascopy.rar

EURSEK:
Download - EURSEK-tickdata-csv-dukascopy.rar

CADJPY:
Download - CADJPY-tickdata-csv-dukascopy.part01.rar
Download - CADJPY-tickdata-csv-dukascopy.part02.rar
Download - CADJPY-tickdata-csv-dukascopy.part03.rar

GBPUSD:
Download - GBPUSD-tickdata-csv-dukascopy.part01.rar
Download - GBPUSD-tickdata-csv-dukascopy.part02.rar
Download - GBPUSD-tickdata-csv-dukascopy.part03.rar
Download - GBPUSD-tickdata-csv-dukascopy.part04.rar

GBPCHF:
Download - GBPCHF-tickdata-csv-dukascopy.part01.rar
Download - GBPCHF-tickdata-csv-dukascopy.part02.rar
Download - GBPCHF-tickdata-csv-dukascopy.part03.rar
Download - GBPCHF-tickdata-csv-dukascopy.part04.rar

AUDJPY:
Download - AUDJPY-tickdata-csv-dukascopy.part01.rar
Download - AUDJPY-tickdata-csv-dukascopy.part02.rar
Download - AUDJPY-tickdata-csv-dukascopy.part03.rar
Download - AUDJPY-tickdata-csv-dukascopy.part04.rar