Wednesday, October 26, 2005

Profit factor walk-through in my backtesting software

I've talked a bit recently about the java software I've been developing for backtesting and automating trades so I thought it would be a good idea to outline some of my recent progress here with companion images.

One of the primary statistics that I'm concerned with when looking at the results of a trading system backtest is the Profit Factor. The Profit Factor is computed as follows:

(PW * AW) / (PL * AL)

Where

  • PW = Probability of a trade being a winning trade

  • AW = Average win size

  • PL = Probability of a trade being a losing trade

  • AL = Average loss size



Profit Factor basically describes the historic profitability of a series of trades. For example, a series of trades with a profit factor of 1.0 is break-even. Another more specific example might be a system that generates trades with a 50% chance of losing and a 50% chance of winning. For this system, the average win size is 3 times the size of the average loss. So:


  • PW = 0.5

  • AW = 3

  • PL = 0.5

  • AL = 1



And when we plug it into the formula, it looks like this:

(0.5 * 3) / (0.5 * 1) = 3

As long as the numbers remain the same in the future, we can expect this hypothetical system with a profit factor of 3 to be very profitable. In fact, the more frequently we trade it, the larger we would expect our compound returns to be. The reason for this can be simply explained by considering two casinos, each with a house edge of 55%. Obviously the higher volume casino will have larger earnings over an identical timeframe.

When backtesting a system, it would be useful to know what the best stoploss and profit target values are for that system, so I wrote a module that computes the profit factor of every combination of stoploss and profit target, and outputs the data to excel where I can graph it easily. Here is the graph for one of the systems I've began to develop (click to enlarge):




One of the major reasons graphs like this are important is so that you can make sure you aren't over-fitting your system to the data set you are testing it on. If you simply have the computer find the very best profit factor out of the batch, you may end up with system parameters that merely take advantage of a coincidental price sequence that never repeats and won't be profitable in the future. Parameter clusters that form broad hills tend to be more robust and reproducible in the future. In the graph above, there are two significant hills, with a spike at the top of the one on the left. I wouldn't necessarily expect the parameters coinciding with the peak on the left to reproduce quite that good of results in the future, but the fact that it is on top of a hill means that it is still a favorable parameter set, and a good candidate to look further in to.

So here are the backtested trade results from one of the parameter sets at the peak of the left hill (click to enlarge):




As you can see in the totals at the top, the longs are more profitable trades to take. That is because the market is trending up in the 3 year data sample that this backtest covers! When I filter out the short trades, only analyzing long trades, the profit factor of the system goes from 3.1250 to 9.6654! However, the long-only system is less profitable over the same three year timespan because it is taking only 28 trades with a profit factor of 9.6, whereas the bi-directional system has 56 trades with a profit factor of 3.1 -- slightly more profitable, and makes money in a wider range of market conditions. So a possible future direction for this simple system of mine might be to find a good way of identifying long-term market uptrends and downtrends as an indicator for overweighting long or short trades.

6 Comments:

At 11:23 AM, Blogger Michael Taylor said...

Great job with your backtesting software. Profit factor is a great measurement tool. I like the way you graphically present your results. Noticed you use Excel. Have you ever tried using R? That's what I'm currently working on in trying to build a simulation trade idea analysis tool. That and a bit of Python. Not sure if there's a Java interface to R like there is with Python though.

Question for you. In addition to profit factor...I also look at the profit mean, standard deviation, max drawdown, and number of trades. Know of any formulas that combine these figures into one?

The sharp ratio will handle mean/sd. There's a modified sharp...but I can't find the exact formula.
I've been experimenting with using mean/(sd*sqrt(#trades))...but just really starting that experiment and formula manipulation. And that still doesn't take into account max drawdown.

Anyway, if you have come across or use anything like what I've mentioned...please share! :)

Take care,

MT

 
At 11:47 PM, Blogger jwu said...

wow....

 
At 3:52 PM, Blogger Michael Taylor said...

I've found a formula that takes into account max drawdown: Sterling Ratio. It's your profits/max drawdown. The higher the ratio the better. Of course, this still doesn't incorporate everything into one formula. I guess, formulas are like progamming languages...someone is always coming up with the One End all be All and still need All to get the End. :)

By the way, one thing regarding the profit factor...very important to include commission costs and slippage into your past trades in order to come up with a realistic result.

And if the past trades analyzed are not actual trades but backtested results...a filter to remove excessive positive tails is required. Which may prove difficult if your system is designed to capture fat tails. :)

In my mean reverting systems I've found profits often 3 standard deviations away from the mean. Of course, this isn't realistic and the underlying data is the culprit.

To combat this I leave negative tails in the historical trades and remove all positive tails 2 standard deviations or more. While this isn't the perfect solution...it helps get closer to what actual results may be (borrowed this method from acrary).

After the positive tails filter and accounting for the house edge (commissions&slippage)...profit factor is a great tool to use in evaluation.

Take care and good luck with your system development.

MT

 
At 4:46 PM, Blogger jontait said...

I'll probably code the sterling ratio into my stats module this week. It seems like a variation of the max % drawdown that Acrary computes. I believe he described it as (drawdown bottom / equity peak). I bet this starts to look pretty nasty when a system is on its way out...

Regarding profits that are 3 or more standard deviations away from the mean, filtering them out is probably a good idea, but for systems with stop-losses and targets as parameters, you aren't going to see the wild profits that need to be filtered out.

 
At 10:27 PM, Blogger Michael Taylor said...

Actually, that's the crazy thing...the first time I noticed the 3 standard dev plus moves was in a profit limit/stop loss system. The system I'm referring to had a 4% profit target and I found several gains over 50%! And also several losses below -50%.

Of course, once I looked at the data it was easy to see what was happening...the profit targets and stop losses will cap those big moves intraday...but not the close to open.

After digging through those anomolies...I found several were indeed actual price jumps. But, several others were the result of junk in the data. Now, I'm referring to stock data. As you can imagine, with over 3,000 stocks on the Nasdaq Exchange and going back 3 to 4 years in backtests...data issues can compound pretty quickly.

There's some code in the Wealth-Lab forums by fundtimer that cleans up data issues apparently "good enough"...that I need to try. One of those things that I'm afraid to turn on the lights in fear of all the roaches that might come spilling out. :)

MT

 
At 11:33 PM, Blogger jontait said...

Michael,

Nice catch on the Close to Open gap. As I'm handling it now, gaps through limits or stops are filled near the open price, but I can see how bad data for the open price could really skew the supposed profitability of a system. I haven't run into this problem yet, but it is better to be safe than sorry, so I will add that filter soon.

-Jon

 

Post a Comment

<< Home