Objective measures of performance

[Maranello]

Are PwC ratings the best statistical measure of performance? Please discuss . . . (and no word limit, so its not an A level essay)

Should we rely more on averages etc?

On a broader note, are there other objective measures of performance that we can consider? Or is it all nonsense anyway and we are much better off with subjective debate?

Will post my thoughts as the thread develops; to kick-start the discussion, some of the posts by Goatman, Milo and yours truly in this thread might get the thinking juices flowing :-)

[Milo]

My major issue with PWC is that a player's match rating is calculated using (amongst other things) combinations of the following criteria

1) How they performed
2) How everyone else performed
3) Who played in the opposition (and what their existing rating is)

Now (1) is clearly the major indicator here. Where it fails for me is that (2) sometimes can shed light on how an individual has performed but more often than not it is (and should be) irrelevant. The game is at a different stage, the bowlers may be different, the pitch different. First innings and fourth innings performances are often totally different and should be treated as such.

(3) is a big problem. A bowler can get given credit for taking wickets in a game that includes some heavily rated players (eg Sehwag and Dravid of India). He may not even bowl to them but statistically PWC 'objectively' rates his performance as if he did. It is no coincidence that Harmison's rise to the top has been as a result of bowling very well against a team that contained the number 1 rated player, Lara, at the time. This is regardless of whether he took Lara's wicket or not (in fact he only dismissed him 4 times in 15 innings). One bowler could dismiss the top 5 whilst the other could get 6-10, they could both end up with 5-60 and they get the same rating. That is a serious flaw of this objectivity.

[Maranello]

How the Ratings are Calculated

1. The weighted average

Traditional averages reflect players' standings over their whole career. The Ratings, on the other hand, put more emphasis on what the player has done in his more recent matches. For example, suppose there are two twin brothers, Steve and Mark, both of these have an average of 50.

Steve 0 10 20 30 40 50 60 70 80 90 100

Mark 100 90 80 70 60 50 40 30 20 10 0

However, in this example the trend in Steve's scores is clearly upwards, while Mark's is the reverse. Using a mathematical principle known as exponential decay, you can produce a weighted average that shows up a player's recent form, and this weighted average is the basis of the PwC Ratings.

In the Ratings formula, each innings counts as 4% less important as you go back in the player's career. In our example above, Steve's weighted average would be about 55 while Mark's would be 45 - both good levels, but Steve would have a higher rating because of his better recent form.

This weighted average explains why players with poor career averages sometimes have high Ratings (and vice versa).

Before calculating the weighted average, the Ratings program evaluates each individual performance, taking into account factors such as the strength of the opposition. These factors differ for the Test and One-Day Ratings.

2. Points

Before the player's weighted average is printed out, it is converted into points, on a scale of 0 to 1000. For example, for a batsman, a weighted average of 60 will give him about 850 points. In both the Test and One-Day Ratings, 900 points indicates outstanding, world beating form and is rarely reached by any player. A player with 700 points is usually inside the world top ten.

3. Changes in points

Like the stock market, a player's Rating can move down as well as up. The higher a player's Rating, the faster it will fall if he loses form. Changes of 100 points after a single game are possible, but usually a player's points change by less than 30 points after a Test match and less than 15 points after a One-Day International.

4. New players

Batsmen and bowlers have to prove themselves before they can make the world top ten. For this reason, a player is only awarded a percentage of his points until he has fully qualified. A Test player reaches his full Rating after playing 40 innings or taking 100 wickets, although an outstanding player can reach the top ten after only a couple of series. Brett Lee of Australia climbed to sixth in the bowling after only six Test matches, but this is a rare achievement. A One-Day player needs to play about 40 matches to get a full Rating, but again, within a year of his first game a good player can make the top 20.

5. All-rounders

We publish an index of the current top five leading all-rounders. This is calculated by multiplying the batting and bowling points together. The reason for multiplying rather than adding is that by definition to be an all-rounder a player has to be selected for both his batting and bowling. Glenn McGrath (900 bowling/100 batting) is not picked for his batting, whereas Craig White (350/450 at the time of writing) is picked for both. Multiplying points together will favour all-round ability over specialist ability. Incidentally, there is a case for saying the all rounder index should actually be the square root of the number we produce, but since this wouldn't affect the order we avoid this extra complication.

6. Missing a game

A player who misses a Test match for his country, for whatever reason, loses 1% of his points, and he loses ½ % of his points if he misses a One-Day match.


Factors used in the Test Match Ratings
The method of calculating the Ratings has been considerably fine-tuned since they were first launched in 1987, but the basic system has remained the same.

The two most important factors taken into account by the Test Ratings are the strength of the opposition and the level of run-scoring in the match. The value of runs scored and wickets taken are adjusted to take account of these factors.

1. Opposition strength

The Ratings adjust the value of a batsman's score depending on the strength of the opposition's bowling over the whole innings (as measured by their Ratings). If McGrath and Lee bowled most of the overs, the batsmen would get a credit of perhaps 10% on their runs, so an innings of 50 might be upgraded to be worth 55. However, if Australia's occasional bowlers bowled most of the overs, the batsmen might lose 20% or more of their runs, valuing an innings of 50 as being worth only 40 runs in normal conditions. (The exact changes depend on the Ratings of the bowlers at the time.)

Meanwhile, a bowler who dismisses a top batsman in form, such as Sachin Tendulkar, might get up to three times as much credit as he gets for dismissing a tailender like Pommie Mbangwa.

2. Level of run-scoring (wrongly known as the "pitch factor")

If both teams make 600 runs, run-scoring was clearly easy in the match (perhaps because of a flat pitch, but no measurement is made of the pitch itself). The Ratings discount runs made in high-scoring conditions, and give credit to runs made in very low scoring matches. Likewise, a bowler who takes 0 for 100 will be penalised less in a high-scoring game than a low-scoring one.

3. Result of the match

The Ratings favour players who perform well when it counts. There is a bonus for players who perform well in victories for their team. A player who makes a high score or takes a lot of wickets in a victory gets a bonus, but he gets no bonus if he makes a low score or doesn't take a wicket. Steve Waugh has benefited from this bonus more than any other batsman (Australian or otherwise) in recent years.

4. Not outs

Not outs can be a distorting factor in conventional averages. In the Test Ratings, a player gets a bonus for a not out, but this diminishes as his score increases. A batsman only gets a little more credit for making 300 not out than for making 300 and out.

[Maranello]

I have temporarily included PwC's description of the ratings system above to facilitate discussion. This is for reference at this stage, and will be removed in line with Forum Guidelines on not including whole articles from other sources.

The above can be accessed on the Ratings website here

[flanflinger]

Originally Posted by Maranello

Meanwhile, a bowler who dismisses a top batsman in form, such as Sachin Tendulkar, might get up to three times as much credit as he gets for dismissing a tailender like Pommie Mbangwa.

Milo,

the key to your point is here. The ratings do take into account who they got out.

The PWC ratings are good at judging who is currently in form, but I feel that over a career you need to include averages. I know they don't take into account the fact that bowling out Zimbabwe is easier than Australia, nut now with the fact that all-teams play each other so often, I think that for a judge of a career the avearge still wins.

[Maranello]

Good comments FF. On your second point, the PwC ratings are best used for finding out who the best is right now, or at any given moment in time (eg who was the best in the world immediately after the Headingley Test, 1981) - for this, I believe they have no substitute.

However, they are not a measure of how good a player was over their whole career, without some tweaking anyway. The PwC "best ever" rating is a measure of performance at one point in time.

So if someone (eg A.N.Other) has one truly outstanding series, they can shoot up 30 or 50 points in the ratings and could end up with a score over 900 in that week's ratings. Statistically, this does not necessarily mean that Mr Other is better over his whole career than someone whose best was only 875.

To find out who was the best over entire careers, we either use averages, which as you point out, are skewed - are runs against Zim equal to runs against Warne & McGrath? Is getting SRT out in his pomp the same as getting Walsh's wicket? Averages suffer from enormous distortions as a result (the other distortions are listed in my post above).

In my view, the best measure of "class" over a career is the weighted average of ratings for the players over each match they played. As a "quick and dirty" one can see what area is under each player's ratings graphs, compute a weighted average per Test and then compare with other other players.

This way, we will compare the weighted average of ratings over their career, and not just the peak they attained. As you point out, the one peak is all very well but it does not tell us of all the possible troughs!

[Milo]

I'd argue whether they can even tell you about he best right now. How can that judgement be made about Harmison when he missed the entire series in SL (the Malthusian checks on a fast bowler, as it were) and hasn't even played against Australia in ages. It tells you nothing other than he bowled very well against West Indies and New Zealand.

Best at the moment? How can we possibly compare with what McGrath and Gillespie are doing on the other side of the world.

[Zainub]

It's certainly not non sense but it's not a joke either. All the theory and mathematics asside my gut feeling has been that the ratings are more of an indicator of present form than anything else. If I was trying to evaluate how good a player has been over say 10-15 years I'd not neccessirily give them the same weightage. They are also some what effective for carrying out comparisions.

Having said all that, what I rely on most is my gut feeling, because (I know I say this so often but it really is so true) statistics reveal just as much as the conceal. No averages or ratings will tell you how a certain bowler has bowled a fiery spell of 10 overs, full of pace, and relentlessly accurate where he's beaten the bat consistently, and still unfortunately not got wicket, perhaps even got catches dropped of his bowlings...from a batsmen's perspective too I don't think any rating will ever tell you just how hard sometimes you have to work to get the mere 40 or 60 odd. Sanjay Manjeraker often tells viewers from his position in the comentary box that how he still ranks his innings of 40 odd against WI still to him seems the toughest innings of his entire career, Inzi will tell you just how important that 100 against lowly Bangladesh was last year....this little things do influence my judgements quite often.