How accurate is that estimate?

(All the examples quoted in this article are over five years old—to minimise the risks to confidentiality.)

Internally we publish many numbers—for example, forecasts of market sizes and estimates of competitor revenues. But in most cases, we don't say how accurate those numbers are likely to be. Arguably we should publish, with each estimate of a number, an upper and lower bound, which define an interval highly likely to contain the true value.

Here is a sample of the maximum errors observed during 2002-2003:

Market Forecast for the year (made just before its start)

• Product Category (e.g. 'External Disk')
  
  
  
  • At the Country level: +/- 27%
    
  • At the EMEA level: +/- 25%
  
  
  
  
• Product Sub-category (e.g. a Server price-band)
  
  
  
  • At the Country level: +/- 34%
    
  • At the EMEA level: +/- 31%

Instances:

• For the year 2002, our forecast in 2002 was that 'External DASD' in the UK would reach $1,226m. But this 2002 estimate was corrected to $895m in 2003.
  
• For the year 2002, our 2002 forecast was that 'External Disk' in EMEA would reach $7,419m. But this 2002 estimate was corrected to $5,551m in 2003.
  
• For the year 2002, the 2002 forecast of 'Servers $25K to $100K' in the UK was $305m. But this 2002 estimate was corrected to $410m in 2003.
  
• For the year 2002, our 2002 forecast of 'Thin Servers' in EMEA was $319m. But this estimate was corrected to $219m in 2003.

A vendor's revenues for a quarter in the recent past

• Product Category (e.g. 'Servers')
  
  
  
  • At the Country level: +/- 47%

Instance:

• For 2Q02, one analyst firm estimated that IBM's server revenues in the UK were $93m. But according to internal accounts, our revenues were $63m.

It's worth commending the company for changing its numbers when they are known to be incorrect. One analyst firm, on the other hand, has historically not changed figures published in its server tracker when shown to be wrong. Instead it will adjust future figures in order to make the running total correct. So if any of the 2Q08 numbers turn out to be wrong, for example, it will adjust the 3Q08 figures to cancel out the error.

I suppose this approach has the advantage that, with each new release, you only need add the new column of data to your spreadsheet—i.e. you never need worry about previous quarters changing—but it does make you worry about the true accuracy of any individual quarter's estimates.