February 23, 2007

10 best ways to lie with metrics

Posted in management, Metrics at 2:31 pm by scottmaxwell

We spend a lot of time with our portfolio companies trying to identify, manage, and track meaningful metrics that, when managed well over long periods of time, will assist in the building of a great company. ( I had a summary post last year on the topic.) As I meet with companies, read press releases, and hear presentations at companies, I find that many would rather present “feel good” metrics rather than metrics that give an accurate picture on the state of their company, including all of the positive and negative trends (or, more importantly, do the hard work of managing to get strong, positive trends for the key drivers of their business). If you want to make your metrics “feel good” (rather than “key drivers”), this post is for you:

10 best ways to lie with metrics

  1. Gather a lot of metrics and then only present the metrics that are positive. If you gather enough data, you will always have metrics that show positive trends. One good way of finding positive metrics is to look for the negative statistics and then look for the bright side. For example, if you lose a few great employees, focus on your costs being below budget. (When you show different statistics over time, you should also congratulate yourself on adjusting the metrics to new ones that are better indicators of your business…this will demonstrate that you are a “learning organization.”)
  2. Gather and present some metrics that are really easy to manage (this will allow you to consistently show some metrics over time that can have really good long term trends). For example, show “leads” or traffic, as you can always come up with ways for buying more (but be careful not to present lead quality, cost of leads, or ROI, as these are more difficult to manage to).
  3. When you are presenting, use many metrics. This will impress the audience that you are truly on top of your business.
  4. Use extremely precise numbers. For example, don’t say “about 100,000 downloads” or “growing roughly 100%,” say “101,243 downloads” or “growing at 102.4%.” The more precision that you use, the more your audience will think that you are a brilliant manager.
  5. Present your metrics quickly. The more quickly you go through your metrics, the less time your audience will have to consider the validity of your metrics.
  6. If your audience asks for some metrics that don’t make you look good, tell your audience that “we don’t break out those numbers” or “our systems don’t allow us to get that data” or “we put our resources against product development and marketing, so we have a very lean staff that doesn’t allow us to gather those numbers” or “I will get back to you with that information.”
  7. Hire a graphic designer to turn your metrics into really good looking charts (3D, movement, many colors, graphics, etc.). Your audience will focus more of their attention on the quality of your slides rather than the data you are presenting. The net effect will be very positive.
  8. Keep some of the metrics “in your pocket” so that you can share some of the metrics “off the cuff.” This really impresses the audience. A good way of doing this is leaving the most recent data out of your presentation and then sharing the most recent data orally.
  9. Prepare each of your functional heads with some of the more detailed data that supports your “feel good” metrics so that during your presentation you can say ” <name of functional head> would you like to expand on this?” the more your team members seem to be behind your metrics, the more believable the data.
  10. Never, ever, ever educate your audience on the small number of key drivers for your business. The simpler and clearer that you are, the easier the audience will grasp and remember how to think about your business and the more frequently they will ask for the same data. This education will completely destroy your future ability to keep your metrics “feel good!”

Note: You can get some additional great ideas on how to obfuscate the truth from the book How to Lie with Statistics, a classic!

About these ads

9 Comments »

  1. Jorge Camoes said,

    Let me focus on a specific set of those “feel good” metrics. Since the classic study by Beattie and Jones “The use and abuse of graphs in annual reports”, back in 1992, I don’t see much improvement, quite the opposite. Annual reports became marketing communication tools and you must dig deep to find useful information. They are the best source for bad examples (I know I can always trust them for that…).

  2. rick said,

    Haha! Great stuff!

    I once worked with a chip designer whose signature technique when questioned about the promised performance of his silicon was to throw a long string of numbers on the whiteboard in a numerator and another long string in the denominator, a few powers of ten here and there, then with a flourish cross out zeros top and bottom – tick tick tick tick tick poof! 14.5 Gbps. or what ever number he wanted it to say.

    No VC ever, even once, said, “wait…is that MIPS or BIPS? and what if your assumption on required instructions is wrong?” They were too busy tripping over their checkbooks giving him money.

  3. Kerry said,

    Excellent post.

    The general problem here is selectively using data to reinforce what you want to see. Instead of using data to discover what you need to see.

    This issue comes up not just in the definition of metrics but also in the definition of markets.

    More discussion on this market definition issue is here:

    http://simplematterofsoftware.typepad.com/weblog/2007/02/fractal_geometr.html

  4. scottmaxwell said,

    Kerry, thanks for the comment. I like the fractal analogy (and the picture!).
    S

  5. justindavies said,

    Scott, great post.

    Kerry (whom commented above) drew me to it after reading one of mine at http://justindavies.wordpress.com/2007/02/13/talk-to-me-in-numbers-applying-metrics-to-business-success/

    A good read on the topic is “Smart Business Metrics” by Phelps.

  6. Shefaly said,

    I am not sure this is news to anybody who has ever:

    1. written a short thesis for a course or published a paper in a peer-review journal (ever wonder why nobody ever reports that data contradicted their hypothesis?)

    2. asked a market research firm to supply numbers to justify a decision already taken through an executive’s gut-feel (!) (common in CPG industry and many others too)

    3. applied for a visa to travel to weird places, where the more the data you supply, the more likely you are to get a visa (best story told by an American UN worker, resident in Germany, wanting to go to Kazakhstan for 1-day – on work of course – who had to supply details of her being married and the proof that her husband was indeed legally divorced)

    However that somebody has written a book is a different league altogether!

  7. oncculm said,

    bOrZRH urfqamfrxvep, [url=http://swvyfeallvzi.com/]swvyfeallvzi[/url], [link=http://qpmnsqxrwzfh.com/]qpmnsqxrwzfh[/link], http://cxmprrnhkdbj.com/

  8. i dont get bout the metrics.. what are u pertaining to???

  9. i dont get it??


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: