Statistics Can Deceive When It Comes to Small-Business Jobs

Surveys, truth, and statistics

Tim Berry

So I'd say that I'm drowning in information lately, except that 1) you wouldn't be very sympathetic, because you are too; and 2) it's not information that oftenit's often half information or fake information, or merely good intentions.

Surveys and Statistics Vs. Truth

We forget so easily that when a survey shows X percent of respondents prefer Y over Z, that doesn't make it true. That means only that this survey, with this group of respondents, came out that way.

Maybe you took statistics in college, maybe not; and maybe you did but (like me) you've forgotten most of it. Remember this, though: If it doesn't start with a random sample, it's not likely to mean that much.

Real-world example: Last month, I found two competing reports on small-business jobs. One had jobs decreasing by a quarter million or so; the other had jobs increasing by a very small percent. Was one wrong? No. They had different samples. The more optimistic one surveyed customers of a Web payroll application, who probably tended to be smaller, and newer, and more in tune with high technology. Maybe.

One thing I remember from my several years in a market research company is that it's very hard to get a truly random sample.

Another thing I remember from those days is that it's also really hard to create survey questions that don't distort the answers. There are unintentionally leading questions throughout most of the surveys I see. Is the real answer the truth or the option that makes the respondents feel best about themselves as they give their answers? For example, all those surveys that ask people what they plan to buy in the next six months: What's the relationship between what you plan to buy and what you actually buy?

Focus Groups Out of Focus

I've been involved in a lot of focus groups gone wrong. A focus group has to have truly representative people to work, but most of them end up with volunteers who frequently go to focus groups. And they often end up reflecting the views of one particularly charismatic or articulate leader in the group, rather than the views of the whole group. And meanwhile, back in the real world, people don't shop in a group. They shop alone.

The Danger of a Bad Steering Wheel

What's particularly dangerous about unintentionally bad research is that it often takes on a special status, as if the research results are higher truth, or absolute truth, instead of just another guess. A manager's own judgment gets subjugated to the power of "what the research shows is . . . " and, way too often, bad decisions are made.

My recommendation? Do the research. But don't believe it unquestioningly. Stay skeptical. Make the search an input, but not the only answer.

Tim Berry is president and founder of Palo Alto Software, founder of, and a co f ounder of Borland International. He teaches starting a business at the University of Oregon. He is author of books and software including Business Plan Pro, published by Palo Alto Software, and The Plan-As-You-Go Business Plan, published by Entrepreneur Press. He has a Stanford M.B.A. degree and degrees with honors from the University of Oregon and the University of Notre Dame. He blogs at Planning Startup Stories and Up and Running.

small business

You Might Also Like