The dirty little secret of employee surveys

They're junk science

January 16, 2013

CALGARY, AB, Jan. 16, 2013/ Troy Media/ – Each year, hundreds of thousands of people, employees from the executive suite to the front lines, are asked to complete an employee survey. Almost a billion dollars worth of engagement surveys are sold in North America each year.

The case for improving employee engagement is impressive. The Hay Group, a major survey provider, says that; “high levels of employee engagement can boost revenue growth by up to two and a half times.” Aon Hewitt notes: “20 per cent of the organization’s (most engaged) employees create 80 per cent of the value” and Gallup, the polling company, claims that “actively disengaged employees . . . cost the American economy up to $350 billion per year in lost productivity“.

Numbers like these get attention. It’s worth spending a billion to get 350 billion back. But is any of it working? Is business getting a return on the investment?

The increasingly obvious answer is ‘No’. Worse, these surveys are likely doing more harm than good.

The first cracks in the employee engagement hype appeared in Peter Hutton’s What Are Your Staff Trying to Tell You?, published in 2008. It concluded that, whatever it was, it wasn’t what the employee survey said it was. Last year, Business of HR blogger John Hollon, called engagement surveys a mindless waste producing terrible results.

Now we know why. According to “Understanding Employee Engagement and Trust; The New Math of Engagement Surveys” to be published next week in the American Society for Quality’s, Journal of Quality and Participation; “The dirty little secret of employee engagement surveys is that they’re largely junk science – placing the marketing objective of telling and selling a good story above the practical and ethical objective of telling the truth.” Statistical methods are misused, corrupting survey results, while providing an air of scientific legitimacy.

Just how bad is it? The statistical methods used to identify important findings in engagement surveys, such as statistical significance tests and regression analysis, are the same methods used in the 1994 best seller ‘The Bell Curve‘ to prove that blacks are intellectually inferior to whites.

Among The Bell Curve recommendations; keeping aboriginals on reserve to limit racial mixing, and using financial incentives to encourage rich white women to have more babies. These were intended to increase the IQ of society. Employee engagement surveys use the same statistical methods and get similarly insightful recommendations geared to increasing the engagement score of the business.

Some organizations base management bonuses on engagement score improvement. It’s not lost on managers that the fastest way to improving engagement is firing anyone suspected of providing negative feedback.

This junk science continues in the ranking of engagement results (i.e.; 100 best companies to work for) and in cherry picking factors comprising engagement models. For example, Gallup found that the pride employees feel in working for a company was among the most important factors influencing engagement. But it’s not in Gallup’s Q12 model. Why? Because it wasn’t deemed ‘actionable’.

Actionable by whom? Perhaps not by those in the front lines, but creating a company people can take pride in should be somebody’s responsibility. Say the top of the house? Cherry-picking produces biased engagement models that sacrifice engagement by ensuring blame is fully allocated to the mid- and lower-levels of the organization.

Should we rid ourselves of the employee survey? No, but business needs to stop the junk science.

We need to get feedback useful in building better, more productive, workplaces. That means gathering improvement-oriented data concerning what people like and what they don’t, what’s working and what isn’t. Employees, and employee feedback, needs to be treated with a little respect.

Do that, and business will soon get some engagement bang for its billion bucks.

Troy Media columnist Robert Gerst is a Partner in Charge of Operational Excellence and Research & Statistical Methods at Converge Consulting Group Inc. He is author of The Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement and numerous articles in peer-reviewed publications.

Read more Bottom Line

Follow the Bottom Line via RSS

This column is FREE to use on your websites or in your publications. However, Troy Media, with a link to its web site, MUST be credited.

0 Responses to "The dirty little secret of employee surveys"

  1. Avatar
    gadgetgreg   January 16, 2013 at 4:07 am

    I also wonder if the anonymous surveys are really anonymous.

  2. Avatar
    Rebecca Everett   January 22, 2013 at 3:53 am

    Your recommendatioh is to “…get feedback useful in building better, more productive, workplaces. That means gathering improvement-oriented data concerning what people like and what they don’t, what’s working and what isn’t…” What instruments or tools exist to do this?

  3. Avatar
    Mark Bullock   March 10, 2013 at 9:38 am

    I agree with your premise.  However, because two methods use the same techniques doesn’t necessarily make them equivalent bad (or good).  I think it’s a logical fallacy to say that because tests of significance or regression models are used by one piece of shoddy science, it necessarily follows that another work will be equally contaminated.

  4. Avatar
    Susan Hunt   April 3, 2013 at 2:47 am

    I fully agree with this article, we are too quick to hang our hat on % results when what we should be doing is listening to our workforce. The tool I use is the comments people give as the primary outcome of our survey this is then used to instigate listening circles – managers listen teams discuss.

  5. Avatar
    Robert Gerst   May 16, 2013 at 1:00 pm

    Hello all,
    My sincerest apologies. For whatever reason, I didn’t notice people had been responding to this column until recently. I should have been much more diligent in following things up. So, late as it is, here are my responses:
    Mark: Your logic is sound, just because two techniques use the same methods doesn’t make them equally good or bad–your are right. But in this case, I’m talking about two methods being used in the same application for which the methods were not designed. In other words, classic examples of bad practice and junk science. The methods I describe certainly have their place, but drawing the type of conclusions they did in each of these cases isn’t one of them. 
    Rebecca: Converge posts a do it yourself guide to doing engagement surveys properly on its Voice of the Employee website. The link is http://voiceoftheemployee.com/?p=554. It details pitfalls as a well as procedural and analytic methods and tools to help do this. As I said its free and no secret marketing or registrations required. Click and its yours. This should help.
    Susan: I sort of agree with you. First feedback/discussion circles are a great idea. But here is where we must be careful. Shoddily analyzed employee engagement surveys can be just as badly interpreted by groups of individuals as by a single manager. Garbage in–garbage out. If survey data is used to help drive discussions, that feedback had better be analyzed correctly or we will be making the same old mistakes except doing it in groups. Again, that free do-it-yourself-guide described above should help.
    Okay, really sorry for this delayed response.
    Bob

  6. Avatar
    Ray Harris   February 26, 2014 at 8:08 am

    Hi Robert, I did not see the article you cited in the ASQ publication. Can you provide another link?

  7. Avatar
    Robert Gerst   February 26, 2014 at 12:26 pm

    Hello Ray,

    The article link is: http://rube.asq.org/quality-participation/2013/01/human-resources/the-new-math-of-engagement-surveys.pdf

    This is an ASQ Journal so you must subscribe to the Journal to get access.

    There is a longer version of the article for the slightly more statistically inclined at:
    http://www.converge-group.net/wp-content/uploads/2012/03/Employee-Survey-as-Statistical-Confidence-Game.pdf

    Bob

  8. Avatar
    Ray Harris   February 27, 2014 at 3:25 am

    Thanks Bob! I was looking at the wrong year. I will read through these references. I do agree the Q12 model is used to assess engagement but like all surveys and data collected that contains errors, we must be able to understand the results and make reasonable cause-effect hypotheses in order to translate observation into meaningful management action. Gallup doesn’t do that but the US government is attempting to increase federal employee engagement at the US Merit Systems Protection Board (mspb.gov) studies link. One report that is going beyond the declarative “Houston, we have a problem…” survey to assess engagement characteristics related to motivation is Federal Employee Engagement: The Motivating Potential of Job Characteristics and Rewards. This report is based on a 2010 MSPB survey of about 42,000 Federal employees regarding their motivation and engagement in the performance of their work. I am still reading the report but I think it may address your closing statement of how to get and measure the relationship between engagement and employee feedback.

  9. Avatar
    Robert Gerst   February 27, 2014 at 8:25 am

    @Ray Harris  Hello Ray, I would be very interested in taking a look at the report if you can send it to me . . . and if it is public of course.

You must be logged in to post a comment Login