Jeffrey Saltzman's Blog

Enhancing Organizational Performance

Employee Survey Interpretation – 102

with 2 comments

“If you were really good you could get all you needed simply by asking one question” Anonymous manager in discussing his organization’s survey with me.

Over the last decade or so there has been increasing pressure on survey providers to make their employee surveys shorter and shorter and more of the items directly actionable. People in general can answer 4-5 questions per minute and so the difference between a long survey, say 100 items, and a shorter one say 50 items is about 10 minutes of someone’s time. More of the concerns around length seem to center on perceptions of the burden placed on the individual and the organization rather then the actual amount of time spent answering questions.  

Let me state something fairly obvious. Surveys can only provide feedback on what was asked about. If you don’t ask a survey participant “do you have the training you need to get your job done”, don’t expect to answer the question from the survey itself, “Do people have the training they need?”

Survey questions can be thought of as falling broadly into 2 categories (plus demographics). One category is called dependent variables and the other is called independent variables. Dependent variables are outcome items (e.g. overall satisfaction, pride, intention to leave) and independent variables are causative or action oriented items. In experiments, independent variables are the factors that a researcher can manipulate to try and tease out cause and effect on the studied outcome or dependent variable. For instance, if I give more training does pride go up, does intention to leave decline?

While a good survey can benefit from a mix of both, one of the consequences of shorter surveys is more of a reliance on dependent items, which can be more broad and general. In some cases independent variables can become or can be treated as dependent variables and to a lesser extent dependent variables can become independent ones.

On surveys one easy way to tell if a question is dependent or independent is to determine if impacting the item’s result is directly actionable. If for instance you ask someone to rate their pride in working for the organization and the result came back poorly, the next step is to think about raising the level of pride in the organization. In order to do that you need to consider the factors within the organization that impact pride. Your ability to impact the level of pride in the organization is determined by your ability to impact those factors that you hypothesize drive pride. Hence the question about pride is a dependent variable and not directly actionable.

Going back to the training item mentioned above, “Do you have the training you need to get your job done”,  if the result from that item was poor, in order to improve on the item you would provide training, hopefully the kind of training people need to get their jobs done. Since the result from this item is directly actionable it is an independent variable.

You can see that if you were to ask about each and every potential specific action that enables job performance you would wind up with a very lengthy questionnaire. You would ask about the training (for your current job, to prepare you for the next job etc.), communications (upwards, downwards, lateral, timeliness, openness, honesty of), decision making (speed and quality), cooperation and teamwork, information systems, equipment and resources, computer systems hardware and software, accuracy of company data, inventory systems, maintenance and repairs, physical working conditions, safety, ethics, clarity of company mission, vision and values, etc. The list could go on and on. And that is the rub. The push to shorten questionnaires leads to a greater reliance on dependent variables and results in fewer directly actionable items being used – in direct opposition to the pressures that organizations are putting on survey providers.  

To shorten the above list somewhat you might ask “do you have what you need to get your job done?” instead of each and every specific about the job. While the result of that item is important it is not directly actionable without asking people in follow up meetings “what do you need in order to get your job done better?” (That is the real power behind survey feedback meetings, to put specifics onto the actions that should be taken).

One approach that can be taken in analyzing survey results is to take one of the dependent variables and regress it against the independent variables (through correlations, regression or modeling techniques). For instance, taking pride as the dependent variable or outcome item, we would statistically determine which other items from the survey (and it is best to limit it to the other independent or actionable survey items) are most influential on causing pride to go up becoming more favorable, or to decline becoming less favorable. These lists of “key drivers” can help answer the questions about how to create positive change in the dependent variable results. It does not negate the underlying truth thought that the survey can only provide results on items that you ask about. There may be other drivers of pride within the organization that are not asked about on the survey and hence can not possibly show up on the key driver list.

Written by Jeffrey M. Saltzman

November 16, 2009 at 8:03 am

2 Responses

Subscribe to comments with RSS.

  1. I once saw a hotel’s feedback card online with a single question: “Would you recommend us to a friend?”. That gives a lot of information right there – but anyone with a stake in the game would want specifics.

    So, no matter how good you get, you cannot measure anything meaningful unless you ask it.


    November 17, 2009 at 11:46 am

  2. Really great article Jeffrey…..

    Nice tips on Employee Survey!!!!

    Employee Surveys

    December 5, 2009 at 6:57 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: