Archive for the ‘Survey Research’ Category
Scott Brooks and I recently published this article which we thought you would enjoy. Jeff
Daniel Kahneman coined the acronym WYSIATI which is an abbreviation for “What you see is all there is”. It is one of the human biases that he explores when he describes how human decision-making is not entirely based on rational thought. Traditionally, economists believed in the human being as a rational thinker, that decisions and judgments would be carefully weighed before being taken. And much of traditional economic theory is based on that notion. Dr. Kahneman’s life’s work (along with his co-author Dr. Amos Tversky) explodes that notion and describes many of the short-comings of human decision-making. He found that many human decisions rely on automatic or knee-jerk reactions, rather than deliberative thought. And that these automatic reactions (he calls them System 1 thinking) are based on heuristics or rules of thumb that we develop or have hard-wired into our brains. System 1 thinking is very useful in that it can help the individual deal with the onslaught of information that impinges on us each and every day, but the risk is when a decision that one is faced with should be thought through rather than based on a knee-jerk reaction.
System 1 decisions are easy, they are comfortable, and unfortunately they can also be wrong. But wrong in the sense that if one learned how to take a step back and allow for more deliberative thought prior to the decision, some of these wrong decisions or judgments could be avoided. A simple example from Dr. Kahneman’s book “Thinking Fast and Slow” will illustrate the point.
“A bat and a ball together cost $1.10. The bat cost $1.00 more than the ball. How much does the ball cost?” Fifty percent of the students who were posed this simple question, students attending either Harvard or Yale got this wrong. Eighty percent of the students who were asked this question from other universities got it wrong. This is System 1 thinking at its finest and most error prone. It is fast, easy, comfortable, lets you come up with a quick answer or decision, but one that is likely wrong. Knowing who reads this blog I’ll let you figure out the answer yourself.
WYSIATI is the notion that we form impressions and judgments based on the information that is available to us. For instance we form impressions about people within a few seconds of meeting them. In fact, it has been documented that without careful training interviewers who are screening job applicants will come to a conclusion about the applicant within about 30 seconds of beginning the interview. And when tested these initial notions are often wrong. Interviewers who are trained to withhold judgment about someone do a better job at applicant screening, and the longer that judgment is delayed the better the decision.
This notion of course flies in the face of Malcolm Gladwell’s best seller “Blink” in which he talks about the wonders of human’s ability to come to decisions instantly and a whole generation of manager’s have eagerly embraced his beliefs - including a few CEO’s I know. Why? It is easy, it is intuitive, it is comfortable and it plays to the notion that I am competent and confident in my work. The only problem is that when put to serious scientific scrutiny, it is often wrong.
A few months ago I introduced this concept to an HR group I was talking to. I explained how untrained HR people in a rush to judgment will jump to conclusions about someone, perhaps too rapidly. One 30-year HR veteran insisted that this may be all well and good but of course did not apply to her. After all, with her 30 years of experience her rush to judgment was of course going to be accurate. She “just knew” who were going to be good employees. I let it drop, and I think I was labeled a trouble-maker by the group. That is a label I can embrace.
We tend to develop stories based on the information at hand; piecing the information we do have into a narrative, often without asking the question, “what information am I missing”? In the area of survey research I have often seen researchers confidently presenting the “drivers” of one type of behavior or another. Say for instance, the drivers of employee engagement. But since the analysis is based on a “within” survey design, the only drivers that can possibly emerge are those that you asked about in the survey in the first place. So the researcher, in designing the 30-50 item survey, is limiting the drivers to those items that they decided to ask about in the first place. The researcher likely has in their head a model of what is important in driving engagement when designing the questionnaire, a model that was designed based on another 30-50 item or fewer questionnaire. It becomes a tautology, it becomes true because I tested it and it came out as true, but the only thing I tested is what I already believed.
There are techniques that can be applied that lead towards more deliberative and better decision-making processes. If you were walking briskly down a busy road and someone asked you “how much is 17 x 24?” you would do what every other human would do to figure that out, you would stop and think.
“The great majority of mankind are satisfied with appearances, as though they were realities, and are often more influenced by the things that seem than by those that are.”
I sometimes think of my job as holding up a mirror to an organization so that it can see not what it wants to see but what it has to see. While people in almost every organization like to think their organizations are unique, most organizations really fall within a few common patterns, with some nuance, when it comes to employee survey findings. Here are some common patterns:
- In most organizations there are virtually no differences to be found when the data is cut by gender, ethnicity, or generation. While there are some slight differences to be found, (e.g. females are often a hair more positive, as are traditional part-timers), on organization performance items like customer focus, quality, leadership, communications, decision making, cooperation and teamwork, looking for differences is like searching for a meaningless needle in the proverbial haystack. When differences are seen it is usually indicative of significant underlying organizational issues.
- The largest differences are most reliably found on an employee survey when you cut the data by occupation or level within the organization. Managers are generally between five and 15 points more favorable (the more senior the more favorable) on survey items than non-managers, depending on the item. They are typically less favorable on customer service/focus, resources and product quality. When the differences are larger it is usually a sign that senior management is living in a different organization than the other employees experience, and that each group would have difficulty seeing the world from the other’s perspective. It is not unusual though for administrative groups to be more favorable than their level suggests.
- The most favorable group completing the employee survey will very reliably be the employees that you just hired. Specifically, those with less than 12 months tenure. It takes the typical organization about 12-18 months to beat that positiveness out of the new employee. Disillusionment with organizational effectiveness, training, advancement opportunity, pay and other frustrations as well as the sense the organization is not as was advertised, drive the numbers down.
- The least positive will be those with around 3-5 years tenure and in many organizations there will be a gradual recovery in positiveness over time, but it usually never gets as positive as the newly hired employee was again. A small number of organizations can buck this trend and what they do in order to accomplish that is very interesting.
- By geography the least positive employees in the world are pretty reliably the Japanese (with a few exceptions). The most positive are those in Latin America and some other parts of Asia (e.g. Indonesia, Thailand, Vietnam). The USA is below Latin American, with the UK trailing the USA and continental Europe lagging behind the UK. Russia and the USA look remarkably similar on a large number of items.
- The best way to predict turnover with an employee survey is by asking the employee directly if they are going to stay or leave. They tend to answer very honestly. If anyone tries to sell you some mumbo jumbo predictive index, walk away. There is nothing better out there than simply asking that one directly.
- The best way to predict customer loyalty is to ask the employee about whether customer issues are resolved quickly.
- The best way to predict accidents is to ask about the safety environment and the emphasis placed on safety. Are you getting the idea yet?
- Employees are generally more critical of product and service quality than your customers are. They see how the sausage gets made.
- Employee Engagement is no magic bullet and is rarely predictive of many critical organizational performance metrics.
While not every organization out there is as unique as perhaps they think they are, there are certainly some organizations that are performing better than others in various aspects. And there are some organizations out there that could really benefit from a well done employee survey, focused on the right things, aimed at improving effectiveness and performance and monitoring employee sentiment and insight. Sometimes all sorts of excuses are given to avoid having to look at the organization squarely in the mirror. “We just had a bad quarter, or are in the middle of a reorganization, or we are unveiling our revamped strategy and vision, or we are rolling out new products” etc.
Organizations making excuses like that, I would bet, would be extremely reluctant to use those same arguments with respect to tracking financial performance stating, “we had a bad quarter so we are going to skip tracking the financials this quarter. We will start again perhaps next quarter when we believe the numbers will look better.” They simply would not be able to get away with it. If people are truly an organization’s most important asset, as is so often stated, how can their opinions, observations, insights, and emotions be ignored?
© 2012 by Jeffrey M. Saltzman. All rights reserved.
Visit OV: www.orgvitality.com
“But if you will it, it is not fantasy” – Theodor Herzl
Recently, virtually every organization that I speak with has put innovation front and center as a necessary, in fact imperative characteristic of their organizational cultures. The thinking is clear. In order to thrive in turbulent environments organizations must innovate, must examine the way they do business, update their products and services, and must differentiate themselves in order to outperform the competition. Competition, environmental stress and the need to survive as an organization spur on innovation.
Innovation is often described as coming in bursts, but much more common is the rolling-up-the-sleeves, hard work, incremental innovation, building on other’s breakthroughs, often in a collaborative fashion, that over the long run can radically change the way things work and the products that an organization offers.
It could easily be argued that innovation occurs naturally among all of the earth’s creatures. Species innovate constantly and naturally in the never ending battle to survive. One species of cuckoo finch has eggs that mimic the coloration of another species so that when the finch deposits their eggs into the other species’ nest, the hatched baby birds are raised by the tricked surrogate parents. As a defense the second species, the tawny-flanked prinias evolved more colorful eggs that looked different form the finch’s eggs but the finches responded by evolving and again mimicking the more colorful eggs. Evolution is innovative.
The mitochondria that inhabit our cells and produce the energy which powers cells originated externally from the cells they now inhabit. They have their own DNA and can reproduce only from their own DNA, indicating that they were a separate life form somewhat like bacteria. Prior to their role in our cells they existed independently. Mitochondria entered into a symbiotic relationship with cells and then that combined organism evolved into the variety of cells that make up human beings. Today we would not be able to survive without these creatures living within our cells and the mitochondria would not exist without us. Evolutionary innovation is often collaborative.
Beyond evolving on a biological front, humans evolve and innovate their behaviors constantly if not quickly, with our ancestors developing new forms of stone tools over millennia, fire being tamed, the taking up of living in shelters of one sort or another, and placing metal strips on the teeth of our children to improve function and appearance.
Many of the inventions that propelled our civilization and were described as deriving from “ah-ha” moments were nothing of the sort. Rather the innovative breakthrough came from groundwork that laid the foundation and was then built upon. Basic innovations often sit dormant until additional development work and insights are gained allowing the innovation to be applied in day-to-day life.
Take Edison’s light bulb for instance. It is often credited to Edison as a singular event. And in fact Edison played a very important role in the light bulb, but without the innovations of those who came both before and after him the light bulb would not have become as wide spread as it has. Here is a chronology of the major milestones.
• The first electric light was made in 1800 by Humphry Davy. When he connected wires to his newly invented battery and a piece of carbon, the carbon glowed, producing light.
• Much later, in 1860, physicist Sir Joseph Wilson Swan was determined to devise a practical, long-lasting electric light. He found that a carbon paper filament worked well, but burned up quickly. In 1878, he demonstrated his new electric lamps in Newcastle, England.
• In 1877, Charles Francis Brush manufactured some carbon arcs to light a public square in Cleveland, Ohio, USA. These arcs were used on a few streets, in a few large office buildings, and even some stores. Electric lights were only used by a few people.
• Thomas Alva Edison experimented with thousands of different filaments to find just the right materials to glow well and be long-lasting. In 1879, Edison discovered that a carbon filament in an oxygen-free bulb glowed but did not burn up for 40 hours. Edison eventually produced a bulb that could glow for over 1500 hours.
• Lewis Howard Latimer improved the bulb by inventing a carbon filament (patented in 1881); Latimer was a member of Edison’s research team, which was called “Edison’s Pioneers.” In 1882, Latimer developed and patented a method of manufacturing his carbon filaments.
• In 1903, Willis R. Whitney invented a treatment for the filament so that it wouldn’t darken the inside of the bulb as it glowed.
• In 1910, William David Coolidge (1873-1975) invented a tungsten filament which lasted even longer than the older filaments. The incandescent bulb revolutionized the world. (Enchanted Learning).
Rather than being the exception the “evolution” of the light bulb is very often how innovation occurs with multiple people contributing, often working collaboratively over a period of time.
There are multiple methods available for measuring the existence of innovation in organizations. You could count the number of patents issued to the organization, or the age of each of its product’s since design, the amount of time that employees spend on innovation, the R&D budget, the headcount assigned to “innovation”, or the perceptions of the customers towards the organization’s products and services as being innovative. One method for measuring the degree of innovation in organizations is through the perceptions of the employees.
Employee surveys will often ask about the “emphasis” on innovation within the organization, but I prefer asking about whether innovation is actually occurring. Critical when measuring innovation through employee surveys is to ask about:
• the generation of innovative ideas;
• the ability to test out those ideas from a funding and other resources standpoint;
• the ability to evaluate innovations to see which one’s should be implemented organization-wide and which ones rejected.
Examining or asking about the reward system is also often very informative as an organization may truly desire to be innovative, but is actually rewarding its employees for playing it safe and not trying new things rather than the innovative efforts desired.
Slack and redundancy are two concepts that are also critical to be in place for an organization to successfully innovate. If an organization is being run in such a tight fashion with no slack so that it can’t try new things, because all resources are dedicated to getting the work done the traditional way, the ability to be innovative does not exist. And likewise if the organization does not have the ability to experiment with new methods, while another redundant process is performing in a traditional fashion, the evaluation of innovative ideas and processes will be very difficult to objectively assess.
Organizational innovation is critical and creating organizational cultures that support innovation rather than suppress it is within reach for all organizations.
© 2012 by Jeffrey M. Saltzman. All rights reserved.
“When I die, I want to die like my grandfather who died peacefully in his sleep. Not screaming like all the passengers in his car.” – Will Rodgers
Go outside and pick up a handful of dirt. Your life and the life of most creatures on this planet is possible not because of what you see in the dirt but largely because of what you don’t see. At the micro-scale the Earth teems with life, much of it overlooked by us as we wend through our daily routines. This largely unnoticed life is more than important to the health of our planet for it forms the very foundation upon which other life can exist. Without the smallest of creatures, the mirco-organisms which create the conditions upon which other life depends, the earth would be barren, unable to support our existence. Being human, a good chunk of us do our utmost to destroy these foundational creatures, or due to our obliviousness, like the grandfather who fell asleep at the wheel, make their existence as tenuous as possible.
Organizational surveys as they are often currently conducted like to measure the “big picture”, but I have begun to wonder if those of us who conduct surveys for a living are missing the leaves because all we tend to focus upon is the beauty of the forest. But without the leaves the forest will die.
The notion is this. There are fundamental underlying conditions that need to be present within an organization for that organization to truly thrive for the long term. These fundamental conditions form the basic building blocks of organizational life and hence success. I think we have intuitively known about this and there are historical questions, which I now think of as primitive attempts to get at these conditions that have existed on surveys for a long time.
Take for instance an item that asked employees to rate their co-workers. This item tries to go beyond the formal structures of the organization, training or lack-thereof that the workers have received, the pay or benefits they obtain etc. It tries to get at something more underlying – do you like/respect/want to be around the people you interact with day-to-day? It was pretty much a constant in surveys up until about 5 or so years ago. The typical response to that item was in the mid-80 percent or higher on favorability. In other words people in general liked their co-workers. It was really unheard of for this item to be unfavorably rated. It just did not happen.
When all else in an organization was rated poorly, you could always turn to the management team and say, well at least people like their co-workers, that is a starting point, a place from which you can build. But now I think it was simply a poorly written question, for while people did indeed like their co-workers it clearly did not get at the fundamentals, the micro-organisms necessary for the organization to thrive. There is also an issue with very positively rated items, in that they have imbedded in their response pattern less information then items that are more moderately rated and hence the organization can learn little from these items. The same could be said of the generally derided item about having a best friend at work.
Physicists use something called an effective theory approach for problem solving. It is an approach that utilizes the theory that makes most sense at the distances or object size that you are studying. Newton’s theory of gravity is just fine if you are on the earth’s surface studying an apple falling from a tree, but Einstein’s approach to gravity is necessary if you want to be able to solve problems that involve planetary distances. And Einstein’s theory does not work at the sub-atomic or quantum level. For that you need string theory. This approach implies that organizational surveys should ask about only those things you could hope to measure and see within the organizational world at the distances or sizes you are studying. The physicist’s approach implies that measuring the forest and ignoring the leaves is just fine if you can show that the forest is what leads to the outcomes that you desire, such as customer repurchase intentions. What is left out is that the forest will in fact perish if the conditions do not exist for the leaves to thrive, to create the forest in the first place.
It is very clear today that many approach organizational surveys without having a real understanding of the unifying state that allows us to measure the basic building blocks of successful organizational functioning, those things that are foundational to creating the conditions that allows an organization to thrive, to become vital over the long term.
I was recently sent a note about a document new Apple employee’s see it goes like this:
“There is work and there’s your life’s work. The kind of work that has your finger prints all over it. The kind of work that you would never compromise on. That you’d sacrifice a weekend for. You can do that kind of work at Apple. People don’t come here to play it safe. They come here to swim in the deep end. They want their work to add up to something. Something big. Something that couldn’t happen anywhere else. Welcome to Apple.”
Whether those conditions exist for each and every employee of Apple is of course a good question and one that I can’t answer here, but the notion that there is more to work than work is very clear. If everyone in every organization felt that way just imagine what we could accomplish.
© 2012 by Jeffrey M. Saltzman. All rights reserved.
The first of Newton’s laws of motion states “a body that is at rest will remain at rest unless an outside force acts on it.” His second law states “a body in motion at a constant velocity will remain in motion in a straight line unless acted upon by an outside force.” Those same laws seem to apply to the world of employee survey action taking.
Some of those who get survey results never seem to get around to taking action based on the survey results they have in hand. And just like a body at rest, they tend to stay at rest doing nothing with the findings.
- The survey results provided may not be definitive enough for them and they may request additional analysis after analysis until they get around to doing just about nothing.
- The survey results may point to action that is difficult or overwhelming and so the easiest path may again be to let things be just as they are and do nothing.
- They survey results may point to behaviors that go against closely held beliefs that the manager may have, so even though the data says one thing, he or she may simply know in their heart the “right thing to do” regardless of the data.
In one study which pointed out some of the obstacles to having action arise from the survey process, (Wiley & Brooks, 2010), the 3 top obstacles to taking action on a survey were identified as:
- Accountability (12%)
- Holding organizational members responsible for their role in the survey program; ownership and clarity of assignment
- Resources (12%)
- Especially time (given the other demands of manager’s job), but other resources as well: training, technical, financial
- Importance (12%)
- Management (especially executive management) attention to and support for survey
But looking on the positive side for a moment, what are the benefits of taking action, even if it may not be the perfect action based on the survey results? If you look at survey data longitudinally and track which employees saw results from a previous survey vs. those who did not (from within one organization), and which ones saw action arising from the survey vs. those who did not, the data strongly suggests that seeing the data and seeing action, drives a very positive shift in the next survey iteration on critical business performance metrics.
- In one organization for instance if 75% or more at the department level could recall actions arising from the survey their average employee engagement score rise by 5 percentage points.
- In that same organization, those departments where less than 50% could recall actions arising from the survey score their employee engagement scores went down by 13 percentage points.
The benefits of taking action, even if it is not the perfect action are very clear. A body in motion tends to stay in motion, and in our fast changing world, staying in motion; constantly improving organizations based on insightful data which is tied to the organizational strategy is a very impactful way to help performance.
© 2012 by Jeffrey M. Saltzman. All rights reserved.
By guest blogger Scott Brooks
Employee engagement is important. It has well-acknowledged meaning and impact. It has the advantage of being universally applicable — virtually any organization with any type of employees should benefit from better measurement and understanding of how these employees connect to and add “oomph” to their work.
However, “engagement” is one possible focus for an employee opinion survey. By itself it does nothing. Like motivation, engagement without direction or ability is easily wasted. Also like motivation, engagement supplies energy. Measuring it alone is like measuring the gas in the gas tank without understanding the engine (aka an organization’s ability to create value). At risk of a metaphor getting out of hand, important elements to measure can also include steering (agility), shock absorbers (resiliency), or GPS (leadership).
Here is another metaphor: A focus on engagement is like a focus on individual concert musicians. A truly strategic employee survey designed to understand and improve the whole symphony experience would likely examine perspectives of the conductor, sheet music, concert hall acoustics, and audience feedback.
A strength of engagement—it’s universal applicability—is also a limitation. If an employee survey is to be strategic, it needs to measure strategic topics. In for-profit organizations, strategic success is based on differentiation from competitors. If Wal-Mart has the same strategy as Target, there is no competitive edge. Embrace all the best practices you want, but you cannot copy your way to competitive differentiation. Engagement, it follows, is not a strategic topic. Not by itself. As a topic it holds no uniqueness for a given organization. How an organization USES its engagement is where the unique, competitive edge will be found. Employee surveys are excellent tools for drilling into not just engagement, but how it can be used. How does an organization really create value? This is where surveys really become strategic and connect with the passions of executives. If a CEO could snap his or her fingers and have all the thousands of employees in an organization trying to improve one thing, what would it be?* Would a CEO choose service, sales relationships, inventing next generation products? Most often, leadership’s focus is on how an organization uses engagement, how value is created, and less on engagement itself. That doesn’t mean it isn’t important. But our job is to start with this focus, this engine, and then connect it to the gas tank.
(* This is one great definition of an employee survey, framing it in terms of action, not measurement.)
Linkage research is the statistical demonstration of relationships between engagement or other survey topics and business performance metrics such as sales, customer satisfaction, employee turnover, or safety. It can help guide businesses on where to focus limited resources in order to maximize success. Embedding surveys (including employee engagement surveys) into a more comprehensive, multi-faceted research program adds more value than when each piece of work is performed independently. Engagement surveys form a piece of the business success puzzle, but business success is a puzzle with many pieces. In fact, in head-to-head tests, the best predictor of customer loyalty is not engagement, but service climate. The best predictor of safety is not engagement, but safety climate. Similar statements can be said of quality, scalability, innovation, and a host of other important organizational outcomes. In short, if you really want to drive improvement in “X,” you had better ask about “X” in your survey.
Where this leads us is that employee surveys should not be simply about employees. Sure, we want to understand employee-centric topics, but employees are valuable sources of information about a much broader array of topics that help us understand and improve organizations. We wouldn’t advocate that all possible topics are always included in employee surveys, certainly. We want to be conscious about what we include and what we exclude. Addressing engagement alone is almost certainly sub-optimizing the effort. It is important not simply to ask employees about themselves, but to ask them about what they see around them. They have a lot to say about work and effectiveness.
For many in the HR marketplace, engagement unfortunately has become an end and not a means. We confuse the tool with an outcome unto itself. In doing so, engagement has become a mono-focus for suppliers interested in high production and also for time-stressed buyers who want something simple. It is certainly easy to create standard products and commodities based on engagement, a reasonably generic topic. But the real goal is sustainable organizational change. That is hard. That is custom. That is uniquely solved for each organization.
© 2012 by Jeffrey M. Saltzman. All rights reserved.