Last week I talked about some of the ways that data is misunderstood and misused. Today I want to circle back to leadership, and talk a bit about how these data issues can have far reaching impact in the workplace, and why every good leader needs to use data responsibly.
I know, I know, two posts in a row about data. But I think this is important for two reasons.
The first is that not everyone has access to the same levels of education, including math and science. There’s a large number of people who don’t have the background to understand how easy it is to misunderstand data, and inadvertently cause harm. The more we talk about it, and the more accessible we can make it, the more everyone benefits.
The second reason is that there are a number of people out there who know exactly what they are doing when they misdirect or abuse data. This happens extremely frequently with those in high levels of power. And I want all of us to have the capability to call out bad data when we see it.
Not all of us are managers, but all of us can be leaders. And a leader, at whatever level, needs to be able to question data being used in bad faith.
Today, I want to break down one of the most frustrating examples I’ve seen of bad data used in bad faith, which was sent out by my agency’s HR department. Not only was the data suspect, the way it was delivered was problematic.
HR did regular employee engagement surveys, which on its own, is not necessarily a bad thing. As defined by Gallup, engaged employees are “those who are involved in, enthusiastic about and committed to their work and workplace”. Doing these surveys is common at workplaces, and often a well-intentioned attempt to take the pulse of the workforce. But like most surveys, it’s extremely important to recognize the limitations of the data that you can collect.
And yet, a few years ago, the department sent out a very cheerful email to all employees. After the most recent employee engagement survey, they were thrilled to report the results. According to them, employee engagement was extremely high. In fact, we even beat the national average as reported by Gallup. We may have assumed that our overwhelmed, overloaded bureaucratic institution with constant turnover was struggling, but in fact, we were in great shape!
Any theories on what the issue might be with a message like this?
It’s a bit of a trick question. Because there’s not just one issue.
Issue one – these surveys are voluntary. Now I don’t necessarily advocate forcing staff to fill out surveys. But tell me this – if you’re feeling disengaged at work, unmotivated, invisible, checked out – how likely do you think you are to do a survey asking you how you feel?
If you’re so overwhelmed with work that you’re doing overtime, and still unable to keep up with workload, how likely are you to stop what you’re doing to fill out a survey?
If you’ve spoken out to management and asked for help and support, and they’ve ignored you, how likely are you to think it’s worth your time to fill out a survey?
This is what is known as a sampling bias.
Let’s look at it this way. Say you want to do a survey to find out if everyone in your office would be willing to chip in for a new microwave. So you sit in the kitchen with your clipboard, and ask people as they come in to heat up their lunch. And you discover that a strong 90% of those people would be willing to give money. So you decide to let your manager know that 90% of the office is in favor of the idea.
But you didn’t ask the entire office. You asked the people who come into the kitchen. You didn’t ask the people who go out to eat, or go home for lunch. And those are the people who are less likely to want to chip in for an appliance they don’t typically use.
You didn’t sample everyone, so you can’t conclude your results apply to everyone.
This is what HR did. They sampled only those employees who had the time, interest, and inclination to provide feedback. And yes, most of those people would come across as engaged.
But that’s not all your employees. And HR should know that.
Issue two – we had huge rates of turnover at this agency. I know, because I worked in personnel for a number of years, and also handled hiring as a manager. It was not uncommon to see workers come in and burn out within a year, or for longer term employees to shift to the private sector.
So this employee engagement survey, even if you could get every employee to voluntarily fill it out, is missing a huge piece of data.
It’s missing the thing we can’t see. The people who aren’t there.
This is another common kind of bias, called survivorship bias. There’s a fantastic article on it written by David McRaney, that I highly recommend.
Essentially, we as humans, have a tendency to ignore what we cannot see, and give extra weight to what we do see.
Say, for example, that you are tasked with doing a survey of all the chairs in the office to see which brands hold up the longest. And you find some chairs that were bought twenty years ago, and yet are still in almost perfect condition. You make note of the brand, and think, wow, this brand is just amazing! We should only buy this brand from now on!
What you’re not seeing? All the chairs from that same brand that broke down and were thrown out years ago. The chairs that didn’t make it past a year or two. The chairs that would lead you to a different decision.
You’re missing an important part of the picture.
And again, this is what HR did. They didn’t do exit surveys, they didn’t follow up with ex-employees. They just looked at what was in front of them and used it to drawn conclusions for everyone. It’s not a true picture of what is happening.
Issue three – the presentation is a problem, as are the power dynamics.
This is not about why the data itself is important, but rather what you try to say with it.
Now it’s entirely possible that the HR team responsible for the survey and email didn’t realize the issues with their data. Which is rather concerning, but it’s possible.
However, it’s one thing for a co-worker to send out some information that could benefit from a second look. It’s a different thing when it comes from those who have power and control over your career.
There is a danger when those in positions of power misrepresent the truth, intentional or not. When people with power use data in bad faith, it has an impact. They’re sending a message, even when it’s not deliberate.
And I can guarantee that HR does know the turnover rates for the agency. I can guarantee that they were very aware of the problems employees faced with workload, stress, burnout, and other office frustrations.
We can give others the grace of assuming good intentions. But it doesn’t give them the right to not be called out for what they say. And HR should be called out for this.
Because there’s a big problem with HR sending out an email that says “you are all really happy and engaged, aren’t we so lucky!” while ignoring all those missing pieces of data.
It sends a strong message that your problems are not important. That they care more about their numbers looking good than delving into the reasons people are leaving. That if you’re not happy, the problem is entirely with you, and not the agency.
This is why I use the term bad faith to describe how data is sometimes used. Because data is good for educating and making informed decisions. But when you’re using it to sugarcoat, minimize problems, or otherwise distract focus from the negative, you’re not using it in good faith.
It’s one thing to want to encourage positive thinking. But when you refuse to acknowledge the negative, and try to throw sparkles over the situation, it’s patronizing. You’re trying to invalidate employees who are not engaged. You’re telling them they’re the minority, based on bad information, instead of trying to actively solve what’s wrong.
I know this may seem like a lot to write about one email, but I think it’s such a good example of how small things can have a big impact. I still remember this email after several years, because of how insulting it felt.
When you are in a leadership position, it’s important to consider these things.
Doing a survey is fine. But be honest about what you’re measuring and what might be missing.
Sending out an encouraging email is great. If there are some positive trends in the data, fantastic. But don’t make grand claims. Don’t insult the intelligence of your employees by pretending one survey proves anything beyond a shadow of a doubt.
And above all, be aware of how you’re presenting your information. If you have power over others, you have to consider that in your message. You will not always be able to see your own bias, but if you are open to criticism, people will let you know.
As with all of the important tools of leadership, this is about being mindful, careful, and smart. A hammer can be used to hurt or to build. Data is no different. Use it wisely.
Absolutely amazing post. Please by all means keep them coming.