A Surprisingly Simple Scaling Superpower: Checklists

On a frozen Thursday afternoon in January, 2009, US Airways Flight 1549 lifted off from LaGuardia Airport into a clear sky. The plane had only completed three minutes of the flight to Charlotte, North Carolina when the plane collided with a flock of Canadian geese, sucking birds into both of the plane’s engines. Mid-air collisions with birds are so common that the airline industry has a name for it - “bird strikes” [1] - and the engines are designed to withstand collisions with smaller birds. But Canada geese are large enough to do serious damage; the airplane’s engines faltered, and the plane began dropping out of the sky.

The passengers and flight crew didn’t make it to Charlotte that afternoon, but every one of them lived to make it home to their families. The Captain, Chesley "Sully" Sullenberger, and First Officer Jeffrey Skiles managed to pull off a near-impossible emergency landing on the Hudson River, which the press dubbed the “Miracle on the Hudson.”

Passengers evacuate Flight 1549 after an emergency landing on the Hudson River.

Passengers evacuate Flight 1549 after an emergency landing on the Hudson River.

The pilot and first-officer demonstrated impeccable skill in executing the emergency landing. In the investigation that followed, experienced pilots attempted to replicate the feat in flight simulators, and crashed repeatedly. But there is a third hero in this story that may surprise you: A checklist.

When the engines failed, Sullenberger and Skiles knew what to do. Sullenberger took the controls, and Skiles grabbed a checklist. The checklist itemized all the essential steps required for an unpowered emergency water landing. This checklist ensured that in the heat of the moment, critical steps weren’t forgotten. This allowed Sullenberger to direct his full attention to the task of landing the plane.

According to Atul Gawande in his insightful book, The Checklist Manifesto, the Miracle on the Hudson is not the only disaster that checklists have prevented. Airlines have developed checklists for almost every procedure required for flying a plane, from the mundane (e.g. procedures for takeoff) to the extremely rare (e.g. unpowered emergency water landings). Pilots and flight crews use these checklists religiously for a simple reason: they prevent catastrophes.

In the Checklist Manifesto, Gawande describes his struggles and successes as he designed and implemented a system of checklists in his own field - surgery. Gawande worked with a team put together by the UN’s World Health Organization to create a checklist that could reduce preventable errors and complications at hospitals all over the world. The checklist worked. The team studied the effects of the checklist at 8 large hospitals in both rich and poor countries [2]. The results of the study, summarized in an article in the New England Journal of Medicine, found that the checklists reduced surgical complications by 36%, and deaths by a whopping 47%.

Checklists as Scaling Superpower

In my consulting practice, I focus on the process of scaling - taking something that you know is great, and making it available to orders of magnitude more people. This process of expanding the scope of an organization requires more people, more collaboration, and an increase in the number of work processes; as a result, it also involves a staggering increase in complexity.

In this environment of extreme complexity, it’s easy to forget important steps and subtasks, or fail to accurately balance competing priorities. Moreover, in complex environments, people cannot carefully think through every decision and must rely on intuition. Unfortunately, relying on intuition alone increases the likelihood of falling prey to a variety of well-known cognitive biases.   

Even if a person is able to deploy sufficient vigilance to remember every step of important tasks, carefully balance competing priorities, and slow down to ensure that bias has not clouded judgement, this vigilance comes at a cost. Exercising vigilance saps cognitive resources, limiting a person’s capacity to perform at her best on other more critical tasks. These are precisely the conditions in which checklists can substantially improve an organization’s performance.

Checklists are helpful for any repeated organizational process. Examples include a checklist of items that need to be included in every presentation for clients, steps for A/B testing, or things that need to be included in every blog post.

Why Checklists Work

The reasons that checklists improve performance vary from situation to situation. Below, I describe the three main ways that checklists can help organizations involved in a scaling effort. In the Checklist Manifesto, Gawande outlines some other ways that checklists can help temporary teams (such as flight crews and surgical teams), but since most organizational teams have fairly stable membership, I won’t cover that here.

Avoid missed steps

When you’re charging towards a goal, dotting i’s and crossing t’s is uniquely frustrating. I know this from my experience conducting research while completing my PhD. The time between generating an interesting hypothesis, and getting the data required to test your hypothesis is agonizingly long. It can take months to design the study, collaborate with colleagues, complete the ethics review, and run the study. Once you have data in hand, you naturally want an answer immediately. But if your excitement gets the better of you, and you cut corners in the analysis process, the results are meaningless. I expect that every researcher falls into this trap occasionally. It ends up slowing down the whole process, resulting in useless fits of excitement or disappointment that turn out unfounded. And a haphazard analysis plan is far more prone to stupid errors. This is a problem waiting for a checklist.

Specifically, two conditions make people more likely to skip steps:

  1. People skip steps when motivated to complete the task as fast as possible. This can happen if you are under time pressure, or are excited to see the results of your effort (e.g. generating a product you’re excited to bring into existence, answering a pressing question, completing a surgery when there are a hospital’s worth sick patients waiting for attention, etc.). In these situations, you are motivated to skip steps so you can finish faster.

  2. Skipping steps is more likely in situations where completing any given step matters only rarely, but when it does, it matters a lot. This is the case in my data analysis example above - any given statistical check rarely changes the results, but every so often it changes them dramatically. This situation also arises at the conclusion of a surgery: it’s extremely rare that surgical implements get forgotten in someone’s body cavity, but it matters enough that it’s probably worth double checking before you stitch a person up.

Taken together, these two conditions can cause a perfect storm that makes skipping routine steps commonplace. First, people are motivated to finish the task quickly. Second, they are able to convince themselves that skipping steps probably isn’t a big deal. Checklists are therefore very effective.

Reduce cognitive load

I discussed the importance of limiting cognitive load in a previous post. Cognitive load is determined by the number of different things you have on your mind at any given time. The more things you are trying to remember or account for, or the more tasks you are trying to complete simultaneously, the greater your cognitive load. This is important because high cognitive load reduces a person’s performance. Trying to do too much at once literally makes you stupider. As a result, checklists can make you smarter.

A checklist outsources the process of remembering critical steps to a little piece of paper. Once you have ticked an item off the list, it can be forgotten immediately; and remembering what’s next is as simple as reading the next line. The cognitively taxing process of vigilance is eliminated, allowing you to focus on what’s important.

Gawande provides an example of this in the Checklist Manifesto. He discusses a hedge fund that uses checklists to great effect. The hedge fund manager implemented a checklist of tasks to complete for the first round of due diligence when evaluating a potential investment. They found that the checklist cut the time required to complete this process in half. I suspect that process speed up because the checklist reduced cognitive load.

Reduce biases

I have discussed cognitive biases in previous posts, and will dedicate future posts to the topic as well. Our brains evolved for hunting, gathering, and mating, not evaluating investment opportunities, or building international sales forces. As a result, when we rely on intuition or automatic cognitive processes there are mistakes that our mind consistently makes. For example, we pay more attention to losses than gains, and our evaluations of people are unreasonably influenced by irrelevant information (such as the person’s attractiveness). If you can identify situations where bias is likely to cloud your judgement, you can use checklists to counteract this bias.

This is the idea behind standardized job interview questions (which essentially comprise a checklist). Such procedures have been shown to reduce bias in the hiring process. Another bias - the Ostrich Effect - is also the reason that I advise my clients to set up regular meetings to evaluate any changes in known risks that they currently face (that is, run through a checklist of risks), and also take time to identify potential new risks.

What Makes a Good Checklist?

According to Gawande, the airlines have made a small industry out of designing effective checklists, and have nearly perfected the art. Although the requirements that airlines have for checklists differ somewhat from the requirements of people who are scaling their business (the two primary differences are 1) flight crews are typically quite temporary teams, and 2) the time available is often more limited for flight crews), many of the principles are more general.

Expect to improve your checklist over time

Airline checklists are tested rigorously using real flight crews in flight simulators and are modified repeatedly until they are ready to be used in actual airplanes. Even then, they are often reissued every few years. This type of rigor may not be necessary for checklists used within your organization, but you should be prepared to refine your checklists over time.

Checklists must be as simple as possible

Don’t include all steps of a process, just enough to tell you what to do. For example, the process of checking that no foreign objects have been left inside a surgical site is different for a knee surgery than it is for a heart surgery, but the checklist doesn’t need to spell that out. Keep checklists brief and straightforward. Only include items that really affect the outcomes you care about.

Make the checklist as general as possible

You want your checklist to be as useful as possible, so make it applicable to as many situations as you can. The checklist designed by the WHO team was meant to be used in any surgery. That doesn’t mean that it wouldn’t necessarily be worthwhile to design another checklist specific for brain surgeries, but if you were to do that, it wouldn’t need to include any of the items already on the general checklist.

Don’t make checklists until you need them

Making a good checklist takes time and forethought, resources that are scarce in an organization in the midst of a scaling effort. Moreover, getting people to use checklists is not always easy (a topic I will discuss in the next section). Getting people to use the checklist will be impossible if you can’t convince people that using it is necessary. So don’t bother investing in making checklists unless the process is frequent enough to warrant its use.

The Problem with Checklists

Armed with evidence that the WHO’s surgical checklist saves lives, you would expect that its usage has spread like wildfire, quickly being adopted by surgeons worldwide. Unfortunately, that’s not exactly what happened.

The problem with checklists is that they only work if people believe in them. If the intended users of the checklist don’t think it’s going to work, they’re not going to use it properly. They’ll cut corners, ticking off steps that weren’t actually completed, or complete the checklist with only some of the people who should be involved. In other words, they’ll act exactly the way they would if there were no checklist at all.

Compounding this problem, people often hate using checklists. It feels like the checklist is busting into your place of business, bossing you around, telling you how to do your job, and taking away your right to make your own choices. And of course, that’s exactly what it’s doing. That’s the point! But importantly, it’s not the whole point. The checklist is supposed to make you better at all the truly complex stuff that isn’t on that checklist, and can’t possibly be boiled down into a checklist. It’s freeing you from the tyranny of the minutia so that you can stay focused on being the master of your craft.

For these reasons, before you implement a checklist in your organization, you need to get the support of the people who are going to use it. This step has often been skipped when implementing the surgery checklist, with administrators and health policy officials dictating its use. In many such cases, the checklist has had zero effect on patients’ health outcomes. If people don’t want to use a checklist, it will not change their behaviour.

Therefore, for a checklist to be successful, follow this checklist:

  1. Before making the checklist, talk with the potential users about how and why a checklist will help. If you can’t convince them of the checklist’s value, don’t bother making one, or at least allow people to opt out.

  2. Get the checklist’s users to help design it.

  3. Test the checklist extensively in a safe environment (e.g. a flight simulator) and revise as necessary.

  4. Once the checklist is in use, encourage feedback from users.

  5. Observe the use of the checklist in action. Revise the list or give additional instructions if it is not being used the way you intended.

If you want more information on checklists, you can check out the WHO’s surgical checklist (pdf), Gawande’s website, or his non-profit dedicated to improving health outcomes for people having surgeries, women giving birth, and people with serious illnesses.

FOOTNOTES

[1] I can only assume that birds would refer to such events by a different, and arguably more accurate name. Probably something along the lines of “airplane strike”.

[2] Specifically, the hospitals used in the WHO checklist study were in Toronto, Canada; New Delhi, India; Amman, Jordan; Auckland, New Zealand; Manila, Philippines; Ifakara, Tanzania; London, England; and Seattle, WA.

**********************************

Geordie McRuer completed his PhD in Organizational Behaviour at the Rotman School of Management and is now the founding partner at Bastet Organizational Strategy, a consultancy that helps organizations successfully plan and execute their scaling strategies. You can learn more about Bastet here.

You can also follow Geordie on Twitter or LinkedIn

Self-Confidence and Arrogance are Opposites

When I talk to clients about leadership, I extol the virtues of exuding confidence. The response that inevitably comes back is “yes, but can’t you be too confident, to the point of arrogance?” I consider this a misunderstanding of definitions. Given the vitriolic, and certainly arrogant leadership of the presumptive Republican nominee in the US, I think this is a good time to differentiate the two. People typically think about low self-confidence, high self-confidence, and arrogance as existing on a continuum, with low self-confidence and arrogance at the extremes, and high self-confidence in a happy middle ground.

I prefer to think about self-confidence and arrogance in terms of “ego-involvement.” People who are highly “ego involved” spend all their time trying to sell other people (and likely themselves) an unambiguous, consistent, and almost certainly bullshit version of themselves. Observing the behaviour of these people feels like watching a performance. The goal of the performance is incessant confirmation of the person’s self-conception, even when this confirmation impedes solving more important problems. Although people who are high on ego-involvement can have low self-esteem (and therefore spend their time curating an image of unlovable ineptitude), these people do not aspire to leadership roles, and so rarely cause major problems within organizations. In my experience, almost all leaders have (reasonably) high self-regard, but often vary substantially in ego-involvement.

Leaders who are high in ego-involvement are arrogant. Leaders who are low on ego-involvement tend to be seen as self-confident. For this reason, I prefer to see arrogance and self-confidence as opposites rather than seeing arrogance as an extreme version of self-confidence. Both would score high on measures of self-esteem; they consider themselves competent and likable. But the difference in ego-involvement cause some stark differences. For example:

Arrogant leaders believe they have everything figured out already. Self-confident leaders believe they will figure things out eventually.

Arrogant leaders avoid people who will challenge them. Self-confident leaders seek such people out.

Arrogant leaders conceal their errors. Self-confident leaders discuss their errors so others can learn from them.

Arrogant leaders shoot the messenger. Self-confident leaders thank them.

Stanford’s Bob Sutton encourages leaders to have “strong opinions that are weakly held.” Although this sounds like Orwellian doublespeak, it’s actually gets to the heart of the difference between arrogance and self-confidence in leadership. To be seen as an effective leader, you need to be decisive. Leaders make decisions, and then take action, and followers expect this from their leaders. It is often the case, especially when the stakes are low, that bad decisions are vastly superior to prolonged indecision. This decisive action is the “strong opinion” part, and both arrogant and self-confident leaders typically display this decisiveness. However, the “weakly held” component separates the self-confident leader from the arrogant one.

For decisive action to be an effective strategy in the long term, the leader needs to be vigilant in looking for error, since quick decisions are often imperfect. If errors are detected, the leader must change course accordingly. In other words, strong opinions must be weakly held. An arrogant leader cannot change course, because it would indicate a flaw in the initial decision. Arrogant leaders have strong opinions that are strongly held. When arrogant leaders spot errors they have made (or more likely, have them pointed out by someone else), the leader typically sweeps this knowledge under the rug, or shoots the messenger. The goal of this response is ego defense, rather than the long-term health of the organization.

An important thing to understand about arrogance and self-confidence is that they are not just personality traits. They can also be influenced by one’s environment. If you are the leader of an organization, your arrogance can quickly trickle down throughout the whole organization. This happens by reducing “psychological safety” within your organization. I’ll be covering the research on psychological safety, how to nurture it within your organization, and what the benefits of psychological safety are in a future post.

A challenge for people who chose leaders - whether it’s a small team selecting a member of the group to lead its nascent organization, an electorate selecting a the leader of a nation, or the board of directors selecting a new CEO - is discerning arrogance from self-confidence. I think this starts by understanding that arrogance and self-confidence are opposites, rather than matters of degree. Of course, we need decisive leaders. But we also need humility, flexibility and motivation to find the best solution, rather than just the solution that the leader can take credit for. These latter criteria can never be met by an arrogant leader.

**********************************

Geordie McRuer completed his PhD in Organizational Behaviour at the Rotman School of Management and is now the founding partner at Bastet Organizational Strategy, a consultancy that helps organizations successfully plan and execute their scaling strategies. You can learn more about Bastet here.

You can also follow Geordie on Twitter or LinkedIn

 

A Startup's Guide to Negotiations

Over this week and last week, I posted a series of three posts on LinkedIn. Together, they provide startup founders an introduction to the basics of negotiation. 

The first post discusses how new employees get their first exposure to your leadership during their negotiation over their job contract.

The second post looks at the role of power in negotiations and helps founders understand and analyze power relationships.

The third post provides a step-by-step guide to planning and executing a successful negotiation.

Why Do Startups Fail?

If you’re looking for a billion-dollar question to answer, you could do worse than, “Why do startups fail?” But of course, you wouldn’t be the first to ask it. Google it, and you’re still getting perfect matches fifteen pages in. Unfortunately, those answers are wholly unsatisfying. You typically get a long list of risks that startups face (e.g. running out of money, running out of time, hiring the wrong people, selling the wrong product, selling the product to the wrong people, etc.), but no guidance on how to balance those risks.

I recently came across a spectacular list of 156 startup postmortems and I’ve been pouring over them ever since. Although you could use this to create another laundry list of risks startups face (and indeed, someone has), it still doesn’t help answer the real question.

However, I read these postmortems, not as a business analyst, or a serial startup founder, but as a researcher of human behaviour in organizations. Through that lens, I don’t just see risks, I see bad decisions. And I see a lot of them.

Seen through a behavioural lens, the primary reason for startup failure is clear: Poorly designed (or more likely, completely absent) human process architecture.

At its fundamental level, a startup is just the same as any other organization: it’s a group of people guided by a series of scripts that guide human processes, including coordination, information processing, and decision making. And if those scripts aren’t designed carefully, the group of people is left to coordinate, process information and make decisions using the scripts they showed up with. And there’s bad news about humans when it comes to these specific skills: the available scripts are shit.

This blog is about the ineffective human process scripts that evolution has left us with, how they torpedo startups with good ideas, good people, and good financing, and how well-crafted human process architecture can help your good idea survive the startup process.

If you want more content than the one post a week you’ll get here, I’ve started up a consulting firm specializing in helping startups with their human process architecture (including decision making, coordination, cultural, and motivational design). Click the Contact tab above to reach me.