Showing posts with label crisis management. Show all posts
Showing posts with label crisis management. Show all posts

Tuesday, July 10, 2012

Crisis in Outer Space! (The True Story of Apollo 13)


My 26th book will be Project: Impossible, an exploration of how people achieved goals any reasonable person would have thought impossible. This week, the true story of the the Apollo 13 mission, continuing last week’s brief history of rocketry and spaceflight. 


Thirteen

Although the United States officially “won” the space race with Apollo 11 on July 20, 1969, there would be five more missions to the Moon, during four of which astronauts walked on the lunar surface. The exception was Apollo 13, the seventh manned mission in the program.

Gordon Cooper and Donn Eisele, originally scheduled for Apollo 13, were passed over by NASA management. The flight crew operations chief selected Alan Shepard, the first American in space, to replace Cooper, but management turned him down because of recent surgery. As a result, NASA selected the backup crew for Apollo 11, who were scheduled for Apollo 14, for this mission. Jim Lovell, who had flown on two Gemini missions and one previous Apollo mission, was to be the mission commander. Fred Haise, a research test pilot, had been on two previous backup crews but had never flown in space. The command module pilot, who would stay in orbit while the other two crew members went down to the lunar surface, was Ken Mattingly. It would also be his first time in space.

Seven days before launch, Mattingly was exposed to measles (as it turned out, he didn’t get them), and was replaced by Jack Swigert from the backup team. It would also be Swigert’s first time in space.

The lead flight director, in overall command, was Gene Kranz.

Crisis in Outer Space

Liftoff for the Apollo 13 mission came on April 11, 1970, at 13:13 Central Standard Time. There was a small hiccup during the launch: the center engine in the second stage had to be shut down early because of a malfunction known as “pogo oscillation.” It had been seen in previous missions, but never so seriously. Automatic cut-offs stopped the problem before it could tear the ship apart; later missions had technical modifications to prevent a reoccurrence. In any event, the remaining engines burned longer, and the vehicle continued to a successful orbit.

Such a problem was hardly unusual. Given the complexity and inherent risk of any space mission, it would have been far more notable had the flight gone off without a hitch. Solving problems was all in a day’s work for NASA’s talented and experienced people.

But what happened next tested their capabilities to the maximum.

About 56 hours after takeoff, with Apollo 13 much closer to the Moon than to the Earth, Mission Control radioed Jack Swigert and asked him to turn on the stirring fans for the hydrogen and oxygen tanks. About a minute and a half later, there was a loud bang. The crew’s first thought was that the lunar module had been struck by a meteoroid.

What had happened was actually much worse. Number 2 oxygen tank had exploded. Later analysis would reveal damaged insulation on the wires to the stirring fan, allowing a short circuit. A large aluminum skin panel on the outside of the ship blew off, damaging an antenna and momentarily interrupting communication with Mission Control. The shock of the explosion caused a break in the number 1 oxygen tank as well. Over the next two hours, the entire oxygen supply of the service module was lost. Complicating matters even more, the fuel cells needed oxygen and hydrogen to generate electricity. The command module was left with backup battery power only.

The damaged Apollo 13 spacecraft


Landing on the Moon was no longer an option. The crew hastily shut down the command module to save its limited power and moved into the lunar module. The new project was how to get the crew back safely to Earth.

What saved the Apollo 13 mission?

The Kranz Dictum

It’s only in the movie version of Apollo 13 that Gene Kranz says the phrase, “Failure is not an option.” The real message came after the 1967 Apollo 1 disaster, in which astronauts Virgin “Gus” Grissom, Edward H. White, and Roger B. Chaffee lost their lives, Gene Krantz addressed his flight control team, establishing what would become known as “The Kranz Dictum.”
Spaceflight will never tolerate carelessness, incapacity, and neglect. Somewhere, somehow, we screwed up. It could have been in design, build, or test. Whatever it was, we should have caught it. 
We were too gung ho about the schedule and we locked out all of the problems we saw each day in our work. Every element of the program was in trouble and so were we. The simulators were not working, Mission Control was behind in virtually every area, and the flight and test procedures changed daily. Nothing we did had any shelf life. Not one of us stood up and said, “Dammit, stop!”
I don't know what Thompson's committee will find as the cause, but I know what I find. We are the cause! We were not ready! We did not do our job. We were rolling the dice, hoping that things would come together by launch day, when in our hearts we knew it would take a miracle. We were pushing the schedule and betting that the Cape would slip before we did. 
 
From this day forward, Flight Control will be known by two words: “Tough and Competent.” Tough means we are forever accountable for what we do or what we fail to do. We will never again compromise our responsibilities. Every time we walk into Mission Control we will know what we stand for. 
Competent means we will never take anything for granted. We will never be found short in our knowledge and in our skills. Mission Control will be perfect. 
 
When you leave this meeting today you will go to your office and the first thing you will do there is to write “Tough and Competent” on your blackboards. It will never be erased. Each day when you enter the room these words will remind you of the price paid by Grissom, White, and Chaffee. These words are the price of admission to the ranks of Mission Control.

The Apollo flight teams had prepared for disaster time and time again. Exercises, simulations, and extensive training all went into achieving the goal of “tough and competent.” This is an essential ingredient in effective crisis management. By preparing for different eventualities and maintaining a high level of readiness, you and your team are in the best possible position to handle a crisis.

However, no matter how good you are, failure is always an option.


Timeline of the events in the Apollo 13 crisis


Working the Problems

Crises differ from more general projects in several ways. First, they are often imposed on the project team with little or no notice. Apollo 13 was going well until suddenly it wasn’t. Crises normally have extreme constraints in time and resources. The clock was ticking with Apollo 13. If problems could not be solved in very short order, the consequences would take hold at once — with fatal results.

While NASA had an extensive supply of spare parts, machine shops, and trained engineers who could have fixed the ship easily, those resources were on Earth, and the problem was more than a hundred thousand miles away.

Evacuating and shutting down the command module was the first order of business, but there were many obstacles yet to be overcome before the crew of Apollo 13 would once again see home. There were plans for aborting an Apollo mission, but some of them were ruled out by the exigencies of the situation. The quickest way home required jettisoning the lunar module, but that was serving as the lifeboat for the crew. The service module integrity was in doubt, so they didn’t want to fire its engine except as a last resort.

That left a circumlunar option, using the Moon’s gravity as a slingshot to send the crippled ship back toward earth. To do that, they needed to make a minor course correction, but debris from the explosion made it impossible to use the onboard sextant device, requiring Jim Lovell to fly the spacecraft using the sun in the cockpit window as an alignment star.

The problems mounted. While there was plenty of oxygen in the lunar module, carbon dioxide removal required the use of lithium hydroxide canisters. While there were enough of them available, the square command module canisters wouldn’t fit in the round LM openings. An engineering team created a kludged-together system using plastic bags, cardboard, and tape, working on an extremely limited time span.

(As an aside, the duct tape and other supplies that made this possible were also part of planning for crisis management: there was a kit containing some basic utility items available for use. One can only imagine the planning that went into deciding exactly what would be part of that kit.)

Power supplies were limited. The LM was rated for two people for a day and a half, and now it would need to accommodate three people for four days. All nonessential power was shut down. Water and food were limited. The crew became dehydrated. Lovell lost 14 pounds.

The team managed to overcome one problem after another, but the toughest technical challenge came at the end of the mission. There had never been a case where the command module had to be powered up after a long sleep, and the flight controllers had to test and write new procedures to accomplish it. (In the movie, that’s the suspenseful scene in which Ken Mattingly, played by Gary Sinise, tries to find a start-up sequence that draws less than 20 watts.) The normal time for a project like that was three months; the team had three days.

By the time the Apollo 13 team reentered the command module, condensation had covered the interior with fine droplets of water. Water was inside the circuit panels as well, and the chance of a short circuit was all too real. Fortunately, the tragedy of Apollo 1 had led to various safeguards against short circuits; there was no problem.

Four hours before landing, the crew jettisoned the service module, and one hour before landing they jettisoned the LM that had served as their lifeboat. As they entered the atmosphere, the heat of reentry created rain inside the command module.

But that was the final hazard.

On April 17, 1970, Apollo 13 splashed down safely near American Samoa.

Apollo 13 Mission Control right after splashdown


Crisis Management and the Impossible Project

What distinguishes a crisis from other kinds of projects is the way it tightens the constraints. Time pressure is normally high, and the nature of the situation normally limits resources that would otherwise be available to the team. These revised constraints are normally established by the situation, not by the will or desire of the project team. In the case of Apollo 13, a procedure that would normally take three months had to be developed in three days, for the simple reason that three days was all they had. Modifying the carbon dioxide removal system would have been trivial on Earth; it was a nail-biting project in space, with only the resources available on the ship able to be used for the job.

Had the mission control team not been well prepared — had Gene Kranz not insisted on “tough and competent” — had simulations by the hundreds not taken place, it’s almost certain that the Apollo 13 mission would have ended in failure.

But that’s the point. To prepare for crisis, prepare early.

By the time the crisis occurs, it’s usually too late.

Tuesday, April 17, 2012

In re Zimmerman

Stages of a Project

In project management, we frequently say that the stages of a project go like this:

1. Enthusiasm
2. Disillusionment
3. Despair
4. Panic
5. Search for the Guilty
6. Punishment of the Innocent
7. Praise for Non-Participants

The key problem comes in Step 5, “Search for the Guilty,” as opposed to the alternative, “Fix the Problem.” The overwhelming need to punish the offender outweighs the need to fix the problem.

In re Zimmerman

These days, I’ve mostly sworn off political topics. I hate that I’ve lost friends that way. But the shooting of Trayvon Martin by George Zimmerman also raises issues relevant to project management, cognitive bias, and fallacies of logic, all of which are legitimate subjects of this blog.

As is too often the case in any sensational media matter, the professional outrage machinery is in full swing. The most extreme and definite opinions on both sides receive the majority of coverage, even though they are a minority of the population, and in response, both defenders and accusers try to delegitimize the comments and opinions of those who disagree.

My goal in writing this is not to start an argument about the relative culpability of either Zimmerman or Martin — and in fact, I’d vastly prefer not to have one. There are many other places you can go to have that argument. Right now, let's look at the nature of those opinions.

In the case of Zimmerman — or, for that matter, any criminal suspect — there are four separate questions:

  1. Should Zimmerman be suspected?
  2. Should Zimmerman have been arrested?
  3. Should Zimmerman be put on trial?
  4. Should Zimmerman be convicted?
The Decision Hierarchy

Decision-making processes are normally hierarchical. There are earlier, tentative decisions we make before we reach our final conclusion. Should we, for example, do a particular project? The initial decision may be to do a feasibility study, or a pilot test. Only when those results are in will we make our final choice and commit resources to the solution.

The legal concept of level of proof is particularly meaningful. Depending on the stage of the process, different levels of proof are required to support a given decision.

Should Zimmerman be suspected? The legal standard for suspecting someone of a crime is a low bar: reasonable suspicion. This standard is based on "specific and articulable facts", "taken together with rational inferences from those facts," to distinguish it from an "unparticularized suspicion" or a hunch. In Zimmerman's case, there's no real doubt that Zimmerman shot Martin. Even Zimmerman admits as much.

In project management, a related question is whether the environment contains opportunities or problems worthy of addressing. Coming up with a target list doesn't mean you've made a final decision, so the burden of proof is low.

Should Zimmerman have been arrested? The standard for an arrest is probable cause. That standard is higher than "reasonable suspicion," but much lower than the standard required for a criminal conviction. The best-known definition of probable cause is "a reasonable belief that a person has committed a crime." Another common definition is "a reasonable amount of suspicion, supported by circumstances sufficiently strong to justify a prudent and cautious person's belief that certain facts are probably true." 

This question, not the question of whether Zimmerman is ultimately found guilty, is the core of the current controversy. Frankly, had Zimmerman been charged and tried at the time — even if he were subsequently acquitted — it’s hard to believe that this case would have ever achieved national prominence. It’s not exactly as if shootings are an uncommon occurrence in the United States.

"Probable cause" also has a role in project management. Feasibility studies aren't free, so you have to pick your shots. Establishing a proof level equivalent to "probable cause" helps you focus on the issues that matter most.

Should Zimmerman be put on trial? Prosecutors can bring charges based on no more than probable cause, but a grand jury, which reviews the case prior to trial, is supposed to make its judgment based on the preponderance of the evidence, that is, whether is charge is more likely to be true than not true. Prosecutors also have the benefit of a more comprehensive investigation than is normally performed at the time an arrest is made. The additional information may well alter an initial determination.

A full-fledged project is the equivalent of a trial, because you have to do all the work, and in the process, you may discover surprises and risks not part of the initial process. The initial project plan rests not on final determinations, but rather on the preponderance of the evidence available in the initiating and planning stages.

Should Zimmerman be convicted? Depending on the charge, two standards may apply: clear and convincing evidence, or proof beyond reasonable doubt. The first standard applies in some civil cases; the second in criminal cases. Zimmerman clearly falls into the second category. Our adversarial legal system is designed to put the strongest burden on the shoulders of those who would convict someone of a crime.

There is a final standard of proof, proof beyond a shadow of a doubt, but that is often an impossible burden to meet, so that standard doesn't apply in a criminal case. Indeed, given that there are only two people with full knowledge, and one of them is dead, the "shadow of doubt" may never be completely eliminated.

At the end of the project, the results must necessarily speak for themselves. Did you solve the issue that led to the project? Perhaps clear and convincing evidence is all you need, but sometimes a higher burden of proof rests on the project manager.

Justified and Unjustified Opinions

If there's one fundamental American right, it's the right to an opinion, but not all opinions are equal. To have a legitimate, justified opinion, you need to have some actual facts and apply an appropriate standard of proof. As long as you're clear about what stage of the decision is currently on the table, saying "yes" to one level doesn't necessarily equate to a final determination.

In an imperfect world, armed with imperfect knowledge, we cannot escape the reality that we must necessarily make some decisions and hold some opinions in advance of all the facts. And that raises the question about what makes an opinion justified and legitimate.

It's often argued that [insert side of the argument] has already convicted [Zimmerman] [Martin], rather than wait for the judicial machinery to work its course. And, of course, some extremists have already rushed to final judgment on the matter. But the vast majority of us have not.

That doesn't mean judgments, even in preliminary stages, are inappropriate. As noted, there's no reasonable doubt of the basic fact that Zimmerman shot Martin. It's hard for anyone to argue that the police should not have looked into the matter.

Can a "prudent and cautious person" reasonably conclude, at this stage of the investigation, that Zimmerman should be arrested? Clearly, the answer is yes. Even without the discredited claim of a racial slur, the 911 call, in which Zimmerman disregarded the recommendation of the dispatcher not to get out of his car, casts a "reasonable amount of suspicion" on Zimmerman's actions.

Does the preponderance of the evidence lean in favor of putting Zimmerman on trial? The Florida prosecutor assigned to the case has determined the answer to be yes, so it's not unreasonable or unjustified for an outsider to agree with it.

It's possible, and even legitimate, to have a definite opinion on the questions of arrest and trial. It's much less justified to have a definite and final opinion about Zimmerman's ultimate guilt. The standard of "reasonable doubt" has not yet been overcome, and will not be until all the evidence — including evidence presented by the defense — is out in the open.

The claim is often made that Trayvon Martin's supporters have already convicted Zimmerman, but that's an exaggeration. It's true that Martin's supporters have generally concluded that Zimmerman should be arrested and tried, but those opinions are valid at a much lower standard of proof. It's not "convicting" Zimmerman to argue that the evidence supports arrest and trial.

Bias and Judgment

When it comes to the final decision, a prudent person should be cautious. You not only need evidence, but the evidence itself often needs to be challenged and validated. But that doesn't mean you can't hold a preliminary opinion, as long as you're willing to modify it in the face of persuasive information to the contrary.

If you rate opinions about Zimmerman on a scale of 10 (certain he's guilty) to 0 (certain he's innocent), people with scores of 10 or 0 are clearly making decisions ahead of the facts. For such people, the outcome of a trial will mean nothing: if the verdict goes their way, they knew it all along, but if the verdict goes against their opinions, it will mean that the trial was rigged and thus invalid. People who are unwilling to revise their opinions in the face of new facts aren't reasonable. Fortunately, their number isn't large.

More common is people whose opinions are 8-2, strongly convinced of Zimmerman's guilt or innocence, but not so locked into their positions that persuasive contrary evidence is incapable of changing their minds. People in this category need to be extremely careful of the actions of cognitive bias in their information processing and decision process. Even if you are trying to be open-minded, once a mind's made up, inertia takes hold. Change is difficult.

People with opinions in the 4-6 range are in less danger from cognitive bias, because their opinions are inherently more tentative. In my case, I am not paying detailed attention to the story, because of the general unreliability of most of what's published at this point. That's not the same as saying I have no opinion, or that I don't lean toward one side, but I'm fully aware that the eventual factual record may not support my tentative ideas. Even so, cognitive bias can have its effects even on people of more moderate opinion, so it's important to stay on one's guard.

But that's opinions about Zimmerman's guilt. Opinions about arrest or trial fall into different standards of proof.

The Process of Proof


After watching innumerable crime shows, we all should have a pretty good idea how the police and trial process is supposed to work. When it goes according to plan, it does a reasonably good job of establishing a factual record in support of a decision. Of course, your mileage may vary.

Even if the story seems straightforward at first glance, investigators still go through the process of reconstructing the story, gathering physical evidence, and taking initial testimony. Raw evidence, of course, is of little use until it’s processed — the body examined, witnesses interviewed in detail, a timeline reconstructed, DNA tests performed, etc. In processing evidence, investigators create a story, a timeline of events and people that ideally reveals the truth of a situation.

Stories, of course, always begin as outlines, and as they take shape and form, you can fill in greater detail. Sometimes, stories surprise you, and you find yourself in an unexpected place. Minor characters (“persons of interest”) become suspects — people with motive, means, and opportunity. Some suspects are ruled out as the process moves forward; other suspects are eventually charged with a crime.

But all that assumes police and prosecutors are doing their jobs properly. To me, the most important question is not whether George Zimmerman committed second degree murder in the shooting of Trayvon Martin, but whether he should have been arrested in the first place.

What's the Real Problem?

Questions about the process will probably not be part of the trial, because it’s Zimmerman, not the police department, who is its subject. One hopes that the Justice Department investigation will address these matters. Again, I don’t claim to know the right answers. But as a management consultant, my job is to ask the right questions. Here are some things I'd like to know.

First, was the behavior of the Sanford Police Department appropriate? To measure "appropriate," consider the following: Did the department follow established protocols? Were those protocols adequate to the situation? Did any special circumstances make it more difficult to follow the protocols? What can we do about that in the future?

If protocols were not followed, why not? Are there management or organizational issues, problems with internal culture, changes in the environment, or other factors? And if so, what can we do to address those issues? Even if the more serious allegations of political interference by Zimmerman’s father (a retired Virginia magistrate) or charges of institutional racism turn out to be true, the reflexive “search for the guilty” is a much less effective response than fix the problem.

Wasting a Crisis

I have a stronger opinion about process in this case than I have about guilt. One of the facts of management consulting is that if you have a really bad outcome, you need to adjust your process as necessary to keep it from happening again.

The Sanford police were, no doubt, shocked at the public response and degree of national interest. Regardless of the assignment of fault or blame, they have to adapt to the new reality that their work will receive increasing scrutiny in the future. The 1982 Chicago Tylenol murders were clearly not the fault of the manufacturer, but the ubiquitous tamper-evident seals on all that we eat or drink date from that crime. And if they truly are at fault, then there’s all the more reason to work harder and better in the future.

Too much emphasis on determining guilt can, unfortunately, detract from the more important matter of change. In the cognitive biases series, I covered my version of the “Semmelweis Effect," the reality that peoples’ views harden when you accuse them of terrible crimes. Moral indignation can backfire, and that does no one any good.

Because it’s so easy to identify the most inflammatory and outrageous pieces on either side, it’s equally easy to miss the large amount of reasonable, proportionate commentary—also on both sides. A crisis, as has been noted, is a terrible thing to waste. That doesn’t mean we want a crisis or welcome it when it comes, but if we waste the crisis, we’re often doomed to experience it again. That’s the worst possible outcome.

It’s clear that something went terribly wrong in the Trayvon Martin case. Less clear is what went wrong, and the purpose of trial and investigation is to establish the narrative in an authoritative manner. Whether the trial and investigation accomplish that goal remains to be seen.

But it’s truly the most important part of the matter. Because once we answer that question, we can move forward to the two questions that matter in the long run:

Why did it happen?

And how can we keep it from happening again?