Painted into a Corner: The crash of XL Airways Germany flight 888T
On the 27th of November 2008, an Airbus A320 undergoing a check flight at the conclusion of a lease agreement plunged into the Mediterranean Sea on approach to Perpignan, France, killing all 7 people on board. The plane was due to be transferred from European charter carrier XL Airways Germany back to its owner, Air New Zealand, who wanted to check that its systems were working properly — only to discover, in tragic fashion, that they were not. As the pilots attempted to test the plane’s low speed protections while on approach, the protections failed and the airplane stalled, plummeting to its doom just 3,000 feet below. Investigators would later piece together a bizarre and complex sequence of events leading up to the disaster, beginning three days earlier when the aircraft was repainted in Air New Zealand livery, which precipitated the simultaneous failure of two critical sensors, disrupting the Airbus’s fly-by-wire logic right as the crew attempted a test maneuver at a dangerously low altitude. The findings raised questions about the judgment of all involved, from the painting crew who rinsed the aircraft with a high-pressure fire hose, to the pilots themselves, who initiated a check of a critical system without considering what they would do if it didn’t work. This is their story — and an account of the lessons to be learned from their fate.
◊◊◊
It is an increasingly prevalent truth that modern airlines don’t own their own airplanes — they lease them instead. Leasing airplanes instead of buying them reduces up-front costs and gives airlines the ability to quickly adjust their capacity to meet demand, making it an attractive option for companies in volatile market niches. At the same time, airlines that do own their aircraft can take advantage of the system by leasing their excess aircraft to other airlines, allowing the airplanes to keep making money even when demand is low. Short-term leases can come complete with pilots and even cabin crew, but longer term leases are usually “dry,” covering only the airplane, which may be fully converted to the lessee’s specifications.
In 2005, Air New Zealand purchased several brand new Airbus A320s, of which it immediately leased three to its own low-cost subsidiary, Freedom Air International. Freedom Air operated the planes for about a year, until Air New Zealand decided to sub-lease two of them — one to TAM Airlines in Brazil, and another to Frankfurt-based charter airline XL Airways Germany. Owned by an Icelandic investment bank, the airline and its sister company XL Airways France catered toward inclusive package tours to holiday destinations around the Mediterranean and North Africa, a relatively fickle business model that made leasing airplanes more attractive than buying them.
Under the leasing agreement, the airplane was repainted in XL Airways livery and was given the new German registration D-AXLA, which it would retain until the contract expired in November 2008.
D-AXLA spent an uneventful two years at XL Airways Germany, ferrying German tour groups to and from places like Sharm el-Sheikh, Marrakesh, and Heraklion. However, it was ultimately decided not to renew the contract, as XL Airways Germany planned to ditch its fleet of three leased A320s to focus on its remaining fleet of Boeing 737s (although it’s unclear whether this was the cause of the contract non-renewal, or a result of it). In any case, D-AXLA was withdrawn from service on November 3rd, 2008, at which time the end-of-contract clauses kicked in. According to the agreement with Air New Zealand, the airplane was to be returned on November 24th (later pushed back to November 27th), following combined 20-month and 40-month airframe inspections, a complete inspection of both engines, repainting in Air New Zealand livery, and an “acceptance flight” or “check flight” to verify the functionality of specified systems.
XL Airways Germany contracted out this maintenance work to a company called EAS Industries, based at Perpignan-Rivesaltes Airport in the coastal French city of Perpignan, near the Spanish border. The aircraft was ferried there from Frankfurt on November 3rd, and the inspections, lasting several weeks, proceeded as planned.
The part of this work most significant to this story was the repainting of the aircraft in Air New Zealand livery. XL Airways had painted the plane white with large, blue “XL” logos on the fuselage, tail, and engines, while Air New Zealand’s color scheme called for a slightly different shade of white, a blue-green tail with a white logo, and the words “Air New Zealand” on the forward fuselage above the window line, along with other minor differences. The work was to be carried out by a team of EAS Industries painters who had undergone an aircraft familiarization course.
Because the amount of paint needed to cover something the size of an A320 is considerable, a coat of paint weighs more than you might think. Too many new coats will start to incur a noticeable performance penalty, so the repainting process normally involves soaking the airplane in chemical paint stripper to remove prior coats, except for composite surfaces, which must be sanded.
In the case of D-AXLA, EAS Industries developed a repainting timeline based on Airbus’s official repainting procedure. This procedure called for a number of steps which boiled down to the following: 1) protect critical sensors, antennas, and other sensitive areas using chemical-resistant polyethylene film and adhesive aluminum barriers; 2) clean the aircraft; 3) strip the paint; 4) rinse the aircraft; 5) sand off any remaining paint; and 6) apply new primer and paint. These activities, and intermediate steps, were carried out in stages between November 5th and November 24th.
As called for in the procedure, the painting crew took care to protect the aircraft during the application of stripper and paint by fitting specialized covers over key sensors. The specified polyethylene and aluminum adhesive barriers were then installed in the gaps to ensure that chemicals couldn’t get underneath the covers.
Meanwhile, the team found that two different iterations of XL Airways livery had been applied on top of the former bright orange livery of Freedom Air International, and it was soon discovered that the available paint stripper was insufficient to completely remove all traces of this hidden, original layer. Consequently, the painting team had to sand off a large number of sticky orange patches every time they stripped a new part of the aircraft. The fuselage itself was stripped, sanded, and painted first, and then the same process was repeated for wings. However, sanding the tops of the wings generated a large amount of dust that rose up and settled on the newly repainted fuselage, which was undesirable, because the technical lettering — registration numbers, warning placards, and so on — had yet to be applied. Consequently, a decision was made to rinse the dust off the airplane before applying the technical lettering on November 24th.
According to the official painting procedures, removal of dust prior to application of a final coat should be done by hand with “anti-dust cloths.” Separately, provisions existed for rinsing the airplane with water at various stages of the process, during which all protective coverings were to remain in place. But wiping dust off the entire airplane with a cloth would be very labor-intensive, and the schedule was coming down to the wire, so the team opted to give the plane a rinse instead. The manual contained detailed rinsing procedures that specified the maximum water pressure and flow rate of the hose, the maximum angle of application of the stream, and other parameters — all very esoteric to the painting crew, no doubt. Instead, they apparently decided that it would be much faster and easier to just rinse the airplane outside the hangar using a fire hose for 15 to 20 minutes. It probably never occurred to them that spraying water on the plane required following any special procedures. After all, it was just water, right?
The fact is, however, that the procedures exist for a reason, regardless of water’s docile reputation. And the most important part of the procedure is to keep the protective covers and seals over the critical sensors, because, as it turns out, getting water in them is bad. This fact was apparently not appreciated by the EAS Industries team, who sprayed down the entire aircraft without ensuring that all of the protective covers were in place.
Official documents don’t make it clear whether there were no covers on the aircraft at all, or only some, but what is known is that no protection whatsoever had been applied to at least two of the aircraft’s three angle of attack sensors.
The angle of attack is the angle of the airplane’s lifting surfaces relative to the oncoming airflow. For example, if the pitch of the airplane is 10 degrees nose up, but the plane is climbing on a slope of 6 degrees, then the angle of attack is 4 degrees. This parameter isn’t directly used by the pilots during the course of a normal flight, but it’s nevertheless critical to aircraft performance, because a higher angle of attack makes the wings generate more lift at any given speed. Furthermore, at very high angles of attack this relationship breaks down, causing the aircraft to lose lift and stall, which makes knowing the angle of attack very important if you don’t want to crash.
In practice, the person whose job it is to know the angle of attack is usually a computer, not a pilot. On most aircraft, if the angle of attack gets too close to the stall threshold, then it becomes the pilot’s problem, so the computer will generate a stall warning (typically by shaking the pilots’ control columns). On the Airbus A320, however, things work a little differently.
The A320, like all modern Airbus models, is a fly-by-wire aircraft, which means that pilot inputs using a side stick are fed to a bank of computers that calculate how to move the control surfaces in a manner and magnitude appropriate to the flight conditions at that particular instant. This makes the A320 rather different (and on balance, easier) to fly than traditional airplanes with which the public is more familiar. For example, if you push your Airbus side stick all the way to the right as fast as you can, then the flight control computers will deflect the ailerons sufficiently to achieve a specified maximum right roll rate, but no more. How much deflection is required to achieve that rate varies with airspeed (and other parameters), but the pilot doesn’t have to think about that, because the computer compensates for it automatically. Non-fly-by-wire aircraft address this issue by increasing the force required to produce a given deflection as speed increases, but there is an undeniable simplicity to the particular way Airbus does it.
The fly-by-wire system described above also allows for the incorporation of “flight envelope protections” designed to prevent the pilot from making any input that would result in a loss of control. Protections exist in all axes, so there’s a maximum bank angle, a maximum negative pitch, a maximum load factor, and so on. And most importantly for this story, there is a maximum angle of attack, located some healthy margin below the point at which the aircraft will actually stall, and which cannot be exceeded no matter how hard the pilots try, in theory. Put a big asterisk on that.
The way the angle of attack protection system works is as follows.
First, three sensors on the exterior of the aircraft measure the angle of attack (henceforth, AOA) in real time. This data is constantly fed to three Air Data Reference (ADR) computers, one for each sensor, which among many other tasks process the AOA data and determine its validity. The three ADRs then transmit this data to the two redundant Flight Augmentation Computers (FACs), which compare the three sources of data in order to verify their consistency. If the data are consistent, then the FAC combines the three measurements into one aircraft-wide AOA value, which it then uses to calculate a whole host of secondary parameters. This AOA value is also checked against the thresholds that will trigger the flight envelope protections — and in particular, two values called “alpha prot” and “alpha max.” At alpha prot, a system called “alpha floor protection” engages, automatically increasing engine thrust and preventing further pitch up inputs. The pilot can however disengage the alpha floor protection. If the AOA then continues to increase, it could reach alpha max, which is a hard ceiling. At that point the flight control computers will command whatever control inputs are necessary to prevent the AOA from increasing past alpha max, and will ignore any attempt by the pilot to penetrate it.
All of these protections ultimately rely on the correct functioning of the angle of attack sensors themselves. The A320 has three sensors, two mounted on the left side of the fuselage and one on the right side, just aft of the forward doors. Each sensor consists of a rotating vane, which, kind of like a wind sock, orients itself to point “downwind,” although in this case the “wind” is the air flowing past the airplane as it flies. A resolver measures the position of the vane, and then the ADR turns that position into an AOA value.
Each AOA vane is attached to a shaft that rotates freely within a set of bearings. These bearings must be almost entirely free of friction at all times in order to ensure that the vane can align itself exactly downwind of the oncoming airflow. If contaminants enter the bearings, or if the bearings corrode, then the AOA vane could stick and start transmitting bad data, which is obviously undesirable. Certification regulations therefore require AOA sensors to be resistant to penetration by fluids, such as rain flowing across the fuselage. This is achieved by means of a labyrinth mounting designed to complicate the flow of liquids into the interior of the sensor. On an unprotected sensor, water would simply flow straight into the gap between the rotating vane and the fixed faceplate beneath it. A labyrinth mounting, by contrast, forces flowing water to travel around a series of corners on its way in, sapping of it of energy before it can come into contact with the bearings, as shown above.
The AOA vanes installed on D-AXLA were manufactured by B. F. Goodrich and had met European and American requirements for resistance to water penetration, but they were not impermeable. During testing, the sensors were subjected to a jet of water applied at the most adverse angle, from a distance of 2.5 meters, for 15 minutes, with a flow pressure of 200 kPa and a flow rate of 450 liters per hour. After the test, the sensor was disassembled, and no water was found inside. This was more than adequate to prove that rain would not get inside the sensor in flight, and indeed, there are no known reports of this happening.
However, the sensors were not designed to prevent penetration by more powerful jets of water, such as, hypothetically, a fire hose. In fact, the fire hose used to wash D-AXLA on November 24th, 2008, had a flow pressure of 300 kPa and a flow rate of 5,500 L/h, far above the values used during certification testing. This extra power and volume was sufficient to propel water all the way through the labyrinth mounting and into the bearings, where it became trapped. The only way to detect this water would have been to disassemble the AOA sensors, which would not normally be done unless a problem was reported in flight. And so the water sat there for the next three days, awaiting its unsuspecting victims.
◊◊◊
With the repainting process complete, the check flight was now the only task remaining before Air New Zealand could take possession of the airplane. Air New Zealand company policy for lease agreements was to conduct a check flight before transferring a plane to a lessee, and another when it was returned, in order to demonstrate that the airplane was in good working order. When Air New Zealand leased D-AXLA and its sister ship in May 2006, it was the first time the airline had ever leased out an Airbus A320, so the company needed to create a check program for this aircraft type that was satisfactory to both the lessees and to itself as the lessor. Considering that the A320s were essentially brand new, Air New Zealand and XL Airways Germany were content to perform some simple, cursory checks of the airplanes’ handling, but TAM Airlines, the other lessee, insisted on a full check program based on the test regime used internally by Airbus to assess in-service aircraft. In response, the Air New Zealand Technical Supervisor asked Airbus for a copy of its In-Service Aircraft Test Manual. But while Airbus did agree to grant Air New Zealand a copy of the manual, it came with a confidentiality clause preventing them from actually using it, for liability reasons. Instead, Air New Zealand decided to draft a check program based on Airbus’s non-proprietary “Customer Acceptance Manual,” or CAM, which was not designed as a testing regime, but rather was used by Airbus test pilots to demonstrate features of the A320 to customers prior to delivery.
Picking and choosing various procedures from the CAM, Air New Zealand drew up an “Operational Flight Check” document for the A320, which it submitted to TAM Airlines and XL Airways Germany for approval. The checks that Air New Zealand chose to include were intended to standardize the program with its existing program for the Boeing 737, but the door was left open for the lessees to request additional checks prior to transfer of the airplanes. Subsequently, discussions between XL Airways and Air New Zealand representatives in Christchurch, New Zealand resulted in the addition to the Operational Flight Check document of a procedure called “low speed configuration full.”
The idea behind this check was to demonstrate the performance of the earlier-described angle of attack protections while in landing configuration with the landing gear and flaps fully extended. Note the word “demonstrate,” because as it was conceived by Airbus in the Customer Acceptance Manual, it was a demonstration, and not a test of functionality. Its purpose was never to verify that the protections were working following maintenance to the aircraft, but it was going to be used that way by Air New Zealand and XL Airways Germany.
According to the protocol, the check was to be accomplished while in descent to the airport at an altitude between 14,000 and 10,000 feet. After extending the landing gear and flaps, the pilot will reduce engine power to idle, then pull back on the side stick to pitch the airplane up, aiming to reduce the airspeed at a rate of one knot per second. As the airspeed falls, lift must remain constant, causing the angle of attack to increase. This increase is allowed to continue until the AOA reaches alpha prot, at which point the alpha floor protection will kick in to increase engine thrust. The pilot should then disconnect the alpha floor protection, but should not allow the AOA to reach alpha max. Once the correct disengagement of the alpha floor is observed, the pilot can pitch down again and restore engine power to end the maneuver.
A few aspects of this procedure are worth analyzing in greater detail.
The first thing to note is that the pilots normally observe the approach to alpha prot and alpha max based on airspeed, not angle of attack. At a given altitude and in a given configuration, airspeed and angle of attack have a predictable relationship, such that the Flight Augmentation Computers can calculate the airspeed at which a particular AOA value will be reached. Thus, the airspeed at which the AOA will reach alpha prot is called Vprot, and the airspeed at which the AOA will reach alpha max is called Vmin. A third speed, VLS, stands for “lowest selectable speed;” it lies somewhere just above Vprot and means what it says on the tin. These three speeds are collectively known as limit speeds. All three are prominently displayed on the pilots’ airspeed indicators, where a red bar rises up from the bottom of the instrument to Vmin; a dashed orange bar rises up further from Vmin to Vprot; and a hollow orange line continues from Vprot to VLS, as shown above. In order to follow the “low speed configuration full” check procedure, the pilot will anticipate the approach of alpha prot and alpha max by watching these limit speed indicators.
The second thing to be aware of is the expected behavior of the aircraft’s stabilizer trim system during the maneuver. As frequent readers of my articles have probably gathered, the stabilizer trim is used to balance the aircraft at a particular pitch angle without the need for the pilot to apply constant pressure to the controls. On a conventional aircraft, if you pull back using the control column, the plane will pitch up, but when you let go, it will pitch back down to wherever it was stable before, unless you adjust the stabilizer trim setting to match the new, higher pitch angle. On the A320, however, the controls work rather like those in a simple flying video game: when you pull back on the side stick, the airplane pitches up, and it stays there when you let go. If you want to pitch back down again, then you grab the stick again and push forward until you’re happy. The A320 does this by engaging its autotrim system, which automatically adjusts the stabilizer trim position to match the pitch angle commanded by the pilot, without the pilot having to think about it at all.
When the AOA approaches alpha prot, however, the stabilizer trim will reach the “trim stop,” and further nose up inputs by the pilot will no longer result in further nose up movement of the stabilizer. The check procedure called for the pilots to observe when the trim reaches the trim stop, something that’s worth keeping in mind as we get closer to the climax of this story.
◊◊◊
As the date of the check flight approached, XL Airways Germany appointed a crew consisting of 51-year-old Captain Norbert Käppel, who was the airline’s air and ground operations manager, and 58-year-old First Officer Theodor Ketzer; the pair each had about 12,000 hours of experience. At the same time, several New Zealanders made their way to Perpignan to help check out the airplane, including 52-year-old Air New Zealand Captain Brian Horrell; three Air New Zealand engineers; and a representative of the New Zealand Civil Aviation Authority.
On November 25th, Captain Käppel contacted EAS Industries to see if they would arrange the paperwork for the check flight on November 27th, a capability that they possessed. However, neither the European Union operations handbook nor XL Airways’ own operations manual contained any official designation for an “acceptance” or “check” flight associated with a leasing agreement, so the terminology relating to the matter was a little bit up in the air. As a result, it so happened that Käppel conveyed the impression that the planned flight was a “training flight,” which EAS Industries stated was outside their purview.
Had the flight been conveyed as a test flight, EAS Industries could have submitted a request for a test flight to the Coordination Unit of the French Flight Test Center, which would in turn have arranged an airspace block for purposes of the test flight under the supervision of a dedicated test controller. The details of the test flight would then have been worked out between the captain of the flight and the duty officer at the Flight Test Center. In France, this is the only way to carry out a “test flight.” But Captain Käppel was unaware of this, and due to the miscommunication with EAS Industries, he instead instructed XL Airways Germany’s flight operations department to file two regular flight plans for “training/ferry” flights, one departing from and returning to Perpignan, and a second from Perpignan back to Frankfurt. The first of these was intended to be the check flight.
Two days later, on the 27th, the crew awoke at 4:30 in the morning and flew as passengers from Frankfurt to Perpignan, where they put in place final preparations for the check flight and return flight to Frankfurt later that day.
At around 11:00, Captain Käppel called the Perpignan control tower to discuss the flight plan that had been filed.
“Question: where do we make the test flight?” he asked. “Do you have an airspace that allowed us to do the test flight?”
“Okay, normally the only test flight which are allowed are test flight with level steady route like if it was a normal flight, you know?” the controller replied. “And it will be done with the ACC [Area Control Center], err, probably the Bordeaux ACC. Okay, just wait a minute, I have a look on your flight plan for test flight this afternoon, I suppose that EAS did it, err, consulting…”
Here the controller revealed her belief that EAS Industries had filed the flight test paperwork with the Flight Test Center, which they had not. Captain Käppel never picked up on this nuance, and the Perpignan controller subsequently told him that “this type of flight should be accepted by the ACC.” Neither party recognized that XL Airways Germany lacked authorization to conduct a test flight in French airspace.
Nevertheless, the snowball kept rolling, and by 14:30 — two hours behind schedule — the team was ready to carry out the flight. Given the delay, it was decided that they would approach Perpignan Airport and do a touch-and-go before continuing to Frankfurt, instead of coming to a full stop and taking off again. That meant that the Air New Zealand engineers and the CAA representative would need to be on board in order to return to Frankfurt, riding as passengers with no particular duties. Meanwhile, Air New Zealand’s Captain Horrell sat in the cockpit jumpseat to oversee the checks with Captain Käppel and First Officer Ketzer.
At 14:33, flight 888T — the T stood for “test” — took off from Perpignan and began climbing to the northwest, headed for the Bay of Biscay. The crew immediately rattled off the first few check items, including timing the landing gear retraction, without any issues. From there the flight plan called for a climb to 31,000 feet, where they would perform checks of the communication, navigation, and flight control systems, followed by a further climb to 39,000, where they would check the functioning of the auxiliary power unit (a backup generator) at its service ceiling. Checks of the high and low speed protections were scheduled for the descent.
However, at 14:48, as flight 888T climbed into high-level airspace controlled from the Bordeaux Area Control Center, the controller there spotted the flight and immediately became suspicious. Earlier that same day, an XL Airways Boeing 737 had also attempted to conduct an acceptance flight out of Perpignan under a regular flight plan, and the controller found that it was burdensome to accommodate their requested maneuvers without a dedicated testing area. He immediately recognized that flight 888T was the same, and he called Perpignan tower to clarify why he had been sent another “disguised test flight.”
Seconds later, the crew reached the stage of the check program that called for tests of the bank angle limitations, which would require altering their course. After initial greetings, Captain Käppel asked, “Triple eight tango, just be advised, could we maintain flight level one eight zero on heading three three zero and do one or two three-sixties? Because we are an acceptance flight, so it could be nice if you could give us some airspace to do a few procedures.”
“I call you back sir,” Bordeaux replied. In the meantime, the crew verified some of the navigation systems.
At 14:52, the controller returned. “Stardust triple eight tango,” he said, using XL Airways’ callsign, “I’m afraid we cannot do test flights in general air traffic, you have to be OAT sir for that.” In this case, general air traffic is distinct from “operational air traffic” (OAT), in that the former represents aircraft engaged in normal transport activities, while the latter requires special treatment. A test flight is normally operational air traffic, with its own airspace and controller set aside by the Flight Test Center, so as not to interfere with general air traffic.
After a brief interruption to work out their clearance, Käppel tried again: “Please bring us to an airspace where we can do, err, few exercise just for about ten minutes,” he asked.
“I am asking sir, but our rules are that you cannot do test in your flight, we can only deal with regular flight plan sir,” said the controller.
Reluctantly, Captain Käppel accepted the controller’s prohibition. Instead, they would simply have to do whatever checks they could get ATC to authorize, and find some other way to accomplish the rest.
Abandoning the precise order and timing of the check program, the crew began to improvise. After finishing the navigation and communications checks, they decided to head back to Perpignan, since there was no point going all the way to the Bay of Biscay if ATC still wouldn’t authorize maneuvers when they got there. But that didn’t mean they wouldn’t try to complete all the checks anyway. As they turned around, they performed the bank angle tests while complying with an air traffic control clearance, confirming that the protections were working. ATC was left none the wiser. And after that, they asked to climb very briefly to 39,000 feet on the way back to Perpignan in order to test the auxiliary power unit — a request that once again rubbed the controller the wrong way. “I understand sir, but you know you are on general aviation and many sectors above, so it’s a bit more difficult for us,” he said. At the center, the controller was forced to repeatedly confer with neighboring sectors to ensure they wouldn’t cross paths with higher level traffic, but the request was ultimately granted.
Flight 888T remained at 39,000 feet for only about two minutes, just long enough to complete the check, and then they began their descent into Perpignan. Prior to reaching 25,000 feet, they carried out the overspeed test, pitching down and accelerating until the overspeed warning sounded, before pulling back; the check went off without a hitch. After that, they signed off with Bordeaux, who left them with a choice admonishment: “Next time, sir, it’ll be better not to do your flight in general aviation, it’s a bit complicated for us,” he said.
“Yes, we know, but we had actually planned our flight plan like this with your local ATC, so sorry for that, and there was misunderstanding, we’ll do differently next time,” said Captain Käppel.
Next, continuing their descent, they completed checks of the bleed air system, then they tested the flight controls in alternate law — more on that later — by switching off both FACs, which also worked as expected. But by this point, they were rushing, struggling to jam all the tests into a reduced timeframe due to ATC’s denial of a dedicated test area. By the time Captain Horrell brought up the low speed test, they were already descending through 12,000 feet, below the 14,000-foot altitude where it was supposed to begin.
Horrell briefed the low speed test in moderate detail. “Low speed is, I’ll read through,” he said. “First it will be landing gear down, flaps go to full, adjust engine power, stabilized… when the aircraft speed is stable and set power at idle, it’ll be manual, and adjust the pitch to maintain deceleration around one degree per second. During deceleration…”
“You want to go all the way down to VLS and alpha prot, right?” Captain Käppel asked.
“Yes,” Horrell continued. “So when you get to the bottom of VLS, you have to pull quite hard to make it go.”
“Yeah, I know, yeah,” said Käppel.
“And then it will go into the alpha floor,” said Horrell. “You need to pitch forward, and err… you’re happy with the disconnect and reengage, and out of alpha floor, that’s it we’re into landing.” Notably, he did not mention what airspeeds corresponded to VLS and Vprot, even though these figures were available in a table at the bottom of the program sheet.
Moments later, flight 888T maneuvered for approach and was cleared down to 6,000 feet. At that point they entered clouds, prompting Captain Käppel to suggest, “I think we will have to do the slow flight probably later.” He understandably was uncomfortable testing the flight controls when he couldn’t see the ground.
“Okay, yeah,” said Horrell.
“Or we do it on the way to Frankfurt, or we even skip it,” Käppel continued. “Sorry.” Horrell didn’t acknowledge the suggestion. It’s worth noting that the check could not have been done on the way to Frankfurt because they were running out of daylight, and they were not allowed to attempt it at night.
However, two minutes later, now on their way down to 2,000 feet, they broke through clouds, and Käppel said, “Down below the clouds, so you want what?”
“To go slower, you mean?” said Horrell.
“Yeah.”
“We need to go slow with err, recovery from… recovery,” Horrell said.
Captain Käppel called for gear down, then asked, “What’s the speed?”
“Just… come right back to the alpha floor activation,” said Horrell.
First Officer Ketzer extended the flaps to the landing configuration, and Käppel leveled off at 3,000 feet to begin decelerating. “We are reducing,” he announced.
“I will say when the trim stops,” Horrell added.
None of the crew knew that they were about to make a terrible mistake.
◊◊◊
Let’s return for a moment to the water in the angle of attack sensors. It had been there since November 24th, and was still there when flight 888T departed, but no anomalies with any systems were observed. In fact, the water initially had no effect at all. But by the time the flight leveled off at 31,000 feet, the outside air temperature had fallen to a frigid -54˚C, and the trapped water started to freeze. The sensor heaters were turned on, as is standard procedure, but the heating system was only designed to keep the vanes themselves free of ice. The bearings where the water had become trapped were not heated at all, so when the water froze, so too did the bearings, preventing the №1 and №2 AOA vanes from moving. The №3 vane was apparently unaffected.
Interestingly, the fact that two of the AOA sensors were frozen initially went unnoticed even by the flight computers themselves. Although the values transmitted by the two vanes had stopped changing, these values were consistent with the airplane’s actual angle of attack in cruise flight. The working №3 sensor also continued to transmit similar values, so the data passed all of the flight augmentation computer’s consistency checks, too.
This state of affairs began to change as flight 888T descended toward Perpignan — but only subtly. During one of the maneuvers, the airplane’s actual angle of attack, recorded by sensor №3, diverged from the frozen, constant value transmitted by sensors 1 and 2, triggering the FAC’s spidey senses. But the way the FAC deals with divergent AOA values is to take the values from the three sensors and eliminate the sensor whose reading is furthest from the median value. Thus, because sensors 1 and 2 were recording identical values, the FAC considered sensor 3 to be the deviant one, and so the computer “rejected” the only working AOA sensor. And because the №3 AOA sensor is a backup only — sensors 1 and 2 are primary — this rejection was not brought to the attention of the crew.
The consequences of this chain of events began to build as Captain Käppel decelerated for the low speed test. As their speed dropped, the angle of attack increased proportionally, but the AOA recorded by sensors 1 and 2 remained frozen at the same value as before. Fascinatingly, the only immediate effect of this divergence was that the flight computers began to underestimate the weight of the airplane. In flight, the aircraft calculates its own weight by solving for weight as a variable in a performance equation that includes angle of attack, airspeed, descent rate, and other parameters. Basically, since the plane was maintaining altitude, its lift must have been constant. However, its airspeed was decreasing, which means less lift, so normally the AOA would have to increase, generating more lift to compensate. But the measured AOA wasn’t changing — so the only way to make the equation work was to assume that the airplane’s weight was rapidly falling! In fact, within a short period, the calculated weight dropped so far below the weight predicted for this phase of flight, taking into account fuel burn, that a “Check GW” [gross weight] message appeared on a secondary cockpit display, but if the pilots even noticed, they likely could not have comprehended its significance.
However, the frozen AOA values soon generated another, more immediately dangerous effect on the pilots’ limit speed indications. As you will hopefully recall, the limit speeds of Vprot and Vmin correspond to the airspeeds at which the FAC predicts the AOA will reach alpha prot and alpha max, respectively. (VLS, for its part, is fixed for a given configuration and was unaffected). Now, can you imagine what might happen to these indications if the AOA reading isn’t changing? You can think of it something like this. If the AOA reading is frozen, then the difference between the current AOA and alpha max is a constant. Therefore, by the transitive property, the difference between the current airspeed and the calculated Vmin (or Vprot) is also a constant. The problem then is that the current airspeed isn’t frozen; in fact, it’s falling. And so, as it falls, the calculated values of Vprot and Vmin (or current airspeed minus the constant) fall at exactly the same rate, perpetually receding ahead of the actual airspeed all the way down to zero. Now also recall that Air New Zealand’s Captain Horrell did not brief the expected value of Vprot — he simply told Captain Käppel to decelerate through VLS and wait for the alpha floor protections to kick in. At this point you’re probably already realizing what’s about to happen.
Indeed, knowing everything I’ve just explained to you, it shouldn’t come as a surprise that when Captain Käppel pulled back on his side stick, he watched the speed drop at the prescribed rate of one knot per second, lower and lower, without ever getting any closer to alpha prot, even as they reached and then passed through VLS. Seconds later, their airspeed reached where Vprot should have been, but because the angle of attack readings were still frozen, the alpha floor didn’t engage, and engine power did not automatically increase.
At this point, Käppel had been maintaining his nose up input on the side stick long enough for the autotrim system to move the stabilizer trim all the way to the fully nose up position. (Remember how we had that whole discussion about autotrim?) Sensing that the trim had reached the stop, Käppel returned his side stick to the neutral position — but the stabilizer trim remained where it was, because it was designed to hold the last pitch attitude commanded by the pilot. If he had then pushed forward on his side stick, the trim would have begun moving again toward nose down, but he didn’t — at least not yet.
At almost exactly the same time as the above, their airspeed plunged through Vprot and reached Vmin. Any slower, and the AOA would increase beyond the stall threshold, and they would fall from the sky. The flight envelope protections would normally prevent the AOA from exceeding alpha max, thus preventing the airspeed from falling below Vmin, but the AOA readings were still frozen at a completely comfortable value, so nothing happened. Instead, the working №3 AOA sensor detected that the AOA had exceeded alpha max, and sent a signal to the flight computers to generate an aural stall warning. This capability remained despite the fact that the FACs had rejected that sensor as inaccurate — a deliberate design decision made with exactly this situation in mind. Consequently, an automated voice began to call out, “Stall! Stall!” over and over, accompanied by a cricket alarm, but none of the flight envelope protections activated.
As soon as he recognized the stall warning, Captain Horrell shouted, “Stop!” At the same time, Captain Käppel advanced the thrust levers to takeoff power, but as the engines spooled up, their airspeed bottomed out at just 92 knots, and the airplane began to stall. As the stall developed, air ceased to flow smoothly over the wings, which also meant that the airplane became unstable in roll, resulting in a sudden bank to the left.
As the plane slewed left, the A320’s entire flight control system suddenly switched from “normal law” to “direct law” — instantly transforming its entire handling philosophy.
Earlier in this article, I took the time to describe how control inputs on the Airbus command a rate of pitch or roll, and computers work out how much to deflect the control surfaces to achieve the desired rate. I also described how flight envelope protections limit the maximum bank angle, AOA, and so on. But all of that only applies when the fly-by-wire control system is in what’s called “normal law” — which it is essentially all of the time. But if certain sensor or computer failures occur, it will no longer be possible to maintain some or all of these advanced features. These certain failures could cause the flight controls to enter “alternate law.” This law really contains several further sub-laws, but in general alternate law means that pitch control works as normal, but there are few or no flight envelope protections, and roll inputs become directly proportional with control surface deflection, without computer modification. But other types of failures may not allow even these residual functions, in which case the flight controls can enter “direct law,” where side stick movement becomes proportional with control surface deflection in all axes. That means that in direct law, the A320 suddenly starts flying rather like a conventional airplane, albeit without realistic force feedback.
In the case of flight 888T, the left roll during the stall resulted in locally different airspeed readings on the left and right sides of the aircraft. Divergent airspeed readings normally cause the flight controls to enter alternate law, but if the landing gear is extended, this will cause the controls to enter direct law instead, for reasons that are beyond the scope of this article. Therefore, since the landing gear was extended for the low speed, full configuration test, direct law suddenly became active right at the moment of the stall. The change in control law was prominently annunciated on the alert display and was accompanied by a master caution warning, but amid the chaos of the moment, the flight crew probably didn’t notice.
Perhaps the most insidious side effect of the switch to direct law was that the autotrim system became unavailable. That meant that the stabilizer trim would remain wherever it was set at that moment, which happened to be full nose up. To get it out of full nose up, Captain Käppel would have needed to use the backup manual trim system, which is used in-flight only in an emergency. Orange text reading “USE MAN PITCH TRIM” appeared on the pilots’ primary flight displays in order to remind them, but unfortunately, they must not have seen it.
Instead, Captain Käppel simply applied the normal stall prevention maneuver, which is to add takeoff power and hold the pitch angle between 0 and 5 degrees. Normally, this should prevent the angle of attack from exceeding the stall threshold, and in fact, for a brief moment it appeared to work, as the stall warning stopped and airspeed started increase. But when Käppel pushed his side stick forward to lower the pitch, he would have expected the stabilizer trim to move with it, as it would during normal law. The controls were, however, in direct law at that moment, so the trim instead remained at full nose up, actively fighting his attempts to pitch down. And within seconds, the very nose high trim setting and the very high engine thrust — which also confers a nose-up moment — combined to send the plane hurtling into a steep and unsustainable climb. Käppel again pushed forward on his sidestick to counter it, but the stabilizer was so badly misconfigured that his feeble efforts were overpowered by the brutal laws of physics. At this point, he desperately needed to trim nose down manually — but tragically, he did not.
As the plane climbed rapidly, its speed plunged again, and Käppel exclaimed in German, “I increase speed, yeah?” Switching to English, he then added, “Ja, it’s pitching up all the time!”
“Stick forward,” Horrell shouted. “Alpha floor, we’re in manual!” He had mistaken Käppel’s thrust increase for the activation of the alpha floor.
“It’s, it’s pitching up, this — ” Käppel exclaimed.
“Are you able to handle this?” First Officer Ketzer asked.
“No!” said Käppel.
The stall warning started blaring again as they approached a stall for the second time in the space of 30 seconds. Moments later, the airplane in fact stalled, reaching an incredible 57 degrees pitch up while in a 59-degree bank to the left. Their altitude peaked at 3,788 feet and their airspeed bottomed out somewhere below 40 knots, far too slowly to sustain flight. Shuddering violently, the airplane then lost lift and began to plunge from the sky.
As it did so, the nose fell through into a dive and the speed began to increase, but there was very little room to recover. First Officer Ketzer repeatedly shouted “Gear up,” and Captain Käppel complied, letting loose a string of indistinct shouts and expletives.
Confused by the loss of control, Horrell said, “What’s wrong here?”
“Flaps up, flaps up!” Käppel shouted. Their airspeed was rapidly increasing as they dived toward the ground, but they weren’t going to make it.
“Speed brakes!” Horrell suggested.
“TERRAIN! TERRAIN!” blared the ground proximity warning system.
Those were the final words on the cockpit voice recording. Moments later, at 15:46 and 7 seconds, pitched 14 degrees nose down and banked 15 degrees to the right, with a forward airspeed of 263 knots and increasing, flight 888T plowed into the Mediterranean Sea five kilometers off the coast of Perpignan, instantly destroying the aircraft and killing all 7 people on board. Only 62 seconds had passed since the first sign of trouble.
◊◊◊
Numerous witnesses on shore and in boats saw the A320’s final plunge, including the crew of a yacht positioned only 500 meters from the point of impact. These witnesses swiftly reported the crash to emergency services, who responded with urgency to the scene, but there was little for them to find. Only light debris remained afloat, including part of the aircraft’s tail, freshly emblazoned with the iconic logo of Air New Zealand. Within hours, it was determined that no one had survived.
In a cruel twist of fate remarked upon by the airline’s CEO, the crash coincidentally came on the 29th anniversary of the infamous crash of Air New Zealand flight 901, New Zealand’s worst peacetime disaster, piling yet more misery on top of an already black date for the company and the country.
◊◊◊
The investigation into the crash involved more than half a dozen interested parties and took the better part of three years, ending in July 2011. The immense breadth and depth of their investigative work is only briefly glimpsed in this article, as comprehensive as it may appear to be. And in the end, the sequence of events that they established was both bizarre and complex, building up to an accident unlike almost any other, and yet also somehow filled with the ghostly reflections of other past and future tragedies.
To summarize, the events began when the painting crew washed the aircraft using a high-pressure fire hose with no protective covers installed on the angle of attack vanes. Water entered the bearings, which subsequently froze when the plane climbed to cruising altitude, jamming two of the three AOA sensors. These sensors provided faulty readings to the flight augmentation computers, which subsequently failed to prevent the airplane from stalling when the pilots attempted to test the stall protection systems. The pilots were unable to recover from the stall in time, largely because the stabilizer trim remained at full nose up all the way down to the water.
Regarding this sequence of events, a number of observations must be made.
First of all, there was a failure of judgment on the ground at EAS Industries. The potential for water to enter the AOA sensor bearings if the sensors were not covered during cleaning had been known to Airbus since the early 1990s, and the danger was spelled out in two letters to operators issued in 1993 and 1998. Furthermore, procedures designed explicitly to prevent this issue were included in the Airbus painting and cleaning procedures, which EAS used as the basis for its own repainting program. However, the painting team supervisor failed to look up this easily available documentation because they didn’t consider washing the airplane to be a “maintenance action” requiring adherence to any particular procedures, which was both poor judgment and a failure of training. Furthermore, the decision to wash the airplane using a fire hose was questionable and appeared to have been made in order to speed up the project timeline.
Meanwhile on the administrative side, investigators noted that the concept of an “acceptance flight” existed within a regulatory gray area that lacked any official definition. Companies around the world were carrying out these types of check flights in connection with lease agreements, but there were no standards applicable to them, and they were often arranged on an ad-hoc basis. The lack of adequate planning and procedures for acceptance flights was made apparent by XL Airways’ failure to acquire the proper permission to perform such a flight in French airspace. Additionally, Air New Zealand based the check items on an Airbus program that was, first of all, designed for trained test pilots, and second, was intended for demonstration purposes only, and not as a test of system functionality. But there was also no rule that said they couldn’t do that, due to the lack of any special regulations applicable to acceptance flights.
Once the flight was underway, the crew suddenly discovered that because of these administrative failings, air traffic control would not grant them permission to perform the maneuvers described in the check program. At that point, the crew made the first of several errors of judgment when they decided to continue the checks out of order and on a compressed timeline. The Airbus customer acceptance manual on which the check program was based contained a warning that “any unplanned change in the programme during the flight could endanger the safety of the flight,” but this language was not included in the documentation provided to the crew by Air New Zealand.
As a result of the flight crew’s improvisation, the low speed test was not done at 14,000 feet as prescribed by the check program. Had it been performed at this altitude, all else being equal, the crew likely would have been able to recover from the stall before hitting the ground — which is exactly why they were supposed to do it at that height in the first place. (It could not be performed higher than that because low speed flight in the landing configuration at high altitudes is not possible.) But by the time they were ready to perform the check, they had already descended to 8,000 feet, and it was not actually begun until 3,000 feet because Captain Käppel didn’t want to perform the check while in clouds.
It must be emphasized that 3,000 feet is far too low to check the functioning of the low speed protections. If the airplane stalls at that altitude, there may not be enough room to recover, as this crew discovered the hard way. Performing the test at that altitude was wildly inappropriate, bafflingly cavalier, and obviously dangerous. No self-respecting test pilot would have attempted it. So why did Horrell and Käppel even try? Why didn’t Käppel go with his own suggestion to skip the procedure? We may never know for sure, but there are a number of factors that might have contributed to this highly ill-advised decision.
The first thing to keep in mind is that an atmosphere of improvisation had already been established. The crew were conducting a highly unusual flight under exceptional conditions, carrying out numerous non-routine procedures without passengers on board — all of which are factors that can lead to greater risk-taking than normal. In fact, according to an FAA study from around the time of the crash, 25% of all accidents involving turbine-powered aircraft occurred during non-revenue flights, and as far as such flights go, this was an especially risky one. So in short, the itinerary itself and the conditions under which it was carried out encouraged a feeling that it was okay to do things that the crew never would have attempted under normal circumstances.
Another aspect to note is that the pilots had no plan in the event that the low speed protections didn’t work, even though they were ostensibly testing the system’s functionality. During real test flights at Airbus, test pilots must anticipate the possibility of system failures and develop a contingency plan to respond, but no such advice was included in the Air New Zealand check program because it was based on a procedure meant for demonstration flights. Even so, the fact that the pilots didn’t do this on their own shows that they were fully convinced of the airplane’s fitness to fly, and the possibility that the system might not work evidently wasn’t considered. This suggests that the pilots didn’t really see themselves as performing a “test flight” at all. It’s possible that they simply approached the check program as a list of boxes to tick off because figures in upper management wanted the boxes ticked, and not because there was any real question about the airplane’s airworthiness.
If the pilots had accounted for the possibility of failure, then they might not have performed the test at all, given the circumstances. But even if they went ahead with it anyway, a more effective briefing might have been sufficient to prevent the accident all by itself. Specifically, the check program sheet contained a table of limit speeds, which stated that VLS in their configuration was 123 knots and Vmin was 107 knots. Had Horrell told Käppel to watch for these particular airspeeds, then he wouldn’t have been caught out by the false limit speed markers on his airspeed indicator, and he might never have lost control of the airplane. Instead, Horrell told Käppel to “just come back to the alpha floor activation,” which was an inappropriate directive in a situation where the functionality of the alpha floor was, strictly speaking, an unknown.
Once all of these mistakes were made, the pilots found themselves in a notoriously tricky situation — flying an Airbus with degraded control laws, on an unstable flight path, with an improper trim setting. Combinations of even two of these factors, let alone all three, have been featured in my previous articles on deadly accidents such as Air France flight 447 and China Airlines flight 140. And with so little altitude to recover, the crew had almost no room for error, and only a few seconds in which to deduce that they were flying in direct law and could only recover by using manual trim. At that point, the outcome was a roll of the dice, and they came up short. But again, they never should have allowed things to get that far in the first place.
◊◊◊
In its final report, the French Bureau of Inquiry and Analysis issued several recommendations calling for stricter regulation of non-revenue acceptance flights, better warnings for flight crews when control laws change, and improved stall recovery training. The latter two of these were already in the spotlight following the crash of Air France flight 447 in June 2009, and major training changes that have since been implemented are considered the result of several major accidents, including XL Airways Germany flight 888T, that collectively contributed to a shift in perspectives. Most notably, none of the flight crew of flight 888T, despite their extensive experience, had undergone upset and recovery training, which teaches recovery from abnormal attitudes in flight. This training is mandatory in most of the world today.
The crash of flight 888T in particular is perhaps most noteworthy for its unique causes and the harsh lessons it conveys about such lofty topics as judgment and caution. All of the pilots aboard the flight were veteran aviators, with well over 10,000 hours apiece and years of A320 experience. They presumably understood how the airplane’s systems worked. They had the skills to develop a safe and reasonable approach to the tests on their itinerary. And they were most certainly capable of treating the flight with the respect that it deserved. But in the end, they were caught out by that most human of faults — the sin of complacency, the unconscious belief that nothing ever goes wrong, and if it does, it only happens to other people. And although their fate lies sealed in stone, some glimmer of good may yet be extracted from tragedy if into that stone we carve a gentle warning for others.
_______________________________________________________________
Don’t forget to listen to Controlled Pod Into Terrain, my new podcast (with slides!), where I discuss aerospace disasters with my cohosts Ariadne and J! Check out our channel here, and listen to our latest episode, in which we break down the incredibly poor decision-making aboard Pinnacle Airlines flight 3701. Alternatively, download audio-only versions via RSS.com, or look us up on Spotify!
_______________________________________________________________
Join the discussion of this article on Reddit
Support me on Patreon (Note: I do not earn money from views on Medium!)
Visit r/admiralcloudberg to read and discuss over 250 similar articles
_______________________________________________________________
Note: this accident was previously featured in episode 63 of the plane crash series on November 17th, 2018, prior to the series’ arrival on Medium. This article is written without reference to and supersedes the original.
