Monday, September 08, 2008

Console Post of the Week: Microsoft's Shame

"Shame" may sound like an unreasonable word, but keep reading.

After years of speculation, Dean Takahashi (whose reputation is unquestioned) has finally published a definitive article on what went wrong with the Microsoft 360. Have you always wondered why the launch machines (and consoles manufactured over the next 18 months) had such disastrously high failure rates?

Now we know. Takahashi has documented it all.

Before we take a look at some of the particulars in the article, let's look at Microsoft's official response:
I asked Microsoft to confirm or deny 35 different facts contained in this story. Instead, I received a formal statement from a Microsoft spokesperson, saying the company had already acknowledged an “unacceptable number of repairs” to Xbox 360 consoles and responded to the hardware failures with a free replacement program. The statement also said, “This topic has already been covered extensively in the media. This new story repeats old information, and contains rumors and innuendo from anonymous sources, attempting to create a new sensational angle, and is highly irresponsible.”

Please note, and this is extremely important, that nowhere in that statement does Microsoft say that any of Takahashi's information is incorrect. It's a an absolutely classic non-denial denial.

Now, let's look at the details.

In August 2005, according to Takahashi, a Microsoft engineer said that production needed to be stopped because of the defect rate. How high was the defect rate?
The defect rate for the machines was an abysmal 68 percent at that point, according to several sources. That meant for every 100 machines that Microsoft’s contract manufacturers, Flextronics and Wistron, made at their factories in China, 68 didn’t work.

Incredible.

There were also issues with microprocessor yield:
The initial yield on the most critical chip, the three-core microprocessor designed and manufactured by IBM, was only 16 percent. For every 100 produced, only 16 worked.

And more:
There were plenty of warning signs. Early reports on the problems were myriad. In an Aug. 30, 2005 memo, the team reported overheating graphics chip, cracking heat sinks, cosmetic issues with the hard disk drive and the front of the box, under-performing graphics memory chips from Infineon (now Qimonda), a problem with the DVD drive, and other things.

As launch drew closer, the problems continued:
The testing machines were not ready, and the battery of tests that they ran were not fully developed. That meant that the testing machines would inspect the Xbox 360s coming off the line and approve them for shipment, even though there were likely flaws.

The test machines were not properly debugged, due to an ill-advised cost-cutting initiative that shaved $2 million from $25 million paid to Cimtek, a test machine maker in Canada. The Microsoft team decided not to pay the consulting fee to Cimtek to build, manage and debug the test machines. Sources familiar with the matter said there were only about 500 test machines at the time of launch, a third of the 1,500 needed.

“There were so many problems, you didn’t know what was wrong,” said one source of the machines. “The [test engineers] didn’t have enough time to get up and running.”

And what about yield?
The Xbox 360’s yield climbed somewhat throughout the fall of 2005, as Microsoft readied the worldwide launch in November. But the “first pass yield” (before machines were taken off the line to be reworked inside the factory) was never over 70 percent.

Post-launch, the problems continued, and there were good reasons for the shortages at retail:
By the end of March 31, Microsoft said it had shipped more than 3.3 million consoles to retailers. There was a growing “bone pile” in a warehouse at Wistron and at a repair center in Texas.

Microsoft had more than 500,000 defective consoles that sat in warehouses. They were either duds coming out of the factory or they were returned boxes, according to inside sources.

Here's more:
During the spring of 2006, the engineers were working on a transition to a new motherboard, code-named Zephyr. But they postponed that transition to work on fixing the “bone pile” issues and “maximize the yield,” according to an email circulated by engineering manager Harjit Singh to the hardware team on March 10, 2006. The yield at that point was an abysmal 50 percent on the first pass. When the bad machines were reworked within the factory, the yield went up to 75 percent –- hardly acceptable.

In an effort to contront this problem internally, Microsoft took an extraordinary step:
Microsoft decided to shut down manufacturing of the Xbox 360 in January, 2007. Between January and June, it didn’t build any new machines. The reason was partly because it made too many machines earlier, but the other reason was to track down the source of its quality problems.

Look, we all knew that Microsoft had their head up their ass when it came to the quality of the 360 hardware, but this is solid evidence of something different: duplicitous behavior. They knowing shipped a console when they already had overwhelming internal evidence that the failure rate was going to be extraordinarily high compared to other consumer electronics devices.

Somehow, Microsoft has tried to position themselves as "doing the right thing" because they've extended warranties on the 360 (for the red ring of death). I think that falls less in the category of "doing the right thing" than "trying desperately to avoid a class action lawsuit." Can you imagine the discovery process if a lawsuit was filed? Microsoft's internal documents would get them killed! I don't see how they would avoid a total recall of the initial 15+ months of shipments, at a minimum.

Even worse, they continued to ship this console for over A YEAR, knowing it was defective, before they finally took serious steps to find the problem. When considered against Microsoft's public statements, the information in Takahashi's article is nothing short of sickening.

And before you say "oh, all companies do those kinds of things"--no, they don't. The failure rate for the 360 may well have been the highest failure rate in the history of consumer electronics devices, and it's certainly the highest, by far, for a gaming console.

Don't you think other companies faced similar problems in the past? How did other companies solve those problems? Well, I'll tell you the one thing they didn't do--they didn't ship the product in that condition.

I'm sure the honks at Microsoft are totally incapable of feeling shame--for anything--but that's what they should be feeling.

Shame.

Site Meter