I think that the idea that data lies is false, and that such a notion is commonly held a sign of lazy intellect. You can present data in different ways to focus on different aspects of a system. And you can make faulty assumptions based on data you look at.

It is true someone can just provide false data, that is an issue you have to consider when drawing conclusions from data. But often people just don’t think about what the data is really saying. Most often when people say data lies they just were misled because they didn’t think about what the data actually showed. When you examine data provided by someone else you need to make sure you understand what it is actually saying and if they are trying to support their position you may be wise to be clear they are not misleading you with their presentation of the data.

Here is some data from Greg Mankiw’s Blog. He wants to make his point that the USA is taxed more on par with Europe than some believe because he want to reduce current taxes. So he shows that while taxes as a percent of economic activity is low in the USA taxes per person is comparable to Europe.

The USA is the 2nd lowest for percent of GDP taxes 28.2% v 27.4% for Japan. But in taxes per person toward the middle of the pack. France which has 46% taxes/GDP totals $15,556 in tax per person compared to $13,097 for the USA. Both measures of taxes are useful to know, in my opinion. Neither lies. Both have merit in providing a understanding of the system (the economies of countries).

This is an interesting video on Deming and American management (by the BBC in 1992): Prophet Unheard. It includes some nice old footage of Deming in Japan. The importance of respect for people is clear and the video also touches on the idea the danger of relying on data (when you do not understand variation and that many important matters and unmeasurable). The video features many snippets of Dr. Deming speaking and includes Don Peterson, Ford CEO; Clare Crawford Mason, If Japan Can, Why Can’t We producer; and Myron Tribus.

Once again the USA was the leading country in manufacturing for 2008. And once again China grew their manufacturing output amazingly. In a change with recent trends Japan grew output significantly. Of course, the 2009 data is going to show the impact of a very severe worldwide recession.

The first chart shows the USA’s share of the manufacturing output, of the countries that manufactured over $185 billion in 2008, at 28.1% in 1990, 27.7% in 1995, 32% in 2000, 28% in 2005, 28% in 2006, 26% in 2007 and 24% in 2008. China’s share has grown from 4% in 1990, 6% in 1995, 10% in 2000, 13% in 2005, 14% in 2006, 16% in 2007 to 18% in 2008. Japan’s share has fallen from 22% in 1990 to 14% in 2008. The USA has about 4.5% of the world population, China about 20%. See Curious Cat Investment blog post” Data on the Largest Manufacturing Countries in 2008.

Even with just this data, it is obvious the belief in a decades long steep decline in USA manufacturing is not in evidence. And, in fact the USA’s output has grown substantially over this period. It has just grown more slowly than that of China (as has every other country), and so while output in the USA has grown the percentage with China has shrunk. The percentage of manufacturing output by the USA (excluding output from China) was 29.3% in 1990 and 29.6% in 2008. The second chart shows manufacturing output over time.

The 2008 China data is not provided for manufacturing alone (the latest UN Data, for global manufacturing, in billions of current USA dollars). The percentage of manufacturing (to manufacturing, mining and utilities) was 78% for 2005-2007 (I used 78% of the manufacturing, mining and utilities figure provided in the 2008 data). There is a good chance this overstates China manufacturing output in 2008 (due to very high commodity prices in 2008).

Hopefully these charts provide some evidence of what is really going on with global manufacturing and counteracts the hype, to some extent. Global economic data is not perfect. These figures are an attempt to capture the economic reality in the world but they are not a perfect proxy. This data is shown in 2008 USA dollars which is good in the sense that it shows all countries in the same light and we can compare the 1995 USA figure to 2005 without worrying about inflation. However foreign exchange fluctuations over time can show a country, for example, having a decline in manufacturing output in some year when in fact the output increased (just the decline against the USA dollar that year results in the data showing a decrease – which is accurate when measured in terms of USA dollars).

If the dollar declines substantially between when the 2008 data was calculated and the 2009 data is calculated that will give result in the data showing a substantial increase in those countries that had a currency strengthen against the USA dollar. At this time the Chinese Renminbi has not strengthened while most other currencies have – the Chinese government is retaining a peg to a specific exchange rate.

Korea (1.8% in 1990, 3% in 2008), Mexico (1.7% to 2.6%) and India (1.4% to 2.5%) were the only countries to increase their percentage of manufacturing output (other than China, of course, which grew from 3.9% to 18.5%).

Failing to do this leads most people to fail to learn from there decisions. It is hard to improve when you don’t learn. The similarity to the PDSA improvement cycle is not a surprise. Both are about learning so you can adopt more effective strategies.

Infusing his entire presentation with humor and fascinating tales of his memories, Box focused on sequential design of experiments. He attributed much of what he knows about DOE [design of experiments] to Ronald A. Fisher. Box explained that Fisher couldn’t find the things he was looking for in his data, “and he was right. Even if he had had the fastest available computer, he’d still be right,” said Box. Therefore, Fisher figured out how to study a number of factors at one time. And so, the beginnings of DOE.

Having worked and studied with many other famous statisticians and analytic thinkers, Box did not hesitate to share his characterizations of them. He told a story about Dr. Bill Hunter and how he required his students to run an experiment. Apparently a variety of subjects was studied [see 101 Ways to Design an Experiment, or Some Ideas About Teaching Design of Experiments]
…
According to Box, the difficulty of getting DOE to take root lies in the fact that these mathematicians “can’t really get the fact that it’s not about proving a theorem, it’s about being curious about things. There aren’t enough people who will apply [DOE] as a way of finding things out. But maybe with JMP, things will change that way.”

George Box is a great mind and great person who I have had the privilege of knowing my whole life. My father took his class at Princeton, then followed George to the University of Wisconsin-Madison (where Dr. Box founded the statistics department and Dad received the first PhD). They worked together building the UW statistics department, writing Statistics for Experimenters and founding the Center for Quality and Productivity Improvement among many other things.

Statistics for Experimenters: Design, Innovation, and Discovery shows that the goal of design of experiments is to learn and refine your experiment based on the knowledge you gain and experiment again. It is a process of discovery. If done properly it is very similar to the PDSA cycle with the application of statistical tools to aid in determining the impact of various factors under study.

There is no true value of anything. There is instead a figure that is produced by application of a master or ideal method of counting or measurement… no true value of the number of inhabitants within the boundaries of (e.g.) Detroit. A count of the number of inhabitants of Detroit is dependent upon the application of arbitrary rules for carrying out the count. Repetition of an experiment or of a count will exhibit variation.

Dr. Deming’s ideas on the theory of knowledge are the least understood and least seen in other management systems. The importance of understanding what data does, and does not tell you, is at least somewhat acknowledged in other management system but is often not found much in the actual practice of management. The execution often glosses over the importance of actually understanding statistics versus using formulas. Just using formulas is dangerous. It may be inconvenient but learning about the traps we can fall into in using data is important.

How often do you see the operational definition used to calculate the data you see with the data you are provided?

Anscombe’s quartet: all four sets are identical when examined statistically, but vary considerably when graphed. Image via Wikipedia.

___________________Anscombe’s quartet comprises four datasets that have identical simple statistical properties, yet are revealed to be very different when inspected graphically. Each dataset consists of eleven (x,y) points. They were constructed in 1973 by the statistician F.J. Anscombe to demonstrate the importance of graphing data before analyzing it, and of the effect of outliers on the statistical properties of a dataset.

Computer hardware and software creators use benchmarks as one tool to compare the performance of alternative products. At times this can be very useful. You can learn what software of hardware is faster and that may be a very valuable factor. However, any measure is determined by the operational definitions used in collecting the measure. And if people have incentives to improve the measured number they often will do just that (improving the measure) rather than improving the system (the measure is meant to serve as a proxy for some function of that system).

Technology companies compete fiercely and claiming the software or hardware is faster is one big area of competition. And the comment on Reddit is claiming one competitor changed some code only to get a better measure (that provides no benefit to customers). The problem with such actions, is they provide no actual value: all they do is make the measure less meaningful as a proxy.

Now it is also perfectly understandable why it would be done – when you are focused on improving the number, it might well be easier to distort the system to provide a better number (used by to measure performance) instead of actual improve the performance. It is easy to see why a company would do this if they want to have marketing claim their products are the fastest.Continue reading →

A few weeks ago, we ran one of the largest multivariate experiments ever: a 1,024 recipe experiment on 100% of our US-English homepage. Utilizing Google Website Optimizer, we made small changes to three sections on our homepage (see below), with the goal of increasing the number of people who signed up for an account. The results were impressive: the new page performed 15.7% better than the original, resulting in thousands more sign-ups and personalized views to the homepage every day.
…
While we could have hypothesized which elements result in greater conversions (for example, the color red is more eye-catching), multivariate testing reveals and proves the combinatorial impact of different configurations. Running tests like this also help guide our design process: instead of relying on our own ideas and intuition, you have a big part in steering us in the right direction. In fact, we plan on incorporating many of these elements in future evolutions of our homepage.

Making things visible is a key to effective management. And data in computers can be easy to ignore. Don’t forget to make data visible. Paul Levy, CEO of Beth Israel Deaconess Medical Center in Boston recently hosted Hideshi Yokoi, president of the Toyota Production System Support Center and wrote this blog post:

Together, we visited gemba and observed several hospital processes in action, looking for ways to reduce waste and reorganize work. It was fascinating to have such experts here and see things through their eyes. Mr. Yokoi’s thoughts and observations are very, very clear, notwithstanding a command of English that is still a work in progress.

The highlight? At one point, we pointed out a new information system that we were thinking of putting into place to monitor and control the flow of certain inventory. Mr. Yokoi’s wise response, suggesting otherwise, was:

“When you put problem in computer, box hide answer. Problem must be visible!”

Google depends on economic principles to hone what has become the search engine of choice for more than 60 percent of all Internet surfers, and the company uses auction theory to grease the skids of its own operations. All these calculations require an army of math geeks, algorithms of Ramanujanian complexity, and a sales force more comfortable with whiteboard markers than fairway irons.
…
Varian tried to understand the process better by applying game theory. “I think I was the first person to do that,” he says. After just a few weeks at Google, he went back to Schmidt. “It’s amazing!” Varian said. “You’ve managed to design an auction perfectly.” To Schmidt, who had been at Google barely a year, this was an incredible relief. “Remember, this was when the company had 200 employees and no cash,” he says. “All of a sudden we realized we were in the auction business.”
…
Google even uses auctions for internal operations, like allocating servers among its various business units. Since moving a product’s storage and computation to a new data center is disruptive, engineers often put it off. “I suggested we run an auction similar to what the airlines do when they oversell a flight. They keep offering bigger vouchers until enough customers give up their seats,” Varian says. “In our case, we offer more machines in exchange for moving to new servers. One group might do it for 50 new ones, another for 100, and another won’t move unless we give them 300. So we give them to the lowest bidder—they get their extra capacity, and we get computation shifted to the new data center.”

Google continues to make bold moves putting faith in their ability to find innovative solutions that others reject as impossible. It is a challenging but interesting path to success, for them, at least.

Revealed Preference: the preference consumers display by their action, in contrast to what they may say they prefer. While surveys may be useful people often say they will do one thing and actually when given the choice to do so, don’t.

Normally what matters is not what people say they want but what they actually will choose. For that reason revealed preference is a better measure than stated preference. Stated preference is often used as a proxy for actual preference (which may be fine) but it is important to understand that it is just a proxy for actual preference.

Various techniques are used to ensure a quality (no red bead) product. There are quality control inspectors, feedback to the workers, merit pay for superior performance, performance appraisals, procedure compliance, posters and quality programs. The foreman, quality control, and the workers all put forth their best efforts to produce a quality product. The experiment allows the demonstration of the effectiveness (or ineffectiveness) of the various methods.

Business schools have shown a remarkable ability to miss the economic catastrophes unfolding before their eyes.

In the late 1990s, their faculties rushed to write paeans to Enron, the firm of the future, the new economic paradigm. The admiration was mutual: Enron was stuffed with Harvard Business School alumni, from Jeff Skilling, the chief executive, down. When Enron, rotten to the core, collapsed, the old case studies were thrust in a closet and removed from the syllabus, and new ones were promptly written about the ethical and accounting issues posed by Enron’s misadventures.
…
Is there a pattern here? Go back to the 1980s, and you find that Harvard MBAs played a big enough role in the insider trading scandals that washed through Wall Street for a former chairman of the SEC to consider it a good move to donate millions of dollars for the teaching of ethics at the school.

Time after time, and scandal after scandal, it seems that a school that graduates just 900 students a year finds itself in the thick of it. Yet there is remarkably little contrition.

Last October, Harvard Business School celebrated its 100th birthday with a global summit in Boston. While Wall Street and Washington descended into an economic inferno, Jay Light, the dean of the school and a board member at the Black-stone private equity group, opened the festivities by shrugging off any responsibility.

“We all failed to understand how much [the financial system] had changed in the past 15 years or so, and how fragile it might be because of increased leverage, decreased transparency and decreased liquidity: three of the crucial things in the world of financial markets,” he said.
…
You can draw up a list of the greatest entrepreneurs of recent history, from Larry Page and Sergey Brin of Google and Bill Gates of Microsoft, to Michael Dell, Richard Branson, Lak-shmi Mittal – and there’s not an MBA between them.

Yet the MBA industry continues to grow, and business schools provide vital income to academic institutions: 500,000 people around the world now graduate each year with an MBA, 150,000 of those in the United States, creating their own management class within global business.

Given the present chaos, shouldn’t we be asking if business education is not just a waste of time, but actually damaging to our economic health?

Business schools unfortunately continue to take a heavily simplistic number (without an understanding of variation) and fad driven approach to management. W. Edwards Deming was against the damage they were causing decades ago, and I see little evidence they have learned from their failures.

Schools are good for making connections and getting a piece of paper. Some companies won’t consider you for some jobs unless you have an document saying you have an MBA. I strongly question the wisdom of only hiring an MBA to do some job. But many companies like to use simple criteria like – without a piece of paper saying you have an MBA we won’t consider you for this job. So if you want a job from them getting that piece of paper is important.

Catalyzing innovation, problem solving, and discovery, the Second Edition provides experimenters with the scientific and statistical tools needed to maximize the knowledge gained from research data, illustrating how these tools may best be utilized during all stages of the investigative process. The authors’ practical approach starts with a problem that needs to be solved and then examines the appropriate statistical methods of design and analysis.
…
* Graphical Analysis of Variance
* Computer Analysis of Complex Designs
* Simplification by transformation
* Hands-on experimentation using Response Service Methods
* Further development of robust product and process design using split plot arrangements and minimization of error transmission
* Introduction to Process Control, Forecasting and Time Series

To look at this question, three colleagues and I conducted an experiment. We presented 87 participants with an array of tasks that demanded attention, memory, concentration and creativity. We asked them, for instance, to fit pieces of metal puzzle into a plastic frame, to play a memory game that required them to reproduce a string of numbers and to throw tennis balls at a target. We promised them payment if they performed the tasks exceptionally well. About a third of the subjects were told they’d be given a small bonus, another third were promised a medium-level bonus, and the last third could earn a high bonus.
…
So it turns out that social pressure has the same effect that money has. It motivates people, especially when the tasks at hand require only effort and no skill. But it can provide stress, too, and at some point that stress overwhelms the motivating influence.

When I recently presented these results to a group of banking executives, they assured me that their own work and that of their employees would not follow this pattern. (I pointed out that with the right research budget, and their participation, we could examine this assertion. They weren’t that interested.)

Singapore is again ranked first for Ease of Doing Business by the World Bank. For some reason they call the report issued in any given year as the report for the next year (which makes no sense to me). The data shown below is for the year they released the report.

So proclaimed statistician George Box 30 years ago, and he was right. But what choice did we have? Only models, from cosmological equations to theories of human behavior, seemed to be able to consistently, if imperfectly, explain the world around us. Until now. Today companies like Google, which have grown up in an era of massively abundant data, don’t have to settle for wrong models. Indeed, they don’t have to settle for models at all.
…
Speaking at the O’Reilly Emerging Technology Conference this past March, Peter Norvig, Google’s research director, offered an update to George Box’s maxim: “All models are wrong, and increasingly you can succeed without them.”
…
There is now a better way. Petabytes allow us to say: “Correlation is enough.” We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

see update, below. Norvig was misquoted, he agrees with Box’s maxim

I must say I am not at all convinced that a new method without theory ready to supplant the existing scientific method. Now I can’t find peter Norvig’s exact words online (come on Google – organize all the world’s information for me please). If he said that using massive stores of data to make discoveries in new ways radically changing how we can learn and create useful systems, that I believe. I do enjoy the idea of trying radical new ways of viewing what is possible.

In this talk we will see that a computer might not learn in the same way that a person does, but it can use massive amounts of data to perform selected tasks very well. We will see that a computer can correct spelling mistakes, translate from Arabic to English, and recognize celebrity faces about as well as an average human—and can do it all by learning from examples rather than by relying on programming.