Ideas from Jeffrey Pfeffer and Robert I. Sutton
Professors of Management at Stanford University
Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-Based Management
Boston, MA: Harvard Business School Press, 2006
1. Most People Rely on Common Sense and Half-Truths
All too often managers believe that a practice works, “not because it is based on sound logic or hard facts, but because managers ‘believe’ it works” based on emotions, prior experience, cultural beliefs, faith, or ideology (p. 10).
Our deeply ingrained beliefs about the world are “quite ‘sticky’ – they resist dis-confirming evidence and persist in affecting judgments and choice, regardless of whether or not they are true” (p. 12).
Half-truths do the most damage because when then there are “treated as completely true and applied in full force to every decision and every action,” they “undermine performance, destroy management careers, and ruin employee well-being.” Half-truths are also “more difficult to debunk that total nonsense because arguments can always be mustered about the times and places they are correct – and then generalized to the wrong settings and times” (p. 25).
Many managers “show little interest in subjecting their business practices and decisions to the same scientific rigor they would use for technical or medical issues” (p. 12). Managers need to use “evidence-based management,” which entails using “better, deeper logic and employing facts” so they can “do their jobs better” (p. 13).
2. What Is Evidence Based Management?
Don’t assume you understand what enhances performance. Instead, ask “the basic question of why something might enhance performance…Why is a particular practice linked to performance improvement – what is the logic?...If you cannot unpack the logic of why things have worked, it is unlikely you will be able to determine whether or not they will work this time.” (pp. 7-9).
Before adopting a new practice, ask “What are the downsides and disadvantages to implementing the practice, even if it is a good idea” (p. 8).
Even if something worked in the past, a “new situation is different from the past,” so it might not work again. Plus, memories or understanding of what worked in the past may be flawed, so “what we ‘learned’ was right in the past may have been wrong, or incomplete, in the first place” (p 9).
Think before you act. “Blindly copying the same approach without considering the underlying problems is just plain dumb” (p. 9).
Often the “facts required for decision making are not readily at hand,” but this is not an excuse to lazily fall back on “benchmarking, past experience, ideology, and conventional wisdom to guide” your decisions. Instead, you should rely on evidence-based methods to look at what data can be gathered quickly based on existing practices, while also “taking steps to gather new and possible more useful information” with better methods in the future (p. 18).
3. Example: The Scientific Evidence Shows that Merit Pay for Teachers Doesn’t Work
Four assumptions behind merit pay (pp. 22-24):
1. Teacher motivation is the most important determinant of student learning and achievement
2. Learning “can be measured reliably and accurately” by standardized tests
3. Teachers are motivated primarily by financial incentives; Thus, more pay will create more motivation, and more motivation and effort will increase student learning and achievement
4. Teaching is a “solo activity” with “little interdependence with others in the school,” so a teacher’s motivation and effort can be clearly measured as the determining factor in student learning and achievement
Conclusion:
“The evidence shows that merit-pay plans seldom last longer than five years and that merit pay consistently fails to improve student performance.” When work requires “even modest interdependence and cooperation,” and teaching requires a great deal cooperation with other teachers, administrators, students, parents, and the community, individual merit rewards “have consistently negative consequences on organizations” (p. 128). Plus, merit pay “provides an incentive for some teachers to game the system” through cheating. Research has shown that teacher “cheating” was “quite sensitive to the size of the incentive provided for enhancing student scores.”
Merit pay is also a dubious policy because teachers do not affect student learning very much, let alone determine student learning. Studies of leaders and managers in multiple settings have found that leaders “often have far less influence over performance than most people think” (p. 192). Scholars have estimated that leaders account for about 10 percent of performance outcomes. This is largely due to the fact that “organizational performance is determined largely by factors that no individual - including the leader - can control” (p. 192).
The evidence is clear that merit-pay is a doubled-edged sword, so it needs to be used carefully and selectively. “If you are going to pay people for doing something, you need to be very, very thoughtful about the possible consequences of the behavior you have just signaled them to do.” Managers need to ask, “what could or might go wrong” with this policy (p. 132). It is best to start small and treat merit-pay initiatives as an experiment to see if it works and if it produces unwanted, unintended consequences.
Furthermore, the use of standardized tests and grades are not conducive to learning and lasting academic achievement. The evidence “strongly suggests that student learn better when they are not graded and certainly not when they are graded on a curve” (p. 35).
And yet, not only does the idea of merit pay persist, it has gained in popularity and spread all across the country in the last two decades since Pfeffer and Sutton wrote their book.
Educational research Anthony Bryk noted, “It is like policy makers suffer from amnesia” (qtd. in Pfeffer & Sutton, 2006, p. 24). Often policy makers, managers, and journalists suffer from “a bizarre collective amnesia,” where the “same things are ‘discovered,’ or at least reported, over and over, which wastes much effort and time” (p. 43). Individual financial incentives “often aren’t that effective” in a range of settings (p. 129), but nobody seems to remember this fact, or believe it.
4. Warning: Evidence-Based Management and Practice is Not Magic
Managers and teachers are professionals, like doctors, who “face one decisions after another. It isn’t possible for even the very best physician – or manager – to make the right decision each time” (p. 27).
No tool or thought process can “improve every managerial decision and action” all the time. The best we can do is not be “seduced” by emotions, prior experience, cultural beliefs, faith, ideology, and half-truths. Instead, we must “use the best data” and “the best logic” to make the best decision we can in each situation.
Do your research. Most likely other people have faced a similar problem and shared their experience and decision-making process. Odds are, there is also data and scientific research on the problem. Study the problem you are facing and see what others already know. Then collect data on your problem and try to figure out why the problem exists and how it might be solved. Remember, as the political scientists James March once said, “Most claims of originality are testimony to ignorance and most claims of magic are testimony to hubris” (qtd. in Pfeffer & Sutton, 2006, p. 45).
Furthermore, don’t just research “best practices.” It is much more instructive to research failed practices. It is important to remember that “crucial evidence is lost” when we “ignore the practices and strategies” that fail (p. 37). Studying only best practices can “lead to flawed and dangerous conclusions” about what will and won’t work.
There are no silver bullets. No magic. Using evidence-based management “is a journey, not a quick-fix technique, and along the road you will encounter hindrances and obstacles” (p. 30). Far too many “solutions” are “presented as costless and universally applicable, with little acknowledgement of potential pitfalls” (p. 47). To practice “evidence-based management you first need to know the truth. And it’s better to know the truth early, when situations can be remedied, than later when it may be too late to do much” (p. 32). But the truth is messy, complicated, and usually incomplete.
Einstein once said, “Not everything that can be counted counts, and not everything that counts can be counted” (qtd. in Pfeffer & Sutton, 2006, p. 40). In my most recent book, I argued that we cannot measure what matters most. Thus, while quantitative data is important, it is not the only kind of data, and it is not always the most useful or enlightening data (p. 40). But be warry of your subjective biases when making observations, as “eyewitness accounts are notoriously unreliable” (p. 49). With qualitative data, try to get as many perspectives as possible.
Also, be warry of the attribution error of placing “excessive weight on individual personality, preferences, and efforts when trying to explain what people (and groups and organizations) do and whey they do it” (p. 102). Most people ignore or discount the “setting, culture, or system” that people act within. Try to find what social, environmental, organizational, institutional, economic, or political influences might be at play.
Know that “winning” ultimately requires “mustering the truth and the best information for making decisions, not deferring to people based on title, rank, or anything else” (p. 31). But more than knowledge, we need the wisdom to know what we know, how we know, and the limitations of our knowledge (pp. 52, 103-104). Do not pretend to know what you do not know.
Also, be aware of and admit your mistakes as early as possible so you can learn from them and fix them (p. 105). Organizations that report more errors are usually more productive. If you want “better performance instead of the illusion of it, you and your people must tell everyone about problems you’ve fixed, point out others’ errors so all can learn, admit your own errors, and never stop questioning what is done and how do to it better” (p. 107).
However, your supervisions might not always be supportive because “facts and evidence are great levelers of hierarchy,” and the evidence might threaten the pride or status of your bosses (p. 31).