Based on years and years of writing about science and studying psychology, here is my Grand Unified Theory for Why It’s So Easy to Overestimate Our Knowledge of a Topic.
Let’s say you want to learn about psychology/architecture/physiology. First, that’s easy! We’re all lifelong students of behavior/buildings/bodies. So you’re starting here:
Then you start reading and filling in that area of your knowledge. Of course it’s not incomplete:
You’re learning. Maybe you’re reading lots of blogs. Some articles. Books. And after all of this, you’re starting to see the same research studies and term being used and repeated, over and over. You know about the Asch studies on conformity. The Stanford prison experiment. The marshmallow test. You’ve seen a lot of the same cognitive biases, over and over. Over and over.
So that’s where a lot of well-read amateurs are. This is the Dunning-Kruger effect: “an illusory superiority that comes from the inability of people to recognize their lack of ability.” You don’t know what you don’t know. It’s easy for a well-read amateur, a weekend warrior, to be unaware of their blindspots for many reasons: first, most people compare themselves to people who know less than they do. This is called a “downward social comparison,” and it’s a great way to make yourself feel better. Maybe you’d get a better grade than that person, or most others—but how much do they know? You don’t have an objective point of view here. Without using an objective measuring stick, it’s easy to fall into complacency. (Especially when learning feels difficult, and you’re getting by just fine with the information you do have.)
So, by this point, you feel very smart and well-read.
Here’s the thing: you’re not.
What you don’t know is that the chart actually looks like this:
Our blindspots are HUGE. HUGE.They’re more like this:
There are a million reasons why a study or entire area of research could be in that unknown white area. A lot of our blindspots exist because of the way that scientific information tends to spread to the general public, which is very similar to the way that cultural items—movies, music, books—become hits. It starts with the creation of a study. We’ll use wine to symbolize a study, because anytime that a study makes it through the process of being peer-reviewed is always a good time to test one’s tolerance for booze:
On its own, a single study will get published in a journal, like the Journal of Personality and Social Psychology, the Journal of Experimental Social Psychology, the Journal of Health Psychology, or Psychological Science. If a study is vibrant, related to something in the news, or particularly interesting, psychology writers might help spread it out into the universe, like so:
Seen: the dropping of knowledge into new territory
It might be a slightly nerdy, technical website or publication that’s not very popular. But what’s in the glass is always easier to read than what’s in the bottle, so it spreads a little more:
Depending on what journalists and editors see and find interesting, the media attention might stop there. Or it might keep going, getting spread out to more places:
Eventually, you start getting a “superstar” effect, where it’s everywhere. This is the marshmallow study, your Stanford prison experiment, your Asch conformity studies: they’re the Taylor Swift of psychology experiments. They’re like memes: you can’t get away.
Because you know so many of these and keep seeing them everywhere, you feel fine filling in your square in like this:
Now, what don’t you see? All of the scientific knowledge, studies, and information that was never written up at all.
Some of this wasn’t published as a study: researchers collect tons of unused data. Labs, funding, timelines, deadlines—all of these things are immensely complicated, and lots of information can get lost in the process.
Perhaps a researcher tried an experiment several times, but didn’t get a statistically significant result. There currently isn’t anywhere to publish information that would ultimately be helpful: “we did this, but nothing happened. Let this be a lesson to other labs, it’s a waste of time!”
Sometimes a study gets published in an obscure journal or doesn’t get picked up. Things happen. And if a study doesn’t get written about right away, the odds of it getting written about in popular media drop off very steeply, very quickly. Even if it’s a great study that links to a new, intriguing line of thought, you might never hear about it.
We know from the world of music that there’s no direct correlation between popularity and quality—but the idea that “it must not be that good if I haven’t heard about it” is all too prevalent. Why certain studies get picked up and become popular boil down to the same factors that influence our decisions on a daily basis: the whims of daily events and the opinions of tastemakers—popular writers, editors at prestigious publications.
HOW CAN WE MINIMIZE BLINDSPOTS?
If you’re doing research or want to learn about a new topic, I recommend learning about things in a systematic, deliberate way in order to minimize your blindspots. Learning requires building a good framework for the area you want to learn about, recognizing patterns, and maintaining a sense of intellectual humility: owning the fact that we all have blindspots.
1. Read very basic texts in the area you’re researching: intro books are your friend. Get an overview. Remember: the area of a subject you’re interested in is just one tiny area of it. Divide a pie chart into several pieces: Social Psychology is always going to be just one area in psychology. It is not 95% of psychology. If it feels like one area represents the bulk of a discipline, that’s because your reading habits are lopsided.
2. Find a syllabus for a class in the more specialized subdivision that you’re interested in. (You can restrict Google searches to education domains for better results.) Try a well-known university and a few less-known ones. (Professors are people with personal, idiosyncratic tastes, after all. They’re understandably biased towards their own line of research.) Read. You can often find free sources online.
3. Know your living sources. Get intimately acquainted with Google Scholar, which lets you see how many times a paper has been cited and how influential it is.
4. Get over your fear of going straight to the source. Identify key peer-reviewed journals in the area that keep popping up. Go there. Read. Read papers and abstracts that interest you. Sign up for email alerts when new articles are published. Brain and Cognition. APA Journals.
5. Review articles and journals are a great way to read about trends from the past few years and theories that link current studies. Review of General Psychology. Annual Review of Psychology. Current Directions in Psychological Science. Neuroscience & Biobehavioral Reviews. These are all things that I love.
6. Identify key researchers in this area. Sign up for alerts on Google Scholar to find out when they publish new articles. Better yet: visit the website of a lab or researcher you’re interested in. Reading how they describe their work, and how their papers have evolved over the years, is a wonderful way to see how ideas evolve over time.
7. Test yourself! Write blog posts and papers. Take quizzes. Reading alone doesn’t mean that you’ll retain all of the information.
8. If you ever find yourself saying “now you’re just complicating things,” or “that’s ridiculous, you’re just splitting hairs,” back up. That’s a sign that you’re missing enough context to understand why something is important. (It’s also a sign of defensiveness because treasured opinions and worldviews are being questions.) Things really are complicated.
9. Researchers are people who have their own cognitive biases. They’re not perfect. The process of collecting data, getting funding, and publishing a paper are all run by humans with their own motives and agendas, even if they’re not aware of what they are. Leave your ego at the door. Stay interested in learning for the sake of learning, rather than learning in order to back up a hunch or idea.
10. Just because multiple people agree says nothing about how accurate those beliefs are. It’s possible for everyone to be wrong at the same time. (See: what we believed a few decades ago.) Science progresses one funeral at a time.
Yes! There’s no better feeling than talking with someone who appreciates your work. Kyle Ingham not only appreciated my book, but he just got it—so many of his favorite quotes and ideas were ones that I was just hoping that others would highlight.
How to make better decisions and improve your situation even when the world seems completely random.
Here’s the podcast.
- People use luck as an explanation when they have no other explanation
- Assigning external causality things that happen takes away our power
- How optimism and confidence can “create” luck
- Why just showing up and being more social can improve your luck
- People do judge books by their covers, and how to use that to your advantage
- Half of your appearance is genetic, half of it is what you do with it
- Our luck has a direct connection to the stability of our self-esteem
- Why changing what you pay attention to can change your life
Thank you so much to Wharton for having me as a guest! Here’s one of my favorite podcast interviews, where I joined hosts Anne Greenhalgh and Mike Useem on Leadership in Action.
Originally aired on SiriusXM Channel 132, Business Radio Powered by The Wharton School. SiriusXM’s Business Radio is a nationally broadcasted, 24/7 radio channel powered by the Wharton School. We feature access to expertise on a wide range of business topics from sports to technology, delivered directly by the world’s top business experts from the Wharton School at the University of Pennsylvania. SiriusXM also has internet streaming options online and via apps for round the clock access. Tune into channel 132 to hear the latest!
Have you ever felt cold in an office? Or noticed how many women complain about the office being chilly?
That’s because the standard room temperature was devised using a 40-year-old man weighing 154 pounds to calculate the default metabolic rate.
In fact, because of biological differences (some in a study here), on average, women prefer the indoor room temperature to be 77. Men? 71. Of course people are unique, but the gender differences in these averages is robust. Robust enough so that you’ve probably noticed it. (Feeling uncomfortably cold in an office can actually take a toll on your productivity.)
The point is that selection bias can impact findings easily, even when we don’t realize it. Selection bias refers to the way that a non-random selection of research subjects can skew (bias) the results. If I was at a basketball game and “randomly” wandered down to the court and measured the height of five players, I shouldn’t use that data to make a claim about the average height of men in the arena.
Selection bias creeps up in psychology studies all the time—even when we don’t realize it.
Surely you’ve heard of The Marshmallow Test. If not: a researcher at Stanford named Walter Mischel began testing kids’ ability to delay gratification. In his seminal study, he placed children in front of a piece of candy. Before he left them alone with it, he told them that if they could hold off eating it until one of the researchers returned—roughly 15 minutes—they’d be ultimately be rewarded with two marshmallows.
The children who were able to wait for the second marshmallow not only scored higher on their SATs, but reached higher levels of career success than their grabby peers. They were less likely to abuse substances or be obese; even their relationships were better. In a huge follow-up study 40 years after the original one, researchers found differences in how the brains of “delayers” and the “nondelayers” responded to rewards.
Well, duh. Isn’t it obvious that the ability to delay gratification is important? James Clear made that the theme in “40 Years of Stanford Research Found That People With This One Quality Are More Likely to Succeed.” Angela Duckworth’s famous TED talk on grit also hammered this idea. Keeping your eyes on the prize over the long term is crucial for success. Grit, the ability to delay gratification, perseverance, self-regulation, self-control—whatever you want to call it—leads to good things. But not always.
The problem with the original study was the selection bias in Mischel’s studies.
“The original results were based on studies that included fewer than 90 children—all enrolled in a preschool on Stanford’s campus.” [Atlantic article]
In other words, the kids who participated in the study did not represent a random sample from the population at large. They were upper middle class and white. Their parents had some affiliation with Stanford—even if just living in that area. They were the perfect example of what Joseph Henreich called WEIRD in a famous paper: Western, Educated, Individual, Rich, and Democratic.
“In restaging the experiment, Watts and his colleagues thus adjusted the experimental design in important ways: The researchers used a sample that was much larger—more than 900 children—and also more representative of the general population in terms of race, ethnicity, and parents’ education.” [Atlantic]
According to Psychological Science: “The results showed that, although children who were able to wait and resist temptation tended to have stronger math and reading skills in adolescence, the association was small and disappeared after the researchers controlled for characteristics of the child’s family and early environment.”
Behavior is the product of both the person and the situation. This is a form of attribution bias: thinking that not waiting for the marshmallow points to ingrained flaws in someone’s character, when in fact not waiting may just be a rational response to an otherwise shitty situation.
What if you, as a kid, had seen people wait for the marshmallow and come up empty-handed? What if an adult who promised you a marshmallow never followed through? What if you had seen that scenario play out several times?
External circumstances and environments exert a compelling impact on our lives, teaching us when and where our efforts are even worthwhile. It’s logical—it’s how we learn. People repeat behaviors that have been rewarded in the past. If you learn that hard work and patience will pay off and make that effort worthwhile, you’re more likely to try.
If you have a hard time waiting for the second marshmallow, consider your past experiences. For example, suffering from a macroeconomic shock (seeing scads of second marshmallows being yanked away) can alter your entire life, if it happens at a critical moment in your life.
In “Growing Up in a Recession,” Italian economists found that this can happen to an entire generation:
We find that individuals who experienced a recession when young believe that success in life depends more on luck than effort, support more government redistribution, and tend to vote for left-wing parties. The effect of recessions on beliefs is long-lasting.
Remember: waiting, being patient, and being able to delay gratification are adaptive behaviors. But they’re much easier to develop and practice if you’ve always had people follow through.
A year ago at this time, I was getting feedback on my book and making a few final tweaks to the website. The designer I hired—well, let’s just say that I learned a lot about who to hire. The big takeaway was this: to avoid making a decision you’ll regret, always start looking for something well before you actually need it.
•Are you single? Don’t date if you’re lonely or really dislike being alone. Desperation is a surefire way to end up in an abysmal, subpar relationship.
•Are you thinking about hiring someone to expand your team? Start putting out feelers. Now.
•Think you’re going to have a big presentation, date, interview? Start browsing for work clothes now, not just the night beforehand.
•Do you want a new job? Don’t wait until you hate going to work to look for a new gig. Don’t wait until your bank account is getting close to zero.
“Choices do more than reveal preferences; they also reflect subtle, yet often quite reasonable, dependencies on the environment.” (source)
Don’t get me wrong—constraints are good! They’re necessary, because if we don’t make a decision, we’ll never get anything done.
But the downfall of a time constraint is that when we think we have to choose—and, like, soon—it can become incredibly easy to make a subpar decision. Our brain is remarkably efficient at factoring in real world constraints, without even realizing it. When you know that any choice is better than no choice, you stop collecting data and start zeroing in on an option, ultimately choosing whatever the best option seems to be—out of whatever options you think you have.
Why else would so many online retailers put up banner ads, counting down how many hours are left in a sale? (Do you really think that Banana Republic isn’t going to sell you that sweater tomorrow?) I’ve seen many friends wind up in relationships best described as “meh” because they’ve fallen victim to the false ticking of their biological clock. We look at someone, think “this is a human who has agreed to spend time with me,” and try to make it work. You might end up taking a subpar job that sets your career back. Out of loneliness, you might become friends with people who don’t share your values. You get that shirt that you never end up wearing again, simply because you needed something that would cover your tattoos the next day and you really hate shopping.
I make the best decisions when I’m not actively looking for something. Most of us do. That’s when we stop unknowingly limiting ourselves to “the three options that happen to be in front of me right now.”
Instead, we’ve waited until something is so great that it grabs our attention, distinguishing itself from everything else in the environment.
To make the best choice? Stop actively looking. Think about “things you’d like to do someday” (hire, date, furnish your living room, take a class), and let that simmer in the back of your mind. Don’t pick something because you feel like you need it: pick something that you really want. Let the thing find you.