Conversations about education are often peppered with the phrase “research shows……” But not all research is alike, and I thought it might be useful to provide a brief overview of educational research methods as I see them.
Educational research studies use both quantitative and qualitative methods. Quantitative research relies on statistical analysis to look for relationships, based on samples that are of sufficient size to produce meaningful results. For example, there are several quantitative studies that have shown a relationship between the SAT or ACT scores of teachers and the reading or math scores of the elementary students that they teach. The higher the teacher’s score, the higher the average student score. Since we know that there are important factors other than teacher scores that affect student scores (such as students’ socioeconomic status and students’ prior achievement, for example) studies like these add those variables into the equation in order to control for them. As my former statistics professor used to say, it’s the beauty of mathematics.
The advantage of quantitative research is that it relies on large samples to discern relationships that hold true in general. Thus, quantitative research can be a good basis for making policy decisions. We might decide, for example, that SAT score would be one factor to take into account in screening applicants for teacher positions. There are a few caveats to bear in mind when thinking about quantitative research, however. First, most researchers will only draw conclusions from results that are statistically significant at the .95 confidence level. That means that there is a less than 5 percent chance that the relationship shown is caused by chance. Five percent is small, but it’s not zero---lots of things happen that have less than a 5 percent probability. Thus, we usually would want to see a number of sound, large-scale studies showing the same relationship before we would make a policy decision. Second, we have to beware of spurious correlations. We can show statistically that there is a relationship between students’ shoe size and intelligence as measured by some tests. But we would be nuts to choose kids with big feet for gifted classes: kids will test higher as they age; their feet just come along for the ride. Finally, what’s true in general doesn’t hold true in every case. It’s important to keep the exceptions in mind and avoid making policy that harms those who don’t fit into the general pattern that the research shows. You have to keep your wits about you as you draw conclusions from, and make policy based on, quantitative studies.
While quantitative research usually draws its strength from large numbers, qualitative research is based on close observation and analysis. Sometimes a case study method is used, in which one or more situations of interest are studied in depth in order to try to understand the mechanisms at work. For example, a teacher quality study might focus on what teachers who are rated highly by their principals do in the classroom, in order to draw generalizable conclusions about the methods and behaviors that characterize good teaching. Qualitative research is particularly useful at teasing out answers to the “why” and “how” questions that quantitative research often can’t answer. Once we suspect, for example, that teachers with higher SAT scores might cause higher student achievement, a good follow-up qualitative study might videotape a group of high SAT teachers, and another group of lower SAT teachers, and try to determine how they differ. Maybe teachers with higher SAT scores understand math better. Or maybe they don’t give up in the face of obstacles, or have better self discipline, or set higher expectations. Understanding that would tell us a lot.
Qualitative studies add insight, richness and depth. What we have to bear in mind about qualitative studies, however, is that they are often based on just a few situations, and that observation is fallible. Some of the most effective teachers might not fit a particular researcher’s model of what good teaching should look like. (This is a particular concern of mine, I confess, since some of the people I’ve learned the most from in my life didn’t fit the typical education school picture of good teaching. And when I ask people to describe the teacher who made the most difference for them, I hear about a remarkable range of approaches and styles, some of which are pretty far out of the mainstream mold.) So you can see that I test research approaches against my own experience and observations, too, which some researchers decry, but seems to me essential and anyway unavoidable. (Sociology majors might remember the insights of C. Wright Mills, who drew on the private problems of the individual to help frame the big problems of social science research.)
In some schools of education, the qualitative and the quantitative researchers don’t mix. It’s a classic liberal/conservative standoff in which the qualitative researchers seem to believe that the quantitative researchers have no hearts, while the quantitative researchers seem to believe that the qualitative researchers have no brains. I think both types of research are important. Even the best tools can be misused, and research tools are no different. The key is to be clear about the benefits and limitations of all our research tools, and to remember that we are dealing with the complex lives of real kids in real schools.
I’m on vacation next week, but the following week I’ll provide a little summary of the current state of research on student achievement, and the growing contributions of the economists who are taking quantitative research to a new level.