## · •There are several fallacies and logical traps to avoid when making causal arguments about correlated variables, to include reverse causation, omitted variables, intervening variables, and spurious correlation.

CHAPTER 2

Theories, Hypotheses, and Evidence

J. Tyler Dickovick; Jonathan Eastwood

· A medical worker undergoes Ebola response training during the outbreak that began in 2014.

N THIS CHAPTER

Introduction to Theories, Hypotheses, and Evidence

Theories

Hypotheses

How Theories Emerge and Are Used

Types of Evidence

Hypothesis Testing

Correlation

Causation

Critiques: Using Theories and Evidence

Empirical Critiques: Using Deviant Cases

Theoretical Critiques: Improving Theories and Hypotheses

The Challenges of Measurement: Biases, Errors, and Validity

THINKING COMPARATIVELY

Qualities of Good Analysis and Argumentation

Step 1: Asking Good Questions: Why?

Step 2: Hypothesis Testing: Generating Good Hypotheses and Testing Them Fairly

Step 3: Balancing Argumentation: Evidence, Originality, and Meaningfulness

In 2014, a deadly outbreak of the Ebola virus struck several countries in West Africa. Guinea, Liberia, and Sierra Leone were especially hard hit. Medical professionals from around the world came to the region in an attempt to contain the outbreak. Some villagers, however, did not welcome them; and in fact, at least one youth group mobilized to fight off the doctors because they noticed something: soon after the doctors arrived, people in a village began to die. As The New York Times reported, some villagers reached a simple conclusion: the doctors bring death, and the way to stop Ebola was to stop the doctors.1

This kind of faulty logic almost certainly proved deadly to some. Examples like this show how important it is to have good theories that can help us understand—not misunderstand—how the world works. Such an example also shows how important it is to interpret evidence correctly. In this story, it was the case that the disease was claiming lives despite the doctors’ best efforts, not that the doctors were doing the killing. Villagers were confusing correlation—observing factors that accompany each other—with causation, or an argument that one thing causes another.

In this chapter, we discuss how theories work. We discuss how to form hypotheses, or educated guesses about what will happen under certain circumstances, and how to avoid certain pitfalls in testing those hypotheses. All this will prepare you better for examining the issues of comparative politics that make up the rest of the book.

## Introduction to Theories, Hypotheses, and Evidence

Social scientists look for convincing answers to important questions about why things happen: why are some countries democratic and others not, or why do revolutions occur, or why do some countries have two main political parties while others have many parties in their legislatures? The first step in comparative politics is asking good research questions about the causes and effects of political events. Chapter 1 gave us an approach—the comparative method—to begin to answer those questions by comparing and contrasting cases, most often different countries or specific events in different countries. We may examine the political party systems of Germany and France, or the communist revolutions in Russia and China, for instance. To do so, we juxtapose the facts of the different cases to make an argument about the similarities and differences between them.

In this chapter, we talk about the tools we need to answer questions, with a focus on two elements that help us to formulate possible answers: theories and hypotheses. We discuss what theories and hypotheses are and how they differ from one another. We then discuss how evidence is used to test hypotheses and theories.

### Theories

Theories are general explanations of empirical phenomena, or explanations about how the world operates. A theory aims to explain more than just one or two cases or examples, and it is typically backed by a considerable number of supporting facts. An explanation or framework in the social sciences will rarely earn the right to be called a theory if we cannot find considerable support for its arguments in the real world.

This may not be the only way you hear the word theory used. There is also a more casual everyday usage to describe a hunch or idea. For instance, imagine a friend who says, “The Chicago Cubs are going to win the World Series this year, that’s my theory.” From a social scientific point of view, this is a prediction, not a theory. It may be a good or a bad prediction, but it is speculative at best, a hopeful guess. Without some logical argumentation and backing in facts, it is not really a theory based on reason and evidence. If, on the other hand, the friend offers a detailed explanation that shows how the team with the strongest pitching routinely wins the World Series, and proceeds to detail how Chicago’s pitching is stronger than that of other teams, then the friend is approaching a general theory about the relationship between two variables: pitching and championships. In this theory, pitching is the cause and winning the championship is the consequence, which is also called the effect or the outcome.

In political science, there are two different types of theory, typically referred to as normative theory and empirical (also known as positive) theory. Normative theory deals with questions of values and moral beliefs. An example might be the question “What is the best kind of political system we could construct?” This is a matter of morals and ethics. Empirical theory, by contrast, deals with empirical questions. An example is “Which factors are most likely to produce a preferred political system?” This is about the factors and variables that cause things to happen. In this book, we are mostly focused on empirical theories: we discuss theory as a general explanation of why things happen.

Web Activities

### Hypotheses

Hypotheses are specific proposed explanations for why an outcome occurs. To answer research questions, we may generate or formulate hypotheses that we think can explain a set of facts upon further research. Hypotheses are not explanations already backed by lots of evidence. Instead, they are possible answers to a question, which we plan to test out by applying them to data, looking at specific observations (or “cases”) to see if there is evidence to support the idea. Informally, you can think of them as hunches. If the hypothesis receives that support from the evidence, it may become a thesis in an argument.2

Developing hypotheses requires us to make imaginative leaps from unanswered questions to possible explanations. Hypotheses can be generated from existing theories in a deductive fashion: starting with general ideas and then testing whether they work on specific examples. For example, say we are asking about why an anti-colonial revolution happened in a certain African country in the 1950s. We may begin our research with a major theory that holds that social revolutions (such as the French Revolution, Russian Revolution, or Iranian Revolution) are caused by the social upheavals produced by modernization. We seek to apply this theory to the African country we are studying. Using the theory as our general model, we might hypothesize that the anti-colonial revolution in the African country was produced by a history of modernization. Another way to think of this sort of approach is to consider it an effort to test an observable implication of the starting theory.3

Not all hypotheses are deduced from general theories, of course. Some can also come from looking at a case that deviates from a particular theory. We can learn a great deal from so-called deviant cases, or “outliers,” that do not do as we might expect. For instance, in many international comparisons, the United States is a deviant case. It has both higher income inequality and greater differences in life expectancy between racial groups (to name just two variables) than one might expect based on its level of economic development. By focusing on some characteristics that make the United States different from other cases, we might sometimes understand general relationships better. For example, perhaps it is not a country’s overall level of economic development that predicts the life expectancy of its people, but individual life chances. By this thinking, U.S. income inequality may help us to account for the fact that high development does not lead to high life expectancy for all U.S. groups.

We often formulate a hypothesis with some initial knowledge of the topic at hand, but we do not want to presume we already know the answer to the questions we are asking. We do not normally aim to create a hypothesis in an inductive way—moving from specific observations to general claims. That is, we don’t do the research, find the answer, then go back and propose our hypothesis (although sometimes our analysis does suggest new hypotheses, and inductive approaches to theory generation do exist). Instead, we formulate our hypothesis with an open mind toward what answers we may find. Our hypotheses may be supported or rejected by the research we do, so there is always the possibility that they are wrong. In fact, most hypotheses are wrong, and rarely if ever can we fully confirm or disprove a hypothesis with limited research.4 The goal is not to pick the correct hypothesis at the outset but rather to learn something from the study we undertake. In fact, many social scientists believe that our knowledge advances more from refuting hypotheses than from defending or supporting them.

Hypotheses and theories inform one another. Theories help guide us in formulating hypotheses, and confirming hypotheses may either support or undermine theories. In general, hypotheses are more tentative and speculative than theories. A specific hypothesis is generated for each research question and is put on the line to be tested in each case. While the evidence from testing a specific hypothesis may support or oppose a particular theory, it usually is insufficient to reject or confirm a theory by itself. Generating a theory is a more elaborate, long-term process than generating and testing a single hypothesis.

After testing hypotheses for a specific study, scholars will typically offer a thesis, a claim to argue on the basis of evidence from research. One can think of a thesis statement that usually appears near the beginning of a well-written scholarly paper. In comparative politics, a thesis is an argument supported by the research evidence that comes from testing a hypothesis. (It is no coincidence that the word hypothesis—with the prefix “hypo”—roughly means something “less than” or “not yet” a thesis.) While a thesis has evidence supporting it, that does not mean it is a full theory. Before achieving the status and prestige of being called a theory, an idea requires ample evidence to support it, typically based on research by many scholars. For the most part, students of comparative politics test hypotheses, make specific claims in the form of theses, and are expected to use evidence to argue in support of their theses, taking account of existing theories. We are informed by theory and can contribute to debates by theorizing, but we rarely craft or falsify entire theories alone.

### How Theories Emerge and Are Used

Theories emerge and are used all the time to explain the world around us. Let us take a prominent theory from beyond comparative politics that has long featured prominently in social and political debate: the theory of evolution. First, we offer a very abridged version of the theory, followed by one specific hypothesis derived from the theory.

Theory (abridged): The origin and development of species are based on a process of natural selection in which organisms with a genetic advantage in a given natural environment thrive and propagate their genes, whereas organisms at a genetic disadvantage will fall out of the gene pool over the long term.

Hypothesis (example): The theory of evolution accounts for humans’ walking on two legs. Human ancestors first began to walk on two legs in African savannahs where grasses grew tall; those walking on two legs had an advantage over similar four-legged mammals because they could better see and more easily flee predators.

In testing this hypothesis—that humans first walked upright to flee predators—a scholar will examine whether the evidence is consistent with such an explanation. The evidence uncovered may include fossils and archaeological evidence. As part of the work, the scholar may note that other prominent arguments are inconsistent with one or more substantial pieces of evidence, thereby making this hypothesis relatively more capable of explaining the observed facts.

Of course, even if the scholar finds that the evidence is consistent with the hypothesis, a single study will not be the end of the story. Counterarguments will emerge. Indeed, the existence and progress of the social sciences depends precisely on the common efforts of the scholarly community to question existing explanations and to provide alternative ones. In our example, some scholars who accept the evolutionary perspective may argue that humans first walked upright to conserve energy while foraging for food. Scholars who reject the theory may also contribute arguments to the debate, and even non-scientists may do so, as long as their arguments are tested empirically. We can pursue the scientific endeavor by further testing related hypotheses to see how the theory changes as a result.

We narrow in on good explanations by finding increasing evidence that certain hypotheses are consistent with the evidence while others are inconsistent with the evidence. We can rarely, if ever, confirm a hypothesis or prove it fully true; rather, we can find that a hypothesis is increasingly viable as we find more and more evidence to support it. Ideally, much like a courtroom lawyer that has the evidence on his or her side, we will make our case “beyond a reasonable doubt” as we defend our claim.

Theories have facts and evidence supporting them, but these are not proof that a theory is valid and correct in all circumstances. Often, a wrong theory will hold sway for a long period of time until it is supplanted by a stronger theory. For example, the earth was long believed to be situated at the center of the universe, and this appeared consistent with many facts, such as the sun and moon rising and falling beyond the horizon each day. However, this theory eventually came into conflict with observations that suggested that the earth revolved around the sun. Both theories persisted for a time until it became clear that the heliocentric (sun-centered) theory best explained the structure of our solar system. Thus, competing theories may coexist, and there may simultaneously be facts and evidence that support a theory and other facts and evidence that contradict the theory. Theories may ultimately fail and be rejected, but ideally theories only “die” when replaced by new ones that better explain existing evidence.5

Theories in political science explain tendencies and help us understand many cases, but there are almost always exceptions to the rules. Nothing in political science works in all cases the way the laws of physics work everywhere on earth. For example, as you will read in chapter 6 (and as briefly mentioned in chapter 1), there are several competing theories to explain why countries become democracies. There is considerable evidence that wealthier countries are likelier to be democratic than poor countries, but this does not mean every rich country will be a democracy and every low-income country will be under authoritarian rule. Rather, the theory of the link between wealth and democracy posits a tendency, much as eating healthy foods and not smoking will tend to increase one’s life expectancy. Not everyone who eats well and avoids smoking will live to old age, and not everyone who smokes and eats junk food will die young. Cause-and-effect relationships in the social sciences are general patterns, not absolute laws. As a result, building theory is an intensive process over an extended period of formulating and testing hypotheses, gathering and examining evidence, and understanding and synthesizing debates. Theories are imperfect but can be improved over time.

Since theories compete with one another as the best explanations of social phenomena, it may be natural to think of scientists competing with one another to come up with the best theory. This is true in part, but the social sciences are also a collective endeavor. In this sense, when a theory is rejected, it represents an advance of our understanding. Even critiques of one scientist’s effort by another scientist are part of the process of testing and contesting the best explanations.

### Types of Evidence

For most students being introduced to comparative politics, the dominant form of evidence will be qualitative, meaning it comes from accounts of historical or contemporary events. For instance, if I wish to examine the hypothesis that the French Revolution of 1789 was caused by the emergence of a self-conscious middle class (bourgeoisie), then I may look to accounts of that class and its attitudes and involvements in political life in the years leading up to 1789 in France, perhaps comparing it to other countries where a revolution did not take place. In this case, my data are not numbers and figures inserted into a spreadsheet but rather the detailed accounts of historical record. I may examine my hypothesis using the facts of who did what, when and where they did it, and how. Qualitative evidence may come from many sources such as written works like constitutions and laws, historical or journalistic accounts or reports, and interviews or surveys of people.

Social scientists also use quantitative data such as statistics and figures as they aim to make inferences, or conclusions based on evidence, about cause and effect. Examples include measures of average incomes or average life expectancies across countries. Such quantitative comparisons may be undertaken using national statistics from government agencies, numerical data from surveys, or data collected by researcher observations. Various data sources may be used to compare and contrast outcomes in different countries. At a more advanced level, such descriptive statistics can be used to formulate and begin to test hypotheses about the causes and effects of differences between countries. Other quantitative research in comparative politics focuses on the construction of formal mathematical models of the strategic behavior of individuals and groups in political situations. Quantitative data differ from qualitative data in their presentation, but both types are used to generate and test hypotheses. While the details of statistical methodologies and formal mathematical modeling are beyond the scope of this book, we work from the premise that both qualitative and quantitative work may be used to categorize and describe differences across cases, but they can also be used to examine hypotheses about the causes of those differences.6

In comparative politics, you will use historical accounts and data more often than you will make predictions about the future. This is because we have real evidence only for things that have happened and not for what might happen. Of course, the past may give us expectations about the future, which is why we hear that those who fail to learn about the past are doomed to repeat it. But in terms of concrete evidence, we cannot know what has yet to happen. For this reason, we work with existing cases to develop hypotheses and theories. For instance, we may hypothesize that China, which is not currently a democratic country, will move toward democracy as it grows wealthier. This hypothesis may come from observations about what has happened in other countries as they have grown wealthier. Well-regarded theories may strongly suggest that China will democratize, and we may expect that it will do so, but to test the hypothesis we will have to await future events. Evidence comes only from events that have happened.

The Qualitative–Quantitative Debate

Spring Festival travel rush in Shenzhen City, China, 2012. Will China move toward democracy as its middle class grows larger? We address this question in the discussion of democratization in chapter 6.

## Hypothesis Testing

The core of comparative politics is examining hypotheses about cause and effect between two or more variables, using observations from different cases. We defined variables in chapter 1 as some measure that can vary from one observation to the next. Examples range from a country’s average income or average life expectancy, to whether a revolution occurred in a given country, to the most prominent religion in a particular state, to the religion of a particular person.

In social science, cause-and-effect arguments are based on examining different variables and how those variables relate to one another and may depend on one another. If country A is wealthy and country B is poor, what does country A have that country B does not that makes it so?9 An explanation will hinge on identifying what variable might cause A to have become rich and B to remain poor. Our goal will be to identify what other variables go alongside wealth that are lacking in countries that are poor and to examine whether those variables made the difference. Our first key distinction here is between correlation and causation.

### Correlation

Correlation measures the association between two variables. When two variables correlate, they are related to one another (or, to separate the words, they “co-relate”). To use a simple example, the temperature in many places will correlate with the month of the year: when it is February in much of the Northern Hemisphere, the temperature will be relatively cold; whereas in July, the temperature will be relatively hot. This does not mean it is impossible to have a hot day in February or a cold day in July, just that there is an association in general. There is thus a correlation between the variable “month of the year” and the variable “temperature.”

If two variables have a positive correlation, they tend to increase together. One increases as the other increases. An obvious example is the income of a person and the amount the person spends on luxury goods. People with low incomes cannot afford to spend money on luxury goods, while the wealthy may spend a large amount on luxury goods. These two variables are positively correlated. A negative correlation is just the opposite and means that as one variable tends to increase, the other tends to decrease. An example might be the number of cigarettes one smokes per day and one’s life expectancy.

Just as we can find a positive correlation between wealth and democracy, we can conversely find a negative correlation between another pair of variables: poverty and democracy. Consider the number of people in a country living on an income below \$2 per day (call this variable the absolute poverty rate) and the level of democracy. In this case, the rich countries have relatively low levels of poverty and high levels of democracy, while many countries in Africa have high levels of poverty and low levels of democracy. When we look at the nearly two hundred or so countries in the world today, these correlations are apparent, even though it should be noted that there are some countries that are rich but not democratic and some that are low-income yet are democratic.

Correlation: Wealth and Democracy

### Causation

Causation exists when one variable causes another. This helps us answer the fundamental questions raised in chapter 1, such as “Why are some countries democracies?” Recall that why questions are often best answered with because answers. As the word because implies, answering why involves explaining causes. Without causal arguments and theories, correlations are just patterns in search of an explanation. When we have causation, we usually have correlation, but the opposite is not true. Failing to distinguish between correlation and causation can lead to a variety of problems, as we will show.

Does the correlation between wealth and democracy prove that getting rich causes democracy to happen? Not necessarily. It may be that this correlation points in the direction of a causal argument, such as wealth → democracy. Or maybe the other way around: democracy → wealth. On the other hand, it may be that the correlation exists, but there is no causal reason for it. It may be simply due to chance that rich countries happen to be democracies. Or there may be other factors that result in both wealth and democracy, so called “confounding variables.”

As it turns out, one of the central theories of comparative politics suggests that countries that grow wealthy are likely to become democratic for specific reasons we detail in chapter 6. The causal argument, beginning with the positive correlation between wealth and democracy, finds that historically, countries have developed a middle class as they have grown wealthier. This middle class, rather than the elite, ends up being a central force that pushes for more rights for all citizens. In poor countries without a middle class, democracy is unlikely to succeed, but growing middle classes in countries that are growing rich have helped bring democracy with them. While the correlation here does have a causal explanation, notice that the correlation needed an argument and logic to bring the story together and to make the fact of the correlation into evidence that supports an argument. Note also that the presence of a causal story here does not mean that the proponents of this view have established that this is the correct causal story.

We cannot assume that all correlations between two variables (call them X and Y) mean that X leads to Y. We will use various examples to illustrate possible relationships between variables. The first of those was the causal argument that X leads to Y (Figure 2.1).10

Figure 2.1 Causal Relationship Between Correlated Variables (X and Y)

Causation: Legislative Elections and the Number of Political Parties in Legislatures

But there are many other possibilities. Figure 2.2 shows some possible relationships between variable X and variable Y that are not the simple causal relationship where X → Y. If we assumed X → Y in each of these cases, we could run into a number of analytical problems.

Figure 2.2 Possible Problems with Causal Arguments About Correlated Variables (X and Y)

We discuss each of these problems in order.

1. (1)Definitional problems and falsifiability problem

The first problem is one that is rarely noted because it apparently involves arguments that are “too correct.” In reality, one common problem is confusing cause and effect between two variables with two variables that are the same by definition. If X is measuring the same thing as Y, they will correlate perfectly. But this is not because X → Y, but rather because X = Y.11 A common problem for comparativists is defining two variables that are so nearly the same that the causal argument is meaningless, or tautological. This definitional problem relates to the problem of falsifiability, which is the idea that for an explanation to be meaningful, it must be contestable. To argue that something is true means something only if there is a chance it could at least possibly be incorrect and could be proved wrong. For instance, say we are asked why a baseball team won a game, and our “analysis” is that the winning team “just scored more runs” than the losing team, or “just got it done.” This argument is correct, in the narrow sense that it is not inaccurate, but it is also meaningless, precisely because it can never be otherwise: scoring more runs over the course of a game and winning the game are one and the same, by definition. By contrast, if we say that none of the world’s democratic countries will ever again succumb to dictatorship, then that argument is falsifiable because a contrary example is possible.

Definitions and Falsifiability: Dictators and Dictatorships

1. (2)Reverse causality problem

The reverse causality problem is rather simpler to understand. Our story at the beginning of the chapter held that villagers noted that contagious epidemics brought doctors into villages and people in the villages began to die, and they concluded that doctors were causing the illness rather than the illness bringing the doctors to the village. In this case, two variables are correlated, but the causal argument linking the two may be the opposite of what we anticipate. Instead of X leading to Y, perhaps Y leads to X. Getting the “causal arrow” pointed in the right direction is essential, and reversing causality has the potential to lead to disastrous consequences.

Reverse Causality: Cancer Rates and Longevity

1. (3)Endogeneity problem

The endogeneity problem is about circularity: It happens when two variables exhibit mutual or reciprocal effects. You may know of a simple expression such as “the chicken and the egg” problem, though endogeneity arises any time variables mutually affect one another. If X and Y correlate and seem to go together, we may be left trying to figure out whether X caused Y to happen or Y caused X to happen. Reasonable people may disagree about which direction the causal arrow goes.12 Endogeneity problems are common in the real world. When we talk of vicious circles (of, say, poverty and dictatorship) or virtuous circles (of, say, economic growth and human development), we are describing a situation in which many important variables are endogenous. Indeed, endogeneity as such is not a problem but a feature of many social and political phenomena. It becomes a problem when we mistakenly claim one variable causes another when the two variables are, in fact, endogenously linked. Even so, social scientists don’t want simply to identify multiple variables as endogenous but to understand more precisely the ways endogenously linked variables interact over time. One of the leading strategies for resolving this dilemma in qualitative research is closely tracing the historical sequence. Where we have good information about when and where things happened, who did them, or how events unfolded, we may be able to determine whether X leads to Y or Y leads to X. If we can identify clearly whether the chicken came before the egg (or vice-versa), we may be able to address this problem. This is not, however, always possible, as the box on education and health shows.

In addition, there are statistical strategies for dealing with this problem that we cannot explore here, but you could learn about them by taking a more advanced social science methods course.

Endogeneity: Education and Health

1. (4)Intervening variable problem

Intervening variables are another potential problem, though they are not always problematic. The situation here is that X leads to Y, but indirectly: the effect of X on Y is mediated through another variable, Z. This is not always a problem. An example is eating fatty foods and having a higher risk of heart disease. Eating lots of fatty foods leads to an accumulation of cholesterol in the arteries, which leads to higher risk of heart disease. Even though there are intervening steps between the actual eating and the risk of heart disease, we can still say eating fatty foods causes a higher risk of heart disease. As long as we can specify the argument and its steps, we do not have an intervening variable problem. The potential problem arises when we miss an intervening variable and this leads us to a wrong interpretation (or, alternatively, if we control for the intervening variable in a statistical model we are using to examine the effect of X on Y). The previous box example illustrates the problem of failing to consider intervening variables (or “causal pathways”).

Intervening Variables: Communism and Democratization

1. (5)Omitted variable problem

We frequently miss or omit variables that should be in our analysis. We observe an empirical relationship between X and Y and assume this means that one creates the other, when in fact both are attributable to a third factor, sometimes also called a confounding or “lurking” variable, because though it is there in the background, we might not see it. If X and Y are positively correlated, it may not be that X → Y or Y → X at all. Instead, some factor Z may lead to both X and Y, thus giving rise to the correlation between them. That is, Z → X and Z → Y. This is a very common problem, and one of the great difficulties of social science involves ruling out likely Z variables of this sort. If you take a course in statistics, you will learn how to deal with this problem if Z is measured. But if Z is not measured, or if we don’t even know to look for it, its presence can “bias” our estimate of X’s effect on Y.

Omitted Variables: Ice Cream Sales and Murders

1. (6)Spurious correlation problem

Finally, there are many variables out there in the world, and some are bound to correlate with one another even in the absence of any causal relationship. Many problems that seem to be of this sort will actually be omitted variable problems upon further investigation, but there are examples of correlations where simply no meaningful causal relationship exists. Lucky superstitions are examples where two variables seem to correlate, but there is no plausible relationship between them. Perhaps your college’s sports team always seems to win when you put your lucky hat on for the game. This correlation may continue for some time, but there is no reasonable scientific explanation linking your hat-wearing tendencies to victory, and no reason to expect that you putting on your hat will lead your team to win the next game. Though the variables “hat wearing” and “victory” may correlate, there is no causation. (Sorry. You can take off your hat now.)

Spurious Correlation: Stock Markets and Butter Production in Bangladesh

## Critiques: Using Theories and Evidence

Evidence can be used to support an argument, but it can also help us counter an argument, and this too is a meaningful contribution to advancing our understanding and knowledge. Evidence can enhance our knowledge by providing a helpful critique of the conventional wisdom. Accordingly, empirical critiques have a prominent place in comparative politics, as do the theoretical critiques they enable.

### Empirical Critiques: Using Deviant Cases

In testing hypotheses, we often hope to find evidence that supports a particular theory. Specifically, we want to find cases that confirm or reaffirm our theory or support our hypothesis. But many interesting advances come from empirical evidence that does not fit a theory well. Deviant cases—those that do not fit a theory or are exceptions or outliers—are very important in advancing social science theory. These cases help us test out why a theory doesn’t work and understand what improvements need to be made to our knowledge. They allow us to make an empirical critique of a theory because the cases do not support it. Much like getting a bad result on a certain test can encourage us to do better where we fell short, so too a deviant case forces us to think about how to improve our arguments.

### Theoretical Critiques: Improving Theories and Hypotheses

Theoretical critiques are new ideas that improve on the logic or reasoning of existing theories. Theory and empirical evidence constantly interact, and where deviant cases help provide an empirical critique, these can help us improve our theories. They often provide the impetus for improvement of the theory. Empirical critiques allow us to advance social science by pointing out anomalies, inconsistencies, and deviations from a theory. Theoretical advances can also come from critiques of the theory itself, through reexaminations of the logic, assumptions, or arguments underpinning it. The following box gives an example of how a theoretical critique emerged from empirical evidence that didn’t fit a theory.

Empirical Critique: Ghana and Modernization Theory (see chapter 6)

Critiques help us craft better arguments and theories. First, they can improve our understanding of scope conditions, or the conditions under which an argument works. Identifying and examining cases that do not fit an argument is a good potential avenue for further research. Second, critiques based on empirical evidence can help improve our concepts and lead to a clearer understanding of what exactly we are studying. For instance, the tiny, oil-rich country of Equatorial Guinea has grown rapidly to become one of the wealthiest countries in Africa, but much of its wealth goes just to the dictator’s family. Studying this empirical example might give us more insight into what exactly a country’s “economic development” means. By identifying weaknesses in arguments and offering alternative explanations, critiques give us better understandings of why things happen.

Theoretical Critique: Dependency Theory in Latin America (see chapter 5)

The Challenges of Measurement: Biases, Errors, and Validity

The challenges of garnering and wielding evidence are multifaceted. Beyond determining how to gather evidence and which pieces to use, we must pay attention to measures and indicators (elements or features suggesting underlying factors). Without careful and thoughtful measurement, we may accidentally introduce biases and errors into an analysis. Bias is a preference for one idea or perspective over another, especially a preference that may result in unbalanced use of evidence or in analytical error.

Bias aside, it is possible to simply make measurement errors, such as by typing the wrong number in a spreadsheet. This kind of error happens more often than you might expect and sometimes in consequential ways. As a well-known example, a spreadsheet data error in work by Harvard economists Carmen Reinhart and Kenneth Rogoff produced erroneous results in highly influential research about government debt and economic growth (note that this error was discovered by a graduate student!).16 Less obvious might be how a measurement cannot fully reflect what it is trying to measure. Most college students have taken standardized tests such as the SAT or ACT, which attempt to measure overall competence in math and language. Scores for most students will fluctuate from one test to the next depending on the specific questions. Whatever their merits, the tests thus have a degree of measurement error in conveying competence. Many social science measures are imperfect, and we should keep this in mind when carrying out our analyses.

A second measurement problem is measurement bias. One example of bias comes from respondents in a survey who are untruthful, whether consciously or subconsciously.17 Another would be if the questions we ask people are interpreted differently by different groups of respondents. Perhaps the most serious form of bias for beginning researchers is seeking to confirm one’s favored hypothesis. This can include a tendency to believe things are a certain way that we want to see them. Imagine that a very ideological capitalist student wants to show that countries with free markets have performed better economically than countries that have more active government involvement in the economy. The eager student knows that the United States performed better than the U.S.S.R. in economic growth rates in the 1980s and uses these cases to “prove” the hypothesis that less government involvement in the economy is better for the economy. Subconsciously, the student may have chosen those cases because he knew what he would find, and that it would support his preference. But looking at the same question in other cases (say, Scandinavia or Canada vs. African countries) might show very different results. The point is not that the student is wrong—in fact, he may be correct—but that the student’s preconceptions biased the research. We must ask research questions and test hypotheses fairly by ensuring the answer is not predetermined.

Even when researchers are careful not to bias their measures, we must consider the problem of measurement validity—that is, whether a given measure effectively captures or represents what is being researched. Indicators that are valid accurately reflect our concept. Informally, validity means measuring what we claim we are measuring. In some cases, this is straightforward, and our measures may be perfectly valid. To measure the “total number of political parties represented in a legislature,” we may simply find a record of every member of the legislature, note which party each member is from, then count the number of distinct parties to which legislators belong. On the other hand, consider the challenge of trying to measure overall health outcomes of a given country. Is life expectancy the right measure for this? Or infant mortality rates (the percent of infants that die before the age of two, for example)? Or rates of asthma, malaria, or HIV/AIDS? In truth, each of these is a valid measure of something specific, but none precisely measures “overall health.”

Measurement Validity: Nationalism in Latin America

Several guidelines can help promote measurements that are accurate, unbiased, and valid. We should strive for valid measurement to the greatest extent possible, but sometimes, when dealing with certain questions and sets of data, we will have to work with imperfect indicators of the concepts that interest us. We should explicitly state our reservations about our measures (and potential biases) when we present our work. This allows others to make their own judgments. In addition, we should be mindful of how our measured variables relate to our concepts and questions. In your own research, you should ask yourself what can actually be measured and whether the measurements actually correspond to the concept you are trying to study.

THINKING COMPARATIVELY: Qualities of Good Analysis and Argumentation

Chapter Summary

Chapter 2 Flashcards

Introduction to Theories, Hypotheses, and Evidence

· •Social scientists use theories, hypotheses, and evidence to build arguments about how the world operates. Theories are general explanations of how empirical phenomena operate across a range of cases. They are typically backed by some evidence. Hypotheses are potential explanations of cause and effect for specific cases. They are designed to be tested using evidence and are often derived from theories.

Hypothesis Testing

· •The central practice in comparative politics is testing hypotheses about causal questions using empirical evidence. This involves measuring variables and seeing how variables correlate across cases.

· •Variables that correlate with one another may have a causal relationship, but not necessarily.

· •There are several fallacies and logical traps to avoid when making causal arguments about correlated variables, to include reverse causation, omitted variables, intervening variables, and spurious correlation.

Critiques: Using Theories and Evidence

· •Political science can advance by developing critiques of existing theories and arguments. Critiques can be empirical, based on demonstrating cases that do not fit a theory, or can be more purely theoretical by using reason and logic to show problems with a theory.

The Challenges of Measurement: Biases, Errors, and Validity

· •Measurement is a leading challenge facing comparative political scientists. Comparativists aim to avoid measurement errors and biases and seek to ensure that measures are valid, or measure what they claim to measure.

Thinking Comparatively

· •Good practices in comparative politics include asking causal “Why” questions, developing unbiased hypothesis tests, and making arguments that are original yet informed by an understanding of existing theories and findings.