Fixing Unnecessary Missingness in Democracy Scores in {peacesciencer}
Source: vignettes/democracy.Rmd
democracy.Rmd
library(tidyverse)
#> ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
#> ✓ ggplot2 3.3.3 ✓ purrr 0.3.4
#> ✓ tibble 3.1.2 ✓ dplyr 1.0.6
#> ✓ tidyr 1.1.3 ✓ stringr 1.4.0
#> ✓ readr 1.4.0 ✓ forcats 0.5.1
#> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
#> x dplyr::filter() masks stats::filter()
#> x dplyr::lag() masks stats::lag()
library(peacesciencer)
library(kableExtra)
#>
#> Attaching package: 'kableExtra'
#> The following object is masked from 'package:dplyr':
#>
#> group_rows
library(stevemisc)
#>
#> Attaching package: 'stevemisc'
#> The following object is masked from 'package:dplyr':
#>
#> tbl_dfMissing data are everywhere in almost all statistical models in the political and social sciences. Canonical cases consider missingness in just the outcome variable, but it’s almost always the case that there are missing data across multiple (if not all) variables in a model. A regression model selects on complete cases and will thus punt rows from consideration in the model where any variable is missing. This is potentially a major problem for statistical inference, though the implication for inference is contingent on both type and scope.1 Generally, unless missing data are missing completely at random and/or missingness pervades less than 5% of the data, the potential for bias in the analysis looms large. Inter-state conflict researchers often do not consider how much this concerns their analysis because the number of observations in a given dyad-year or state-year analysis will still be in the high thousands. However, the remaining number of observations is less the point than the percentage missing and excluded from the analysis.
From my experience with dyad-year and state-year analyses, the biggest offender here will be the democracy scores for states. The reasons for this are multiple and mostly stem from the use of Polity data. The Polity project, for all I can tell, may not necessarily be the oldest cross-national data set on democracy for scholars who have been doing peace science analyses for the past three or four decades. However, it is certainly one of the oldest cross-national data sets on democracy and one of the oldest with great coverage into the 19th century. This would coincide with the Correlates of War state system data and the emergence of the militarized inter-state dispute (MID) data in the 1980s and 1990s. The democracy score that researchers would use was almost always the polity2 variable in the data, which added the democracy score and autocracy score together into a 21-item [-10:10] index. Shortcomings here, though, are multiple, leaving aside comments about whether we can adequately understand democracy as a battery of executive constraints and execute-level open competition. For one, the Polity project only considers states with a population of at least a million whereas state system membership (certainly CoW) has a population threshold of 500,000. This means a localized sample of post-World War II conflicts won’t have any Eastern Caribbean observations in it and the 1983 invasion of Grenada won’t appear in the analyses. Further, the Polity project is also replete with interregnum observations, which are often treated as missing data because the missing codes operate outside the 21-item index.
Consider the ccode_democracy data I provide in peacesciencer as illustrative of what’s at stake. You can (and should for transparency’s sake) see the underlying code that generates this data set on the Github repository for this package. Briefly, this is a data set that takes Version 10 of the Vdem data and the 2017 version of the Polity data (along with Xavier Marquez’ UDS extensions, more on that later) and standardizes both to Correlates of War state system membership data.
#library(tidyverse)
#library(peacesciencer)
ccode_democracy
#> # A tibble: 16,731 x 5
#> ccode year v2x_polyarchy polity2 xm_qudsest
#> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 2 1816 0.367 9 0.707
#> 2 2 1817 0.37 9 0.707
#> 3 2 1818 0.365 9 0.707
#> 4 2 1819 0.362 9 0.707
#> 5 2 1820 0.349 9 0.707
#> 6 2 1821 0.336 9 0.707
#> 7 2 1822 0.341 9 0.707
#> 8 2 1823 0.345 9 0.707
#> 9 2 1824 0.345 9 0.707
#> 10 2 1825 0.341 9 0.707
#> # … with 16,721 more rowsDoing this highlights just how much missingness there is in our democracy data. For example, let’s standardize these data to all observations between 1816 and 2010 and see how much of the data are missing.
ccode_democracy %>%
filter(between(year, 1816, 2010)) %>%
summarize(perc_missing = sum(is.na(polity2))/length(polity2))
#> # A tibble: 1 x 1
#> perc_missing
#> <dbl>
#> 1 0.0850Over 8% of the Polity data are missing in the CoW state system. Here would be the observations affected, starting with the states that don’t appear at all in the Polity data.
# library(kableExtra)
ccode_democracy %>%
filter(between(year, 1816, 2010)) %>%
group_by(ccode) %>%
mutate(nobs = n()) %>%
filter(is.na(polity2)) %>%
group_by(ccode) %>%
summarize(n = n(),
nobs = unique(nobs),
years = str_c(year, collapse = ", ")) -> missing_obs
missing_obs %>%
filter(n == nobs) %>%
mutate(country = countrycode::countrycode(ccode, "cown", "country.name")) %>%
select(ccode, country, n, nobs) %>%
kbl(., caption = "CoW States that Never Appear in the Polity Data") %>%
kable_styling(position = "center", full_width = F, bootstrap_options = "striped")| ccode | country | n | nobs |
|---|---|---|---|
| 31 | Bahamas | 38 | 38 |
| 53 | Barbados | 45 | 45 |
| 54 | Dominica | 33 | 33 |
| 55 | Grenada | 37 | 37 |
| 56 | St. Lucia | 32 | 32 |
| 57 | St. Vincent & Grenadines | 32 | 32 |
| 58 | Antigua & Barbuda | 30 | 30 |
| 60 | St. Kitts & Nevis | 28 | 28 |
| 80 | Belize | 30 | 30 |
| 221 | Monaco | 18 | 18 |
| 223 | Liechtenstein | 21 | 21 |
| 232 | Andorra | 18 | 18 |
| 240 | Hanover | 30 | 30 |
| 273 | Hesse Electoral | 51 | 51 |
| 275 | Hesse Grand Ducal | 52 | 52 |
| 280 | Mecklenburg Schwerin | 25 | 25 |
| 331 | San Marino | 19 | 19 |
| 338 | Malta | 47 | 47 |
| 395 | Iceland | 67 | 67 |
| 403 | São Tomé & Príncipe | 36 | 36 |
| 511 | Zanzibar | 2 | 2 |
| 591 | Seychelles | 35 | 35 |
| 781 | Maldives | 46 | 46 |
| 835 | Brunei | 27 | 27 |
| 935 | Vanuatu | 30 | 30 |
| 946 | Kiribati | 12 | 12 |
| 947 | Tuvalu | 11 | 11 |
| 955 | Tonga | 12 | 12 |
| 970 | Nauru | 12 | 12 |
| 983 | Marshall Islands | 20 | 20 |
| 986 | Palau | 17 | 17 |
| 987 | Micronesia (Federated States of) | 20 | 20 |
| 990 | Samoa | 35 | 35 |
These seem like uncontroversial omissions. Perhaps no one will miss Monaco or Tonga in a dyad-year model on conflict onset. However, there are several problematic omissions here. The exclusion of Hanover, the two Hesses (sic), and Mecklenburg means there will be some important conflict-dyad omissions for the various wars of German unification. Belize has (I would argue) a prominent, conspicuous, and interesting spatial rivalry with Guatemala. Guatemala has at points, has refused to acknowledge Belize’ right to exist, but Belize is no stranger to initiating low-level disputes on the border as well. The Eastern Caribbean omissions mean every observation in the 1983 invasion of Grenada will be dropped. Trinidad and Tobago has a territorial conflict with Venezuela regarding its oil-rich maritime boundary. It even had a violent coup attempt in 1990, which appears in the UCDP data.
Here are the observations for which there is only situational missingness in the Polity data.
missing_obs %>%
filter(n < nobs) %>%
mutate(country = countrycode::countrycode(ccode, "cown", "country.name")) %>%
select(ccode, country, everything()) %>%
kbl(., caption = "Situational Missigness in the Polity Data") %>%
kable_styling(position = "center", full_width = F, bootstrap_options = "striped")| ccode | country | n | nobs | years |
|---|---|---|---|---|
| 41 | Haiti | 1 | 134 | 1915 |
| 42 | Dominican Republic | 4 | 110 | 1914, 1915, 1916, 1924 |
| 70 | Mexico | 3 | 180 | 1846, 1847, 1863 |
| 91 | Honduras | 4 | 112 | 1907, 1912, 1919, 1924 |
| 93 | Nicaragua | 2 | 111 | 1926, 1927 |
| 135 | Peru | 2 | 172 | 1881, 1882 |
| 140 | Brazil | 2 | 189 | 1822, 1823 |
| 210 | Netherlands | 1 | 191 | 1940 |
| 211 | Belgium | 3 | 177 | 1914, 1939, 1940 |
| 212 | Luxembourg | 2 | 88 | 1940, 1944 |
| 225 | Switzerland | 32 | 195 | 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847 |
| 235 | Portugal | 4 | 195 | 1816, 1817, 1818, 1819 |
| 255 | Germany | 2 | 151 | 1945, 1990 |
| 265 | German Democratic Republic | 2 | 37 | 1989, 1990 |
| 267 | Baden | 3 | 56 | 1816, 1817, 1818 |
| 269 | Saxony | 1 | 52 | 1848 |
| 310 | Hungary | 2 | 93 | 1944, 1956 |
| 329 | Two Sicilies | 1 | 46 | 1861 |
| 345 | Yugoslavia | 2 | 131 | 1915, 1916 |
| 346 | Bosnia & Herzegovina | 16 | 19 | 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 |
| 350 | Greece | 4 | 181 | 1916, 1917, 1918, 1919 |
| 355 | Bulgaria | 1 | 103 | 1913 |
| 360 | Romania | 1 | 133 | 1916 |
| 366 | Estonia | 1 | 43 | 1918 |
| 367 | Latvia | 2 | 43 | 1918, 1919 |
| 385 | Norway | 1 | 102 | 1940 |
| 390 | Denmark | 1 | 191 | 1940 |
| 452 | Ghana | 3 | 54 | 1957, 1958, 1959 |
| 500 | Uganda | 1 | 49 | 1979 |
| 530 | Ethiopia | 2 | 109 | 1936, 1941 |
| 552 | Zimbabwe | 5 | 46 | 1965, 1966, 1967, 1968, 1969 |
| 616 | Tunisia | 60 | 112 | 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1956, 1957, 1958 |
| 640 | Turkey | 4 | 195 | 1918, 1919, 1920, 1921 |
| 645 | Iraq | 7 | 79 | 2003, 2004, 2005, 2006, 2007, 2008, 2009 |
| 651 | Egypt | 28 | 102 | 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882 |
| 652 | Syria | 1 | 63 | 1958 |
| 660 | Lebanon | 15 | 65 | 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004 |
| 690 | Kuwait | 3 | 50 | 1961, 1962, 1990 |
| 700 | Afghanistan | 20 | 92 | 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 |
| 710 | China | 11 | 151 | 1860, 1861, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945 |
| 712 | Mongolia | 3 | 90 | 1921, 1922, 1923 |
| 740 | Japan | 1 | 145 | 1945 |
| 750 | India | 3 | 64 | 1947, 1948, 1949 |
| 771 | Bangladesh | 1 | 40 | 1971 |
| 800 | Thailand | 1 | 124 | 1941 |
| 811 | Cambodia | 9 | 58 | 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987 |
| 812 | Laos | 1 | 58 | 1953 |
| 817 | Republic of Vietnam | 9 | 22 | 1954, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972 |
| 940 | Solomon Islands | 1 | 33 | 2003 |
Some of these are really important omissions! Bosnia & Herzegovina only has democracy scores for its first three years in the system. Hungary has missing data incidentally coinciding with the year of the Soviet invasion in 1956. Tunisia’s entire first spell in the CoW state system is missing, as is the first few years of its reappearance in the 1950s. Over 40% of the short-lived Republic of Vietnam’s democracy data are missing. Missingness in Europe coinciding with the end of World War I is going to drop a lot of important conflicts of the time (see: Latvia in 1918 and 1919). The Dominican Republic’s democracy scores go missing, just in time to drop the U.S. invasion of it from any analysis. Incidentally, a lot of missing observations here are going to collide with major conflicts. Perhaps a researcher can sidestep some of these problems by lagging the democracy variable a year, but that won’t fix all of them. After all, some countries are effectively born in/from conflict (e.g. Bosnia, India, Pakistan, the two Koreas, and more). If a researcher is not careful here, there will be more missingness in the data than meets the eye.
There’s no reason to accept these missing observations as a cost of doing business with Polity. That’s why peacesciencer brings in multiple data sets that have better coverage than Polity. The most unique of these comes from Xavier Marquez’ extension of the Unified Democracy Scores (UDS) data. The UDS data were designed to be a sort of “compromise” between competing measures of democracy, but the underlying statistical model—the graded response model—serves as a missing data fix too. If, say, the Polity project does not have observations of a particular state in a given year, the model leans on other inputs to derive a democracy estimate. The standard error of the estimate increases with missing inputs, but a lot of measures of democracy correlate highly regardless. The ensuing estimate, drawn from a standard normal distribution, serves as a solid estimate of the level democracy in a given year. It’s included in ccode_democracy as xm_qudest.
Here are a few cases of Xavier Marquez’ filling in missing data while also passing the look test. First, here’s Turkey.2
# library(stevemisc)
ccode_democracy %>%
# select(-v2x_polyarchy) %>%
filter(ccode == 640) %>%
gather(var, val, -ccode, -year) %>%
ggplot(.,aes(year, val)) +
theme_steve_web() +
facet_wrap(~var, nrow= 3, scales = "free") + geom_line() +
labs(y = "", x = "",
title = "Various Democracy Scores for Turkey, 1816-2010")
#> Warning: Removed 3 row(s) containing missing values (geom_path).
Here’s Afghanistan, a mostly non-democratic country throughout its history.
# library(stevemisc)
ccode_democracy %>%
# select(-v2x_polyarchy) %>%
filter(ccode == 700) %>%
gather(var, val, -ccode, -year) %>%
ggplot(.,aes(year, val)) +
theme_steve_web() +
facet_wrap(~var, nrow= 3, scales = "free") + geom_line() +
labs(y = "", x = "",
title = "Various Democracy Scores for Afghanistan, 1919-2010")
#> Warning: Removed 3 row(s) containing missing values (geom_path).
Here’s Portugal, a country that democratized at various points in its history.
# library(stevemisc)
ccode_democracy %>%
# select(-v2x_polyarchy) %>%
filter(ccode == 235) %>%
gather(var, val, -ccode, -year) %>%
ggplot(.,aes(year, val)) +
theme_steve_web() +
facet_wrap(~var, nrow= 3, scales = "free") + geom_line() +
labs(y = "", x = "",
title = "Various Democracy Scores for Portugal, 1816-2010")
#> Warning: Removed 7 row(s) containing missing values (geom_path).
Here’s a correlation matrix for the three democracy estimates across time and space. Notice that Marquez’ UDS extensions correlate with polity2 at .94 and with the Vdem data at .91. That’s better than the correlation between polity2 and the Vdem data.
ccode_democracy %>%
select(v2x_polyarchy:ncol(.)) %>%
cor(use="complete.obs")
#> v2x_polyarchy polity2 xm_qudsest
#> v2x_polyarchy 1.0000000 0.8599225 0.9158889
#> polity2 0.8599225 1.0000000 0.9413074
#> xm_qudsest 0.9158889 0.9413074 1.0000000More importantly, look at the data coverage of the UDS extensions vis-a-vis alternatives.
ccode_democracy %>%
filter(between(year, 1816, 2010)) %>%
summary
#> ccode year v2x_polyarchy polity2
#> Min. : 2.0 Min. :1816 Min. :0.0080 Min. :-10.0000
#> 1st Qu.:200.0 1st Qu.:1917 1st Qu.:0.1430 1st Qu.: -7.0000
#> Median :359.0 Median :1967 Median :0.2350 Median : -3.0000
#> Mean :393.6 Mean :1949 Mean :0.3436 Mean : -0.4115
#> 3rd Qu.:616.0 3rd Qu.:1991 3rd Qu.:0.5450 3rd Qu.: 7.0000
#> Max. :990.0 Max. :2010 Max. :0.9230 Max. : 10.0000
#> NA's :1008 NA's :1257
#> xm_qudsest
#> Min. :-1.76974
#> 1st Qu.:-0.43923
#> Median : 0.08278
#> Mean : 0.25368
#> 3rd Qu.: 0.97599
#> Max. : 2.98554
#> NA's :4
ccode_democracy %>%
filter(between(year, 1816, 2010)) %>%
filter(is.na(xm_qudsest)) -> missing_xm
missing_xm
#> # A tibble: 4 x 5
#> ccode year v2x_polyarchy polity2 xm_qudsest
#> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 300 1918 NA 0 NA
#> 2 329 1861 NA NA NA
#> 3 616 1881 0.026 NA NA
#> 4 651 1882 0.074 NA NAMarquez’ UDS extensions are missing for just four cases: Austria-Hungary in 1918, Two Sicilies in 1861, Tunisia in 1881, and Egypt in 1882. That’s better coverage than the alternatives. A user can accept these as missing observations since they involve just four cases. Alternatively, a user can knock off three of those with some kind of parlor trick like this.
M1 <- lm(xm_qudsest ~ polity2, data=ccode_democracy)
M2 <- lm(xm_qudsest ~ v2x_polyarchy, data=ccode_democracy)
pol_intercept <- broom::tidy(M1)[1, 2] %>% pull()
pol_coef <- broom::tidy(M1)[2, 2] %>% pull()
vdem_intercept <- broom::tidy(M2)[1, 2] %>% pull()
vdem_coef <- broom::tidy(M2)[2, 2] %>% pull()
ccode_democracy %>%
mutate(impute_pol = pol_intercept + polity2*pol_coef,
impute_vdem = vdem_intercept + v2x_polyarchy*vdem_coef) %>%
mutate(imputed = case_when(
is.na(xm_qudsest) & !is.na(polity2) ~ impute_pol,
is.na(xm_qudsest) & !is.na(v2x_polyarchy) ~ impute_vdem,
TRUE ~ xm_qudsest
)) %>% select(-xm_qudsest) %>%
left_join(missing_xm, .)
#> Joining, by = c("ccode", "year", "v2x_polyarchy", "polity2")
#> # A tibble: 4 x 8
#> ccode year v2x_polyarchy polity2 xm_qudsest impute_pol impute_vdem imputed
#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 300 1918 NA 0 NA 0.273 NA 0.273
#> 2 329 1861 NA NA NA NA NA NA
#> 3 616 1881 0.026 NA NA NA -0.728 -0.728
#> 4 651 1882 0.074 NA NA NA -0.582 -0.582Whatever you choose to do here, a conflict researcher should take missing data seriously in their democracy estimates. peacesciencer does, which is why the add_democracy() function in this package does more than merge in Polity data.
“Type” refers to the categories of missingness, whether missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). MCAR refers to cases where, say, a respondent withheld reporting their income in a survey because they flipped a coin and it came up heads. MAR refers to missingness that depends on observations that are recorded elsewhere (e.g. a respondent withholds reporting their income because they generally don’t trust people, having previously communicated that in another survey prompt). MNAR refers to circumstances in which the missing data depend on the missing values themselves (e.g. high-income earners typically do not like reporting how much they earn). “Scope” refers to just how much missingness is present in the data, given the full set of observations.↩︎
Recall the three variables under consideration here work on different scales though all communicate the same underlying continuum of most autocratic to most democratic. The
polity2variable is on a [-10,10] scale with one-point increments. The Vdem’ polyarchy data are on a continuum from 0 to 1. The UDS estimates are standardized to a mean of 0 and a standard deviation of 1 where democracy is understood as latent phenomenon.↩︎