Some Comments on (Weighting) Alliance Measures of Foreign Policy Similarity

Not all alliances look like the Franco-Russian Alliance. Most Russian alliances don't look like it either.

I’m preparing for a yearly update of {peacesciencer}. Peace science data tend to update rather slowly, so this update is more about efficiency and reducing bloat. However, there will be one feature I’m adding to the package that I started last year around this time. {fpsim} is a relatively new R package of mine with some toy functions for creating measures of dyadic foreign policy similarity. This functionality is for the most part already available in {peacesciencer}, but the data will be more current. Alliance coverage will be available through 2018 and UN voting coverage will be available through 2022.

With it, I wanted to riff on something to which I’m deeply indebted to Frank Häge. His 2011 article in Political Analysis is currently the data included in {peacesciencer} and the article is the backbone of how I advise people to think about measures of dyadic foreign policy similarity. I wanted to add some comments to what his article communicates that mostly reiterate what I think his main takeaways are. In this case, though, they concern how to think about alliances, and weighting alliance data, for calculating measures of dyadic foreign policy similarity. What I say here is largely supported by his 2011 article, though might be a bit more opinionated (or more clumsily stated). Briefly: we use alliance data for measures of dyadic foreign policy similarity because we have data on them, not because they are a great source of information in and of themselves for the task at hand. I don’t think they’re clear measures of dyadic foreign policy similarity because states sign alliances and pledge things for any number of reasons. They are a somewhat anachronistic measure of dyadic foreign policy similarity that doesn’t communicate the quantitative information we think they do. Weighting them by system capabilities just introduces one anachronism to another for more current observations. You can still do this. Just be mindful of what you are doing and for what time frame you’re doing it.

Here are the R packages I’ll be using in this post, even if I’ll try not to litter this post with too much R code.

library(tidyverse)     # for most things
library(kableExtra)    # for tables
library(fpsim)         # for measuring dyadic foreign policy similarity
library(peacesciencer) # for data on capabilities
library(stevethemes)   # for my own vanity/themes

theme_set(theme_steve())

Here’s a table of contents.

1. Some Caveats About Alliances as Source of Foreign Policy Similarity
2. Some Caveats About Treating Alliance Pledges as Ordinal
3. Some Caveats About Weighting Alliance Ties by Capabilities
4. Conclusion

Alrightie then…

Some Caveats About Alliances as Source of Foreign Policy Similarity

Generating measures of dyadic foreign policy similarity impels us to ask ourselves what source information we should use to generate such a measure across time and space. If we could play god, return to 1816, and create such a measure in real time to guide us for the next 200+ years, perhaps we could be more flexible in the construction of such a measure. Perhaps an elite- or mass-level survey could approximate “affinity” (if not “similarity”) with some kind of average thermometer rating, though the logistics of such an instrument would be so vast it would be an impossibility to create. There’s perhaps a content/sentiment analysis of newspaper articles discussing various countries in various newspapers of note (e.g. Times of London in the UK, New York Times in the United States) that could work.¹ However, there are a lot of assumptions about what makes it to print and who is the intended audience that would need to be addressed in such a measure, which is still more about “affinity” than “similarity.” If we wanted “similarity”, we’re stuck in the 19th century where pacific cross-border activity and inter-governmental organizations were only beginning to take off. Technology was both opportunity and limitation. Power politics was the norm and alliances were featured components of a (European) state’s foreign policy. Why not use alliances, then? Data on them would take time to marshal, but it’s not impossible. In fact, we already have that data given our interest in so-called realpolitik policies and the balance of power.

Alliances are peculiar things in the study of international relations. The typical person might offer NATO or the Warsaw Pact if pressed to give an example of an alliance, though those would be far from typical in the world of alliances. For one, the Warsaw Pact had eight members, NATO currently has 32 members, but 86% of alliances in v. 5.1 of the ATOP data are bilateral. Pressed to define what alliances do, a person might again reiterate those examples to emphasize the primarily defensive nature of alliances or their so-called “balancing” effects. The Franco-Russian Alliance might be the ideal type of this kind of logic initially conjured. Born from a changing European situation in which a unified Germany was upending the post-Napoleonic status quo, it was an explicit anti-German alliance in which both sides pledged defense if Germany attacked (or supported the attack) of one of the two members. However, that is again not the full scope of what alliances do. Per ATOP (v. 5.1), only about 35% of all alliances include a defense pledge. 16% of alliances include a pledge of neutrality (in the even of a potential conflict with a third-party not included as a signatory in the agreement). Just over 10% are outright offensive pacts in which signatories pledge conflict against another state or set of states.² More interestingly, 56% of all alliances pledge non-aggression and 49% pledge to consult in the event of some crisis. In other words, two or more states often sign what we call alliances precisely because they do not like each other very much. States can sign alliances for any number of reasons, like wresting or settling disputed territory from a state after a conflict. Basically, our initial impressions of what alliances “look like” betray what they ultimately do. Earnestly paraphrasing p. 238 here, alliances are written agreements that states sign to coordinate military policies on issues that could lead to armed conflict. That could even be conflict with the states signing the alliance.

For an illustration of the myriad reasons states may enter into an alliance, consider the conspicuous case of Russia. Russia is the no. 1 alliance-signer of all time. As of version 5.1 of the data, Russia has been a member of 157 alliances all time sine 1815. That’s almost as much as no. 2 (France, 91) and no. 3 (United Kingdom, 69) combined. However, a scan of the data highlights the important heterogeneity of alliances and complicates what we think is a nice and clear signal we hope to extract from them. Consider the following table, which lists all 12 alliances that were active for Russia in the second half of the 19th century (1850-1899).

A few interesting patterns emerge here. For one, these alliances were overwhelmingly focused on Russia’s immediate neighbors following the third and final partition of Poland. Of these 12, five included Austria-Hungary and an additional three were conducted with just Germany (or its predecessor state, Prussia). Russian relations with both states weren’t always hostile at this point, but were complicated to say the least. The 1833 Treaty of Münchengrätz was an important signal of joint Austrian-Russian cooperation on the issue of the Ottoman Empire and its ongoing decline. That would be the simple story of that alliance, and one that maps nicely to the information we think alliances communicate to those of us interested in dyadic foreign policy similarity. The Franco-Russian alliance (ATOP ID: 1385) would communicate the same thing. However, this treaty importantly had to at least partly repair relations between the two that soured as a result of the Greek war of independence and reiterate the major coordination problem (also involving Prussia/Germany) that came from conspiring Poland out of existence. The dates don’t exactly line up as I remember reading about it, but ATOP ID no. 1315 looks like The League of Three Emperors. This three-party pact signals cooperation on the issue of the former/current Poland, but it also belies the somewhat massive misgivings all three states had with each other near this point. Austria-Hungary and Prussia/Germany had just fought a war in 1866 and its leadership was more than worried about the implications of an aggressive German ethnonationalism for its multinational empire of which ethnic Germans were just one part. It was an alliance with only the pretense of foreign policy similarity. The fallout of this league’s final rupture was renewed combat for Russia against the Ottoman Empire, which enraged the Austrians enough to make common cause with Germany immediately thereafter (no matter the renewal of the league in 1881). That Austrian-German dual alliance from 1879 expanded to the so-called Triple Alliance in 1882 when Italy joined for a multilateral defense pact. Yes, that Italy, famously good friends with Austria-Hungary at this moment in time. In these cases, and the fallout cases, Russian alliances communicate dyadic foreign policy similarity as much as they’re signaling diplomatic misgivings and dyadic foreign policy dissimilarity.

Contrast Russian alliance behavior in the second half of the 19th century with all Russian alliances that were active as of through 2000. Note that just two of these 61 cases are no longer active. ATOP ID no. 4300 was a Russian-Turkmenistani alliance that ended on April 23, 2002, ostensibly replaced by ATOP ID no. 5000. ATOP ID no. 4385 ended on July 16, 2001 and was seemingly replaced by ATOP ID no. 4980. I’ve condensed a few of the more massive multilateral ones for sake of legibility.

I will defer to the Russia expert or alliance expert in how they read this information, if they would like to interject, but here would be my reading of this information as an IR scholar and someone thinking about what this information means for measures of dyadic foreign policy similarity. The historical Russia may have used alliances to deal with complicated questions involving rivals and would-be enemies, or to signal some kind of focused foreign policy hive mind in the case of the Franco-Russian alliance. The modern Russia seems to increasingly use what we code as “alliance” as a means of bilateral relations management. The information one might extract from the 1997 alliance with Belarus seems pretty straightforward, but should we understand the spate of non-aggression pledges Russia and China offer to each other as equivalent to ones made to Italy in 1994? Should the Treaty on Basic Relations between the Russian Federation and Belize³ be understood as an equivalent non-aggression pledge to the one Russia apparently made to Uzbekistan in 2004? Or Moldova in 2001? Simply, I do not think a non-aggression pledge from Russia today means the same thing it did 150 years ago. Alliances may be more prevalent today than they were 150 years ago, but the information we may hope to extract from them has changed considerably. There is a similar comment to be made about the absence of a formal defense pledge between Russia and China in these data, as there is between Russia and Armenia from Aug. 29, 1997 through 2018. A similar theme would emerge concerning the absence of an alliance of any form between Israel and the United States as of 1991, even though the increased similarity between both is quite evident over the past 30 years.

To return to the motivating question of why we should use alliances as information about dyadic foreign policy similarity, the answer seems to collapse to a basic and dissatisfying answer of “because we have data on them.” The simple story we like to tell about alliances betrays the myriad and heterogeneous reasons why states sign them. The standard prism by which we think of what alliances set out to accomplish have never been faithful to what they actually do, and I can raise an identical complaint about the simplistic stories about arms races. Alliances have never really said what we want them to say about dyadic foreign policy similarity, and I do not think they say the same thing across time and space. Alliances are more common today, but their salience may arguably be less than what it was before the world wars. If we ignore how noisy of a measure they were, I still think they are a somewhat anachronistic measure of dyadic foreign policy similarity. If you need to do something with observations before 1946, that information is available and I won’t impugn its use. None of this should be considered a complaint about the alliance data, either in its CoW or ATOP form. The data sets faithfully communicate what alliances are, no matter the differences between them. I just think they’re less than adequate, but certainly available, for a measurement of dyadic foreign policy similarity. Just be mindful what you’re doing with it.

Some Caveats About Treating Alliance Pledges as Ordinal

My misgivings about treating alliance pledges as communicating some kind of ordinal information are much spicier than my misgivings about using them as source information for measures of dyadic foreign policy similarity. I brought this up previously in a 2021 blog post related to this topic.

What follows is a gigantic caveat since I’ve always hated this train of thought when I was first reading about it in graduate school, even if it never occurred to me how to operationalize Tau-b with the data. Calculating Tau-b like this builds in an assumption that you can construct an ordinal measure of an alliance commitment. The basic train of thought is defense > (neutrality AND/OR non-aggression) > entente > no alliance at all. In their article, Signorino and Ritter (1999) go to great lengths to acknowledge why this is problematic but you should do it anyway. Häge offers the more reasonable take that you should not at all do this and instead use one of his binary measures since you should not think of alliance pledges as ordinal in any way in this typology. {peacesciencer} basically follows Häge’s recommendation and encourages sensible defaults, even if less sensible defaults are available in the data Häge calculated.

I would welcome the reader to dig into the data and read about the conceptualization and operationalization of alliance pledges to better understand this point. I am not terribly inclined to challenge an argument that a mutual defense pledge is “more” than a pledge of non-aggression or consultation, but is that pledge conditional on a particular third-party? Does it come with a pledge of active military support? That would be useful information to consider in evaluating such a pledge. I would, however, challenge a claim that a neutrality pledge is equivalent to a non-aggression pledge. They are categorically different commitments allies make to each other. The first identifies a third party from which one of the signatories pledges neutrality in the event of a conflict involving the signatory and the third party. The other routinely captures contentious questions involving the allies themselves or, potentially, generic pledges of a bilateral relationship with nothing else interesting happening (in the case of the Russia-Belize non-aggression pledge from 2018). I’m also not sold on an argument that a neutrality or non-aggression pledge is “more” than a pledge of consultation (or “entente” in the CoW data). It might vary from case to case, but a more aggressive critic could just as well read that a pledge of consultation in the event of a crisis may suggest a stronger commitment to diplomacy than forgoing the use of force as an option. It’s at least promising diplomacy first. A critical reader well-versed in the alliance literature is free to balk at that interpretation, and I’m not offering it myself, but the counterclaim under contention here is that non-aggression is “more” than consultation. They could just as well be equivalent. Further, the illustrative case of Russia above suggests that the presence of an alliance, even (especially?) non-aggression pledges, often signal more foreign policy dissimilarity than the absence of an alliance.

You really should not interpret pledges as communicating quantitative information or that the alliance data permit the construction of an ordered-categorical scale based on pledges included in an alliance. A simple binary indicator communicating whether there is some kind of agreement will have to do, given the circumstances.

Some Caveats About Weighting Alliance Ties by Capabilities

One particular measure of dyadic foreign policy similarity—Signorino and Ritter’s (1999) S—offered a means to capture how important some alliance ties were. Whereas their measure is a measure of “distance” while Kendall’s Tau-b is more of a measure of the net probability of concordance for an ordered set of observations, their formula permits elevating some ties as more important than others. Using the (questionably) valued alliance data, you could potentially weight these alliances by the target states’ capabilities. The obvious choice here would be the Composite Index of National Capabilities (CINC) measure from the Correlates of War project.

Take Table 6 of Signorino and Ritter (1999), for example. This table is included in {fpsim} as gmyrus14. In the following data frame, gmy communicates the alliance portfolio for Germany in 1914 for all European states at this time and rus communicates the alliance portfolio for Russia for all European states at this time. The syscap communicates the CINC score Signorino and Ritter report for the state in the state column, proportional to all of Europe. Further, each state is maximally committed to defending itself.

gmyrus14
#> # A tibble: 20 × 4
#>    state syscap   gmy   rus
#>    <chr>  <dbl> <dbl> <dbl>
#>  1 GMY     0.25     3     0
#>  2 RUS     0.21     0     3
#>  3 UKG     0.2      0     1
#>  4 FRN     0.08     0     3
#>  5 AUH     0.08     3     0
#>  6 ITA     0.05     3     1
#>  7 BEL     0.03     0     0
#>  8 SPN     0.03     0     0
#>  9 TUR     0.02     0     0
#> 10 NTH     0.01     0     0
#> 11 SWD     0.01     0     0
#> 12 RUM     0.01     3     0
#> 13 POR     0.01     0     0
#> 14 SWZ     0.01     0     0
#> 15 GRC     0        0     0
#> 16 DEN     0        0     0
#> 17 YUG     0        0     0
#> 18 BUL     0        0     0
#> 19 NOR     0        0     0
#> 20 ALB     0        0     0

We can reproduce the second part of their Table 6 below with functions in {fpsim} and arrive at the basic takeaway they report. Kendall’s Tau-b suggests almost no association and their unweighted S score suggests a moderate level of similarity. However, much of that is arguably a function of their agreement for some of the most inconsequential system members (for whom neither had an alliance). Weighted by the target state’s CINC score (to elevate the “important” system members in Europe), a moderate degree of dissimilarity emerges. There would be more face validity in such a measure, given the world war that followed between the two.

# Tau-b...
taub(gmyrus14$gmy, gmyrus14$rus)
#> [1] 0.03031695
# Unweighted *S* measure, absolute distances (actually the default in srs())
srs(gmyrus14$gmy, gmyrus14$rus, distances = 'absolute')
#> [1] 0.4
# Weighted *S* measure, absolute distances, by CINC
srs(gmyrus14$gmy, gmyrus14$rus, distances = 'absolute', weights = gmyrus14$syscap)
#> [1] -0.46

On paper, this is reasonable. However, its application is rather questionable upon further review. For one, consider what comprises a CINC score. A CINC score is (unarguably?) our most common quantitative measure of power in the international system. Formally, you’d construct it as follows.

..where:

• \(tpr_{it}\) = total population ratio of country i in year t
• \(upr_{it}\) = total urban population ratio of country i in year t
• \(ispr_{it}\) = iron and steel production ratio of country i in year t
• \(ecr_{it}\) = primary energy consumption ratio of country i in year t
• \(mer_{it}\) = military expenditure ratio of country i in year t
• \(mpr_{it}\) = military personnel ratio of country i in year t

If, say, a hypothetical country has 100% of the world’s population, 100% of the world’s urban population, produces 100% of the world’s iron and steel, consumes 100% of the world’s energy, and is 100% of the world’s expenditure and personnel, their CINC score would be 1. In these data, the absolute most powerful a state has ever been was the United States in 1945. Its CINC score of .384 suggests that the United States at the tail end of World War 2 was about 38.4% of the entire international system’s latent and actual capacity for war. Tuvalu in 2012 has the minimum CINC score of about 0.000000244. Both would make sense, prima facie.

I like this measure a fair bit for what I think it ultimately is. This is a 19th century measure of “power” and to what extent a state in the 19th century could marshal enough resources to the battlefield given the then-timely questions of what industrialization and urbanization were doing for a country’s growth. “Post-industrialization”, I’m not sure this measure conveys the same kind of information it did in the 19th century. Consider the following plot, which identifies the top 10 states by CINC scores for 1816 (start of the data), 1850, 1900, 1950, 2000, and 2016 (end of the data).

I welcome alternative interpretations here, but I’m gathering more face validity in a much smaller, much more European system in the 19th century driven by questions of industrialization and accelerated urbanization. I see less face validity over time, especially as the measure seems to elevate the countries with the much larger populations. There is a reasonable question to ask about the “Rise of China” and whether we believe that has since happened, quantitatively. I’m not sure we should have reason to believe it happened by 2000. I also don’t believe we have reason to think that Brazil is more powerful than the United Kingdom, even though we will have to concede that Brazil has around three times the population of the United Kingdom. Since CINC is unweighted, those two population measures are a third of the measure that follows. I already think alliances are an anachronistic measure for a measure of dyadic foreign policy similarity in the present. Weighting the measure by a CINC score introduces another anachronism to it, like a hat on a hat.

There are other peculiarities too. The data on capabilities has always been historically skewed to the right. Very few states have proportionally that much weight. As the state system has expanded in size (i.e. as empires ended), the relative weight at the top necessarily decreases. For example, the top 3 states in capabilities in 1816 (the United Kingdom, Russia, and France) combined for 61.8% of capabilities in a system of just 23 states. In 2016, the top three states (China, the United States, and India) combined for 45% of capabilities in a system of 195 states. Even then, the concentration at the top is rather impressive, and worrisome for a measure like this. Except for a stretch from 1990 to 2008, more than 50% of the system’s capabilities are concentrated in the top five in any given year.⁴ Under those circumstances, the dyadic foreign policy similarity of Israel and Jordan in 1950 is much more about how they agree on India than it is about how they agree on Egypt.

The right skew implies, and the data certainly illustrate, that most states—especially new states, themselves typically small states—possess almost no capabilities relative to the international system. In 1816, 11 of 23 sovereign states had less than 1% of capabilities proportional to the international system. That’s about 48% of the system in that year. In 2016, 176 of 195 states had less than 1% of capabilities. That’s over 90% of the system. I’m not sure I’m comfortable with discarding that much information to identify the more “important” cases.

If you read Signorino and Ritter (1999), you’ll be sympathetic to what weighting is trying to do. It’s trying to identify the “important” foreign policy ties and downweight irrelevant foreign policy ties/concordance/agreement. However, I echo Häge (2011) that weighting measures of dyadic foreign policy similarity is a second-best solution. It’s second-best to other metrics that better model chance-corrected agreement that he offers, like Cohen’s (1960, 1968) kappa or Scott’s (1955) pi. Weighting by capabilities just discards too much information and gives too much weight to great powers and/or states that are conspicuously high on capabilities (e.g. India). It relies on what I feel is an anachronistic measure of power to weight what I think is also an anachronistic source of information for measuring dyadic foreign policy similarity.

Conclusion

Here are the basic takeaways for people interested in measuring dyadic foreign policy similarity across time and space. For one, alliances are not a great source of information for dyadic foreign policy similarity. They are an available source of information for dyadic foreign policy similarity, but not a great source of information. I think this importantly follows because our standard prism for thinking about what alliances do and why states sign them have never squared with the reality of what alliances actually are and what they do. States often sign alliances because they are dissimilar in their foreign policy. Further, resist any and all urges to believe alliances communicate quantitative information, or even ordinal information. The most you can do, given the circumstances, is identify whether states agree that some target is important enough to merit an agreement of some form. That’s about it. Further, alliances are already an anachronistic source of information for dyadic foreign policy, and a questionable at that. Weighting by capabilities introduces yet another anachronism that does not scale well to the present. It’s arguably better than not weighting at all, but it is not better than measures like Cohen’s (1960, 1968) kappa or Scott’s (1955) pi for a chance-corrected measure of agreement. Use one of those, and, of those, use Cohen’s kappa. Scott’s (1955) pi builds in a strong assumption that the baseline propensity of forming a tie in a dyad is the same for both members. Either one is better than S or Tau-b.