Choice or circumstance? Adjusting measures of foreign policy similarity for chance agreement

The similarity of states' foreign policy positions is a standard variable in the dyadic analysis of international relations. Recent studies routinely rely on Signorino and Ritter's (1999, Tau-b or not tau-b: Measuring the similarity of foreign policy positions. International Studies Quarterly 43:115-44) S to assess the similarity of foreign policy ties. However, S neglects two fundamental characteristics of the international state system: foreign policy ties are relatively rare and individual states differ in their innate propensity to form such ties. I propose two chance-corrected agreement indices, Scott's (1955, Reliability of content analysis: The case of nominal scale coding. The Public Opinion Quarterly 19:321-5) pi and Cohen's (1960, A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37-46) kappa, as viable alternatives. Both indices adjust the dyadic similarity score for a large number of common absent ties. Cohen's kappa also takes into account differences in individual dyad members' total number of ties. The resulting similarity scores have stronger face validity than S. A comparison of their empirical distributions and a replication of Gartzke's (2007, The capitalist peace. American Journal of Political Science 51: 166-91) study of the 'Capitalist Peace' indicate that the different types of measures are not substitutable.