Coefficients of Agreement

Description

A coefficient of agreement, as proposed by van der Eijk (2001) , quantifies how strongly responses on an ordered rating scale are concentrated versus dispersed or polarized. Unlike other measures such as the standard deviation, which are sensitive to skewness and the location of the distribution, this coefficient is specifically designed to reflect the peakedness or central agreement of a distribution. The measure ranges from –1 (perfect polarization) to 0 (uniform distribution) to +1 (perfect agreement).

This approach constructs this coefficient by decomposing the empirical distribution into uniform “layers” and then averaging the individual agreement levels of these layers. This method is considered more robust and interpretable than standard deviation-based measures because it better captures the nature of agreement, especially in ordinal data.

Operationalization

The coefficient of consensus is constructed in three main steps. First, any empirical distribution on an ordered rating scale is decomposed into “layers” with each layer consisting of a simple pattern of equal frequencies across selected response categories. Second, each layer is evaluated for its degree of agreement based on how concentrated the responses are. The overall agreement score than is computed as a weighted average of the agreement scores of all layers, with each layer weighted by the proportion of responses it contains.

Since values of the agreement coefficient AA are not always easily interpretable, a polarization index PP is often used as a reversed and standardized version. This transformation maps the index onto a [0, 1] scale, where 0 represents perfect agreement and 1 represents perfect polarization.

Use cases

Publications that use this measure: