Question 1: What is the problem with using the percentage agreement as a measure of agreement between observers? Do percentage agreements between 67 and 76 represent a high degree of agreement, as concluded?
When two observers record the presence or absence of something, they will always agree on some cases unless they are deliberately setting out to disagree. If they record completely at random, without the actual presence of the characteristic influencing their decision, the proportion of cases for which they agree is expected to be at least one half. This occurs when both record presence for 50% of case; otherwise the agreement will be greater. When both observers record the presence of the characteristic for a high proportion of cases, or both for a low proportion of cases, the proportion of cases for which they agree will be much greater than 50%. For example, if both record presence for a random 80% of cases, we expect agreement for a proportion = 0.82 + 0.22 = 0.68 or 68%. (The proportion where both record presence would be 0.8 times 0.8 and the proportion where both record absence would be 0.2 times 0.2. The agreement is the sum of these.) Hence the percentages reported may represent little more agreement than we would expect by chance. They do not represent a high degree of agreement
Back to question.
This page maintained by Martin Bland.
Last updated: 1 March, 2007.