Exercise: observer agreement about sex

This website is for students following the M.Sc. in Evidence Based Practice at the University of York.

183 students were observed twice by different student observers. These measured height (mm), arm circumference (mm), head circumference, and pulse (beats/min) and recorded sex and eye colour. They entered these into a computer file. Eye colour and sex were entered as numerical codes.

The following table shows sex as recorded by two observers:

Sex recorded by
first observer Sex recorded by
second observer Total
female male
female 118 1 119
male 1 63 64
Total 119 64 183

Sex recorded by first observer	Sex recorded by second observer	Total
female	male
female	118	1	119
male	1	63	64
Total	119	64	183

This is the output from SPSS 16, where kappa is a statistic available from crosstabs:

Symmetric measures

Value Assym. Std.
Error^a
Approx T^b
Approx. Sig.
Measure of agreement
Kappa 0.976 0.017
13.203
.000

N of Valid Cases 183
a: Not assuming the null hypothesis.
b: Using the asymptotic standard error assuming the null hypothesis.

Symmetric measures
	Value	Assym. Std. Error^a	Approx T^b	Approx. Sig.
Measure of agreement	Kappa	0.976	0.017	13.203	.000
N of Valid Cases	183
a: Not assuming the null hypothesis. b: Using the asymptotic standard error assuming the null hypothesis.

This is the output from a Stata command for Cohen's kappa:

. kap sex1 sex2

             Expected
Agreement   Agreement     Kappa   Std. Err.         Z      Prob>Z
-----------------------------------------------------------------
  98.91%      54.52%     0.9760     0.0739      13.20      0.0000

Question 1:

What is meant by “Agreement” and “Expected agreement”?

Check suggested answer 1.

Question 2:

What does kappa mean and what can we conclude?

Check suggested answer 2.

Question 3:

What is “Z”?

Check suggested answer 3.

Question 4:

“Prob>Z” is the P value. What is it testing?

Check suggested answer 4.

Question 5:

How is it possible for kappa to be less than 1.00 for sex?

Check suggested answer 5.

The following is a data listing for the two students who appear to have changed sex:

Student A
First Second
height 1858 1607
arm 234 258
head 565 535
pulse 66 72
eye hazel brown
sex male female

Student A
	First	Second
height	1858	1607
arm	234	258
head	565	535
pulse	66	72
eye	hazel	brown
sex	male	female

Student B
First Second
height 1717 1711
arm 247 251
head 539 536
pulse 84 72
eye green green
sex female male

Student B
	First	Second
height	1717	1711
arm	247	251
head	539	536
pulse	84	72
eye	green	green
sex	female	male

Question 6:

What do you think has happened in the case of Student A?

Check suggested answer 6.

Question 7:

What do you think has happened in the case of Student B?

Check suggested answer 7.

Back to Measurement in Health and Disease index.

To Martin Bland's home page.

This page maintained by Martin Bland.
Last updated: 21 July, 2008.