How do I compare observed numbers correct to those expected by chance in a multi-choice task?
Suppose a questionnaire has a set of k possible responses to N questions of which one of the k choices is the correct response. Assuming by chance each of the k responses to each question is equally likely the number expected by chance equals N/k.
Suppose we observe x $$\leq$$ N responses for a patient. We wish to see how likely x responses are given the patient responds at random to k choices on each of N questions.
To do this we can assume the number of correct responses, x, follows a Poisson distribution, Po(m) of general form:
$$P(X=x|\mbox{random responses}) = \frac{mtext{x}}{x!}etext{-m}$$
where $$m$$ is the expected total number of correct responses for N questions. Since this equals N/k we can rewrite the above as:
$$P(X=x|\mbox{random responses}) = \frac{(N/k)text{x}}{x!}etext{-(N/k)}$$
one-tailed p-value = $$P(X \leq x) = \sum_text{1}^text{x}P(X=x|\mbox{random responses})$$ = 0.5(two-tailed p-value)
For large N the Poisson distribution is approximately Normally distributed with mean and variance both equal to N/k so we can analogously obtain a one-tailed p-value as:
$$P(X \leq x) = Probit(\frac{x-(N/k)}{\sqrt{N/k}})$$.
Example
Suppose we have 14 questions each with 3 possible responses of which only one is correct and a patient gets a total score of 3 correct responses. We wish to determine how likely it is we would observe 3 or fewer responses given the patient has responded at random (1 sided p-value).
The expected number of correct responses assuming the patient is responding at random is 14/3=4.67.
Using the Poisson distribution
P(X $$\leq$$ 3)
= P(X = 0) + P(X = 1) + P(X = 2) + P(X = 3)
$$ = (1 + 4.67 + \frac{4.67text{2}}{2} + \frac{4.67text{3}}{6})e^text{-4.67}$$
= 0.31.
We can think of the p-value as the sum of numbers correct which are less likely assuming random responses than the observed one. It turns out that P(X $$\geq$$ 6) also account for less likely occurrences given random responses than the observed 3 correct.
Since P(X $$\leq$$ 3) is approximately equal to P(X $$\geq$$ 6) because the Poisson distribution is symmetric the two-tailed p-value = 0.31 x 2= 0.62.
We conclude there is no evidence to suggest a score of 3 on a three-choice task of 14 questions differs from chance.
Using the Normal approximation to the Poisson distribution:
$$P(X \leq 3) = Probit (\frac{3-4.67}{\sqrt{4.67}}) = Probit(-0.772) = 0.22$$ with a two-sided p-value of 0.44. The two-tailed probability equals $$P(X \leq 3) + P(X \geq 6.33)$$ since ths Normal distribution probabilities are symmetric about the mean of 4.67. of course we can't observed 6.33 correct responses but this is an approximation to the Poisson distribution!
We again conclude there is no evidence to suggest a score of 3 on a three-choice task of 14 questions differs from chance. These calculations may be computed using a [attachment:multichoice.xls spreadsheet.]
[WORK IN PROGRESS: Exact two-tailed p-value for the Poisson distribution giving the chance of numbers correct occurring which are no more likely than the m=number correct observed assuming random responses.]
For N > 30 the Poisson and Normal approximations should closely agree. For N < 30 the Poisson is preferable as it is a discrete distribution assuming, as in this example, only integer values can occur.