How do I compare observed numbers correct to those expected by chance in a multi-choice task?
Suppose a questionnaire has a set of k possible responses to N questions of which one of the k choices is the correct response. Assuming by chance each of the k responses to each question is equally likely the number expected by chance equals N/k.
Suppose we observe x $$\leq$$ N responses for a patient. We wish to see how likely x responses are given the patient responds at random to k choices on each of N questions.
To do this we can assume the number of correct responses, x, follows a Poisson distribution, Po($$\mu$$) of general form:
$$P(X=x|random responses) = \frac{\mutext{x}}{x!}etext{-\mu}$$
where $$\mu$$ is the expected total number of correct responses for N questions. Since this equals N/k we can rewrite the above as:
$$P(X=x|random responses) = \frac{(N/k)text{x}}{x!}etext{-(N/k)}$$
For large N the Poisson distribution is approximately Normally distributed with mean and variance both equal to N/k.
so
$$P(X=x) = INV(\frac{x-(N/k)}{\sqrt{N/k}})$$
where INV is the inverse Normal function.
Example
Suppose we have 14 questions each with 3 possible responses of which only one is correct and a patient gets a total score of 3 correct responses. We wish to determine how likely it is we would observe 3 or fewer responses given the patient has responded at random (1 sided p-value).
The expected number of correct responses assuming the patient is responding at random is 14/3=3.67.
Using the Poisson distribution
P(X $$\leq$$ 3) = P(X = 0) + P(X = 1) + P(X = 2) + P(X = 3) =