=  How do I compare observed numbers correct to those expected by chance in a multi-choice task? =

Suppose a questionnaire has a set of k possible responses to N questions of which one of the k choices is the correct response. Assuming by chance each of the k responses to each question is equally likely the number expected by chance equals N/k.

Suppose we observe x $$\leq$$ N responses for a patient. We wish to see how likely x responses are given the patient responds at random to k choices on each of N questions.

To do this we can assume the number of correct responses, x, follows a Poisson distribution, Po($$\mu$$) of general form:

$$P(X=x|random responses) = \frac{\mu^text{x}}{x!}e^text{-\mu}$$ 

where $$\mu$$ is the expected total number of correct responses for N questions. Since this equals N/k we can rewrite the above as:

$$P(X=x|random responses) = \frac{(N/k)^text{x}}{x!}e^text{-(N/k)}$$

For large N the Poisson distribution is approximately
Normally distributed with mean and variance both equal to N/k.

so

$$P(X=x) = INV(\frac{x-(N/k)}{\sqrt{N/k}})$$

where INV is the inverse Normal function.

__Example__

Suppose we have 14 questions each with 3 possible responses of which only one is correct and a patient gets a total score of 3 correct responses. We wish to determine how likely it is we would observe 3 or fewer responses given the patient has responded at random (1 sided p-value).

The expected number of correct responses assuming the patient is responding at random is 14/3=3.67.

Using the Poisson distribution

P(X $$\leq$$ 3) = P(X = 0) + P(X = 1) + P(X = 2) + P(X = 3) =