=  How do I compare observed numbers correct to those expected by chance in a multi-choice task? =

Suppose a questionnaire has a set of k possible responses to N questions of which one of the k choices is the correct response. Assuming by chance each of the k responses to each question is equally likely the number expected by chance equals N/k.

Suppose we observe x $$\leq$$ N responses for a patient. We wish to see how likely x responses are given the patient responds at random to k choices on each of N questions.

To do this we can assume the number of correct responses, x, follows a Poisson distribution, Po(m) of general form:

$$P(X=x|\mbox{random responses}) = \frac{\m^text{x}}{x!}e^text{-m}$$ 

where $$\mu$$ is the expected total number of correct responses for N questions. Since this equals N/k we can rewrite the above as:

$$P(X=x|\mbox{random responses}) = \frac{(N/k)^text{x}}{x!}e^text{-(N/k)}$$

one-tailed p-value = $$P(X \leq x) = \sum_text{1}^text{x}P(X=x|\mbox{random responses})$$
= 0.5(two-tailed p-value)

For large N the Poisson distribution is approximately
Normally distributed with mean and variance both equal to N/k so we can analogously obtain a one-tailed p-value as:

$$P(X \leq x) = Probit(\frac{x-(N/k)}{\sqrt{N/k}})$$.

__Example__

Suppose we have 14 questions each with 3 possible responses of which only one is correct and a patient gets a total score of 3 correct responses. We wish to determine how likely it is we would observe 3 or fewer responses given the patient has responded at random (1 sided p-value).

The expected number of correct responses assuming the patient is responding at random is 14/3=4.67.

Using the Poisson distribution

P(X $$\leq$$ 3) = P(X = 0) + P(X = 1) + P(X = 2) + P(X = 3)

$$ = (1 + 4.67 + \frac{4.67^text{2}}{2} + \frac{4.67^text{3}}{6})e^text{-4.67}$$

=  0.31.

Since P(X $$\leq$$ 3) = P(X $$\leq$$ 14-3) because the Poisson distribution is symmetric the two-tailed p-value = 0.31x2= 0.628.

We conclude there is no evidence to suggest a score of 3 on a three-choice task of 14 questions differs from chance.

Using the Normal approximation:

$$P(X \leq 3) = Probit \frac{3-4.67}{\sqrt{4.67}} = Probit(-0.772) = 0.22 with a two-sided p-value of 0.44.

We again conclude there is no evidence to suggest a score of 3 on a three-choice task of 14 questions differs from chance.

For N > 30 the Poisson and Normal approximations should closely agree. For N < 30 the Poisson is preferable as it is a discrete distribution assuming, as in this example, only integer values can occur.