Seto's Page

　It is well known that the count of events occurring randomly in time or space follows a Poisson distribution.

　The probability \(P(\lambda, k)\) that an event expected to be observed \(\lambda\) times is observed exactly \(k\) times is given by:$$P(\lambda, k) = \frac{\lambda^k e^{-\lambda}}{k!}$$This is the Poisson distribution. The expected value of this distribution is \(\lambda\), and the variance is also \(\lambda\). For example, even if a station is used by an average of 10,000 people per day, the probability that the number of users on a given day is exactly 10,000 is only 0.4% ( \(= \displaystyle\frac{10000^{1000} e^{-10000}}{10000!}\) ).

　Now, we casually wrote “expected to be observed \(\lambda\) times,” but in general, knowing the true value of \(\lambda\) is difficult (unless God tells us). In most cases, we have no choice but to accumulate observations and estimate a value believed to be close to the true value (the maximum likelihood estimate) or a range that contains the true value. When a range that contains the true value with a certain probability can be estimated, that range is called a confidence interval. For example, a 95% confidence interval means the range where the probability of obtaining a value less than or equal to the observed value is 2.5% or less, and the probability of obtaining a value greater than or equal to the observed value is also 2.5% or less.

　For a Poisson distribution, when \(\lambda\) is sufficiently large, the distribution closely resembles a normal distribution with mean \(\lambda\) and variance \(\lambda\). In this case, the confidence interval follows the familiar normal distribution expressions: \([\lambda – \sqrt{ \lambda}, \lambda + \sqrt{ \lambda}] \) is the 68% confidence interval, and \([\lambda – 2\sqrt{ \lambda}, \lambda + 2\sqrt{ \lambda}] \) is the 95% confidence interval.

　On the other hand, when \(\lambda\) is small, the normal distribution approximation breaks down. This is because, for example, if an event was observed 0 times, the above approach would give an error of 0 (meaning the phenomenon never occurs), which is clearly wrong. Computing the confidence interval for a Poisson distribution requires solving rather complex integrals, but fortunately there exists a convenient open-source statistical analysis software called R (http://www.r-project.org/). The table below shows the upper and lower bounds of the confidence interval computed for small values of \(\lambda\). In R, you can calculate this immediately with a command like poisson.test(some_number, conf.level = 0.68).