# The Chi-Squared Distribution, Part 1

The chi-squared distribution has a simple definition from a mathematical standpoint and yet plays an important role in statistical sampling theory. This post is the first post in a three-part series that gives a mathematical story of the chi-squared distribution.

This post is an introduction which highlights the fact that mathematically chi-squared distribution arises from the gamma distribution and that the chi-squared distribution has an intimate connection with the normal distribution. This post lays the ground work for the subsequent post.

The next post (Part 2) describe the roles played by the chi-squared distribution in forming the various sampling distributions related to the normal distribution. These sampling distributions are used for making inference about the population from which the sample is taken. The population parameters of interest here are the population mean, variance, and standard deviation. The population from which the sample is take is assumed to be modeled adequately by a normal distribution.

Part 3 describes the chi-squared test, which is used for making inference on categorical data (versus quantitative data).

These three parts only scratches the surface with respect to the roles played the chi-squared distribution in statistics. Thus the discussion in this series only serves as an introduction on chi-squared distribution.

_______________________________________________________________________________________________

Defining the Chi-Squared Distribution

A random variable $Y$ is said to follow the chi-squared distribution with $k$ degrees of freedom if the following is the density function of $Y$.

$\displaystyle f_Y(y)=\frac{1}{\Gamma(\frac{k}{2}) \ 2^{\frac{k}{2}}} \ y^{\frac{k}{2}-1} \ e^{-\frac{y}{2}} \ \ \ \ \ \ \ y>0 \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (1)$

where $k$ is a positive integer. In some sources, the distribution is sometimes named $\chi^2$ distribution. Essentially the distribution defined in $(1)$ is a gamma distribution with shape parameter $\frac{k}{2}$ and scale parameter 2 (or rate parameter $\frac{1}{2}$). Note that the chi-squared distribution with 2 degrees of freedom (when $k=2$) is simply an exponential distribution with mean 2. The following figure shows the chi-squared density functions for degrees of freedom 1, 2, 3, 5 and 10.

Figure 1 – Chi-squared Density Curves

Just from the gamma connection, the mean and variance are $E[Y]=k$ and $Var[Y]=2k$. In other words, the mean of a chi-squared distribution is the same as the degrees of freedom and its variance is always twice the degrees of freedom. As a gamma distribution, the higher moments $E[Y^n]$ are also known. Consequently the properties that depend on $E[Y^n]$ can be easily computed. See here for the basic properties of the gamma distribution. The following gives the mean, variance and the moment generating function (MGF) for the chi-squared random variable $Y$ with $k$ degrees of freedom.

$E[Y]=k$

$Var[Y]=2 k$

$\displaystyle M_Y(t)=\biggl( \frac{1}{1-2t} \biggr)^{\frac{k}{2}} \ \ \ \ \ \ t<\frac{1}{2}$

_______________________________________________________________________________________________

Independent Sum of Chi-Squared Distributions

In general, the MGF of an independent sum $Y_1+Y_2+\cdots+Y_n$ is simply the product of the MGFs of the individual random variables $X_i$. Note that the product of Chi-squared MGFs is also a Chi-squared MGF, with the exponent being the sum of the individual exponents. This brings up another point that is important for the subsequent discussion, i.e. the independent sum of chi-squared distributions is also a chi-squared distribution. The following theorem states this fact more precisely.

Theorem 1
If $Y_1,Y_2,\cdots,Y_n$ are chi-squared random variables with degrees of freedom $k_1,k_2,\cdots,k_n$, respectively, then the independent sum $Y_1+Y_2+\cdots+Y_n$ has a chi-squared distribution with $k_1+k_2+\cdots+k_n$ degrees of freedom.

Thus the result of summing independent chi-squared distributions is another chi-squared distribution with degree of freedom being the total of all degrees of freedom. This follows from the fact that if the gamma distributions have identical scale parameter, then the independent sum is a gamma distribution with the shape parameter being the sum of the shape parameters. This point is discussed in more details here.

_______________________________________________________________________________________________

The Connection with Normal Distributions

As shown in the above section, the chi-squared distribution is simple from a mathematical standpoint. Since it is a gamma distribution, it possesses all the properties that are associated with the gamma family. Of course, the gamma connection is far from the whole story. One important fact is that the chi-squared distribution is naturally obtained from sampling from a normal distribution.

Theorem 2
Suppose that the random variable $X$ follows a standard normal distribution, i.e. the normal distribution with mean 0 and standard deviation 1. Then $Y=X^2$ follows a chi-squared distribution with 1 degree of freedom.

Proof
By definition, the following is the cumulative distribution function (CDF) of $Y=X^2$.

\displaystyle \begin{aligned}F_Y(y)=P[Y \le y] &=P[X^2 \le y]=P[-\sqrt{y} \le X \le \sqrt{y}]=2 \ P[0 \le X \le \sqrt{y}] \\&=2 \ \int_0^{\sqrt{y}} \frac{1}{\sqrt{2 \pi}} \ e^{-\frac{x^2}{2}} \ dx \end{aligned}

Upon differentiating $F_Y(y)$, the density function is obtained.

\displaystyle \begin{aligned}f_Y(y)=\frac{d}{dy}F_Y(y) &=2 \ \frac{d}{dy} \int_0^{\sqrt{y}} \frac{1}{\sqrt{2 \pi}} \ e^{-\frac{x^2}{2}} \ dx \\ &=\frac{1}{\Gamma(\frac{1}{2}) \ 2^{\frac{1}{2}}} \ y^{\frac{1}{2}-1} \ e^{-\frac{y}{2}}\end{aligned}

Note that the density is that of a chi-squared distribution with 1 degree of freedom. $\square$

With the basic result in Theorem 1, there are more ways to obtain chi-squared distributions from sampling from normal distributions. For example, first normalizing a sample item from normal sampling and then squaring it will produce a chi-squared observation with 1 degree of freedom. Similarly, by performing the same normalizing in each sample item in a normal sample and by squaring each normalized observation, the resulting sum is a chi-squared distribution. These are made more precise in the following corollaries.

Corollary 3
Suppose that the random variable $X$ follows a normal distribution with mean $\mu$ and standard deviation $\sigma$. Then $Y=[(X-\mu) / \sigma]^2$ follows a chi-squared distribution with 1 degree of freedom.

Corollary 4
Suppose that $X_1,X_2,\cdots,X_n$ is a random sample drawn from a normal distribution with mean $\mu$ and standard deviation $\sigma$. Then the following random variable follows a chi-squared distribution with $n$ degrees of freedom.

$\displaystyle \sum \limits_{j=1}^n \biggl( \frac{X_j-\mu}{\sigma} \biggr)^2=\biggl( \frac{X_1-\mu}{\sigma} \biggr)^2+\biggl( \frac{X_2-\mu}{\sigma} \biggr)^2+\cdots+\biggl( \frac{X_n-\mu}{\sigma} \biggr)^2$

_______________________________________________________________________________________________

Calculating Chi-Squared Probabilities

In working with the chi-squared distribution, it is necessary to evaluate the cumulative distribution function (CDF). In hypothesis testing, it is necessary to calculate the p-value given the value of the chi-squared statistic. In confidence interval estimation, it is necessary to determine the the critical value at a given confidence level. The standard procedure at one point in time is to use a chi-squared table. A typical chi-squared table can be found here. We demonstrate how to find chi-squared probabilities first using the table approach and subsequently using software (Excel in particular).

The table gives the probabilities on the right tail. The table in the link given above will give the chi-squared value (on the x-axis) $\chi_{\alpha}^2$ for a given area of the right tail ($\alpha$) per df. This table lookup is illustrated in the below diagram.

Figure 2 – Right Tail of Chi-squared Distribution

For df = 1, $\chi_{0.1}^2=2.706$, thus $P[\chi^2 > 2.706]=0.1$ and $P[\chi^2 < 2.706]=0.9$. So for df = 1, the 90th percentile of the chi-squared distribution is 2.706. The following shows more table lookup.

df = 2, $\chi_{0.01}^2=9.210$.
$P[\chi^2 > 9.210]=0.01$ and $P[\chi^2 < 9.210]=0.99$
The 99th percentile of the chi-squared distribution with df = 2 is 9.210.

df = 15, $\chi_{0.9}^2=8.547$.
$P[\chi^2 > 8.547]=0.9$ and $P[\chi^2 < 8.547]=0.1$
The 10th percentile of the chi-squared distribution with df = 15 is 8.547.

The choices for $\alpha$ in the table are limited. Using software will have more selection for $\alpha$ and will give more precise values. For example, Microsoft Excel provides the following two functions.

=CHISQ.DIST(x, degree_freedom, cumulative)

=CHISQ.INV(probability, degree_freedom)

The two functions in Excel give information about the left-tail of the chi-squared distribution. The function CHISQ.DIST returns the left-tailed probability of the chi-squared distribution. The parameter cumulative is either TRUE or FALSE, with TRUE meaning that the result is the cumulative distribution function and FALSE meaning that the result is the probability density function. On the other hand, the function CHISQ.INV returns the inverse of the left-tailed probability of the chi-squared distribution.

If the goal is to find probability given an x-value, use the function CHISQ.DIST. On the other hand, if the goal is to look for the x-value given the left-tailed value (probability), then use the function CHISQ.INV. In the table approach, once the value $\chi_{\alpha}^2=x$ is found, the interplay between the probability ($\alpha$) and x-value is clear. In the case of Excel, one must choose the function first depending on the goal. The following gives the equivalent results for the table lookup presented above.

=CHISQ.DIST(2.706, 1, TRUE) = 0.900028622
=CHISQ.INV(0.9, 1) = 2.705543454

=CHISQ.DIST(9.21, 2, TRUE) = 0.989998298
=CHISQ.INV(0.99, 2) = 9.210340372

=CHISQ.DIST(8.547, 15, TRUE) = 0.100011427
=CHISQ.INV(0.1, 15) = 8.546756242

_______________________________________________________________________________________________
$\copyright \ 2016 - \text{Dan Ma}$