Variance of $n$ Pairwise Correlated r.v.'s

It’s well-known that the average of $n$ i.i.d. r.v.’s with variance $σ^{2}$ has variance $\frac{1}{n} σ^{2}$ .

Suppose that the r.v.’s $X_{i}$ ’s are only identically distributed with variance $σ^{2}$ and pairwise correlation $ρ$ , then we have:

V (\frac{1}{n} \sum_{i = 1}^{n} X_{i}) = \frac{1}{n^{2}} (n σ^{2} + n (n - 1) ρ σ^{2}) = \frac{σ^{2}}{n} + \frac{(n - 1) ρ σ^{2}}{n} = ρ σ^{2} + \frac{1 - ρ}{n} σ^{2} .

This equation suggests that as the size $n$ increases, though the second term disappears, the first term remains constant. So the averaging over more and more correlated r.v.’s will not keep improving the variance.

Wait a second! Our derivation of the above equation doesn’t depend on sign of $ρ$ . But the above equation seems to fail when $ρ < 0$ since the variance would then be negative for a sufficiently large $n$ . What’s wrong with it?

Let’s consider the simple case when $n = 3$ and $ρ = - 1$ . First of all, is it even possible to construct such 3 r.v.’s? We see that since $X_{i}$ ’s are identically distributed, if $ρ_{X_{1}, X_{2}} = - 1$ and $ρ_{X_{1}, X_{3}} = - 1$ , we have $X_{2} = - X_{1}$ and $X_{3} = - X_{1}$ . But this contradicts the fact that $ρ_{X_{2}, X_{3}} = - 1$ .

Actually, we cannot have arbitrary number of r.v.’s that are identically distributed with a negative pairwise correlation. We will show it by using the fact that the determinant of a correlation matrix must be positive semi-definite.

Let $ρ > 0$ and the correlation matrices for the case of positive and negative pairwise correlations are denoted by:

Σ (ρ) = (\begin{matrix} 1 & ρ & \dots & ρ \\ ρ & 1 & ⋱ & ⋮ \\ ⋮ & ⋱ & ⋱ & ρ \\ ρ & \dots & ρ & 1 \end{matrix}) and Σ (- ρ) = (\begin{matrix} 1 & - ρ & \dots & - ρ \\ - ρ & 1 & ⋱ & ⋮ \\ ⋮ & ⋱ & ⋱ & - ρ \\ - ρ & \dots & - ρ & 1 \end{matrix}),

respectively.

Then we have the following:

\begin{aligned} det Σ (ρ) & = det ((\begin{array}{c} 1 - ρ \\ ⋱ \\ 1 - ρ \end{array}) + (\begin{array}{c} \sqrt{ρ} \\ \sqrt{ρ} \\ \sqrt{ρ} \end{array}) (\begin{array}{c} \sqrt{ρ} & \sqrt{ρ} & \sqrt{ρ} \end{array})) \\ = (1 + (\begin{array}{c} \sqrt{ρ} & \sqrt{ρ} & \sqrt{ρ} \end{array}) (\begin{array}{c} (1 - ρ)^{- 1} \\ ⋱ \\ (1 - ρ)^{- 1} \end{array}) (\begin{array}{c} \sqrt{ρ} \\ \sqrt{ρ} \\ \sqrt{ρ} \end{array})) (1 - ρ)^{n} \\ = (1 + \frac{n ρ}{1 - ρ}) (1 - ρ)^{n} > 0, \forall ρ \in (0, 1), \end{aligned}

where we used the matrix determinant lemma: $\det (A + {u v}^{⊤}) = (1 + v^{⊤} A^{- 1} u) \det (A)$ . It shows that $det Σ (ρ)$ is always positive definite for $ρ \in (0, 1)$ . So there’s no problem with such identically distribution r.v.’s with a positive pairwise correlation $ρ$ .

Now, let’s compute determinant of $Σ (- ρ)$ :

\begin{aligned} det Σ (- ρ) & = det ((\begin{array}{c} 1 + ρ \\ ⋱ \\ 1 + ρ \end{array}) + (\begin{array}{c} \sqrt{ρ} \\ \sqrt{ρ} \\ \sqrt{ρ} \end{array}) (\begin{array}{c} - \sqrt{ρ} & - \sqrt{ρ} & - \sqrt{ρ} \end{array})) \\ = (1 + (\begin{array}{c} - \sqrt{ρ} & - \sqrt{ρ} & - \sqrt{ρ} \end{array}) (\begin{array}{c} (1 + ρ)^{- 1} \\ ⋱ \\ (1 + ρ)^{- 1} \end{array}) (\begin{array}{c} \sqrt{ρ} \\ \sqrt{ρ} \\ \sqrt{ρ} \end{array})) (1 + ρ)^{n} \\ = (1 + \frac{- n ρ}{1 + ρ}) (1 + ρ)^{n} . \end{aligned}

To ensure $det Σ (- ρ) \geq 0$ , we need the condition that

1 + \frac{- n ρ}{1 + ρ} \geq 0 \Leftrightarrow 1 + ρ \geq n ρ \Leftrightarrow ρ \leq \frac{1}{n - 1} \Leftrightarrow - ρ \geq \frac{1}{1 - n} .

Thus, we see that, for $n$ r.v.’s that are identically distributed with a negative pairwise correlation $- ρ$ , $- ρ$ actually cannot be smaller than $\frac{1}{1 - n}$ . For example, 3 identically distributed r.v.’s cannot have a pairwise correlation smaller than $- \frac{1}{2}$ and this bound becomes $- \frac{1}{3}$ for 4 identically distributed r.v.’s.

Lastly, we make two more observations based on the above findings.

For identically distributed r.v.’s $X_{i}$ ’s with pairwise correlation $ρ$ , what are the maximum and minimum values of $ρ$ ?

\frac{1}{1 - n} \leq ρ \leq 1.

How do we construct such $X_{i}$ ’s with pairwise correlation $ρ$ ?

Well, for a positive pairwise correlation, we’ve showed that $Σ (ρ)$ is always positive definite. Thus, there exists a decomposition $Σ (ρ) = C C^{'}$ for some matrix $C$ . Then, for any i.i.d. r.v.’s $Z_{i}$ ’s with unit variance, the linear transformation $X = C Z$ should have the desired variance-covariance matrix since

V (X) = V (C Z) = C V (Z) C^{'} = C C^{'} = Σ (ρ) .

For a negative pairwise correlation $- ρ$ , as long as $- ρ > \frac{1}{1 - n}$ , we know $Σ (- ρ)$ is also positive definite, thus the above construction also works in this case.

Variance of n Pairwise Correlated r.v.'s Ibuki

Variance of $n$ Pairwise Correlated r.v.'s Ibuki