Python numpy.corrcoef() RuntimeWarning: invalid value encountered in true_divide c /= stddev[:, None]

Question

It seems that corrcoef from numpy throw a RuntimeWarning when a constant list passed to the corrcoef() function, for example the below code throw a warning :

import numpy as np
X = [1.0, 2.0, 3.0, 4.0]
Y = [2, 2, 2, 2]
print(np.corrcoef(X, Y)[0, 1])

Warning :

/usr/local/lib/python3.6/site-packages/numpy/lib/function_base.py:3003: RuntimeWarning: invalid value encountered in true_divide
  c /= stddev[:, None]

Can anyone explain why it's throwing this error when one of the lists is constant and how to prevent this error when a constant list is passed to the function?

Answer 1

Correlation is a measure of how well two vectors track with each other as they change. You can't track mutual change when one vector doesn't change.

As noted in OP comments, the formula for Pearson's product-moment correlation coefficient divides the covariance of X and Y by the product of their standard deviations. Since Y has zero variance in your example, its standard deviation is also zero. That's why you get the true_divide error - you're trying to divide by zero.

Note: It might seem tempting, from an engineering standpoint, to simply add a very small quantity (say, a value just above machine epsilon) onto one of the entries in Y, in order to get around the zero-division issue. But that's not statistically viable. Even adding 1e-15 will seriously derange your correlation coefficient, depending on which value you add it to.

Consider the difference between these two cases:

X = [1.0, 2.0, 3.0, 4.0]

tiny = 1e-15

# add tiny amount to second element
Y1 = [2., 2.+tiny, 2., 2.]
np.corrcoef(X, Y1)[0, 1] 
-0.22360679775

# add tiny amount to fourth element
Y2 = [2., 2., 2., 2.+tiny]
np.corrcoef(X, Y2)[0, 1]
0.67082039325

This may be obvious to statisticians, but given the nature of the question it seems like a relevant caveat.

Answer 2

If you would just like to suppress the warning, you can modify the function like this:

import numpy as np
X = [1.0, 2.0, 3.0, 4.0]
Y = [2, 2, 2, 2]
with np.errstate(divide="ignore", invalid="ignore"): 
    print(np.corrcoef(X, Y))

That should make it robust to divisions by zero and NaN values.

Answer 3

The Pandas .corr is more versatile in such conditions and even presence of NaN(s), so first option could be to convert the lists to pandas series and use the pandas correlation and the response would be a nan;

import pandas as pd
X = [1.0, 2.0, 3.0, 4.0]
Y = [2, 2, 2, 2]
print(pd.Series(X).corr( pd.Series(Y)))

If you want to stick to numpy then you can check the standard deviation of series using an if statement, and proceed for correlation only when they are more than zero. Apparently here we have no valid output because one list has no variation, but the concept could be applied to other cases.

import numpy as np
X = [1.0, 2.0, 3.0, 4.0]
Y = [2, 2, 2, 2]
if np.std(Y)==0 or np.std(X)==0 :
    print ('The correlation could not be computed because the standard deviation of one of the series is equal to zero')
else:
    print(np.corrcoef(X, Y)[0, 1])

Answer 4

Here is a "fixed" version of np.corrcoef:

def pearsonccs(samples):
    C = np.cov(samples)
    diag = np.diag(C)
    N = np.sqrt(np.outer(diag, diag))
    N[N == 0] = 1
    return C / N

All it does is replacing the zeroes in the covariance matrix with one to avoid the divide-by-zero-error.