Skip to main content
Astronomy

Return to Answer

added 458 characters in body
Source Link
David Hammen
David Hammen
  • 34.1k
  • 3
  • 74
  • 126

Should I expect a decrease in the reduced chi-square values with increasing n?

You should see a decrease up to the point where you start overfitting -- assuming that your assumption that the correct model is some kind of polynomial is correct. If it's not a polynomial, adding ever more terms might by luck capture ever more of the unmodeled variation.

Suppose one has captured data at $n>4$ points along some domain of interest, each with a bit of measurement error. Suppose the true model is a quadratic. The regression will improve with each step in going from a constant to a line to a quadratic to a cubic, so the chi-squared value will decrease with each step (but not much for the last step from a quadratic to a cubic), The reduced chi-squared value will decrease nicely on moving from a constant to a line to a quadratic. It won't reduce by much, or it might even increase on that last step from a quadratic to a cubic. That's a sign of overfitting (and also of using the wrong model).

Suppose one has lots of measurements (e.g., $n=1000$) and the measurement noise is not particularly small. Going from a constant to a line to a quadratic still results in a nice stepwise decrease in the reduced chi-square value. Here the step to a cubic might also result in a decrease because the addition of the cubic term might happen to capture some of the measurement noise and because dividing by 996 is not all that different from dividing by 997.

Should I expect a decrease in the reduced chi-square values with increasing n?

You should see a decrease up to the point where you start overfitting -- assuming that your assumption that the correct model is some kind of polynomial is correct. If it's not a polynomial, adding ever more terms might by luck capture ever more of the unmodeled variation.

Suppose one has captured data at $n>4$ points along some domain of interest, each with a bit of measurement error. Suppose the true model is a quadratic. The regression will improve with each step in going from a constant to a line to a quadratic to a cubic, so the chi-squared value will decrease with each step (but not much for the last step from a quadratic to a cubic), The reduced chi-squared value will decrease nicely on moving from a constant to a line to a quadratic. It won't reduce by much, or it might even increase on that last step from a quadratic to a cubic. That's a sign of overfitting (and also of using the wrong model).

Should I expect a decrease in the reduced chi-square values with increasing n?

You should see a decrease up to the point where you start overfitting -- assuming that your assumption that the correct model is some kind of polynomial is correct. If it's not a polynomial, adding ever more terms might by luck capture ever more of the unmodeled variation.

Suppose one has captured data at $n>4$ points along some domain of interest, each with a bit of measurement error. Suppose the true model is a quadratic. The regression will improve with each step in going from a constant to a line to a quadratic to a cubic, so the chi-squared value will decrease with each step (but not much for the last step from a quadratic to a cubic), The reduced chi-squared value will decrease nicely on moving from a constant to a line to a quadratic. It won't reduce by much, or it might even increase on that last step from a quadratic to a cubic. That's a sign of overfitting (and also of using the wrong model).

Suppose one has lots of measurements (e.g., $n=1000$) and the measurement noise is not particularly small. Going from a constant to a line to a quadratic still results in a nice stepwise decrease in the reduced chi-square value. Here the step to a cubic might also result in a decrease because the addition of the cubic term might happen to capture some of the measurement noise and because dividing by 996 is not all that different from dividing by 997.

Source Link
David Hammen
David Hammen
  • 34.1k
  • 3
  • 74
  • 126

Should I expect a decrease in the reduced chi-square values with increasing n?

You should see a decrease up to the point where you start overfitting -- assuming that your assumption that the correct model is some kind of polynomial is correct. If it's not a polynomial, adding ever more terms might by luck capture ever more of the unmodeled variation.

Suppose one has captured data at $n>4$ points along some domain of interest, each with a bit of measurement error. Suppose the true model is a quadratic. The regression will improve with each step in going from a constant to a line to a quadratic to a cubic, so the chi-squared value will decrease with each step (but not much for the last step from a quadratic to a cubic), The reduced chi-squared value will decrease nicely on moving from a constant to a line to a quadratic. It won't reduce by much, or it might even increase on that last step from a quadratic to a cubic. That's a sign of overfitting (and also of using the wrong model).