Question about Statistical measures

Question

I have recently tried parameter estimation for nonlinear ODE using non-linear fitting techniques. I learned about Statistical measures like p-tests, t-tests, $R^2$ squared, adjusted $R^2$ square, etc. I tried to go through some literature and found a few things (please correct me if any of my observation is false):

If a p-value reported from a t-test is less than 0.05, then that result is said to be statistically significant.

The model with the lowest AIC offers the best fit.

3)The best model for the data is the one with the lowest BIC value.

I want to explore it more and would be grateful if anyone could guide me to a good book/notes where I can get more details.

Answer 1

A good online source is StatLect: https://www.statlect.com/fundamentals-of-statistics/model-selection-criteria

A couple cautions though as you proceed (to ensure valid estimates):

1. How are you fitting your model to the data? Are you just using least squares? Have you considered maximum likelihood or Bayesian approaches?
2. Do you have a sense of what the errors for your model would look like? Are they symmetric and unbounded (e.g. a Normal distribution?). You can't really apply inference to a numerical technique - you need to model the variability/measurement errors with a distribution of some sort to start to apply inference.
3. Check the residuals of your fit and apply a test for normality and also check if they are serially correlated (e.g., high error followed by high error?)

There's more to assessing a model than calculating a p-value, especially if you really want to be sure you are not being fooled by randomness.