A model can be classified as parametric or non-parametric. How are models classified as parametric and non-parametric models? What is the difference between the two approaches?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Parametric Methods
A parametric approach (Regression, Linear Support Vector Machines) has a fixed number of parameters and it makes a lot of assumptions about the data. This is because they are used for known data distributions, i.e., it makes a lot of presumptions about the data.
Non-Parametric Methods
A non-parametric approach (k-Nearest Neighbours, Decision Trees) has a flexible number of parameters, there are no presumptions about the data distribution. The model tries to "explore" the distribution and thus has a flexible number of parameters.
Comparision
Comparatively speaking, parametric approaches are computationally faster and have more statistical power when compared to non-parametric methods.
answered Sep 27, 2020 at 2:27
$\begingroup$
$\endgroup$
5
I provided some details but the most important excerpt is from Stuart Russell and Peter Norvig's AIMA book:
A learning model that summarizes data with a set of parameters of fixed size (independent of the number of training examples) is called a parametric model. No matter how much data you throw at a parametric model, it won’t change its mind about how many parameters it needs.
For nonparametric models ask yourself a question: What is the number of parameters of the decision tree?
As the decision tree is an example of nonparametric model the number of parameters in a decision tree depends on the quantity of the data. The more data we have means more parameters.
-
1$\begingroup$ What are the parameters of a decision tree? $\endgroup$– ado sarCommented Oct 29, 2023 at 14:30
-
$\begingroup$ Criterion such as Gini, and max deep are consider as hyper parameters, DT do not have weights and biases in the same way as neural networks or other linear models and DT are considered as non parametric models. $\endgroup$– prostiCommented Oct 31, 2023 at 11:53
-
$\begingroup$ I didn't ask about hyperparameters, I asked about "parameters". The distinction between parametric and non-parametric models boils down to whether the number of parameters varies with data. So, in a decision tree what qualifies as a parameter? If we don't have a definition of what is a "parameter" in a decision tree, then statements like "the number of parameters in a decision tree vary/grow with data" are meaningless. The parameters of a decision tree could be the regions of the tree and these indeed can vary with different amount of data. $\endgroup$– ado sarCommented Oct 31, 2023 at 17:01
-
$\begingroup$ So you would like to say that DT is a parametric model. LOL. $\endgroup$– prostiCommented Oct 31, 2023 at 19:02
-
$\begingroup$ Apologize, I realized that my second comment does not make sense and I will remove it. Nevertheless, is there any definition of what a parameter, not hyperparameter is in a decision tree? $\endgroup$– ado sarCommented Oct 31, 2023 at 19:28