Questions tagged [regularization]
Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
1,428
questions
0
votes
0
answers
8
views
Reducing Variance with Regularization in LOOCV for Small Datasets
I have a small dataset and I am considering using Leave-One-Out Cross-Validation (LOOCV) to evaluate my model. I understand that cross-validation, in general, is a method to assess a model's ...
- 1
0
votes
1
answer
21
views
Would it be possible to use regularization methods as a feature selection method and then use machine learning models to analyses data?
My data is RNA-seq data with more than 14000 features and the problem is binary classification. Then the total sample is 50 and p>>n. When I use Elasticnet method with train and test data, the ...
0
votes
0
answers
15
views
difference between l2 penalty and l2 loss in SAE
I was reading this paper from Anthropic https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html and in the paper loss is defined like this :$$
L = \mathbb{E}_x \left[ \| x - \hat{x} \|...
- 3
1
vote
0
answers
22
views
Multi-task learning-Loss function
0
I am training a convolutional autoencoder with two velocity fields (2D array) as inputs and outputs. These fields represent wind velocities in both the x and y directions within a square domain. My ...
- 11
1
vote
0
answers
31
views
Ask a coding problem for the equivalence of unconstrained Optimization with L1 Regularization
I recently read a statistics paper: DAGs with NO TEARS: Continuous Optimization for Structure Learning
It has an unconstrained problem:
$$\min_\theta F(\theta)+\lambda || \theta||_1$$, where $$F(\...
- 21
0
votes
1
answer
46
views
why does LASSO regression return unstandardized coefficients [closed]
I have more general questions that does not refer to a coding issue.
Why does LASSO regression require standardization of the predictors but return unstandardized coefficients (glmnet function - https:...
- 1
1
vote
0
answers
14
views
penalized package [closed]
Has anyone used penalized package? I was using it for lasso in Cox regression, with time-varying coefficients.
The problem is when I made a plot with ...
- 11
-1
votes
1
answer
23
views
How does a neural network differentiate between a neuron that outputs 0 and a dropped-out one?
How does a network differentiate between a neuron with output 0 and a dropped-out neuron (this neuron might output a non-zero value but due to dropout it outputs 0)?
- 477
3
votes
1
answer
37
views
What is the boundary curve for $λ_1$ and $λ_2$ that give at least a 0 component in elastic net?
Define the elastic net estimate:
$
\hat{\beta}^{\lambda_1, \lambda_2} = \arg \min_{\beta \in \mathbb{R}^p} \left( \frac{1}{2n} \| y - X\beta \|_2^2 + \lambda_1 \ \frac{1}{2} \|\beta \|_2^2 + \lambda_2 ...
- 311
1
vote
0
answers
37
views
how to approximate the eigendecomposition of a correlation matrix when the data have been standardized?
Context
I am working to develop a penalized regression framework that will scale up to analyzing high dimensional data with a certain correlation structure. Let $X$ represent an $n \times p$ matrix of ...
0
votes
0
answers
34
views
Is there any test I can apply to the data to tell whether the adaptive LASSO or the LASSO is likely to perform better in prediction?
Is there a. test I can perform on a sample that will tell me if coefficients estimated using the LASSO, the adaptive LASSO, or the relaxed adaptive LASSO are likely to give better (in the mean squared ...
- 3,157
0
votes
0
answers
22
views
Confidence intervals ODE ridge regression
i want to find confidence intervals for a least squares loss which is L2 regularized. I have only found something for linear problems, but in my case i want to estimate ordinary differential equation ...
- 1
6
votes
1
answer
114
views
Valid confidence intervals in GAM’s using shrinkage estimation
In this blog article:
https://www.fharrell.com/post/improve-research/
it states:
“The frequentist paradigm does not provide confidence intervals or p-values when parameters are penalized”.
I was ...
- 859
0
votes
0
answers
21
views
Testing difference between two models using WAIC and degrees of freedom of WAIC
I am conducting Bayesian penalised regression, horseshoe specifically, in R using the bayesreg package see here. One model is nested within the other, i.e. to the second model I have simply used all ...
- 2,043
3
votes
0
answers
29
views
Is there a likelihood penalization or (im)proper prior to remove estimation bias for gamma parameters?
So I am learning that maximum likelihood estimation of the parameters for a gamma distribution are biased. As far as I understand there is no guarantee in general that there exists a prior (or base ...
- 9,401