Questions tagged [lagrange-multipliers]
The method of Lagrange multipliers finds critical points (including maxima and minima) of a differentiable function subject to differentiable constraints.
73
questions
1
vote
0
answers
19
views
Derivation of dual formulation of support vector regression
I'm trying to derive the dual formulation of epsilon-insensitive support vector regression. I think my derivation is correct, but I can't match it up to a result for the dual that I've seen given in ...
- 235
4
votes
1
answer
58
views
Likelihood-ratio and score tests of a (non)linear combination of coefficients
The likelihood-ratio and score test are typically used for simple scalar hypotheses such as $\beta_1 = 0$ or $\beta_1 = \beta_2 = 0$. How can we test a linear combination of coefficients using the ...
- 1,824
1
vote
0
answers
36
views
Support Vector Classifiers for Overlapping Classes
I am currently studying support vector classifiers (SVC), more specifically, the solution to the Lagrangian (Wolfe) dual function with the help of the book "The Elements of Statistical Learning&...
- 11
2
votes
2
answers
80
views
How is the Representer theorem used in the derivation of the SVM dual form?
This is the primal form of the SVM hypothesis :
$$
h _{\mathbf{\vec w}, b}(\mathbf{\vec x}^{(i)}) = \mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b
$$
The Representer theorem as formulated here ...
- 115
0
votes
0
answers
11
views
Necessary condition for constrained optimization
Suppose $X=(X_1,\cdots,X_k)$ follows the multinomial distribution with a known size $n$ and an unknown probability vector $(p_1,\cdots,p_k)$.
Find the necessary conditions for the solution to the ...
- 287
0
votes
0
answers
90
views
Estimating the parameter of a Bernoulli distribution using probabilistic modeling and the MAP estimation
Suppose you tossed a coin multiple times. Sometimes you got heads and other times you got tails. You recorded your experiment in a dataset $ X$. Now you want to estimate the parameter θ (which ...
- 272
1
vote
0
answers
41
views
Why is there only a box constraint on alpha and not on mu when solving the dual problem of soft linear SVM?
I am currently learning about the linear SVM in the non-separable case. In the dual representation, we introduce the Lagrange multipliers μk and αk (see also this source: https://...
1
vote
0
answers
21
views
Are solutions to the Lagrangian multipliers ($\alpha_i$) in a hard-margin SVM unique?
An intermediate step in the derivation of the hard-margin SVM's dual form is as follows:
I also know that $a_i$ for all points not on the margin boundary is 0, which makes sense; they must be zeroed ...
1
vote
0
answers
28
views
Min max formulation conversion to max min formulation. Reason?
Question is based on the screenshot attached. Based on paper here.
I am not being able to understand why min max formulation (eq 4) is first converted to max min formulation (eq 5). Is it something ...
- 11
0
votes
0
answers
24
views
Subspace test for multivariate normal distribution [duplicate]
Subspace test for multivariate normal distribution
Suppose $X_1, X_2,\ldots, X_n$ are i.i.d. observations from a multivariate normal distribution $N(\mu,\Sigma)$ where $\Sigma$ is known. Furthermore, ...
- 1
1
vote
0
answers
77
views
Can we convert the optimization of a loss function with regularization to the Lagrangian, constrained optimization *before* solving the optimization?
It is shown here that the optimization of a loss function with regularization,
$$\text{argmin}_b L(X,b) + c ||b||_p \phantom{aaaaaaaaaaaaaaaaaaaaaaaa} (*)$$
is equivalent to the constrained ...
- 483
0
votes
0
answers
31
views
Cannot understand the bound of Lagrangian parameter in SMO
I'm trying to understand SMO, but stuck to the part of bound for Lagrangian parameters.
In the SMO paper(https://www.microsoft.com/en-us/research/uploads/prod/1998/04/sequential-minimal-optimization....
2
votes
1
answer
245
views
How to maximize the ELBO in coordinate ascent variational inference
In the lecture by D.Blei: https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf
Variational inference is explained and he shows how to derive the optimal ...
- 449
0
votes
1
answer
49
views
deriving the optimal distribution
Let the input variable $X \in \mathcal{X}$ and the target variable $Y \in \mathcal{Y}$. For a fixed hypothesis $h \in \mathcal{H}$ I want to solve
\begin{equation}
\min_{p(X,Y)} \int_{\mathcal{X}}\...
- 127
1
vote
1
answer
38
views
incorporating a distribution constraint in a minimisation objective
For a given (convex) hypothesis $h \in \mathcal{H}$, and the variables $X \in \mathcal{X}$ and $Y \in \mathcal{Y}$ I have the following optimisation problem:
\begin{equation}
\min_{p(X,Y)} \int_{\...
- 127