Hinge loss

In machine learning, the hinge loss is a loss function used for training classifiers. The hinge loss is used for "maximum-margin" classification, most notably for support vector machines (SVMs).^[1]

For an intended output t = ±1 and a classifier score y, the hinge loss of the prediction y is defined as

${\displaystyle \ell (y)=\max(0,1-t\cdot y)}${\displaystyle \ell (y)=\max(0,1-t\cdot y)}

Note that ${\displaystyle y}${\displaystyle y} should be the "raw" output of the classifier's decision function, not the predicted class label. For instance, in linear SVMs, ${\displaystyle y=\mathbf {w} \cdot \mathbf {x} +b}${\displaystyle y=\mathbf {w} \cdot \mathbf {x} +b}, where ${\displaystyle (\mathbf {w} ,b)}${\displaystyle (\mathbf {w} ,b)} are the parameters of the hyperplane and ${\displaystyle \mathbf {x} }${\displaystyle \mathbf {x} } is the input variable(s).

When t and y have the same sign (meaning y predicts the right class) and ${\displaystyle |y|\geq 1}${\displaystyle |y|\geq 1}, the hinge loss ${\displaystyle \ell (y)=0}${\displaystyle \ell (y)=0}. When they have opposite signs, ${\displaystyle \ell (y)}${\displaystyle \ell (y)} increases linearly with y, and similarly if ${\displaystyle |y|<1}${\displaystyle |y|<1}, even if it has the same sign (correct prediction, but not by enough margin).

The Hinge loss is not a proper scoring rule.

While binary SVMs are commonly extended to multiclass classification in a one-vs.-all or one-vs.-one fashion,^[2] it is also possible to extend the hinge loss itself for such an end. Several different variations of multiclass hinge loss have been proposed.^[3] For example, Crammer and Singer^[4] defined it for a linear classifier as^[5]

${\displaystyle \ell (y)=\max(0,1+\max _{y\neq t}\mathbf {w} _{y}\mathbf {x} -\mathbf {w} _{t}\mathbf {x} )}${\displaystyle \ell (y)=\max(0,1+\max _{y\neq t}\mathbf {w} _{y}\mathbf {x} -\mathbf {w} _{t}\mathbf {x} )},

where ${\displaystyle t}${\displaystyle t} is the target label, ${\displaystyle \mathbf {w} _{t}}${\displaystyle \mathbf {w} _{t}} and ${\displaystyle \mathbf {w} _{y}}${\displaystyle \mathbf {w} _{y}} are the model parameters.

Weston and Watkins provided a similar definition, but with a sum rather than a max:^{[6] [3]}

${\displaystyle \ell (y)=\sum _{y\neq t}\max(0,1+\mathbf {w} _{y}\mathbf {x} -\mathbf {w} _{t}\mathbf {x} )}${\displaystyle \ell (y)=\sum _{y\neq t}\max(0,1+\mathbf {w} _{y}\mathbf {x} -\mathbf {w} _{t}\mathbf {x} )}.

In structured prediction, the hinge loss can be further extended to structured output spaces. Structured SVMs with margin rescaling use the following variant, where w denotes the SVM's parameters, y the SVM's predictions, φ the joint feature function, and Δ the Hamming loss:

${\displaystyle {\begin{aligned}\ell (\mathbf {y} )&=\max(0,\Delta (\mathbf {y} ,\mathbf {t} )+\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {y} )\rangle -\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {t} )\rangle )\\&=\max(0,\max _{y\in {\mathcal {Y}}}\left(\Delta (\mathbf {y} ,\mathbf {t} )+\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {y} )\rangle \right)-\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {t} )\rangle )\end{aligned}}}${\displaystyle {\begin{aligned}\ell (\mathbf {y} )&=\max(0,\Delta (\mathbf {y} ,\mathbf {t} )+\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {y} )\rangle -\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {t} )\rangle )\\&=\max(0,\max _{y\in {\mathcal {Y}}}\left(\Delta (\mathbf {y} ,\mathbf {t} )+\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {y} )\rangle \right)-\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {t} )\rangle )\end{aligned}}}.

The hinge loss is a convex function, so many of the usual convex optimizers used in machine learning can work with it. It is not differentiable, but has a subgradient with respect to model parameters w of a linear SVM with score function ${\displaystyle y=\mathbf {w} \cdot \mathbf {x} }${\displaystyle y=\mathbf {w} \cdot \mathbf {x} } that is given by

${\displaystyle {\frac {\partial \ell }{\partial w_{i}}}={\begin{cases}-t\cdot x_{i}&{\text{if }}t\cdot y<1,\0円&{\text{otherwise}}.\end{cases}}}${\displaystyle {\frac {\partial \ell }{\partial w_{i}}}={\begin{cases}-t\cdot x_{i}&{\text{if }}t\cdot y<1,\0円&{\text{otherwise}}.\end{cases}}}

However, since the derivative of the hinge loss at ${\displaystyle ty=1}${\displaystyle ty=1} is undefined, smoothed versions may be preferred for optimization, such as Rennie and Srebro's^[7]

${\displaystyle \ell (y)={\begin{cases}{\frac {1}{2}}-ty&{\text{if}}~~ty\leq 0,\\{\frac {1}{2}}(1-ty)^{2}&{\text{if}}~~0<ty<1,\0円&{\text{if}}~~1\leq ty\end{cases}}}${\displaystyle \ell (y)={\begin{cases}{\frac {1}{2}}-ty&{\text{if}}~~ty\leq 0,\\{\frac {1}{2}}(1-ty)^{2}&{\text{if}}~~0<ty<1,\0円&{\text{if}}~~1\leq ty\end{cases}}}

or the quadratically smoothed

${\displaystyle \ell _{\gamma }(y)={\begin{cases}{\frac {1}{2\gamma }}\max(0,1-ty)^{2}&{\text{if}}~~ty\geq 1-\gamma ,\1円-{\frac {\gamma }{2}}-ty&{\text{otherwise}}\end{cases}}}${\displaystyle \ell _{\gamma }(y)={\begin{cases}{\frac {1}{2\gamma }}\max(0,1-ty)^{2}&{\text{if}}~~ty\geq 1-\gamma ,\1円-{\frac {\gamma }{2}}-ty&{\text{otherwise}}\end{cases}}}

suggested by Zhang.^[8] The modified Huber loss ${\displaystyle L}${\displaystyle L} is a special case of this loss function with ${\displaystyle \gamma =2}${\displaystyle \gamma =2}, specifically ${\displaystyle L(t,y)=4\ell _{2}(y)}${\displaystyle L(t,y)=4\ell _{2}(y)}.

• Multivariate adaptive regression spline § Hinge functions

• Loss functions
• Support vector machines

• Articles with short description
• Short description is different from Wikidata