Continuous Bernoulli distribution

In probability theory, statistics, and machine learning, the continuous Bernoulli distribution^{[1] [2] [3]} is a family of continuous probability distributions parameterized by a single shape parameter ${\displaystyle \lambda \in (0,1)}${\displaystyle \lambda \in (0,1)}, defined on the unit interval ${\displaystyle x\in [0,1]}${\displaystyle x\in [0,1]}, by:

${\displaystyle p(x|\lambda )\propto \lambda ^{x}(1-\lambda )^{1-x}.}${\displaystyle p(x|\lambda )\propto \lambda ^{x}(1-\lambda )^{1-x}.}

The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders,^{[4] [5]} for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, ${\displaystyle [0,1]}${\displaystyle [0,1]}-valued data.^{[6] [7] [8] [9]} This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, ${\displaystyle \{0,1\}}${\displaystyle \{0,1\}}-valued data.

The continuous Bernoulli also defines an exponential family of distributions. Writing ${\displaystyle \eta =\log \left(\lambda /(1-\lambda )\right)}${\displaystyle \eta =\log \left(\lambda /(1-\lambda )\right)} for the natural parameter, the density can be rewritten in canonical form: ${\displaystyle p(x|\eta )\propto \exp(\eta x)}${\displaystyle p(x|\eta )\propto \exp(\eta x)}.

Given a sample of ${\displaystyle N}${\displaystyle N} points ${\displaystyle x_{1},\dots ,x_{n}}${\displaystyle x_{1},\dots ,x_{n}} with ${\displaystyle x_{i}\in [0,1],円\forall i}${\displaystyle x_{i}\in [0,1],円\forall i}, the maximum likelihood estimator of ${\displaystyle \lambda }${\displaystyle \lambda } is the empirical mean,

${\displaystyle {\hat {\lambda }}={\bar {x}}={\frac {1}{N}}\sum _{i=1}^{n}x_{i}.}${\displaystyle {\hat {\lambda }}={\bar {x}}={\frac {1}{N}}\sum _{i=1}^{n}x_{i}.}

Equivalently, the estimator for the natural parameter ${\displaystyle \eta }${\displaystyle \eta } is the logit of ${\displaystyle {\bar {x}}}${\displaystyle {\bar {x}}},

${\displaystyle {\hat {\eta }}={\text{logit}}({\bar {x}})=\log({\bar {x}}/(1-{\bar {x}})).}${\displaystyle {\hat {\eta }}={\text{logit}}({\bar {x}})=\log({\bar {x}}/(1-{\bar {x}})).}

The continuous Bernoulli can be thought of as a continuous relaxation of the Bernoulli distribution, which is defined on the discrete set ${\displaystyle \{0,1\}}${\displaystyle \{0,1\}} by the probability mass function:

${\displaystyle p(x)=p^{x}(1-p)^{1-x},}${\displaystyle p(x)=p^{x}(1-p)^{1-x},}

where ${\displaystyle p}${\displaystyle p} is a scalar parameter between 0 and 1. Applying this same functional form on the continuous interval ${\displaystyle [0,1]}${\displaystyle [0,1]} results in the continuous Bernoulli probability density function, up to a normalizing constant.

The Beta distribution has the density function:

${\displaystyle p(x)\propto x^{\alpha -1}(1-x)^{\beta -1},}${\displaystyle p(x)\propto x^{\alpha -1}(1-x)^{\beta -1},}

which can be re-written as:

${\displaystyle p(x)\propto x_{1}^{\alpha _{1}-1}x_{2}^{\alpha _{2}-1},}${\displaystyle p(x)\propto x_{1}^{\alpha _{1}-1}x_{2}^{\alpha _{2}-1},}

where ${\displaystyle \alpha _{1},\alpha _{2}}${\displaystyle \alpha _{1},\alpha _{2}} are positive scalar parameters, and ${\displaystyle (x_{1},x_{2})}${\displaystyle (x_{1},x_{2})} represents an arbitrary point inside the 1-simplex, ${\displaystyle \Delta ^{1}=\{(x_{1},x_{2}):x_{1}>0,x_{2}>0,x_{1}+x_{2}=1\}}${\displaystyle \Delta ^{1}=\{(x_{1},x_{2}):x_{1}>0,x_{2}>0,x_{1}+x_{2}=1\}}. Switching the role of the parameter and the argument in this density function, we obtain:

${\displaystyle p(x)\propto \alpha _{1}^{x_{1}}\alpha _{2}^{x_{2}}.}${\displaystyle p(x)\propto \alpha _{1}^{x_{1}}\alpha _{2}^{x_{2}}.}

This family is only identifiable up to the linear constraint ${\displaystyle \alpha _{1}+\alpha _{2}=1}${\displaystyle \alpha _{1}+\alpha _{2}=1}, whence we obtain:

${\displaystyle p(x)\propto \lambda ^{x_{1}}(1-\lambda )^{x_{2}},}${\displaystyle p(x)\propto \lambda ^{x_{1}}(1-\lambda )^{x_{2}},}

corresponding exactly to the continuous Bernoulli density.

An exponential distribution restricted to the unit interval is equivalent to a continuous Bernoulli distribution with appropriate^{[which? ]} parameter.

The multivariate generalization of the continuous Bernoulli is called the continuous-categorical.^[10]

• Continuous distributions
• Exponential family distributions

• Webarchive template wayback links
• Articles with short description
• Short description matches Wikidata
• All articles with specifically marked weasel-worded phrases
• Articles with specifically marked weasel-worded phrases from November 2022