Hoeffding's lemma

In probability theory, Hoeffding's lemma is an inequality that bounds the moment-generating function of any bounded random variable, implying that such variables are subgaussian. It is named after the Finnish–American mathematical statistician Wassily Hoeffding.

The proof of Hoeffding's lemma uses Taylor's theorem and Jensen's inequality. Hoeffding's lemma is itself used in the proof of Hoeffding's inequality as well as the generalization McDiarmid's inequality.

Statement

Let X be any real-valued random variable such that $a\leq X\leq b$ almost surely, i.e. with probability one. Then, for all $\lambda \in \mathbb {R}$,

$\mathbb {E} \left[e^{\lambda X}\right]\leq \exp {\Big (}\lambda \mathbb {E} [X]+{\frac {\lambda ^{2}(b-a)^{2}}{8}}{\Big )},$

or equivalently,

$\mathbb {E} \left[e^{\lambda (X-\mathbb {E} [X])}\right]\leq \exp {\Big (}{\frac {\lambda ^{2}(b-a)^{2}}{8}}{\Big )}.$

Proof

The following proof is direct but somewhat ad-hoc. Another proof with a slightly worse constant are also available using symmetrization.

Proof

Let $\mu =\mathbb {E} [X]$. Since the conclusion involves $b-a$, without loss of generality, one may replace $X$ by $X-\mu$, $a$ by $a-\mu$, and $b$ by $b-\mu$, which leaves the difference $b-a$ unchanged, and assume $\mathbb {E} [X]=0$, so that $a\leq 0\leq b$.

Since $e^{\lambda x}$ is a convex function of $x$, we have that for all $x\in [a,b]$,

$e^{\lambda x}\leq {\frac {b-x}{b-a}}e^{\lambda a}+{\frac {x-a}{b-a}}e^{\lambda b}$

So,

${\begin{aligned}\mathbb {E} \left[e^{\lambda X}\right]&\leq {\frac {b-\mathbb {E} [X]}{b-a}}e^{\lambda a}+{\frac {\mathbb {E} [X]-a}{b-a}}e^{\lambda b}\\&={\frac {b}{b-a}}e^{\lambda a}+{\frac {-a}{b-a}}e^{\lambda b}\\&=e^{L(\lambda (b-a))},\end{aligned}}$

where $L(h)={\frac {ha}{b-a}}+\ln(1+{\frac {a-e^{h}a}{b-a}})$. By computing derivatives, we find

$L(0)=L'(0)=0$ and $L''(h)=-{\frac {abe^{h}}{(b-ae^{h})^{2}}}$.

From the AMGM inequality we thus see that $L''(h)\leq {\frac {1}{4}}$ for all $h$, and thus, from Taylor's theorem, there is some $0\leq \theta \leq 1$ such that

$L(h)=L(0)+hL'(0)+{\frac {1}{2}}h^{2}L''(h\theta )\leq {\frac {1}{8}}h^{2}.$

Thus, $\mathbb {E} \left[e^{\lambda X}\right]\leq e^{{\frac {1}{8}}\lambda ^{2}(b-a)^{2}}$.

Statement

This statement and proof uses the language of subgaussian variables and exponential tilting, and is less ad-hoc.

Let $X$ be any real-valued random variable such that $a\leq X\leq b$ almost surely, i.e. with probability one. Then it is subgaussian with variance proxy norm $\|X\|_{vp}\leq {\frac {b-a}{2}}$.

Proof

By the definition of variance proxy, it suffices to show that its cumulant generating function $K(t):=\log E[e^{t(X-E[X])}]$ satisfies $K''(t)\leq (b-a)^{2}/4$. Explicit calculation shows $$K''(t)={\frac {E[(X-E[X])^{2}e^{t(X-E[X])}]}{E[e^{t(X-E[X])}]}}-\left({\frac {E[(X-E[X])e^{t(X-E[X])}]}{E[e^{t(X-E[X])}]}}\right)^{2}$$Notice that the quantity ${\frac {E[(X-E[X])e^{t(X-E[X])}]}{E[e^{t(X-E[X])}]}}=E\left[(X-E[X]){\frac {e^{t(X-E[X])}}{E[e^{t(X-E[X])}]}}\right]$ is precisely the expectation of a random variable obtained by exponentially tilting $X-E[X]$. Let this variable be $Y_{t}$. It remains to bound $Var[Y_{t}]\leq (b-a)^{2}/4$.

Notice that $Y_{t}$ still has range $[a-E[X],b-E[X]]$. So translate it to $Z_{t}:=Y_{t}-{\frac {a+b-E[X]}{2}}$ so that its range has midpoint zero. It remains to bound $Var[Z_{t}]\leq (b-a)^{2}/4$. However, now the bound is trivial, since $|Z_{t}|\leq {\frac {b-a}{2}}$.

Given this general case, the formula $\mathbb {E} \left[e^{\lambda (X-\mathbb {E} [X])}\right]\leq e^{\frac {\lambda ^{2}(b-a)^{2}}{8}}$ is a mere corollary of a general property of variance proxy.

See also

• Hoeffding's inequality
• Bennett's inequality