Lecture 18: Chernoff Bounds

The comment at the end about the moment bounds being at least as good as the Chernoff-type bounds is actually a really simple idea. Say we’re working with non-negative random variables. Recall the Chernoff uses something like

\displaystyle  \Pr[ X \geq \lambda ] \leq \inf_{t \geq 0} \frac{ \mathop{\mathbb E}[e^{tX}] }{ e^{t\lambda} }

We could have instead used the “best” moment bound

\displaystyle  \Pr[ X \geq \lambda ] \leq \min_{k \geq 0} \frac{ \mathop{\mathbb E}[X^k] }{ \lambda^k }

The claim (from this 1995 paper by Philips and Nelson) is

\displaystyle  \inf_{t \geq 0} \frac{ \mathop{\mathbb E}[e^{tX}] }{   e^{t\lambda} } \geq \min_{k \geq 0} \frac{ \mathop{\mathbb E}[X^k] }{   \lambda^k }

This means whatever bound we could prove by taking the moment-generating function (on the left), it will never be better than the best moment bound.

Why is the claim true? Suppose the RHS minimum is {\alpha}. Then {\mathop{\mathbb E}[X^k] \geq \alpha \lambda^k}, for all {k \geq 0}. And hence, for any {t > 0},

\displaystyle  \begin{array}{rl}    \mathop{\mathbb E}[e^{tX}] &= \sum_{k \geq 0} \frac{\mathop{\mathbb E}[ (tX)^k ]}{k!} \\   &= \sum_{k \geq 0} \frac{t^k \mathop{\mathbb E}[ X^k ]}{k!} \\   &\geq \sum_{k \geq 0} \frac{ t^k (\alpha \lambda^k)}{k!} = \alpha   e^{t\lambda}.    \end{array}

Which means the LHS infimum is also at least {\alpha}. Easy! The paper extends this to general random variables (using a simple conditioning trick), and also gives other results, including examples where the moment bound can be much better.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s