Lecture 21: SVDs

A few things about today’s lecture:

  • This one is kinda important, I meant to discuss it but forgot. We defined {v_1} as the best-fit one-dimensional subspace, then {v_2} subject to having fixed {v_1}. This greedy approach is nice, but one worry is that maybe if we optimized over two dimensional spaces, we’d get a better fit than the space spanned by {v_1, v_2}.

    Thankfully not: the best {2}-dimensional subspace (i.e., the one that maximizes the sum of squared projections of the {a_i}s onto it) is indeed spanned by {v_1, v_2}.

    Why? Suppose there was another {2}-dimensional subspace {W}. Pick an orthonormal basis {w_1, w_2} for {W} such that {w_2 \perp v_1}. (Clearly such a basis exists.) Now, we know that {\|Aw_1\| \leq \|Av_1\|}, by our greedy choice of {v_1}. Also, {\|Aw_2\| \leq \|Av_2\|}, by our greedy choice of {v_2} (we could have chosen {w_2 \perp v_1} but chose {v_2} instead). So

    \displaystyle  \|Aw_1\|^2 + \|Aw_2\|^2 \leq \|Av_1\|^2 + \|Av_2\|^2

    and hence the sum of squared projections onto {W} are no longer than those onto span{(v_1, v_2)}. A similar proof works to show that the space spanned by {\{v_1, \ldots, v_k\}} is the best-fit {k}-dimensional subspace.

  • For a square symmetric matrix {A = V\, diag(\lambda_1, \ldots, \lambda_n)\, V^T}, the definition of {f(A)} is indeed for all functions {f: {\mathbb R} \rightarrow {\mathbb R}}:

    \displaystyle  f(A) = V \, diag(f(\lambda_1), \ldots, f(\lambda_n)) \, V^T.

    It may not be interesting for some functions {f}, of course.

  • Melanie pointed out that if we want to find the best-fit affine subspace, we should imagine the origin to be at the center-of-gravity of the points in {A}, and find the best-fit linear subspace through that. A proof is given in her thesis, for example.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s