CSC 699 Class Schedule

Details in sections 1.2.6 are less critical, but the meaning of equation 1.68 is important. Details of 1.5.5 and 1.6 are less critical, but try to get some sense of the main information theory concepts if you don't already know them.

Introduction to probability

Selected solutions

Details in section 2.3 are quite technical. You will need to use some judgment about what to study depending on your background. However, try to get some of the main points regardless. The Robbins-Monro method is not so important for a first cut on this material. The same goes for section 2.3.8 which can be skipped if you like.

Selected solutions
(compiled by Kate, Joe, and Kobus)

This is the first chapter that goes well beyond "review", and so we will break it into two pieces.

Solutions
Provided/compiled by Lennie and Prasad.

Solutions
Compiled by Anarug and Ernesto

Appendix E covers the method of Lagrange multipliers. This is an important optimization technique that has, among its many applications, maximizing the likelihood in the derivation of most EM equations. I found Bishop's treatment very clear.

The key points for this week are:

• A general approach (EM) to missing variable problems. The missing variables are denoted Z in the book. If you are clustering, what is "missing" variable is which cluster is responsible for each point.
• The specific example of Gaussian mixture model (GMM), that is useful, and serves as the canonical example of EM.
• An important approach to clustering (K-means / GMM).

Possible difficulties:

• The index notation (e.g. the exponent z_k) in equations starting on page 431. This is a notational trick. Because z_k has only one instance that is 1, the others being zero, when used as an exponent of terms in a product, it "selects" the appropriate factor.
• The Q function. Sometimes this creates confusion. The Q function is never evaluated. It is just what you consider maximizing when you derive the EM equations. The maximal value is what you actually compute (M-Step).

Link to supplementary file with two probability computation tricks.

The first may be useful for the assignment. The second is useful if you ever want to do large scale EM computations.

Solutions
Compiled by Siou and John.

The rest of the chapter is useful and interesting, and time spent skimming it is not wasted, but our goal for this week is understanding the basic HMM.

Note: Equation 13.68 looks like it has a typo. (As do 13.65 and 13.70).

Solutions
Compiled by Igor and Rui

Link to a nice alternative treatment.

The key points for this week are:

• Sampling probability distributions in general (Section 11.1.1).
• Using a proposal distribution to help sample difficult distributions to sample, and general difficulties with sampling due to high dimensionality and other factors (Section 11.1.2-11.1.4).
• MCMC (e.g. Metropolis Hastings)---this is a very important tool!
• Gibbs sampling.

Solutions

From a technical perspective, we will focus on the material through 12.2. Furthermore, some of the material on probabilistic PCA is more technical than desired for a first exposure. Nonetheless, try to get the main points. In particular, make sure you understand Figures 12.9 and 12.12. If you are relatively new to PCA, most of your energy should be spent on the pages up to and including 572.

If you are already familiar with kernel methods, 12.3 may be of interest; otherwise wait until after the break. ICA (covered in 12.4) is of interest, but the coverage is very brief. 12.4.3 covers important ideas, but does not go into technical depth. Worth a skim if you have time.

Section 3.5 will be omitted from the presented part; however you might find it interesting.
I did not find section 3.1.5 very interesting. The result is pretty obvious, and I have never been tempted to fit the same model to data that is different enough to be represented by different dimension. Perhaps I just don't it?

Section 3.3.3 may seem a bit mysterious at first if you have never seen it before, but it is well worth the effort to understand.

You should try to understand most of the figures in the assigned sections. I think they provide a lot of intuition on the topic.

Solutions

We will focus our attention on the material up to and including page 206. We will pay some attention to 4.3.4 and 4.3.5 which should be skimmed. The rest of the chapter is for keeners only.

Solutions

We will focus our attention on 5.1, 5.2, 5.3, 5.5 up to 5.5.3 (inclusive), 5.5.6, and 5.6. (Alternatively: Skip 5.4, 5.5.4, 5.5.5, and 5.7).

Solutions

We will focus on 6.1 and 6.2, and 6.4 up to 6.4.2 inclusive.

Late breaking suggestion: Have a look at 12.3.

No solutions due to special informal format for this week.

Read this also:

Finally, you might find some of the issues better addressed in this supplemetary tutorial.