NBDSM

Schedule

7/6/17 - Talk and Discussion: Steve Young - Boltzmann Machines and Statistical Mechanics.

PREREADINGS:

MacKay - Information Theory, Inference, and Learning Algorithms Chapter 43 on the Boltzmann machine. Chapter 42 on Hopfield networks.

Media:hinton_lect11.pdf Media:hinton_lect12.pdf Lecture notes from Hinton's Coursera class. Good overview of Boltzmann machines and Hopfield nets. You can sign up for the free course and watch the accompanying videos here. They're also on Youtube.

What

nBDSM is the noiseBridge Deepnet and Statistical Mechanics working group. We meet weekly to learn, teach, and discuss topics at the intersection of AI/deep learning and statistical mechanics. Note that we have a non-trivial overlap with The One, The Only Noisebridge DreamTeam.

We're focused on theory. Implementation is fun too, but has its own set of (mostly orthogonal) skills that we'll cover only lightly.

Prerequisites

Our discussions are at upper division to grad level in machine learning and statistical mechanics. To be able to get something out of them, you should know

• linear algebra (at the level of D. Lay's book)
• single and multi-variable calculus, vector calculus, Lagrange multiplers, Taylor expansions (all of Stewart's textbook).
• basics of statistics, including bayesian
• statistical mechanics (at the level of McGreevy's MIT lecture notes)

There are plenty of other places to learn this stuff. Eg you can review your probability, stats and linear algebra from chapters 2 and 3 of Goodfellow.

Links

Check out these cool links

• A great talk by Ganguli at last year's deep learning summer school in Montreal.
• Anything recent by Ganguli at the Neural Dynamics and Computation Lab as well.
• Calculated Content
• The venerable colah's blog
• Stat Mech//Machine Learning conference 2017 at Berkeley: smml:2017
• Les Houches 2013 school on Statistical physics, Optimization, Inference and Message-Passing algorithms. Fairly advanced. Proceedings
• Videos from Geoff Hinton's neural net course on Coursera.
• A blog post about exponential families that demonstrates the sort of intuition we're trying to build.
• Media:maxEntChap10.pdf Intro to principle of Maximum Entropy

Papers

High Level Overviews

Good large scale overview of why the stat mech side is important

• Advani et al. - Stat mech of complex neural systems and high dimensional data - arXiv:1301.7115v1

Less emphasis on the physics, more emphasis on the stat mech <-> statistical inference connection.

• Mastromatteo - On the typical properties of inverse problems in stat mech - arXiv:1311.0910v1

Interesting papers

• Chen et al. - On the Equivalence of Restricted Boltzmann Machines and Tensor Network States - arXiv:1701:04831v1
• Mehta et al. - An exact mapping between the Variational Renormalization Group and Deep Learning - arXiv:1410.3831
• Saxe et al. - Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - arXiv:1312.6120

Books

• A great place to find books and articles is Library Genesis. I use these links for books and articles: [1], [2].
• Huang's text is the bronze standard for grad level stat mech.
• Chandler's text is supposedly great for stat mech, although I haven't read it.
• Engel - Statistical Mechanics of Learning I haven't looked at this yet, but it seems promising.
• MacKay - Information Theory, Inference, and Learning Algorithms Link here
• Bishop - Pattern Recognition and Machine Learning
• Goodfellow et al. - Deep Learning
• Mezard - Information, Physics, and Computation
• Jaynes - Probability Theory, The Logic of Science An excellent introduction to Bayesian reasoning and probability theory in general, from a very pedagogical, opinionated point-of-view. Dives into the motivations behind some information theory as well.

Ideas for future talks

Here's some ideas for future talks. If you want to present one of these,

A) Feel free to be advanced as you like -- assume an audience of graduate students.

but

B) Don't feel pressured to go any faster than you want. If you think you can give a pedagogical 'for dummies' talk in the course of an hour and a half, go for it!

• Derive capacity of Hopfield net and understand this limitation intuitively
• Explain similarity/relationship/identity of Bayesian inference and maximum entropy formalism.
• Deep intuitive dive on Lagrangian duals and what they really do/mean in the context of statistical inference/machine learning/stat mech