Fineline: Created page with " == SCHEDULE == === 7/6/17 - Talk and Discussion: Steve Young - Boltzmann Machines and Statistical Mechanics. === PREREADINGS: [https://libgen.unblocked.srl/book/index.php?..."

2017-07-07T22:27:57Z

Created page with " == SCHEDULE == === 7/6/17 - Talk and Discussion: Steve Young - Boltzmann Machines and Statistical Mechanics. === PREREADINGS: [https://libgen.unblocked.srl/book/index.php?..."

New page

== SCHEDULE ==
=== 7/6/17 - Talk and Discussion: Steve Young - Boltzmann Machines and Statistical Mechanics. ===

PREREADINGS:

[https://libgen.unblocked.srl/book/index.php?md5=E70CC484C7FF51073859B15779162C25 MacKay - Information Theory, Inference, and Learning Algorithms] Chapter 43 on the Boltzmann machine. Chapter 42 on Hopfield networks.

[[Media:hinton_lect11.pdf]] [[Media:hinton_lect12.pdf]] Lecture notes from Hinton's Coursera class. Good overview of Boltzmann machines and Hopfield nets. You can sign up for the free course and watch the accompanying videos [https://www.coursera.org/learn/neural-networks here]. They're also on [https://www.youtube.com/playlist?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9 Youtube].

== What ==

nBDSM is the noiseBridge Deepnet and Statistical Mechanics working group. We meet weekly to learn, teach, and discuss topics at the intersection of AI/deep learning and statistical mechanics. Note that we have a non-trivial overlap with The One, The Only [https://www.noisebridge.net/wiki/DreamTeam Noisebridge DreamTeam].

We're focused on theory. Implementation is fun too, but has its own set of (mostly orthogonal) skills that we'll cover only lightly.

== Prerequisites ==

Our discussions are at upper division to grad level in machine learning and statistical mechanics. To be able to get something out of them, you should know

*linear algebra (at the level of [https://libgen.unblocked.srl/book/index.php?md5=79185948E755CC9B9611346E586CD050 D. Lay's book])
*single and multi-variable calculus, vector calculus, Lagrange multiplers, Taylor expansions (all of Stewart's textbook).
*basics of statistics, including bayesian
*statistical mechanics (at the level of [http://mcgreevy.physics.ucsd.edu/s12/index.html McGreevy's MIT lecture notes])

There are plenty of other places to learn this stuff. Eg you can review your
probability, stats and linear algebra from chapters 2 and 3 of [https://libgen.unblocked.srl/book/index.php?md5=E4B2AB0EF22458F94C835D4D2397034E Goodfellow].

== Links ==
Check out these cool links

* A great [http://videolectures.net/deeplearning2016_ganguli_theoretical_neuroscience/ talk] by Ganguli at [https://sites.google.com/site/deeplearningsummerschool2016/ last year's deep learning summer school] in Montreal.
* Anything recent by [https://arxiv.org/find/stat/1/au:+Ganguli_S/0/1/0/all/0/1 Ganguli] at the [https://ganguli-gang.stanford.edu/ Neural Dynamics and Computation Lab] as well.
*[https://calculatedcontent.com/ Calculated Content]
* The venerable [http://colah.github.io/ colah's blog]
* Stat Mech//Machine Learning conference 2017 at Berkeley: [https://smml.io/ smml:2017]
* [http://www.lps.ens.fr/~krzakala/LESHOUCHES2013/home.htm Les Houches] 2013 school on Statistical physics, Optimization, Inference and Message-Passing algorithms. Contains links to papers/talks.
* [https://www.youtube.com/playlist?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9 Videos] from Geoff Hinton's neural net course on Coursera.
* A [http://bactra.org/weblog/361.html blog] post about exponential families that demonstrates the sort of intuition we're trying to build.
* [[Media:maxEntChap10.pdf]] Intro to principle of Maximum Entropy

== Papers ==

=== High Level Overviews ===

Good large scale overview of why the stat mech side is important
* Advani et al. - Stat mech of complex neural systems and high dimensional data - [https://arxiv.org/abs/1301.7115v1 arXiv:1301.7115v1]

Less emphasis on the physics, more emphasis on the stat mech <-> statistical inference
connection.
* Mastromatteo - On the typical properties of inverse problems in stat mech - [https://arxiv.org/abs/1311.0190v1 arXiv:1311.0910v1]

=== Interesting papers ===

* Chen et al. - On the Equivalence of Restricted Boltzmann Machines and Tensor Network States - [https://arxiv.org/abs/1701.04831v1 arXiv:1701:04831v1]
* Mehta et al. - An exact mapping between the Variational Renormalization Group and Deep Learning - [https://arxiv.org/abs/1410.3831 arXiv:1410.3831]
* Saxe et al. - Exact solutions to the nonlinear dynamics of learning in deep linear neural networks - [https://arxiv.org/abs/1312.6120 arXiv:1312.6120]

== Books ==
* A great place to find books and articles is [https://www.reddit.com/r/Scholar/comments/3bs1rm/meta_the_libgenscihub_thread_howtos_updates_and/ Library Genesis]. I use these links for books and articles: [https://libgen.unblocked.srl/], [https://libgen.unblocked.li/scimag].
* [https://libgen.unblocked.srl/book/index.php?md5=836019C9EC85D594F380D1D898B59713 Huang's] text is the bronze standard for grad level stat mech.
* [https://libgen.unblocked.srl/book/index.php?md5=05528E3012BCB03B7E0142343E58D0C6 Chandler's] text is supposedly great for stat mech, although I haven't read it.
* [https://libgen.unblocked.srl/book/index.php?md5=AFEE8F3BEF62DC5C7A191451259AD8EB Engel - Statistical Mechanics of Learning] I haven't looked at this yet, but it seems promising.
* [http://www.inference.org.uk/itila/p0.html MacKay - Information Theory, Inference, and Learning Algorithms] Link [https://libgen.unblocked.srl/book/index.php?md5=E70CC484C7FF51073859B15779162C25 here]
* [https://libgen.unblocked.srl/book/index.php?md5=E4B2AB0EF22458F94C835D4D2397034E Goodfellow et al. - Deep Learning]
*[https://libgen.unblocked.srl/book/index.php?md5=3E76F8F5189A047550CA9020D97848E4 Mezard - Information, Physics, and Computation]

Deepnet - Revision history

Fineline: Created page with " == SCHEDULE == === 7/6/17 - Talk and Discussion: Steve Young - Boltzmann Machines and Statistical Mechanics. === PREREADINGS: [https://libgen.unblocked.srl/book/index.php?..."