Look Who it Was Random video

Workshop context setting; Phase transitions in distributed by Partha Mitra

522 views · Published 10 January 2018 · 1:00:42 · Indexed 22 September 2026

Channel: International Centre for Theoretical Sciences · 2018 · Science & Technology

Watch on YouTube

Statistical Physics Methods in Machine Learning

DATE: 26 December 2017 to 30 December 2017
VENUE: Ramanujan Lecture Hall, ICTS, Bengaluru
The theme of this Discussion Meeting is the analysis of distributed/networked algorithms in machine learning and theoretical computer science in the "thermodynamic" limit of large number of variables. Methods from statistical physics (eg various mean-field approaches) simplify the performance analysis of these algorithms in the limit of many variables. In particular, phase-transition like phenomena appear where the performance can undergo a discontinuous change as an underlying parameter is continuously varied. A provocative question to be explored at the meeting is whether these methods can shed theoretical light into the workings of deep networks for machine learning.

The Discussion Meeting will aim to facilitate interaction between theoretical computer scientists, statistical physicists, machine learning researchers and mathematicians interested in these questions. Examples of specific topics to be covered include (but are not limited to) problems such as phase transitions in optimization and learning algorithms, matrix approximation, mixing in large networks, sub-linear time algorithms, learning theory and non convex optimization. The meeting will allow structured and and unstructured interactions among the participants around the main theme.

 

*Participation is by invitation only.

CONTACT US: [email protected]

PROGRAM LINK: https://www.icts.res.in/discussion-meeting/SPMML2017

Table of Contents (powered by https://videoken.com)
0:00:00 Start 
0:00:07 Workshop context setting; Phase transitions in distributed
0:02:34 Polynomial Theory of Complex Systems
0:03:37 Google's answer to this is AutoML (learning the network architectures)
0:04:03 Brain Circuits are Nature's Algorithms
0:06:00 Physics strikes back (in ML/ AI)..
0:09:17 Vector spaces/Linear Algebra: 
0:10:06 Linearizion by lifting to a high dimensional space (Koopman linearizion - classical mechanics; SVMs)
0:10:11 Symmetry groups: "Convolutional" networks (ie, builds in translational invariance) learns the filters;
0:11:13 "Thermodynamics of Big Data"
0:14:01 Brains are distributed, noisy networks.
0:14:19 Two case studies (in the large N limit)
0:14:44 In the era of "parts lists", it is important to remember that:
0:18:21 Qualitatively new phenomena appear in the large-N or thermodynamic limit
0:18:30 N -∞
0:19:57 Phase transitions in Statistical Physics
0:20:31 Phase transitions in Theoretical Engineering (Communications, Computation, Controls, Machine Learning)
0:21:37 Communication Networks
0:21:52 Case Study 2:
0:26:42 Intuitive idea : update with an average over the neighboring sites:
0:27:13 Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
0:28:04 Relation between network consensus distributed addition
0:28:33 Consider & = 1 , nearest neighbors on a line
0:29:18 Consensus dynamics with additive noise: Initial variance reduction (blue: t=0 red: t=10 delta=1/2)
0:29:40 Long term increase of variance (black curve; notice "long wavelength" fluctuations dominate)
0:29:52 Variance as a function of time (d=1 consensus)
0:30:11 Higher dimensions = spectral dimension
0:31:35 1st Term
0:32:47 Car platoon with relative position and relative velocity feedback
0:35:11 Q. Would memory in the controller help? (= dynamical feedback, integral control)
0:35:57 Theorem:
0:36:42 How about multiplicative noise?
0:37:08 Exact conservation case
0:39:03 "Average conservation" case
0:39:29 Parameter uncertainty margin for stability in d2 is 0
0:41:13 Case Study 2: Feature Selection in Multivariate Regression
0:45:30 Algorithmic Phase Boundaries Differ:
0:47:22 Phase diagram for Basis Pursuit (Donoho) For uncorrelated Gaussian H: E[H, Hu]=yM6, 6,
0:49:32 There are several approaches to computing the phase transition boundary for Lasso with uncorrelated Gaussian
0:50:01 Phase boundaries and critical exponents for
0:50:24 Phase transition curve for Elastic Net (differs from Basis Pursuit curve)
0:51:53 Theory vs Simulations (Elastic Net)
0:52:21 For the finite noise case (Basis Pursuit), MSE shows a minimum at A ~
0:53:21 Background work: Distribution of singular values for Gaussian with correlated entries
0:53:55 Theoretical Phase Boundary
0:54:09 Combining Consensus and Feature Selection problems ..
0:55:40 Specific Conclusions
0:56:39 Broad Conclusions
0:58:46 Support
0:58:57 Q&A

More from this channel