tag:blogger.com,1999:blog-3722233.post5146589104526336803..comments2022-11-27T20:31:33.169-06:00Comments on Computational Complexity: Learning About LearningLance Fortnowhttp://www.blogger.com/profile/06752030912874378610noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-3722233.post-85248923307130912452017-01-06T16:18:29.821-06:002017-01-06T16:18:29.821-06:00Re "how much of an art machine learning still...Re "how much of an art machine learning still is" - a Deep Neural Network has a lot of engineering choices to make, not just gradient descent methods and threshold functions but the large scale architecture of the connections among layers, where there's a CNN, an RNN, an LSTM, etc. In comparison, SVMs, Random Forests, and regression have just a handful of hyper-parameters to tweak.Mitchhttps://www.blogger.com/profile/06352106235527027461noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-35322944953089123012017-01-06T15:08:51.129-06:002017-01-06T15:08:51.129-06:00"Where were all these cool tools when I was a..."Where were all these cool tools when I was a kid?" - When you were a kid, the disgruntled gray hairs were envious of your tools like a keyboard and screen. <br /><br />When 'kids these days' are old they'll be marveling at the youngsters that seem to just twitch their eyelids to pilot their interstellar spacecraft.<br /><br />I'm not saying you have gray hair.Mitchhttps://www.blogger.com/profile/06352106235527027461noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-42777173142937252402017-01-06T12:08:38.333-06:002017-01-06T12:08:38.333-06:00As a follow-up, the above concrete computational c...As a follow-up, the above concrete computational considerations in regard to rank-jumping in tensor network representations are surveyed abstractly in <a href="http://www.scottaaronson.com/blog/?p=3095#comment-1725988" rel="nofollow">"Yellow Book" comment #91</a> on Scott Aaronson's <i>Shtetl Optimized</i> essay "My 116-page survey article on P vs. NP" (of Jan 03 2017).<br /><br />As a followup, I will attempt to compose these two perspectives — abstract-with-concrete and Yellow Book-with-pragmatic — in a <i>MathOverflow</i> and/or <i>TCS StackExchange</i> question regarding "Yellow Book" descriptions of rank-jumping in practical computational simulations. Such attempted compositions — from me or anyone — can rightly be appreciated as tributes to a small-yet-vital community, namely the proprietors of mathematical weblogs.<br /><br />Math weblogs require of their proprietors a sustained personal commitment that (as it seems to me and many) crucially nourishes the vitality of the 21st century's diverse STEAM enterprises. In particular, math weblogs crucially nourish the hopeful enthusiasm of Yellow Book Era STEAM-students — hundreds of millions of 21st century STEAM-students, YIKES! :) — who will inherit and, as we can hope and even reasonably foresee, apply fresh Yellow Book understandings in marvelously extending our 21st century's great STEAM-enterprises. <br /><br />This New Year's appreciation of math weblogs, and heartfelt gratefulness for the sustained efforts of their oft-underappreciated proprietors, is therefore extended.John Sidleshttps://www.blogger.com/profile/16286860374431298556noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-89913104935874971582017-01-06T05:20:11.893-06:002017-01-06T05:20:11.893-06:00------
Lance asks "How many nodes should you...------<br /><b>Lance</b> asks "How many nodes should you have in your network? How many levels? Too many may take too long to train and could cause overfitting. Too few and you don't have enough parameters to create the function you need."<br />------<br />Algorithmic answers to these questions center upon the notion of "rank-jumping" (as at least some portions of the literature call it). <br /><br />Specifically in regard to the rank-jumping literature, a notably student-friendly multi-reference multi-example survey is Vin de Silva and Lek-Heng Lim's "Tensor rank and the ill-posedness of the best low-rank approximation problem" (<i>SIAM Journal on Matrix Analysis and Applications</i>, 2008).<br /><br />The de Silva/Lim survey has been concretely helpful (to me) in upgrading quantum simulation codes that, dynamically and adaptively, raise-and-lower the ranks of tensor representations. Algorithms that once were <i>ad hoc</i>, evolve to be more nearly universal and natural (and stable too). <br /><br />Sweet! Hoorah for "Team Yellow Book"! :)<br /><br />Further suggestions in regard to this "Yellow Book" literature — whether in the language of "rank jumps" or "topological closure" or any other <i>GAGA</i>-esque terminology — would be welcome to me and many. It's been plenty challenging (for me at least) to reduce this literature's beautiful insights to concrete algorithmic practice.<br />John Sidleshttps://www.blogger.com/profile/16286860374431298556noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-82611918315635238932017-01-06T01:41:10.665-06:002017-01-06T01:41:10.665-06:00The idea behind convolution net is as follows: thi...The idea behind convolution net is as follows: think of images and a box. Let's define a feature over the pixels in the box like the existence of a vertical line and have a neural network for it. Now the location of this box doesn't matter for the feature, so if you are looking for vertical lines in an image you can just use the same network for all of them, you can share the weights between networks for the feature. This saves a lot of weights and makes training and inference practical.<br /><br />Many important papers in machine learning are about intelligent ways for saving computation time. You really don't want the number of computation steps to grow superlinearly with respect to network depth, input size, ... that would make training and inference infeasible in practice. CNNs are the reason deep learning worked in practice and beat all previous algorithms in image recognition by a large margin. Machine learning requires a good deal of engineering to have practical algorithms that you can actually run and test, even constant factors matter.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-43069924497994512412017-01-06T01:12:46.103-06:002017-01-06T01:12:46.103-06:00https://arxiv.org/abs/1611.01578
https://arxiv.or...https://arxiv.org/abs/1611.01578<br /><br />https://arxiv.org/abs/1505.00521<br /><br />https://media.nips.cc/Conferences/2015/tutorialslides/wood-nips-probabilistic-programming-tutorial-2015.pdfAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-39317540753339580962017-01-05T22:21:51.586-06:002017-01-05T22:21:51.586-06:00Have you tried TF's playground?Have you tried <a href="http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.20483&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false" rel="nofollow">TF's playground</a>?Yuvalhttp://yuvalpinter.comnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1563602768827240062017-01-05T14:00:38.165-06:002017-01-05T14:00:38.165-06:00"Convolution nets has a special first layer t..."Convolution nets has a special first layer that captures features of pieces of the image."<br /><br />This is not correct. Convolutional neural networks have many convolutional layers, anywhere, not necessarily at the first layer. These layers exploit locality and translation invariance, two important properties of image-like data. Here is a stylized example showing how convolutional neurons can recognize higher-and-higher level abstractions, from edges through noses to faces:<br /><br />https://i.stack.imgur.com/Hl2H6.pngDániel Vargahttps://www.blogger.com/profile/10626372642275036283noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-70954635466084591932017-01-05T10:55:05.605-06:002017-01-05T10:55:05.605-06:00Small correction: recurrent nets represent time de...Small correction: recurrent nets represent time dependencies (or more generally, dependencies along any DAG), not feedback loops. Each computation of a recurrent net can be unfolded into a feed-forward net of depth O(input size).Fernando Pereirahttps://www.blogger.com/profile/05849361902113771573noreply@blogger.com