Claude Shannon (1916-2001)

Claude Shannon was born hundred years ago Saturday. Shannon had an incredible career but we know him best for his 1948 paper A Mathematical Theory of Communication that introduced entropy and information theory to the world. Something I didn't know until looking him up: Shannon was the first to define information-theoretic security and show that one-time pads are the one and basically only code that secure.

Entropy has a formal definition, the minimum expected number of bits to represent the output of a distribution. But I view information as a more abstract concept of which entropy is just one substantiation. When you think of concepts like conditional information, mutual information, symmetry of information, the idea of an underlying distribution tends to fade away and you begin to think of information itself as an entity worth mentioning. And when you look at Kolmogorov Complexity, often called algorithmic information theory, the measure is over strings, not distributions, yet has many of the same concepts and relationships in the entropy setting.

Computational Complexity owes much to Shannon's information. We can use information theory to get lower bounds on communication protocols, circuits, even upper bounds on algorithms. Last spring the Simons Institute for the Theory of Computing had a semester program on Information Theory including including a workshop on Information Theory in Complexity Theory and Combinatorics. Beyond theory, relative entropy, or Kullback–Leibler divergence, plays an important role in measuring the effectiveness of machine learning algorithms.

We live in an age of information, growing dramatically every year. How do we store information, how do we transmit, how do we learn from it, how do we keep it secure and private? Let's celebrate the centenary of the man who gave us the framework to study these questions and so much more.

Some short bits from the Gathering for Gardner Conference

I attended G4G12 (Gathering for Gardner) a conference that meets every 2 years (though the gap between the first and second was three years) to celebrate the work of Martin Gardner. Most of the talks were on Recreational mathematics, but there were also some on Magic and some are hard to classify.

Martin Gardner had a column in Scientific American called Mathematical Games from 1956 to 1981. His column inspired man people to go into mathematics. Or perhaps people who liked math read his column. The first theorem I ever read outside of a classroom was in his column. It was, in our terminology, a graph is Eulerian iff every vertex has even degree.

For a joint review of six G4G proceedings see here. For a joint review of six books on recreational math including three of Gardner's, see here. For a review of a book that has serious math based on the math he presented in his column see here.

The talks at G4G are usually 6 minutes long so you can learn about a nice problem and then work on it yourself. Their were a large variety of talks and topics. Many of the talks do not have an accompanying paper. Many of them are not on original material. But none of this matters--- the talks were largely interesting and told me stuff I didn't know.

64=64 and Fibonacci, as Studied by Lewis Caroll, by Stuart Moshowitz. This was about a Lewis Caroll puzzle where he put together shapes in one way to get a rectangle of area 65, and another way to get a square of area 64, The following link is NOT to his talk or a paper of Moshowitz, but it is about the problem: here

How Math can Save your life by Susan Marie Frontczak. This was part talk about bricks and weights and then she stood on the desk and sang this song (thats not her signing it).

Twelve ways to trisect and angle by David Richeson. This was NOT a talk about cranks who thought they had trisected and angle with straightedge and compass. It was about people who used ruler, compass, and JUST ONE MORE THING. I asked David later if the people who trisected the angle before it was shown impossible had a research plan to remove the ONE MORE THING and get the real trisection. He said no- people pretty much knew it was impossible even before the proof.

The Sleeping Beauty Paradox Resolved by Pradeep Mutalik. This paradox would take an entire blog post to explains so here is a pointer to the wikipedia entry on it: here. AH, this one DOES have a paper associated to it, so you can read his resolution here

Larger Golomb Rulers by Tomas Rokicki. A Golomb Ruler is a ruler with marks on it so that the all of the distances between marks are distinct. The number of marks is called the order of the ruler. Construction a Golumb ruler is easy (e.g., marks at the 1,2,4,8,... positions I think works). The real question is to get one of shortest length. They had some new results but, alas, I can't find them on the web.

Chemical Pi by John Conway.  There are people who memorize the first x digits of pi. John Conway does something else. He has memorized the digits of pi and the chemical elements in the following way:

HYDROGEN  3.141592653 HELIUM next 10 digits of pi LITHIUM etc

that is, he memorized the digits of pi by groups of 10 and separated by the chemical elements in the order they are on the Periodic table. He claims this makes it easier to answer questions like: What is the 87th digits of pi. He also claims it gives a natural stopping point for how many digits of pi you need to memorize (need? maybe want). (ADDED LATER WHEN I CORRECTED HELIUM TO HYDROGEN: here are some mnemonic devices:  here.

This post is getting long so I may report on more of the talks in a later post.

The Master Algorithm

We see so few popular science books on computer science, particularly outside of crypto and theory. Pedro Domingos' The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake the World, despite the hyped title and prologue, does a nice job giving the landscape of machine learning algorithms and putting them in a common text from their philosophical underpinnings to the models that they build on, all in a mostly non-technical way. I love the diagram he creates:

Working out from the inner ring are the representations of the models, how we measure goodness, the main tool to optimize the model and the philosophies that drove that model. The book hits on other major ML topics including unsupervised and reinforcement learning.

In the bullseye you can see the "Master Equation" or the Master Algorithm, one learning algorithm to rule them all. The quest for such an algorithm drives the book, and Domingos describes his own, admittedly limited attempts, towards reaching that goal.

I diverge from Domingos in whether we can truly have a single Master Algorithm. What model captures all the inner-ring models above: circuits. A Master Algorithm would find a minimum-sized circuit relative to some measure of goodness. You can do that if P = NP and while we don't think circuit-minimization is NP-hard, it would break cryptography and factor numbers. One of Domingos' arguments states "If we invent an algorithm that can learn to solve satisfiability, it would have a good claim to being the Master Algorithm". Good luck with that.

Its hard to tell if a problem is hard. Is this one hard?

Here is a problem I heard about at the Gathering for Gardner. Is it hard? easy? boring? interesting? I don't know.

Let N={1,2,3,...}

PROBLEM: parameters are s (start point) and f (not sure why to call it f). both are in N

Keep in mind the sequence, in order, of operations:

DIVIDE BY f, SUBTRACT f, ADD f, MULTIPLY by f.

form the following sequence of numbers in N

a(0)= s

Assume a(0),...,a(n) are known. Let A = {a(0),...,a(n)}. N-A are the elements in N that are NOT in A.

If a(n)/f is in N-A then a(n+1)=a(n)/f

Else

If a(n)-f is in N-A then a(n+1)=a(n)-f

Else

If a(n)+f is in N-A then a(n+1)=a(n)+f

Else

If a(n)*f is in N-A then a(n+1) = a(n)*f

Else

If none of the above holds then the sequence terminates.

Lets do an example! Let a=14 and f=2

14, 7, 5, 3, 1, 2, 4, 6, 8, 10, 12, 24, 22, 11, 9, 18, 16, 32, 30, 15, 13, 26, 28, 56, 54, 27, 25, 23, 21,

19, 17, 34, 36, 38, 40, 20,  STOP since 10, 18, 22, 40 are all on the list.

Lets do another example! Let a=7, f=2

7, 5, 3, 1, 2, 4, 6, 8, 10, 12, 14, 16, 18, 9, 11, 13, 15, 17, ... (keeps going)

If f=2 and you get to an odd number x so that ALL of the odds less than x have already appeared but NONE of the odd numbers larger than x have appeared, then the sequence will go forever
with x, x+2, x+4, ...

QUESTIONS and META QUESTIONS

1) Can one characterize for which (s,f) the sequence stops.

2) Is it decidable to determine for which (s,f) the sequence stops.

3) Both (1) and (2) for either fixed s or fixed f.

4) Are the above questions easy?

5) Are the above questions interesting?

There are four categories:

Easy and Interesting- Hmmm, if its TOO easy (which I doubt) then I supposed can't be interesting.

Easy and boring.

Hard and interesting. This means that some progress can be made and perhaps connections to other mathematics.

Hard and Boring. Can't solve and are not enlightened for the effort.

Who Controls Machine Learning?

After AlphaGo's victory, the New York Times ran an article The Race Is On to Control Artificial Intelligence, and Tech’s Future.

A platform, in technology, is essentially a piece of software that other companies build on and that consumers cannot do without. Become the platform and huge profits will follow. Microsoft dominated personal computers because its Windows software became the center of the consumer software world. Google has come to dominate the Internet through its ubiquitous search bar. If true believers in A.I. are correct that this long-promised technology is ready for the mainstream, the company that controls A.I. could steer the tech industry for years to come.

I then tweeted "Can a company control AI? More likely to become a commodity." The major machine learning algorithms are public knowledge and one can find a number of open-source implementations including Google's own TensorFlow that powered AlphaGo. What's to stop a start-up from implementing their own machine learning tools on the cloud?

Some of my readers' comments forced me to rethink my hasty tweet. First, Google, Microsoft and Amazon can create ML infrastructure, cloud hardware that optimizes computational power and storage for machine learning algorithms to get a level of data analysis that one couldn't replicate in software alone.

More importantly, Google etc. have access to huge amounts of data. Cloud companies can provide pretrained machine learning algorithms. Google provides image classification, voice transcription and translation. Microsoft offers face and emotion detection and speech and text analysis. One could imagine, in the absence of privacy issues, Google taking your customer data, matching with data that Google already has on the same customers to draw new inferences about how to market to those customers better.

With almost all our computing heading to the cloud, cloud computing providers will continue to compete, and provide continuing better tools in machine learning and beyond. Eventually will one company "control AI"? That would surprise me but we may still end up with an AI oligarchy.

What Rock Band Name would you choose?

I looked up my colleague Dave Mount on Wikipedia and found  that he was a drummer for the glam rock band Mud. He informed me that (a) on Wikipedia he is David Mount and (b) if he had a rock band it would be named Fried Purple Ellipsoids.

This  set off an email discussion where people said what their rock band name would be. I noticed that many ideas for names had variants. For example, my favorite for Ramsey Theorists: The Red Cliques could be

The Red Cliques

Red Clique

Bill Gasarch and the Red Cliques!

Clique!

So below I list one variant of each name but keep in mind that there are others.

 The Hidden Subgroups

 Amplitudes with Attitude 

 Schrodinger's cat (I wonder if this IS a rock band already)

 The Red Cliques

 Fried purple ellipsoids

 Fried green ellipsoids

 BIG A-G-T

 The Biconnected Sets

 PRAM!

BPP!  (I wonder if any complexity class would work.)

SAT (I wonder if other one-word problems would work. TSP!)

Karp and the reductions

Avi and the derandomizers 

 Aravind and the  Expanders

(Could replace Karp, Avi, and Aravind with others, but these are the first that
came to mind. Plus THE EXPANDERS was Aravind Srinivasan's idea.)

The MIT Logarhythms (This is a real acapella group see here.)

The Discrete Logarhythms

RSA!

The Oracles

The Interactive Proofs

The Natural Proofs

Fried Green Proofs

If we expand to include math we get lots more, so I'll just mention one real one: The Klein Four, an acapella group.

SO- what would YOUR rock band name be?

It's All About the Jobs

In the April CACM Moshe Vardi asks Are We Headed toward Another Global Tech Bust? I agree with some of Vardi’s points, mostly that VC money chasing after unicorns (potential billion-dollar start-ups) will not continue at its heavy pace and we’re already seeing a slow down. But I disagree with Vardi’s assessment that “we should brace ourselves for another global tech and enrollment bust” in computer science. I suspect we’ll see more of a reality check, but that reality looks extremely strong.

Vardi claims that “It is the dream of joining a unicorn that probably attracts many students to study computing”. It’s not just the unicorns bringing students to computer science, but essentially a 100% employment rate for CS graduates looking for a job in the field, many receiving six-figure starting salaries. Few, if any, other disciplines can claim full employment after the bachelor’s degree. Industry is desperate to hire computing professionals in machine learning, cloud computing, cybersecurity, mobile computing, automation, robotics and data science, among others. Not just the computing companies but every industry that deals with data, which is pretty much every industry. Unicorns may become rarer but we won’t see a decline in demand for computer science students until we automate ourselves out of a job.

Take a look at this chart from Ed Lazowska's Where The Jobs Are – 2016 Edition. Those CS jobs won't fill themselves.

Are Perfect Numbers Bigger than Six initial sums of odd cubes (answered)

(NONE of this is my work. In fact some of it is on Wikipedia.)

In my last blog I noticed that

28 = 1³  + 3³

496= 1³ + 3³ + 5³ + 7³

noting that 28 and 496 are the 2nd and 3rd perfect numbers.

I asked if 8128, the next perfect number is also an initial sum of odd cubes. It is!

8128 = 1³ + 3³ + ... + 15³

I also asked if there was something interesting going on .The answer is YES but not that interesting.

All of the math with proofs are  here. I sketch below.

Known Theorem  1: n is an even perfect number iff n is of the form (2^p-1)(2^p- 1) where 2^p-1 is prime.

Known Theorem  2: 1³ + 3³ + 5³ + ... + (2(m-1)+1)³ = m²(2m²-1).

Interesting theorem: if n is an even perfect number larger than 6 and p is the p from Known Theorem 1 then n is the sum of the first  2^(p-1)/2 odd cubes.

Why this is less interesting: The proof does not use that n is perfect. It holds for any number of the form 2^p-1(2^p-1) where p is odd.

So the theorem has nothing to do with perfect numbers. Oh well.

Are perfect numbers bigger than 6 initial sums of odd cubes?

I pose two questions today (Monday April 4).

I will post the answers tomorrow (Tuesday April 5).

Feel free to comment about the answers. If you don't want clues look at the comments.
If I need to clarify something I will do it in the main post So, to reiterate- feel free to leave spoilers but if you want to avoid reading them, don't read the comments.

Note:

The first four perfect numbers are 6, 28, 496, 8128

28 = 1³ + 3³

496 = 1³ + 3³ + 5³ + 7³

Is 8128 the sum of the first six odd cubes? No, and that is not one of my questions.

Questions:

1) Is there a k such that 8128  is the sum of the first k odd cubes?

2) Is there something interesting going on here?

The Machine Learning Turk

Google's AlphaGo took the world by storm when it won its match with Lee Sedol but Demis Hassabis now acknowledges the dark truth. Google wanted to promote its cloud computing division as Amazon AWS and Microsoft Azure have quite the head start. Google needed a killer app that would bring users to Google Cloud and decided they could win if they had the best machine learning tools. They bought Deepmind, run by Hassabis, and needed a showcase event and decided to focus on Go, a game yet to be conquered by computers. Hassabis and his team used clever machine learning techniques on top of Monte Carlo Tree Search but only made mild improvements to the game. Google was growing desperate so a plan was hatched.

Using a modern version of the mechanical turk, an 18th century chess playing automaton that secretly hid a human inside playing the game, Hassabis enlisted Japanese Go player Yuta Iyama to secretly choose the moves for AlphaGo. Iyama, who worked with Google when they agreed to remove Iyama's embarrassing Karaoke videos from YouTube, didn't have to physically be in the machine but relayed the moves by a method Hassabis wouldn't reveal. AlphaGo, secretly getting its moves from Iyama, easily dispatched the European champion in October.

Hannabis and his team wrote up their failed algorithms and found it shockingly easy to fool the Nature editors and reviewers. Yann LeCun of Facebook looked at the Google's team's Nature paper and didn't see that much different from what Facebook had tried. "I just figured Google had chosen better parameters to make their program successful. At the time I should have realized what Google was up to."

Google took a risk challenging Lee Sedol but Sedol, not realizing he was really facing Iyama, played the wrong style of game and lost the match four games to one.

Will this revelation hurt the future of AI? "Machine learning continues to change society, but when it comes to Go," said LeCun, "Alpha fools".