Friday, January 26, 2018

From Art to Science

Q: Why did the neural net cross the road?
A: Who cares along as it got to the other side.

Whit Diffie and Martin Hellman in their classic 1976 paper talk about the transition of cryptography:
Theoretical developments in information theory and computer science show promise of providing provably secure cryptosystems, changing this ancient art into a science.
Indeed we have some very strong theoretical foundations for cryptography, though they still need to rely on unproven hardness assumptions. Often we trade off formal proven techniques for efficiency, in for example AES and Bitcoin. Here we use "science" in another sense: extensive testing of the hardness of these protocols. Creating and breaking cryptographic protocols used to be an art, having the right intuitions to make things work (for now). Today, breaking security rarely involves breaking a cryptographic protocol but rather due to bad implementations and easy-to-guess or phished passwords.

For neural networks we still live in an art regime. We have great methods to minimize the error for a given network (though no formal proofs as such), one still needs to design the right network with the appropriate topology and various tricks like regularization to prevent overfitting. Several members of the theoretical CS community have taken on the task to turn the art into a science but getting us to where we sit in cryptography will take some time. Crytography took thousands of years to get from an art to science, hopefully we can get there faster for machine learning.

This debate came to a head at the recent NIPS conference. Ali Rahimi in his test-of-time talk described the current state of ML research as "alchemy". Yann LeCun gave a strong response
Sticking to a set of methods just because you can do theory about it, while ignoring a set of methods that empirically work better just because you don't (yet) understand them theoretically is akin to looking for your lost car keys under the street light knowing you lost them someplace else.
Imagine if we never did any cryptography until we had the science in place.

Proving theorems in mathematics is an art more than a science. Because of incompleteness in some sense it has to be. If mathematics ever became automated it would be a whole lot less fun.

6 comments:

  1. Is science an art?

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  2. "Proving theorems in mathematics is an art more than a science."

    Just wait til AlphaZero (or somesuch) is turned on it...

    ReplyDelete
  3. Crytography took thousands of years to get from an art to science, hopefully we can get there faster for machine learning.

    I guess you meant "hopefully we can get there faster for neural networks". Machine learning as a whole has pretty strong scientific foundations.

    ReplyDelete
  4. Ellsworth Kelly and Mark Rothko over-glorified color/texture swatches are art. Proofs are science. Were they ever subjective?

    ReplyDelete
  5. 'If mathematics ever became automated it would be a whole lot less fun'

    This is so not true. I find automated (partially at least) formal proofs to be so much fun that I need to be really careful not to get addicted to it.

    ReplyDelete