The current state of computer science reminds me of the early excitement of the Internet in the mid-90's. By the beginning of the 90's, computers landed in many homes and most offices. Computers had modems, connected through the phone lines. Various online service networks, CompuServe, Prodigy, AOL, MSN, started up which provided some basic information and communication but were mostly walled gardens.
But then came the mosaic web browser in 1993. There weren't many web pages and they looked awful. There is a special place in hell for whomever invented the <blink> tag. But the ability to link to other pages, local or anywhere else on the web, was a game changer. I quickly created a web page so people could download my papers, to try it out and because I got tired of responding to email requests for them. People experimented with all different kinds of things on the web. Companies tried to figure out what to put on web pages. The online service networks reluctantly put browsers on their sites.
In the mid-90's the Internet was exciting but messy. Search engines would brag about the number of pages they searched but the ranking lacked relevance. Every CS job talk in every subarea, including theory, focused on the Internet. VC money flowed into Internet-related companies no matter how silly. It wasn't until Google using the PageRank algorithm gave us a good search engine, the dot-com bust drove out the bad companies and cloud computing gave us good platforms that we got to the Internet we have today, for better or for worse.
We're at that messy stage with machine learning. We can see the game-changing potential of the technology but far too many problems limit our ability to use them. VC money flows into new ML startups while our traditional Internet companies are shedding jobs. Will the transformers paper be this generation's PageRank or do we need another trick to take the next step? If the Internet is any guide, we'll get past this early stage, the market will shake out, only to develop even more challenges down the line.
I am surprised by one sentence: "Every CS job talk in every subarea, including theory, focused on the Internet". I do not think this is reflected in the papers from these years (for theory papers). I mean, did theorists really work on internet-related questions? I feel (but prove me wrong!) that the current trend is much more pervasive than internet was.
ReplyDeleteMany theorists didn't work directly work on Internet related papers but most found a way to connect their work to the Internet in their job talks. Lots of theorists also did work on Internet related papers. Most notably that's what led to Akamai.
Delete"Will the transformers paper be this generation's PageRank or do we need another trick to take the next step?"
ReplyDeleteAs you can guess, my take on this is that the transformer/LLM thing's problematic relationship with reality (namely that it has absolutely no relationship whatsoever to reality, causality, reason, reasoning, or even basic arithmetic) will end up being seen as fatal.
Machine learning, on the other hand, is just statistics. Now, statistics is hard, and using AI/ML buzzwords to avoid doing the hard work of getting one's statistics (and models) right looks to me to be problematic right now, but for the nonce, there are lots of places (i.e. data sets) that have correlations in them that we haven't seen yet, and even though correlation is not causation, some of these are things that will be worth looking at.
So when I see "we used AI techniques to do X", I get suspicious. One example the other day was in metallurgy (my undergrad minor). Metals mix when in the liquid phase and when you cool a mixture, strange things happen (phase diagrams of even bimetallic mixtures can be very complex). Since there are a lot of metals (much of the periodic table), there are a lot of strange things that can happen, and using statistical methods to look for mixtures of multiple different metals that end up having interesting mechanical properties when cooled is pretty cool, since it's a way larger space than one can deal with empirically/experimentally. You need direction, and the folks doing the work figured out how to make statistics (oops, "AI") help them find direction. I don't see how this can be called "AI", of course. And I don't see how the terms "AI" or "ML" are helpful/useful in describing this work. But it got them a paper in Science, so I guess they made the right decision.
"But the ability to link to other pages, local or anywhere else on the web, was a game changer."
ReplyDeleteGopher had the ability to link to other Gopher pages, local or anywhere else with a Gopher server.
"It wasn't until Google using the PageRank algorithm gave us a good search engine"
AltaVista was pretty great.
I have used both Gopher and Altavista back in the days.
ReplyDeleteBoth are great examples of an often forgotten phenomenon: it is not important to be first, you have to be first who is good enough.
ML is inherently messy because of the No Free Lunch theorem.
ReplyDelete