Wednesday, May 22, 2024

Peer Review

Daniel Lemire wrote a blog post Peer Review is Not the Gold Standard in Science. I wonder who was claiming it was. There is whole section of an online Responsible Conduct in Research we are required to take on peer review which discussing its challenges: "honesty, objectivity, quality control, confidentiality and security, fairness, bias, conflicts of interest, editorial independence, and professionalism". With apologies to Winston Churchill, Peer Review is the worst form of measuring academic quality, except for all of the others.

Peer review requires answering two questions.

  1. Has the research been done properly?
  2. What is the value of the research?
For theoretical research, the first comes down to checking the proofs, which sounds like an objective check. Here we have a "gold standard", formalizing the proof so it can be verified in a proof system like Lean. That's a heavy burden so we generally only require authors to give enough details so it's clear that we could formalize the proof given enough time. That becomes subjective and reviewers, especially for conferences, may not have the time or inclination to check the details of a 40-page proof. Maybe one day AI can take a well-written informal proof and formalize it for a proof system.

But the second question is almost entirely subjective. How does the work advance previous research? What value does it give to a field and how does it set up future research? Different researchers will give different opinions. And then there are the people who consciously or unconsciously cheat, helping their friends get papers accepted to citations rings. As we focus on metrics to judge researchers, too many people will game the system to pump up those metrics.

In 2013, NeurIPS had over 13,000 submission for 3500 slots. Even with the best or reviewer's intentions, it's impossible to maintain any sense of consistency for these large volume conferences.

Despite the problems with peer review, you'd hate to us a different system, say delegating the reviewing to some AI process, even if it could lead to more consistency. I suspect many reviews are being delegated anyway.

Peer review grew in importance as journals and conferences had to make choices to fill a limited proceedings. These days we have the capacity to distribute every papers. So perhaps the best form of measuring academic quality is no review at all.

Sunday, May 19, 2024

I don't do that well when the Jeopardy category is Math

Bill and Darling are watching Jeopardy.

DARLING: Bill, one of the categories is MATH TALK. You will kick butt!

BILL: Not clear. I doubt they will have the least number n such that R(n) is not known. They will ask things easy enough so that my math knowledge won't help.

DARLING: But you can answer faster.

BILL: Not clear. 
--------------------------------------------
Recall that in Jeopardy they give the answers and you come up with the question.
Like Sheldon Cooper I prefer my questions in the form of a question. 
Even so, I will present the answers that were given on the show (that sounds funny), then 
I will provide the questions (that sounds funny), what happened, and what I would have gotten right. 

$400
ANSWER: Its a demonstrably true mathematical statement; Calculus has a ``Fundamental'' one.
QUESTION: What is a Theorem?
WHAT HAPPENED: Someone buzzed in and said AXIOM. This one I knew the answer and would have won!

$800
ANSWER: Fire up the engines of your mind and name this solid figure with equal and parallel circles at either end. 
QUESTION: What is a Cylinder?
WHAT HAPPENED: Someone buzzed in with the correct answer. I had a hard time parsing this one and only got it right in hindsight. This one I would have lost on. Note that the phrase Fire up your engines is supposed to make you think of Fire on all cylinders. This did not help me.

$1200
ANSWER: Multiply the numerator of one fraction by the denominator of another (and vice versa) to get the ``cross'' this. 
QUESTION: What is a Product?
WHAT HAPPENED: I got this one very fast. So did the contestant on the real show. Not clear what would happened if I was there.

$1600
ANSWER: See if you can pick off this term for the point at which a line or curve crosses an axis. 
QUESTION: What is an Intercept?
WHAT HAPPENED: Someone buzzed in with the correct answer. I really didn't know what they were getting at. Even in hindsight the answer does not seem right, though I am sure that it is. The phrase pick off this term is  supposed to remind me of something, but it didn't. Lance happened to read a draft of this post and did the obvious thing: asked ChatGPT about it. ChatGPT said that in football a pick off is an interception. To see the ChatGPT transcript see here.

$2000
ANSWER: In 19-5=14 19 is the minuend; 5 is this other ``end''
QUESTION: What is a  Subtrahend?
WHAT HAPPENED: Someone buzzed in with the correct answer. The answer was news to me. It is correct; however, I am not embarrassed to say I never heard these terms. Spellcheck thinks that minuend and subtrahend words. This is similar to when I was not smarter than a fifth grader (see blog post here). 

----------------------------------------------------------------
So the final tally:
The $400 question I would have gotten right
The $1200 question I might have gotten right if I was fast on the buzzer

But that's it. Why did I do so badly? 
1) Two of the ones I got wrong were phrased in funny ways. I thought so anyway. And note that they did not use advanced math knowledge, so my math knowledge didn't help. (This is not a complaint- it would be bad if they used advanced math knowledge. Like when a crossword puzzle my wife was working on wanted  Log-Man and it began with N and I knew Napier. Why was that in a crossword puzzle for laypeople? Because  Napier has a lot of vowels in it.)

2) One of them I really did not know the math knowledge. Is it arrogant to say that if there is a math question on Jeopardy where I don't know the answer then its a bad question? I leave that as an exercise for the reader. 

On questions about  presidents, vice presidents, or American history, I do well.

On questions about novelty songs  (sometimes comes up) I do very well. (One question was about this song here. The question: here.)

But math... not so much. 

For computer science questions I also do not do that well, but I've learned some common abbreviations that I did not know: 

BIT: Binary Integer (A reader named Anonymous, who makes many comments, pointed out that BIT is actually Binary Digit. I have a possibly false memory of Jeopardy telling me Binary Integer. Either my memory is wrong or Jeopardy is wrong. But Anonymous is right- its Binary Digit.) 

HTTP: Hypertext Transfer Protocol

HTML: Hyper Text Markup Language

FORTRAN: Formula Translation

Those were more interesting than learning about minuend and subtrahend, terms I had never heard before and won't hear again unless I catch a rerun of Jeopardy (at which time I will get it right).




Wednesday, May 15, 2024

Jim Simons (1938-2024)

Jim Simons passed away Friday at the age of 86. In short he was a math professor who quit to use math to make money before it was fashionable and used part of his immense wealth to start the Simons Foundation to advance research in mathematics and the basic sciences.

While his academic research focused on manifolds, Simons and his foundation had theoretical computer science as one of its priorities and helped fund and promote our field on several fronts.

Foremost of course is the Simons Institute, a center for collaborative research in theoretical computer science. Announced as a competition in 2010 (I was on team Chicago) with the foundation eventually landing on UC Berkeley's campus. At the time, I wrote "this will be a game changer for CS theory" if anything proven to be an understatement over the last dozen years.

Beyond the institute, the Simons Foundation has funded a number of theorists through their investigator and other programs.

Let's not forget Quanta Magazine, an online science publication funded by the foundation without subscriptions or paywalls while science journalism has been seeing cuts elsewhere. Quanta has been particularly friendly to the computational complexity community such as this recent article on Russell and his worlds.

The Simons Foundation will continue strong even without its founder. But as we see challenges in government funding, how much can or should we count on wealthy patrons to support our field?

Read more on Jim Simons from Scott, Dick, the foundation and the institute.

Saturday, May 11, 2024

What is Closed Form? The Horse Numbers are an illustration

In the book Those Fascinating Numbers by Jean-Marie De Konick they find interesting (or `interesting') things to say about many numbers. I reviewed the book in a SIGACT News book review column here. The entry for 13 is odd: 13 is the third Horse Number.  The nth Horse number is the number of ways n horses can finish a race. You might think: OH, that's just n!. AH- horses can tie. So it's the number of ways to order n objects allowing ties. 

Is there a closed form for H(n)? We will come back to that later. 

0) The Wikipedia Entry on horse races that ended in a dead  heat is here. They list 78 dead heats (two horses tie for first place) and 10 triple dead heats (three horses tie for first place). For the horse numbers we care if (say) two horses tie for 4th place. In reality nobody cares about that. 

1) I have found nowhere else where these numbers are called The Horse Numbers. 

2) They are called the Ordered Bell Numbers. The Wikipedia entry here has some applications.

3) They are also called the Fubini Numbers according to the Ordered Bell Number Wikipedia page.

4) I had not thought about the Horse Numbers for a long time  when they came up while I was making slides for the proof that  (Q,<) is decidable (the slides are here).

5) There is an OEIS page for the Horse Numbers, though they are called the Ordered Bell Numbers and the Fubini Numbers. It is here. That page says H(n) is asymptotically \(\frac{1}{2}n!(\log_2(e))^{n+1}\) which is approx \(\frac{1}{2}n!(1.44)^{n+1}\).

6) There is a recurrence for the Horse Numbers:

H(0)=1

H(1)=1

H(2)=3

For all  \(n\ge 3\) we split H(n) into what happens  if i horses are tied for last place (choose i out of n) and if the rest are ordered H(n-i) ways. Hence

\( H(n) = \binom{n}{1}H(n-1) + \binom{n}{2}H(n-2) +  \cdots  + \binom{n}{n}H(0) \)

Using \(\binom{n}{i} = \binom{n}{n-i}\) we get

\( H(n) = \binom{n}{0}H(0) + \binom{n}{1}H(1) +  \cdots  + \binom{n}{n-1}H(n-1) \)

STUDENT: Is there a closed form for H(n)?

BILL: Yes. Its H(n).

STUDENT: That's not closed form.

BILL: Is there a closed form for the number of ways to choose i items out of n?

STUDENT: Yes, \(\binom{n}{i}\) or \( \frac{n!}{i!(n-i)!}\) 

BILL: Does that let you compute it easily? No. The way you compute \(\binom{n}{i}\) is with a recurrence. The way you compute H(n) is with a recurrence. Just having a nice notation for something does not mean you have a closed form for it. 

STUDENT: I disagree! We know what n! is!

BILL: Do not be seduced by the familiarity of  the notation. 



Wednesday, May 08, 2024

Favorite Theorems: Dichotomy

A constraint satisfaction problem has a group of constraints applied to a set of variables and we want to know if there is a setting of the variables that make all the constraints true. In CNF-Satisfiability the variables are Boolean and the constraints are ORs of variables and their negations. In graph coloring, the variables are the colors of the nodes and the constraints, corresponding to edges, are two variables must be different. These problems lie in NP, just guess the values of the variables and check the constraints. They are often NP-complete. They are sometimes in P, like 2-coloring graphs. But they are never in between--all such problems are either in P or NP-complete.


Ladner's Theorem states that if P \(\neq\) NP then there exists a set in NP that is not in P and not NP-complete. Ladner's proof works by blowing holes in Satisfiability, an unsatisfying construction as it gives us a set that is NP-complete on some input lengths and easy on others. One could hope that some version of a constraint satisfaction problem could lead to a more natural intermediate set but dichotomy theorems tell us we need to look elsewhere.

In 1978, Thomas Schaefer gave a dichotomy theorem for satisfiability problems, basically CSP problems over Boolean variables. In 1990, Pavol Hell and Jaroslav Nešetřil showed a dichotomy result for homomorphisms of undirected graphs as described in my 2017 blog post. In 1998 Tomás Feder and Moshe Vardi formalized the constraint satisfaction dichotomy conjecture and expressed it as homomorphisms of directed graphs. The blog post described a claimed but later retracted solution to the dichotomy conjecture. Bulatov and Zhuk announced independent and different correct proofs later that year. In 2020 Zhuk received the Presburger Award for his paper (Bulatov was too senior for the award). 

Sunday, May 05, 2024

May the fourth be with you. Too many -days?

(This post was inspired by Rachel F, a prior REU-CAAR student, emailing me wishing me a happy Star Wars Day.) 

 I am writing this on May 4 which is Star Wars day. Off the top of my head I know of the following special days (I exclude official holidays, though the term official has no official meaning.)

Jan 25: Opposite Day Wikipedia Link

Feb 2: Groundhog Day Wikipedia Link

Feb 12: Darwin Day Wikipedia Link

March 14: Pi Day Wikipedia link

May 4: Star Wars Day Wikipedia Link

April 22: Earth Day Wikipedia link

April 25: Take your Child to Work Day Wikipedia Link

Sep 21: National Cleanup Day Wikipedia Link

Sept 22: Hobbit Day Wikipedia Link

Oct 1: International Coffee Day Wikipedia Link

Oct 8: Ada Lovelace Day Wikipedia Link

Oct 16: Boss's Day  Wikipedia Link

Oct 23: Mole Day Wikipedia Link

Nov 13: Sadie Hawkins Day Wikipedia Link

Sept 19: Talk like a Pirate Day Wikipedia Link

A few notes

1a) Oct 23 is also Weird Al's birthday.

1b) May 4 is also Edward Nelson's birthday (he invented the problem of  finding the chromatic number of the plan). See my post (actually a guest post by Alexander Soifer) on the problem here for more information on that.

1c) I left off St. Patrick's Day (March 17) and International LGBT + Pride day (June 28) and many others.  Those left off are so well known that they are official where as I was looking for unofficial holidays. But see next point.

2) The Wikipedia entry for Talk Like a Pirate Day says it's a parodic holiday. The entries on the others holidays use terms like unofficial. I prefer unofficial since ALL holidays are made up, so the only real question is which ones are recognized. But even that is problematic since one can ask recognized by who? Also, despite collecting parody music and videos for the last 50 years, I have never heard the term parodic. Therefore it is not a word. Spellcheck agrees!

3) Darwin Day should be Darwin-Lincoln day since they were both born on Feb 12. In fact,they were both born in 1809. Most famous same-birthday-and-year pair ever. Second place is Lenny Bruce and Margaret Thatcher (Oct 13, 1925). 

4) The page on Pi Day mentions Tau Day, but Tau day has no page of its own. Tau is \(2\pi\) which some say comes up more often then \(\pi\) and hence should be THE constant. Some say that \(2\pi i\) comes up so often that it should be THE constant. However, there can't really be a day to celebrate it.(I blogged about is-tau-better-than-pi here.)

5) In the future every day will be some kind of day. The Future Is NOW: Website of Fun Holidays

Are the holidays on the list real? Depends what you mean by real. Because of the web anyone can post a list of anything and its just one person's opinion. I do not know who controls that website but even if I did, it would be hard to say YES THOSE ARE REAL or NO THOSE ARE NOT. 

One could say that to be a real DAY, it has to be on Wikipedia. But there are two problems with this:

a) Goodhart's law. When a measure becomes a target it stops being a measure. If I want Jan 15 to be Bagel and Lox Day, I'll make a page for it.

b) I'm still waiting for Raegan Revord, who has played Missy on Young Sheldon for 7 years, to get a Wikipedia Page. So what hope does Polar Bear Plunge day (Jan 1) have for getting a Wikipedia Page? 



Wednesday, May 01, 2024

Our Obsession with Proofs

Bullinger's post on this blog last week focused on Vijay Vazirani's public obsession of finding a proof for the 1980 Micali-Vazirani matching algorithm. But why does Vijay, and theoretical computer science in general, obsess over proofs? 

You can't submit a paper to a theory conference without a claimed complete proof, often contained in an appendix dozens of pages long. Often we judge papers more on the complexity of the proof than the statement of the theorem itself, even though for a given theorem a simpler proof is always better.

A proof does not make a theorem true; it was always true. The Micali-Vazirani algorithm is no faster with the new proof. Would we have been better off if the algorithm didn't get published before there was a full proof?

We're theoretical computer scientists--doesn't that mean we need proofs? Theoretical economists and physicists don't put such an emphasis on proofs, they focus on models and theorems to justify them.

Once a senior CS theorist told economists that his group had given the first full proof of a famous economics theorem and wondered why the economists didn't care. The economists said they already knew the theorem was true, so the proof added little to their knowledge base.

More than one journalist has asked me about the importance of a proof that P \(\ne\) NP. A proof that P = NP would be both surprising and hopefully give an algorithm. While a proof that P \(\ne\) NP would be incredibly interesting and solve a major mathematical challenge, it wouldn't do much more than confirm what we already believe.

I'm not anti-proof, it is useful to be absolutely sure that a theorem is true. But does focusing on the proofs hold our field back from giving intuitively correct algorithms and theorems? Is working out the gory details of a lengthy proof, which no one will ever read, the best use of anyone's time? 

As computing enters a phase of machine learning and optimization where we have little formal proof of why these models and algorithms work as well as they do, does our continued focus on proofs make our field even less relevant to the computing world today?