Monday, June 23, 2003

Whose hands are on the FOCS cover?

FOCS
Cover Image Sometimes I learn interesting aspects of the history of theory from this weblog. On an old post on conference covers, Pekka Orponen left a comment saying that Alvy Ray Smith originally designed the FOCS cover back in 1973, when the conference was SWAT: Switching and Automata Theory. Here is Smith's story about the cover where he states the hands are his.

Smith had three papers in SWAT before moving into computer graphics and co-founding companies such as Pixar. So next time you are looking up a FOCS paper take a look at the hands of a one-time theorist who made a difference.

Wednesday, June 18, 2003

Walter Savitch

After the FCRC meetings I attended were concluded, I headed up to UCSD for the celebration of Walter Savitch for his sixtieth birthday and upcoming retirement. He gained his fame in complexity for Savitch's Theorem that shows "P=NP" for space.

I learned quite a bit at the meeting. Walt Savitch was Steve Cook's first student, his only student while Cook was at Berkeley in his pre-Toronto pre-"SAT is NP-complete" days. Also as Cook said, Savitch is the only student he has had with a theorem named after him. That theorem made up a good part of Savitch's Ph.D. thesis. At the celebration Cook gave an overview on propositional proof systems.

After coming to UCSD, Savitch did some work on computational linguistics and one of the leaders of the field, Aravind Joshi, gave a talk on combining trees to keep the structure when parsing sentences.

Savitch is probably best known now in computer science for his textbooks in introductory programming that likley many of you have used.

Congrats Walt on a fine career and here's hoping retirement doesn't slow you down.

Tuesday, June 17, 2003

FOCS 2003

The list of accepted papers for FOCS 2003 has been posted. As usual, many (but not all) of the papers are available on the authors' web sites.

Monday, June 16, 2003

Boosting

As promised I added links to the papers in the post on the STOC business meeting. Let me say some more words on the winner of the Gödel prize

Valiant developed the concept of PAC (Probably Approximably Correct) learning as roughly where a learner sees a small number of labelled examples from a distribution and with high confidence will generate a hypothesis that with high probability will correctly label instances drawn from the same distribution.

A strong learner has confidence close to 100%; a weak learner has confidence only slightly better than 50%. Schapire, using a technique called boosting, showed how to convert a weak learner to a strong learner. This is a wonderful theoretical result but the algorithm had problems that made it difficult to implement.

In their Gödel prize winning paper, A decision-theoretic generalization of on-line learning and an application to boosting, Freund and Schapire develop the adaboost algorithm that solves many of these issues and has become a staple of the theoretical and practical machine learning community.

Boosting has its own web site where you can find much more information about the algorithms and applications.

Saturday, June 14, 2003

Alonzo Church

Alonzo Church was born a hundred years ago today in Washington, DC. Church is best known for the λ-calculus, a simple method for expressing and applying functions that has the same computational power as Turing machines.

With Rosser in 1936, he showed that λ-expressions that reduce to an irreducible normal form have a unique normal form. In that same year he showed the impossibility of decided whether such a normal form existed.

Church's thesis, which he states as a definition: "An effectively calculable function of the positive integers is a λ-definable function of the positive integers."

Again in 1936, Kleene and Church showed that computing normal forms have the equivalent power of the recursive functions of Turing machines. And thus the Church-Turing thesis was born: Everything computable is computable by a Turing machine.

The λ-calculus also set the stage for many of the functional programming languages like lisp and scheme.

Alonzo Church passed away on August 11, 1995 in Ohio.

Friday, June 13, 2003

Thoughts on FCRC

I have mixed feelings about the Federated Computing Research Conference. It is a good idea to get many different areas of computer science together. I do get to see many people I haven't seen in years who went into non-theoretical areas of CS.

On the other hand 2200 participants made the place quite crowded and it seemed to take away from the informal atmosphere of most theory conference. Since STOC and Electronic Commerce had nearly a complete overlap I jumped back and forth between talks never really feeling fully part of either conference.

For the first time the Complexity conference was not part of FCRC because 2003 is a Europe year for Complexity. In an informal poll I took of STOC people interested in complexity most liked having both conferences at the same place but would rather that happen in isolation, like last year in Montreal, rather than as part of the much larger FCRC meeting.

In what seems to be a trend in CS conferences, wireless internet was made available at the conference site. As you walked around you would pass many people sitting on chairs and on the ground hunched over their laptops disconnected from the conference and connected into another world. Seemed a bit depressing but I too found the net hard to resist--it is always tempting to simply open my laptop and connect, checking email and posting to this weblog.

Tuesday, June 10, 2003

The Business Meeting

Update (9/3/03): STOC 2004 CFP

Last night was the business meeting for STOC. This used to be a raucous affair with major battles over issues like whether to have parallel sessions, but now in its old age is more for distributing information like

  • Awards: Godel Prize for best paper published in a journal in last seven years: Yoav Freund and Rob Schapire for their adaboost algorithm.
    Knuth Prize for lifetime research activity: Miklos Ajtai.
    STOC Best Paper: Jointly to Impagliazzo and Kabanets for "Derandomizing Polynomial Identity Tests Means Proving Circuit Lower Bounds" (you read it here first) and Oded Regev for "New Lattice Based Cryptographic Constructions".
    The Danny Lewin Best Student Paper Award: Thomas Hayes for "Randomly Coloring Graphs of Girth at Least Five" Thomas Hayes is from my once and future department at U. Chicago.
  • Future Conferences: FOCS 2003 in Boston, STOC 2004 in Chicago and FOCS 2004 in Rome, its first time outside North America. On Friday, a day after agreeing to return to Chicago, Janos Simon asked me to give the presentation on STOC 2004, which is always fun. Laszlo Babai, another member of the Chicago faculty, will be PC chair of that meeting.
  • Registration numbers, NSF report, progress on scanning in old papers and redoing submission software. I'm not going to do a full business meeting report--someone else does that and will eventually appear in SIGACT news.

Monday, June 09, 2003

Howdy from San Diego

This week I'm at the Federated Computing Research Conference (FCRC), a combination of thirty conferences and workshops with 2200 participants. I'm here for the theory conference (STOC) and Electronic Commerce.

Last night, Adleman, Rivest and Shamir gave their Turing award lecture, each giving twenty minutes of an hour long talk. Their basic them on how cryptology has changed in the last 25 years:

  1. Cryptography is now done publicly rather than in secret. This has led to researchers building on each others ideas to create better and better encryption schemes and protocols. But also it has allowed more people to attack these protocols and weed out the bad ones.
  2. Cryptography has moved from art to science. Now we have protocols based on mathematical ideas like number theory instead of just creating seemingly complexity functions.
Adi Shamir made other interesting comments like that perfect cryptography is impossible, though very good cryptography can be had at a modest cost. Most attacks on practical implementations of cryptographic protocols work on the implementation as opposed to the protocol.

The lectures were taped and may show up on-line someday. I'll let you know if I find them there--definitely recommended viewing.

Friday, June 06, 2003

Back to Chicago

A personal note: I have accepted an offer to return to the computer science department of the University of Chicago starting this fall. As NEC Labs has been moving its focus away from basic research, it is time for me to go back to an academic life.

I plan to keep this weblog going in Chicago as long as I have things to say.

Tuesday, June 03, 2003

Foundations of Complexity
Lesson 19: The Immerman-Szelepcsenyi Theorem

Previous Lesson

In this lesson we will prove the Immerman-Szelepcsényi Theorem.

Theorem (Immerman-Szelepcsényi): For reasonable s(n)≥ log n, NSPACE(s(n))=co-NSPACE(s(n)).

Let M be a nondeterministic machine using s(n) space. We will create a nondeterministic machine N such that for all inputs x, N(x) accepts if and only if M(x) rejects.

Fix an input x and let s=s(|x|). The total number of configurations of M(x) can be at most cs for some constant c. Let t=cs. We can also bound the running time of M(x) by t because any computation path of length more than t must repeat a configuration and thus could be shortened.

Let I be the initial configuration of M(x). Let m be the number of possible configurations reachable from I on some nondeterministic path. Suppose we knew the value of m. We now show how N(x) can correctly determine that M(x) does not accept.

Let r=0
For all nonaccepting configurations C of M(x)
  Try to guess a computation path from I to C
  If found let r=r+1
If r=m then accept o.w. reject
If M(x) accepts then there is some accepting configuration reachable from I so there must be less than m non-accepting configurations reachable from I so N(x) cannot accept. If M(x) rejects then there is no accepting configurations reachable from I so N(x) on some nondeterministic path will find all m nonaccepting paths and accept. The total space is at most O(s) since we are looking only at one configuration at a time.

Of course we cannot assume that we know m. To get m we use an idea called inductive counting. Let mi be the number of configurations reachable from I in at most i steps. We have m0=1 and mt=m. We show how to compute mi+1 from mi. Then starting at m0 we compute m1 then m2 all the way up to mt=m and then run the algorithm above.

Here is the algorithm to nondeterministically compute mi+1 from mi.

Let mi+1=0
For all configurations C
  Let b=0, r=0
  For all configurations D
    Guess a path from I to D in at most i steps
    If found
      Let r=r+1
      If D=C or D goes to C in 1 step
        Let b=1
  If r<mi halt and reject
  Let mi+1=mi+1+b
The test that r<mi guarantees that we have looked at all of the configurations D reachable from I in i steps. If we pass the test each time then we will have correctly computed b to be equal to 1 if C is reachable from I in at most i+1 steps and b equals 0 otherwise.

We are only remembering a constant number of configurations and variables so again the space is bounded by O(s). Since we only need to remember mi to get mi+1 we can run the whole algorithm in space O(s).

Wednesday, May 28, 2003

Open Questions from Hopcroft and Ullman

On page 281 of the 1979 edition the classic theory text of Hopcroft and Ullman lies two tables describing closure and decidability properties of various formal languages. Four entries in the table are labelled by "?" meaning the answer was not known. Those entries are
  1. Are context-sensitive languages closed under complementation?
  2. If L is context-sensitive is MIN(L) context sensitive? MIN(L) is the set of strings in L who do not have proper prefixes in L.
  3. Is it decidable whether the complement of a given context-sensitive language is context-sensitive?
  4. Is it decidable whether two given deterministic context-free languages are equal?
We now know the answers to all of these question are "Yes."
  1. In 1988, Neil Immerman and Róbert Szelepcsényi independently showed that nondeterministic space is closed under complement. Since CSLs are equivalent to nondeterministic linear space we have that CSLs are also closed under complement. Immerman and Szelepcsényi received the 1995 Gödel Prize for this result. We will cover Immerman-Szelepcsényi in the next Foundations lesson.
  2. By using Immerman-Szelepcsényi, one can create a nondeterministic linear time algorithm to check that x is in L and that all proper prefixes of x are not in L.
  3. Trivially true by Immerman-Szelepcsényi.
  4. This was the hard one. Only in 1997 did Géraud Sénizergues succeed in showing that the problem is decidable. For this work he received the 2002 Gödel Prize. Here is the simplified version of his result, a mere 58 pages.

Thursday, May 22, 2003

Computing's Lost Allure?

The New York Times today had an article on the shrinking number of computer science majors in American universities. Let me give you my take on this.

First of all this should be no surprise. People, consciously or unconsciously, follow the money. Computer science majors were a very hot property in the late 90's and now they are less so. So less people are going into computer science.

My advice: Don't follow the money. You will be trying to time the market four years down the road. I remember as an undergrad seeing many of my fellow freshman going into Chemical Engineering because it was a hot area. Four years later many of them had trouble finding or keeping a job and had to move to other areas. Some of them even became.......lawyers.

Best to do what you enjoy. If you enjoy it you will have a better chance to succeed. Worry about the job market when you get there.

Wednesday, May 21, 2003

Celebration for Walter Savitch

Just got this announcement. And we just discussed Savitch's Theorem in my last Foundations Lesson.

The Department of Computer Science and Engineering at the University of California, San Diego, is sponsoring an event on June 12, 2003, on the occasion of Professor Walt Savitch's birthday and in recognition of his thirty-four years of teaching Computer Science at UCSD. This event will feature two distinguished speakers, Professor Stephen Cook (University of Toronto) and Professor Aravind Joshi (University of Pennsylvania) with a reception in honor of Professor Savitch concluding the program. The event will take place at Eucalyptus Point (Room B) in Thurgood Marshall College on the UCSD campus. Please see the department web site for a link to more information.

This event immediately follows the STOC conference which is also being held in San Diego as part of FCRC.

Friday, May 16, 2003

Turing Machines and Godel's Theorems

A little recursion theory can make Gödel's Theorems intuitively easy.

Let A be the set of <M> such that M does not accept the input <M>. By diagonalization techniques we can show A is not recursively enumerable.

Now let us fix a logical theory like ZFC set theory. The actual theory does not matter much. Let B be the set of <M> such that there is a proof in ZFC of the statement "M does not accept input <M>." B is recursively enumerable since we can just try all possible proofs.

Now B is clearly a subset of A so there must be an input <M> in A-B. For this <M>, we have that M does not accept input <M> but there is no proof of this fact. We now have a true mathematical statement with no proof. This is Gödel's first theorem.

We can be more constructive. Since B is recursively enumerable there is some Turing machine N that computes B. Suppose that N accepts <N>. This means <N> is in B which implies there is a proof that <N> does not accept N, a contradiction.

So <N> cannot accept N which means <N> is not in B. So we have now constructed an N such that the statement "N does not accept <N>" is true but not provable.

But wait a minute, didn't I just give a proof that N does not accept <N>? Actually I had to make the assumption that ZFC is consistent. If ZFC is inconsistent then every statement (true or false) is provable so B would not be a subset of A and my whole argument falls apart.

If ZFC could prove that ZFC is consistent then I would not have to make any assumption and would have a contradiction. Thus ZFC cannot prove its own consistency. This is Gödel's second theorem.

Logicians tell me I'm cheating: I had to assume something technically stronger than consistency for this argument to work. Still these proofs illustrate the amazing power of Turing machines to make Gödel's theorems easier to understand.

Wednesday, May 14, 2003

Foundations of Complexity
Lesson 18: Savitch's Theorem

Previous Lesson | Next Lesson

Unlike what we believe for time, there is a polynomial relation between deterministic and nondeterministic space.

Savitch's Theorem (1970): NSPACE(s(n)) is contained in DSPACE(s2(n))

Proof: Recall the definition of tableau (as described in Lesson 12). Let N be a nondeterministic machine that uses s(n) space. We can represent the computation of an input x of length n by a tableau where each configuration has size O(s(n)) and there are at most m = cs(n) configurations for some constant c.

We will create a deterministic machine M(x) to determine whether the tableau is proper and thus N(x) accepts. First we need a subroutine CHECK to determine whether one configuration can reach another in 2t steps. We do not have enough space to write the entire tableau but instead we do a divide and conquer approach: Try all possible middle configurations and recurse on each half.

CHECK(CONF1,CONF2,t)
\* Output TRUE if CONF1 can get to CONF2 in at most 2t steps *\
If t=0 then 
        {if (CONF1 = CONF2
         or CONF1 goes to CONF2 in one step on machine N)
         then output TRUE else output FALSE}
For each CONF
  {If CHECK(CONF1,CONF,t-1) and CHECK(CONF,CONF2,t-1) then output TRUE}
Output FALSE

We can implement CHECK by only having to store a constant number of configurations at each level of the recursion with a recursive depth of t for a total space of O(ts(n)).

Let CONF0 be the initial configuration encoding x. We can now give our main routine.

MAIN
Let r be the smallest integer at least log2(cs(n))
For each CONF in an accepting state
   {If CHECK(CONF0,CONF,r) then output TRUE}
Output FALSE

Total space used: O(log2(cs(n))s(n)) = O(s2(n)).

Monday, May 12, 2003

The Nerd Shot

Many years ago I was commuting home on the train with my wife and one of her colleagues. I showed them the group picture from a Dagstuhl I had attended that they recently sent me. My wife and her friend spent much of the train ride laughing at the collection of nerdly looking people in the photo. Ever since then we have jokingly called a conference group picture the nerd shot.

Computer scientists, especially in Europe, love to take pictures of other computer scientists. Problem is computer scientists are not particularly photogenic nor, for the most part, are they good photographers. The worst is the group photo, where we are herded like cattle to some enclosed place where we stand until all the strays are rounded up and finally the photo is taken, sometimes several times by several people.

Some things have changed over time. Most photos are now taken digitally and get posted quickly, sometimes before the conference is over. My kids enjoy playing "Where's Daddy?" especially if I am out of town. But still the group photo experience remains the same.

I was talking to my wife on the phone at the last Dagstuhl and told her it was time for the nerd shot. She said I should fix my hair and I replied "What? You want me to stand out?"

Without further adieu here is that nerd shot.

Thursday, May 08, 2003

Foundations of Complexity
Lesson 17: Space Complexity

Previous Lesson | Next Lesson

In addition to time, computer scientists also worry about the memory or space that a Turing machine uses. Roughly one can measure space as the number of tape squares used by at Turing machine. We would like to talk about space bounds like log n and still be able to read the whole input so we need a slightly different model.

We now allow our Turing machine to have a read-only input tape, one or more work tapes and for a Turing machine computing a function, a write-only output tape. On the input tape the head can move back and forth but it cannot change the values in any cell. On the output tape the head can only move right, writing as it moves. The amount of space a Turing machine uses on input x is the number of cells of the work tape that it uses. We will assume a space bound of at least log n since we need log n bits to describe the location of the the input head pointer.

In real world terms, think of your computer accessing "the internet". You can still reach many pages even though you cannot store the entire internet on your computer.

Any machine that runs in time t(n) clearly also runs in space t(n). If a machine uses space s(n) and it halts then it will halt in time cs(n) for some constant c. Otherwise the machine will repeat a configuration and run forever.

We define the classes DSPACE(s(n)) as the set of languages that are accepted by Turing machines using at most O(s(n)) space. NSPACE(s(n)) is the same for nondeterministic machine. We will always assume s(n)≥log n and s(n) is an easily computable function.

Common space classes include L=DSPACE(log n), NL=NSPACE(log n), PSPACE=∪kDSPACE(nk) and NPSPACE=∪kNSPACE(nk).

Unlike the P versus NP question, we know quite a bit about the relationship of deterministic versus nondeterministic space through the following two results.

  • Savitch's Theorem: NSPACE(s(n))⊆DSPACE(s2(n)). In particular this means NPSPACE = PSPACE.
  • Immerman-Szelepcsényi Theorem: If L is in NSPACE(s(n)) then the complement of L is also in NSPACE(s(n)).
We will discuss these theorems in more detail in upcoming lessons.

Tuesday, May 06, 2003

Universal Search

Psst. Want to know the fastest algorithm for factoring? I can give you an algorithm that is within a constant multiplicative factor of the best possible factoring algorithms.

Actually this is an idea due to Levin. Let p1, p2, ... be a list of all programs. On some input m simulate program p1 for half of your computation time, p2 for a quarter of the time, p3 for an eighth of the time, etc., until one of these programs outputs a factor of m. If pi is the fastest algorithm for factoring then our algorithm will run in time at most 2i times the running time of pi. The multiplicative factor 2i is independent of m but unfortunately could be quite large.

Marcus Hutter gives another algorithm that has a multiplicative factor of 5 but has a large additive constant. The trick is to spend some of your time searching for a proof that an algorithm is correct and runs in certain amount of time. You then only need to simulate the provably fastest algorithm found so far.

Hutter's algorithm works only as fast as the provably best algorithm with a provable running time. It could very well be the case that there is some good heuristic for factoring that does not have a provable running time or proof of correctness. Levin's technique will capture this case.

Of course, neither of these papers gives a practical algorithm as the constants involved go beyond huge. Nevertheless it is still interesting to see the theoretical possibilities of universal search.

Sunday, May 04, 2003

FCRC

Even the largest theoretical computer science conferences draw at most a couple of hundred people. Many (but not all) other areas of computer science also do not draw large numbers of participants. To have a larger and more noticeable conference and possibly to get press attention, the CS community decided to hold a joint conference covering many different areas in computer science. In 1993, the first Federated Computing Research Conference was held in San Diego.

The press didn't show but with some cross-collaboration the conference was considered a mild success. After stops in Philadelphia and Atlanta, FCRC returns to San Diego June 7-14. The 2003 FCRC includes the main spring theory conference (STOC), Electronic Commerce (EC), Computational Geometry, Principle and Practice of Parallel Programming and many others. Registration deadline is May 7.

For the first time the Complexity Conference will not be part of FCRC as 2003 is a Europe year for us. I am planning to attend FCRC for STOC and the EC meeting. If you attend FCRC stop by and say hi.

Thursday, May 01, 2003

History's Loss/Mathematics' Gain

Some excitement at Schloss Dagstuhl this week. Localized high winds tore the metal plating off the roof of much of the new building Wednesday evening. Much of that roof sits in the courtyard--quite a sight. The good news is no one was injured and property damage, besides the roof, was minimal. A few of us had to switch rooms but the workshop goes on.

Let me finish off this centennial week with a story of Kolmogorov that I have heard from different sources so some variation of this story is likely true. Kolmogorov initially wanted to be an historian. He looked at the question as to whether taxes in Russia in the middle-ages were collected at the house level or at the village level. He analyzed tax data and showed that that data had a much simpler description if taxes were collected at the village level. (You can see the seeds of Kolmogorov complexity here). He presented these results to the history faculty to great applause. Asking them whether he could publish such a paper he was told, "You only have one proof. You cannot publish in a history journal without at least two more proofs of your hypothesis." And so Kolmogorov left history for a field where one proof suffices.