Smith had three papers in SWAT before moving into computer graphics and co-founding companies such as Pixar. So next time you are looking up a FOCS paper take a look at the hands of a one-time theorist who made a difference.
Computational Complexity and other fun stuff in math and computer science from Lance Fortnow and Bill Gasarch
Monday, June 23, 2003
Whose hands are on the FOCS cover?
Wednesday, June 18, 2003
Walter Savitch
I learned quite a bit at the meeting. Walt Savitch was Steve Cook's first student, his only student while Cook was at Berkeley in his pre-Toronto pre-"SAT is NP-complete" days. Also as Cook said, Savitch is the only student he has had with a theorem named after him. That theorem made up a good part of Savitch's Ph.D. thesis. At the celebration Cook gave an overview on propositional proof systems.
After coming to UCSD, Savitch did some work on computational linguistics and one of the leaders of the field, Aravind Joshi, gave a talk on combining trees to keep the structure when parsing sentences.
Savitch is probably best known now in computer science for his textbooks in introductory programming that likley many of you have used.
Congrats Walt on a fine career and here's hoping retirement doesn't slow you down.
Tuesday, June 17, 2003
FOCS 2003
Monday, June 16, 2003
Boosting
Valiant developed the concept of PAC (Probably Approximably Correct) learning as roughly where a learner sees a small number of labelled examples from a distribution and with high confidence will generate a hypothesis that with high probability will correctly label instances drawn from the same distribution.
A strong learner has confidence close to 100%; a weak learner has confidence only slightly better than 50%. Schapire, using a technique called boosting, showed how to convert a weak learner to a strong learner. This is a wonderful theoretical result but the algorithm had problems that made it difficult to implement.
In their Gödel prize winning paper, A decision-theoretic generalization of on-line learning and an application to boosting, Freund and Schapire develop the adaboost algorithm that solves many of these issues and has become a staple of the theoretical and practical machine learning community.
Boosting has its own web site where you can find much more information about the algorithms and applications.
Saturday, June 14, 2003
Alonzo Church
With Rosser in 1936, he showed that λ-expressions that reduce to an irreducible normal form have a unique normal form. In that same year he showed the impossibility of decided whether such a normal form existed.
Church's thesis, which he states as a definition: "An effectively calculable function of the positive integers is a λ-definable function of the positive integers."
Again in 1936, Kleene and Church showed that computing normal forms have the equivalent power of the recursive functions of Turing machines. And thus the Church-Turing thesis was born: Everything computable is computable by a Turing machine.
The λ-calculus also set the stage for many of the functional programming languages like lisp and scheme.
Alonzo Church passed away on August 11, 1995 in Ohio.
Friday, June 13, 2003
Thoughts on FCRC
On the other hand 2200 participants made the place quite crowded and it seemed to take away from the informal atmosphere of most theory conference. Since STOC and Electronic Commerce had nearly a complete overlap I jumped back and forth between talks never really feeling fully part of either conference.
For the first time the Complexity conference was not part of FCRC because 2003 is a Europe year for Complexity. In an informal poll I took of STOC people interested in complexity most liked having both conferences at the same place but would rather that happen in isolation, like last year in Montreal, rather than as part of the much larger FCRC meeting.
In what seems to be a trend in CS conferences, wireless internet was made available at the conference site. As you walked around you would pass many people sitting on chairs and on the ground hunched over their laptops disconnected from the conference and connected into another world. Seemed a bit depressing but I too found the net hard to resist--it is always tempting to simply open my laptop and connect, checking email and posting to this weblog.
Tuesday, June 10, 2003
The Business Meeting
Last night was the business meeting for STOC. This used to be a raucous affair with major battles over issues like whether to have parallel sessions, but now in its old age is more for distributing information like
- Awards: Godel Prize for best paper published in a journal in last
seven years: Yoav Freund and Rob Schapire for their adaboost
algorithm.
Knuth Prize for lifetime research activity: Miklos Ajtai.
STOC Best Paper: Jointly to Impagliazzo and Kabanets for "Derandomizing Polynomial Identity Tests Means Proving Circuit Lower Bounds" (you read it here first) and Oded Regev for "New Lattice Based Cryptographic Constructions".
The Danny Lewin Best Student Paper Award: Thomas Hayes for "Randomly Coloring Graphs of Girth at Least Five" Thomas Hayes is from my once and future department at U. Chicago. - Future Conferences: FOCS 2003 in Boston, STOC 2004 in Chicago and FOCS 2004 in Rome, its first time outside North America. On Friday, a day after agreeing to return to Chicago, Janos Simon asked me to give the presentation on STOC 2004, which is always fun. Laszlo Babai, another member of the Chicago faculty, will be PC chair of that meeting.
- Registration numbers, NSF report, progress on scanning in old papers and redoing submission software. I'm not going to do a full business meeting report--someone else does that and will eventually appear in SIGACT news.
Monday, June 09, 2003
Howdy from San Diego
Last night, Adleman, Rivest and Shamir gave their Turing award lecture, each giving twenty minutes of an hour long talk. Their basic them on how cryptology has changed in the last 25 years:
- Cryptography is now done publicly rather than in secret. This has led to researchers building on each others ideas to create better and better encryption schemes and protocols. But also it has allowed more people to attack these protocols and weed out the bad ones.
- Cryptography has moved from art to science. Now we have protocols based on mathematical ideas like number theory instead of just creating seemingly complexity functions.
The lectures were taped and may show up on-line someday. I'll let you know if I find them there--definitely recommended viewing.
Friday, June 06, 2003
Back to Chicago
I plan to keep this weblog going in Chicago as long as I have things to say.
Tuesday, June 03, 2003
Foundations of Complexity
Lesson 19: The Immerman-Szelepcsenyi Theorem
In this lesson we will prove the Immerman-Szelepcsényi Theorem.
Theorem (Immerman-Szelepcsényi): For reasonable s(n)≥ log n, NSPACE(s(n))=co-NSPACE(s(n)).
Let M be a nondeterministic machine using s(n) space. We will create a nondeterministic machine N such that for all inputs x, N(x) accepts if and only if M(x) rejects.
Fix an input x and let s=s(|x|). The total number of configurations of M(x) can be at most cs for some constant c. Let t=cs. We can also bound the running time of M(x) by t because any computation path of length more than t must repeat a configuration and thus could be shortened.
Let I be the initial configuration of M(x). Let m be the number of possible configurations reachable from I on some nondeterministic path. Suppose we knew the value of m. We now show how N(x) can correctly determine that M(x) does not accept.
Let r=0 For all nonaccepting configurations C of M(x) Try to guess a computation path from I to C If found let r=r+1 If r=m then accept o.w. rejectIf M(x) accepts then there is some accepting configuration reachable from I so there must be less than m non-accepting configurations reachable from I so N(x) cannot accept. If M(x) rejects then there is no accepting configurations reachable from I so N(x) on some nondeterministic path will find all m nonaccepting paths and accept. The total space is at most O(s) since we are looking only at one configuration at a time.
Of course we cannot assume that we know m. To get m we use an idea called inductive counting. Let mi be the number of configurations reachable from I in at most i steps. We have m0=1 and mt=m. We show how to compute mi+1 from mi. Then starting at m0 we compute m1 then m2 all the way up to mt=m and then run the algorithm above.
Here is the algorithm to nondeterministically compute mi+1 from mi.
Let mi+1=0
For all configurations C
Let b=0, r=0
For all configurations D
Guess a path from I to D in at most i steps
If found
Let r=r+1
If D=C or D goes to C in 1 step
Let b=1
If r<mi halt and reject
Let mi+1=mi+1+b
The test that r<mi guarantees that we have looked at all
of the configurations D reachable from I in i steps. If we pass the
test each time then we will have correctly computed b to be equal to 1
if C is reachable from I in at most i+1 steps and b equals 0
otherwise.
We are only remembering a constant number of configurations and variables so again the space is bounded by O(s). Since we only need to remember mi to get mi+1 we can run the whole algorithm in space O(s).
Wednesday, May 28, 2003
Open Questions from Hopcroft and Ullman
- Are context-sensitive languages closed under complementation?
- If L is context-sensitive is MIN(L) context sensitive? MIN(L) is the set of strings in L who do not have proper prefixes in L.
- Is it decidable whether the complement of a given context-sensitive language is context-sensitive?
- Is it decidable whether two given deterministic context-free languages are equal?
- In 1988, Neil Immerman and Róbert Szelepcsényi independently showed that nondeterministic space is closed under complement. Since CSLs are equivalent to nondeterministic linear space we have that CSLs are also closed under complement. Immerman and Szelepcsényi received the 1995 Gödel Prize for this result. We will cover Immerman-Szelepcsényi in the next Foundations lesson.
- By using Immerman-Szelepcsényi, one can create a nondeterministic linear time algorithm to check that x is in L and that all proper prefixes of x are not in L.
- Trivially true by Immerman-Szelepcsényi.
- This was the hard one. Only in 1997 did Géraud Sénizergues succeed in showing that the problem is decidable. For this work he received the 2002 Gödel Prize. Here is the simplified version of his result, a mere 58 pages.
Thursday, May 22, 2003
Computing's Lost Allure?
First of all this should be no surprise. People, consciously or unconsciously, follow the money. Computer science majors were a very hot property in the late 90's and now they are less so. So less people are going into computer science.
My advice: Don't follow the money. You will be trying to time the market four years down the road. I remember as an undergrad seeing many of my fellow freshman going into Chemical Engineering because it was a hot area. Four years later many of them had trouble finding or keeping a job and had to move to other areas. Some of them even became.......lawyers.
Best to do what you enjoy. If you enjoy it you will have a better chance to succeed. Worry about the job market when you get there.
Wednesday, May 21, 2003
Celebration for Walter Savitch
The Department of Computer Science and Engineering at the University of California, San Diego, is sponsoring an event on June 12, 2003, on the occasion of Professor Walt Savitch's birthday and in recognition of his thirty-four years of teaching Computer Science at UCSD. This event will feature two distinguished speakers, Professor Stephen Cook (University of Toronto) and Professor Aravind Joshi (University of Pennsylvania) with a reception in honor of Professor Savitch concluding the program. The event will take place at Eucalyptus Point (Room B) in Thurgood Marshall College on the UCSD campus. Please see the department web site for a link to more information.
This event immediately follows the STOC conference which is also being held in San Diego as part of FCRC.
Friday, May 16, 2003
Turing Machines and Godel's Theorems
Let A be the set of <M> such that M does not accept the input <M>. By diagonalization techniques we can show A is not recursively enumerable.
Now let us fix a logical theory like ZFC set theory. The actual theory does not matter much. Let B be the set of <M> such that there is a proof in ZFC of the statement "M does not accept input <M>." B is recursively enumerable since we can just try all possible proofs.
Now B is clearly a subset of A so there must be an input <M> in A-B. For this <M>, we have that M does not accept input <M> but there is no proof of this fact. We now have a true mathematical statement with no proof. This is Gödel's first theorem.
We can be more constructive. Since B is recursively enumerable there is some Turing machine N that computes B. Suppose that N accepts <N>. This means <N> is in B which implies there is a proof that <N> does not accept N, a contradiction.
So <N> cannot accept N which means <N> is not in B. So we have now constructed an N such that the statement "N does not accept <N>" is true but not provable.
But wait a minute, didn't I just give a proof that N does not accept <N>? Actually I had to make the assumption that ZFC is consistent. If ZFC is inconsistent then every statement (true or false) is provable so B would not be a subset of A and my whole argument falls apart.
If ZFC could prove that ZFC is consistent then I would not have to make any assumption and would have a contradiction. Thus ZFC cannot prove its own consistency. This is Gödel's second theorem.
Logicians tell me I'm cheating: I had to assume something technically stronger than consistency for this argument to work. Still these proofs illustrate the amazing power of Turing machines to make Gödel's theorems easier to understand.
Wednesday, May 14, 2003
Foundations of Complexity
Lesson 18: Savitch's Theorem
Unlike what we believe for time, there is a polynomial relation between deterministic and nondeterministic space.
Savitch's Theorem (1970): NSPACE(s(n)) is contained in DSPACE(s2(n))
Proof: Recall the definition of tableau (as described in Lesson 12). Let N be a nondeterministic machine that uses s(n) space. We can represent the computation of an input x of length n by a tableau where each configuration has size O(s(n)) and there are at most m = cs(n) configurations for some constant c.
We will create a deterministic machine M(x) to determine whether the tableau is proper and thus N(x) accepts. First we need a subroutine CHECK to determine whether one configuration can reach another in 2t steps. We do not have enough space to write the entire tableau but instead we do a divide and conquer approach: Try all possible middle configurations and recurse on each half.
CHECK(CONF1,CONF2,t)
\* Output TRUE if CONF1 can get to CONF2 in at most 2t steps *\
If t=0 then
{if (CONF1 = CONF2
or CONF1 goes to CONF2 in one step on machine N)
then output TRUE else output FALSE}
For each CONF
{If CHECK(CONF1,CONF,t-1) and CHECK(CONF,CONF2,t-1) then output TRUE}
Output FALSE
We can implement CHECK by only having to store a constant number of configurations at each level of the recursion with a recursive depth of t for a total space of O(ts(n)).
Let CONF0 be the initial configuration encoding x. We can now give our main routine.
MAIN
Let r be the smallest integer at least log2(cs(n))
For each CONF in an accepting state
{If CHECK(CONF0,CONF,r) then output TRUE}
Output FALSE
Total space used: O(log2(cs(n))s(n)) = O(s2(n)).
Monday, May 12, 2003
The Nerd Shot
Computer scientists, especially in Europe, love to take pictures of other computer scientists. Problem is computer scientists are not particularly photogenic nor, for the most part, are they good photographers. The worst is the group photo, where we are herded like cattle to some enclosed place where we stand until all the strays are rounded up and finally the photo is taken, sometimes several times by several people.
Some things have changed over time. Most photos are now taken digitally and get posted quickly, sometimes before the conference is over. My kids enjoy playing "Where's Daddy?" especially if I am out of town. But still the group photo experience remains the same.
I was talking to my wife on the phone at the last Dagstuhl and told her it was time for the nerd shot. She said I should fix my hair and I replied "What? You want me to stand out?"
Without further adieu here is that nerd shot.
Thursday, May 08, 2003
Foundations of Complexity
Lesson 17: Space Complexity
In addition to time, computer scientists also worry about the memory or space that a Turing machine uses. Roughly one can measure space as the number of tape squares used by at Turing machine. We would like to talk about space bounds like log n and still be able to read the whole input so we need a slightly different model.
We now allow our Turing machine to have a read-only input tape, one or more work tapes and for a Turing machine computing a function, a write-only output tape. On the input tape the head can move back and forth but it cannot change the values in any cell. On the output tape the head can only move right, writing as it moves. The amount of space a Turing machine uses on input x is the number of cells of the work tape that it uses. We will assume a space bound of at least log n since we need log n bits to describe the location of the the input head pointer.
In real world terms, think of your computer accessing "the internet". You can still reach many pages even though you cannot store the entire internet on your computer.
Any machine that runs in time t(n) clearly also runs in space t(n). If a machine uses space s(n) and it halts then it will halt in time cs(n) for some constant c. Otherwise the machine will repeat a configuration and run forever.
We define the classes DSPACE(s(n)) as the set of languages that are accepted by Turing machines using at most O(s(n)) space. NSPACE(s(n)) is the same for nondeterministic machine. We will always assume s(n)≥log n and s(n) is an easily computable function.
Common space classes include L=DSPACE(log n), NL=NSPACE(log n), PSPACE=∪kDSPACE(nk) and NPSPACE=∪kNSPACE(nk).
Unlike the P versus NP question, we know quite a bit about the relationship of deterministic versus nondeterministic space through the following two results.
- Savitch's Theorem: NSPACE(s(n))⊆DSPACE(s2(n)). In particular this means NPSPACE = PSPACE.
- Immerman-Szelepcsényi Theorem: If L is in NSPACE(s(n)) then the complement of L is also in NSPACE(s(n)).
Tuesday, May 06, 2003
Universal Search
Actually this is an idea due to Levin. Let p1, p2, ... be a list of all programs. On some input m simulate program p1 for half of your computation time, p2 for a quarter of the time, p3 for an eighth of the time, etc., until one of these programs outputs a factor of m. If pi is the fastest algorithm for factoring then our algorithm will run in time at most 2i times the running time of pi. The multiplicative factor 2i is independent of m but unfortunately could be quite large.
Marcus Hutter gives another algorithm that has a multiplicative factor of 5 but has a large additive constant. The trick is to spend some of your time searching for a proof that an algorithm is correct and runs in certain amount of time. You then only need to simulate the provably fastest algorithm found so far.
Hutter's algorithm works only as fast as the provably best algorithm with a provable running time. It could very well be the case that there is some good heuristic for factoring that does not have a provable running time or proof of correctness. Levin's technique will capture this case.
Of course, neither of these papers gives a practical algorithm as the constants involved go beyond huge. Nevertheless it is still interesting to see the theoretical possibilities of universal search.
Sunday, May 04, 2003
FCRC
The press didn't show but with some cross-collaboration the conference was considered a mild success. After stops in Philadelphia and Atlanta, FCRC returns to San Diego June 7-14. The 2003 FCRC includes the main spring theory conference (STOC), Electronic Commerce (EC), Computational Geometry, Principle and Practice of Parallel Programming and many others. Registration deadline is May 7.
For the first time the Complexity Conference will not be part of FCRC as 2003 is a Europe year for us. I am planning to attend FCRC for STOC and the EC meeting. If you attend FCRC stop by and say hi.
Thursday, May 01, 2003
History's Loss/Mathematics' Gain
Let me finish off this centennial week with a story of Kolmogorov that I have heard from different sources so some variation of this story is likely true. Kolmogorov initially wanted to be an historian. He looked at the question as to whether taxes in Russia in the middle-ages were collected at the house level or at the village level. He analyzed tax data and showed that that data had a much simpler description if taxes were collected at the village level. (You can see the seeds of Kolmogorov complexity here). He presented these results to the history faculty to great applause. Asking them whether he could publish such a paper he was told, "You only have one proof. You cannot publish in a history journal without at least two more proofs of your hypothesis." And so Kolmogorov left history for a field where one proof suffices.