The Noisy Channel Model for Unsupervised Word Sense Disambiguation

Seminar presentation at ITU on our CL paper.
Full post...

A tutorial on language modeling with SRILM

Full post...

From Eternity to Here by Sean Carroll

Why is the future different than the past? The fundamental rules of physics seem as symmetric in time as they are for different directions of space, yet you can't unscramble an egg and you can't remember the future. The answer seems to lie in the concept of entropy, which tends to increase over time in a closed system unlike energy and momentum which stay constant. Yet, entropy is one of these concepts that lead to quite a bit of confusion, sort of like (and sometimes for the same reasons as) probability. Sean Carrol's wonderful book tackles the mysteries of entropy and arrow of time. Here are some entropy puzzles and some further reading:

Entropy puzzles:

• If you find yourself in a universe with low entropy it is as likely to have come from a higher entropy state as it is to evolve into a higher entropy state. So how do you know you are not a Boltzmann brain?
  
• Take a box with a partition in it, with gas A on one side, gas B on the other side, and both gases are at the same temperature and pressure. Remove the partition. If gas A and B are different gases, there is an entropy that arises due to the mixing. If the gases are the same, no additional entropy is calculated. What if you thought they were the same gas and years later it was discovered that they happened to be two different isotopes? (See Gibbs paradox and E.T. Jaynes' paper). More generally this microstate / macrostate business seems completely user defined and arbitrary, so how can it have real physical effects?
  
• In a reversible system there must be just as many paths that decrease the entropy as that increase the entropy. Why don't we observe as many of the first type as the second?
  
• A gas squeezed in the corner of a room will tend to spread thereby increase its disorder and entropy. If we add an attractive force like gravity matter seems to clump together rather than spread out. How does clumping together increase entropy?
  
• A rotting plant turns into dust and gas which increases disorder and entropy. A seed turns a bunch of gas and dust into a full grown tree which seems to decrease entropy. This can only happen because the seed is not a closed system and is using the energy from the sun and ends up increasing the overall entropy of the universe at the end. When, how, and why does this type of thing happen?
  

Further reading:

• From Eternity to Here: Sean Carroll's book that inspired this post.
  
• Labyrinth of Time: Michael Lockwood's book on the arrow of time with a bit more philosophy and a bit less black hole physics.
  
• Time and Chance: David Z. Albert's book gives one of the clearest discussions of thermodynamics (both classical and statistical) and its relation to the arrow of time.
  
• Good and Real: Gary L. Drescher's book that tackles not only the arrow of time, but quantum indeterminacy, consciousness, free will, and ethics. This is one of my favorite books which inspired a few earlier posts.
  
• E. T. Jaynes has a number of articles that clarify some of the mysteries. See also these links.
  
• Permutation City: One of my favorite sci-fi novels by Greg Egan explores the nature of time, simulation and reality. Maybe instants are not ordered in time at all, the state of our short term memory seems to give this impression.
  
• Max Tegmark at MIT came up with the Mathematical universe hypothesis which is reminiscent of the Dust theory that underlies Permutation City. See also Hans Moravec's essay.
  

Full post... Related link

Bologna Translation Service (2011-2013)

The Bologna Translation Service is designed for semi-automatic translation of course syllabi and study programmes from 9 languages - Dutch, English, Finnish, French, German, Portuguese, Spanish, Swedish and Turkish - into English. It helps speed up translation and revision processes, reducing costs and human resources at the same time. The project has received funding from the European Community (ICT-PSP 4th call) under Grant Agreement no 270915 from March 2011 to March 2013. Project partners included Cross Language (Belgium), Convertus (Sweden), ALS - Traslan Teoranta (Ireland), Eleka Ingeniaritza Linguisitikoa (Spain), and Koç University (Turkey).

Full post...

Plausibility vs. Inference

Here are two examples of common sense judgement:

(1) I saw the Statue of Liberty flying over New York. => I was flying over New York.
(2) I gave the book to Mary. => Mary has the book.

The first one involves syntactic disambiguation. Either the subject or the object could be doing the flying (consider "I saw the airplane flying over New York.") Our "common sense" tells us that the Statue is too big to fly, so I am the more likely one flying.

The second one involves a semantic inference. The meaning of give involves a physical transfer or a transfer of possession, as a consequence the item given ends up with the recipient. Our "common sense" is full of such little factoids (here is another: "Oswald killed Kennedy." => "Kennedy is dead.") which let us see beyond what is explicitly stated in the text.

I want to emphasize that these examples are qualitatively different and calling them both "common sense judgements" may be confusing. The first one is a plausibility judgement (which is more likely to fly: me or the statue?). The second one is an exact inference, i.e. "give" definitely causes transfer and "kill" definitely causes death. To solve the first one we need a model of what is more likely to be happening in the world. To solve the second one we need more traditional inference of what entails what.

Disambiguation problems in computational linguistics (word sense disambiguation, resolving syntactic ambiguities, etc.) rely on plausibility judgements, not exact inference. A lot of work in AI "common sense reasoning" will not help there because traditionally reasoning and inference work focus on exact judgements.

As far as I can see nobody is working on plausibility judgements explicitly. Researchers use corpus statistics as a proxy to solve disambiguation problems. This may be obscuring the real issue: I think the right way to do linguistic disambiguation is to have a model of what is plausible in the world.

Full post...