Turing Test and Loebner Prize Competition

A couple of weeks ago, the 2006 Loebner Prize competition was held. Back when I was at Harvard in the early 1990s,  the annual Loebner Prize Competition was created as the first real-life version of the "Turing Test," described in Turing's article "Computing Machinery and Intelligence" to answer the question "Can Computers Think?" Turing wrote:

It is proposed that a machine may be deemed intelligent, if it can act in such a manner that a human cannot distinguish the machine from another human merely by asking questions via a mechanical link.

The test is a natural extension of his earlier work on Universal Turing Machines, which can simulate any other machine, to the human mind. The test, however, is controversial. Searle famously counterargued with the Chinese Room experiment. Also, Turing himself made an unsuccessful prediction that the Turing test would be passed by 2000, but tech visionary Ray Kurzweil has willingly bet that Turing Test will be passed by 2029.

The first competition made big news on campus, especially since it was held locally, and I followed the results of the initial competition closely. Since I had a prior interest in AI and natural language processing, I envisioned that one day that I might be the one to actually win the $100,000 prize; unfortunately, the terms of the grand prize has since been expanded to include audio and visual input. The direction of my work has been increasingly intersecting with the aims of the competition, and so maybe one day (perhaps in ten years) it might actually compete.

Instead of summarizing the competition, I'll refer to the text in the Loebner Prize website:

The Loebner Prize for artificial intelligence ( AI ) is the first formal instantiation of a Turing Test. The test is named after Alan Turing the brilliant British mathematician. Among his many accomplishments was basic research in computing science. In 1950, in the article Computing Machinery and Intelligence which appeared in the philosophy journal Mind, Alan Turing asked the question "Can a Machine Think?" He answered in the affirmative, but a central question was: "If a computer could think, how could we tell?" Turing's suggestion was, that if the responses from the computer were indistinguishable from that of a human,the computer could be said to be thinking. This field is generally known as natural language processing.

In 1990 Hugh Loebner agreed with The Cambridge Center for Behavioral Studies to underwrite a contest designed to implement the Turing Test. Dr. Loebner pledged a Grand Prize of $100,000 and a Gold Medal (pictured above) for the first computer whose responses were indistinguishable from a human's. Such a computer can be said "to think." Each year an annual prize of $2000 and a bronze medal is awarded to the most human-like computer. The winner of the annual contest is the best entry relative to other entries that year, irrespective of how good it is in an absolute sense.

Here's a sample transcript from 1996 contest winner. There are many other transcripts available from the prize website. A conversation with some of these contestants can be had online: [TuringHub] [ALICE] [Jabberwacky] [Others]

My computer science professor Stuart Shieber, who, by the way, stoked my interest in natural language, wrote an critique of the contest, "Lessons from the Restricted Turing Test," in the Communications of the ACM journal.

Stuart noted the technology used by most, if not all, of the contestants are still very primitive. They are basically variants of the 1966 computer program ELIZA, just with a larger database of responses. These programs don't do any natural language parsing--relying instead of simple string searches and manipulation--and don't do any logical inferencing.

For example, a long time winner of the annual competition was the ALICE (ArtificiaL IntelligenCE) chatterbot. I downloaded the open-source software six years ago and was dismayed to see how primitive its technology was. Essentially, the program consisted of a database of common questions (with some wildcard support) and canned answers. Another winner, MegaHal, uses statistical methods (Markov models) to generate responses based on prior data such as movie dialogues, encyclopedias, popular quotatons, and hand-crafted sentences. 

Despite the simplistic technology, a few judges over the history of the competition have been fooled into thinking a computer to be a person, and, interestingly, some humans have been thought to be computers. Curiously, the original ELIZA did fool the assistant of its creator, Weizenbaum, into revealing personal information as did another low-tech program in this webpage detailing "How my program passed the Turing Test!"

The Loebner Prize Competition has mostly relied on tricks, such as simulating typing speed and entering non-sequitors, rather than smarts. If I ever had some free time in the distant future and joined the competition, I would rely on genuine attempt to replicate intelligence. It's the thought that counts.

Although Turing's 2000 prediction of a successful Turing Test failed to come true, Turing did rightly predict that computer programs would eventually defeat men in chess. The ability to play chess was Turing's hallmark example of human intelligence. Interestingly, "Turing Test and Intelligence," claims that in chess the Turing Test has already been passed.

Kasparov claims to be able to distinguish computer play from human play. During a simultaneous event over the Internet, he once stopped playing a certain game, saying that he was up against a computer (it was supposed to be human opposition). He was losing at the time! Kasparov has also claimed, while losing against a computer, that the computer was being given human assistance!!

It's interesting to see modern variations of the Turing Test include CAPTCHA's for preventing robots from posting spam, and Amazon's mechanical Turk. (Turk, by the way, was a chess playing machine centuries ago that was powered by a hidden human being.) 

Comments

[Wes]

John McCarthy, AI Pioneer, refutes Searles argument in http://reddit.com/goto?rss=true&id=jnf7

[WaterBreath]

It is interesting to me that Turing, being such a respected mathematician, logician, etc., should have had such an anthropocentric view of intelligence and AI.

Why should interaction with an intelligent machine be indistinguishable from an intelligent human? If the machine turns out to be "smarter", i.e. it doesn't make grammatical errors, it isn't restricted to understanding just a few cultural contexts, that would certainly reveal it as a machine, without ruling out intelligence.

Another interesting line of thought is to consider the nature of intelligence in other forms of life. What if humans were not the only sentient beings on earth? Would interaction with an intelligent dolphin be indistinguishable from that with an intelligent human? What about an intelligent dog? A bird? I would expect not. And why should expect otherwise from a machine, which is a vastly more different "organism"?

I am a strong believer that the nature of interaction between two beings is dependent on countless subtle contexts. Not the least of which is the physical forms of the entities. I would expect a sentient animal to seem much more "human" than a truly sentient machine. If for no other reason than that a machine, as we know it now, has no encpasulating tactile surface, no motility, no articulated limbs. Additionally, the senses and actuations it does have at its disposal work very different from humans', and in some cases aren't possessed by humans at all. The interface between a machine and the world has very little in common with a human, and so it means that a machine leads a very different "life" than a human. Which would again point to an inherent "distinguishability".

I would expect none of this to prevent us from creating a truly sentient machine. Even one that we can converse with, teach, and learn from.... With the caveat that, in order to achieve it, we cannot ignore the inherent differences. To focus on the humanity of an intelligent machine, we restrict ourselves, and it. We are placing artificial limitations on the endeavor. Yes, we must give the machines at least some common interaction mechanism (either audio or visual is in this respect a minimum, I think). But we should also not shy away from incorporating all of the machine's sense and actuation mechanisms to their full potential. An intelligent machine's visual communication should not be restricted to sign language motions displayed on its screen, for example, but should utilize the full plasticity of the monitor as a communication medium. And it should consider its network connection as an equally valid form of direct and "natural" communication with other devices.

However, if we choose to pursue this broader path to "designed" intelligence, we should not be surprised if the result is quite readily and unapologetically "distinguishable" from humanity.

[DarkMinerva]

In his book Behind Deep Blue: Building the Computer that Defeated the World Chess Champion (ISBN 0-691-09065-3), Feng-hsiung Hsu explains that the whole Human vs. Computer contest is really a competition between two humans (Human vs. Human Programmer). I've always found it interesting that a pioneer in artificial intelligence would emphasize such a disclaimer (at least to the level Hsu does in his book). For instance, he writes about his aversion for the media touting the famous Deep Blue vs. Kasparov chess matches as evidence computers are becoming smarter than humans. For somebody, like me, who believes computers will someday be more intelligent than humans, Hsu’s book was sobering to say the least.

Today I believe we will not see true artificial intelligence until we have a full understanding of our own intelligence. How can we even begin to match something that we don’t have a full picture of? Until we have a thorough understanding of the human brain, all AI attempts will be a bunch of guesses and “tricks”.

[Johnnymushio]

I agree with waterbreath about us not really knowing a thing about our own brain. It was always my own response to the question.

[huoyangao]

...
In Turing Test Two, two players A and B are again being questioned by a human interrogator C. Before A gave out his answer (labeled as aa) to a question, he would also be required to guess how the other player B will answer the same question and this guess is labeled as ab. Similarly B will give her answer (labeled as bb) and her guess of A's answer, ba. The answers aa and ba will be grouped together as group a and similarly bb and ab will be grouped together as group b. The interrogator will be given first the answers as two separate groups and with only the group label (a and b) and without the individual labels (aa, ab, ba and bb). If C cannot tell correctly which of the aa and ba is from player A and which is from player B, B will get a score of one. If C cannot tell which of the bb and ab is from player B and which is from player A, A will get a score of one. All answers (with the individual labels) are then made available to all parties (A, B and C) and then the game continues. At the end of the game, the player who scored more is considered had won the game and is more "intelligent".
...

http://turing-test-two.com/ttt/TTT.pdf