How did Watson get so smaht?

At yes­terday afternoon’s Pro­files in Inno­va­tion lec­ture, IBM Watson cre­ator David Fer­rucci explained (very quickly I might add — that man talks fast) how the super­com­puter came to think like a human and beat Jeop­ardy cham­pion Ken Jen­nings at the TV quiz show last year.

Okay, maybe “think” isn’t quite the right word. But this is what I found so inter­esting: the team didn’t set out to build a com­puter that thinks like a human at all, but in the end, to Jen­nings’ own sur­prise, Watson’s process of rea­soning, which is based com­pletely on math­e­mat­ical algo­rithms, is very sim­ilar to a human’s process of reasoning.

To get Watson to “think,” the team used machine learning, which really isn’t all that new or exciting on its own. Amazon and Net­flix, for example, have been doing this for years. Each time you give the web­site feed­back on your pref­er­ences, it learns a little more about you and gets better and better at making book or movie rec­om­men­da­tions that you will actu­ally enjoy.

But with Watson it wasn’t that simple. Because Jeop­ardy ques­tions use com­plex lan­guage and aren’t at all straight­for­ward, Watson had to dig into masses and masses of data to gen­erate answers. The data, Fer­ucci explained, wasn’t simple five star rat­ings, either. It came from long tex­tual pas­sages from a variety of sources. Watson had to sift through all the pas­sages the team gave it over four years of training and use some­thing called “plau­sible infer­ence” to figure out what the right answer could be.

Example: Lin­coln once said: “Trea­sury sec­re­tary Chase just sub­mitted this to me for the third time. Guess what, pal? This time I’m accepting it.” What is “this”? Someone else in the audi­ence, not me, quickly said “his res­ig­na­tion,” which the man was able to infer from the sen­tence based on his knowl­edge that Lin­coln was a pres­i­dent, that trea­sury sec­re­taries only submit a few things to pres­i­dents, and from the tone of voice it prob­ably wasn’t some great bill pro­posal. When this ques­tion was posed to a class­room of six graders the common answer wasn’t “his res­ig­na­tion” but “a face­book friend request.”

It all comes down to our expe­ri­ence. A sixth grader has a much dif­ferent expe­ri­ence of the world than an old dude who’s lived through many frus­trating polit­ical administrations.

And so where does that leave a computer?

The more infor­ma­tion the team feeds the com­puter the more it has the chance to learn from it. But how does it then decide which infor­ma­tion to pay atten­tion to and which to ignore. Fer­rucci said that Watson was made up of not just one machine learning algo­rithm, as in the case of Net­flix or Amazon, but sev­eral hun­dred. Each con­tributes a tiny bit of infor­ma­tion to the final answer, and each has a dif­ferent con­fi­dence level.

Say ten of us are sit­ting around a table and are asked the same ques­tion. Each of our expe­ri­ences will lead us to a sep­a­rate, although pos­sibly iden­tical, answer. And each of us will have a dif­ferent level of con­fi­dence in our answer. Watson aver­ages up all of those answers and the con­fi­dences they are sub­mitted with to gen­erate his final answer.

And then of course it gets even more inter­esting. Jeop­ardy players only answer ques­tions a cer­tain number of times. The typ­ical winner answers around 50% of the ques­tions. They know what they don’t know. And when they do answer, they do so cor­rectly a majority of the time. So Watson had to mimic this bal­ancing act. Yet another algo­rithm taught it to answer ques­tions when the average con­fi­dence was above a cer­tain threshold. And of course, that threshold would change based on how well he was doing in the game.

So, I have to run to the annual RISE Con­fer­ence now (Research, Inno­va­tion, and Schol­ar­ship Expo) and find out which of our North­eastern stu­dents will be the next David Fer­rucci. But isn’t it inter­esting that while the Watson team didn’t set out to mimic human thinking behav­iors, the best way for it to answer Jeop­ardy ques­tions nat­u­rally evolved to do just that?