How to keep them coming back for more

Meet Tanya — the rela­tional agent Var­doulakis used for her study. Image cour­tesy of Think­stock and Laura Pfeifer Vardoulakis.

It totally blew my mind.” That’s what grad­uate stu­dent Laura Pfeifer Var­doulakis said of her encounter with work taking place in Tim­othy Bickmore’s lab in the Col­lege of Com­puter and Infor­ma­tion Sci­ence. Bick­more is one of the few researchers starting to develop med­ical tech­nolo­gies that target patients and indi­vid­uals instead of clin­i­cians. “When I came here I was still thinking about it from the doctor’s side,” said Var­doulakis, who worked at a med­ical soft­ware com­pany before coming to grad­uate school. “He was working on it from the patient side and it blew my mind.” Bickmore’s team designs computer-​​based “rela­tional agents,” or ani­mated com­puter char­ac­ters that look and act like humans. They’re nurses and other health care workers that live inside the com­puter screen and do every­thing from walking patients through hos­pital dis­charge papers to pro­viding com­pan­ion­ship to older adults. They stand to do a lot for pre­ven­ta­tive med­i­cine but Var­doulakis real­ized there was still more to be done to opti­mize their impact.

The two main issues are how do you keep people using the technology–or wanting to use it–over a long period of time, and then, two, how do you get good quality data if you’re not using a sensor?” A lot of things, such as ques­tions about emo­tional state, fall into this cat­e­gory of stuff that can’t be easily quan­ti­fied. Var­doulakis wanted to design a study that could shed light on whether rela­tional agents could help in some of these more abstract issues. Per­haps a series of ques­tions pre­sented by an avatar would be more com­pelling than a simple, internet-​​based text survey. If so, then per­haps avatars would be a good tool for lon­gi­tu­dinal studies in which par­tic­i­pants are asked to answer ques­tions at reg­ular points over a long time period. But she also wanted to look at some other fac­tors as well. For instance, if survey par­tic­i­pants are offered money are they more likely to keep coming back and answering the ques­tions each week? Or what if they’re offered mean­ingful feed­back about their responses, would that do the trick?

After she con­ducted sev­eral studies with older adults in the Boston area, for her thesis Var­doulakis decided to con­duct a larger-​​scale study with the com­mu­nity right out­side her lab door: col­lege stu­dents. The number one health con­cern for col­lege stu­dents is alcohol con­sump­tion, so she chose to make this the topic of her survey. Starting with a stan­dard ques­tion­naire used by the Office of Pre­ven­tion and Edu­ca­tion at North­eastern, or OPEN, Var­doulakis designed an online survey as well as a rela­tional agent that asked the par­tic­i­pants ques­tions directly with its auto­mated voice. Half of her 375 par­tic­i­pants used the agent, while the other half used the text survey. She also ran­domly assigned each person to one of two feed­back groups and one of two mon­e­ti­za­tion groups. Each time they returned for another week of inqui­si­tion, half of the par­tic­i­pants would receive a com­pletely non-​​judgemental analysis of their survey answers the pre­vious week, while the other half received no feed­back All of the par­tic­i­pants were entered into a drawing for a $25 amazon gift card each week, but only half were reminded of the drawing each time they returned.

So, what did all of this tell Var­doulakis? First, she found that two things seemed to impact a person’s adher­ence to the study. First, the dura­tion of their first ses­sion was the biggest deter­mi­nant. Regard­less of the delivery method and whether or not they got infor­ma­tional feed­back or reminders about the gift card drawing, par­tic­i­pants who took less than 200 sec­onds to answer the first survey were more likely to stick with it over the dura­tion of the 16-​​week study.

The second deter­mi­nant was whether or not the par­tic­i­pant was a drinker. If they reported drinking at least one alco­holic bev­erage during the course of the study, Var­doulakis checked them off as drinkers. In the text group, drinkers were more likely to stick with it than non-​​drinkers and vice versa for the agent group. When it comes to self reported data, it’s well known that people tend to do some­thing called “max­i­mizing their social desir­ability.” “We’re more likely to up-​​play our pos­i­tives and down­play our neg­a­tives,” said Var­doulakis. This makes some amount of sense when inter­acting with a real human…we don’t like to be judged. But the same thing seemed to be hap­pening with the rela­tional agent. There should have been no sta­tis­tical dif­fer­ence between the text and agent group, but the fact that there was sug­gests that par­tic­i­pants may have been feeling sub­con­sciously judged by the com­puter screen char­acter. I can under­stand that — Bickmore’s team has spent years trying to make their agents as life-​​like as pos­sible. They even study real clin­i­cians and try to emu­late the facial expres­sions and man­ner­isms people use when inter­acting with patients.

Var­doulakis wanted to believe that feed­back would be a pow­erful tool to pro­mote adher­ence as well as pro­viding quality data. Unfor­tu­nately, she said, it kind of turned out the oppo­site. Par­tic­i­pants who were told they would receive feed­back about their answers tended to report fewer alco­holic bev­er­ages than the non-​​feedback group. They were also less likely to report neg­a­tive con­se­quences as a result of their drinking (e.g., missing a class, get­ting into a fight, etc). Keep in mind this was on their very first encounter with the survey, so they haven’t even seen that feed­back yet, they just know it’s gonna happen some day. “It’s almost a social desir­ability issue, where you’re pre­senting your­self to your­self,” said Vardoulakis.

Overall, the study pro­vided some good infor­ma­tion for people designing novel patient-​​facing health­care tech­nolo­gies. It’s obvi­ously very impor­tant for people to main­tain anonymity when dealing with sen­si­tive sub­jects like alcohol use, this has been known for a long time in the field. But devel­opers may not have real­ized just how impor­tant. Social desir­ability isn’t only rel­e­vant in the face of our fellow humans, it’s also appar­ently a factor when we’re dealing with machines.

The field of patient-​​facing health­care tech­nolo­gies is posi­tioned to make a big impact on the field. But in order for any of these pro­grams and devices to be truly valu­able, espe­cially from a health­care per­spec­tive, they need to be sup­ported by reli­able studies. I think that this one from Var­doulakis drives that point home. We might have some strong biases about what we expect to be suc­cessful but until we put it out there and test it, we can’t say a thing. Of course, Var­doulakis’ work is only a begin­ning and addresses a very spe­cific pop­u­la­tion. More work needs to be done in the same vein to show exactly where and how these tools can be useful.

And what about those first two ques­tions she asked her­self (how do we keep people coming back and how do we ensure they pro­vide mean­ingful data)? Well,  a handful of things: For one, first impres­sions matter – the assess­ments should not take a lot of effort to com­plete, said Var­doulakis. Sec­ondly, when dealing with poten­tially sen­si­tive health topics, text-​​based inter­faces are prefer­able over agent-​​interfaces.. Third, pro­viding feed­back might back­fire – people might sensor them­selves more if they know they’ll receive feed­back. And finally, although I didn’t men­tion it above, the mon­e­tary incen­tive was pow­erful in get­ting people to com­plete the survey once, but its effect wore off rapidly and didn’t seem to matter long-​​term.