All models are wrong, some are useful’

Image via FlexSim Healthcare.

Image via FlexSim Healthcare.

There are good models and there are bad models.

For example, how many times have you pur­chased what amounted to be a garbage sack because it looked so much like a beau­tiful dress on the air­brushed model in the pic­ture online? If it’s half as many times as I have, then you know that models are not the real deal.

Whether it’s an archi­tec­tural model of the new building going up down­town, the model train you’ve got set up in your par­ents’ base­ment, or the man­nequins and air­brushed photos in our glossy mag­a­zines, models are some­times fairly inaccurate.

In con­trast, last week med­ical pro­fes­sionals trav­eled from five states (Mass., Conn., R.I., N.Y., Maine) to a health­care sim­u­la­tion work­shop at North­eastern to learn how to build models of health­care sys­tems that actu­ally are rep­re­sen­ta­tive and useful to help them improve crit­ical problems.

When it comes to a com­pu­ta­tional model, “just because it’s not real, doesn’t mean it can’t be used in real and useful ways,” said indus­trial and mechan­ical engi­neering pro­fessor James Ben­neyan, director of the Health­care Sys­tems Engi­neering Insti­tute, at a recent two-​​day work­shop for Boston area health­care sys­tems per­sonnel to teach them how to use com­puter sim­u­la­tion to improve crit­ical prob­lems. He quoted a pop­ular adage among his lot, uttered some years ago by sta­tis­ti­cian George E. P. Box: “All models are wrong, some are useful.”

The work­shop was part of Benneyan’s grant from the Cen­ters for Medicare and Med­icaid to create a national net­work of cen­ters that partner with local health­care sys­tems to help improve crit­ical prob­lems with sys­tems engi­neering methods. While I was there I met a few people from the Center for Clin­ical Excel­lence at Brigham and Women’s Hos­pital, an in-​​house con­sul­tancy group that aims to help the hos­pital estab­lish more effec­tive quality, safety, and oper­a­tions mea­sures. They told me that they often use sim­u­la­tion among their clin­ical staff to look at prac­tices and iden­tify more effi­cient approaches. But for them, sim­u­la­tion has always been of the “table-​​top” sort.

In health­care sys­tems engi­neering, prac­ti­tioners can draw from a number of tools in their arsenal. Most of the prob­lems they encounter, Ben­neyan likes to say, are pretty basic and can be solved with simple approaches. But when you get into the com­plex, weedy problems—like how to better manage of ICUs and emer­gency depart­ments, schedule pri­mary care appoint­ments, or redesign screening poli­cies for common diseases—basic approaches no longer cut the mustard.

I’d never heard of table-​​top sim­u­la­tion before the work­shop, but it’s exactly what it sounds like. You sim­u­late the real world sce­nario on the table: Think of Stannis Baratheon’s painted table in Game of Thrones or the XO’s war room map in Bat­tlestar Galac­tica. You can play out what did happen and then make changes to see how it might have hap­pened if you’d done some­thing differently.

This is all well and good until the number of vari­ables in your scenario—and their unreliability—starts rising. In the health­care world, these are often quite sig­nif­i­cant, as it deals with per­haps the least reli­able entity on the planet: humans.

How do we design good processes given there’s so much out there we can’t con­trol?” Ben­neyan asked the work­shop par­tic­i­pants. Well, you stop playing around with the table-​​top-​​models and pull out the big guns: com­puter models.

Lou Keller, an expert in com­puter sim­u­la­tion who has been involved in sim­u­la­tion of health­care sys­tems his entire career span­ning mil­i­tary and civilian health­care sys­tems, walked the par­tic­i­pants through the FlexSim Soft­ware which he helped develop, as an example tool for these more com­plex sit­u­a­tions. The number-​​one rule, he said, is to always model what is sup­posed to happen first,” he said. “Once you know what’s sup­posed to happen then you can go looking for what it is that’s keeping it from happening.”

Health­care costs our nation $3 tril­lion each year. Ben­neyan says that one third of that could be elim­i­nated by dealing with the waste and inef­fi­cien­cies built into the system. The most common method for doing so is sim­u­la­tion. “These are really crit­ical prob­lems we are trying to help our health­care col­leagues with, so I’m thrilled with the interest and engage­ment,” said Ben­neyan. “For me, this also was a lot of fun, as it’s the first time I’ve taught with someone who was a bit of a mentor when I was starting my career.”

The mis­sion of the Health­care Sys­tems Engi­neering Insti­tute at North­eastern is to have broad national impact on health­care improve­ment and redesign through research, edu­ca­tion, and appli­ca­tions of sys­tems engi­neering methods.