Boulder Community Hospital computer records back on line - but something does not add up

This post is in followup to my March 20, 2013 post "Boulder Community Hospital computer system crash: Either you're in control of your information systems, or they're in control of you".

At a March 24, 2013 Denver Post article "Boulder Community Hospital computer records back on line" the following statements are made:

The computer system that Boulder Community Hospital uses to manage patient records, which had been down for almost two weeks, is now up and running again, hospital officials said Saturday.

Meditech, the system used by the hospital to manage patient records, went down March 12 and affected the hospital, its Foothills campus, eight laboratories and six imaging centers. It was put back into full service at about 3 p.m. Friday, according to hospital spokesman Rich Sheehan.

Sheehan said an investigation showed the outage was a result of a malfunction in one of the main computer servers ... the hospital has replaced the hard drives for the server that failed and are inspecting the remaining servers ... [the failure] resulted in the system being unable to access patient information. The malfunction affected both the primary server and a backup server kept off-site.

A hard drive failure led to a two-week outage of an entire EHR system and its offsite backup server?  A mission-critical system in a hospital is so fragile that a hard drive failure caused a two week outage?

If so, that itself shows, at best, poor overall system design with regard to reliability and redundancy (any server worth its salt has hard drives in a failure-tolerant configuration e.g., RAID), but also is not quite credible on its face.  A remote server should not be taken down by the failure of a local server.  I suspect the failure was more than just a hard drive failure, including software bugs or configuration errors, mass hardware and/or network failure, or even sabotage.

The following statement also lacks believability on its face:

... All patient data was recovered except for an eight-hour period the day of the outage. Sheehan said the hospital had to re-create, re-enter and validate the patient information for that eight-hour period before the system could resume normal operations.

If an information system is down for two weeks, there's two weeks worth of data lost.

... Sheehan said the hospital has replaced the hard drives for the server that failed and are inspecting the remaining servers. The hospital is also now doing data backups every four hours as opposed to every six hours, and is planning on doing hourly backups by the end of the week.

Replacing a failed hard drive is an inadequate precaution.  A 'system redundancy makeover' seems in order for when the next hard drive fails.   Hard drives have a very well known MTBF (mean time between failure) and annual failure rate.  (The very Seagate ST3750528AS hard drive in the PC I am typing this blog post on has an Annualized Failure Rate of 0.34%, per the manufacturer's publicly-available literature.) 

... An independent consulting firm also has been hired to conduct an investigation. The hospital said it expects a report within a few weeks. 

As other organizations are using Meditech products, Joint Commission Safety Standards (as I wrote in a 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards" available at this link) call for sharing the results of that report with other organizations.  I had discussed this letter numerous times with senior Joint Commission leadership.

Will sharing of the independent consultant firm's report happen?  Probably not.

However, rest assured the Plaintiff's attorneys of Colorado will request it in malpractice suits that arose during the time period of outage.

-- SS

Boulder Community Hospital computer system crash: Either you're in control of your information systems, or they're in control of you

Yet another health IT crash, "prolonged" this time, from some unspecified "glitch":

Boulder Community Hospital computer system crash frustrates patients
Officials say it could take until Friday for outage to be resolved
By Brittany Anas
Camera Staff Writer
Posted:   03/18/2013 07:23:23 PM MDT
Updated:   03/18/2013 07:24:16 PM MDT

A prolonged computer system outage is preventing Boulder Community Hospital from accessing patient records -- making it difficult for people to schedule surgeries, get test results and make appointments for routine blood work.

Meditech, the system used by the hospital to manage patient records, went down in the middle of last week. It could take the hospital until Friday to get the system back up, said Rich Sheehan, spokesman for Boulder Community.

That fits my definition of "prolonged."

While information technology officials are investigating what caused the outage, Sheehan said patient records are protected and hospital officials don't believe they've been hacked. 

That's not very reassuring, considering the length of the outage.

The outage affects the hospital, its Foothills campus, eight laboratories and six imaging centers. 

Patients are on its face put at-risk ... for example, I know of several deaths of infants and adults from delayed x-ray reports alone ... but the clinicians, not the IT seller or hospital IT staff, are liable.

"We know medical care is important to people, so we understand the concerns those in the community have," Sheehan said. "We have a lot of people working on this, doing the best they can to solve this problem in a safe manner and as quickly as possible." 

"We know medical care is important to people?"  No, really?

In the meantime, the hospital is using manual paper record-keeping systems and traditional paper charts for its inpatients. Hospital officials say the system allows them to continue treating patients, provide diagnostic services and collect important clinical information that will be entered later into each patient's electronic health record.

But that concerns Eroca Lowe, whose mother was in the hospital Thursday through Sunday with gallbladder pain.

Lowe said the outage made it extremely difficult for doctors and nurses to do their jobs while hunting down lab results. She criticizes the hospital for not having a backup computer system and resorting to paper records.

"That's not a hospital in 2013," she said.

It's a good bet the paper records and HIM personnel managing them are not what they used to be pre-computer.

... Dina Huber said it took her and her significant other six days to schedule an appointment for a hernia surgery because the system used for scheduling is down.

"If they can't keep their computer system running, how can we trust them to perform surgery?" Huber said.

Fortunately, surgeons perform surgery ... not computers, IT staff or management.  Doctors, as the enablers of healthcare, don't need computers to save lives.

However, making their job harder is not a good idea.

A physician who works at Boulder Community Hospital, speaking on the condition of anonymity, said he doesn't think the outage is compromising the health or safety of patients. But, he said, the backup response "seems a little haphazard, and it's not an organized plan." He said physicians are left chasing down records.

If a prolonged outage "is not compromising safety", then why did the hospital spend tens of millions on computers?

Sheehan said the hospital is prioritizing accuracy and patient safety while getting the records system up and running. 

Once it's running again, there is significant risk of data now recorded manually being lost, thus again increasing error risk.

"We apologize for the delays, but this was an unavoidable situation," Sheehan said.

("We apologize for the chilly water, but this was an unavoidable situation." - Captain of the HMS Titanic?)

If an injury occurs, how will that sound to a jury?

Let me answer that:  like bull***.  

My response to Mr. Sheehan and Meditech, and the IT personnel involved:  "Either you're in control of your information systems, or they're in control of you."

It seems the latter clearly applies here.

I pray nobody gets injured ... and that the principals don't end up before plaintiff attorneys I've educated on the issues of bad health IT.

-- SS