Another in the "Health IT Crashed, But Patient Care Was Not Compromised" Series - NHS Greater Glasgow and Clyde

We must have health IT to prevent stupid doctors and nurses from making mistakes.  Without it, patients are at the mercy of dreaded Paper Records, which by definition cannot ever be used to provide quality care.

But when the IT goes down, Patient Care Has Not Been Compromised.  This line should be trademarked, as it's seen so often.  (I even have an indexing tag for it, see this query link: http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised.)

Care is never, never compromised when the IT goes belly up en masse due to information technology malpractice.  Care is only compromised by paper, no matter how good the paper records and its human stewards are.

Here's the latest example that made it to the news, in Scotland:

1 October 2013
BBC News
Appointments postponed after major IT failure at NHSGGC (NHS Greater Glasgow and Clyde)

Hundreds of outpatient appointments and a number of operations had to be postponed after computer systems failed at Scotland's biggest health board.

NHS Greater Glasgow and Clyde said technicians were working through the night to fix a "major IT problem" which occurred on Tuesday morning.

It affected staff access to clinical and administrative systems.

Delaying hundreds of appointments and delaying surgeries at up to 10 major hospitals seems on its face to represent "compromised care."

The health board apologised to patients and said all appointments would be rescheduled.

In total, 288 outpatient appointments, four planned inpatient procedures, 23 day surgery cases and 40 chemotherapy sessions were postponed.

There was also some delay in calls to the switchboard being answered.

The problem may have affected up to 10 major hospitals across the health board area.

One wonders how these appointments and surgeries were triaged for delay.  Clearly the downed computer was of no help.

Here's that wonderful line:

But emergency operations were not compromised - neither were community services.

So when does computer failure actually compromise patient care?  I'd like to see some hospital executive with a spine for once admit that IT malpractice does disrupt patient care, create distractions, and thus create safety risk.  Considering the domain, however, I doubt I'll ever see that.

A spokeswoman said: "Our technical staff are working flat out to resolve this.

It should never have happened in the first place.

"The problem relates to our networks and the way staff can connect to some of our clinical and administrative systems.

Well, sick patients really appreciate that explanation.

It was not clear how long the disruption would last.

NHSGGC said if it did continue, people who were scheduled for treatment would be contacted directly.

Per a computer guru from my past:  "Either you're in control of your information systems, or they're in control of you."

In this instance, the latter clearly applies.

In healthcare, having your information systems in control of you is, sooner or later, going to be deadly.

-- SS

Setback for Sutter after $1B EHR crashes (in followup to post "RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals")

At my July 12, 2013 post "RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals" (http://hcrenewal.blogspot.com/2013/07/rns-say-sutters-new-electronic-system.html) I reproduced a California Nurses Association warning about rollout of an EHR at Sutter:

RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals

Introduction of a new electronic medical records system at Sutter corporation East Bay hospitals has produced multiple problems with safe care delivery that has put patients at risk, charged the California Nurses Association today.

Problems with technology are not unique to health care ...  [What is unique to healthcare IT is the complete lack of regulation - ed.]

In over 100 reports submitted by RNs at Alta Bates Summit Medical Center facilities in Berkeley and Oakland, nurses cited a variety of serious problems with the new system, known as Epic. The reports are in union forms RNs submit to management documenting assignments they believe to be unsafe.

Patient care concerns included computerized delays in timely administration of medications and contact with physicians, ability to properly monitor patients, and other delays in treatment.  Many noted that the excessive amount of time required to interact with the computer system, inputting and accessing data, sharply cuts down on time they can spend with patients with frequent complaints from patients about not seeing their RN.  [Note: patients are not given the opportunity for informed consent about the risks, nor opt-out of EHR use in their care - ed.]

In related posts I'd observed such concerns being ignored by hospital management.  See header of the aforementioned post.

Now we have this:  a major system crash.

Healthcare IT News
Setback for Sutter after $1B EHR crashes

'No access to medication orders, patient allergies and other information puts patients at serious risk'

SACRAMENTO, CA | August 28, 2013

The nearly $1 billion electronic health record system at Sutter Health in Northern California crashed early this week, leaving nurses and clinical staff unable to access any patient information for a full day.
he Epic EHR was down Aug. 23 due to a planned upgrade for eight hours during which nurses and other clinical users could read medication orders and patient histories but had to record new data on paper to then be re-entered into the system later, according to reports from the California Nurses Association, part of National Nurses United, the largest nurses union in the U.S.

Then, on the morning of Aug. 26 at approximately 8 a.m., the system crashed completely, at which time nurses, physicians and hospital staff had no access to patient information, including what medications patients were taking or required to take and all vital patient history data. 
 
The Epic EHR went dark at several Sutter Health locations, including Alta Bates Summit Medical Center, Eden Medical Center, Mills-Peninsula Hospital, Sutter Delta, Sutter Tracy, Sutter Modesto and numerous affiliated clinics.

"This incident is especially worrisome," said Bonnie Castilla, RN, CNA legislative director, in a news release. "No access to medication orders, patient allergies and other information puts patients at serious risk.  These systems should never be relied upon for protecting patients or assuring the delivery of the safest care."  [This is no exaggeration.  My mother was nearly killed by such a problem when caused by an EHR defect (link) that I averted at the last minute by happenstance of showing up at the right time, while she recuperating from another EHR-related disaster - ed.]   

Some RNs also reported that Sutter failed to provide adequate support during the blackout and didn't implement adequate backup planning.
 
"All information such as medication administration records and patient histories was outdated by two to three days," said Mike Hill, Alta Bates Summit RN, in a news release. "There were no orders that could be seen of any kind through the day so nurses called for what they needed."

Kevin Sweat, an emergency room RN at Eden Medical Center, a Sutter affiliate, also weighed in on the blackout, saying that in his 12 years as an RN he has never seen a system crash like this. 
  

Worse, clinicians must now serve their Cybernetic Master to perfection, or be whipped (apparently to improve morale):

... "We have been on Epic for 5 months now, and we can no longer have incorrect orders, missing information or incorrect or missing charges. Starting on September 1st, errors made in any of the above will result in progressive discipline," according to another hospital memo sent to staff.

In the setting of dire warnings by the nurses of EHR dangers several months back that were likely largely ignored, if any patient was harmed or killed as a result of this latest fiasco, the corporate leadership has literally begged to be sued for negligence, in my view.

However I'm sure a press release soon will claim that "patient care has not been compromised."

Of course this includes now and moving forward, even with informational gaps all over the place.

-- SS

Aug. 29, 2013 additional thought:

The punishment for not being a 'perfect' user of this EHR is the ultimate "blame the user" (blame the victim?) game, considering the pressures of patient care in hospitals in lean times - partly due to EHR expense! - and EHRs that have not been formally studied for usability and are poorly designed causing "use error" (that is, a poor user experience promotes even careful users to make errors).  Cf. definition of bad health IT:

Bad Health IT ("BHIT") is defined as IT that is ill-suited to purpose, hard to use, unreliable, loses data or provides incorrect data, causes cognitive overload, slows rather than facilitates users, lacks appropriate alerts, creates the need for hypervigilance (i.e., towards avoiding IT-related mishaps) that increases stress, is lacking in security, compromises patient privacy or otherwise demonstrates suboptimal design and/or implementation.

The study of usability is getting underway only now via NIST but will likely be done in an industry-friendly way due to health IT politics.

-- SS

Aug. 29, 2013 addendum

There have been numerous comments over at HisTalk (at http://histalk2.com/2013/08/27/news-82813/) defending the outage as not EPIC's fault.   From the point of view of clinicians - and more importantly, patients - it doesn't matter what component of the hospital's entire "EHR" (an anachronistic term used for what is now a complex enterprise clinical resource and clinician command-and-control system) went down. 

Aside from all the EPIC issues the nurses have been complaining about (see earlier July 12, 2013 post linked above), the larger problem is that IT malpractice occurred.  The term "malpractice" is used in medical mishaps; I see no reason why it does not apply to major outages of mission critical healthcare information technology systems.

IT malpractice in healthcare kills.

These are the types of nurses I'd want caring for me and mine.  Letting this kind of snafu go "anechoic" does not promote proper management remedial education on Safety 101 and on health IT risk, two areas of education that management appears to desperately need in hospitals.

-- SS

RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals

Add the following from Sutter East Bay Hospitals to nurses' and physicians' complaints at Marin General Hospital (http://hcrenewal.blogspot.com/2013/05/marin-general-hospitals-nurses-are.html), Affinity Medical Center (http://hcrenewal.blogspot.com/2013/06/affinity-rns-call-for-halt-to-flawed.html), Contra Costa County (http://hcrenewal.blogspot.com/2012/08/contra-costas-45-million-computer.html), San Francisco Department of Public Health (http://hcrenewal.blogspot.com/2010/11/avatar-fails-no-not-cameron-movie-but.html), and others:

For Immediate Release 
July 11, 2013
Contact-  Charles Idelson, 510-273-2246

RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals

Introduction of a new electronic medical records system at Sutter corporation East Bay hospitals has produced multiple problems with safe care delivery that has put patients at risk, charged the California Nurses Association today.

Problems with technology are not unique to health care – pilots of the ill fated Asiana airline that tragically crashed at San Francisco International Airport July 6 told federal investigators that an automatic throttle failed to keep the jetliner at the proper speed for landing, the Los Angeles Times reported July 9.  [What is unique to healthcare IT is the complete lack of regulation - ed.]

In over 100 reports submitted by RNs at Alta Bates Summit Medical Center facilities in Berkeley and Oakland, nurses cited a variety of serious problems with the new system, known as Epic. The reports are in union forms RNs submit to management documenting assignments they believe to be unsafe.

Patient care concerns included computerized delays in timely administration of medications and contact with physicians, ability to properly monitor patients, and other delays in treatment.  Many noted that the excessive amount of time required to interact with the computer system, inputting and accessing data, sharply cuts down on time they can spend with patients with frequent complaints from patients about not seeing their RN.  [Note: patients are not given the opportunity for informed consent about the risks, nor opt-out of EHR use in their care - ed.]

"EPIC is a system that is so cumbersome to use for nurses and physicians, that we often feel as though we are caring for a computer, not a patient,” said Thorild Urdal, an RN at Alta Bates Summit’s hospital in Berkeley. “It delays care and treatment, the program is naturally counter-intuitive and it was clearly not designed in concert with nurses and physicians." [Clinicians end up caring for an "iPatient", as others have noted - ed.]

"The Epic program developed and implemented by Sutter is neither nurse or patient friendly,” said Alta Bates Summit Oakland RN Mike Hill. “Epic does not enhance my ability to chart instead it takes time away from the bedside and my patients and preventing me from providing the absolute best care that they and I expect from me as a nurse."

Sutter CEO Pat Fry last year told the San Francisco Business Times that Sutter will spend $1 billion on Epic, a system that has sparked controversy at several other hospitals, including a Contra Costa facility where several RNs cited serious medical errors in testimony to county supervisors last August.

At Alta Bates Summit specific incidents directly related to Epic problems included:

• A patient who had to be transferred to the intensive care unit due to delays in care caused by the computer.  [It's happenstance they did not have to be transferred to the morgue - ed.]
• A nurse who was not able to obtain needed blood for an emergent medical emergency.
• Insulin orders set erroneously by the software.
• Missed orders for lab tests for newborn babies and an inability for RNs to spend time teaching new mothers how to properly breast feed babies before patient discharge.
• Lab tests not done in a timely manner.
• Frequent short staffing caused by time RNs have to spend with the computers.
• Orders incorrectly entered by physicians requiring the RNs to track down the physician before tests can be done or medication ordered.
• Discrepancies between the Epic computers and the computers that dispense medications causing errors with medication labels and delays in administering medications.
• Patient information, including vital signs, missing in the computer software.
• An inability to accurately chart specific patient needs or conditions because of pre-determined responses by the computer software.
• Multiple problems with RN fatigue because of time required by the computers and an inability to take rest breaks as a result.
• Inadequate RN training and orientation.

These "incidents" are certainly capable of causing harms or fatalities.  One wonders if hospital executives are providing the usual refrain that these are just "glitches" (http://hcrenewal.blogspot.com/search/label/glitch) and that patient care has not been compromised (http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised).

A bit more background follows:

... Hospitals nationally are spending tens of billions of dollars on technology systems, especially on electronic health records (EHR) programs for which they also receive federal financial incentives.

EHR programs are paraded as a panacea for reducing medical errors and cutting costs, but in life the promise is falling short in both areas.

A RAND corporation analysis earlier this year said visions of savings and improved efficiency in patient care have had what the New York Times called “mixed results, at best.”

The U.S. Food and Drug Administration has acknowledged getting hundreds of reports of problems involving health information technology including numerous patient injuries and deaths.

Some examples seen at hospitals across the country:

• At Marin General Hospital in Northern California, RNs called on the Marin Healthcare District board to delay implementation of their EHR system. "Orders are being inadvertently passed to the wrong patients. People have gotten meds when they've been allergic to them. This is dangerous," Marin RN Barbara Ryan said in comments reported by the Marin Independent Journal.
• In Chicago, the Chicago Tribune in 2011 reported on a patient death at Advocate Lutheran General hospital after an automated machine prepared an intravenous solution containing a massive overdose of sodium chloride — more than 60 times the amount ordered by a physician.
• At Affinity Medical Center RNs in Massillon, Oh. RNs in June raised multiple objections to the hurried introduction of an EHR system. Subsequently, they have cited medication errors, delays in care, problems with documentation, computers crashing, and other concerns.

I am simply the reporter here.

-- SS

Marin General Hospital nurses warn that new computer system is causing errors, call for time out

- Posted on the Healthcare Renewal Blog May 17, 2013 -

Of course, the ever-present euphemism for life-threatening EHR malfunctions and defects, i.e., "glitches" are the cause (http://hcrenewal.blogspot.com/search/label/glitch):

Marin General Hospital nurses warn that new computer system is causing errors, call for time out

By Richard Halstead
Marin Independent Journal
Posted:   05/15/2013 04:07:49 PM PDT

Nurses at Marin General Hospital have asked administrators to put implementation of a new computerized physician order entry system on hold until glitches can be worked out and more training provided to nurses and doctors who use it.

Nearly a dozen nurses attended the regularly scheduled meeting of the Marin Healthcare District board Tuesday night at Marin General to voice their concerns. The district board oversees Marin General, but it does not involve itself in the hospital's day-to-day operations.

"Orders are being inadvertently passed to the wrong patients. People have gotten meds when they've been allergic to them. This is dangerous," said Barbara Ryan, a Marin General registered nurse, who works in pediatrics and the intensive care nursery. "We're not asking you to get rid of it. We're asking you to place it on hold."

Orders passed to wrong patients?  No problem, just a glitch!  Meds people are allergic to?  Just a glitch.  Dangerous?  No way.  It's just a glitch!

But Lee Domanico, who serves as the CEO of both Marin General and the Marin Healthcare District, said, "I'm confident that in spite of the implementation issues, we have a system today that is safer for patients than our old paper system, and it will get even safer as we gain experience with it and work to fix some of the glitches we've experienced."

Where's the data backing up that assertion, I ask?  The actual risks of paper records don't seem to be robustly documented anywhere.

Ryan, who serves as the California Nurses Association/National Nurses United representative, was one of four Marin General nurses who spoke during the public comment portion of the meeting. Ryan said the nurses warned in advance of the system's roll-out on May 7 that nurses and doctors had insufficient knowledge of the system. Ryan said due to problems with the software nurses had been unable to open the program at home to practice using it.

And yet the rollout happened anyway?  That seems to me to be reckless indifference to the concerns of clinicians.

"Lo and behold the problems that we were worried about have happened," Ryan said. "We're looking at two-hour preps for surgery and two- to three-hour discharges; skilled nursing facilities calling back saying, this really doesn't make sense; the wrong meds ordered on the wrong patients and then given to the wrong patients; the inability for nurses to be able to see what the doctor ordered and double-check it."

Of course, I might add, patient safety was not compromised, the other common refrain of EHR glitch-excusers ... see below.

Ryan said nurses have and will continue to file "assignment despite objection" forms due to the system. Nurses file the forms to document formal objections to what they consider an unsafe, or potentially unsafe, patient care assignment.

"We will take patients but we will object to the assignment because it is unsafe," Ryan said. "This system is making it unsafe."

These will be exceptionally helpful in court to any patients injured or killed as a result of these "glitches" and EHR rollout that occurred despite direct warnings from clinical experts.

Marin General nurse Susan Degan said, "This is not about resistance to change. It's about accountability. My most important role is that of patient advocate. I am held accountable when errors are made."

Domanico acknowledged there have been some technical problems with the Paragon system, including making it possible for nurses to open from home. And he said the software is not faster than the old paper system.  [Considering it's acknowledged all the way up to the highest levels of HHS that current EHR's slow physicians down, one wonders if anyone in this organization thought an EHR would actually increase speed? - ed.]

About the "resistance to change" canard, see my essay "Doctors and EHRs: Reframing the 'Modernists v. Luddites' Canard to The Accurate 'Ardent Technophiles vs. Pragmatists' Reality" at http://hcrenewal.blogspot.com/2012/03/doctors-and-ehrs-reframing-modernists-v.html .

"So yes," Domanico said, "it is causing stress for nurses who have heavy workloads, who are learning how to use it, particularly in areas where we need to speed up the computer."

What?  "Speed up the computer?"  They've spent tens if not hundreds of millions for an EHR, and the computer's too slow?

Actually, I think what this CEO in an obvious display of health IT ignorance is trying to say is that we have to do something about the system's poor usability, which sort of mimics what the Board Chair of the American Medical Assocation just said (http://hcrenewal.blogspot.com/2013/05/ama-finally-on-board-with-ehr-views.html).

Also - clinician stress promotes error.

But Domanico challenged the suggestion that patient safety at Marin General had been compromised.

In fact, there is no way the issues described above cannot be compromising patient safety, on its face. (http://hcrenewal.blogspot.com/search/label/Patient%20care%20has%20not%20been%20compromised).

"I would have no hesitation about entering this hospital tonight," he said.

As a VIP, of course, this CEO would get special treatment.  Thanks a lot.

I would NOT want to be a patient there under these conditions, unless perhaps I had a 24x7 medically-skilled advocate/bodyguard.

Board member Ann Sparkman, who previously served as in-house counsel at Kaiser Permanente, said nurses at Kaiser struggled at first when a new computer system was introduced there.

Sparkman said, "It's just to be expected."

This seems a rather bizarre appeal to common practice (http://www.nizkor.org/features/fallacies/appeal-to-common-practice.html).

The stunning ignorance of this board member about proper mission-critical IT safety testing and implementation, such as performed in pharma, aerospace, etc. is, quite frankly, shocking.

Further, an attitude that life-threatening "glitches" are "just to be expected" by a member of the Board of Directors, with fiduciary responsibilities regarding hospital operations, is grossly negligent in my opinion, and completely ignores patient's rights.

Unbelievable.

One wonders if any formally-trained medical informatics experts were in leadership roles in this project.

-- SS

When "Human Error" Causes EHR Downtime, Who is Liable For Patient Injuries That Result?

In the Pittsburgh Post-Gazette was this story of yet another EHR "glitch":

March 9, 2013 12:17 am 

Human error the cause of UPMC electronic issue

A systemwide problem with UPMC's information systems Wednesday left electronic patient records and other data inaccessible for about three hours. A UPMC spokeswoman said the hospitals "immediately went to manual backup systems, and we quickly identified and fixed the problem." She said there was no indication that patient care was compromised by the incident, which was due to human error.

I will presume the "human error" was not a physician or nurse pressing the wrong button, but a "human error" involving the servers or IT infrastructure such as a botched system upgrade, action that caused a server room power fault, etc.

UMPC is a very large system as their webpage shows, showing approximately fifteen major facilities.

The now-expected "patient care has not been compromised" line was provided to the Gazette, a line so commonly heard after EHR outages that I  use it as a Healthcare Renewal indexing tag (see this query link).

The following questions arise:

• What, exactly, was the "human error" and why was there no fault tolerance built into these mission-critical systems to account and compensate for it, such as via redundancy?

• If paper is so bad as a record-keeping medium that hundreds of billions of dollars are being spent to replace it, then how can patient care not be compromised, especially when multiple hospitals unexpectedly and without warning have to return to its use? 

• How can a very large hospital system rapidly declare that "patient care was not compromised" without a thorough and comprehensive patient review, accounting for possible delayed negative outcomes (by way of just a few simple examples, due to medication or imaging delays?)

•  Who is liable for any adverse patient outcomes that occurred related to the sudden unavailability of past records:  the clinicians?  The "human" who committed the computer-related error?  The corporation, either for direct negligence in implementing and mandating use of a system prone to mass outage by human error, or vicariously for the negligence and/or misconduct of its information technology employees and/or agents?

•  How many "outages" will it take before some patient is outright, no-doubt-about-it harmed or killed?  Do we want to find out, or is a priority to have redundancies so these systems don't crash?

I, for one, would not want to have a family member be "crashing" at the time of a sudden, unexpected system-wide EHR outage.

-- SS

Allegheny Health System Computer Crash (Again) and Paper Backups

I reported on a health IT crash in my May 2011 post "Twelve Hour Health IT Glitch at Allegheny General Hospital - But Patients Unaffected, Of Course..."

Now, there's this at the same healthcare system:

Computer system at West Penn Allegheny restored after crash 
Liz Navratil
Pittsburgh Post-Gazette
October 2, 2012

The computer system at West Penn Allegheny Health System crashed about noon today, temporarily leaving doctors and nurses to work off of paper records instead.

Kelly Sorice, vice president of public relations for the health system, said all systems have since been restored. She said the servers crashed about noon today when the system experienced a power surge.

Doctors in the health system keep paper copies of almost all of their records so they can reference them during power outages or scheduled maintenance times, Ms. Sorice said.

Some systems were up eight hours later and others were expected to come online overnight, according to a report at HisTalk.

Assuming the statement about "doctors keep paper copies of almost all their records" was not spin control regarding skeletal paper records, a question arises.

Why, exactly, spend hundreds of millions of dollars on computing if paper records are kept, and are perfectly sufficient to accomplish the following, the usual refrain in health IT crash scenarios?

Ms. Sorice said she did not know of any procedures that had been rescheduled and added that, "Patient care has not been compromised."

As a physician/ham radio enthusiast who did an elective in Biomedical Engineering in medical school, I also want to know:

1)  What caused the “power surge?”
2)  Why were the systems not protected against a “power surge?”
3)  Exactly how did the “power surge” affect the IT?

Note: I've created a new, searchable indexing term for HIT outage stories with the usual refrain along the lines that "patient care has not been compromised." 

See this query link using the new indexing term.

-- SS

Addendum Oct. 3:

Australian EHR reseacher and professor Dr. Jon Patrick opines:

Even if [the paper records are] skeletal they suggest an endemic lack of confidence. I think the hospital spokesperson hasn't seen the implication of their statement.

-- SS

UK: Another Example of IT Malpractice With Bad Health IT (BHIT) Affecting Thousands of Patients, But, As Always, Patient Care Was "Not Compromised"

At my Dec. 2011 post "IT Malpractice? Yet Another "Glitch" Affecting Thousands of Patients. Of Course, As Always, Patient Care Was "Not Compromised" and others, I noted:

... claims [in stories regarding health IT failure] that "no patients were harmed" ... are both misleading and irrelevant:

Such claims of 'massive EHR outage benevolence' are misleading, in that medical errors due to electronic outages might not appear for days or weeks after the outage ... Claims of 'massive EHR outage benevolence' are also irrelevant in that, even if there was no catastrophe directly coincident with the outage, their was greatly elevated risk. Sooner or later, such outages will maim and kill.

Here is a prime example of why I've opined at my Sept. 2012 post "Good Health IT (GHIT) v. Bad Health IT (BHIT): Paper is Better Than The Latter" that a good or even average paper-based medical record keeping system can facilitate safer and better provision of care than a system based on bad health IT (BHIT).

Try this with paper:

NHS 'cover-up' over lost cancer patient records

Thousands awaiting treatment were kept in the dark for five months when data disappeared

Sanchez Manning
The Independent
Sunday 30 September 2012

Britain's largest NHS trust took five months to tell patients it had mislaid medical records for thousands of people waiting for cancer tests and other urgent treatments. Imperial College Healthcare NHS Trust discovered in January that a serious computer problem and staff mistakes had played havoc with patient waiting lists.

It's quite likely the "serious computer problem" far outweighed the impact of "staff mistakes", as disappearing computer data does so in a "silent" manner.  One does not realize it's missing as there's not generally a trail of evidence that it's gone.

About 2,500 patients were forced to wait longer on the waiting lists than the NHS's targets, and the trust had no idea whether another 3,000 suspected cancer patients on the waiting list had been given potentially life-saving tests. Despite the fact that the trust discovered discrepancies in January and was forced to launch an internal review into the mess, including 74 cases where patients died, it did not tell GPs about the lost records until May.

That is, quite frankly, outrageous if true and (at least in the U.S.) might be considered criminally negligent (failure to use reasonable care to avoid consequences that threaten or harm the safety of the public and that are the foreseeable outcome of acting in a particular manner).

Revelations about the delay prompted a furious response yesterday from GPs, local authorities and patients' groups. Dr Tony Grewal, one of the GPs who had made referrals to Imperial, said doctors should have been told sooner to allow them to trace patients whose records were missing. "The trust should have contacted us as soon as it was recognised that patients with potentially serious illnesses had been failed by a system," he said. "GPs hold the ultimate responsibility for their patient care."

That is axiomatic.

The chief executive of the Patients Association, Katherine Murphy, added: "This is unacceptable for any patient who has had any investigation, but especially patients awaiting cancer results, where every day counts. The trust has a duty to contact GPs who referred the patients. It's unfair on the patients to have this stress and worry, and the trust should not have tried to hide the fact that they had lost these records. They should have let the GPs know at the outset."

Unfair to the patients is an understatement,  However, if one's attitude is that computers have more rights than patients, as many on the health IT sector seem to with their ignoring of patient rights such as informed consent, lack of safety regulation, and lack of accountability, then it's quite acceptable.

The trust defended the delay in alerting GPs, arguing that it needed to check accurately how much data it had lost before making the matter public. It said a clinical review had now concluded that no one died as a result of patients waiting longer for tests or care.

That would be perhaps OK if the subjects whose "data had been lost" through IT malpractice were lab rats.

Despite this, three London councils – Westminster, Kensington and Chelsea, and Hammersmith and Fulham – are deeply critical of the way the trust handled the data loss. Sarah Richardson, a Westminster councillor who heads the council's health scrutiny committee, said that trust bosses had attempted to "cover up" the extent of the debacle. "Yes, they've done what they can but, in doing so, [they] put the reputation of the trust first," she said. "Rather than share it with the GPs, patients and us, they thought how can we manage this information internally. They chose to consider their reputation over patient care."

As at my Oct. 2011 post "Cybernetik Über Alles: Computers Have More Rights Than Patients?", to be more specific, they may have put the reputation of the Trust's computers first. 

Last week, it was revealed that Imperial has been fined £1m by NHS North West London for the failures that led to patient data going missing. On Wednesday, an external review into the lost records said a "serious management failure" was to blame for the blunder.

Management of what, one might ask?

Imperial's chief financial officer, Bill Shields, admitted at a meeting with the councils that the letter could have been produced more quickly. He said that, at the time, the trust had operated with "antiquated computer systems" and had a "light-touch regime" on elective waiting times.

Version 2.0A will, as again is a typical refrain, fix all the problems.

Terry Hanafin, the leading management consultant who wrote the report, said the data problems went back to 2008 and had built up over almost four years until mid-2011. Mr Hanafin said the priorities of senior managers at that time were the casualty department and finance.

Clinical computing is not business computing, I state for the thousandth time.  When medical data is discovered "lost", the only response should be ... find it, or inform patients and clinicians - immediately.

He further concluded that while the delays in care turned out to be non-life threatening, they had the potential to cause pain, distress and, in the case of cancer patients, "more serious consequences" ... The trust said it had found no evidence of clinical harm and stressed that new systems have now been implemented to record patient data. It denied trying to cover up its mistakes or put its reputation before concerns for patients. "Patient safety is always our top priority," said a spokesman.

"More serious consequences" is a euphemism for horrible metastatic cancer and death, I might add.  The leaders simply cannot claim they "found no evidence of clinical harm" regarding delays in cancer diagnosis and treatment until time has passed, and followup studies performed on this group of patients.

This refrain is evidence these folks are either lying, CYA-style, or have no understanding of clinical medicine whatsoever - in which case their responsibilities over the clinic need to be ended in my opinion.

I, for one, would like to know the exact nature of the "computer problem", who was responsible, and if it was a software bug, how such software was validated and how it got into production.

-- SS

Oct. 1, 2012 Addendum:

What was behind the problems, according to another source?   

Bad Health IT (BHIT):

Poor IT behind Imperial cancer problems
e-Health Insider
28 September 2012
Rebecca Todd

An independent review of data quality issues affecting cancer patient referrals to Imperial College Healthcare NHS Trust has identified “poor computer systems” as a key cause of the problem.

The review’s report highlights the trust’s use of up to 17 different IT systems as causing problems for patient tracking.

However, it says the trust should be aware of the risks of [replacing the BHIT and] moving to a single system, Cerner Millennium, because of reported problems in providing performance data after similar moves at other London trusts.

In January 2012, the report says the NHS Intensive Support Team was reviewing the way reports on cancer waiting times were created from Imperial’s cancer IT system, Excelicare.

The team discovered that almost 3,000 patients were still on open pathways who should have been seen within two weeks. In May, letters were sent to GPs to try and ascertain the clinical status of around 1,000 patients.

BHIT must be forbidden from real-world deployments, and fixed rapidly or dismantled (as Imperial College Healthcare NHS Trust appears to be doing), although the "solution" might be just as bad, or worse, than the disease.

-- SS

From the University of Chicago EHR Helpdesk Call Line

I was alerted this morning (Aug. 23rd) to this message currently in the telephone message of the CBIS [Chicago Biomedicine Information Services] Service Desk at University of Chicago Medical Center:

"Thanks for calling the CBIS Service Desk.  Your call is very important to us. We are currently experiencing troubles with our Citrix logon.  It may log you on under a different profile.  Please check before you go any further when you're logging in to Citrix."

Citrix is a computer program that allows remote access to information systems.

I imagine the meaning of "log you on under a different profile" means "logging you on as a different user."

The chances of a security breach (ability of unauthorized user to peer into patient's charts they have no business seeing), unauthorized history/order manipulation, or even misidentification error (e.g., a clinician inadvertently acting upon a patient of some other clinician who has a similar name to their own patient) and other distracting work disruptions due to the inconveniences this "trouble" creates are worrisome.

One wonders how every user is being informed of this problem, as not everyone makes it a habit to call the service desk before logging in to clinical systems...

But, alas, this is just a "glitch" (the euphemism used by technophiles for malignant software defects), and, of course, patient safety is never compromised by "glitches."

Patient Safety Will Not Be Compromised, We Predict ... So Say Us All.

-- SS

8/29/12 Addendum:

Apparently the problem was finally solved between 5:30 PM and 9 PM CST on August 27.   I first became aware of it at around 8 AM EST August 23.  Brings to life the line "either you are in control of your information systems, or they are in control of you."

Also, see the comment thread to this post here, specifically the comments starting at August 28, 2012 12:16:00 PM EDT, to see yet another demonstration of the illogic, unserious attitudes and feelings of entitlement towards patient risk and transparency characteristic of the health IT industry.  The anonymous commenter also alleges to have firsthand knowledge of the problem, suggesting they are from U. Chicago, but this cannot be confirmed.

-- SS

ONC and Misdirection Regarding Mass Healthcare IT Failure

In my keynote address to the Health Informatics Society of Australia in Sydney recently, I cautioned attendees including those in government to be wary of healthcare IT hyper-enthusiast misdirection and logical fallacy (a.k.a. public relations).

In the LA Times story "Patient data outage exposes risks of electronic medical records" on the Cerner EHR outage I wrote of in my post "Massive Health IT Outage: But, Of Course, Patient Safety Was Not Compromised" (the title, of course, being satirical), Jacob Reider, acting chief medical officer at the federal Office of the National Coordinator for Health Information Technology is quoted.  He said:

"These types of outages are quite rare and there's no way to completely eliminate human error."

This is precisely the type of political spin and hyper-enthusiast misdirection I cautioned the Australian health authorities to evaluate critically.

As comedian Scott Adams humorously noted regarding irrelevancy, a hundred dollars is a good price for a toaster, compared to buying a Ferrari.

Further, when you're the patient harmed or killed, or the victim is a family member, you really don't care how "rare" the outages are.

Airline crashes are "rare", too.   So, shall they just be tolerated as a "cost of doing business" and spun away?

(As I once wrote, the asteroid colliding with Earth that caused the extinction of the dinosaurs was a truly "rare" event.)

It seems absurd for me to have to point out that paper, unless there is a mass outbreak of use of disappearing ink, or locally hosted clinical IT, do not go blank en masse across multiple states and countries for any length of time, raising risk across multiple hospitals greatly, acutely and simultaneously.   Yet, I have to point out this obvious fact in the face of misdirection.

Locally hosted health IT, of course, can only cause "local" chart disappearances.  "Local" is a relative term, however, depending on HC organization size, as in the example of a Dec. 2011 regional University of Pittsburgh Medical Center (UPMC) 14-hour outage affecting thousands here.

Further, EHR's and other clinical IT, whether hosted locally or afar, had better offer truly major advantages, without major risks and disadvantages, over older medical records technologies before exposing large numbers of patients to an invasive IT industry and the largest unconsented human subjects experiment in history.

Unfortunately, those basic criteria are not yet apparent with today's systems (see for instance this reading list).

EHR's and other clinical IT, forming in reality an enterprise clinical resource management and clinician workflow control apparatus, have introduced new risk modes including mass chart theft (sometimes tens of thousands in the blink of an eye); also, mass chart disappearances as in this case - all not possible with paper.

At the very least, if hospitals want enterprise clinical resource management and clinician workflow control systems, these should not be relegated to a distant third party.  Patients are not guinea pigs upon whom to test the ASP software model ("software as a service") that, upon failure for any reason, threatens their lives.

Finally, these complications are a further example why this industry cannot go on without meaningful oversight.  The unprecedented special medical device regulatory accommodations must end.

-- SS

Massive Health IT Outage: But, Of Course, Patient Safety Was Not Compromised

Having been 'Down Under' in Sydney addressing the Health Informatics Society of Australia on the need to slow down their national health IT program - and on the need to think critically about HIT seller public relations exaggerations and hubris - and being very busy, I missed this quite stunning story of a major health IT outage.

Just a typical "glitch":

Some lessons from a major outage
Posted on July 31, 2012
By Tony Collins

Last week Cerner had a major outage across the US. Its international customers might also have been affected.

InformationWeek Healthcare reported that Cerner’s remote hosting service went down for about six hours on Monday, 23 July. It hit “hospital and physician practice clients all over the country”. Information Week said the unusual outage “reportedly took down the vendor’s entire network” and raised “new questions about the reliability of cloud-based hosting services”.

A Cerner spokesperson Kelli Christman told Information Week,

“Cerner’s remote-hosted clients experienced unscheduled downtime this week. Our clients all have downtime procedures in place to ensure patient safety.  [Meaning, for the most part, blank paper - ed.] The issue has been resolved and clients are back up and running. A human error caused the outage.  [I don't think they mean human error as in poor disaster recovery and business continuity engineering - ed.]  As a result, we are reviewing our training protocol and documented work instructions for any improvements that can be made.”

Christman did not respond to a question about how many Cerner clients were affected. HIStalk, a popular health IT blog, reported that hospital staff resorted to paper [if that was true, that paper was OK in an unplanned workflow disruption of major proportions, then why do we need to spend billions on health IT, one might ask? - ed.] but it is unclear whether they would have had access to the most recent information on patients.

One Tweet by @UhVeeNesh said “Thank you Cerner for being down all day. Just how I like to start my week…with the computer system crashing for all of NorCal [Northern California].”

Tony Collins is a commentator for ComputerWorldUK.com.  He's quoted me, as I wrote in my May 2011 post Key lesson from the NPfIT - The Tony Collins Blog.

This incident brings to life longstanding concerns about hospitals outsourcing their crucial functions to IT companies.  

Quite simply, I think it's insane, at least in the foreseeable future, as this example shows.

It also brings to mind the concerns that health IT, as an unregulated technology, causes dangers in hospitals with inadequate internal disaster and business continuity functions aside from fresh sheets of paper.  Such capabilities would likely be mandatory if health IT were meaningfully regulated.

The Joint Commission, for example, likely issued its stamp of approval for the affected hospitals, hospitals who had outsourced their crucial medical records functions to an outside party that sometimes went mute.  If someone was injured or died due to this outage, they would not care very much about the supposed advantages.

There's this in the article:

... “Issue appears to have something to do with DNS entries being deleted across RHO network and possible Active Directory corruption. Outage was across all North America clients as well as some international clients.”

Of course, patient safety was not compromised.

Finally:

Imagine being a patient, perhaps with a complex history, in extremis at the time of this outage.  

I, for one, do not want my own medical care nor that of my relatives and friends subject to cybernetic recordkeeping unreliability and incompetence like this, and the risk it creates.

-- SS

Aug. 8, 2012 addendum:

The Los Angeles Times covered this outage in a story aptly entitled "Patient data outage exposes risks of electronic medical records."

They write:

Dozens of hospitals across the country lost access to crucial electronic medical records for about five hours during a major computer outage last week, raising fresh concerns about whether poorly designed technology can compromise patient care.

My only comment is that the answer to this question is rather axiomatic.

They also quote Jacob Reider, acting chief medical officer at the federal Office of the National Coordinator for Health Information Technology, who said:

"These types of outages are quite rare and there's no way to completely eliminate human error"

This is precisely the type of political spin and misdirection I cautioned the Australian health authorities to evaluate critically.

Paper, unless there is a mass outbreak of use of disappearing ink, or locally hosted clinical IT, do not go blank en masse across multiple states and countries for any length of time, raising risk across multiple hospitals greatly, acutely and simultaneously.  (Locally hosted IT outages only cause "local" mayhem; see my further thoughts on this issue here).

-- SS

North Bristol Hits Appointment Problems: Another "Our Lousy IT Systems Screwed Up, But Patient Safety Was Never Compromised" Story

At my Dec. 2011 post "IT Malpractice? Yet Another "Glitch" Affecting Thousands of Patients. Of Course, As Always, Patient Care Was "Not Compromised" referencing prior posts, I wrote:

... At my Nov. 2011 post "Lifespan (Rhode Island): Yet another health IT glitch affecting thousands - that, of course, caused no patient harm that they know of - yet" I wrote:

There's been yet another health IT "glitch" that, of course, caused no patients to be harmed. See other "glitches" here, here, here and at other posts which can be found by searching this blog on the banal term 'glitch'.

Another "our clinical IT crapped out , BUT ... patient care/safety was never compromised" story just arose:

North Bristol hits appointment problems
E-Health Insider
11 January 2012
Rebecca Todd

Clinicians working at North Bristol NHS Trust have expressed concern about disruption to patient care, which they say is caused by appointment problems following the go-live of a new Cerner Millennium electronic patient record system.

I would have entitled the article "North Bristol hit by IT-created appointment problems."

Reported problems include patients being booked into non-existent clinic appointments or not being told about scheduled operations, resulting in some operations being cancelled.

No patient care compromise possible there. Who, after all, needs a timely operation? It frees up a lot of money for IT golf tournaments to let those of no value to society (i.e., the old, and those who will not admit computers in healthcare with deterministically revolutionize medicine because, well, they're magic) simply die due to delayed or cancelled surgery...

Ehealth Insider understands that some of the problems relate to the way the trust configured the EPR system; including setting up dummy clinics for which appointment letters were subsequently sent out.

It's never the software or computer's fault.

As a matter of fact, I have not seen any official response to the work of Dr. Jon Patrick at U. Sydney on the many software engineering flaws of another product of the same company. His work is entitled "A study of an Enterprise Health information System" and is at this link: http://sydney.edu.au/engineering/it/~hitru/index.php?option=com_content&task=view&id=91&Itemid=146. Do they have Class Action lawsuits in Australia?

In a regional BBC news report, aired on Monday evening, anonymous hospital clinicians called the implementation a “complete shambles” and said it represented a “potential danger” to patients.

According to the BBC report, the problems meant patients were being booked for impossible appointment times, such as 12.05 am, and quoted correspondence saying staff and the system were both on the “verge of meltdown."

The clinician comments are anonymous since non-anonymous reporting would get the clinicians declared health IT apostates, and then excommunicated. Non-anonymous 'whistleblowers' could also fear being sued due to possible gag clauses - the kind of clause hospital executives sign in violation of their fiduciary responsibilities to their staff and to patients. (See my 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards"at this link and the much-expanded essay on the same themes "Health Care Information Technology Vendors' Hold Harmless and Keep Defects Secret Clauses" at this link.)

Martin Bell, director of IM&T at the trust, confirmed to EHI that North Bristol had experienced some “unexpected problems” in the past few weeks with some of the outpatient appointments and theatre lists.

Bell stressed, however, that patient safety had not been compromised and that this continued to be the top priority.

There's that line again. Perhaps it's part of some hospital administrator JournoList-recommended catchphrase for describing how safety was not compromised during a major workflow disruption?

He said the problems were not down to the software itself, but due to “implementation and data migration difficulties in some clinics."

Right. Quite credible.

“Our information management and technology team, supported by our suppliers BT and Cerner, have been working very hard to sort out any initial issues as quickly as possible and we are already seeing improvements,” he said.

Congratulations are due. They are seeing "improvements" in dangerous clinical IT malfunctions that should never have to have been seen in the first place, if the statement is true.

“Many wards, our two minor injuries units and the Emergency Department, are successfully using the new system." The trust is one of the largest in the South of England, with more than 1,000 beds.

Just give them time.

EHI understands that as part of the Millennium implementation, dummy clinics were set up. Patients were then sent appointment letters for these clinics in error.

EHI also understands that some patients had also not turned up for scheduled operations because they had not been informed about the booking.

Bell apologised to patients who had been “inconvenienced during this transition period” and said staff had shown real dedication to continue to deliver patient care.

What if someone had been inconvenienced into their grave, or ends up there as a result of delays? On what wavelength will the apology be transmitted?

“We firmly believe that the new system, once fully implemented, will improve services for our patients and provide real value,” he said.

That seems to be the mantra, but delivery on such promises are rare. See "Pessimism, Computer Failure, and Information Systems Development in the Public Sector" as here, Public Administration Review 67;5:917-929, Sept/Oct. 2007, Shaun Goldfinch, University of Otago, New Zealand. The article is a cautionary article on IT that should be read by every healthcare executive documenting the widespread nature of IT difficulties and failure, the lack of attention to the issues responsible, and recommending much more critical attitudes towards IT.

A £69m contract for BT to deliver Cerner Millennium to three new, or ‘greenfield’ sites in the South of England was agreed in April 2010, under the auspices of the National Programme for IT in the NHS.

That would not be the failed National Programme for IT in the NHS, the NPfIT what went PfffT, would it?

North Bristol was the last of these three sites to go-live with the system in December last year.

It followed Oxford University Hospitals NHS Trust, which went live a week earlier, and Royal United Hospital Bath NHS Trust, which was the first to go-live in July.

Cerner said it was working closely with North Bristol and BT on the recent implementation of Millennium.

“In complex and large deployments, especially when migrating from two different systems, it is always anticipated that it would take time for the new system to bed-in,” it said in a statement.

The patients are given full informed consent on this issue, right? Right?

“Across much of the trust, the deployment has worked well. However, this is a major change management project and there have been some difficulties with outpatient appointments.

“Although this is not a problem with the software, Cerner is working in partnership with BT and trust staff to resolve any issue as quickly as possible.”

Link: BBC News

Right. Perhaps this software and claim needs testing - in a court of law.

The only thing missing is the word "glitch", though I am including that term in this posting's index, since I consider it another story in the ever-growing health IT "glitch" series.

-- SS

Addendum:

A reader sent me this comment:

How can anyone claim the problems at N. Bristol are unexpected. they are EXACTLY the same problems encountered in Taunton five years ago.

The Somerset Trust had sixteen cancelled go live dates and when Cerner 'Millennium" (note: they never defined which Millennium...) was switched on the whole hospital went into slow-motion.

Appointments could not be made at out-patient reception desks while patients waited and therefore had to be posted on. Twenty-four whole time equivalent clerks had to be employed to manage the back-up of appointment requests. So much for enhanced efficiency and cost savings.

The only possible response to this news is again to remind people of Einstein's famous definition of insanity: "repeating the same thing again and again and expecting a different result."

As for other Trusts, why no news of transformed performance by Cerner's systems at other Cerner implemented sites, Berkshire, Newcastle, Kingston, Oxford etc. The only 'good' news we get is that the system has been switched on.

If any of this expensive activity had really produced data, efficiency or cost gains, we would be drowning in Cerner press releases, the silence can only mean one thing, that their system is performing as poorly at other sites as it has in the South West.

Contrast this with the output and data produced openly by Birmingham University Hospital from its in-house created IT system.

Unfortunately one can only draw one serious conclusion about the whole Cerner/ NHS debacle - to paraphrase Mr. Clinton - "It's the (imho substandard) software, stupid!

This story needs serious investigation ... Recently US news items have started to discount the supposed efficiency gains for e-Health implementations and started to emphasize data capture and patient safety as the imperative for switch on. Unfortunately for Cerner supporters (and other vendors) the US Institute of Medicine's recent report stated unequivocally that there was (to everyone's apparent surprise) no quality evidence that e-Health improved patient safety.

I would contend no drug, therapeutic equipment or operation would or could be implemented in secondary care in the absence of critical and peer reviewed evidence of benefit [emphasis mine - ed.] that has characterized the rush to switch-on substandard IT solutions in English NHS hospitals.

I note that critical, peer-reviewed evidence, especially based on prospective randomized clinical trials as opposed to anecdotal, weak retrospective observational studies, have been deemed unnecessary in health IT.

Yet serious case reports of risk and injury from credible sources are deemed the true "anecdotes" and discounted. As I've written before, the science of medicine is nearly entirely lacking in the domain of health IT.

To put it in the words of James Le Fanu (channeling Sherlock Holmes) in his very apropos essay entitled "The Case of the Missing Data: The Dog That Didn't Bark", details on contrary strands of evidence that could reasonably have been expected to appear in evidential text are absent.

-- SS

IT Malpractice? Yet Another "Glitch" Affecting Thousands of Patients. Of Course, As Always, Patient Care Was "Not Compromised."

At my Nov. 2011 post "Lifespan (Rhode Island): Yet another health IT glitch affecting thousands - that, of course, caused no patient harm that they know of - yet" I wrote:

There's been yet another health IT "glitch" that, of course, caused no patients to be harmed. See other "glitches" here, here, here and at other posts which can be found by searching this blog on the banal term 'glitch'.

Add another case to the health IT glitch file, under the "do we feel lucky today?" patient risk category.
From the Pittsburgh Post-Gazette (I am quoted):

Computer outage at UPMC called 'rare' Systemwide disruption potentially dangerous, expert warns Saturday, December 24, 2011 By Jonathan D. Silver, Pittsburgh Post-Gazette

UPMC's electronic medical records system for inpatients went offline for more than 14 hours at nearly all its hospitals in the region, marking what the health system called a "rare" outage, but one that it claims did not harm patients.

First, as my aforementioned Nov. 2011 post and its contained links point out, these events are not as "rare" as they should be. (The asteroid colliding with Earth that caused the extinction of the dinosaurs - now that's a "rare" event.)

Second, as multiple posts on this blog have pointed out, the claims that "no patients were harmed" is both misleading and irrelevant:

Such claims of 'massive EHR outage benevolence' are misleading, in that medical errors due to electronic outages might not appear for days or weeks after the outage, depending on what information was corrupted/lost/misindentified/or otherwise mishandled after it is 'backloaded' once the system is up. All it takes is one med lost to cause misery and death. (I can speak about that from unfortunate personal experience.

Claims of 'massive EHR outage benevolence' are also irrelevant in that, even if there was no catastrophe directly coincident with the outage, their was greatly elevated risk. Sooner or later, such outages will maim and kill.

The outage affected a system designed by Cerner Corp., a global electronic records company, and customized by UPMC that doctors and nurses rely on for communication about patient records, medical orders and prescriptions.

It was unavailable from about 8:45 p.m. Thursday to 11 a.m. Friday at almost all of UPMC's hospitals except for Children's and UPMC Hamot in Erie, spokeswoman Wendy Zellner said.

"This is rare. This kind of widespread, extensive downtime would be rare," Ms. Zellner said.

Doctors and nurses continued to have access to patients' electronic records through backup systems, she said. They also had to resort to using old-fashioned paper records for documentation and orders.

"These things happen. They have really well spelled-out procedures for what to do when something goes down," Ms. Zellner said.

She acknowledged that doctors and nurses faced some challenges.

Faced 'some challenges?' In other words, care was compromised by the outage and the 'challenges' were to avoid medical error (and, of course, to make sure billing was unaffected):

Compromised -

a. To expose or make liable to danger, suspicion, or disrepute

b. To reduce in quality, value, or degree; weaken or lower.

Thousands of patients were affected, again reinforcing my point about how IT can and does greatly amplify the risks of paper -- as in my Rhode Island post -- such as errors and confidentiality breaches.

I cannot, for example, think of a single instance where thousands of paper records went unavailable simultaneously (unless, that is, someone lost the key to the Medical Records department), were made available to identity thieves en masse, or where thousands of medical orders were scrambled or truncated in a relatively short period of time as in Rhode Island.

These amplified risks could wipe out any advantages of EHR's over paper in a microsecond.

A partial list of facilities apparently affected in this latest episode of EHR mayhem, from this list:

• Magee-Womens Hospital of UPMC
• UPMC Bedford Memorial
• UPMC East
• UPMC Horizon
• UPMC McKeesport
• UPMC Mercy
• UPMC Montefiore
• UPMC Northwest
• UPMC Passavant
• UPMC Presbyterian
• UPMC Shadyside
• UPMC St. Margaret
• Western Psychiatric Institute and Clinic of UPMC

That accounts for several thousand active patients, I am sure.

(12/28 Addendum: Bed counts of PA hospitals are here. Searching on "University of Pittsburgh Medical Center", it can be seen that thousands of beds were indeed involved.)

"Whenever people aren't working in their native system and workflow I have to believe that is more cumbersome for the clinicians, but these folks are well-trained in what to do when these things happen."

This seems at best an insensitive and perhaps even inhumane bit of P.R. More "cumbersome" for the clinicians? What about the poor patients? How would Ms. Zellner feel, I wonder, if it were her mother, child or significant other on the Operating Room table or having an acute MI when the EHR/CPOE systems went down?

Ms. Zellner said UPMC's public relations staff was unaware of the outage until contacted by a reporter.

It appears P.R. is not very high on the list for receiving information when a crisis arises. I may have known of the outage before they did.

The outage was caused by a "bug" or glitch in software designed by a vendor affiliated with Cerner, Ms. Zellner said. She refused to identify the company.

"We're not trying to point fingers at different vendors. It's a database bug, that's all I can tell you."

(That is, it's not our fault, it's the fault of the database vendor. Hospitals, I regret to inform you - you are responsible for unapproved medical devices used in your facilities, no matter what the source.)

And there's that word "glitch" again, accompanied by the equally banal "bug."

It's just a "bug." Cute little critter!

Me again in the Post-Gazette:

Scot M. Silverstein, a doctor and assistant professor Healthcare Informatics at Drexel University in Philadelphia, disagreed with the use of the terms "bug" and "glitch."

"What occurred here was a disruptive, potentially dangerous major malfunction of a life-critical enterprise medical device," he said.

Somehow, when a clinician makes a mistake, the terms "bug" and "glitch" are never used. In fact, when clinicians fail to meet accepted professional standards of healthcare practice, it is called "malpractice."

I think we can all agree that a major near-full-day outage of an enterprise EHR affecting multiple hospitals and thousands of patients does not meet accepted professionals standard of life-critical computing practice. Yet, all this merits is the word "glitch." It seems to me that if patients are harmed by, in reality, what is (on its face) IT malpractice during such events, not only the clinicians affected should be held liable.

Ms. Zellner said the problem was not a "crash" of the system because there were alternate methods used to cope that prevented patient care from being compromised.

The usual refrain. Let me repeat my definition of "compromised:"

Compromised -

a. To expose or make liable to danger, suspicion, or disrepute

b. To reduce in quality, value, or degree; weaken or lower.

A simple question - if extended EHR outages like this never seem to "compromise" care, then why not eliminate health IT entirely and spend the hundreds of millions saved on patient care?

"This is not a crash of Cerner either," Ms. Zellner said. "I think a crash is, 'Oh my God, the sky is falling,' nobody can get anything."

I leave it to the readers to ascertain the computer expertise levels and reasonableness of what Ms. Zellner thinks a "crash" is.

Technicians from UPMC, Cerner and the third company [the 'mystery' database company? - ed.] worked together on-site to identify and fix the problem. Ms. Zellner said she did not know why it took 14 hours to fix and the underlying cause was still unclear.

"They know what the problem is and I believe it's been fixed, but we really don't know what triggered it," Ms. Zellner said. "I think the next step would be some actual software upgrades."

They "don't know what triggered the 'problem'" - is a proper translation that they have no idea what went wrong?

In fact, regarding another Cerner EHR system which was extensively studied (see "A Study of an Enterprise Information System" at this link), Dr. Jon Patrick came to the conclusion that one of the sources of catastrophic failures is poor software engineering that has made the behavior of the studied system "non-deterministic." Further, software upgrades are not protected from incremental changes made by maintenance and customization staff, and may introduce even more instability.

A software upgrade without clearly understanding "what triggered the problem" is simply asking for more trouble. (My bet, however, is that they attempt it anyway.)

A Cerner representative could not be reached for comment.

What's to say?

How about this:

Dr. Silverstein said based on what he was told about the computer outage, it means that hospital medical staff would have been unable to update patient charts and probably would not have been able to issue any orders through the system during the time it was off line.

He also questioned how up-to-date the hospital's redundant records were.

Repeating UMPC's statement from the article that appeared after I gave my quotes to the reporters: "Doctors and nurses continued to have access to patients' electronic records through backup systems, [the UPMC spokesperson] said. They also had to resort to using old-fashioned paper records for documentation and orders."

My stated fears of disruption and increased risk due to compromised care seem well-grounded.

In May, Allegheny General Hospital had to shut its electronic medical records computer system down because of problems with the vendor's hardware.

The hospital used backup procedures to continue care for patients, including using paper orders and record-keeping.

Wait ... I thought I'd heard these events were "rare." Two in the same city within six months?

---------------------------

Truth be told:

The primary rule in computing is:

Either you are in control of your information systems, or they are in control of you.

Clearly the latter was the case here.

The following questions arise:

• Was the software containing the "bug" properly vetted before being used on live patients? This is not just the vendor's obligation.
• If it was not vetted properly, why not?
• Was it an "upgrade" or patch? (If so, the same vetting rules apply.)

Further, the soft-selling of these incidents must end. The use of terms such as "bug" and "glitch" must also end. What occurred here, echoing my newspaper quote, was a disruptive, potentially catastrophic major malfunction of a life-critical enterprise medical device.

System-wide EHR crashes are not merely ‘glitches’ or ‘bugs.’ They need to be considered, as in medicine itself, as 'never events.' From AHRQ:

The term "Never Event" was first introduced in 2001 by Ken Kizer, MD, former CEO of the National Quality Forum (NQF), in reference to particularly shocking medical errors (such as wrong-site surgery) that should never occur. Over time, the list has been expanded to signify adverse events that are unambiguous (clearly identifiable and measurable), serious (resulting in death or significant disability), and usually preventable.

Further, re: "patient care was never compromised." How do they know that? In fact, this is 'spin' and word games on its face. By definition, if CPOE and chart updating was unavailable, patient care was compromised, where "compromised" means "increased levels of risk for error were created, requiring workarounds."

Further, as mentioned earlier, harms might not show up for some time. Lost orders, corrupted data, errors of omission or commission transcribing backup paper records into the computer ("backloading"), etc. can take their toll later. Post-outage vigilance is essential, putting even more stress on clinicians that increases likelihood of further error and that they certainly do not need. Clinicians are stressed enough already.

Finally:

IT personnel have not only deliberately inserted themselves into clinical affairs (e.g, via the HITECH Act of 2009), they have also done so with a stunning arrogance and unproven braggadocio about their systems "revolutionizing" medicine (whatever that means).

Indeed, they need to accept the medical responsibility and obligations this territorial intrusion entails.

On its face, this massive outage was the result of issues that did not meet accepted professional standards of IT practice for life-critical environments. Res ipsa loquitur.

Something was not vetted properly, there was a lack of redundancy, the IT personnel were NOT in control of their systems.

Just as when physicians don't provide care that meets accepted professional standards of healthcare, this incident and others like it are, by definition, a result of IT malpractice.

If patients are harmed, IT personnel and their management (often non-IT C-level officers) involved in this system need to be held accountable.

If they can't take the clinical heat (as clinicians do daily since the time they enter medical or nursing school), then they need to get out of the clinical kitchen.

-- SS

Note: see this take on these matters at the HIStalk blog:

UPMC’s Cerner systems go down for 14 hours at most campuses last Thursday and Friday, forcing them to go back to paper. The PR person blamed “a database bug,” which makes the above Oracle press release from this past summer a particularly fun read. Cerner and UPMC have an atypical vendor-customer relationship since they’ve invested big money together in innovation projects and UPMC runs a Cerner implementation business overseas.

Now we know who the unnamed "mystery database vendor" is...

-- SS

Dec. 29, 2011 Addendum:

Was UPMC acting as a "proving ground" for some Oracle-Cerner-UPMC experimental health IT technology that resulted in the crash? The claim of being an IT "proving ground" has been made in the past:

Pittsburgh Tribune
May 2, 2006UPMC partners with technology provider

The University of Pittsburgh Medical Center is taking another step in a quest to commercialize new medical technology.

UPMC on Monday signed a three-year deal with health care information technology provider Cerner Corp. to develop and market medicine-related technological advances. Both parties will contribute $10 million in cash, services and intellectual property to the effort.

The deal is a smaller version of an April 2005 deal between UPMC and information technology behemoth IBM.

As is the case in the IBM deal, UPMC will serve as a built-in proving ground for jointly developed technologies and products, with Cerner marketing the products and UPMC awarded a share of profits.

As I wrote at "Proving Ground for IT Tests On Children: Pioneers in Health IT, or Pioneers in Ignoring the Past?":

"A hospital and patients are not a learning lab for HIT vendors. The appropriate "proving ground" for new medical technology is the controlled clinical trial where participants (in this case, patients and healthcare professionals alike) have freedom of choice whether or not to participate, and a chance to give (or deny) consent after being fully informed of potential risk."This is a fundamental human rights issue.
-- SS