by Yujan Shrestha, MD on July 14, 2021
“Rubber ducking” is the practice of talking to inanimate objects to think through bugs.
Are you curious about how to resolve defects in regulated medical device software? Are you working on a defect resolution process for your own company? If so, this article is for you.
A key component of any ISO 13485 compliant quality management system (QMS) is a defect resolution process. While quality metrics can vary from organization to organization, we can all agree that timely and thorough defects resolution should be one of them.
This document details the SKUASH debugging methodology for medical device software defect investigation and documentation. We have used the approach for several years on client projects, including an industry leading medical-device company with hundreds of installations around the world and a multi-tiered support organization. Not only is a documented procedure for evaluating complaints required by the regulations, but it is also a good engineering practice.
The medical diagnostic process is similar to software debugging in many ways:
Pop quiz: What does the “P” in HIPAA stand for? Privacy? Protection? Patient? It actually stands for “Portability”. Why all the fuss about portability? It can save lives. Physicians have been concerned about portability well before the digital age. A patient’s medical history has a universally accepted schema that allows physicians to quickly review a medical history. Need to know if the patient smokes? That is in the “social history” section. Need to know why the team made a decision for a critical care patient? That is in the “assessment and plan” section. The standardized schema facilitates safe handoffs and fast patient-record review. It is sort of like an API that healthcare providers use to transfer state when they speak to each other.
When it comes to organizing information, debugging patients and debugging software are similar. Why not have a schema for debugging? Wouldn’t it be nice to quickly load another engineer’s debugging state into your brain?
A thorough and well-documented defect-review procedure is required by medical-device regulations, including the US quality systems regulation (21 CFR Part 820 subpart I) and ISO 13485 section 8.2.2. We have investigated hundreds of medical device software defects over the years and have come to appreciate a structured mental framework for tackling difficult bugs. We use the SKUASH methodology to iterate on a solution and document the outcome. While it is probably overkill for those little bugs that come up during the development loop—which are below the purview of regulation anyway—it is especially useful for those tricky bugs that take more than a working day to solve. Overall, it is a communications tool that fits modern messaging, collaborative problem solving, and record maintenance when developing medical devices.
Now let us go into detail for each of the letters of the SKUASH acronym: situation, knowns, unexplained, assumptions, solutions, and hypotheses.
Take a few minutes to explain the situation in sufficient detail to replicate the issue in a controlled environment. If replication is not realistic, then describe the current state of the software and execution environment with as much detail as possible. What was the user doing when they encountered the bug? What is the version of the software used? Is there any local context about the environment needed for replication? Has the user’s IT department tampered with the code and decided it was okay to live edit some code on the server (this has actually happened). Have our support engineers already tried a few troubleshooting scripts? Has the customer’s IT department recently updated a PACS version that could be related to the error? The context of the situation should be subjective and conversational. Use the customer’s own words when possible.
List relevant facts. Initially, this is just the relevant details from the situation. While most knowns will have some degree of uncertainty, keep the threshold high so the team can reliably act on them.
Items may be added to this list by investigative activities such as:
Items may be removed from this list:
Why is this list useful?
List observations that were anomalous but have no immediate explanation. Also list any questions that are not answerable with the objective evidence at hand. A common trap for engineers—myself included—is to try to explain every anomaly to avoid the unsatisfactory feeling of leaving a stone unturned. This “depth first search” approach could end up wasting tons of time chasing down threads that could lead to technically interesting discoveries but do not make any progress to solving the problem. A “breadth first search” forces you to be a bit more scientific about which investigative threads you go down. Your time is limited and valuable. Time wasted going down rabbit holes or scratching academic itches is time taken away from shipping potentially life-saving products.
Items may be added to this list:
Items may be removed from this list:
Should you take away just one item from this article, let it be this: Never assume you are out of assumptions. You should revisit your assumptions often to mitigate confirmation bias. Confirmation bias is a major contributing factor to medical errors and even airline disasters. It is easy to chalk up conflicting evidence to laboratory errors or faulty sensors. Software engineers are not immune to this cognitive bias. Just replace “patient” with “bug” and the comparison holds up pretty well.
There is a natural urge for engineers to jump into hypothesis generation and testing. We are problem solvers after all. I encourage you to take a step back and enumerate your assumptions first. You will be surprised with how many assumptions you identify that will change the course of your investigation before you even start. Skipping this step is like pouring a foundation without a soil test.
It is also useful to revisit your “knowns” to see if they are assumptions instead. Nothing is absolute. Every known has a non-zero probability of being an assumption in disguise. The higher you climb the investigation ladder, the more important it is to question the stability of the “knowns” that you are standing on.
Items may be added to this list:
Items may be removed from this list:
It is worthwhile to come up with potential solutions to the problem even before you have begun investigation. There are a few reasons for this:
Items may be added to this list:
Here you list out potential hypotheses that could explain the root cause of the bug. Your investigative efforts should focus on proving hypotheses. If you run out of hypotheses, brainstorm some more by reviewing the other SKUASH elements. Failing that, you may need to do some untargeted investigation to prospect for more information with the goal to help you generate more hypotheses.
Items may be added to the list by:
Items may be removed from the list by:
Software engineering as a whole is both an art and a science. Debugging, however, is more of a science than an art. So why not use the scientific method and develop a shared framework for communication? In summary, the SKUASH method facilitates efficient problem solving and communication among engineers and customers alike.
Has this article helped you SKUASH a bug? Please subscribe to get more articles like this.
Need help SKUASH-ing your next milestone? Please check out our services.
We send out tips about once a month.
Articles about software development, AI, signal and image processing, medical regulations, and other topics of interest to professionals in the medical device software industry.
You may view previous articles here.
The Innolitics team, and experts we collaborate with, write all of our articles.