Biomedical Breakthrough

How Industrial Logic helped a medical service company restore confidence and quality.

Like the Sword of Damocles, fear of an audit by the Food and Drug Administration (FDA) hung over the heads of employees at a major medical service company. While audits by the FDA exist to protect the public safety, they can create a lot of anxiety, since failing an audit can lead to the company receiving a public warning letter. If that happens, it can affect a stock price, negatively impact market share, or, in extreme cases, even shut down a company. This fear affected every aspect of their work and culture.

The fear of failing an FDA audit is so high that most companies regularly conduct their own internal audits, which are every bit as strenuous. The purpose of these internal audits are to find potential problems before the FDA does. One of these internal audits performed against the Kappa team’s last major release (Release 9) discovered quality issues severe enough to fail an FDA audit. If not addressed correctly and in a timely manner, these problems could have serious consequences for the company.

Background

When working with complex medical software with a long history, there are bound to be problems. Having previously embraced an agile approach, the Kappa team’s journey took a disappointing turn over the previous 5-6 years. The once productive agile team had lost its way. In addition to a massive turn-over in people, good engineering practices had been replaced by silos and hand-offs. As a result, releases had become huge, slow, and full of bugs.

Formal Verification of Release 8 found over 80 major defects. Release 9 had nearly doubled that. Each defect found would need to be fixed and the software as a whole re-verified before it could be shipped, creating massive delays. Formal Verification isn’t supposed to be the first time things are tested. It is meant to be the last.

With the releases having become so bug-ridden and having failed an internal audit, senior leadership turned to Industrial Logic to help the Kappa team improve the quality of their next big software update, called release 10.

Challenges

The engagement began with an onsite Readiness Assessment by our team of coaches, which unearthed the following dysfunctions:

benefit

Fear of FDA audit hindered change and innovation.

benefit

Siloed teamwork caused progress delays & hand-offs between disciplines.

benefit

Normalization of Deviance - accustomed to errors.

benefit

Disconnect between what was specified, what was developed, and what was tested.

benefit

Testers juggled tasks from different developers and multiple projects.

benefit

Stakeholder demands expanded scope and delayed releases.

Solutions

Industrial Logic’s coaches identified the following remedies to address the team’s deepest dysfunctions:


Forming Dedicated Cross-Functional Teams:

  • The Industrial Logic coaches guided the Kappa Team in forming ensembles (balanced teams) composed of at least two developers, at least one tester, and a business analyst which fostered collaboration, collective responsibility, improved quality, lower WIP, and decreased cycle time.

Moving Into A Collaborative Work Mode:

  • The ensemble worked together to get a story finished with shared responsibility from start to finish. The members would help each other with tasks which may not be strictly theirs from a reporting structure. (i.e. Developers might help the tester’s test. Tester’s might research information needed for the Developers.)
  • Audit Fear got allayed since now the group shouldered the burden for completing a work item correctly, rather than each team member feeling like they were shouldering the concern alone.

Reformulating Requirements As Tests:

  • The Industrial Logic coaches first had the team rewrite their backlog items into User Stories (complete pieces of functionality) then they had them reformulate them into tests used as specifications or “test tables” to outline scenarios for testing. The ensembles would then work to get those specifications working as passing tests.

Attention To Technical Excellence:

  • The Developers learned test-driven development, micro-testing, and refactoring.
  • They put together a set of instructions to properly configure their IDE to compile and run their system.

Eliminate Waste:

  • The product backlog was streamlined, old defects resolved, and unnecessary items removed.
  • Eliminated story point estimation and moved the ensembles to working only on small user stories which were on the order of 1-3-days, and tracked them on a Kanban board.
  • Introduced the team to Probabilistic Forecasting using Monte Carlo simulations.

Moving to Smaller, More Frequent Releases:

  • Fixed date, variable scope, maintenance releases were established. Batches of deficiencies from the prior release were addressed and pushed out each quarter. The various stakeholders were able to negotiate what was most important to be included in the next quarterly release.
  • This eliminated the churn and Work in Progress at a project level which helped the teams to manage both streams of work.

Results and Benefits

Industrial Logic joined forces with the Kappa team and together unleashed a tidal wave of improvements. From the beginning, the driving force behind this partnership was an unwavering commitment to quality. Release 10 boasted an impressive array of new features that could intimidate even the most seasoned software professionals. In addition to developing those new features, the Kappa team fixed major quality problems in the legacy code, and adopted a safer, more effective software development process.

Critical defects discovered during Formal Verification can add weeks or months to the testing timeline as they prevent the software from being released to the field. Release 10 had the audacity to reveal only a solitary defect. You read that right—just one! This stunning accomplishment stands in stark contrast to the testing results for previous releases.


“Our team loved what Industrial Logic helped us do. They trained and coached our staff to use a better, more repeatable way of developing quality software, which delivered far better results for our business.” Director, Systems Development, Kappa

Release 10’s extremely low defect rate is now a badge of honor for everyone involved. It's not just about one release outperforming the others; it's about a team establishing a renewed commitment to excellence and having the knowledge, skills, and support needed to deliver quality results.

By dramatically slashing defect rates, Industrial Logic coaches helped the Kappa team reshape the testing timeline, making days and weeks vanish like magic.

Release 10 demonstrated how Industrial Logic helped the Kappa Team improve quality, reduce time spent fixing defects, and protect the company from failed FDA audits. We call that better software sooner.

The Kappa team's transformation stands as a testament to the power of continuous systematic improvement. The dramatic results came from applying the following ideas:

  • Forming Dedicated Cross-Functional Team
  • Moving into a Collaborative Work Mode
  • Reformulating Requirements As Tests
  • Increasing Automated Test Coverage
  • Attention to Technical Excellence
  • Stripping the Backlog of Unneeded Items
  • Moving to Smaller, More Frequent Release
  • Limiting Work In Progress