Like the Sword of Damocles, fear of an audit by the Food and Drug Administration (FDA) hung over the heads of employees at a major medical service company. While audits by the FDA exist to protect the public safety, they can create a lot of anxiety, since failing an audit can lead to the company receiving a public warning letter. If that happens, it can affect a stock price, negatively impact market share, or, in extreme cases, even shut down a company. This fear affected every aspect of their work and culture.

The fear of failing an FDA audit is so high that most companies regularly conduct their own internal audits, which are every bit as strenuous. The purpose of these internal audits are to find potential problems before the FDA does. One of these internal audits performed against the Kappa team’s last major release (Release 9) discovered quality issues severe enough to fail an FDA audit. If not addressed correctly and in a timely manner, these problems could have serious consequences for the company.
caduceus

Background

failed_audit

When working with complex medical software with a long history, there are bound to be problems. Having previously embraced an agile approach, the Kappa team’s journey took a disappointing turn over the previous 5-6 years. The once productive agile team had lost its way. In addition to a massive turn-over in people, good engineering practices had been replaced by silos and hand-offs. As a result, releases had become huge, slow, and full of bugs.

Formal Verification of Release 8 found over 80 major defects. Release 9 had nearly doubled that. Each defect found would need to be fixed and the software as a whole re-verified before it could be shipped, creating massive delays. Formal Verification isn’t supposed to be the first time things are tested. It is meant to be the last.

With the releases having become so bug-ridden and having failed an internal audit, senior leadership turned to Industrial Logic to help the Kappa team improve the quality of their next big software update, called release 10.

Challenges

The engagement began with an onsite Readiness Assessment by our team of coaches, which unearthed the following dysfunctions:

Culture

self_protective_behavior.png

FDA Audit fear would loom over every possible improvement which created a resistance to change or improve.

It seemed like unless a feature was tested manually, and appeared in the test trace matrix, the test didn't "count".

In some cases the “record of a test passing” was more important than whether the test still passed.

Teamwork

alone-small.png

Instead of face to face conversations, collaboration had been reduced to comments written in Jira over many weeks.

The team batted work back and forth between development and verification until the item was finally called "Done".

The team used “pipelining” with the Business Analysts working in one Sprint, the Developers picking up stories in the next Sprint, Testers testing a story in the following Sprint. If issues were found they were fixed in a following Sprint, and tested again following that. This way of working caused confusion and massive delays.

Quality

normalization-of-deviance.png

Normalization of Deviance - the team had become accustomed to many things not working correctly. Their IDEs were not configured correctly, compiler, and system logs throw lots of errors.

When time to start the release process many critical defects would be found and verification would need to restart after fixes were applied.

Protection against defect regressions was missing, additionally manually testing this large / complex application was not sustainable.

Specifications

curious_not_furious_small.png

Developers would implement technical tasks then hand them off to a tester who would be forced to figure out what portions of a partial use case specification that task mapped to for their test.

Defects were treated as isolated events, with developers fixing them and testers creating new tests. However, no changes were applied to the original feature specification or its corresponding test.

Workflow

slow_qa.png

Testers would be forced into context switching depending on which technical task from a developer was ready for testing.

Everyone being assigned to multiple projects led them to describe themselves as “great multi-taskers”. In reality it was high-WIP and lacked the benefit of focus.

Product Planning

bad_data.png

With such lengthy release cycles, the “Last Train Effect” emerges—each stakeholder wants their “baggage” (features) on the next train affecting both release dates and morale.

The product backlog contained thousands of items which were of questionable value, poorly defined, but the team feared deleting them.

Deficiencies in the previous release of the product kept interrupting work for the next release with small emergency projects that needed to be addressed first.

Solutions

Industrial Logic’s coaches identified the following remedies to address the team’s deepest dysfunctions:

benefit
Forming Dedicated Cross-Functional Teams:

The Industrial Logic coaches guided the Kappa Team in forming ensembles (balanced teams) composed of at least two developers, at least one tester, and a business analyst which fostered collaboration, collective responsibility, improved quality, lower WIP, and decreased cycle time.

mob_dondont
Moving Into A Collaborative Work Mode:

The ensemble worked together to get a story finished with shared responsibility from start to finish. The members would help each other with tasks which may not be strictly theirs from a reporting structure. (i.e. Developers might help the tester’s test. Tester’s might research information needed for the Developers.)

Audit Fear got allayed since now the group shouldered the burden for completing a work item correctly, rather than each team member feeling like they were shouldering the concern alone.

TDD_story
Reformulating Requirements As Tests:

The Industrial Logic coaches first had the team rewrite their backlog items into User Stories (complete pieces of functionality) then they had them reformulate them into tests used as specifications or “test tables” to outline scenarios for testing. The ensembles would then work to get those specifications working as passing tests.

customization
Attention To Technical Excellence:

The Developers learned test-driven development, micro-testing, and refactoring.

They put together a set of instructions to properly configure their IDE to compile and run their system.

remove
Eliminate Waste:

The product backlog was streamlined, old defects resolved, and unnecessary items removed.

Eliminated story point estimation and moved the ensembles to working only on small user stories which were on the order of 1-3-days, and tracked them on a Kanban board.

Introduced the team to Probabilistic Forecasting using Monte Carlo simulations.

deployment
Moving to Smaller, More Frequent Releases:

Fixed date, variable scope, maintenance releases were established. Batches of deficiencies from the prior release were addressed and pushed out each quarter. The various stakeholders were able to negotiate what was most important to be included in the next quarterly release.

This eliminated the churn and Work in Progress at a project level which helped the teams to manage both streams of work.

Results and Benefits

Industrial Logic joined forces with the Kappa team and together unleashed a tidal wave of improvements. From the beginning, the driving force behind this partnership was an unwavering commitment to quality.

Release 10 was very large. Not only did it correct deficiencies in prior versions of the software, but it also expanded its interoperability with Laboratory Information Systems. As the release date approached, stakeholders continued to add scope to an already large set of features.
specifications
“Our team loved what Industrial Logic helped us do. They trained and coached our staff to use a better, more repeatable way of developing quality software, which delivered far better results for our business.” Director, Systems Development, Kappa

defects

Critical defects discovered during Formal Verification can add weeks or months to the testing timeline as they prevent the software from being released to the field. Release 10 had the audacity to reveal only a solitary defect. You read that right—just one! This stunning accomplishment stands in stark contrast to the testing results for previous releases.

Release 10’s extremely low defect rate is now a badge of honor for everyone involved. It's not just about one release outperforming the others; it's about a team establishing a renewed commitment to excellence and having the knowledge, skills, and support needed to deliver quality results.

By dramatically slashing defect rates, Industrial Logic coaches helped the Kappa team reshape the testing timeline, making days and weeks vanish like magic.

Summary

Release 10 demonstrated how Industrial Logic helped the Kappa Team improve quality, reduce time spent fixing defects, and protect the company from failed FDA audits. We call that better software sooner.

The Kappa team's transformation stands as a testament to the power of continuous systematic improvement. The dramatic results came from applying the following ideas:

  • Forming Dedicated Cross-Functional Team
  • Moving into a Collaborative Work Mode
  • Reformulating Requirements As Tests
  • Increasing Automated Test Coverage
  • Attention to Technical Excellence
  • Stripping the Backlog of Unneeded Items
  • Moving to Smaller, More Frequent Release
  • Limiting Work In Progress
passed_audit