When working with complex medical software with a long history, there are bound to be problems. Having previously embraced an agile approach, the Kappa team’s journey took a disappointing turn over the previous 5-6 years. The once productive agile team had lost its way. In addition to a massive turn-over in people, good engineering practices had been replaced by silos and hand-offs. As a result, releases had become huge, slow, and full of bugs.
Formal Verification of Release 8 found over 80 major defects. Release 9 had nearly doubled that. Each defect found would need to be fixed and the software as a whole re-verified before it could be shipped, creating massive delays. Formal Verification isn’t supposed to be the first time things are tested. It is meant to be the last.
With the releases having become so bug-ridden and having failed an internal audit, senior leadership turned to Industrial Logic to help the Kappa team improve the quality of their next big software update, called release 10.
The engagement began with an onsite Readiness Assessment by our team of coaches, which unearthed the following dysfunctions:
FDA Audit fear would loom over every possible improvement which created a resistance to change or improve.
It seemed like unless a feature was tested manually, and appeared in the test trace matrix, the test didn't "count".
In some cases the “record of a test passing” was more important than whether the test still passed.
Instead of face to face conversations, collaboration had been reduced to comments written in Jira over many weeks.
The team batted work back and forth between development and verification until the item was finally called "Done".
The team used “pipelining” with the Business Analysts working in one Sprint, the Developers picking up stories in the next Sprint, Testers testing a story in the following Sprint. If issues were found they were fixed in a following Sprint, and tested again following that. This way of working caused confusion and massive delays.
Normalization of Deviance - the team had become accustomed to many things not working correctly. Their IDEs were not configured correctly, compiler, and system logs throw lots of errors.
When time to start the release process many critical defects would be found and verification would need to restart after fixes were applied.
Protection against defect regressions was missing, additionally manually testing this large / complex application was not sustainable.
Developers would implement technical tasks then hand them off to a tester who would be forced to figure out what portions of a partial use case specification that task mapped to for their test.
Defects were treated as isolated events, with developers fixing them and testers creating new tests. However, no changes were applied to the original feature specification or its corresponding test.
Testers would be forced into context switching depending on which technical task from a developer was ready for testing.
Everyone being assigned to multiple projects led them to describe themselves as “great multi-taskers”. In reality it was high-WIP and lacked the benefit of focus.
With such lengthy release cycles, the “Last Train Effect” emerges—each stakeholder wants their “baggage” (features) on the next train affecting both release dates and morale.
The product backlog contained thousands of items which were of questionable value, poorly defined, but the team feared deleting them.
Deficiencies in the previous release of the product kept interrupting work for the next release with small emergency projects that needed to be addressed first.
Industrial Logic’s coaches identified the following remedies to address the team’s deepest dysfunctions:
The Industrial Logic coaches guided the Kappa Team in forming ensembles (balanced teams) composed of at least two developers, at least one tester, and a business analyst which fostered collaboration, collective responsibility, improved quality, lower WIP, and decreased cycle time.
The ensemble worked together to get a story finished with shared responsibility from start to finish. The members would help each other with tasks which may not be strictly theirs from a reporting structure. (i.e. Developers might help the tester’s test. Tester’s might research information needed for the Developers.)
Audit Fear got allayed since now the group shouldered the burden for completing a work item correctly, rather than each team member feeling like they were shouldering the concern alone.
The Industrial Logic coaches first had the team rewrite their backlog items into User Stories (complete pieces of functionality) then they had them reformulate them into tests used as specifications or “test tables” to outline scenarios for testing. The ensembles would then work to get those specifications working as passing tests.
The Developers learned test-driven development, micro-testing, and refactoring.
They put together a set of instructions to properly configure their IDE to compile and run their system.
The product backlog was streamlined, old defects resolved, and unnecessary items removed.
Eliminated story point estimation and moved the ensembles to working only on small user stories which were on the order of 1-3-days, and tracked them on a Kanban board.
Introduced the team to Probabilistic Forecasting using Monte Carlo simulations.
Fixed date, variable scope, maintenance releases were established. Batches of deficiencies from the prior release were addressed and pushed out each quarter. The various stakeholders were able to negotiate what was most important to be included in the next quarterly release.
This eliminated the churn and Work in Progress at a project level which helped the teams to manage both streams of work.
“Our team loved what Industrial Logic helped us do. They trained and coached our staff to use a better, more repeatable way of developing quality software, which delivered far better results for our business.” Director, Systems Development, Kappa
Critical defects discovered during Formal Verification can add weeks or months to the testing timeline
as they prevent the software from being released to the field. Release 10 had the audacity to reveal
only a solitary defect. You read that right—just one! This stunning
accomplishment stands in stark contrast to the testing results for previous releases.
Release 10’s extremely low defect rate is now a badge of honor for everyone involved. It's not just about one release outperforming the others; it's about a team establishing a renewed commitment to excellence and having the knowledge, skills, and support needed to deliver quality results.
By dramatically slashing defect rates, Industrial Logic coaches helped the Kappa team reshape the testing timeline, making days and weeks vanish like magic.
Release 10 demonstrated how Industrial Logic helped the Kappa Team improve quality, reduce time spent fixing defects, and protect the company from failed FDA audits. We call that better software sooner.
The Kappa team's transformation stands as a testament to the power of continuous systematic improvement. The dramatic results came from applying the following ideas:
