The Scatter-Gather Process

There is a prevalent way of managing people in a software process that involves a fair amount of work to ensure that each person gets tasks that are appropriate to their talent, knowledge, skill, and experience.

For a given feature or modification to a software product, a senior technical person will work out a design that is likely to be successful and fit into the business architecture. This master-maker will then divide the work into tasks that need to be completed by individuals.

Often a good master technician will consider their team’s skills and divide the work along individual skill lines. Some will divide the work along component lines and then consider which team member will be given which part to work on based on the components they know best.

The work is assigned and then the team members perform the new work when they complete their current task list, asking questions of the senior person as needed, and hopefully (in many but not all organizations) will test their pieces independently.

alt_text

all work is broken into tickets and assigned to individual contributors

This work is done in individual branches in a version control system (such as git) and tracked in a ticket system (such as Jira).

When all of the pieces are completed and the code has been reviewed, then the parts are integrated and merged into a common development branch. After testing the pieces together, they are merged into a main (trunk) branch or a release branch.

In other words,

  • The work is first divided into a number of pieces
  • Those pieces scattered to various team members
  • Once all of the pieces are complete they are gathered
  • Those pieces of software are merged/integrated
  • The combined code is is tested
  • When the combined solution works it is deployed

alt_text

Work broken down, scattered and gathered.

We call this style of working scatter-gather development.

What’s Appealing About It?

This system is appealing to people for several reasons:

alt_text

Ideally, parallelism should result in faster completion.
  • Individuals can work independently (mostly) with little chit-chat or coordination between them.
  • The majority of the work is done in parallel, ideally.
  • Work products are assigned to individuals at their level of expertise, which utilizes resources efficiently
  • Individuals are responsible and accountable for their pieces of work, which makes it easier to tell the skill levels and strengths (or weaknesses) of each person for performance management purposes.

Ideally, we should see work being done quickly with maximum parallelism, since each task is given to the person who can best complete it in time, and if the change is well-designed and the tasks are well-understood, then assembling the change at the end should not be too difficult or untimely.

Problems with Scatter-Gather

The system has a few notable problems to call out:

alt_text

Rather than being done in parallel, work is spread in time
  • Time Spread: The work generally isn’t done in parallel because each person has a list of tasks related to various stories. The lengths and order of task lists may vary per person. It may take the work longer to complete because the work is spread out across to-do lists.
  • Silos Form: Each person tends to be given work in a given space that they know well. Individuals tend to collect very specific and deep knowledge of a limited fragment of the system. Work tends to queue for specialists, delaying completion.
  • Brick Making (based on an old parable called out in a Huffington Post article): Devs are involved in making pieces of code rather than end-to-end integration and frequent releases.
  • Collaboration Discouraged: individualized, personalized tasks are not very conducive to ensemble programming. Also, since each developer has their work to do they can’t join on someone else’s task without putting their own at risk of being unfinished. The assignment of individual tasks is called unteaming for this reason. alt_text
Developers are too busy to collaborate
  • Burden on Pre-designer: Not only must the leader do the design work upfront (often alone) but also they must understand how best to assign that work to team members. They are also on the hook to answer any questions about implementation, integration, and the business case as the story progresses.
  • Late testing: The feature doesn’t exist as a complete end-to-end flow until the every piece is finished, so end-to-end testing is necessarily delayed.
    alt_text
Late integration and testing means late surprises.
  • Lost Opportunity: Team members don’t see the overall design at the same time, so their ability to spot flaws or opportunities is stymied.
  • Late Surprises: Any flaws in the original design or the workers’ interpretations of the design will show up late, during integration. If problems require rework, that rework will disrupt the task list of the individual who must fix the defect. alt_text
A late rejection disrupts the developer's workflow, making remaining items later.
  • Obscured Status: Progress of any whole job is distributed and isolated. When will all of the parts be started? When will all the parts be complete so that we can integrate them? In calendar time, when can we expect to see this feature in production? alt_text
Predicting completion is complicated and uncertain.

Prediction is Hard and Getting Harder

Companies in the software space try to manage the predictability and quality issues by making more precise estimates, but this is futile. It isn’t a low-quality estimate that causes the predictability problem.

The biggest problem with predictability is accidentally built into the scatter-gather system.

How much schedule slippage may be expected to accumulate across all the developers who are involved in completing the feature, considering that they will work on any given story at different times, any of them may be disrupted by late issues or emergencies, and any integration issues won’t be spotted until all the pieces are finished?

Odds Not On Our Side

The whole idea of splitting work across developers with individual schedules suggests some multiplication of risks.

Scattering whole jobs might mitigate some risks: if we assign N jobs, then even if some number of them don’t complete (X jobs) then at least N-X jobs will complete and be deliverable.

However, in scatter-gather, it’s not deliverable jobs that are launched but pieces of jobs. If even one piece of a feature isn’t completed, the whole feature is undeployable.

Consider you are given a piece of work to do. It’s one job, and it’s within your capability, but it has to be finished on schedule. Exactly two outcomes are possible: either it is completed in time, or it isn’t.

If the work is divided into two parts and assigned, then each piece may complete on schedule or not. For work to be successful, both parts must be finished. There are 3 other possibilities (only A finished, only B finished, or neither finished). There is one way to succeed and three ways to fail.

A Completed B Completed
Yes Yes
Yes No
No Yes
No No

If you divide work into 3 pieces, there are 8 possibilities with only one successful outcome.

A B C
Yes Yes Yes
Yes Yes No
Yes No Yes
Yes No No
No Yes Yes
No Yes No
No No Yes
No No No

With 4 parts split out, that’s 1 out of 16 possible outcomes (not illustrated).

With 5 parts split out, that’s 1 out of 32 possible outcomes (also not illustrated).

Having alignment on the ordering of the work can help alleviate some of the risks here. If everyone works on the same first task, then on the second when the first is done, then we can expect some of the work to be fully completed. If we have different priorities for different people then it’s likely that we’ll end up with all our tasks only partially complete.

If we’re going to coordinate our ordering this way then we might as well just work on the jobs together, most crucial job first.

More Difficult At Scale

With scatter-gather, the features are shuffled together like a deck of cards and individuals take on their parts of the jobs at different times.

No one person on the team knows what is necessary to complete the feature by the end of the week, or the end of the month unless there are per-story dashboards they can refer to. This requires extra coordination and some dedicated displays (most tools are capable of producing such a dashboard).

If we were all working on the same feature, we would be focused and only have one active feature to talk about. With many features in play, people may talk to us about the work we have already done or work we are going to be doing (interrupting the work we are now doing).

This is made worse when the feature is master-minded at a high level and then split across teams and not just individuals in the same team. Often teams have different priorities, which makes it more likely that work will be partially completed, untestable, and non-deployable.

Recognize that this system optimizes for busyness: developers are all very busy, working hard, and being highly utilized, while most of the features they have worked on are stalled waiting for other pieces to be done.

Again, the lack of coordination that obscures progress and delays the completion of whole features is a built-in aspect of the scatter-gather system, not a failing of the developers in the system.

What can we do to move things along?

We could pressure/motivate the developers to complete the work, but since they are all very busy that will motivate them to cut corners, which likely means more round-trips to and from testing and a higher likelihood of error or technical debt.

We could reprioritize the work, but doing so means that other features will wait longer as their work is being delayed (best case) or interrupted (worst case). The scatter-gather system is built on dividing the development organization’s focus, so if we use scatter-gather we must accept that we won’t be able to maintain clear focus.

If we choose to work in a scatter-gather way, then we have to accept the problems that the system imposes.

Alternatively, we could choose not to work in a scatter-gather way.

Instead of spreading pieces of work among people, we can gather the people to do whole units of work.

A multidisciplinary team can do the design together, recognizing issues and opportunities along the way. They can also do the work in deployable, valuable units with end-to-end integration and testing as a normal part of working.

Where work crosses team boundaries, team members from different teams can work the same feature together, perhaps building the back-end functionality as the front-end functionality is being built; perhaps building the on-device part as the in-the-cloud part is being built. Perhaps building microservices in parallel.

Because only one feature is in flight at a time, and it’s always runnable, status tracking is a breeze and the lead time is as short as possible. Even if the team decides to split up the work, it is actually being done in parallel with high-bandwidth communication and frequent integration.

Because the team will do the work together, no master technician has to pre-design and pre-split every feature.

Quite often the teams will work together in esemble using techniques such as mob programming or swarming.

It’s Just That Easy?

No.

The work of the individual becomes inseparable from the work of the team, which makes the whole individual performance management process difficult.

Knowledge sharing and personal improvement happen quickly and naturally. In one quarter of working together they may learn more than they did in the prior 2 years, but how can you be sure since you can’t separate their contributions from those of their peers?

When doing a feature at a time, you can’t report that all 12 features are in progress. This can make for some difficult conversations as one member of the product community realizes that their work is queued behind other community members’ work.

Teams may find working together to be unnatural after years of working as individual contributors in a scatter-gather system.

  • They may miss solitude and disconnection.
  • They may not like working with their peers.
  • They may struggle with time zones.
  • They may feel less efficient when they spend more time deliberating and discussing implementation instead of just following specs.
  • They may find that they’re spending more time learning and less time typing, and this may make them feel guilty.
  • They may miss the adrenaline rush of having too many things to do at once.
  • They may fear their weaknesses being exposed to the team
  • They may feel that the team is slowing them down

These problems are usually overcome if there is sufficient psychological safety among team members, and if the team is supported by managers who understand the value of a leaner and more collaborative process (and who are tired of the struggles of scatter-gather).

You have to choose the set of problems your context can tolerate and the set of benefits that it wants to receive.

Will it work for you? Who can say? Without trying you can’t discover what issues your local team will have working in your local culture. However, many organizations have found the simpler system of whole teams doing whole work to be very satisfying and of great value to the product community.

Maybe it’s worth a try.