Pattern Languages Revisited

 Author: Joshua Kerievsky
Started: 01/09/99
Updated: 05/29/99, 01/07/00
  State: Unfinished

This work is an attempt to show how we can improve both the FORM and CONTENT of our pattern languages.

Below, we will look at examples of some pattern language that have met with some success (meaning they've actually helped people and organizations) but which could be a lot more successful (more well-known and used), if their CONTENT and FORM were improved.

We'll begin with James O. Coplien's A Development Process Generative Pattern Language. What I intend to show is how a small selection of patterns in this pattern language can be rewritten to be much more effective at communicating the same ideas. What I'm about to do is by no means an attempt to one-up "The Cope", for whom I have a lot of respect. My purpose is to delineate deficiencies in both the FORM and CONTENT of current pattern languages, and to offer suggestions for improvement.

So, here are the first three patterns from James' pattern language, in their original form. Following these, I'll present my rewrites of them and explain what I've done.

Size the Organization

Problem: How big should the organization be?

Context:

You are building a software development organization to meet competitive cost and schedule benchmarks. The first release of the end product will probably be more than 25,000 lines of code.

The development organization is embedded in the context of a larger organization, usually that of the sponsoring enterprise or company (see the Rationale).

This applies specifically to organizations that create software, and may not as directly apply to organizations whose job is integrating existing software, for example.

Forces:

Too large, and you reach a point of greatly diminishing returns.

Large organizations are ships that are hard to steer.

Too small, and you don't have critical mass.

Overly small organizations have inadequate inertia and can become unstable.

Size affects the deliverable non-linearly. Communication overhead goes up as the square of the size, which means that the organization becomes less cohesive as the square of the size while the "horsepower" of the organization goes up only linearly.

New people interact much more strongly with others in the organization, and interact more broadly across the organization, than mature members of the society who have assimilated its context.

Solution:

By default, choose 10 people; experience has shown that suitably selected and nurtured small team can develop a 1,500 KSLOC project in 31 months, and a 200 KSLOC project in 15 months, or a 60KSLOC project in 8 months. Do not add people late in development, or try to meet deadlines worked backward from a completion date.

Resulting Context:

An organization where nearly everyone can have "global knowledge." We have found empirically that most roles in a project can handle interactions with about six or seven other roles; with 10 people, you can almost manage total global communications (and a fully connected network may not be necessary). On the other hand, 10 people is a suitable "critical mass."

This pattern is closely coupled to the patterns that support organizational growth below, Phasing it In and Apprentice. This pattern also interacts with Prototype.

Design Rationale:

Keeping an organization small makes it possible for everybody to know how the project works. Projects that do well have processes that adapt, and processes adapt well only if there is widespread buy-in and benefit. The dialogue necessary to buy-in and benefit can accrue only to small organizations. Tom DeMarco has noted that everybody who is to benefit from process should be involved in process work and process decision-making.

Having 10 people at the start is probably overkill, but it avoids the expense and overhead of adding more people later.

This pattern is for large projects. Projects larger than 25KSLOC can rarely be done by an individual (see Solo Virtuoso).

Different management styles (leadership-based, manager-based) lead to the success or failures of different organization types (democracy, republic, oligarchy). Leadership-based management styles help these organizations work; the compelling need for a democracy is less felt, because project members feel they are well-represented under an appropriate management style. [This may be a new pattern about management style.]

A single team is better than a collection of sub-teams. The faster a team breaks up into sub-teams worrying about their own responsibilities rather than those of the larger team, the less effective the enterprise will be as a whole.

Many software development cultures support technical manager groups up to this size. Adding more people would force a group split, which can cause a large decrease in productivity, all other things being equal.

Further study might evaluate the relationship between this pattern and Alexander's "The Distribution of Towns" [Alexander 2] and related patterns. Here, we stipulate that the social organization must be small; it reflects a ``Subculture Boundary'' [Alexander 13] and ``Identifiable Neighborhood'' [Alexander 14]. Alexander emphasizes the grander architectural context that balances support for the ecology with the economies of scale that large towns can provide, while supporting the xenophobic tendencies of human nature. Small organizations like that being built here rarely exist in isolation, but in the context of a broader supporting organization. This relationship to the larger organization invokes Patron.

As to adding people late in development, staff-month myths abound. One manager writes: "On [one] project, I grew from 10 to 20 people to meet a customer contract....with new people, [I] wound up three months late because of `absorption' of new folks into the organization."

Self-Selecting Team

Problem: Self-selecting team

Context:

You are building a software development organization to meet competitive cost and schedule benchmarks.

You are staffing up to meet a schedule in a given market.

Forces:

Empowerment depends on competency and the distribution of knowledge and power.

The worst team dynamics can be found in appointed teams.

The best team dynamics can be found in self-selecting teams.

Broad interests (music and poetry) seem to align with successful team players.

Solution:

Build self-selecting teams, doing limited screening on the basis of track record and broad interests.

Resulting Context:

An empowered, enthusiastic team willing to take extraordinary measures to meet project goals.

Solo Virtuoso

Problem: How big should the organization be?

Context:

You are building a software development organization to meet competitive cost and schedule benchmarks. The first release of the end product will probably be less than 25 KSLOC.

Rapid growth is not anticipated after the first release

Forces:

Some select individuals are able to build entire projects by themselves.

Organization size affects the deliverable non-linearly. Communication overhead g oes up as the square of the size, which means that the organization becomes less cohesive as the square of the size while the "horsepower" of the organization goes up only linearly.

Solution:

Do the entire design and implementation with one or two people.

Resulting Context:

An organization limited to small developments. Though there is a singleton development role, other roles may be necessary to support marketing, toolsmithing, and other functions. The productiv ity of a suitably chosen singleton developer is enough to handle sizable projects; here, we establish 25KSLOC as a limit, but see the rationale below for further parameters.

The pattern is not "License to Hack". Don't give up technical review, validation,and verification at appropriate times in development (Review the Architecture, Engage Customers).

This approach is rarely applicable, as it doesn't resolve some of the forces (many of them applicable) mentioned in Size the Organization. Where those forces don't apply, this pattern is a big win.

Design Rationale:

There are numerous examples of successful single-person developments. The dynamics of this development are different from those for a small team. The productivity of a single individual can be higher than a collection of several productive individuals. We have seen single-person developments generate 25KSLOC of deliverable code in 4 months (a craft interface for a telecommunication system); two-person developments do 135 KSLOC in 30 months. Many of these adhered faithfully to all stipulated reviews and verification steps.

Success, of course, depends on choosing the right person. Boehm notes a 20-fold spread between the least and most effective developers. A telecommunications developer recently told me that "having the right expertise means the difference between being able to solve a problem in a half hour, and never being able to solve the problem at all."

This pattern reduces the "truck number" of the organization (the number of people whose absence would threaten the organization were any one of them hit by a truck).

OK, now we'll rewrite these patterns using the form that the authors of APL use. Again, the reason we are doing this is not to one-up The "Cope", for whom we have a lot of respect, but to help point the community towards what we believe is a better form of expression for important knowledge.

Size the Organization**

Large software projects (greater than twenty-five thousand lines of code) are seldom delivered on time and within budget when the development team is too large or too small.

There are two arguments that have led us to this conclusion: 1. There are limits to the size of software development teams that allow them to work effectively. 2. Adding people late to a project rarely helps complete that project on time and within budget.

1. If a software development team is too large, you can reach a point of greatly diminishing returns. We have found empirically that an organization's size affects a deliverable non-linearly. Communication overhead goes up as the square of the size, which means that the organization becomes less cohesive as the square of the size while the "horsepower" of the organization goes up only linearly.

In addition, if the organization is too small, the team won't have critical mass and productivity will suffer. Projects larger than 25KSLOC can rarely be done by a SOLO VIRTUOSO and overly small organizations have inadequate inertia and can easily become unstable.

However, experience has shown that a suitably selected and nurtured small team of around 10 people can provide a suitable critical mass with a capacity to develop a 1,500 KSLOC project in 31 months, a 200 KSLOC project in 15 months, or a 60KSLOC project in 8 months.

Keeping the organization small makes it possible for everybody to have knowledge of how the project works ("global knowledge"). We have found empirically that most roles in a project can handle interactions with about six or seven other roles; with 10 people, you can almost manage total global communications (and a fully connected network may not be necessary).

Projects that do well have processes that adapt, and processes adapt well only if there is widespread buy-in and benefit. The dialogue necessary to buy-in and benefit can accrue only to small organizations. Tom DeMarco has noted that everybody who is to benefit from process should be involved in process work and process decision-making.

Further study might evaluate the relationship between this pattern and Alexander's "THE DISTRIBUTION OF TOWNS" [Alexander 2] and related patterns. Here, we stipulate that the social organization must be small; it reflects a ``SUBCULTURE BOUNDARY'' [Alexander 13] and ``IDENTIFIABLE NEIGHBORHOOD'' [Alexander 14]. Alexander emphasizes the grander architectural context that balances support for the ecology with the economies of scale that large towns can provide, while supporting the xenophobic tendencies of human nature. Small organizations like that being built here rarely exist in isolation, but in the context of a broader supporting organization. This relationship to the larger organization invokes PATRON.

2. Adding people late to a project rarely helps complete that project on time and within budget.

One manager writes: "On [one] project, I grew from 10 to 20 people to meet a customer contract....with new people, [I] wound up three months late because of `absorption' of new folks into the organization."

Many software development cultures support technical manager groups up to around 10 people. Adding more people would force a group split, which can cause a large decrease in productivity, all other things being equal. We have also found that a single team is better than a collection of sub-teams. The faster a team breaks up into sub-teams worrying about their own responsibilities rather than those of the larger team, the less effective the enterprise will be as a whole.

Therefore:

By default, choose ten people to establish critical mass in the development of large software systems and avoid adding individuals late in the game, or trying to work backwards from a completion date.

Having 10 people at the start of a large project can be overkill, but it avoids the expense and overhead of adding more people later. However, once a core team establishes an identity, it can grow graciously by PHASING IT IN or using one or more APPRENTICE. The organization can generate knowledge early on by building and throwing away a PROTOTYPE.

Self-Selecting Team**

The worst team dynamics can be found in appointed teams.

There are no perfect criteria for screening team memebers. Yet broad interests (music and poetry for, example) seem to indicate successful team players. Teams peopled with such individuals are often willing to take extraordinary measures to meet project goals.

However, when such interests are ignored, or when team members are appointed, team dynamics can suffer, greatly diminishing the productivity of a team.

Therefore:

Create enthusiastic teams by empowering people to self-select their teams. Do limited screening on the basis of track record and broader interests.

A SOLO VIRTUOSO or APPRENTICE may self-select a team.

Solo Virtuoso**

When a smaller software project (less than twenty-five thousand lines of code) is overstaffed, communication overhead inceases and talented individuals, who could produce the software entirely on their own, are bridled, their "horsepower" diminished.

We have said that organizational size affects the deliverable in a non-linear manner (SIZING THE ORGANIZATION). We have also observed that communication overhead goes up as the square of the size, which means that the organization becomes less cohesive as the square of the size while the "horsepower" of the organization goes up only linearly.

The question then is, what organizational size works best for smaller software projects?

The answer depends on the individual(s) involved in the project. The productivity of a single individual can be higher than that of a collection of productive individuals. We have seen single-person developments generate 25KSLOC of deliverable code in 4 months (a craft interface for a telecommunication system); two-person developments do 135 KSLOC in 30 months. Many of these adhered faithfully to all stipulated reviews and verification steps.

Boehm notes a 20-fold spread between the least and most effective developers. A telecommunications developer recently told me that "having the right expertise means the difference between being able to solve a problem in a half hour, and never being able to solve the problem at all."

The result of using a solo virtuoso is an organization limited to small development. Though there is a singleton development role, other roles may be necessary to support marketing, toolsmithing, and other functions. The productivity of a suitably chosen singleton developer is enough to handle sizable projects; here, we establish 25KSLOC as a limit.

Therefore:

Do the entire design and implementation with one or two of your most effective developers.

This pattern is not a "License to Hack." Solo Virtuoso's may still be subject to technical reviews, validation, and verification at appropriate times in the development cycle -- REVIEW THE ARCHITECTURE, ENGAGE CUSTOMERS.

Commentary

    Problem How big should the organization be?

I write:

    Large software projects (greater than twenty-five thousand lines of code) are seldom delivered on time and within budget when the development team is too large or too small.

Clearly there is a great degree of difference between these two problem statements. Cope's doesn't really make me feel uncomfortable - when I read it I'm not completely aware of the problem at hand: I only know that I'm being asked to consider the size of an organization. In my problem statement, I come right out and say what the problem is when teams are the wrong size. When I read the sentence, it makes me feel uncomfortable. I think "uh-oh, I better make sure I have the right number of folks on a project, otherwise I'm doomed."

(Author's note: Alas! I still haven't finished this paper. If you are finding it very valuable, perhaps you can offer me some encouragement to finish it).

To Do Explain how Cope's initial diagram does little to communicate the pattern non-verbally. I find it difficult to understand. Find something better for the re-written version. Explain how Cope's pointer's to Prototype and the other patterns fails to adequately explain why those patterns are important. Compare examples from Alexander. Explain how we added the asterisks to the name of the pattern, and the importance of this Explain how how we were tempted to rename the pattern Small Teams or 10-Person Teams to better reflect its meaning. Explain how Cope's language mixes us a lot of what CA and company would place either before or after the diamonds - that is, not in the explanatory text of the pattern itself. A quote from one of Tom DeMarco's of Fred Brook's excellent works would enhance Size The Organization pattern nicely. In Self-Selecting teams, the problem is mistated - the problem is when teams are not self-selecting, not that their is no perfect criteria for screening team members.