The term paper for the class I’m taking (Paradigmatic Software Development) required that we write a review of a article that has appeared in a software engineering journal. I spent about 45 minutes trying to poke around the journal locator that UMass Dartmouth outsourced before giving up (they don’t let you search by topic, how lame is that?) and then remembering that Google has an academic focused search. Scholar.google.com found a bunch of articles about ‘open source’ and ‘software engineering’, I chose one entitled “Two Case Studies of Open Source Software
Development: Apache and Mozilla“. My review of the paper is over 7 pages long, you can download the entire thing in MS Word format or read the extended entry below.

—————————————-

At the beginning of his now famous essay “The Cathedral and the Bazaar”, Eric Raymond asked the question:
“Who would have thought even five years ago that a world class operating system could coalesce as if by magic out of part-time hacking by several thousand developers scattered all over the planet, connected only by the tenuous strands of the Internet?”
And indeed, no one did think it possible. But the ‘magic’ mentioned in his question turned out to be bigger than the operating system it spawned. The authors of the paper “Two Case Studies of Open Source Software Development: Apache and Mozilla”, set to investigate the claim, made by the proponents of the open source software development, that the open source style of software development (the ‘magic’)
“…has the capacity to compete successfully, and in perhaps many cases displace, traditional commercial development methods.”
The authors posited that there are four main differences between a commercial software development project and an open source development project. Stated in the terms of this course, those four things are:
* The “team” is mostly composed of volunteers, sometimes hundreds or even thousands of them.
* Work isn’t assigned by a project manager; people do what they want to do.
* There is no master design document or design phase.
* There is no explicit timeline for completing new features, no list of items to be included in the next build — in short there is no plan that says who does what when.

The authors set out to test these differences quantitatively and did so, writing up their results in a paper submitted in June of 2000. In their initial research, the authors studied the ubiquitous Apache web server and then in a follow up two years later, the Mozilla project, which has recently produced the Firefox web browser. Of each, they asked six questions:
1. What processes were used to develop the application?
2. How many people wrote code for new functionality? How many people reported problems? How many people repaired defects?
3. Were these functions carried out by distinct groups of people, that is, did people primarily assume a single role? Did large numbers of people participate somewhat equally in these activities, or did a small number of people do most of the work?
4. Where did the code contributors work in the code? Was strict code ownership enforced on a file or module level?
5. What is the defect density of the code?
6. How long did it take to resolve problems? Were high priority problems resolved faster than low priority problems? Has resolution interval decreased over time?

They then tested the results of those questions against seven hypotheses:
1. Open source developments will have a core of developers who control the code base. This core will be no longer than 10 to 15 people, and will create approximately 80% of more of the new functionality.
2. For projects that are so large that 10 to 15 developers cannot write 80% of the code in a reasonable time frame, a strict code ownership policy will have to be adopted to separate the work of additional groups, creating, in effect, several related OSS projects.
3. In successful open source developments, a group larger by an order of magnitude than the core will repair defects, and yet a larger group (by another order of magnitude) will report problems.
4. Open source developments that have a strong core of developers but never achieve large numbers of contributors beyond the core will be able to create new functionality but will fail because of a lack of resources devoted to finding and repairing defects.
5. Defect density in open source releases will generally be lower than commercial code that has only been feature-tested, that is, received a comparable level of testing.
6. In successful open source developments, the developers will be users of the software.
7. OSS developments exhibit very rapid responses to customer problems.

Before diving into the results of their study, I wanted to elucidate how I thought their research was related to the Paradigmatic Software Development course. First, I believe that the open source software development process described above is itself a paradigm (defined as a model or pattern, an archetype, for solving problems), probably fitting under the umbrella of the development and process paradigms like the waterfall model and agile processes. The open source software development process has produced some of the most widely used software in the world (Sendmail, Bind, Linux, Apache, etc.), which means that it merits a place in any software development textbook, especially one that examines paradigms. More importantly, the proponents of the open source style of development claim that their process may even displace the traditional style of software development, in effect challenging many of the ideas presented in this course.

Second, the open source software development process seems to share characteristics with the agile software development paradigm. It eschews planning, places a high value on collaboration, spends less time on documentation and elevates the people involved rather than specific tools or processes. Which came first? Is the open source software development process a loose implementation of the agile development paradigm? Or did the agile development paradigm come about as a result of the open source movement?

Finally, in attempting to answer the six questions mentioned above, the paper attempts to quantify certain aspects of the open source software development process, which I believe would qualify as a form of software metrics. Because their research was mainly concerned with the process and not the software, the authors extracted their data from three sources: the publicly available (and archived) developer email list, the source code repository (which maintains a record of every change made to the system) and the problem reporting database (more commonly known as the bug tracker).

So what were the answers to the six questions? The first question, which asked what processes were used when developing the applications, wasn’t one that could be answered simply by analyzing email archives, source code commit logs and bug databases and thus is qualitative, rather than quantitative. The processes the authors examined were split into six areas: roles and responsibilities, identifying work to be done, assigning and performing development work, prerelease testing, inspections, and managing releases. I’ll attempt to summarize the results of those five qualitative areas and then continue with the other five quantitative questions.

Roles and Responsibilities
The Apache Group was composed entirely of volunteers, started with 8 members, had twelve members during the period of time the authors were examining the process and at the conclusion of the research had twenty five members. All of the members had at least one other “day” job, which meant that the development process had to be decentralized and asynchronous. Email lists were the sole method of communication between the members; a simple voting system was put in place to help resolve conflicts. Each member had free reign to commit source code to any part of the project. Each developer used an iterative process when developing new functionality or modifications starting with discovery (elicitation), problem assignment (who will work on this), problem solving, testing, code review and finally source code commit.

In contrast, the Mozilla group was composed of twelve members, all of which were full time staff, but only four of which spent a significant portion of their time writing code (the other staff were responsible for QA, releases, bug tracking tools, etc.). Source code commit access was granted by the group but there was a model of code ownership, a member had to get the permission of the module owner before committing source code to a specific module.

Identifying Work to Be Done
The Apache Group used a three pronged approach to requirements elicitation phase: a developer email list, a problem reporting database system and Apache specific USENET newsgroups. The developer email list was the primary place of discussion for new functionality and patches for problems; problems were mostly reported in the problem reporting database system and the team relied on volunteers to manually parse and download with USENET newsgroups for potential new functionality and problems.

The Mozilla team actually maintained a roadmap document which specifies which features will be included in future releases as well as a timeline for the releases. Members and volunteers are encouraged to provide feedback and input into the roadmap and the timeline, but the final determination is made by the Mozilla group. Problems are added to a Mozilla created problem reporting database called Bugzilla.

Assigning and Performing Development Work
Both the Apache Group members and the Mozilla group members focus on areas of the code base where they have the most expertise or they are the most familiar. Both groups actively encourage potential solutions to problems from their respective communities but the actual work is always performed by a member of the group.

Prerelease Testing
After a solution has been identified and produced, the member applies the change on his or her local copy of the application to verify the functionality. The Apache Group (at the time of writing) maintained no regression or system tests, although they did maintain a thorough code review system. The Mozilla group runs a build on a daily basis for several platforms. A failed build results in a report being sent to the team; the producer of the failure is “hassled” until it gets fixed. Additionally, the Mozilla group maintains six separate test teams, each of whom take responsibility for separate areas of the product.

Inspections
The Apache Group members review every modification to a stable release before the fix or new functionality is applied to the source code repository. Changes to a development release are applied and then reviewed by both the group and the community at large, which results in valuable feedback. In the Mozilla group, inspections are performed by module owners in the context of the module and then by “superreviewers”, who review a modification from the standpoint of the entire system.

Managing Releases
The members of the Apache Group rotate amongst themselves the duties of the release manager, whose job is to identify any outstanding problems related to this release, get those problems fixed and guard access to the source code for that release (in effect making sure that no one destabilizes a stable product). In contrast, the Mozilla group runs a continuous build process using a product they built called Tinderbox, which automatically produces nightly builds of the product and approximately once a month issues a “milestone”. The milestone is a more manual process that involves members of the team freezing the code for a short period of time and eliminating outstanding bugs. The milestone process is driven by a subset of the Mozilla team known as the “drivers”.

Size of the Community
The authors tried to answer this question by breaking it up into three parts: users who contributed code, users who reported problems and users who repaired defects. During the period of time that authors analyzed, there were 182 people who contributed to 695 fixes, 249 people contributed over 6,000 code submissions and over 3,000 people contributed to approximately 3,900 problem reports, but only 591 of those reports could be directly traced to a change in the code base.

The group of people who contributed code to the Mozilla project was a bit larger: 486 users contributed code and 412 users contributed code that could be traced to fixes that were incorporated into the system. However, their problem report database was comparatively enormous: over 6,800 people contributed to about 58,000 problem reports, 11,616 of which can be traced to changes in the code.

Work Distribution
The authors concluded that approximately 83% of the modification requests and more than 88% of the added lines of code were added by the top 15 developers working in the Apache Group. However, the statistics also showed that the larger development community (users not part of the top 15 mentioned above) were more involved in the defect fixing process, with the top 15 developers only contributing to 66% of the fixes.

The distribution of fixes and code submissions in the Mozilla project was spread out over a larger group of users than the Apache Group with closer to 50 users submitting approximately 83% of the submissions to the code base. Even with an internal team devoted to testing, at some points during the study over 60% of the modification requests came from the community rather than the core group.

Code Ownership
As mentioned above, the members of the Apache Group were reported to have free reign to commit source code to any part of the project they saw fit. The authors, under the assumption that some sort of code ownership would reveal itself even with the above process, examined the code submission reports recovered from CVS commit logs and found their assumption to be wrong. Of the modules in the system with more than 30 changes, more than 95% had more than one developer making more than 10% of the changes. In contrast, the Mozilla group has an enforced code ownership policy where the owner is responsible for:
“… fielding bug reports, enhancement requests, patch submissions… The owner should facilitate good development, as defined by the developer community.”

Defects
The authors used a slightly modified KLOC measurement (instead of counting defects per thousand lines of code delivered they chose to count defects per thousand lines of code added) to compare the defect density of the Apache project and the Mozilla project against four other commercial projects they were able to acquire data for. Additionally, because the Mozilla group had not yet released a final product to the public, there was no post release data available for it. Curiously, both the Apache project and the Mozilla project had a lower defect density rate on the postfeature test than the corresponding commercial projects, but the Apache project had a higher defect density rate on the postrelease test than all of the commercial projects, which may be a result of the large number of installations.

Priority
Both the Apache and Mozilla problem reporting databases were analyzed to determine the amount of time it took from the report of the problem to the creation of a fix and release of the fix into the next release. The authors took the time to categorize the problems reported in the Apache problem reporting database into ‘Core’, ‘Most Sites’, ‘OS’ and ‘Major Optional’, assuming that items in the ‘Core’ would affect more users and thus should be fixed faster (it was generally assumed that the priority a single user enters will not be reflective of what the group thinks of the same issue). The authors turned out to be right about their assumption: problems reported in the ‘Core’ were fixed faster than problems in the ‘Most Sites’ category, problems in the ‘Most Sites’ were fixed faster than problems in the ‘OS’ category and so on. Problems reported in the Mozilla project were not re-categorized, but were instead analyzed on the basis of their entered priority. Again, the members of the project fixed issues faster that had a higher priority, although in this case the priority was what the end user actually entered.

Additionally, the authors split up each project into two periods of time (for the Apache project, before and after January 1, 1997) and then analyzed the average amount of time it took from problem discovery to problem resolution. In both projects, the average amount of time it took decreased from the first period to the second period, perhaps reflective of an increase in the number of volunteers for the project.

Contributions to Software Engineering
The sheer number of open source projects available on the internet today (almost 100,000 on sourceforge.net alone) speaks volumes about the contributions that the open source software development process has already made to software engineering. This particular article objectively confirms what certain people have been saying about the open source development process, specifically that the more users / volunteers that can look at the code, the higher the quality of the software. Additionally, it asserts that small groups of developers (anywhere from 10 to 15 developers) can operate quickly and effectively using only an ad hoc means of coordinating their work and that the benefits of the open source development process will typically come from areas where coordination isn’t a big factor. Finally, the authors hypothesize that for software where the developers are also highly knowledgeable users of the software product they are producing, it may make sense to experiment with the open source style of software development where new features are selected and implemented by the developers on the project, rather than by product management or marketing.

The idea that “the more eyeballs the less bugs there will be” was penned by Eric Raymond as part of his essay entitled “The Cathedral and the Bazaar” and is possibly one of the most compelling arguments for the open source development process. As mentioned above, the authors of the article were able to test his theory quantitatively and the results were eye opening. Not only were the postfeature defect density rates for both the Apache project and the Mozilla project well below the defect density rate of the four commercial projects they tested against, they both accomplished this feat while still maintaining a high rate of productivity. These facts alone should be enough to convince proponents of classical software engineering to reexamine their testing processes.

Another discovery of the article was the high level of productivity achieved by the core developers of both the Apache project and the Mozilla project while maintaining little in the way of formal process, elicitation, specification and design documents, very few (if any) face to face meetings and a geographically distributed, purely volunteer (in the case of the Apache Group) workforce. The authors did mention that some amount of the productivity gains should be attributed to the low-interdependency of the modules in the Apache project (while the Mozilla project was not as independent and did require formal code inspections). This low rate of interdependency of modules (synonymous with the ‘coupling’ design attribute), while already acknowledged as a ‘good thing’ in software design, might be emphasized even more in software development projects as not only a design attribute that eases software maintainability, but as a tool that can be used to make small teams of developers much more productive.

Last, the authors propose that some of the productivity and quality gains seen in both the Apache project and the Mozilla project should be attributed to the vast domain knowledge of the developers because the developers (it was assumed) were also users of the product. This idea should influence that way that developers are hired and assigned to projects (if it hasn’t already), in that consideration should be given to developers whose domain knowledge of the project may be greater but whose skills are better, on par or worse than those being considered for the same position. The article seems to suggest that developers with a large amount of domain knowledge and the trust of their existing team can and should be set loose to select, implement and test features that they believe to be necessary or interesting, rather than a product manager or marketing team making the decision.

My Opinion / Short Term Impact / Long Term Impact
I chose this paper because I’m interested in the open source software development process and because, at least so far, there have been and continue to be a number of open source software development successes, Mozilla and Apache being two of the most visible. I believe this paper empirically validates some of the tenets of the open source development process (many eyes make all bugs shallow), while showing that others may only work in certain situations and under certain conditions (the Mozilla project needed to have explicit code ownership and longer release cycles because their modules were more interdependent). Will the open source software development process displace traditional commercial development methods? Not any time soon. There are many other important factors that companies must consider before committing to an open source development process, chief among those being the issue of intellectual property. With that said, I’d recommend this paper to anyone starting a new project using the open source development process or transitioning to it.

The short term for this paper is almost over (written in 2002, this being May 2005), Apache continues to be the most used web server on the planet. Mozilla Firefox was recently released from beta to rave reviews. The ‘process’ appears to still be working. How will the results of this paper stand up in the long term? As I mentioned above, I don’t believe that the open source software development process will ever completely or even generally replace the traditional software development process (with it’s requirements, design, implementation, testing, etc..). However, I do believe that the percentage of software development projects that release their source code for the benefits of “many eyes” will increase. Companies and teams will begin to include volunteers in the process, helping them to increase the quality of their software. I also believe that progressive companies that embrace the agile development process will begin to mix in some of the tenets of the open source development process, especially in regards to the way in which work is assigned and completed. These companies will create smaller teams, maybe even combining resources available around the world, give them access to email and ad hoc collaboration tools (like Wikis) and will focus on modularizing their software, while increasing the speed at which they’re able to make changes and the quality of their software.