Process Architecture Study: Documentation
Experiment Description
Back to the Overview

As previously mentioned the three designers were to design the process architecture using their own methodology. The following sections outline some important characteristics of the experiment and its participants.

Participants

The study involved three designers. Their personal characteristics can be summarized in the following way: The OO Designer and the OS Designer are at comparable levels of experience within software design and development. The ML Designer is less experienced but within this paradigm it was practically impossible to find a more experienced developer who was able to participate.

Problem Statement

Each designer were to design the process architecture for a software system controlling a set of elevators in a building. In doing so, they were required to use their ``own'' paradigm as a development methodology.

The lift control problem was chosen simply because it was used by Guindon et al. [2] in their experiment. They describe it as a classical standard problem in software specification and software requirements research.

Our experiment focussed on a specific sub-activity of design. Thus we anticipated problems with the original problem statement because it lacked details that were necessary to know when one was to design the process architecture. This expectation led us to provide an extended problem statement that included a specification of the available hardware.

Data Collection

None of the three designers knew the problem in advance. They were given two hours to solve it. This time limit is similar to the one used by Guindon et al. [2]. The designers were instructed to think aloud during their design process. Within the two-hour session they were expected to produce a solution that could be handed over to another person for later evaluation. All three design sessions were videotaped. ]

The sessions started with a brief introduction to the purpose and the problem statement. In addition to the problem statement, they were only given paper and pencil to work with. The papersheets produced were enumerated to enable later identification and relation to the video recordings. The introduction was given by an observer who was present during the whole session. He also administrated the papersheets, controlled the video, kept track of time, and answered clarifying questions from the designers. The first author of this article acted as the observer in all three sessions.

The process of data analysis was divided into two separate part that are described in the following two subsections.

Data Analysis

The first part of the data analysis focussed primarily on the design process. This analysis comprised the following steps:
  1. The videotapes were examined with the purpose of producing a brief overview of each designer's process. The process was described as a sequence of activities where a change from one activity to another was identified as a situation in which the designer broke the on-going line of reasoning. This is closely related to the notion of breakdown, cf. [2].
  2. The activities were each characterized by the problem that the designer dealt with and the concepts used to analyze and solve this specific problem.
  3. The approach taken by the designer in each activity was described and related to the previous activities as well as relevant techniques within the paradigm.
These steps were developed because of the highly dense and complex nature of video material as a documentation medium. The purpose of the first step was to reduce this complexity by analysing and summarising the contents of each videotape in a chronological description. For each of the three sessions, this description was elicited by viewing the videotape four times, and it amounts to approximiately six pages. These process descriptions are not included in this article.

The second step reflects the main purpose of the study as stated in Section 1. Concepts and issues were identified among the three designers and plotted into a table that is used in the following sections of this article. Finally, the third step provided an understanding of the design process by describing how actvities were carried out and how the task was approached by each designer.

Product Evaluation

The second part of the data analysis focussed only on the solutions produced by the three designers. Their solutions were evaluated by two independent reviewers with the following characteristics: The documentation produced by the three designers was examined and evaluated independently by the two reviewers. The reviewers each used an hour on the examination and evaluation of the documents. The documents were handed over and they both used approximiately half an hour on the examination and gave an oral evaluation afterwards. In addition, the reviewers were asked to mark the three solutions with a grade representing how well the assignment had been solved.