Evaluating benefits of requirement categorization in natural language specifications for review improvements

Abstract

One of the most common ways to ensure the quality of industry specifications is technical review, as the documents are typically written in natural language. Unfortunately, review activities tends to be less effective because of the increasing size and complexity of the specifications. For example at Mercedes-Benz, a specification and its referenced documents often sums up to 3.000 pages. Given such large specifications, reviewers have major problems in finding defects, especially consistency or completeness defects, between requirements with related information that are spread over large or even different documents. The classification of each requirement according to related topics is one possibility to improve the review efficiency. The reviewers can filter the overall document set according to particular topics to check consistency and completeness between the requirements within one topic. In this paper, we investigate whether this approach really can help to improve the review situation by presenting an experiment with students reviewing specifications originating from Mercedes-Benz with and without such a classification. In addition, we research the experiment participants’ acceptance of an automatic classification derived from text classification algorithms compared to a manual classification and how much manual effort is needed to improve the automatic classification. The results of this experiment, combined with the results of previous research, lead us to the conclusion that an automatic pre-classification is an useful aid in review tasks for finding consistency and completeness defects.