Phases of Formal Review

Phases of a formal review

In contrast to informal reviews, formal reviews follow a formal process. 

A typical formal review process consists of six main steps:

1. Planning
2. Kick-off
3. Preparation
4. Review meeting
5. Rework
6. Follow-up

Planning

The review process for a particular review begins with a 'request for review' by the author to the moderator (or inspection leader). 

A moderator is often assigned to take care of the scheduling (dates, time, place and invitation) of the review.

On a project level, the project planning needs to allow time for review and rework activities, thus providing engineers with time to thoroughly participate in reviews.

For more formal reviews,e.g. inspections, the moderator always performs an entry check and defines at this stage formal exit criteria.

The entry check is carried out to ensure that the reviewers' time is not wasted on a document that is not ready for review. 

A document containing too many obvious mistakes is clearly not ready to enter a formal review process and it could even be very harmful to the review process. 

It would possibly demotivate both reviewers and the author. Also, the review is most likely not effective because the numerous obvious and minor defects will conceal the major defects.

Although more and other entry criteria can be applied, the following can be regarded as the minimum set for performing the entry check:

• A short check of a product sample by the moderator (or expert) does not reveal a large number of major defects. 

• The document to be reviewed is available with line numbers.The document has been cleaned up by running any automated checks that apply.

• References needed for the inspection are stable and available.

• The document author is prepared to join the review team and feels confident with the quality of the document.

If the document passes the entry check, the moderator and author decide which part of the document to review. 

The maximum number of pages depends, among other things, on the objective, review type and document type and should be derived from practical  experiences within the organization. 

For a review, the maximum size is usually between 10 and 20 pages.

In formal inspection, only a page or two may be looked at in depth in order to find the most serious defects that are not obvious.

After the document size has been set and the pages to be checked have been selected, the moderator determines, in cooperation with the author, the composition of the review team. 

The team normally consists of four to six participants, including moderator and author.

To improve the effectiveness of the review, different roles are assigned to each of the participants. These roles help the reviewers focus on particular types of defects during checking. 

This reduces the chance of different reviewers finding the same defects. 

The moderator assigns the roles to the reviewers. Within reviews the following focuses can be identified:

• focus on higher-level documents, e.g. does the design comply to the requirements;
• focus on standards, e.g. internal consistency, clarity, naming conventions, templates;
• focus on related documents at the same level, 
• focus on usage, e.g. for testability or maintainability.

Kick-off

An optional step in a review procedure is a kick-off meeting. 

The goal of this meeting is to get everybody on the same wavelength regarding the document under review and to commit to the time that will be spent on checking.

Also the result of the entry check and defined exit criteria are discussed in case of a more formal review. 

In general a kick-off is highly recommended since there is a strong positive effect of a kick-off meeting on the motivation of reviewers and thus the effectiveness of the review process.

During the kick-off meeting the reviewers receive:

• A short introduction on the objectives of the review and the documents. 
• The relationships between the document under review and the other documents (sources) are explained, especially if the number of related documents is high.

Role assignments, checking rate, the pages to be checked, process changes and possible other questions are also discussed during this meeting. 

Of course the distribution of the document under review, source documents and other related documentation, can also be done during the kick-off.

Preparation

The participants work individually on the document under review using:

• the related documents, 
• procedures, 
• rules and 
• checklists provided. 

The individual participants identify:

• defects, 
• questions and 
• comments, 

according to their understanding of the document and role.

All issues are recorded, preferably using a logging form.

The annotated document will be given to the author at the end of the logging meeting. Using checklists during this phase can make reviews more effective and efficient.

A critical success factor for a thorough preparation is the number of pages checked per hour. This is called the checking rate. 

The optimum checking rate is the result of a mix of factors, 

• including the type of document, 
• its complexity, 
• the number of related documents and 
• the experience of the reviewer.

Usually the checking rate is in the range of five to ten pages per hour, but may be much less for formal inspection, e.g. one page per hour. 

During preparation, participants should not exceed this criterion. 

By collecting data and measuring the review process, company-specific criteria for checking rate and document size can be set, preferably specific to a document type.

Review meeting

The meeting typically consists of the following elements (partly depending on the review type): 

• logging phase
• discussion phase 
• decision phase

During the logging phase the issues, e.g. defects, that have been identified during the preparation are mentioned page by page, reviewer by reviewer and are logged either by the author or by a scribe.

A separate person to do the logging (a scribe) is especially useful for formal review types such as an inspection.

To ensure progress and efficiency, no real discussion is allowed during the logging phase. If an issue needs discussion, the item is logged and then handled in the discussion phase. 

A detailed discussion on whether or not an issue is a defect is not very meaningful, as it is much more efficient to simply log it and proceed to the next one. 

Furthermore, in spite of the opinion of the team, a discussed and discarded defect may well turn out to be a real one during rework.

Every defect and its severity should be logged. The participant who identifies the defect proposes the severity.

Severity classes could be:

• Critical: defects will cause downstream damage; the scope and impact of the defect is beyond the document under inspection.
• Major: defects could cause a downstream effect (e.g. a fault in a design can result in an error in the implementation).
• Minor: defects are not likely to cause downstream damage (e.g. non-compliance with the standards and templates).

In order to keep the added value of reviews, spelling errors are not part of the defect classification. Spelling defects are noted, by the participants, in the document under review and given to the author at the end of the meeting or could be dealt with in a separate proofreading exercise.

During the logging phase the focus is on logging as many defects as possible within a certain time frame. To ensure this, the moderator tries to keep a good logging rate (number of defects logged per minute). 

In a well-led and disciplined formal review meeting, the logging rate should be between one and two defects logged per minute.

For a more formal review, the issues classified as discussion items will be handled during this meeting phase. Informal reviews will often not have a separate logging phase and will start immediately with discussion.

Participants can take part in the discussion by bringing forward their comments and reasoning. As chairman of the discussion meeting, the moderator takes care of people issues.

Reviewers who do not need to be in the discussion may leave, or stay as a learning exercise. The moderator also paces this part of the meeting and ensures that all discussed items either have an outcome by the end of the meeting, or are defined as an action point if a discussion cannot be solved during the meeting.The outcome of discussions is documented for future reference.

At the end of the meeting, a decision on the document under review has to be made by the participants, sometimes based on formal exit criteria. The most important exit criterion is the average number of critical and/or major defects found per page (e.g. no more than three critical/major defects per page).

If the number of defects found per page exceeds a certain level, the document must be reviewed again, after it has been reworked. If the document complies with the exit criteria, the document will be checked during follow-up by the moderator or one or more participants. Subsequently, the document can leave the review process.

If a project is under pressure, the moderator will sometimes be forced to skip re-reviews and exit with a defect-prone document. Setting, and agreeing, quantified exit level criteria helps the moderator to make firm decisions at all times.

In addition to the number of defects per page, other exit criteria are used that measure the thoroughness of the review process, such as ensuring that all pages have been checked at the right rate. The average number of defects per page is only a valid quality indicator if these process criteria are met.

Rework

Based on the defects detected, the author will improve the document under review step by step. Not every defect that is found leads to rework. It is the author's responsibility to judge if a defect has to be fixed.

If nothing is done about an issue for a certain reason, it should be reported to at least indicate that the author has considered the issue. Changes that are made to the document should be easy to identify during follow-up. Therefore the author has to indicate where changes are made.

Follow-up

The moderator is responsible for ensuring that satisfactory actions have been taken on all (logged) defects,process improvement suggestions and change requests.

Although the moderator checks to make sure that the author has taken action on all known defects, it is not necessary for the moderator to check all the corrections in detail. If it is decided that all participants will check the updated document, the moderator takes care of the distribution and collects the feedback.

For more formal review types the moderator checks for compliance to the exit criteria. In order to control and optimize the review process, a number of measurements are collected by the moderator at each step of the process. Examples of such measurements include:

• number of defects found
• number of defects found per page
• time spent checking per page
• total review effort, etc.

It is the responsibility of the moderator to ensure that the information is correct and stored for future analysis.