Plugtests™ & Standards

What do Plugtests™ have to do with standardization?

They take place while the standard is still in the making. As long as there are changes to the standard, there will be interest in Plugtests™. Once the standard has settled, participation at Plugtests™ drops. In contrast, interoperability testing as done by e.g. test houses or network operators comes later in the process, i.e. when the implementation is about to be ready for the mass market.

How can Plugtests™ enhance the quality of specifications?

Plugtests™ take place early in the process, namely while the standard is still being drafted. They dont guarantee interoperability, but they increase the probability of interworking. They enable the identification of areas for improvements, errors and ambiguities within the standard.

Do products get certified at Plugtests™?

No

Are engineers testing their products or the standard?

To test a standard, it has to be implemented in software and/or hardware. Engineers arrive with prototypes of their implementations at Plugtests™. They find errors in both the standard and their prototypes. From experience, all implementations undergo changes during the interoperability event. There is no way that they can go onto the market right after the event.

Are test specifications needed?

Yes. Test specifications are, in general, written in prose.

Don't well written standards make Plugtests™ superfluous?

The better standards are written and the less options they have, the easier it is to achieve interoperability. Good standards don't make Plugtests™ superfluous, but good standards necessitate fewer of them.

Why not use formal testing methods (conformance tests) instead of Plugtests™?

Discussions on how to validate a specification often take on the character of a 'religious issue', i.e. one where there are very strong divergent views with no realistic chance of reconciliation. Theoretically, if the standard is tight and has only a limited number of options, implementations that pass conformance tests will automatically be interoperable. In practice, however, this is generally not the case. Likewise, if two implementations are interoperable, they may not necessarily conform to the standard.
Conformance tests and Plugtests™ are very different things. They complement each other nicely. We like to compare them to food and water: you need both.

What is the feedback mechanism from Plugtests™ into the standards process?

This depends very much on the standards group. Our favourite example is that of IPv6, the new Internet Protocol to replace the current version, IPv4. When IPv6 was in its hot (standards making) phase, the working group chairs for the standards work of IPv6 participated in the interoperability event, which took place a week prior to the standards meetings. Bugs and ambiguities found in the standard were discussed right at the Plugtests™ and reported to the standards group. ITU-T SG16 (H.323 etc.) has an 'Implementers Guide' that is to a large extent the result of H.323 Plugtests™ and discussions on a mailing list dedicated to developers.
The feedback loop to the standards body is not automatic. Care has to be taken that information learned at Plugtests™ does not get lost.