NFV Plugtest: a great success to help NFV move forward

I'm the Vice-Chair of the Testing, Implementation and Open Source (TST) Working Group at the ETSI NFV ISG.

In the test industry for the past 20 years, I have been working at Ixia for the past 11 years, always in the product management team for wireless test products. Most of my experience has been with mobility testing: GSM, UMTS, LTE, etc.

ETSI, through its Center for Testing and Interoperability (CTI) recently held its first ever NFV Plugtests^TM event in January. In addition to the wise decision to hold it in Leganes, Spain, just outside of beautiful and sunny Madrid, I would qualify the event a technical success. I explain why below.

There were 31 participating organizations, and several supporting organizations, including 4 key NFV open source communities: OPNFV, OSM, Open-O and Open Baton.

3 types of functions under test (FUT) were defined for the Plugtests:

• VNFs
• MANO, comprised of the NFVO and a VNFM together
• NFVI + VIM taken together as a platform

So for each test session, there was one MANO, one NFVI/VIM and at least one VNF being tested together. Test sessions lasted 3 hours.

The test plan, developed by the ETSI CTI, consisted of 26 test cases, classified in 5 groups:

1. Setup (onboarding) and instantiation
2. Scale
3. Scale VNF
4. Update
5. Terminate and teardown

Each test case had a clear purpose, a specific configuration, references to the applicable specifications, a list of features that each FUT must support in order to attempt the test case (applicability statement), and pre-conditions.

The results for the setup and termination groups were near perfect, achieving almost 100% success for both. The results for the 3 other groups, while encouraging, showed that there is still work to be done on those areas. This is to be expected, since the scale operations are quite complex, and the specifications have not been completed yet.

A deeper dive into the results of the scaling test groups reveals a few things. There were a total of 16 test cases related to scaling both NS (8 test cases) and VNF (8 test cases): there were 2 test cases (scale out and scale in) for each of the 4 trigger types for the scaling (manual by an operator, VNF indicator, VIM indicator and VNF/EM request). 31% of the sessions were able to attempt the manual trigger, for which there was a 100% success rate. For the other three trigger types, very few sessions were able to attempt the test cases, mostly because one of the FUTs involved in the test session did not support the feature.

As for the update group, which consisted of test cases to start and stop a VNF in an existing NS, and to add and remove a VNF from an NS, results were encouraging. 68% of the start/stop test cases were executed, with a success rate of about 96%. 33% of the add/remove test cases were attempted, with a 100% success rate.

Given the state of the ETSI NFV ISG specifications, which have just recently completed stage 2 (information modeling and procedure definitions), I think this is a positive result. The very complex features involved with auto-scaling, which are highly dependent on the complete specifications, were the majority of the test cases that were not attempted. That can be expected at this stage.

Some people would suggest that negative results from a Plugtest is an indication of failure. As a test professional for the past 20 years, I would argue the exact opposite: discovering issues early in the process is the best thing that can ever happen. Borrowing an expression that I learned from people in open source communities: you want to fail fast, so you can correspondingly correct fast. It might be bad grammar, but it’s excellent engineering...

Also, one of the main objectives of the Plugtest was to provide feedback to the ETSI NFV ISG on those specifications. The timing is appropriate, since stage 3 work (data modeling based on stage 2 specifications) is in progress, the input is valuable. Multiple issues were identified where clarifications are required, and these will be fed back into the NFV ISG.

Given the high level of industry participation and interest shown in the Plugtest, and that the feedback received by those participants indicated that they felt there was a very high value of having gone through the exercise, I can really qualify the Plugtest as having been a success. It gives the industry a good indication of where NFV is right now: healthy, active, moving beyond the teething pains of onboarding difficulties expressed in the past, and now moving on to tackle very complex features like autoscaling. The bar is poised to be raised for the next Plugtest.

The first ETSI NFV Plugtests report is available on the ETSI portal.