Testbeds are a place for pre-competitive collaboration on a practical level and thus complement theoretical cooperation in standardization and practical in-house product development. The focus is on the emerging ecosystem, and standards are seen as a necessary means to an end. A testbed typically pursues the following goals:

• Establishment of the ecosystem: The creation of a new ecosystem requires a holistic view of the standards as well as the "trappings". It is important to bring the standards into the future context in an application-oriented manner. Gaps in standardization, understanding of use, best practices and topics such as usability can be advanced.
• Supporting standard development: Early prototyping and testing of approaches undergoing standardization allows them to be validated and optimized at an early stage.
• Ensuring interoperability : Cross-manufacturer interoperability is crucial for the success of TSN as a whole and therefore also for individual companies. Early testing and cooperation ensures this requirement.
• Support for product development: A very pragmatic motivation for entering into early prototyping is the expected head start in future productions.
• A place for innovation: An open and early exchange between technology suppliers and users not only ensures the success of future solutions, it also creates the space for creative new solutions through the exchange of requirements and technological advances.

Working in the testbed

The activities of a testbed are just as broad as its objectives. Participants come from different areas: Chip manufacturers, network infrastructure providers, software developers and increasingly also manufacturers of industrial control systems, components and software are involved. Of central importance are joint meetings on site, the plugfests, which take place up to six times a year. It goes without saying that the participants are also in regular contact and continuous testing is also increasingly possible.

At the Plugfests, typically 3-day events, the participants of the testbed come together for practical work, exchange and discussion. Practical activities at a Plugfest typically include:

• Ad-hoc testing: In particular, very early prototype implementations of new functions are typically tested between two or more vendors without making special preparations or creating test plans for this. The aim is to provide feedback for product development as early as possible, sometimes also specifically for feedback into standardization.
• Testing with test tool manufacturers: The systematic way to check standard compatibility is by means of special test tools and test plans. The manufacturers of these tools are typically represented at plugfests and use the interaction to gain a better understanding of market requirements.
• Interoperability tests: Interoperability is typically tested by networking devices across manufacturers. The aim is to test as many different combinations as possible, but also to achieve the widest possible coverage of potential test cases. However, compared to testing with tools, these are often more difficult to stimulate and monitor.
• Proof-of-concept: In the early stages of standardization, proof-of-concepts can be very valuable for further development. Testbeds provide the ideal framework for cross-manufacturer approaches in particular.
• Joint demonstrators: Joint demonstrators are developed to validate and publicly present the ecosystem.

Topics in the testbed

The topics discussed in the testbed are as diverse as the TSN ecosystem and evolve over time. At the beginning of the testbed activities, the technical focus was on the basic standards of TSN: time synchronization using 802.1AS and time-based scheduling within the framework of 802.1Qbv. In addition, the focus was also on publicizing the technology and the first manufacturer-independent proof-of-concepts. With the achievement of a certain level of maturity and the increasing acceptance of TSN, the topics expanded increasingly. Configuration was added as a central topic - a complex and highly political issue! Furthermore, a number of TSN standards came to the fore, particularly with regard to frame preemption and redundancy. Across standards, there has been an increasing shift from a proof of concept of interoperability to a search for potential interoperability problems.

Figure 2: Overview of the work topics in the TSN testbed

© ISW

The increasing focus on the TSN ecosystem can also be seen in the growing importance of protocols and technologies below and especially above TSN. As TSN is particularly important for many participants for the use of OPC UA down to field level, the interaction between OPC UA - PubSub in particular - and TSN is also moving further into focus.

In line with the evolving topics, there is a continuous need for coordination and further development of the testing procedure. In particular, the increasing complexity, the maturity of the products and the growing number of participants require increasing systematization. This was also the focus of the involuntary forced break from on-site activities, which was attempted to be covered by means of virtual plugfests. As a result, the interoperability rack approach was further developed and optimized.

Interoperability in rack format

Carrying out interoperability tests systematically and efficiently is significantly more complex than classic tests for verifying standards. Several factors contribute to this: A large number of devices leads to almost any number of topologies, which can potentially affect the function. Performing targeted tests can be very complex and the coverage very limited, as it is often difficult to stimulate the devices to behave in a certain way. Similarly, monitoring the tests and performing a detailed analysis is a major challenge. On the other hand, testing needs to be usable with as little overhead as possible, work for devices with different characteristics, from the switch to the endpoint, and deliver reproducible results.

Figure 3: Schematic representation of the interoperability rack

© ISW

The development of a suitable approach has long been the focus of the testbed and the subject of ongoing discussions. At the center of the solution is the Interoperability Rack, which has been implemented and improved in many iterations. In the recent past, when it was not possible to hold real plugfests, the focus of many virtual meetings was on consolidating and optimizing the rack. At the real plugfest in fall 2021, the results were implemented on site and now form the basis for further activities.

The interoperability rack, which houses all the devices required for testing, can be divided into the following areas

• Basic infrastructure: The basis of the rack is a classic server rack which is supplemented by integration options for a wide range of devices, from top-hat rail-based industrial components to prototyping boards. The rack supplies the devices with power and connectivity, including VPN access.
• Core topology: In the core of the rack there is a fixed topology of switches and endpoints, which offers a variety of slots to which other devices can be connected.
• Devices-under-test (DUT): Devices that are to be tested are also part of the rack, but can also be connected externally if required, for example during a plugfest.
• Topology patch field: Both the available slots of the core topology and the ports of the DUTs are routed to a patch field. This allows different topologies to be plugged in very easily. Combinations of DUTs and core topology are referred to as scenarios.
• Test tools: The rack is equipped with a variety of test tools. These include devices from the relevant manufacturers as well as other systems such as a pulse-per-second monitor. Another tool developed is the so-called Interoperability Application, which is executed by most endpoints and is used for testing and monitoring.

Test plans exist for the efficient use of the rack, for example for specific functions such as time synchronization or events such as the failure of a link. For systematic interoperability testing, test plans are carried out in specific
scenarios and the results are analyzed. During a plugfest, as many test plans as possible are combined with permuting scenarios; between events, the rack runs in continuous operation, which allows for long-term tests. Participants can also perform remote firmware updates and test their function.

Configuration - a central challenge

The success of TSN depends crucially on usability, which in turn is directly linked to configuration. According to 802.1Qcc, there are three basic approaches to configuration - centralized, distributed and hybrid - with all variants based on other standards that are not yet fully available. The focus of the testbed is on the centralized approach.

Regardless of the approach, the configuration has several dimensions:

• What should be configured? It must be clarified which standards are to be used in which configuration for which use case.
• How should it be configured? It must be clarified how a configuration gets from the point of decision, for example a central network management (CNC), to the affected devices and how they deal with it, both temporally and logically.

While the answer to the first question requires fundamental decisions regarding the use of TSN and is therefore highly political, the second is much more technical. Accordingly, there is a lot of discussion about the first question and one or two experiments or even proof-of-concepts have been carried out. However, systematic testing focuses primarily on technical implementation.

For example, cross-manufacturer tests on the configuration of schedules on switches from different manufacturers have already been carried out in the testbed by a central CNC using NETCONF and YANG. Tests on communication between CUC and CNC have also been successful.

Over the course of the year, a significant expansion of activities is planned with regard to testing the configuration and the focus will be on many more YANG modules in particular. In addition to pure logic and data, aspects such as timing will also be increasingly considered. Another aspect to be mentioned is endpoint configuration, which is an important topic for 2022 thanks to major advances in standards.

Achieving more together

The author: Florian Frick is Group Leader 'Real-Time Communication' at ISW Stuttgart.

© University of Stuttgart

Even though a lot has already been achieved in the testbed in recent years, the TSN ecosystem still has a long way to go. This will hopefully be increasingly accompanied by face-to-face meetings again this year. In addition to many systematic interoperability tests, the work will focus in particular on configuration. The positive experiences of the participants show that the collaboration is worthwhile both for them and for the TSN ecosystem.