In today's networked factory, sensors, devices and machines generate massive amounts of data in a wide variety of formats. Depending on the type of production and the end product, they come from the warehouses for raw materials and intermediate products, are supplied by machines and automated production systems or from sensors that measure weights, dimensions, speeds or other physical variables. Conveyor belts continuously report their running speed, quality assurance systems detect problems or image systems store photos. This information is highly heterogeneous in terms of quantity, format, protocol or meaning.

But that's not all. The production of goods has changed fundamentally in recent years. The local production of complex products is increasingly being replaced by globally distributed manufacturing processes because criteria such as personnel costs, proximity to the customer or the availability of expertise have to be taken into account. Even medium-sized companies often operate ten or more factories with several hundred machines and a high volume of time series data.
However, these primarily business and organizational decisions must then also be mapped by IT, and this merging of operational data (often SQL data from the ERP) with timeseries data (often sensor or measurement data) is an essential key.

Considerable potential for process optimization

This constellation offers enormous potential for optimizing processes. However, relevant market observers such as McKinsey and Forbes assume that more than two thirds of projects in the field of IoT or digital transformation do not lead to the desired (technical and/or economic) success. This raises the question of the reasons.

Unsurprisingly, the availability of the necessary skills and expertise is the key factor. As is to be expected with new technologies, this tends to be limited or difficult to obtain on the market. Secondly, successful implementation generally requires a change in corporate culture: change management and a more agile approach must replace the traditional silo mentality, and that is not easy. The third and very important reason, however, is the inadequacy of the existing IT and data infrastructure and data management.

Overcoming conventional restrictions

The essential backbone of such complex systems is the approach of creating a central, freely scalable IoT data pool for IoT data, which is capable of collecting massive amounts of data from any number of data sources, ensuring intelligent and fast use of this data via a standard SQL interface and allowing unlimited scaling of (raw) data in a cost-efficient manner. Because storage alone is not enough.

The packaging manufacturer Alpla uses an industry-compatible time series management system from Crate.

© Crate.io

What is needed is a system that analyzes the information in real time and in a stream, but can also combine the historical data volumes at the same time and provide them in a suitable form.
This leads to new requirements for the architecture of the underlying databases. In fact, conventional databases and infrastructure technologies were not developed for the world of machine data on an IoT scale. Historically, the respective application generation determined the necessary IT performance.
In the days of locally operating ERP or MES systems, the IT environment was defined by mainframes, client/server technologies and relational SQL databases. The volume of data was rather limited, as was the variety of data and the required speed. The primary concern was to ensure data consistency, reliability and standardization.

With the advent of the internet and the cloud, including other requirements such as mobile, open source and big data, the performance weaknesses of this approach quickly became apparent. Consequently, distributed and document-oriented NoSQL databases were created that were also able to meet the demands of web-scale architectures. The volume of data increased continuously, as did the processing speed. The variety of data also increased. Systems were required that guaranteed high availability, enabled the fast and flexible development of applications and could support a massive number of users.

However, the industrial IoT with its machine-driven data volumes heralded a new level of digital transformation. Today, new solutions not only have to perform in terms of data volumes, but also in terms of combining the traditional data worlds (keywords: SQL, ERP) with billions of data records from sensors and other IoT devices, all linked from factories that may be distributed worldwide as data sources, and with enormous scaling requirements.
This necessary flexibility combined with cost efficiency makes it possible to support holistic management concepts. And this against the backdrop of extreme data volumes (from terabytes to petabytes) and the highest demands (response in milliseconds) on speed with a high variety of data formats to be managed. Solutions are therefore required that enable fast data acquisition and real-time analysis, can be easily scaled to the growing number of locations and machines to be controlled and have the necessary flexibility to work both in the cloud and locally.

The best of two worlds

Relational databases such as SQL are basically collections of two-dimensional tables with a fixed structure. Each row represents a data record, which in turn can be uniquely identified by a fixed index. The number of columns and the file type must be defined before the data is fed in. These strict regulations push this type of database to its limits, especially with large amounts of data and high parallel loads. On the other hand, SQL-capable databases certainly have advantages. These include the wide availability of programming know-how and flexible integration into existing structures.

The Notes app and real-time reports can be used to optimize the production process on site.

© Crate.io

So what could be more obvious than a solution that combines the best of both worlds?
Innovative concepts of distributed databases such as CrateDB overcome the restrictions by combining the convenience of an SQL database with the flexibility and scalability of NoSQL.

The advantages are impressive: you can process any structured or unstructured data type, store images and geospatial data, scale without limits and still process complex SQL queries in real time. This makes sense from both a functional and operational perspective: they scale horizontally on clusters of inexpensive servers to process even millions of records per second, and distributed processing, data partitioning on disk and columnar indexes with in-memory performance enable time-series queries in milliseconds, even when hundreds of thousands of clients are querying and inserting data simultaneously.

Availability, scaling and reliability are guaranteed at all times thanks to a shared-nothing architecture and a cluster can be scaled up or down as required. On this architectural basis, IoT projects can now be implemented that are many times faster and more economical than conventional databases. However, simply recording, analyzing and storing the data is not enough. The database is ultimately only a means to an end, albeit a crucial one.

High-performance data ingestion and storage are a basic requirement, but they need interfaces to other applications, firstly to enrich the recorded data with other information and secondly to make it available in a suitable form. This requires the appropriate interfaces and tools, for example for visualizing the information, correlating it or generating notifications for appropriately defined analysis results.

Economy and functionality

Traditional SQL database approaches (including newer in-memory systems) for storing large amounts of data with fast queries were often priced at a level that went beyond the economic scope of any IoT use case. This is just as often the case with NoSQL systems, also because special expertise is required in addition to the cloud requirements. The combination of these worlds, based on an extremely optimized architecture that has been specially developed for IoT use cases, creates the solution here. It often happens that the use of an optimized database results in 20 to 30 times faster queries and at the same time the resulting cloud costs are 50 to 75 % lower.

The industrial IoT with its machine-driven data volumes is ushering in a new level of digital transformation.

© Crate.io

Provision via the cloud and 24/7 support are of paramount importance for the cost-effective use and smooth operation of an appropriately configured data management solution. Most companies can hardly afford to operate their own data centers, including massive investments in the infrastructure and the maintenance of a business-critical application by their own employees with the necessary skills. Accordingly, solutions that meet the requirements of the IoT must be available as a managed cloud solution, but still offer the option of edge deployment wherever the circumstances do not permit cloud operation.

The use of a cloud-based and managed data platform that is developed and optimized to cope with the complex requirements of a machine-driven environment can contribute in many ways to bringing innovative IoT projects to a positive conclusion. It ensures comprehensive data handling, including the ability to grow easily in line with requirements. And it alleviates the know-how problem by focusing on using IoT and machine data intelligently instead of managing the database. With the demonstrable benefits in terms of efficiency, cost-effectiveness and process improvement, it provides the necessary arguments for acceptance within the company.