From raw data to usable information: visibility and observability

(To Marco Rottigni)
12/07/21

Once upon a time there was traditional IT ... In a well-defined and limited company perimeter, servers, routers, switches, workstations, laptops, firewalls, and various other digital entities coexisted.

This ecosystem had very specific, guarded and defended connections with the outside world. The work of the security team and its captain (the figure of Chief Information Security Officer - CISO) was to define, implement and monitor guidelines and behavioral policies to avoid accidents, infections and theft of sensitive data; at the same time guaranteeing the operational continuity and efficiency of the systems supporting the business.

In recent years, two planetary macro-phenomena have had an impact on this digital biodiversity:

I'm talking about digital transformation and sanitary emergency, an explosive combination that has advanced this ecosystem in an extremely short and often explosive way, transforming a quiet provincial agricultural town into the heterogeneous and species-rich jungle of central Madagascar. Generating - as we often hear at events and conferences - a need for digital visibility; because you cannot manage, control, monitor, secure and protect what you don't know you have or don't see.

Absolutely pertinent, true, current theme… but incomplete in its articulation.

If we list the various environments that characterize an organization of any sector and size today, we find tangible proof of this explosion of species that I mentioned earlier: traditional IT is certainly still part of it, flanked by mobile entities such as smartphones and tablets; containerized environments, that is, built on agile platforms in order to fragment the functionalities necessary for them to become usable on-demand with a self-adjusting scale based on the number of users; cloud projects, in turn divided into infrastructure as a service (IaaS), platforms as a service (PaaS) e software as a service (SaaS); traditional applications, re-engineered to become usable via the web; ultra-specialized tools connected via the Internet, which characterize the Internet of Things , Industrial Internet of Things.

To silence the need to interface with operating systems (eg. Windows XP) and interfaces (eg. RS232 serial, Centronics parallel ...) which, although obsolete, represent the need in some industrial environments, bound to this obsolescence by unsustainable upgrade costs or by non-existent updates.

This partial list provides a good approximation of how complex the challenge of digital visibility for a modern company.

The solution that is often thought of is the integration of two systems: a digital inventory that is kept up to date automatically as much as possible - synchronized with a tracking and change management system, also known as Change Management DB or CMDB.

The goal of these platforms is to make the entire digital landscape visible, perhaps allowing a categorization of species - automated or not based on grouping rules; the two use cases that are normally satisfied are the distribution of sensors for collecting all the metadata from the resources and the categorization in logical perimeters carried out centrally.

Sometimes raising the degree of specialization of the sensor itself. It is not possible to use the same techniques for very different environments, such as laptops on the move, application containers and SaaS environments.

Building this visibility, it remains of fundamental importance the attention to how the collected data - however crude and insignificant if taken in an atomic way - combined can become sensitive and indicative of an exposed attack surface; this attention must also be paid to the verification of the security and confidentiality model with which these data are processed in the three moments of their existence: transit, rest, processing.

On the side of visibility, however, the concept of observability.

Observability starts where visibility ends, understood as the availability of the telemetry collected by the sensors, to increase its value.

Usually it requires that the raw data collected are indexed and normalized, in order to make them processable by a visualization engine, then we proceed to query and organize the results in the dashboard.

Observability therefore allows you to compose the raw data into views, to form the information you are interested in tracing - in the format in which you want to view it.

In the choice of technological platforms to support the processes, it becomes important to take care of this characteristic and above all to evaluate its flexibility: aggregating normalized metadata on the digital inventory makes it possible to support countless business processes: software and hardware distribution verification, forensic investigations, evaluation of the safety posture, and so on.

A final feature characterizing a good observability is the enrichment of the context of a metadata: starting from the metadata collected by the sensor, a catalog of information not detectable by the sensor, but curated by research, is combined.

For example, it is possible to combine the list of identified software with information on the end-of-life or end-of-support - thus supporting purchasing or modernization processes, as well as descendant investment or modernization programs ...

Or combine information about how vulnerable it is, or even when it has been attacked in recent months, with a detected network service - supporting remediation or risk mitigation prioritization processes.

Those involved in digital inventory are increasingly confronted with an increasing degree of complexity of environments and variety of digital species.

The combination of visibility and observability generates a holistic value in the effectiveness of the query of the monitored population, increasing the speed and agility in tracking and monitoring aggregate information of interest for the processes to be supported.

So that all this does not remain a technological exercise in style, these two skills must be related to the cultural growth of the company - translated into awareness of the human factor of being part of an information security chain: what in the Anglo-Saxon world is known as name of "knowledge management".

In fact, personnel and corporate culture must grow together with technological development, understanding the possible risks and potentials; each carrying out their role with the awareness of working in an increasingly agile, complex and interferential environment between the various systems.

Photo: author / web