End-to-End Visibility – What Is It Really?

End-to-end visibility for many is the holy grail of observability. That said, AI is probably the new holy grail – as it is for all domains today – and we’ll dive into that topic down the track. When presented with the concept of end-to-end, we’re conditioned to think about distributed tracing because that gave us the end-to-end view of a transaction. We conjure up views of elegant looking distributed traces with spans representing the different steps of a transaction’s journey – hopefully with no gaps! This is classic distributed tracing, popularised by the open-source community and commercial vendors alike. It’s a mature concept now, with head and tail sampling and more all in consideration to provide a very useful asset.

This article is not about distributed tracing, though it is part of any robust observability solution. Rather, it's about challenging what we mean by "end-to-end" and considering whether the boundaries of observability should extend to every instrumented device and system that keeps a business running.

Traditional observability looks at servers, applications, perform synthetic checks, browser interfaces, mobile applications and the interactions across these, providing insights and monitoring based on this data. The observability market has many vendors who have been around for over a decade that have done a good job of delivering solutions in this area, so we are spoilt for choice in tools. It’s a mature market that has solved many problems and continues to innovate, leading with IA. Discipline in execution is still required though to arrive at successful solution – another story for another time.

Switching gears now and getting to crux of this article. Have you ever gone into a meeting room to see that the Teams device is offline? You want me to rock up to the office – well the devices had better be working! Going to the roads licencing facility during lunch? You’re on borrowed time and the screen showing your number in the queue is out? Such things are important for customer and staff satisfaction.

While you should strive for end-to-end visibility, you should also redefine end-to-end as your business from one corner to another. It’s much more than those traditional items I’ve mentioned above. Banks and government agencies have kiosks and monitors in public facilities. All sorts of businesses run security platforms like Sentinel, Palo Alto and others. There is networking equipment, cameras, meeting room devices, air-conditioning units, UPS devices and good old printers. The internet of things (IOT) arrived a long time ago and resulted in effectively all of this business-critical infrastructure today being instrumented and addressable. It means we have access to signals (eg. SNMP), logs and APIs to round out the view of our business.

Businesses will have varying combinations of these devices, and if you want that best view of your operating environment, then frankly you need to see it all or as much as possible. Thankfully, the observability vendors offer some coverage with pre-built integrations to provide support for the more common devices, acquiring the telemetry for you. Unlocking the other devices may need some creativity and code and what self-respecting developer doesn’t want that challenge?

A data sink is where data goes and is a core competency of any good observability vendor. These platforms are always open for business when it comes to data, so use their logging and metrics APIs to get this data to their platform. Once it’s there, you use it as you would use any other observability data. It’s more data and yes, observability has a reputation for being a big-ticket item, but for the most part the data is simple and not voluminous. Its heartbeats, temperatures and the like, and the vendors are pretty good at filtering data and only keeping what’s necessary or keeping it for a limited period.

Another consideration is ensuring any future devices you purchase are IOT enabled. Know that you can scrape useful information before you make these purchases. The device vendors may offer remote monitoring as part of their package, and it may be comprehensive and valuable. Some will let you see the information, but in the end its data in many places. Rather than decommissioning the vendors solutions, compliment them and give yourself peace of mind because you now have a consolidated view of devices that you didn’t have before.

Observability is understanding a system’s behaviour by observing its outputs. Getting outputs from your traditional sources and these less conventional ones and understanding them is true end-to-end visibility across your business, so take a walk around your floors and facilities and realise the value of that data hiding in plain sight. If you'd like to explore how to bring these overlooked signals into a unified observability strategy, the AC3 team would be happy to help you uncover and operationalise the telemetry already available across your environment. Get in touch with AC3 today.