Latency Analysis using Grafana and raintank-probe

Real time and historical latency analysis in a Telco environment is an important troubleshooting tool that can help the NOC and Ops teams to identity networking issues. Issues like fiber cuts, errors in backbone links, wrong configuration in routing protocols and link saturations can affect customers applications running over those links. Being able to proactively identify issues before customers applications are affected due to latency or packet loss is one of the most important activities of NOC teams.

Typical monitoring tools (Opensource based open Nagios or commercial ones like Solarwinds, Zabbix, etc) are centralized and usually only provide ICMP analysis from one or two geographic locations which is not enough when trying to identify latency between the different markets or POPs.

There are several commercial and open source solutions that allow you to deploy internal probes in such a way that you can send ICMP tests every 60 seconds and send alarms via email or web hooks to a chat application like slack or Hangouts chats. An interesting one is Worldping which is a cloud based solution that can be integrated as plugin with Grafana and have the following advantages:

  • Cloud based
  • You don’t have to deploy an internal VM or public cloud VM and install Linux and Grafana
  • Grafana upgrades
  • Hosted Grafana supports community plugins which means that can be used not only for Worldping
  • Additional to ICMP tests the probe supports DNS, HTTP and HTTPS
  • You can use external probes additional to the private ones

Steps to deploy a hosted Grafana with Worldping plugin:

  1. Create an account in grafana.net
  2. Install and enable Worldping plugin the the hosted Grafana
  3. Install raintank in the internal probes. You can use any Linux server, either a vm or baremetal server or use a Raspberry Pi with Raspbian. Make sure that Go is installed
  4. Create API Key
  5. Configure key in internal probes under probe.ini and start daemon
  6. Configure Internal probes in Worldping (Should appear automatically if the probe list is key was configured correctly)
  7. Configure Endpoints in Worldping
  8. Configure alarms
  9. Enjoy and share dashboards with your team

Grafana Dashbaord example

Other commercial and open source alternatives:

Thousand eyes

PRTG

Vaping (Interesting alternative to Smokeping based on Python)

Author: Manuel Marin Garcia

Network architecture, infrastructure and Open source enthusiast. I move packets. Opinions are my own

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s