Observability

This folder provides a docker compose file to set up an observability stack based on a set of Grafana tools (Grafana, Tempo, Loki and Mimir) and Prometheus.

The information is expected to conform to OpenTelemetry specifications and will come through an instance of the Opentelemetry collector.

This will be listening at localhost:4317.

Goal and Limitations

The resources provided in this folder are intended only to explore and understand what is happening in your application during development, what are the options to make the application observable, and to monitor it locally.

• It is intended to be run in a local workstation.
• None of the configuration seen in this folder is intended for production use.

Please check the notes section.

Deployment

Create the needed local directories for the volumes by running:

$ mkdir -p tempo/data loki/data mimir/data

Then, select the Docker image tags appropriate for you by creating an .env file based on .env.example.

Note that all the projects that are part of this solution are changing at a very fast pace, so if you diverge too much from the sample tags in .env.example you may have to fix the configuration of the different components. Refer to the official documentation in each case.

Finally, create the stack by executing docker compose up -d. This will download the appropriate images, create the containers and expose 2 endpoints on your local host machine:

• A Grafana for visualisation on localhost:3000
• An OpenTelemetry collector listening at localhost:4317.

The OpenTelemtry collector is listening to information in OpenTelemetry structured format, and also transported through GRPC. Example ptions for the Java Agent that work are:

-Dotel.service.name=YourAppNameGoesHere
-Dotel.javaagent.debug=true
-Dotel.metrics.exporter=otlp
-Dotel.logs.exporter=otlp
-Dotel.traces.exporter=otlp
-Dotel.exporter.otlp.protocol=grpc
-Dotel.exporter.otlp.endpoint=http://localhost:4317/
-Dotel.propagators=tracecontext,baggage,b3,b3multi

All the applications' components should send their telemetry to the OpenTelemetry collector. For that, they need to be run with the OpenTelemetry agent properly configured.

All the images are pulled from official sources. Check the docker-compose.yaml file.

Notes

• Neither authentication, nor authorisation or transport security or encryption at rest have been applied. This is consistent with the discussion in Goal and Limitations.
• This set up uses gRPC where possible. Some documentation for using either HTTP (in the case of Tempo) or gRPC (in the cases of Mimir and Loki) have not been found.
• Prometheus and Mimir won't work at all at the moment of this writing if we try to run the containers with a custom UID and GID.
  • Amongst other things, this means that the content under mimir/data/ may belong to root:root depending on your docker installation.