#web #database
Observability
→ Grafana
Grafana provides:
- exploring (online queries)
- alerting (as a replacement of alertmanager)
- dashboarding
It integrates smoothly with loki (logging) and prometheus (metrics).
→ Loki
Loki stores logs either on local or object storage.
It handles various log sources.
I found the most proficient way is to use journal (systemd logs) in json format. It is possible to redirect lot of tools into systemd. For eg:
- docker
- postgres
- nginx
→ Prometheus
Prometheus can get metrics from system by pulling informations or by making them push informations to a gateway. The grafana interface can be plugged on the prometheus interface.
Overall config:
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
- job_name: 'matrix'
metrics_path: "/_synapse/metrics"
static_configs:
- targets: ['localhost:9101']
→ Installation
On archlinux, install prometheus and grafana packages. Both have a systemd service, on ports 9090 and 3000. The config files are /etc/prometheus and /etc/grafana.ini.
→ Use timescaledb
Timescaledb can replace the default database backend. The promscale extension improves the performances.
→ Linux Metrics
On archlinux, install prometheus-node-exporter package and activate the service. It will provide a metric http endpoint on port 9100. Then configure prometheus to fetch the metrics. You can import the grafana dashboard.
→ Matrix
The grafana configuration json can be found here. Configure synapse homeserver.yaml:
listeners:
- port: 9101
type: metrics
bind_addresses: ['0.0.0.0']
enable_metrics: true
