Docs/eBPF features

eBPF features

The Lighthouse agent can watch its host from the Linux kernel with eBPF: which services run there, what they connect to, why a process died and how requests behave, without changing your services or adding code.

Each feature is a switch in the Lighthouse's Settings, under eBPF features. Under every switch the agent on each node reports whether the feature runs, and why not when it does not.

What each feature shows

Feature in SettingsWhat you getWhere you see it
Service discoveryThe services listening on the host, suggested monitors for them, and who connects to whomLighthouse page (Services), Map
Incident contextOOM kills, crashes and failed connections around an incidentThe incident page
Network qualityRTT, retransmits and connect time towards each dependency; DNS failures and latency per resolverMap, Metrics
Per-process attributionTop consumers of CPU, memory, file I/O, disk and network, including short-lived processes; disk latencyMetrics
Real trafficRequest rate, errors and latency per service and routeMetrics > Traffic, Map
TracesRequest traces from Real traffic, filtered and sampled on the hostTraces
Check tracesA trace of every check run: DNS, connect, TLS and time to first byteThe monitor page

Real traffic and Traces need one more component on the host, OpenTelemetry eBPF Instrumentation (OBI): see Real traffic and Traces. Check traces come from the agent's own checks and need no eBPF.

Your plan decides which features you can switch on; one outside it shows Not in your plan. See pricing.

Requirements

eBPF features run on Linux only, and need all of:

  • Kernel 5.8 or newer with BTF (the file /sys/kernel/btf/vmlinux exists), or a RHEL-family 4.18 kernel with the eBPF features backported. The agent probes the kernel instead of trusting its version number.
  • x86-64, x86 or arm64. Agents on 32-bit ARM and riscv64 run without eBPF; everything else works.
  • The privileges: root with the BPF and PERFMON capabilities (or SYS_ADMIN on kernels before 5.8) and the host's process view. How to grant them depends on how the agent is installed, see below.
  • Kernel lockdown not in confidentiality mode.

On a host that misses one of these, the features report why in Settings and the rest of the agent (checks, host metrics) keeps working.

Turn it on

First give the agent the privileges for your install type, then switch the features on in the Lighthouse's Settings.

Linux host

The systemd service that the install script writes already has the privileges. Switch the features on in Settings; the agent picks them up with its next heartbeat.

Docker

The default image runs as a non-root user without capabilities. Start the agent as root with the eBPF capabilities and the host's PID namespace:

docker run -d --name lighthouse --restart=unless-stopped \
  --user 0 --pid=host --cap-drop ALL \
  --cap-add BPF --cap-add PERFMON --cap-add SYS_PTRACE --cap-add DAC_READ_SEARCH \
  --cap-add NET_ADMIN --cap-add SYS_RESOURCE \
  -e LIGHTHOUSE_TOKEN=<your-token> \
  -v lighthouse-data:/var/lib/lighthouse \
  ghcr.io/statusharbor/lighthouse:latest

On kernels before 5.8 replace --cap-add BPF --cap-add PERFMON with --cap-add SYS_ADMIN. Keep the lighthouse-data volume: it holds the agent's identity across container upgrades.

Kubernetes

eBPF runs in the chart's per-node DaemonSet (chart 1.20 or newer), which the Console's install command turns on. Those node pods run as root with the eBPF capabilities and the host's PID namespace; the central pod stays unprivileged and never loads eBPF programs.

If your release was installed from an older chart, upgrade it with the new chart's defaults:

helm upgrade lighthouse oci://ghcr.io/statusharbor/charts/lighthouse \
  -n status-harbor \
  --reset-then-reuse-values \
  --set daemonset.enabled=true

On a cluster that enforces Pod Security Admission, allow privileged pods in the namespace:

kubectl label namespace status-harbor pod-security.kubernetes.io/enforce=privileged

GKE Autopilot and EKS Fargate do not allow these settings. To keep eBPF off the nodes whatever Settings say, add --set daemonset.ebpf.enabled=false.

Real traffic and Traces

Real traffic is measured by OpenTelemetry eBPF Instrumentation (OBI), which runs next to the agent. Traces come from the same OBI. Without OBI, Settings shows OBI binary or sidecar: no OBI configured under both. Set OBI up for your install type, then switch on Real traffic (and Traces) in Settings.

OBI never writes into your traffic: its context propagation is always off. Only the service, route template, method, status class and SQL operation type leave the host; headers, bodies, query strings, raw paths and SQL text never do.

Kubernetes

The chart runs OBI as a second container on each node. It is off by default; turn it on:

helm upgrade lighthouse oci://ghcr.io/statusharbor/charts/lighthouse \
  -n status-harbor \
  --reset-then-reuse-values \
  --set daemonset.obi.enabled=true

Each node then runs an otel/ebpf-instrument container (requests 50m CPU and 300 MiB memory, limit 768 MiB). The Lighthouse's Settings show the exact command with your current version.

Linux host

  1. Download the OBI release for your architecture (v1.0.0, the version the agent is tested with) from the OBI releases and put the binary at /usr/local/bin/obi.

  2. Point the agent at it in /etc/lighthouse/lighthouse.yaml:

    agent:
      obi:
        path: /usr/local/bin/obi

    (or set LIGHTHOUSE_OBI_PATH=/usr/local/bin/obi in the service's environment).

  3. Restart the agent: sudo systemctl restart lighthouse.

The agent starts OBI itself, restarts it if it exits and stops it with the agent. OBI instruments services on all ports; to limit it, set agent.obi.ports (for example "80,443,8000-8999").

Docker

Run OBI as a second container that shares the agent container's network, so the agent reads OBI on loopback, and tell the agent where OBI is:

docker run -d --name lighthouse-obi --restart=unless-stopped \
  --network container:lighthouse --pid=host \
  --user 0 --cap-drop ALL \
  --cap-add BPF --cap-add PERFMON --cap-add SYS_PTRACE --cap-add DAC_READ_SEARCH \
  --cap-add CHECKPOINT_RESTORE --cap-add NET_RAW \
  -e OTEL_EBPF_PROMETHEUS_PORT=19400 \
  -e OTEL_EBPF_OPEN_PORT=1-65535 \
  -e OTEL_EBPF_BPF_CONTEXT_PROPAGATION=disabled \
  -e OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:19401/v1/traces \
  -e OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
  otel/ebpf-instrument:v1.0.0

Add -e LIGHTHOUSE_OBI_METRICS_URL=http://127.0.0.1:19400/metrics to the agent's docker run (from Docker above) and recreate it.

Status in Settings

Under each feature, every node reports one of: running, degraded (running without some kernel features), disabled, failed or unsupported, with the reason:

ReasonWhat to do
Kernel has no BTF type informationUse a kernel built with BTF (most distributions ship it from kernel 5.8 on).
Agent lacks CAP_BPF and CAP_PERFMON (or CAP_SYS_ADMIN)Grant the privileges for your install type.
Kernel lockdown is in confidentiality modeeBPF cannot run under this lockdown mode.
A required host feature is missingRead the line under it. For no OBI configured, set up OBI.
Failed to load after several attemptsThe agent stopped retrying; its log says why.
Turned off on the host (LIGHTHOUSE_EBPF=off)Remove that setting on the host.
Not enabled for this LighthouseSwitch the feature on in Settings.

A node that has not reported for a while shows unknown with its last report.

Turn it off

Switch the features off in Settings: the agent unloads them with its next heartbeat. To keep eBPF off on a host whatever Settings say, set LIGHTHOUSE_EBPF=off (or agent.ebpf: off in lighthouse.yaml): the local setting always wins and loads nothing.

What is never collected

Command-line arguments, environment variables, file contents and network payloads are never read. DNS monitoring reads only the 12-byte DNS header, never the names queried or the answers. The agent's eBPF programs are not pinned, so the kernel removes them when the agent exits. The full list of what each feature collects is in the agent's README.