Docs/eBPF features
eBPF features
The Lighthouse agent can watch its host from the Linux kernel with eBPF: which services run there, what they connect to, why a process died and how requests behave, without changing your services or adding code.
Each feature is a switch in the Lighthouse's Settings, under eBPF features. Under every switch the agent on each node reports whether the feature runs, and why not when it does not.
What each feature shows
| Feature in Settings | What you get | Where you see it |
|---|---|---|
| Service discovery | The services listening on the host, suggested monitors for them, and who connects to whom | Lighthouse page (Services), Map |
| Incident context | OOM kills, crashes and failed connections around an incident | The incident page |
| Network quality | RTT, retransmits and connect time towards each dependency; DNS failures and latency per resolver | Map, Metrics |
| Per-process attribution | Top consumers of CPU, memory, file I/O, disk and network, including short-lived processes; disk latency | Metrics |
| Real traffic | Request rate, errors and latency per service and route | Metrics > Traffic, Map |
| Traces | Request traces from Real traffic, filtered and sampled on the host | Traces |
| Check traces | A trace of every check run: DNS, connect, TLS and time to first byte | The monitor page |
Real traffic and Traces need one more component on the host, OpenTelemetry eBPF Instrumentation (OBI): see Real traffic and Traces. Check traces come from the agent's own checks and need no eBPF.
Your plan decides which features you can switch on; one outside it shows Not in your plan. See pricing.
Requirements
eBPF features run on Linux only, and need all of:
- Kernel 5.8 or newer with BTF (the file
/sys/kernel/btf/vmlinuxexists), or a RHEL-family 4.18 kernel with the eBPF features backported. The agent probes the kernel instead of trusting its version number. - x86-64, x86 or arm64. Agents on 32-bit ARM and riscv64 run without eBPF; everything else works.
- The privileges: root with the BPF and PERFMON capabilities (or SYS_ADMIN on kernels before 5.8) and the host's process view. How to grant them depends on how the agent is installed, see below.
- Kernel lockdown not in confidentiality mode.
On a host that misses one of these, the features report why in Settings and the rest of the agent (checks, host metrics) keeps working.
Turn it on
First give the agent the privileges for your install type, then switch the features on in the Lighthouse's Settings.
Linux host
The systemd service that the install script writes already has the privileges. Switch the features on in Settings; the agent picks them up with its next heartbeat.
Docker
The default image runs as a non-root user without capabilities. Start the agent as root with the eBPF capabilities and the host's PID namespace:
docker run -d --name lighthouse --restart=unless-stopped \
--user 0 --pid=host --cap-drop ALL \
--cap-add BPF --cap-add PERFMON --cap-add SYS_PTRACE --cap-add DAC_READ_SEARCH \
--cap-add NET_ADMIN --cap-add SYS_RESOURCE \
-e LIGHTHOUSE_TOKEN=<your-token> \
-v lighthouse-data:/var/lib/lighthouse \
ghcr.io/statusharbor/lighthouse:latestOn kernels before 5.8 replace --cap-add BPF --cap-add PERFMON with
--cap-add SYS_ADMIN. Keep the lighthouse-data volume: it holds the
agent's identity across container upgrades.
Kubernetes
eBPF runs in the chart's per-node DaemonSet (chart 1.20 or newer), which the Console's install command turns on. Those node pods run as root with the eBPF capabilities and the host's PID namespace; the central pod stays unprivileged and never loads eBPF programs.
If your release was installed from an older chart, upgrade it with the new chart's defaults:
helm upgrade lighthouse oci://ghcr.io/statusharbor/charts/lighthouse \
-n status-harbor \
--reset-then-reuse-values \
--set daemonset.enabled=trueOn a cluster that enforces Pod Security Admission, allow privileged pods in the namespace:
kubectl label namespace status-harbor pod-security.kubernetes.io/enforce=privilegedGKE Autopilot and EKS Fargate do not allow these settings. To keep eBPF
off the nodes whatever Settings say, add
--set daemonset.ebpf.enabled=false.
Real traffic and Traces
Real traffic is measured by OpenTelemetry eBPF Instrumentation (OBI), which runs next to the agent. Traces come from the same OBI. Without OBI, Settings shows OBI binary or sidecar: no OBI configured under both. Set OBI up for your install type, then switch on Real traffic (and Traces) in Settings.
OBI never writes into your traffic: its context propagation is always off. Only the service, route template, method, status class and SQL operation type leave the host; headers, bodies, query strings, raw paths and SQL text never do.
Kubernetes
The chart runs OBI as a second container on each node. It is off by default; turn it on:
helm upgrade lighthouse oci://ghcr.io/statusharbor/charts/lighthouse \
-n status-harbor \
--reset-then-reuse-values \
--set daemonset.obi.enabled=trueEach node then runs an otel/ebpf-instrument container (requests 50m
CPU and 300 MiB memory, limit 768 MiB). The Lighthouse's Settings show
the exact command with your current version.
Linux host
-
Download the OBI release for your architecture (v1.0.0, the version the agent is tested with) from the OBI releases and put the binary at
/usr/local/bin/obi. -
Point the agent at it in
/etc/lighthouse/lighthouse.yaml:agent: obi: path: /usr/local/bin/obi(or set
LIGHTHOUSE_OBI_PATH=/usr/local/bin/obiin the service's environment). -
Restart the agent:
sudo systemctl restart lighthouse.
The agent starts OBI itself, restarts it if it exits and stops it with
the agent. OBI instruments services on all ports; to limit it, set
agent.obi.ports (for example "80,443,8000-8999").
Docker
Run OBI as a second container that shares the agent container's network, so the agent reads OBI on loopback, and tell the agent where OBI is:
docker run -d --name lighthouse-obi --restart=unless-stopped \
--network container:lighthouse --pid=host \
--user 0 --cap-drop ALL \
--cap-add BPF --cap-add PERFMON --cap-add SYS_PTRACE --cap-add DAC_READ_SEARCH \
--cap-add CHECKPOINT_RESTORE --cap-add NET_RAW \
-e OTEL_EBPF_PROMETHEUS_PORT=19400 \
-e OTEL_EBPF_OPEN_PORT=1-65535 \
-e OTEL_EBPF_BPF_CONTEXT_PROPAGATION=disabled \
-e OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:19401/v1/traces \
-e OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
otel/ebpf-instrument:v1.0.0Add -e LIGHTHOUSE_OBI_METRICS_URL=http://127.0.0.1:19400/metrics to the
agent's docker run (from Docker above) and recreate it.
Status in Settings
Under each feature, every node reports one of: running, degraded (running without some kernel features), disabled, failed or unsupported, with the reason:
| Reason | What to do |
|---|---|
| Kernel has no BTF type information | Use a kernel built with BTF (most distributions ship it from kernel 5.8 on). |
| Agent lacks CAP_BPF and CAP_PERFMON (or CAP_SYS_ADMIN) | Grant the privileges for your install type. |
| Kernel lockdown is in confidentiality mode | eBPF cannot run under this lockdown mode. |
| A required host feature is missing | Read the line under it. For no OBI configured, set up OBI. |
| Failed to load after several attempts | The agent stopped retrying; its log says why. |
| Turned off on the host (LIGHTHOUSE_EBPF=off) | Remove that setting on the host. |
| Not enabled for this Lighthouse | Switch the feature on in Settings. |
A node that has not reported for a while shows unknown with its last report.
Turn it off
Switch the features off in Settings: the agent unloads them with its
next heartbeat. To keep eBPF off on a host whatever Settings say, set
LIGHTHOUSE_EBPF=off (or agent.ebpf: off in lighthouse.yaml): the
local setting always wins and loads nothing.
What is never collected
Command-line arguments, environment variables, file contents and network payloads are never read. DNS monitoring reads only the 12-byte DNS header, never the names queried or the answers. The agent's eBPF programs are not pinned, so the kernel removes them when the agent exits. The full list of what each feature collects is in the agent's README.