Docs/Traces
Traces
Traces show where a request spent its time: which service handled it, which calls it made and which one failed or was slow. Status Harbor collects them three ways and shows them in one waterfall:
- Check traces - every Lighthouse check run becomes a trace.
- Service traces - the Lighthouse agent records your services' requests with eBPF, without code changes.
- SDK traces - your own OpenTelemetry SDKs send spans with an API token.
Find them on a monitor's page (Check traces), in an incident's details, in the service map's panels and on the Traces page in the console (console.statusharbor.io/traces).
Reading a trace
On the Traces page, pick a service and narrow the list by operation, duration ("Slower than"), status or time range. New spans can take up to a minute to appear in this list, so it checks again every 15 seconds for two minutes after you open it or change a filter. A trace opened from a monitor, an incident or a link loads right away.
A trace opens as a waterfall: every span is a bar placed on the trace's timeline and indented under its parent, coloured by service and red when it failed. When a trace failed, a banner names the span where the error starts. Small ticks on a bar mark the span's events. The expand button beside close opens the trace full screen, with the span details beside the tree.
Select a span to see:
- The error message, in red at the top, when the span failed with one.
- Facts: duration, start within the trace, status, kind, service, host and the instrumentation scope (library name and version).
- Attributes of the span.
- Events, in time order. An exception shows its type and message; open Stack trace to read where it was thrown.
- Links to other spans, such as a message consumer's link to the producer that sent it. Open linked trace loads the producer's trace with that span selected; Back to the previous trace returns to the span you came from. A link inside the same trace offers Go to span.
- Resource: the attributes of the service that sent the span (version, environment, instance and SDK). It is folded until you open it, like the scope's own attributes.
Check traces
Each run of a Lighthouse check is a trace with one child per phase:
| Phase | Meaning |
|---|---|
dns | Resolving the target's name |
connect | Opening the TCP connection |
tls | The TLS handshake |
request | From sending the request to the first response byte |
response | Reading the body |
A failed run is marked as an error and listed first. Tick Send trace
header on an HTTP check (it is off by default) and the check adds a
W3C traceparent header to its request. When the target continues it,
the target's spans appear inside the check's trace: you see the check,
your service and its database call in one waterfall. Targets that
continue it:
- Go services traced by service traces.
- Services instrumented with an OpenTelemetry SDK, in any language.
Python, Node.js and Java services traced only by service traces start a trace of their own instead: Lighthouse never writes into your traffic, and continuing the header there would need it to. Their requests are still traced, with each request's outgoing calls as its children.
Without the header the check still gets its trace, with every phase; only the target's own spans are missing. Some strict APIs reject unknown headers, which is why it is opt-in per check.
Turn check traces on or off per Lighthouse under Lighthouses → your Lighthouse → Settings → Check traces. They are on by default where your plan includes them.
Service traces
With Real traffic on (OpenTelemetry eBPF Instrumentation next to the agent, see Real traffic and Traces), switch on Traces in the Lighthouse's Settings. The agent then receives your services' request spans on the host - HTTP, gRPC, SQL, Redis and Kafka
- and each request's outgoing calls from the same process become its children.
Before anything leaves the host the agent:
- Filters every span to an allow-list: templated span names and
routes (
/orders/{id}), method, status code, duration, the service, a client call's peer address and port, database system, operation and table, exception type. - Drops query strings, full URLs, SQL text, header values, bodies, exception messages and the addresses of callers to your services.
- Samples per trace: every failed trace is kept; within a budget of 5 spans a second per host, the slowest requests go first and a sample of the rest fills what is left.
Lighthouse never writes into your traffic: the instrumentation runs with trace context propagation off. Calls between services on one host join one trace; across hosts, a trace continues only where your own code or a check's header carries the context.
Send traces from your OpenTelemetry SDKs
Your own services can send spans with any OpenTelemetry SDK over
OTLP/HTTP, with or without a Lighthouse. Create a traces:write token
and point the SDK's exporter at https://otlp.statusharbor.io/v1/traces:
OpenTelemetry SDKs has the settings, examples for
Node.js, Python, Go, Java and Kubernetes and a troubleshooting table.
Plans and retention
| Check traces | Service and SDK traces | |
|---|---|---|
| Included from | Starter | Pro |
| Search back | Your plan's lookback | Your plan's lookback |
Spans are kept for 30 days and are not backed up. Your plan sets how far back you can search and how many spans per second your team can send; see Plans & limits.
Trace storage
Your plan also sets how much storage your team's traces may use, in GB: the compressed size of your services' spans. Check traces do not count; the spans of your own services inside a check trace (a target that continued the check's trace header) do. The Traces page shows the usage under the title once your plan has a size, and you get an alert at 80% and when it is full.
When it is full, Status Harbor keeps accepting new spans and deletes your oldest days first, until your traces take less than 90% of the size. Today is never deleted, and check traces stay for the full 30 days.
When your plan's size goes down (a downgrade, the end of a trial or a change to the plan), nothing is deleted for 7 days: the Traces page and an alert show the date. Moving to a larger plan applies at once.
Turning it off
Switch Traces or Check traces off in the Lighthouse's Settings; the agent stops at its next heartbeat (within a minute). Deleting a Lighthouse deletes its spans; deleting a team deletes all of the team's spans.