Hub observability
Hub exports traces and logs with one Node.js OpenTelemetry SDK (NodeSDK, which owns the tracer and logger providers). Next.js already emits route spans (instrumentation, OpenTelemetry). App logs go through TelemetryLogger.
This page is Hub only. API metrics, file logging, and Endatix:Telemetry live on API observability. Hub does not export metrics.
Hub does not use @vercel/otel. That package is traces-first and only registers a LoggerProvider if you pass logRecordProcessors. Hub needs Azure Monitor exporters, undici instrumentation (so traceparent reaches the API), and a global LoggerProvider so TelemetryLogger records reach an exporter.
What to set
Telemetry is off until at least one exporter is set. Both may be set; then traces and logs fan out to both. If one exporter cannot be created (for example a malformed connection string), it is logged and skipped and the others still start.
Stdout (process log stream — SWA / App Service Log stream, kubectl logs): one line at boot, Telemetry SDK started in Azure AppInsights mode (or OTel / Azure AppInsights + OTel). That line is console.log, not an App Insights row.
App Insights traces: Next.js HTTP spans land in requests / dependencies even when no app log has been emitted yet. After boot Hub writes one TelemetryLogger info record (logger = instrumentation) so traces is not empty if the logs pipeline works. Restart the instance if you are looking at an old process.
Azure Application Insights. Runtime app setting only — not a Static Web Apps build variable. HTTP spans land in requests / dependencies. TelemetryLogger records land in traces, except error / critical calls that pass an Error: those carry exception.type and exception.stacktrace and land in exceptions.
OTLP endpoint for traces and logs. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_ENDPOINT override it per signal, and either one on its own also turns on export for that signal only.
grpc (usually port 4317), http/protobuf or http/json (usually 4318). OTEL_EXPORTER_OTLP_TRACES_PROTOCOL / OTEL_EXPORTER_OTLP_LOGS_PROTOCOL override it per signal. Hub defaults to grpc, like the API, rather than the spec's http/protobuf. An unknown value logs a warning and uses grpc.
Auth for a vendor endpoint, e.g. api-key=… or Authorization=Bearer …. Sent as gRPC metadata or HTTP headers.
TLS is read from the standard variables: OTEL_EXPORTER_OTLP_CERTIFICATE for a private CA and OTEL_EXPORTER_OTLP_CLIENT_KEY / OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE for mTLS.
gRPC plaintext. true / false decides for any non-https:// endpoint; OTEL_EXPORTER_OTLP_TRACES_INSECURE / OTEL_EXPORTER_OTLP_LOGS_INSECURE override per signal. Unset, http:// and a scheme-less host:port (e.g. otel-collector:4317) are plaintext — Hub's long-standing default, where the OTel SDK alone would try TLS. https:// is always TLS.
Rate-limits sampled traces per second (0 = no limit), using Azure Monitor's RateLimitedSampler, which records the sample rate so App Insights still extrapolates request and dependency counts. Unset: a standard OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG applies if set; otherwise 5 traces/s when Azure is configured (the Azure Monitor distro default) and every trace for OTLP only.
service.name / App Insights cloud_RoleName.
Extra resource attributes on both spans and logs, e.g. deployment.environment.name=staging,service.namespace=endatix. Hub also adds host and process attributes. It does not set deployment.environment.name itself.
true (any case) skips SDK start, as the OTel env spec defines. Other values, including 1, do not disable. Stdout: OpenTelemetry SDK disabled (OTEL_SDK_DISABLED). TelemetryLogger then behaves as if no exporter were set.
OpenTelemetry SDK diagnostics to stdout (ALL, VERBOSE, DEBUG, INFO, WARN, ERROR, NONE) — not App Insights traces, and not TelemetryLogger severity. Leave unset in production (SDK default). Set WARN or ERROR only while chasing export failures; DEBUG / VERBOSE / ALL on a short-lived test slot. INFO is noisy in prod (batch export chatter).
Print TelemetryLogger records to stdout in production. With no running log exporter, it forces the plain console fallback (development already has it). With a running log exporter, it adds one JSON object per line to stdout alongside the export — see Stdout.
Hub sets 1 at start unless you set it, which turns off Next.js's own fetch span. Hub → API calls are traced by undici instead, which also carries traceparent; with both, every call would be two dependencies rows. Set 0 to keep Next's span as well.
Next.js built-in OTel verbosity. Compose and Helm set 0 when OTLP is on. Not a Hub exporter switch.
Master list: hub/.env.example. Catalog of every Hub key: Hub environment variables.
Choosing a setup by host
| Host | Set | Notes |
|---|---|---|
| Azure Static Web Apps, App Service, Container Apps | APPLICATIONINSIGHTS_CONNECTION_STRING | Exporters only, not the Azure Monitor distro: no Live Metrics or performance counters. |
| AWS (ECS, EKS, EC2) | OTEL_EXPORTER_OTLP_ENDPOINT → ADOT Collector | The collector forwards to X-Ray and CloudWatch. Add cloud attributes with its resourcedetection processor. |
| Google Cloud (Cloud Run, GKE) | OTEL_EXPORTER_OTLP_ENDPOINT → an OpenTelemetry Collector (sidecar or agent) | Google's OTLP endpoint needs Google credentials, which Hub does not attach. Let the collector authenticate. |
| OTLP vendor (Grafana Cloud, Honeycomb, …) | endpoint, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_HEADERS | Use http/protobuf for endpoints that accept only OTLP/HTTP. |
| Self-hosted (Grafana Alloy, Collector, Tempo, Loki) | OTEL_EXPORTER_OTLP_ENDPOINT | gRPC or HTTP receiver; match the protocol to the port. |
| Local Docker | nothing — Compose sets it | Aspire Dashboard, see below. |
Azure
- Put
APPLICATIONINSIGHTS_CONNECTION_STRINGon the app (runtime), not the GitHub/Oryx build env. - Do not run the classic site agent (
ApplicationInsightsAgent_EXTENSION_VERSION=~3) alongside Hub OTel. The agent collects HTTP on its own; it does not subscribe to the OTel Logs API, so you get duplicate requests and still no app logs from it. - Confirm boot: process Log stream shows
Telemetry SDK started in Azure AppInsights mode. In App Insights (wait a minute for ingest):
traces
| where cloud_RoleName == "endatix-hub"
| where message startswith "Telemetry SDK started"
| order by timestamp desc
- Application logs (same table; logger name
instrumentationfor lifecycle):
union traces, exceptions
| where cloud_RoleName == "endatix-hub"
| order by timestamp desc
Use requests and dependencies for HTTP spans.
AWS, Google Cloud, Helm, and self-hosted
Point OTEL_EXPORTER_OTLP_ENDPOINT at a collector (ADOT, Grafana Alloy, vanilla Collector) and set OTEL_EXPORTER_OTLP_PROTOCOL to match its receiver. Collector-to-backend auth, sampling, and cloud resource attributes belong in the collector.
Helm (hub/helm/values.yaml) defaults otel.enabled to false. When enabled, the chart sets OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL (otel.protocol, default grpc), and NEXT_OTEL_VERBOSE. An empty endpoint with otel.enabled=false is the same as no export.
docker compose up already points Hub at the Aspire Dashboard — local development on the API page.
Stdout
With an exporter set, Hub writes nothing extra to stdout by default, so a host that also collects container stdout does not ingest every record twice.
Set TELEMETRY_CONSOLE_FALLBACK=true when the host's log stream is where you look (SWA log stream, kubectl logs, CloudWatch, Cloud Logging). Each record is one JSON line with timestamp, severity, logger, body, traceId, spanId, and attributes, so log agents keep it as one entry. If log export is configured but did not start (every log exporter failed to build, or the SDK failed), TelemetryLogger prints to stdout instead of dropping records. Traces-only OTLP (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT with no logs endpoint) has no log exporter: TelemetryLogger uses the console only in development or with TELEMETRY_CONSOLE_FALLBACK=true.
Redaction
- Log attributes: string values under a credential-like key (
authorization,cookie,token,secret,password,api-key,connection string) become[REDACTED]on every destination. Booleans and numbers stay, sohasToken: trueis kept. - Spans: secret query parameters — storage SAS
sig, S3 / GCS signatures and credentials,token,access_token,id_token,refresh_token, OAuthcode— are redacted inurl.full,url.query,http.url,http.target, span names and exception events. Exceptions recorded throughTelemetryTracerare redacted too.
Shutdown
Next.js handles SIGTERM / SIGINT: it stops accepting connections, drains in-flight requests, then calls process.exit(143) (or 130). Hub emits Telemetry flushing on SIGTERM (or SIGINT) and flushes. It then holds Next's final process.exit until the SDK has shut down, at most 5 seconds, so spans and logs from the requests that finished during the drain are exported too. Before a signal arrives, process.exit behaves normally.
Telemetry never ends the process. Uncaught exceptions and unhandled promise rejections are logged as errors; Next.js keeps serving.
Traces to the API
Node 18+ fetch is undici, which bypasses node:http. Next.js's own fetch span does not add traceparent to the request, so Hub registers UndiciInstrumentation: Hub → API calls carry traceparent and join one trace. Next's duplicate fetch span is off by default (NEXT_OTEL_FETCH_DISABLED). Do not add browser FetchInstrumentation.
Noisy requests Hub serves — /_next/*, static files, /api/health, robots.txt, sitemap — are dropped for the whole trace, including the Next.js render spans under them, for Azure and OTLP alike. A static file served by the Next.js router produces only two URL-less wrapper spans (NextServer.getRequestHandler, NextServer.getServerRequestHandler); those are dropped when nothing else happened in the trace. Internal metric.* spans and calls to telemetry.nextjs.org are dropped too. Outgoing calls are never dropped for their path: fetching an image or a .json file is a real dependency.
RSC requests (?_rsc=) are traced. They are App Router navigations that render server components and call the API, so their spans and errors matter; sampling bounds the volume. Next.js marks them with next.rsc.
One limit: sampling decides when a request's first span starts, before the URL is known, so static-file requests still count against TELEMETRY_TRACES_PER_SECOND even though they export nothing. Browsers cache /_next/static files as immutable, and serving them from a CDN removes this entirely.
Next leaves the OTel and Azure exporter packages unbundled (serverExternalPackages) so instrumentation and request handlers load the same modules the SDK registered with.
App logs
Use TelemetryLogger from @/features/telemetry. Do not console.log in application code — those lines never reach Azure or OTLP.
For Endatix API failures, map with toResult(...) and pass logMessage / loggerName. Expected 403/404/validation stay quiet; unexpected failures log safe scalars only (status, error code, endpoint) — never tokens, bodies, or raw API detail.
PDF render timeout is a warn (expected deadline), not an exception.