← LLM Observability & Online EvalAll roles
LLM observability · Full-timeRemote
LLM Observability Engineer
SeniorApply by 2026-12-31
LLM ObservabilityOpenTelemetryOnline EvalPythonTrace AnalysisDataset Refresh
Apply nowThe work
Wire production traces to online scores and feedback loops that refresh offline goldens. Implement with client-chosen tooling (for example Phoenix, Langfuse, LangSmith, or in-house pipelines) — not as a vendor reseller.
You will
- Instrument LLM apps for traceability without logging prohibited PII
- Define online eval sampling strategies and alert thresholds tied to runbooks
- Promote production failures into offline regression suites on a regular cadence
- Partner with SRE on SLOs for model-backed endpoints
You likely have
- 5+ years in observability, SRE, or ML platform engineering
- Experience with trace-backed debugging of non-deterministic systems
- Working knowledge of at least one LLM observability ecosystem tool
- Bias toward action — dashboards that trigger dataset updates, not shelfware
Candidate apply
Apply for LLM Observability Engineer
Six fields to start. Your profile is saved locally for faster re-apply across roles.
LLM Observability Engineer
Apply