instrumenting-with-mlflow-tracing

Add or repair MLflow tracing in Python or TypeScript applications, including tracing required for evaluation. Covers framework autologging, custom spans, backend setup, and persisted trace verification.

1,056
installs: 0 up / 0 down, score 0.00
votes: 82
repo stars
Updated: less than a month ago
In the catalog since: 21 Oct 2026
From: mlflow/skills

Get it through your agent npx skills add mlflow/skills@instrumenting-with-mlflow-tracing

Skill instructions

Rendered from the published SKILL.md, 92 lines.

Instrument MLflow tracing

Reuse working tracing for investigation or evaluation.

Prerequisites

Check entry point, language/framework, SDK version, tracking URI, experiment, and deployment identity.
Preserve destinations and providers. Explicit application destinations override inherited environment values; assistant telemetry may use another experiment.
Read Python or TypeScript for the runtime.

Step 1: Check the destination (Databricks only)

For OSS or SageMaker, keep the configured backend and proceed to Step 2.

Read UC setup before editing; validate auth/workspace.
Reuse the explicit destination; otherwise resolve schema choice before provisioning.
Discover an accessible warehouse automatically. Default to UC; legacy storage requires an explicit request.

Step 2: Enable framework tracing

Enable the matching integration before execution. LangGraph uses mlflow.langchain.autolog(), not mlflow.langgraph.autolog().
Do not wrap automatically traced nodes, tools, or models again. Reserve manual spans for uncaptured work or an outer application boundary.

Step 3: Capture application work

Trace meaningful root, LLM, retrieval, tool, routing, and external I/O (files/queues). Skip utility, configuration, and logging noise.
Capture actual relevant inputs/outputs explicitly for manual spans; redact before export.

Step 4: Run one request

Run one representative request; record start/end epoch milliseconds.
Save its trace ID immediately in-process: mlflow.get_last_active_trace_id() in Python; the active span's ID in TypeScript.
Write/flush the trace ID to a file or captured stdout before readback; retain it if the process is cancelled. Do not rerun to recover an ID.

Verification

Fetch a persisted application trace, not a local span or old trace.
Set a 60-second wall-clock process deadline for readback, including flush, SDK retries, requests, and backoff. Terminate at expiry unless the user explicitly requests deeper troubleshooting.
Tool yields/thread-wait timeouts do not cancel calls. Without cancellation, report backend verification as unperformed.
Flush once (mlflow.flush_trace_async_logging() / await mlflow.flushTraces()); fetch the saved ID.
Without an ID, search once in the configured experiment with run-time bounds, max_results=1, and spans included.
Every UC search needs trace.timestamp_ms; result limits do not bound scans. Python search_traces() has no start_time keyword.

For Python readback, adapt this recipe to the saved run values and enforce the deadline externally:

mlflow.flush_trace_async_logging()
if trace_id:
    trace = mlflow.get_trace(trace_id)
else:
    traces = mlflow.search_traces(
        locations=[experiment_id],
        filter_string=(f"trace.timestamp_ms >= {run_start_ms} "
                       f"AND trace.timestamp_ms <= {run_end_ms}"),
        max_results=1, return_type="list", include_spans=True,
    )
    trace = traces[0] if traces else None
assert trace is not None, "No persisted trace read back"
assert run_start_ms <= trace.info.request_time <= run_end_ms
assert trace.data.spans, "No spans captured"

Inspect expected root/graph, model, retrieval, and tool spans and actual I/O. Confirm the application's request, not scorer or assistant telemetry.
Check experiment/location from returned metadata, or the bound UC location if no experiment ID exists.
Reuse returned spans; stop after success. Available agent-evaluation/scripts/validate_tracing_runtime.py may replace this check under the same deadline, not add another audit.
Retry at most once after a concrete quick fix within the original budget.
Stop on missing provider keys, auth/permission errors, warehouse timeouts, or RESOURCE_EXHAUSTED/429.
No parallel searches, repeated agent runs, direct UC SQL, raw REST, alternate credentials, or warehouse fallbacks.
Report operation/error, captured ID, successes, and unverified results. Request the missing prerequisite, not secrets.
Keep instrumentation changes if backend verification is blocked; pause evaluation until tracing is verified.

Return the verified trace ID/link and observed spans. For Databricks use the verified trace URL rules, not an experiment-only link. Do not label an unverified link as proof.

References

Deployment, async export, sampling: production.
Async/thread context, redaction, caught errors or links: advanced patterns.
Cross-service parent propagation: distributed tracing.
User ratings tied to traces: feedback collection.
Unreadable non-chat root previews: root span previews.
Explaining captured behavior: debug-agent; implementing a requested behavior fix: fix-agent-issue, carrying the trace evidence. These are optional follow-on tasks.
Other integrations: automatic tracing docs. Check installed signatures/types.
Missing/moved topic: llms.txt. Docs unavailable: use local recipes and installed API introspection; stop if correctness remains uncertain.

Get it through your agent

Call the Graph-of-Skills MCP server's retrieve_skill_bundle tool and it returns this skill's text inside your budget.

{
  "tool": "retrieve_skill_bundle",
  "arguments": {
    "query": "instrumenting-with-mlflow-tracing"
  }
}

New to this? Set up Graph-of-Skills or get a free key. The text on this page comes from its authors: how skills are sourced and what we check. If none fits your task, your agent can write a skill for your own library. To keep a skill from your own library on a machine, see how to install one of your skills.

Share on X Share on LinkedIn Share on Bluesky Share on Hacker News

Share on Reddit

Source

Source repository mlflow/skills
Listing on skills.sh

From mlflow/skills via skills.sh

Related skills

phoenix-tracing
OpenInference semantic conventions and instrumentation for Phoenix AI observability. Use when implementing LLM tracing, creating custom spans, or deploying to production.
1,041 installs
otel-instrumentation
OpenTelemetry SDK setup, custom instrumentation, and sending data to Honeycomb. Trigger phrases: "instrument my app", "add tracing", "set up OpenTelemetry", "configure OTel", "add custom spans", "add attributes to spans", "send traces to Honeycomb", "set up OTLP", "configure sampling", "add span events", "add span links", "set up tracing for [any language]", "configure the OTel Collector".
3,633 installs
phoenix-tracing
OpenInference semantic conventions and instrumentation for Phoenix AI observability. Use when implementing LLM tracing, creating custom spans, or deploying to production.
1,719 installs
observability
Instruments assistant-ui AI SDK backend routes with Langfuse, LangSmith, or Helicone, and renders independent trace data with the experimental @assistant-ui/react-o11y SpanResource and SpanPrimitive APIs. Use it when traces are missing, serverless spans are dropped, an AI SDK call needs provider metadata, Helicone is not proxying a provider, or a trace tree or timeline needs rendering. For streaming transport use streaming, for initial application wiring use setup, for Assistant Cloud persistence use cloud, and for copied styled trace UI use elements.
4,044 installs
arize-instrumentation
Adds Arize AX tracing to an LLM application for the first time. Follows a two-phase agent-assisted flow to analyze the codebase then implement instrumentation after user confirmation. Use when the user wants to instrument their app, add tracing from scratch, set up LLM observability, integrate OpenTelemetry or openinference, or get started with Arize tracing.
1,079 installs

More from mlflow/skills

mlflow-onboarding
Onboards users to MLflow by determining their use case (GenAI agents/apps or traditional ML/deep learning) and guiding them through relevant quickstart tutorials and initial integration. If an experiment ID is available, it should be supplied as input to help determine the use case. Use when the user asks to get started with MLflow, set up tracking, add observability, or integrate MLflow into their project. Triggers on "get started with MLflow", "set up MLflow", "onboard to MLflow", "add MLflow to my project", "how do I use MLflow".
1,119 installs
agent-evaluation
Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong/incomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).
1,062 installs
searching-mlflow-docs
Searches and retrieves MLflow documentation from the official docs site. Use when the user asks about MLflow features, APIs, integrations (LangGraph, LangChain, OpenAI, etc.), tracing, tracking, or requests to look up MLflow documentation. Triggers on "how do I use MLflow with X", "find MLflow docs for Y", "MLflow API for Z".
1,059 installs
analyzing-mlflow-trace
Analyzes a single MLflow trace to answer a user query about it. Use when the user provides a trace ID and asks to debug, investigate, find issues, root-cause errors, understand behavior, or analyze quality. Triggers on "analyze this trace", "what went wrong with this trace", "debug trace", "investigate trace", "why did this trace fail", "root cause this trace".
1,012 installs

Content is provided as published by its creators; you are responsible for how you use it.

To request removal, use the contact form.