NeMo Compass
The goal of NeMo Compass is to efficiently find signals in agent traces that lead to meaningful improvements in quality, cost, safety, privacy, and/or user satisfaction.
A critical input to improving any agent is understanding what it’s doing, especially once you turn it on in production. Evals often capture only a small subset of the tasks an agent actually needs to be good at. Agent developers have discovered that production trace analysis is an essential part of improving an agent, but today there's no standard for how this type of analysis gets validated and measured.
NeMo Compass is an applied research project with a goal of improving the tools and techniques available to developers for trace analysis at scale. It uses a range of techniques to identify different types of issues in a large set of traces, then summarizes findings as actionable insights. We call each of those discovery techniques "Compass Signals."
| Compass Signal | Question It Answers | How It Works |
|---|---|---|
| Anomaly Detection | What outlier behaviors exist in a set of agent traces? | Scores outliers with an Isolation Forest and reports feature-level deviations |
| Tool Issues | What recurring problems occur when the agent uses tools? | Checks tool calls and results for invalid arguments, failures, missing results, and unproductive retries |
| User Sentiment | What recurring complaints do users have about the agent? | Screens recorded human messages with a classifier, then reviews recurring complaints in context |
| ETHOS.md Divergence | Where does the agent’s behavior conflict with its stated rules and goals? | Compares traces with the agent’s ETHOS.md document and investigates possible gaps |
| Grader failures | What behaviors recur in traces with poor evaluation results? | Uses recorded scores and feedback to select traces, then investigates shared failure patterns |
Learn more about each current Compass Signal: Architecture
We are sharing the implementation at this stage to encourage feedback and testing by others as we continue to improve on it.
Start here
Install uv, then choose a guide:
| Your starting point | Guide |
|---|---|
| Try it before connecting your data | Run the example |
| Braintrust project logs and experiments | Braintrust |
| LangSmith traces and feedback | LangSmith |
| Langfuse traces and scores | Langfuse |
| MLflow traces and assessments | MLflow |
| NeMo Platform Intake | Intake |
| NeMo Gym rollout trajectories | NeMo Gym |
| JSONL or ATIF files | Trace files |
| Another trace store | Convert your traces |
Install from PyPI as nemo-compass; the command is compass.
If compass is not found after installation, run uv tool update-shell
and restart your terminal.
Code analysis (code_base) also requires ripgrep
on PATH. Install it separately with brew install ripgrep on macOS or
sudo apt-get install ripgrep on Debian/Ubuntu; the Python package does not install it.
After your first run
- Read your results — understand insights, skipped Compass Signals, and saved output.
- Compass Signals — supply business rules, configure sentiment, or check findings against code.
- Data and model access — configure your inference endpoint and understand where data goes.
- Configuration reference — overrides, limits, and repeatable runs.
Purpose and expectations · Development · Releasing · Architecture · Dev Notes
Research Topics
Questions we are exploring
- How can developers efficiently find recurring, fixable agent problems in trace datasets containing hundreds of thousands or millions of executions?
- How can developers estimate how often a problem occurs and how much it matters without asking an LLM to inspect every trace?
- How can trace evidence, evaluation results, and read-only checks against agent code help distinguish real problems from misleading patterns?
- How should we measure whether an insight is correct, useful, and worth a developer's time to investigate?
This repository explores these questions. The architecture guide explains what each part does and where judgment or coverage limits remain.
Help shape the direction
We are especially interested in feedback from teams building agent observability or evaluation tools, and from developers investigating their own agents:
- Which types of recurring issues are hardest to identify today?
- How do you decide whether an identified issue matters to your agent?
- What would make a finding more "actionable": examples, frequency, estimated impact, something else?
- How would you ideally use these tools in your own workflows? What would you need to integrate it?
- What types of additional insights or findings would be most valuable to you?
If an NVIDIA contact shared this preview with you, please send your feedback through that contact and ask them to route it to the NeMo Compass research team. Otherwise, use the NVIDIA Developer contact form with NeMo Compass research preview feedback in the subject. Please avoid sending raw traces, prompts, credentials, or user data.
Contributing
This project is currently not accepting code contributions, pull requests, or GitHub issues from external contributors. Please do not submit external pull requests or open issues, including bug reports and feature requests.
For research feedback, use the channels described below. To report a security vulnerability, follow SECURITY.md.
License
Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This project is licensed under the Apache License, Version 2.0. See NOTICE for project attributions, THIRD_PARTY_LICENSES.md for human-readable dependency license disclosures, and third_party/licenses.jsonl for the machine-readable inventory.
1 maintainer
Metadata
1 maintainer
Release files for nemo-compass 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| nemo_compass-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for nemo_compass-0.2.1-py3-none-any.whl |
Release files / nemo_compass-0.2.1-py3-none-any.whl
| Download URL | nemo_compass-0.2.1-py3-none-any.whl |
|---|---|
| Size | 4.7 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9768aa054570258985a30faa3131787bc67ffe7729742ac6b58a5fe5e3eb6aee
|
|
BLAKE2b-256 checksum How to use checksums |
15dd1b9e56d64c242504e61a9b9a8216fbb00b7de01057095093da7093ad03b0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Security for nemo-compass 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
