OMK

English | 简体中文

Observe. Measure. Know.

OMK makes every knowledge change in your AI application evidence-backed.

Observe real-world performance, measure version differences, and determine whether the change is effective and the version is ready to ship.

Same model. Same evaluation samples. Only the knowledge artifact changes.

DeepSeek Harness users: install OMK as a native bundle, reuse the current profile for controlled evaluations, and open persisted DSH task trajectories in Studio. Set up the DSH host plugin →

📖 Full documentation: oh-my-knowledge.pages.dev (searchable, English / 简体中文)

What OMK helps you know

Quick start

npm i -g oh-my-knowledge
omk init demo && cd demo
omk eval --control code-review-v1 --treatment code-review-v2 --dry-run
omk eval --control code-review-v1 --treatment code-review-v2

Runs out of the box — no edits needed first. omk init scaffolds two skill variants and three sample cases; --dry-run previews the sealed task plan and estimated calls; omk eval runs the controlled A/B and opens the authenticated Core run in Studio. Once it runs, swap in your own skills and cases.

Prerequisite: configure one authenticated model runtime (Codex CLI, Claude Code, or an API executor; see Requirements). Inside a Codex task in the ChatGPT desktop app, omk automatically selects codex, reads the model from ~/.codex/config.toml, and uses the same Codex model as the default judge. Claude is not required.

To make Codex the default in regular terminals, add the preference to your shell profile (for example ~/.zshrc):

export OMK_EXECUTOR=codex
# Optional: export OMK_MODEL="your-codex-model"

Without OMK_MODEL, omk reads the model from ~/.codex/config.toml. You can still pass --executor codex --model <codex-model> per command. Pass --judge-models or set OMK_JUDGE_MODELS only when you want a different judge.

The first run has only 3 cases, so the verdict will usually be UNDERPOWERED (insufficient data) — that's a normal starting point, not an error; grow to ~20+ cases before trusting a ship/no-ship call.

The CLI notifies you when a newer version is available (at most once per 20h); set OMK_SKIP_UPDATE_CHECK=1 to silence it permanently.

Walkthrough: 5-minute quickstart guide (recommended for first-time users; includes demo → own skill → verdict actions). More runnable examples (Skill Map, A/B, offline executor, agent runtime, RAG) live in the repo's example gallery.

Deeper: who omk is for · CLI reference · how it works · eval sample format · executors · artifact layout

Inspect one Codex task

If you only want to see what happened behind one Codex conversation, you do not need to run observe ingest first:

omk studio

Studio opens the local Codex conversation overview at http://127.0.0.1:7799 by default. Select a conversation, then a task, to open Task Trajectory. Its four lanes — Conversation, Actions, Results, and Knowledge — show the request, AI responses, tool calls, tool returns, and observable context, with drill-downs into normalized events and raw logs.

Running tasks are prioritized and update live. While Following, the trajectory advances smoothly as events arrive; after you inspect an earlier point, Studio keeps your position and offers View updates. Old logs without a terminal event are marked End status not recorded instead of remaining live forever.

Task Trajectory only reconstructs facts observable in the log. It does not reveal or infer hidden reasoning. See Observe production traces for the full model.

The OMK loop

OMK is for authors and maintainers of LLM knowledge artifacts who need a release decision, not for passive end-users of a skill. The main loop is deliberately controlled:

change a prompt / RAG / skill / agent artifact
→ run omk doctor before evaluation
→ run omk eval with the same model and the same samples
→ read the report / Studio evidence
→ promote a proven version or evolve a candidate
→ observe real usage and draft gap-derived samples for review

The first value is the pre-ship doctor → eval decision. The long-term value is the closed loop: observe surfaces production gaps, sample --from-traces drafts regression samples for human review, and reviewed drafts can become fixed eval samples that make the next eval harder to game.

Use inside AI Coding Agents

Install the official omk Agent Skill to let your coding agent run omk workflows from natural language:

omk install omk-agent-skill

By default, omk installs only into detected local targets it explicitly supports: Codex/AGENTS when ~/.codex or ~/.agents exists, and Claude Code when ~/.claude exists. Use --to all to force every target omk currently knows, or --dest for a custom skill root.

Use inside Claude Code

When the omk skill is available in Claude Code, you can invoke it directly:

/omk eval              # evaluate the artifact(s) in the current project
/omk evolve            # auto-iterate to improve a skill
/omk sample            # generate or fill test cases

These slash commands are natural-language entry points — the agent reads the conversation context to figure out which skill to operate on. You can also just say "compare v1 vs v2 for me" or "improve this artifact" and omk picks the right command.

Use inside Codex

Codex does not support Claude Code style /omk ... slash commands. Ask the agent to run the omk CLI directly. Inside a Codex task, omk automatically selects the Codex runtime and locally configured model:

omk eval
omk evolve skills/my-skill.md   # one-shot: doctor → (auto-generate samples if missing) → self-iterate
omk sample skills/my-skill.md

You can also describe the goal in natural language, such as "compare v1 vs v2" or "generate test cases for this skill".

eval, doctor, sample, evolve, and the LLM-enhanced observe review share the same runtime resolution. Once Codex is selected, the default judge reuses the evaluated Codex model instead of falling back to claude:haiku.

omk evolve is a one-shot loop: it runs the doctor gate first, auto-generates eval samples when the target skill has none, then self-iterates. For a brand-new skill, just run omk evolve skills/foo.md.

Why this tool

Knowledge engineering creates a versioning problem: every prompt, RAG recipe, skill, agent, or workflow can change behavior without changing application code. When someone asks "can we ship v2, and why?", a prettier answer or a higher anecdotal success rate is not enough.

omk treats the knowledge artifact as the variable under test: same model, same evaluation samples, only the artifact changes. That makes the comparison explainable, repeatable, and suitable for CI or release review.

Why omk over alternatives

omk's moat is a default-on safety net: Bootstrap CI and length-debias are normal measurement behavior, missing evidence fails closed, and explicit Gold comparison provides judge ↔ human alpha calibration. Need a hosted SaaS dashboard? Choose LangSmith. Want quick local prompt iteration without statistics? Choose promptfoo. Shipping to production and someone will ask "why should I trust this number?" Choose omk.

RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full comparison with 7 tools across 25+ dimensions: docs/reference/comparison.md.

Features

Run inside an existing DeepSeek Harness

Install OMK as a DSH bundle to reuse the profile's model, credentials, tools, and sandbox:

dsh plugin --profile web add oh-my-knowledge
dsh --profile web

Inside DSH:

• /omk eval eval.yaml runs every sample in an isolated DSH session while OMK owns the report and statistics;
• /omk observe lists recent terminal sessions;
• /omk observe <session-id> reads a consistent snapshot and returns its Studio Task Trajectory URL.

Observe uses the profile's sessionPersistence directly, so users do not export or locate JSONL / SQLite files. The first version is offline-only and does not live-follow a session that is still being written. See the executor guide and observe guide.

Connect an MCP client (experimental)

Positioning: OMK MCP is an active knowledge-feedback interface, not a conversation monitor. OMK MCP alone cannot automatically monitor or subscribe to complete conversations in Codex, ChatGPT, or another client. OMK receives a record only after the client, model, or component actively calls save_observation with authorized content. An Agent Skill may automatically recognize a potential feedback moment, but saving it still requires user confirmation and an explicit MCP tool call.

omk-mcp is a client-neutral stdio MCP server. Codex and other local MCP clients can start it directly; a private host can compose the exported Streamable HTTP adapter. The client calls save_observation only after the user explicitly asks to record feedback, appends the feedback and optional evidence under .omk/observe-inbox/captures/, and can render an inline MCP Apps review card for a human verdict and regression-sample draft.

In Codex, explicitly invoke the OMK Skill to submit the current knowledge feedback:

$omk feedback

The explicit invocation itself confirms the save. The agent selects the most recent clear issue from the visible conversation and calls save_observation with confirmedByUser: true; it asks first when the candidate is ambiguous. This shortcut is not a CLI command, and it does not automatically review the observation, draft a sample, or write to a gold set.

omk-mcp

Every record carries coverageStatus: partial: OMK observes its tool boundary, submitted feedback, and optional evidence, but not the full conversation, other tool calls, or hidden reasoning. Continuous monitoring requires a host that is authorized to access an event stream and actively forwards those events; that is a host-integration capability, not an OMK MCP capability. Conversation IDs, turn IDs, and idempotency keys are hashed rather than persisted verbatim. For a private-host Streamable HTTP integration, see Compose the OMK MCP integration.

Documentation

The full docs are published at oh-my-knowledge.pages.dev — searchable, with an English / 简体中文 switcher. Key pages:

• How it works — input compilation, sealed Core execution, analysis, persistence, and Studio projections
• Eval sample format — sample schema, scoring formulas, 30+ assertion types, custom JS assertions
• CLI reference — all top-level commands with bash examples and flag tables
• Evaluation Core cutover — BREAKING-SCHEMA storage, resume, Studio, Gold, managed-evidence, and evolve migration
• Executors & artifact layout — built-in / custom executors; how variant resolves to an artifact + runtime context
• How-to guides — evaluate an agent (project runtime context) and use non-Claude models (GLM / Qwen / DeepSeek / Moonshot / Ollama)
• Observe & inspect task trajectories — browse local Codex conversations, drill into one task, and follow its observable execution live
• Quickstart — first-time five-minute walkthrough
• Example gallery — a set of runnable examples in the repo, arranged simplest-to-richest
• Sample design spec — capability / construct / provenance metadata; industry-gap mapping
• Statistical rigor — why Bootstrap CI / Gold agreement / length-debias / evidence coverage matter
• Comparison with 7 tools — 25+ dimensions across promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai
• Evidence-gated management — managed records, lifecycle states (installed / measurable / promoted / stale), install → eval → measurable → promote → rollback

Environment variables

Requirements

• Node.js >= 22
• At least one authenticated model runtime:
  • Codex: install and authenticate the Codex CLI (npm i -g @openai/codex); Codex tasks in the ChatGPT desktop app select it automatically
  • Claude: install and authenticate Claude Code
  • API / other executors: configure them as described in Executors
• Advanced claude-sdk / codex-sdk executors are optional and are not downloaded by the base OMK install. Install the matching SDK in the same local project or global npm prefix only when you select one; see Executor prerequisites.

Security notice

This tool is designed for local trusted environments (dev machines, CI pipelines). The following features execute local code — make sure inputs come from a trusted source:

Recommendations:

• Do not expose the local report server on the public internet (no auth)
• Don't use third-party eval-samples you haven't vetted
• Custom assertions have a 30-second timeout but no sandbox isolation

See GitHub Releases for release notes. Contributions welcome — see CONTRIBUTING.