JevPolicy: Deterministic Decisions from LLM Judgments
An open-source TypeScript decision runtime that turns probabilistic LLM judgments into versioned, deterministic, replayable, and observable application decisions.
Pipeline
Highlights
- ▸Code calculates, the model judges, policy decides: the LLM answers typed questions, and ordered YAML rules make the actual decision.
- ▸Provider timeouts and invalid responses map only to explicit policy fallbacks. They never silently become an allow.
- ▸Recorded decisions replay against new policies without calling the model again, and shadow mode tests a candidate policy on live traffic.
- ▸Semantic policy diffs, calibration reports against ground-truth outcomes, and an OpenTelemetry observer that never logs state or prompts.
The idea
LLMs are good at judgment (“is this ticket about billing?”) and bad at being the final authority on what an application does. JevPolicy splits those responsibilities. The application supplies state and facts, the model answers typed questions with probabilities, and versioned YAML rules turn those signals into a decision plus an audit trace. The host application stays responsible for carrying out any action.
That fits workflows like support routing, moderation, approval gates, risk review, and escalation: places where you want model judgment but need to explain, reproduce, and change the decision logic without retraining anything.
Built for changing policies safely
- Replay. Successful evaluations record question-fingerprinted signals, so a past decision can be re-run against a new policy offline.
- Shadow evaluation. A candidate policy runs on the same provider response as the live one. Both are recorded, and only the active decision drives side effects. Incompatible policy pairs are rejected before the provider is called.
- Semantic diff. Rules and preconditions are matched by ID, and reordering is reported separately, because rule order decides which decision wins.
- Calibration. Ground-truth outcomes live in a separate append-only file and join back to decision records, so you can measure how good the signals actually are.
Privacy defaults
Raw state recording is off by default. Policies opt into full or redacted recording, and the OpenTelemetry observer emits spans and metrics about policy identity, decisions, and fallbacks while deliberately excluding state, facts, prompts, and provider responses.