-
Notifications
You must be signed in to change notification settings - Fork 1
docs: add proposed server-side policy evaluation tutorial - #338
docs: add proposed server-side policy evaluation tutorial #338pbeckham wants to merge 41 commits into
Conversation
Claude finished @pbeckham's task in 3m 50s —— View job
Reviewing PR #338 — Server-side policy evaluation tutorial
- Read CLAUDE.md and skill instructions
- Read the main tutorial file (1156 lines)
- Read navigation config changes
- Read other changed docs files (
working_with_controls.mdx) - Verify navigation entries and links
- Post review findings
Review summary
Thorough, well-structured design-preview tutorial that has clearly benefited from 41 commits of iterative review. The page is honest about its preview status (banner, callouts, and consistent "proposed" framing), and the writing is direct and precise. Most findings from earlier review rounds have been addressed. Three items remain worth noting.
Findings
Improvement
1. Non-existent schema URLs in curl commands — tutorials/evaluate_policies_server_side.mdx:607,677
Three schema URLs are referenced as downloadable resources and used in curl commands:
https://docs.kosli.com/schemas/evaluation-input/trail/v1.json(lines 583, 607)https://docs.kosli.com/schemas/evaluation-input/trails/v1.json(line 584)https://docs.kosli.com/rego/capabilities/v1.json(line 677)
These don't exist in this repo or on the live site. A reader following the type-checking or capabilities-checking sections will get 404s. Since the page is a design preview, this is expected — but the curl commands present them as working instructions without any caveat. Consider adding "(not yet published)" to the table rows or an <Info> note near the commands to set expectations.
(Previously flagged by a reviewer; acknowledged as deferred.)
2. Relative link in mockups/policies-ui/README.md:4
The mockups README contains ../../tutorials/evaluate_policies_server_side.mdx — a relative link. CLAUDE.md requires root-relative paths. This file isn't published by Mintlify so it won't break the site, but it would break if the directory structure changes. Minor, since it's a non-published support file.
Suggestion
3. Page length and Diátaxis classification — tutorials/evaluate_policies_server_side.mdx
At ~1,156 lines this is very dense for a tutorial. The "Start here" section and audience-based <CardGroup> help, but the page functions as a hybrid tutorial + reference + specification. The first three <Step> elements (author, ship, evaluate) are genuinely tutorial-shaped; the later sections (managing policies, viewing decisions, the Rego contract, offline replay) read more as reference material.
This is understandable for a design preview seeking feedback on the whole surface — consolidating everything makes review easier. When the feature ships and the page is no longer a preview, consider splitting the Rego authoring contract and the manage/audit sections into separate reference or how-to pages, leaving the tutorial focused on the core loop.
What looks good
- Navigation: New page correctly registered in
config/navigation.jsonunder "Evaluation" group. - Links: All 11 unique internal page links and 4 image paths resolve to existing files. No broken links, no relative links in the published content.
- Frontmatter:
titleanddescriptionpresent; description is 130 characters (under the 160-character limit). - Writing style: Active voice, imperative mood for instructions, "Kosli" used correctly (never "the Kosli platform"), American spelling throughout.
- Components: Appropriate use of
<Steps>,<Accordion>,<CardGroup>,<Warning>,<Info>,<Tip>,<Note>,<Frame>. - Images: All four policy screenshots are real mockups (120-220KB), not placeholders — addressed from earlier review feedback.
- Design preview framing: The
<Warning>banner at the top is clear, and the "not yet available" / "proposed" language is consistent throughout. The in-scope / out-of-scope split is explicit and well-reasoned. working_with_controls.mdxchanges: Fixes a broken prerequisite link (/getting_started/service-accounts→/getting_started/authenticating_to_kosli) and clarifies archive/unarchive semantics. Both are clean improvements.- Mockups directory: Well-organized with a
.gitignorethat correctly excludes generated CSS and HTML from the repo, preventing Mintlify from picking them up. - Cross-page anchors: All 3 anchored links to
working_with_controls.mdxsections (#creating-a-control,#recording-a-decision-against-a-control,#requiring-decisions-in-flow-templates) point to existing headings.
Overall: 0 critical, 2 improvement, 1 suggestion. The page is in good shape for a design preview. The outstanding items are minor and can be addressed when the feature ships.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
tooky
commented
Jul 29, 2026
Thoughts
- "Running evaluation inside Kosli changes what a decision means"
- Why does it change what it means?
- I’m not sure why it keeps talking about "a core loop"?
- Probably need a new role or two for control and policy management
- I’m not sure if we should assume
--flowis defaulted into context onkosli evaluate policy- The note "a command that names its source and its destination is easier to read back months later than one relying on a default." Is describing a potential footgun??
- What’s the behaviour without sync? How can the customer check for status of evaluation that is async?
- "Be deliberate about which flows may satisfy a production control." - wondering if this feels a bit alarmist to include at this stage. I think maybe there’s an FAQ section we could add?
- "A decision carries two timestamps. " Really keen that we make sure this has been seen widely by the team. I might need to see some examples to help understand what makes sense.
- Maybe it would need stating explicitly that request time would be used as the query time for the trail data
- Alternative we could consider is that if an evaluation request time doesn’t return the latest moment we invalidate the request somehow?
- Did you consider any alternatives for trail and flow "co-ordinates" that the
<flow>/<trail>format (e.g. trail=build-ci/build-41)? - "summarize it into JSON when you make the attestation and pass it as --user-data" <- this may depend on attestation type, its probably part of the main
attestation_datafor a custom attestation type schema? - Have been experimenting with some different policy output ideas for richer feedback, we may be able to provide "libarary" rego they can choose to use along with their rego, maybe we should think about how this would work now. e.g see Claude conversation:
- Probably not v1 - but we might have to support multiple Engine and Rego versions? For backwards compatibility purposes? Perhaps the policy should be published with a version it is tested against? Compatible with?
--dry-runin Kosli cli usually does not hit the server? Should we use a different flag- Can we evaluate with
--assertbut no decision recorded
- Can we evaluate with
- Evaluate a local policy file – wondering about recording the sha256 of the policy?
- Thinking about the "Review a policy change" section and whether we should help people create trails for their policy management processes
- I really like that we’ve gone straight for policy bundles - what does that require from us for validating the bundle on upload?
- Does the "Re-run a decision offline" section need to be updated to account for policy bundles?
- Out of scope says: Composable policies — composing many authored policies into one versioned policy? Is that different from a policy bundle
AI generation tells
- Opening para has a few rhetorical flourishes (e.g. "one it can vouch for")
- "The whole loop in two commands", "Publish a policy, then evaluate it. That is the feature:"
- Some of the language feels a bit chatty "worth reading before you send feedback, so you don’t spend it on something already deferred."
- "Note that --context is doing real work here:"
Review flagged several phrases as reading like generated marketing copy rather than documentation. Each is replaced with the underlying fact. - "records the decision itself — one it can vouch for" states the claim instead of the mechanism; say what is recorded alongside the decision. Same for the comparison table's "a decision Kosli computed and can vouch for". - "The whole loop in two commands" / "That is the feature:" sells rather than informs. - "worth reading before you send feedback, so you don't spend it on something already deferred" — the Feedback section already covers this. - "Note that --context is doing real work here:" editorialises about the sentence that follows it. - "core loop" was a drafting label that meant nothing to readers; three uses replaced with what the section actually covers. - "changes what a decision means" asserted significance without saying what changed. It is who computed the decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--context defaulted to the trail named by --flow / --trail. Review pointed out that the note defending the default was describing a footgun rather than removing one: with the default in place, a command that reads and writes different trails looks almost identical to one that does not. Makes --context required, the same decision already taken for --trail and for the same reason — this command does not infer trails. Every example on the page already passed it explicitly, so no example changes. This also removes the warning that existed only to manage the hazard the default created; three lines of plain description replace it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The --context cell was a ~700 character paragraph inside a table, which does not read on narrow viewports. The cell now states what the flag is and that it is required; the reference-type syntax, repeat semantics, and room for future context types move to prose under the table. Also drops a "by default the decision is recorded on the same trail whose facts you evaluate" left over from when --context had a default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Result" names what an evaluation produces better than "output", and pairs with the input schema: input schema in, result schema out. Renames the seven uses that mean the policy's return value, including the "Output" section heading and "output schema" in the validation list. The four that mean something else are unchanged: the --output flag on kosli list decisions and kosli get evaluation-context, OPA's print/trace output, and the draft banner's "commands, flags, and output shown". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A policy can be published as a bundle of several Rego files, but two places still assumed a single file. "Re-run a decision offline" exported the policy as one policy.rego and replayed it with --data, which cannot round-trip a multi-file bundle. The export is now a policy/ directory laid out as published, replayed with --bundle so the same command works for one file or many. Says that the layout is preserved so the digest on the decision can be verified, not just the outcome re-run. The "composable policies" scope bullet read as a description of a bundle, which is what prompted the question. It now names the distinction: composing separately published policies is out of scope, one policy spanning several files is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The publish step offered a directory of Rego files, but every other mention of --policy said "file" only — so a reader who took up that offer had no documented way to validate or ad-hoc-evaluate what they wrote. --policy now consistently takes a single .rego file or a directory holding one policy's bundle, on create, validate and evaluate. States that a directory is one policy's bundle, not a collection of separate policies, which "a directory of them" left ambiguous. Renames "Evaluate a local policy file" to "Evaluate a local policy" since it is no longer file-only, and updates the one link to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The validation list was written for a single file, leaving it unclear whether a multi-file bundle is checked as one unit. States that it is, without specifying the mechanism, and that publishing runs the same checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two terms a bundle author needs that the page left implicit. Tests: the page states twice that tests are stored but never executed, which depends on Kosli telling a test file from a policy file. Names the convention (*_test.rego) so a misnamed file does not silently become evaluated policy. Roots: previously appeared only inside the policy-tab screenshot alt text, never defined. Glossed where the tab's contents are described. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two {/* TODO */} comments did not render, so the open questions they held
were visible only in source. The surrounding prose reads complete without
them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows the earlier pass on flagged phrasing with a sweep for the same patterns elsewhere. Each cut removes a verdict about a fact, not the fact. - "the strongest form of the audit story, because it doesn't ask anyone to trust Kosli to check Kosli" — the sentence after it states the point. - "nobody has to trust that version 3 is still what version 3 was" — the preceding clause already says a digest proves the bytes. - "a summary with no attachment behind it is just another self-reported number" — a closer, after the reason was already given. - "what makes a server-side decision worth more than a self-reported one" — keeps who can do what, drops the judgement. - "This is what makes publishing safe to run" — leads with the fact: publishing is idempotent. - The control owner's question was framed twice, ~80 lines apart, in the same construction. Keeps the one in the review workflow, where it is actionable. - Drops the trade-booking analogy from the two-timestamp explanation, and a filler "Note that". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Evaluation is asynchronous by default, but the page never showed what the default returns and gave no way to observe it — every example passed --sync or --assert. Review asked what the behaviour is without --sync and how a customer checks status. The evaluation context answers both, since it exists from the moment an evaluation is queued and already holds the requested and recorded times. - Shows what the asynchronous command returns: the context that will carry the outcome. - Adds "Check an asynchronous evaluation": kosli get evaluation-context reports STATUS (queued, completed, failed) and names the decision it produced. Also documents --show-input for returning the policy input that was evaluated, matching the flag of the same name on the client-side evaluate commands so jq '.input' and kosli evaluate input keep working. - States that a typed error lands on the context, so a broken policy is distinguishable from a non-compliant artifact without --sync. Corrects the fixture export from --output fixture to --fixture. In the CLI, --output selects a format ([table, json, markdown]) and never content, so overloading it here contradicted every other kosli get. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review read "Be deliberate about which flows may satisfy a production control." as alarmist for a design preview of a feature nobody can use yet. The sentence added a directive tone without adding information — the rest of the callout already states the gap and what to do about it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review asked for the request timestamp to be stated explicitly as the query time for trail data, and for a worked example to make the model concrete. States the mechanism: Kosli fixes the trail moment (which attestations are in scope) at the requested/enqueue time, then queries and serializes that data only when the evaluation executes — always as of the fixed timestamp, not as of whatever the trail contains by then. This is what makes a queued evaluation deterministic regardless of queue delay, and is why no drift-detection or invalidation mechanism is needed for a fact that lands mid-queue: the query never looks at "now," only at the timestamp fixed at enqueue. Replaces the abstract "facts arrive mid-flight" paragraph with a worked example using the same evaluation context (ec_7f3a91c2) and timestamps already introduced in "Check an asynchronous evaluation", so the same example carries through both sections instead of introducing new arbitrary numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review pointed out that recommending --user-data for a summarized attachment may depend on attestation type — it's probably part of the main attestation_data for a custom attestation type schema. Checked against the CLI reference: --attestation-data exists only on kosli attest custom, documented as the custom attestation data validated against the type's schema. --user-data is documented as "additional data" on every attestation command, never the schema-validated payload. So the earlier guidance to always use --user-data was incomplete: for a custom attestation type you define, --attestation-data is the schema- validated field and the better fit for a recurring summary shape (finding counts, severities, package IDs). --user-data remains correct for built-in types (generic, snyk, sonar, and others) that have no schema field for the summary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review asked what --dry-run's "evaluate without recording" actually adds once every other way to test a policy is accounted for: - Rego correctness and schema checks: kosli validate policy — no trail data touched, nothing recorded. - Behavioral correctness: Rego unit tests / opa test, including regression fixtures already exported from past decisions. - Behavior against real trail data: "Evaluate a local policy" already covers this. It records a decision, but the page already says to treat that as expected — the Warning right below it states these decisions cite an unpublished source, should be treated as iterating rather than audit evidence, and that the beta is soliciting feedback on marking or restricting them. Once that one real evaluation exists, its exported fixture can be re-run locally with opa eval against any number of different candidate policies, for free, with nothing further recorded. --dry-run also collided with the CLI-wide convention (--dry-run means "no data sent, exits 0 regardless of errors" everywhere else), which this turn's discussion had been trying to fix by renaming. Removing the overlapping capability is simpler than renaming it. Drops the --dry-run flag-table row, the "Run an ad-hoc evaluation" section, and the dangling --dry-run mention at the end of "Evaluate a local policy". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Separate evaluating a policy from claiming a control is satisfied. Omitting --control runs the evaluation and records the result without writing a decision, so a policy can be checked against real facts without adding to the set of decisions an auditor samples. The destination flags become required-together-with --control rather than unconditionally required. Rename evaluation-context to evaluation, since it is now the primary record of a run rather than a byproduct of a decision. Document that an evaluation reaches a terminal status once and is never revised, that clients branch on STATUS rather than on field presence, and add --wait so the async path does not require a hand-written poll loop. Abandoned evaluations terminate with a typed error instead of sitting in queued. Resolve three open questions the change makes answerable: local policies can no longer record decisions (publishing is the org-admin gate), built-in errors abort rather than evaluating to undefined, and print/trace output is discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Improvement: These schema URLs (evaluation-input/trail/v1.json, trails/v1.json in the table above, and rego/capabilities/v1.json at line 677) don't exist yet — the curl commands will 404 on the preview site. Since the rest of the page is carefully framed as a design preview, consider adding "(not yet published)" after each URL in the table, or an <Info> note near the curl blocks, so readers who try to follow along hit an explanation rather than a silent failure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Improvement (minor): This is a relative link (../../tutorials/...). CLAUDE.md requires root-relative paths. Since this README isn't published by Mintlify it won't break the site, but it would break if the directory structure changes.
What
Documentation-driven development for the server-side Kosli Evaluate project (kosli-dev/server#6267). Adds a tutorial describing the proposed V1
kosli evaluate policyflow, published so it can be shared internally and with design-partner customers for feedback before the feature is built.This is a design preview — nothing in it is runnable today. The page is marked as such throughout.
Scope (this draft)
Core happy path for a governance platform engineer:
allowmandatory).kosli evaluate policyruns server-side and records a decision against a control; synchronous.attestation_ids, policy version, params, timestamp; includes the T / T+1 / T+2 temporal-correctness case.An explicit out-of-scope-for-V1 section keeps reviewer feedback on target (managing policies, testing/validation, reporting/audit views, unhappy-path inspection, composable policies, non-trail targets, scheduled evals, risk measurement).
Design notes surfaced for feedback
kosli attest decisionandkosli evaluate trailflags.--compliantand evidence (--attachments/--user-data) are intentionally not flags — the server derives compliance and captures the policy/report/violations as output.kosli create policy, which collides with the existing environment-policy command — flagged inline as an open question.inputshape is borrowed from the client-side tutorial and may differ server-side.Draft because
Content describes proposed behavior and will change as the design firms up; opening as draft to gather review before it's merged/published.
🤖 Generated with Claude Code