Lock Server TRACE Evidence#
TRACE support in the Lock Server accepts signed evidence records from authenticated producers, verifies who signed them, stores them unchanged, and lets authorized clients retrieve them by record or by governed execution. This page describes what the feature does and does not establish, how to configure and operate it, and its API.
Purpose#
TRACE helps answer four practical questions about a governed agent action:
- What capability was authorized?
- What enforcement point attempted or performed the invocation?
- What did each evidence producer sign?
- Which records belong to the same governed execution?
The Lock Server provides attributable, tamper-evident records and basic correlation. The core evidence path is:
authorization decision → capability invocation → optional runtime effect
A policy decision point (PDP) records what was authorized (AUTHORIZATION_DECISION). Cedarling support for producing TRACE records is in progress. The enforcement point, agent gateway, or tool gateway that mediates the action records CAPABILITY_INVOKED. An agent's own statement that it invoked a capability is not equivalent to enforcement-point evidence. An observer can record RUNTIME_EFFECT.
What TRACE does not claim#
- No completeness. TRACE does not claim that an execution's evidence is complete or that a producer reported every event.
- No truth. A valid signature shows who signed a record and that the signed fields have not changed. It does not show that the content is true, independently observed, or that enforcement succeeded.
- No causation. TRACE does not claim that a recorded action caused a claimed real-world effect. Lock does not infer that an authorization caused an invocation because their timestamps are close. Producers state causal relationships explicitly in
parent_record_ids. - No settlement or anchoring. Records are not labeled "settled", "complete", or externally anchored, and the Lock receipt chain has no signed checkpoints or external anchor (see Receipt chain).
Each record kind establishes one narrow claim and nothing stronger:
| Record | Establishes | Does not establish |
|---|---|---|
AUTHORIZATION_DECISION |
The PDP signed that it returned this decision under the referenced policy. | That the action occurred or was appropriate. |
CAPABILITY_INVOKED |
The enforcement point asserts it mediated the invocation. | That the target produced the claimed effect. |
RUNTIME_EFFECT |
The observer asserts that it observed the effect. | That the effect was authorized. |
| Lock receipt | Lock accepted these bytes at this position in its receipt chain. | That the producer's assertion is true or complete. |
Roles#
| Party | Responsibility |
|---|---|
| Policy decision point (PDP), for example Cedarling (TRACE support in progress) | Decides whether a capability is allowed and signs an AUTHORIZATION_DECISION record. |
| Enforcement point (PEP, agent gateway, tool gateway) | Mediates the action and signs a CAPABILITY_INVOKED record. |
| Evidence producer | Signs the record it creates. A PDP or enforcement point is the producer of its own records. |
| Submitting client | The OAuth client that sends a signed record to Lock. Usually the producer. An approved forwarder may submit for a producer if its binding allows that producer. |
| Lock Server | Verifies, stores, correlates, and returns evidence. Lock does not decide whether the governed capability should run. |
Three independent checks apply. Do not confuse them:
| Check | Question answered | Mechanism |
|---|---|---|
| Capability authorization | May the agent perform this action on this resource? | Cedarling or another PDP. The result is evidence that TRACE records. |
| TRACE API access control | May this client submit or read evidence through Lock? | OAuth access token, TRACE scope, and Lock endpoint policy. |
| Record verification | Who signed this record, and have the signed fields changed? | Producer key lookup and Ed25519 signature verification. |
Scopes#
| Scope | Endpoints |
|---|---|
https://jans.io/oauth/lock/trace.write |
POST /audit/trace, POST /audit/trace/bulk |
https://jans.io/oauth/lock/trace.readonly |
GET /audit/trace/records, GET /audit/trace/records/{record_id}, GET /audit/trace/executions/{trace_execution_id} |
https://jans.io/oauth/lock/trace.admin |
/audit/trace/admin/* (producer keys and chains) |
The client created by the Lock setup script receives scopes from the config-api Lock plugin, not from the Lock Server's own OpenAPI file. A client that uses TRACE needs these three scopes granted to it explicitly.
Evidence domains#
An evidence domain is the isolation boundary inside one Lock deployment. Lock derives it from the authenticated OAuth client and that client's configured binding. A producer never chooses the domain in the record.
- Producer keys, record identity, execution correlation, indexes, and retrieval are all scoped to the domain.
- A key registered in one domain gives no authority in another. A producer that serves several domains needs an explicit registration in each.
- Readers and administrators see only the domain their client is bound to. Lock never searches other domains.
- A client with no
clientDomainBindingsentry falls back todefaultEvidenceDomainIdonly for its domain. It gets no forwarding permission: see "Default-domain fallback" under Configuration. - A signed
evidence_domain_idin the body is ignored and the record is flaggedreserved_member_flag(see Single record). Lock never uses it for routing or storage.
Configuration#
TRACE is configured in the traceConfiguration block of the Lock Server dynamic configuration. The full property table is in Lock Server Configuration and Operation. The values that matter most:
"traceConfiguration": {
"enabled": true,
"defaultEvidenceDomainId": "default",
"defaultAllowedProducerIds": [],
"clientDomainBindings": [
{
"clientId": "2200.00000000-0000-0000-0000-000000000000",
"evidenceDomainId": "default",
"allowedProducerIds": ["cedarling-fleet-1", "gateway-fleet-1"]
}
],
"latenessThresholdSeconds": 300,
"policyMismatchGraceSeconds": 300,
"policyReleaseRetentionDays": 30,
"maxFutureSkewSeconds": 300,
"maxRequestBytes": 262144,
"maxBulkRecords": 100,
"maxBulkRequestBytes": 4194304,
"correlationRetryLimit": 3,
"chainRegistrationMode": "PRE_REGISTERED",
"cryptoProvider": "JAVA",
"receiptAllocationRetryLimit": 8,
"batchIngestEnabled": false,
"rangeAllocationRetryLimit": 32,
"rangeAllocationBackoffMs": 5,
"rangeAllocationMaxBackoffMs": 250,
"batchVerifyThreads": 0,
"postCommitCorrelationThreads": 0,
"receiptRepairIntervalSeconds": 300,
"pendingReceiptTimeoutSeconds": 120
}
Client-to-domain bindings. Lock resolves the domain for an authenticated client_id in this order: an explicit entry in clientDomainBindings, then defaultEvidenceDomainId if set, otherwise the request fails with 403 client_not_bound. The same rule applies to submitters, readers, and administrators.
Default-domain fallback. A client that falls back to defaultEvidenceDomainId has no forwarding permission of its own. Its allowlist is defaultAllowedProducerIds (default: empty), never "*". With the default empty list, an unbound client can read and administer the default domain but cannot submit any record: every submission fails with 403 producer_not_allowed. To let unbound clients submit for specific producers, list those producers in defaultAllowedProducerIds. To let a client forward for a producer other than itself, give the client an explicit clientDomainBindings entry.
Forwarding. A client may submit a record only if the binding's allowedProducerIds contains the signed producer string exactly, or contains "*". Otherwise the request fails with 403 producer_not_allowed.
Limits. Lock applies the request limits (maxRequestBytes, maxJsonDepth, maxStringLength, maxArrayLength, maxObjectMembers) before any cryptographic work. A request over a limit fails with 400 invalid_request. For bulk requests, maxBulkRecords bounds the array length and maxBulkRequestBytes bounds the whole body (always clamped to 16 MiB). maxRequestBytes still bounds each record in the batch.
Lateness. A record is flagged late_flag: true when Lock receives it more than latenessThresholdSeconds after its signed_at. Lateness alone never rejects a record. A signed_at that is more than maxFutureSkewSeconds ahead of Lock's clock is rejected with 400 invalid_assertion and reason signed_at_in_future. The two properties are independent.
Future skew. maxFutureSkewSeconds defaults to 300. A value of 0 or less falls back to 300. Lock compares whole seconds on its own clock: signed_at equal to now + maxFutureSkewSeconds is accepted, one second more is rejected. A rejection stores nothing and uses no receipt position. The producer can fix its clock and re-sign with a corrected signed_at, or retry the unchanged assertion once Lock's clock reaches the permitted skew window. There is no lower bound: a signed_at in the past is never rejected. The check is a plausibility test on a producer claim. It does not make an accepted signed_at accurate. Use receipt_sequence for Lock receipt order. received_at is Lock's observation time.
Neighbor-update retries. After a record is stored, Lock updates derived flags on neighboring records. Each update is a compare-and-swap write against the neighbor's version. On a version conflict, Lock re-reads the neighbor, recomputes the flag from its current state and retries, up to correlationRetryLimit times. When the budget is exhausted Lock logs a warning and gives up on that update. It never fails the request.
Chain registration mode. chainRegistrationMode is PRE_REGISTERED (default) or FIRST_OBSERVED. See Chain registration modes. A value that is not one of the two names is treated as PRE_REGISTERED and logged once at WARN.
Cedarling as a producer. Cedarling needs traceConfiguration settings that differ from the defaults described in the "Default-domain fallback" and "Chain registration mode" paragraphs above. With the defaults, Lock refuses every record from Cedarling. Before you connect Cedarling, read Auth Server and Lock Server setup for the values to set.
Crypto provider. cryptoProvider selects the implementation that parses producer keys, canonicalizes assertions, verifies signatures and computes receipt hashes. Values are matched case-insensitively:
JAVA(default): BouncyCastle, in the Lock process. This is the implementation Lock shipped with.NATIVE:jans-trace-core, the Rust TRACE library intended for producers, reached through thejans-trace-javabinding.
Both providers produce the same content_digest, key thumbprint, receipt hash and storage keys. Switching the provider does not invalidate the stored receipt chain.
Lock fails closed. When the selected provider is not available, because the native library is missing or its self-test fails, every TRACE submission returns 500 crypto_unavailable. Lock never falls back to the other provider. Look for the startup log line TRACE crypto provider: type=<type> available=<true|false> detail=<detail>. The FIPS WAR ships without the native provider, so NATIVE fails closed there.
NATIVE rejects small-order Ed25519 public keys that BouncyCastle accepts. After you switch to NATIVE, a registered key of that kind fails with unknown_producer_key or stored_key_unusable. Check registered keys before you switch.
A value that is not one of the two names is treated as JAVA and logged once at WARN: Unknown traceConfiguration.cryptoProvider '<value>', using JAVA.
Policy store release check. policyMismatchGraceSeconds (default 300) is how long after the last node stops serving a policy store release records judged under it still count as matching; see the acceptance window below. policyReleaseRetentionDays (default 30) is how long a release no node serves any more is kept before it is deleted. A mismatch only sets policy_mismatch_flag; it never rejects a record. See Policy store release check.
Batch ingest. batchIngestEnabled (default false) changes how Lock writes a request. Off, Lock ingests each element on its own: about three database commits per record (claim the receipt, insert the record, settle the receipt). On, Lock verifies and plans every element first, then writes the receipts and the records of the whole request in one database transaction; a single submission is written as a batch of one, with one commit. batchVerifyThreads and postCommitCorrelationThreads (both default 0) move verification and the post-commit flag updates of the batch path to bounded thread pools. rangeAllocationRetryLimit (default 32), rangeAllocationBackoffMs (default 5) and rangeAllocationMaxBackoffMs (default 250) bound how often, and after what random wait, a batch retries its receipt range when another node took a position first; receiptAllocationRetryLimit (default 8) applies only to the per-record path. See Batch ingest for the behavior and for how to enable and roll back.
Timers. The receipt repair timer runs every receiptRepairIntervalSeconds. It settles receipt claims that stayed PENDING longer than pendingReceiptTimeoutSeconds, up to 500 per run (see Receipt chain and Deployment constraints). Lock derives its ingest deadline from pendingReceiptTimeoutSeconds. It has no setting of its own. Setting enabled to false makes every TRACE endpoint return a structured invalid_request error with reason trace_disabled and stops the timer.
Registering producer keys and chains#
Lock accepts a producer key only through the admin API. It never accepts a public key because it accompanied a record. Possession of a signing key does not let the holder define its own producer identity. Registration, in the administrator's evidence domain, grants the authority.
The examples below use a token with the trace.admin scope:
HOST=https://lock.example.org
API="$HOST/jans-lock/api/v1/audit/trace"
TOKEN=$(curl -s -u "$CLIENT_ID:$CLIENT_SECRET" "$HOST/jans-auth/restv1/token" \
-d grant_type=client_credentials \
-d scope=https://jans.io/oauth/lock/trace.admin | jq -r .access_token)
producer_id can contain /, so the admin API carries it in request bodies and query parameters, never in path segments.
Register a key#
curl -s -X POST "$API/admin/producer-keys" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{
"producer_id": "cedarling-fleet-1",
"kid": "cedarling-fleet-1-2026-01",
"public_key_jwk": { "kty": "OKP", "crv": "Ed25519", "x": "<base64url public key, 32 bytes>" },
"authorized_event_kinds": ["AUTHORIZATION_DECISION"],
"valid_from": "2026-01-01T00:00:00Z",
"valid_until": null
}'
producer_id is the stable logical producer identifier, for example cedarling-fleet-1. It is not a name/semver string. Software version and build travel as separate signed metadata (see Single record), so a software upgrade does not force a new producer identity, key or chain. The identifier is 1 to 255 characters, starts with a letter or digit, and then uses letters, digits and . _ : / -. Lock compares it byte for byte and never normalizes it.
authorized_event_kinds is required. It is a non-empty list of AUTHORIZATION_DECISION, CAPABILITY_INVOKED and RUNTIME_EFFECT without duplicates. A missing or empty list, an unknown kind or a duplicate returns 400 invalid_key with reason authorized_event_kinds. The scope is immutable. To change it, register a new kid.
The response is 201 with the stored key, including evidence_domain_id, authorized_event_kinds and revoked_at: null. Registering the same (producer_id, kid) again returns 409 key_already_exists. Keys are create-only: rotate by registering a new kid, and retire a key by revoking it. A JWK that is not {kty: OKP, crv: Ed25519} with a 32-byte x, or dates that do not parse, return 400 invalid_key.
A key verifies a record only when, at verification time:
valid_fromhas passed,valid_untilis unset or in the future, andrevoked_atis unset or in the future. Otherwise Lock rejects the record with400 key_not_valid.- The record's
trace.event_kindis in the key'sauthorized_event_kinds. Otherwise Lock rejects the record with400 unknown_producer_keyand reasonevent_kind_not_authorized, exactly as for an unknown key. Nothing is stored or indexed. This check runs before any signature work.
A key row that was stored before authorized_event_kinds existed authorizes no event kinds. Such a producer must register a new kid.
The verification result stored with each record names the exact kid, the key thumbprint and the time of verification (see Retrieval). A later revocation, expiry or change to the key never changes it.
Revoke a key#
curl -s -X POST "$API/admin/producer-keys/revoke" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{ "producer_id": "cedarling-fleet-1", "kid": "cedarling-fleet-1-2026-01" }'
The response is 200 with the key and its revoked_at. Revoking an already revoked key is idempotent. An unknown key returns 404 record_not_found.
Register a producer chain#
Every record belongs to a producer chain. With the default PRE_REGISTERED mode the chain must be registered before the producer's first record. The model is producer_id → producer_instance_id → one or more producer_chain_id values.
curl -s -X POST "$API/admin/producer-chains" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{
"producer_id": "cedarling-fleet-1",
"producer_instance_id": "cedarling-001",
"producer_chain_id": "chain-01JABC9Z0K"
}'
The response is 201 and includes "genesis_kind": "pre_registered_genesis". A duplicate returns 409 chain_already_exists. In PRE_REGISTERED mode a record whose chain is not registered is rejected with 400 chain_not_registered, at every sequence position.
One producer instance can own several chains at once: register each producer_chain_id separately. Chains are independent. Each has its own sequence numbers, its own gap, link and equivocation findings, and its own prev_record_hash links. Records of different chains can be interleaved in any order, in single or bulk submissions, and a batch can span several chains of the same producer. The Lock receipt sequence still increases per evidence domain across all chains.
Chain registration modes#
| Mode | Behavior |
|---|---|
PRE_REGISTERED (default) |
A chain exists only after an administrator registers it. A record on an unregistered chain is rejected with 400 chain_not_registered. |
FIRST_OBSERVED |
A genesis record (sequence_number 1 with prev_record_hash set to the all-zero sentinel) on an unregistered chain registers the chain. Registration happens after the key, event-kind scope and signature checks succeeded, and before the record is stored. registered_by is the submitting client. Chains registered through the admin API keep working as before. |
In FIRST_OBSERVED mode:
- A record that fails any admission check (schema, forwarding, key, key validity, event-kind scope, signature) never registers a chain.
- Any other record on an unregistered chain is still rejected with
400 chain_not_registered. This includes a producer that restarted mid-chain, or records that arrive out of order before the genesis record. Lock does not register a chain from a non-genesis record. - A genesis position without the sentinel, or the sentinel at a non-genesis position, is still
400 invalid_genesis. - If two genesis records for one chain arrive at the same time, both are accepted: one registers the chain and the other finds it registered.
- Changing the mode affects only later submissions. Lock does not change existing chain registrations.
The chain list shows how each chain came to exist in genesis_kind: pre_registered_genesis for chains registered through the admin API (and for chains stored before this field existed), or first_observed_genesis for chains opened by a genesis record. A first_observed_genesis chain only says that the first genesis record Lock saw for that chain opened it. It says nothing about earlier producer history, and Lock does not claim to hold a complete history of the chain.
Producer deployment patterns. Keys identify trusted producers, chains identify independently sequenced record streams, and versions describe software. They are different identifiers. How a producer holds its key and chain state varies:
| Pattern | Key and chain state |
|---|---|
| Embedded or server Cedarling | The host supplies the key and keeps durable chain state. |
| Sidecar Cedarling | The sidecar owns the key and one or more chain writers. |
| Ephemeral pod | Uses a fleet or workload-provisioned key and opens a first_observed_genesis chain for that pod or writer. |
| WASM Cedarling | Should return the assertion to a trusted host for signing. A key held in a browser gives weaker producer assurance. |
Restarting a producer does not by itself mean a new chain. A producer that durably retained its producer_chain_id, last sequence_number and previous record hash should continue its existing chain. A new chain is needed when that state is unavailable, when a new independently sequenced writer is created, or when the producer rotates deliberately. Lock only checks the records it receives. The signed chain_link member can reference the previous chain head. Lock checks its shape and that only a genesis record carries it. Lock does not validate the link against the prior chain.
List keys and chains#
curl -s "$API/admin/producer-keys?producer_id=cedarling-fleet-1" -H "Authorization: Bearer $TOKEN"
curl -s "$API/admin/producer-chains" -H "Authorization: Bearer $TOKEN"
Both return 200, with { "keys": [...] } and { "chains": [...] }. The producer_id filter is optional. Each chain entry carries evidence_domain_id, producer_id, producer_instance_id, producer_chain_id, registered_by, genesis_kind and created_at.
Submitting records#
Single record#
POST /jans-lock/api/v1/audit/trace
Authorization: Bearer <access token with trace.write>
Content-Type: application/json
The body is the complete signed assertion. For an AUTHORIZATION_DECISION it looks like this:
{
"producer": "cedarling-fleet-1",
"producer_version": "1.4.2",
"kid": "cedarling-fleet-1-2026-01",
"record_id": "9f3e9e2a-6b0e-4b2c-9f6e-3a2f7b0c9d41",
"trace": {
"eat_profile": "tag:jans.io,2026:trace-v1",
"event_kind": "AUTHORIZATION_DECISION",
"signed_at": 1781138542,
"trace_execution_id": "exec-01JABCXYZQK8P5N9F2C7R3T4V6",
"execution_authority": "spiffe://example.org/agent/planner",
"subject": {
"workload_id": "spiffe://example.org/agent/planner",
"workload_instance_id": "04bbe9ef-a853-417c-a2b8-5328b62936e2"
},
"event": {
"decision": {
"action": "Acme::Action::\"Pay\"",
"resource_type": "Acme::Payment",
"outcome": "ALLOW"
},
"tokens": [
{ "issuer": "https://accounts.example.org", "token_type": "access_token", "jti": "9c9f2e77-9a3e-4b62-9a2a-2f6e6e2a9f3e" }
]
},
"policy": {
"policy_store_id": "https://example.org/policy-stores/payments",
"policy_store_version": "1.2.3",
"policy_language": "cedar",
"policy_language_version": "4.4.0",
"bundle_hash": "sha256:<64 hex characters>"
},
"runtime": { "pdp_id": "cedarling-001" }
},
"producer_chain": {
"producer_id": "cedarling-fleet-1",
"producer_instance_id": "cedarling-001",
"producer_chain_id": "chain-01JABC9Z0K",
"sequence_number": 1,
"prev_record_hash": "sha256:0000000000000000000000000000000000000000000000000000000000000000"
},
"parent_record_ids": [],
"signature": "<base64url Ed25519 signature>"
}
Key points:
produceris the stable logical producer identifier. It must equalproducer_chain.producer_idbyte for byte.kidis part of the signed content.producer_version, when present, is a string of 1 to 255 characters. It is signed and stored unchanged, and Lock does not interpret it.record_idis a producer-generated UUID or ULID. Record identity is(evidence domain, producer, record_id), so two producers can use the samerecord_id.trace.policy.policy_store_id,policy_store_versionandbundle_hashare at most 255 characters each (length:<path>). Lock indexes them for search by policy store release.trace_execution_idandexecution_authorityare required and travel together. Lock correlates records by them. Lock neither creates them nor authorizes the execution. They are different fromsubject.workload_id: the authority scopes the execution identifier, and the workload is the subject that takes part in it.trace.subject.workload_idis required (non-empty string, at most 255 characters) forAUTHORIZATION_DECISIONandCAPABILITY_INVOKED.workload_instance_idis optional. ForRUNTIME_EFFECTthe subject is optional and is validated when present.- Producers sign capability facts, not a governance label. An
AUTHORIZATION_DECISIONcarries a singulardecisionand aCAPABILITY_INVOKEDcarries a singularinvocation. Each hasactionandresource_type(the Cedar action and resource type, never a resource id) and anoutcome. See Event kinds. Lock does not resolve capability ids. tokens[]holds references only:issuer,token_type, and eitherjtiorfingerprint. Never send a raw bearer token. Lock rejects material that looks like a JWT. Token identity is(domain, issuer, jti), never a barejti.- A producer should not sign a member that Lock derives or assigns itself:
evidence_domain_id,cnf,content_digest,receipt_sequence,received_at,prev_receipt_hash,coverage_gap_flag,chain_link_failure_flag,equivocation_flag,late_flag,policy_mismatch_flag,reserved_member_flag,unattributed_flag,candidate_flag,key_resolved,key_temporally_valid,key_not_revoked,key_authorized_for_producer,producer_authorized_for_claim,key_thumbprint,key_authorization_ref,verified_at,trust_tier,capability_resolution,capability_idandcapability_ids. Lock checks the top level,trace,trace.policy,trace.event, the singulardecisionandinvocation, and each entry ofdecisions[]andinvocations[]. It does not checkproducer_chain,parent_record_ids[],trace.subject,trace.runtimeortokens[], so a member with a reserved name there is not flagged. A record that carries one is not rejected: the member is ignored, Lock uses its own derived values, and the record is accepted withreserved_member_flag: true(GovOps profile §12). The member stays in the stored and returnedassertion, because the signature covers it; its value there is producer-supplied and ignored, and Lock's own values are iningestionandverification. The paths are stored and returned asreserved_members, for example["trace.trust_tier"]. Only the first 32 are kept, and the flag staystruewhen more were found. Lock logs the total count. Acnfmember never supplies, replaces or limits a key. Lock resolves the verification key only from the producer key registry, never from a key in the body. producer_chaincan carry an optionalchain_linkmember that references the previous chain head. It must be a JSON object, and only a genesis record (sequence_number1) can carry it. On any other record Lock returns400 invalid_genesiswith reasonchain_link_on_non_genesis. Lock checks thatprev_producer_chain_id, when present, is a non-empty string. It keeps the member unchanged inside the signed assertion, returns it unchanged on retrieval, and does not validate the link against the prior chain.- The first record of a chain has
sequence_number1 andprev_record_hashset tosha256:followed by 64 zeros. Any other record needs the previous record'scontent_digestinprev_record_hash.
Optional profile members#
Lock checks the JSON type of each optional member below when the producer signs it. A member of the wrong type returns 400 invalid_assertion with reason type:<path>, empty:<path>, length:<path> or invalid:<path>, in the same way as the required members. A null value counts as absent. Lock does not interpret the values and stores them only inside the signed assertion.
| Location | Member | Accepted shape |
|---|---|---|
trace |
execution_epoch_id, previous_epoch_id, parent_execution_id |
Non-empty string of at most 255 characters. |
trace |
operation_ids |
Array of non-empty strings. |
trace |
session_id, session_issuer |
Non-empty strings. session_issuer is required when session_id is present (missing:trace.session_issuer) and follows the token issuer rule (https, or http for localhost). |
trace |
evidence_origin, measurement_point |
Non-empty strings. Lock does not restrict the values. |
trace |
recorder |
Object or string. |
trace |
attestation_ref |
Object. |
trace |
workload_authentication |
Object. The members authentication_layer, method, authenticated_workload_id and proof_ref are strings when present. |
trace |
model |
Object. The members provider, model_id and version are strings when present. |
trace.subject |
human_sponsor, delegating_org |
String or object. |
trace.policy |
enforcement_mode |
enforce, advisory, silent or declared. Checked wherever Lock validates trace.policy: always for AUTHORIZATION_DECISION, and when present for CAPABILITY_INVOKED. |
Event kinds#
| Event kind | Producer | Required event data |
|---|---|---|
AUTHORIZATION_DECISION |
Cedarling or another PDP | Exactly one of a singular trace.event.decision {action, resource_type, outcome} or trace.event.decisions[]. A complete trace.policy block. trace.runtime.pdp_id. trace.subject.workload_id. |
CAPABILITY_INVOKED |
PEP, agent gateway or tool gateway | Exactly one of a singular trace.event.invocation {action, resource_type, outcome} or trace.event.invocations[]. trace.event.enforcement_point_id. trace.subject.workload_id. trace.policy is optional. When present, it must be complete, like a decision's. |
RUNTIME_EFFECT |
Target system or trusted observer | trace.event.effect_status (required): planned, dispatched, confirmed, failed or reverted. trace.event.outcome, required for every status except planned. trace.event.result_digest, required when the status is confirmed. It carries no decision, decisions, invocation, invocations or trace.policy. It normally has a produced_effect parent in parent_record_ids. Lock accepts it without one and notes a warning at debug level. |
Rules for RUNTIME_EFFECT:
effect_statusis lowercase and matches exactly. Lock stores nothing extra for it: there is no index attribute and no response member. The value stays in the signed assertion.outcomeis free text. It is optional forplanned. When present on aplannedrecord, it must still be a non-empty string.result_digestissha256:followed by 64 lowercase hex characters. It is required forconfirmedand optional for the other statuses. Lock checks the format from the record bytes alone. It does not compare the digest with a real result.- Lock rejects a violation with
400 invalid_assertionand one of these reasons:
| Reason | Cause |
|---|---|
missing:trace.event.effect_status |
The member is absent. |
type:trace.event.effect_status |
The value is not a string. |
invalid:trace.event.effect_status |
The value is not one of the five names. This includes a different letter case, for example CONFIRMED. |
missing:trace.event.outcome |
outcome is absent and the status is not planned. |
empty:trace.event.outcome |
outcome is an empty string. |
missing:trace.event.result_digest |
The status is confirmed and the digest is absent. |
invalid:trace.event.result_digest |
The digest is not in the sha256: form. |
A minimal trace.event for a confirmed effect:
"event": {
"effect_status": "confirmed",
"outcome": "payment_posted",
"result_digest": "sha256:<64 lowercase hex characters>"
}
Rules for the capability facts:
actionandresource_typeare non-empty strings of at most 512 characters.- For
AUTHORIZATION_DECISION,outcomeisALLOWorDENY. ForCAPABILITY_INVOKED,outcomeisSUCCESSorFAILURE. The vocabularies are distinct, so an invocation withALLOWor a decision withSUCCESSreturns400 invalid_assertionwith reasoninvalid:trace.event.invocation.outcome(orinvalid:trace.event.decision.outcome, or the indexed plural path such asinvalid:trace.event.invocations[0].outcome). - An atomic batch evaluation can use the plural form instead. Each entry carries a
decision_idorinvocation_id(non-empty, at most 256 characters, unique within the array) and repeatsaction,resource_typeandoutcome. The record-leveltrace.event.outcomeis not allowed in either form. - Operation-binding digests are optional members of
trace.event:request_digest,input_digestandtool_call_digestonAUTHORIZATION_DECISION, andrequest_digest,tool_call_digestandresult_digestonCAPABILITY_INVOKED(andresult_digestonRUNTIME_EFFECT, whereconfirmedrequires it). A present digest issha256:followed by 64 lowercase hex characters. Any other string returns400 invalid_assertionwith reasoninvalid:trace.event.<member>, and a non-string value returnstype:trace.event.<member>. Lock stores them in the signed assertion and does not interpret them.trace.policy.bundle_hashkeeps its own rule, which also accepts otheralgorithm:digestforms. - Both the singular and the plural member in one record, and a
resource_idinside a fact, are rejected with400 invalid_assertion(reason such asforbidden:trace.event.decisions). A signedcapability_idsis a Lock-derived member: it is ignored and flagged withreserved_member_flag.
Lock validates the schema for the declared kind and rejects fields that contradict it. Lock does not infer that an authorization caused an invocation because their timestamps are close. Producers state causal relationships in parent_record_ids.
Signature scope and digest scope#
The two scopes are intentionally different:
| Value | Computed over |
|---|---|
signature |
Ed25519 signature over the RFC 8785 (JCS) canonical form of the assertion without its top-level signature member. |
content_digest |
sha256: plus lowercase hex SHA-256 over the JCS canonical form of the complete assertion, including signature. |
content_digest is computed by Lock. It drives replay and conflict detection and is not a signature check.
Response#
{
"accepted": true,
"producer_id": "cedarling-fleet-1",
"record_id": "9f3e9e2a-6b0e-4b2c-9f6e-3a2f7b0c9d41",
"content_digest": "sha256:...",
"receipt_sequence": 108422,
"received_at": "2026-06-11T00:41:02.123Z",
"idempotent_replay": false,
"coverage_gap_flag": false,
"chain_link_failure_flag": false,
"equivocation_flag": false,
"late_flag": false,
"policy_mismatch_flag": false,
"reserved_member_flag": false,
"reserved_members": []
}
202 Accepted means the record was verified and durably stored. Lock verifies in a fixed order: OAuth check, domain derivation, JSON structure and limits, forwarding check, key resolution (validity window, then event-kind scope), signature verification, digest, then the idempotency, chain, and equivocation checks. A record that fails verification never enters storage or any retrieval index.
received_at is an RFC 3339 UTC timestamp with exactly three fractional digits and a Z suffix, for example 2026-06-11T00:41:02.123Z. The receipt and verification timestamps in API output (received_at, verified_at) use this form. Timestamps in the admin API (created_at, revoked_at) are RFC 3339 UTC and may omit a zero fraction.
| Condition | Result |
|---|---|
Same identity, same content_digest |
202 with the original receipt and idempotent_replay: true. No second receipt is created. |
Same identity, different content_digest |
409 record_conflict. The first record is never overwritten. |
| Missing predecessor in the chain | Accepted with coverage_gap_flag: true. |
Predecessor present but its digest does not match prev_record_hash |
Accepted with chain_link_failure_flag: true. |
Same chain position, different record_id |
Both are kept and flagged equivocation_flag: true. |
Bulk submission#
POST /jans-lock/api/v1/audit/trace/bulk takes a JSON array of assertions, each in the single-record shape, with the same trace.write scope. A high-throughput producer such as Cedarling can use it instead of one request per record.
- One producer per batch. Every element must carry the same signed
producer. A batch that mixes producers is rejected as a whole with400 invalid_requestand reasonbulk_mixed_producers, and nothing is ingested. Elements without a readableproducerare not counted and fail on their own. - Per record, not all or nothing. Every element is processed as an independent single submission, in array order. One element's failure never rejects the others. Replay, conflict, gap and equivocation behavior is per record and identical to the single-record path.
- Several chains in one batch. Records in one batch can advance several
producer_chain_idchains of the producer. Each chain is checked on its own. - Resubmitting a whole array is safe: records that were already committed replay idempotently.
A processed batch returns 207 Multi-Status, whatever the per-item results. Accepted entries carry the single-record acceptance members inline. Rejected entries carry the error:
{
"results": [
{
"index": 0, "accepted": true,
"producer_id": "cedarling-fleet-1", "record_id": "9f3e9e2a-6b0e-4b2c-9f6e-3a2f7b0c9d41",
"content_digest": "sha256:...", "receipt_sequence": 108422, "received_at": "2026-06-11T00:41:02.123Z",
"idempotent_replay": false, "coverage_gap_flag": false, "chain_link_failure_flag": false,
"equivocation_flag": false, "late_flag": false, "policy_mismatch_flag": false,
"reserved_member_flag": false, "reserved_members": []
},
{
"index": 1, "accepted": false,
"producer_id": "cedarling-fleet-1", "record_id": "1b2c3d4e-0000-4000-8000-000000000000",
"error": { "error": "invalid_signature", "error_description": "...", "reason": "signature_mismatch" }
}
]
}
index is the position in the submitted array, and results keeps request order. record_id and producer_id are present on a rejected entry when they could be read. The error ids are the ones in the error catalog.
Request-level failures return an ordinary structured error with a real HTTP status and no results: malformed JSON, a body that is not an array (not_array), an empty array (empty_bulk_records), more than maxBulkRecords elements (max_bulk_records_exceeded), a body over maxBulkRequestBytes (bulk_body_too_large), mixed producers (bulk_mixed_producers), TRACE disabled, and client_not_bound. Receipt sequences are allocated in array order but are not guaranteed to be contiguous within a batch, because a rejected item claims no receipt. With batch ingest enabled, the new records that a request stores in one successful batch transaction get consecutive sequence numbers. A replayed record keeps its original sequence number. When a request falls back to per-record ingest, its new records are not guaranteed consecutive numbers.
Error catalog#
Errors use the structured body {"error": "<id>", "error_description": "<text>", "reason": "<detail>"}. The reason member appears when error reasons are enabled and names the offending field or check. OAuth failures (401, 403 from the protection layer) come from Lock's standard filters and are not part of this table.
| Error id | HTTP | When |
|---|---|---|
invalid_request |
400 | Malformed JSON, a limit exceeded, an unreadable body, bad query parameters, TRACE disabled (trace_disabled), or a bulk envelope problem such as bulk_mixed_producers (see Bulk submission). |
invalid_assertion |
400 | A schema violation in the assertion. reason names the field. A signed_at ahead of Lock's clock by more than maxFutureSkewSeconds has reason signed_at_in_future. A RUNTIME_EFFECT without a valid effect_status has a reason from Event kinds. |
unsupported_profile |
400 | trace.eat_profile is not tag:jans.io,2026:trace-v1. |
producer_mismatch |
400 | producer does not equal producer_chain.producer_id. |
unknown_producer_key |
400 | No registered key for the domain, producer, and kid, or the record's event_kind is outside the key's authorized_event_kinds (reason event_kind_not_authorized). |
key_not_valid |
400 | The key is outside its validity window or revoked. |
invalid_signature |
400 | Ed25519 verification failed, or the signature is not valid base64url. |
chain_not_registered |
400 | The producer chain is not registered. |
invalid_genesis |
400 | The genesis position and prev_record_hash are inconsistent. |
client_not_bound |
403 | The client resolves to no evidence domain. |
producer_not_allowed |
403 | The client's binding does not allow the signed producer. |
record_not_found |
404 | No matching record in the domain. Also returned by the admin revoke call for an unknown key. |
execution_not_found |
404 | No records for the execution in the domain. |
record_conflict |
409 | The same identity exists with a different content_digest. |
ambiguous_identifier |
409 | A bare record_id or trace_execution_id matches more than one producer or authority in the domain. |
key_already_exists |
409 | Admin: duplicate key registration. |
chain_already_exists |
409 | Admin: duplicate chain registration. |
invalid_key |
400 | Admin: the JWK is not a 32-byte Ed25519 OKP key, the dates are invalid, or authorized_event_kinds is missing, empty, unknown or duplicated. |
storage_failure |
500 | Persistence failed. The reason names the case; see the table below this one. |
crypto_unavailable |
500 | The selected crypto provider (cryptoProvider) is not available or has no Ed25519 support. Lock fails closed and does not fall back to the other provider. |
Reasons of storage_failure that come from the receipt protocol (see Receipt chain):
| Reason | When | Stored? | Producer action |
|---|---|---|---|
receipt_allocation_exhausted |
No receipt position could be claimed within receiptAllocationRetryLimit tries (per-record path), or no receipt range within rangeAllocationRetryLimit tries (batch path; the batch then falls back to the per-record path, so a response shows this reason only when that path also runs out of tries). |
No. | Send again. |
receipt_allocation_interrupted |
The request thread was interrupted during the backoff between two receipt range attempts of the batch path (rangeAllocationBackoffMs above 0), for example while Lock stops. Lock fails closed: the elements without an outcome get this reason and are not retried on the per-record path. |
No. | Send again. |
batch_verification_abandoned |
Lock stopped while the element was still queued for verification on the batchVerifyThreads pool. |
No. | Send again. |
receipt_claim_expired |
The request passed its ingest deadline before the record insert. Lock settles the claim VOID. |
No. | Send again. The new request claims a new position. |
race_record_missing |
Another request won the identity race and Lock cannot read its record. Lock settles the claim VOID. |
No. | Send again. |
receipt_void_after_insert |
The record was stored after its claim became VOID. |
Yes. The record is kept. | Do not expect success from a resend. See Late insert after VOID. |
Canonical propagation wire profile#
This table is informational. It fixes one mapping for the base correlation fields so that independently written producers stay correlatable. Lock does not read these headers, baggage keys or token claims. Lock only reads the signed trace_execution_id, execution_authority and subject members of the assertion. A host, sidecar, PEP or gateway translates the carrier into the producer's input.
| TRACE field | HTTP header | OTel baggage key | Token claim |
|---|---|---|---|
trace_execution_id |
Audit-Trace-ID |
audit.trace_id |
https://jans.io/trace/execution_id |
execution_authority |
Audit-Execution-Authority |
audit.execution_authority |
https://jans.io/trace/execution_authority |
workload_id |
Audit-Workload-ID |
audit.workload_id |
https://jans.io/trace/workload_id |
workload_instance_id |
Audit-Workload-Instance-ID |
audit.workload_instance_id |
https://jans.io/trace/workload_instance_id |
trace_execution_idandexecution_authorityalways travel together.- Cedarling receives normalized values through its authorization API. It does not parse headers, tokens or baggage.
- Unsigned headers and baggage are correlation hints, not authoritative claims. A trusted ingress component should validate or replace externally supplied values instead of forwarding them.
- Signed token claims bind more strongly, but their issuer and signature must still be validated.
- When carriers disagree, the precedence is: validated token claim, then trusted ingress or host input, then unsigned header, then OTel baggage. The conflict should be recorded and not silently resolved.
The component that creates the governed unit of work (an agent runtime, a gateway, a PEP, an application host or a scheduler) creates the identifier pair and propagates it. Neither Cedarling nor Lock generates it.
Retrieval#
All retrieval endpoints need the trace.readonly scope and read only the caller's evidence domain.
# One record. producer_id should be supplied.
curl -s "$API/records/9f3e9e2a-6b0e-4b2c-9f6e-3a2f7b0c9d41?producer_id=cedarling-fleet-1" \
-H "Authorization: Bearer $READ_TOKEN"
# The direct records of one execution. execution_authority should be supplied.
curl -s "$API/executions/exec-01JABCXYZQK8P5N9F2C7R3T4V6?execution_authority=spiffe%3A%2F%2Fexample.org%2Fagent%2Fplanner&start=0&count=100" \
-H "Authorization: Bearer $READ_TOKEN"
A record response has three separate parts: the assertion (in a parsed form and as exact text), the verification result and the ingestion metadata.
{
"producer_id": "cedarling-fleet-1",
"record_id": "9f3e9e2a-6b0e-4b2c-9f6e-3a2f7b0c9d41",
"assertion": { "...": "the producer-signed record" },
"assertion_raw": "{\"...\": \"the producer-signed record, exact text\"}",
"content_digest": "sha256:...",
"verification": {
"signature_valid": true,
"key_id": "cedarling-fleet-1-2026-01",
"key_thumbprint": "sha256:...",
"verified_at": "2026-06-11T00:41:02.123Z",
"algorithm": "Ed25519",
"key_authorized": true
},
"ingestion": {
"receipt_sequence": 108422,
"received_at": "2026-06-11T00:41:02.123Z",
"prev_receipt_hash": "sha256:...",
"receipt_hash": "sha256:...",
"receipt_state": "COMMITTED",
"coverage_gap_flag": false,
"chain_link_failure_flag": false,
"equivocation_flag": false,
"late_flag": false,
"policy_mismatch_flag": false,
"reserved_member_flag": false,
"reserved_members": []
}
}
assertion_rawis the decoded JSON text of the accepted request. Lock decodes the body as UTF-8, rejects malformed UTF-8 and a leading byte-order mark, stores the text unchanged and returns it unchanged. It applies no normalization: whitespace, member order, number spelling, and the choice between a literal character and a\uXXXXescape stay as sent. Encodingassertion_rawas UTF-8 gives back the bytes of the request. For a record sent in a bulk request, the text is the source text of that array element. Lock never adds resolved claims, status, or receipt data to it. It is the source for archives and offline verification.assertionis the same text parsed as a JSON value. It is a re-emission, so member order, whitespace and number formatting can differ from the signed bytes. Use it when you want structure andassertion_rawwhen you need the original text. If the two differ,assertion_rawdecides.- Both forms verify. The signature and
content_digestare defined over the JCS (RFC 8785) form. The JCS form ofassertionand ofassertion_rawis identical because parsed values, member names and array order are preserved and because the parser enforces JCS-compatible input. - Number and Unicode rules that follow from this:
1e2,100.0and100are the same JCS number, and so are-0and0. Do not sign an integer above 2^53-1: Lock uses its exact digits, and a verifier that rounds to a double computes a different digest. A number that overflows a double (1e400),NaN,Infinity,+1,01,.5and1.are rejected. A literal character and its\uXXXXescape are the same value. Precomposed and combining forms of a character are different values: Lock does not normalize. A lone surrogate escape is rejected withinvalid_requestand reasonnot_canonicalizable. Duplicate member names are rejected, also when one is written as an escape. maxRequestBytesbounds the size ofassertion_raw. TheMEDIUMTEXTandTEXTstorage columns are larger.verificationrecords the admission decision Lock made atverified_at. It is written once at ingestion and never recomputed from the current key registry.key_thumbprintpins the exact key, independently of the non-uniquekidselector: it issha256:plus the lowercase hex SHA-256 of the RFC 7638 required-members JSON of the key ({"crv":"Ed25519","kty":"OKP","x":"<x>"}).key_authorizedis a single boolean. It states that the key existed, was inside its validity window, was not revoked, was registered for theproducer_idand was authorized for the record'sevent_kind. Every stored record haskey_authorized: true, because a record that fails any admission check is rejected and never stored. Later revocation, expiry or re-scoping of the key never rewrites these values. Records stored before these two members existed omit them.ingestionis Lock-derived metadata.ingestion.receipt_stateisCOMMITTED, orPENDINGfor the short time before Lock settles the entry. Lock reads it from the receipt entry when it builds the response. It is not stored on the record. A stored record never showsVOID, with one exception: a record stored by a late insert afterVOIDshowsVOID.
If producer_id (or execution_authority) is omitted and the bare identifier matches more than one producer (or authority) in the domain, Lock returns 409 ambiguous_identifier. It never guesses.
Search by policy store release#
GET /audit/trace/records (no record_id) returns the records whose trace.policy names a policy store release. Use it to answer "which decisions were made with release X".
curl -s "$API/records?policy_store_id=https%3A%2F%2Fexample.org%2Fpolicy-stores%2Fpayments&policy_store_version=1.2.3&received_from=2026-06-01T00:00:00Z&start=0&count=100" \
-H "Authorization: Bearer $READ_TOKEN"
| Parameter | Meaning |
|---|---|
policy_store_id |
Exact trace.policy.policy_store_id. |
policy_store_version |
Exact trace.policy.policy_store_version. |
bundle_hash |
Exact trace.policy.bundle_hash. |
received_from, received_to |
Inclusive bounds on received_at, RFC 3339. |
start, count |
Paging, as for executions. |
At least one of policy_store_id or bundle_hash is required; otherwise Lock returns 400 invalid_request with reason missing:policy_store_id. The response is {"ordering": "receipt_sequence", "records": [...]} with record envelopes in receipt order. No match returns 200 with an empty list. Only records that carry trace.policy (AUTHORIZATION_DECISION, and CAPABILITY_INVOKED when present) are indexed. Records stored before this index existed are not returned.
The execution response carries execution_authority, trace_execution_id, "ordering": "receipt_sequence", "completeness": "not_assessed", and records, ordered by Lock receipt sequence. start (default 0) and count (default 100, maximum 1000) page the list. The execution view contains the directly associated records only. It claims no global event order, does not traverse child executions, and does not assess completeness.
Flags explained#
Flags are Lock-derived findings about a record. Lock flags a condition instead of silently dropping or rejecting the record.
| Flag | Meaning | Cleared later? |
|---|---|---|
coverage_gap_flag |
The record's predecessor in its producer chain (sequence_number - 1) has not been seen. Records can arrive out of order. |
Yes. When the missing position arrives and links correctly, the flag on its successor clears. |
chain_link_failure_flag |
The predecessor is present but its content_digest does not match this record's prev_record_hash. A genuine break in the chain. When several records hold the predecessor position (equivocation), one matching content_digest is enough. Lock checks this record when it arrives and again each time a record arrives at the predecessor position, against all records present there at that time. Once set, the flag stays, so a matching predecessor that arrives after a nonmatching one does not clear it. |
No. The conflicting predecessor already exists, so later arrivals cannot resolve it. It stays visible as an integrity finding. |
equivocation_flag |
Two or more records with different record_id values claim the same chain position. Lock stores and flags all of them and never selects a winner. |
No. |
late_flag |
Lock received the record more than latenessThresholdSeconds after its signed_at. Fixed at ingestion. |
No. |
policy_mismatch_flag |
The record's trace.policy names a policy store Lock serves, with a release this node did not serve at signed_at or within policyMismatchGraceSeconds before it. Fixed at ingestion. See Policy store release check. |
No. |
reserved_member_flag |
The producer signed a member Lock derives itself, such as evidence_domain_id or trust_tier. Lock ignored it and used its own values; reserved_members lists the paths. Fixed at ingestion. |
No. |
Lock updates the flags on neighboring records after it stores a new record. These updates are best-effort and eventually consistent. If one fails, Lock logs it and still accepts the new record. A reader can see a flag change shortly after a record is accepted.
After the commit Lock reads the predecessor, own and successor positions of the new record once more. Lock computes the flags of a record from the chain as it was before the record was written, so a neighbor that another request or node stored at the same time can be missing from that computation. With the second read Lock corrects the new record's own coverage_gap_flag (cleared when the predecessor is now stored) and chain_link_failure_flag (set when the predecessor is now stored and does not match), and it corrects a successor that arrived while the record was being written. These corrections only clear a coverage gap and only set a chain link failure, never the other way round. A flag of the new record can therefore change within this post-commit window, after the response when postCommitCorrelationThreads is set. The response reports the flags the record was stored with, on both write paths.
A producer-chain break or gap is a statement about the records Lock has received. It is not a statement about what the producer did.
Policy store release check#
Lock serves policy store archives to producers (GET /policy, GET /policy/stores) and computes each archive's bundle_hash as sha256: plus the lowercase hex SHA-256 of the archive bytes. Lock reloads policy stores every 30 seconds. On every reload each node records the releases it serves in the shared TRACE store (jansTracePolicyRelease, one row per policy_store_id and bundle_hash, not scoped to an evidence domain): the first time a release is served it is inserted, and after that its last-served time moves forward at most once a minute. Updates use optimistic locking and never move the last-served time back or the first-served time forward, so a replica with a lagging clock cannot shorten a release's history. A release that is no longer served stops moving forward. Because the history is in the database, every replica sees the same releases, and it survives restarts and rolling updates.
When an accepted record's trace.policy.policy_store_id is a store this node serves as an archive, Lock judges the record's release at its received_at, or at its signed_at when the record is flagged late_flag. Using the receipt time means a producer cannot backdate signed_at to slip a retired release past the check; it can shift the time by at most latenessThresholdSeconds, and beyond that the record is flagged late. The release is accepted when:
- this node serves that
bundle_hashnow; or - its stored last-served time is within the acceptance window before that time. The window is the larger of
policyMismatchGraceSecondsandlatenessThresholdSeconds, plus 130 seconds: a stored last-served time can trail the real one by the one-minute update interval, two 30-second reloads and the 10-second cache. With the defaults the window is 430 seconds, so a record signed just before a rollout and received up to five minutes later still matches, and a release another replica still serves never looks retired.
Otherwise, and also when a matching release has a known policy_store_version that differs from the record's, the record is flagged policy_mismatch_flag. A record signed under the previous release during a rollout, or uploaded shortly after it, is not flagged. The record is always accepted.
Limits:
- Before the first policy store load, Lock serves no archive and flags nothing.
- A store this node does not serve is never compared, because producers may load stores from other sources.
- Lock caches found history rows for 10 seconds (at most 1000, least recently used evicted). A release it has not found is remembered as missing for one second, or until this node publishes a reload, so a release another replica starts serving is seen within a second and an unknown hash repeated by a producer costs at most one store read per second.
- Releases no node has served for
policyReleaseRetentionDaysare deleted by the receipt repair timer, 500 per run, except a release this node serves. Each delete only removes the row if it is still that old, so a release another replica refreshed in the meantime is kept, and replicas pruning at the same time do not interfere. A record later naming a deleted release is flagged. - The check is best-effort: if it or the store fails, Lock logs a warning and accepts the record without the flag.
The flag is a statement about what Lock served. It does not show that the producer used a tampered policy.
Receipt chain#
For every accepted record Lock assigns, within the evidence domain:
receipt_sequence: an increasing sequence number.received_at: Lock's receipt time.prev_receipt_hash: the hash of the preceding receipt entry.receipt_hash: the hash of this entry.
By default Lock writes the receipt entry and the record as two separate rows: it claims the position, inserts the record, then settles the entry. It is not one atomic write (see Deployment constraints). With batch ingest enabled, a request writes its receipt entries and records in one transaction instead, unless it falls back to per-record ingest.
Receipt states#
Every receipt entry has one of three states. The state is not part of the receipt hash.
| State | Meaning |
|---|---|
PENDING |
Lock claimed the position and has not settled it. Every per-record entry starts here. Batch entries are inserted as COMMITTED and never start here. |
COMMITTED |
The record that carries this entry is stored. |
VOID |
Lock allocated the position and abandoned it. Normally no record carries it. |
The only allowed transitions are PENDING to COMMITTED and PENDING to VOID. COMMITTED and VOID are terminal and never change. Applying the state an entry already has succeeds and changes nothing. Applying the other terminal state is a conflict and changes nothing. Each transition is a compare-and-swap on that one entry. On a version mismatch Lock re-reads the entry and applies the rules again, up to a fixed bound. If the retries run out, the entry stays PENDING and the repair timer settles it.
The retrieval response carries the state as ingestion.receipt_state (see Retrieval).
How Lock settles an entry#
A claim is satisfied if and only if the record stored under the claim's identity carries this receipt sequence and this content_digest. A record under the same identity with another receipt sequence belongs to another claim and does not satisfy it.
| Situation | Lock settles the entry as |
|---|---|
| The record insert succeeded. | COMMITTED. If this write fails, the entry stays PENDING and the repair timer settles it. |
| The insert reported a failure, but a re-read shows the record is stored under this claim (for example a timeout after the database committed). | COMMITTED. The request succeeds. |
| The insert failed and the re-read shows no record. | VOID. The request fails. |
| The insert failed and the re-read failed too. | Left PENDING. The request fails with the original failure. The repair timer settles it. |
| Another request already stored the same identity. | VOID. The answer comes from the stored record. |
The repair timer finds a PENDING entry older than pendingReceiptTimeoutSeconds (counted from received_at) and the claim is satisfied. |
COMMITTED. |
| The repair timer finds such an entry and the claim is not satisfied. | VOID. |
The repair timer never touches a PENDING entry younger than pendingReceiptTimeoutSeconds.
Ingest deadline. When Lock claims a position, it notes the time on its own clock. If its clock is past that time plus half of pendingReceiptTimeoutSeconds (60 seconds with the default of 120) just before the record insert, Lock does not insert. It settles the entry VOID and fails the request with 500 storage_failure and reason receipt_claim_expired. Nothing is stored. The producer can send the record again, and the new request claims a new position. The deadline is half of the timeout, so it stays below the age at which the repair timer may void a claim. The other half covers clock differences between nodes, a database round trip and the time between receipt of the request and the claim.
Late insert after VOID#
The deadline narrows the window in which a record is inserted after the repair timer voided its claim. It cannot close the window, because the database ORM has no multi-entry transaction. A worker that passes the check and then stalls until the repair timer voids its claim can still insert the record. In that case:
- Lock never deletes the record. Records are immutable: a retry may already have been told the record was accepted, and other nodes may have derived flags from it.
- The entry stays
VOID. The stored record carries a receipt that isVOID, and retrieval showsingestion.receipt_stateVOID. - Lock does not update the flags of neighboring records because of this record. It logs the case at error level.
- The request fails with
500 storage_failureand reasonreceipt_void_after_insert. - A resend of the same content does not succeed. Lock does not acknowledge a record with a
VOIDreceipt as a duplicate. It fails the same way. A resend with different content gets409 record_conflict. - The record identity stays blocked until an operator acts. An administrative operation that gives the record a new receipt position is not implemented (see Deferred features).
This case needs a worker that stalls past the deadline, so it is rare.
Receipt hash#
The receipt hash binds the receipt profile tag receipt_profile (tag:jans.io,2026:lock-trace-receipt-v1), evidence_domain_id, receipt_sequence, received_at, producer_id, record_id, content_digest, and prev_receipt_hash. The list is closed for this profile: no other member is hashed, and the receipt state is not hashed. The hash is sha256: followed by the lowercase hex SHA-256 of the UTF-8 bytes of the JCS canonical form of those eight members. receipt_profile is a string constant in every hash input. A different receipt_profile value means a different hash input, and a change to this member set needs a new profile tag. received_at enters the hash as a JSON string in the RFC 3339 form with three fractional digits and Z (2026-06-11T00:41:02.123Z), and receipt_sequence as a JSON integer. A value shown without milliseconds would not reproduce the receipt hash. The first entry in a domain uses the all-zero sentinel sha256:0000000000000000000000000000000000000000000000000000000000000000 as its prev_receipt_hash.
A writer can claim position n+1 while position n is still PENDING. The prev_receipt_hash of n+1 is the hash of entry n, whatever state n is in. Settling an entry never changes the hashes of the chain.
What a verifier can conclude#
From the receipt chain alone, a verifier can conclude:
- the hash of each entry, by recomputing it from the eight members;
- the linkage: each
prev_receipt_hashequals the hash of the entry before it; - the density: every sequence from 1 to the head has an entry; and
- that Lock allocated each position.
The chain alone does not prove the state of an entry. The state is outside the hash, so the hash does not prove that an entry is COMMITTED or VOID. A verifier needs more for these conclusions:
| Conclusion | What the verifier needs |
|---|---|
| A record exists. | An online lookup in the same evidence domain. |
| A record is authentic. | Check the record's signature and content_digest. |
| An authenticated statement of an entry's state. | A separately signed assertion. TRACE does not define one yet. |
A VOID entry keeps the chain continuous. It does not prove cryptographically that Lock accepted no record at that position.
A verifier must accept a VOID entry whose hash and prev_receipt_hash linkage verify and which has no stored record. It is a position that Lock allocated and then abandoned, not a missing entry. A verifier must not report it as a gap. With an online lookup, a verifier reports these as defects:
- a
COMMITTEDentry with no stored record, or whose record carries another receipt sequence or anothercontent_digest; and - a
VOIDentry whose identity has a record that carries this entry's receipt sequence andcontent_digest(a late insert).
A VOID entry whose identity has a record with another receipt sequence is not a defect. That record belongs to the claim that won the identity race.
What the receipt chain does not protect against
The receipt chain is local tamper evidence only. It has no signed checkpoints and no independent external anchor. It does not protect against an operator who can rewrite the whole chain, including every hash. Do not describe the receipt chain as proof of integrity against such an operator, and do not describe TRACE evidence as settled, complete, or externally anchored.
Producer order, Lock receipt order, and causal relationships are separate things. Producer order is what one producer signed in sequence. Receipt order is when Lock accepted records. parent_record_ids carries the causal relationships a producer asserted. None of them is a global event clock.
gRPC transport#
TRACE is also available over gRPC, next to the REST API. The gRPC methods call the same code as the REST endpoints, so validation, evidence domains, receipts, error ids and audit logging are identical. The definitions are in lock-server/service/src/main/proto/grpc/trace.proto.
gRPC is served in the mode set by grpcConfiguration.serverMode (bridge, plain_server or tls_server). The front proxy must route the two TRACE services, as it does for the audit service:
io.jans.lock.trace.TraceServiceio.jans.lock.trace.TraceAdminService
The Apache template of the VM installer and the all-in-one nginx configuration include them. In the Helm charts each service has its own route and its own flag, so Lock Audit, Lock Trace and Lock Trace Admin can be exposed independently:
| Service | Helm flag |
|---|---|
io.jans.lock.audit.AuditService |
gateway-api.routes.lockAuditEnabled |
io.jans.lock.trace.TraceService |
gateway-api.routes.lockTraceEnabled |
io.jans.lock.trace.TraceAdminService |
gateway-api.routes.lockTraceAdminEnabled |
All three require global.auth-server.lockEnabled: true. A custom proxy needs the same rules.
| gRPC method | REST equivalent | Scope |
|---|---|---|
TraceService/SubmitRecord |
POST /audit/trace |
trace.write |
TraceService/SubmitBulkRecords |
POST /audit/trace/bulk |
trace.write |
TraceService/GetRecord |
GET /audit/trace/records/{record_id} |
trace.readonly |
TraceService/SearchRecords |
GET /audit/trace/records |
trace.readonly |
TraceService/GetExecution |
GET /audit/trace/executions/{trace_execution_id} |
trace.readonly |
TraceAdminService/RegisterProducerKey |
POST /audit/trace/admin/producer-keys |
trace.admin |
TraceAdminService/ListProducerKeys |
GET /audit/trace/admin/producer-keys |
trace.admin |
TraceAdminService/RevokeProducerKey |
POST /audit/trace/admin/producer-keys/revoke |
trace.admin |
TraceAdminService/RegisterProducerChain |
POST /audit/trace/admin/producer-chains |
trace.admin |
TraceAdminService/ListProducerChains |
GET /audit/trace/admin/producer-chains |
trace.admin |
Send the access token in the authorization: Bearer <token> metadata entry. The scope and Cedarling policy checks are the ones of the matching REST endpoint.
- Assertions travel as bytes.
SubmitRecordRequest.assertionandSubmitBulkRecordsRequest.recordscarry the exact bytes the producer signed, and Lock hands them unchanged to the REST pipeline. Do not re-serialize the assertion on the client. - Retrieval.
RecordEnvelope.assertion_rawis the exact accepted text.RecordEnvelope.assertionis the same value re-emitted as JSON, as in REST. - Optional fields. An unset optional field behaves like an absent REST query parameter.
SearchRecordsRequestandGetExecutionRequestusestart0 andcount100 when unset. - Bulk results.
BulkResultEntry.flagsandreceipt_sequenceare set only for accepted entries, anderroronly for rejected ones. - Message size. Lock accepts inbound gRPC messages of up to
maxBulkRequestBytes(4 MiB by default, at most 16 MiB) plus 64 KiB of envelope. The server reads the value at startup, so restart Lock after you change it. The limit applies to every gRPC method, including the audit service. The servlet bridge (bridgemode) accepts at least 10 MiB. A bulk body overmaxBulkRequestBytesbut inside the transport limit fails withINVALID_ARGUMENT; a larger message is rejected by gRPC withRESOURCE_EXHAUSTED.
Errors use the gRPC status code that matches the HTTP status of the REST error. The status description is the structured JSON error body described in the error catalog.
| HTTP status | gRPC status |
|---|---|
| 400 | INVALID_ARGUMENT |
| 401 | UNAUTHENTICATED |
| 403 | PERMISSION_DENIED |
| 404 | NOT_FOUND |
| 409 | ALREADY_EXISTS, or FAILED_PRECONDITION for ambiguous_identifier |
| 500 | INTERNAL |
An unexpected server error returns INTERNAL with the description Internal error and no detail. A gRPC method that has no protected REST counterpart is rejected with PERMISSION_DENIED.
Example with grpcurl, run from the root of the Janssen repository (the -import-path folder holds trace.proto):
grpcurl -import-path jans-lock/lock-server/service/src/main/proto/grpc -proto trace.proto \
-H "authorization: Bearer $TOKEN" \
-d '{"record_id": "9f3e9e2a-6b0e-4b2c-9f6e-3a2f7b0c9d41"}' \
jans.example.org:443 io.jans.lock.trace.TraceService/GetRecord
Deployment constraints#
Multi-node deployments. The database ORM gives Lock two things for TRACE: per-entry compare-and-swap through PersistenceEntryManager.updateWithVersion (it throws VersionMismatchException; added in commit 393cd48bac9, PR #15255) and primary-key uniqueness on insert. It has no general multi-entry transaction and no sequence allocator. The SQL backend has an atomic batch insert, which only batch ingest uses. Lock does not target Couchbase or Spanner for TRACE, because they do not implement the compare-and-swap. The protocol below is the selected protocol under these persistence APIs. It is not the only possible protocol.
- A node reads the highest receipt, in any state, and proposes the next sequence number.
- It inserts a
PENDINGreceipt claim at that position. If another node claimed it first, the node re-reads and retries, up toreceiptAllocationRetryLimittimes (default 8), then fails with500 storage_failureand reasonreceipt_allocation_exhausted. - It stores the record, which carries the receipt sequence. Then it settles the claim as
COMMITTED. If storing fails, it settles the claim as described in Receipt chain.
A node that stops between steps leaves a PENDING claim. The repair timer settles claims older than pendingReceiptTimeoutSeconds: COMMITTED if the claim is satisfied, otherwise VOID. VOID entries keep the chain dense and verifiable, because the receipt state is not part of the receipt hash.
Storing the record and appending its receipt entry is not a single atomic write. This has these consequences:
- A failed request leaves no stored record and no index entry. It can leave a
VOIDreceipt entry at the allocated position. AVOIDentry stays in the chain permanently, because theprev_receipt_hashof the next entry binds its hash. - A
PENDINGentry is visible between the claim and its settlement. After a node stops, it stays until the repair timer settles it. That takes up topendingReceiptTimeoutSecondsplusreceiptRepairIntervalSeconds(defaults 120 and 300 seconds), as long as the backlog fits one repair run of 500 entries. - The one exception to "no stored record" is the late insert after
VOID: the record stays, showsreceipt_stateVOID, and the request fails.
Request size and column size. maxRequestBytes limits the size of one request body. The stored assertion is kept as text in a MEDIUMTEXT column on MySQL and a TEXT column on PostgreSQL. Keep maxRequestBytes and the per-record limit inside the bulk path well below the size of that column type for your database, and raise the limits only after checking it. Many TRACE identifiers are stored in VARCHAR columns of fixed width, which Lock's validation keeps the values within.
Upgrading. Batch ingest needs no schema change and no migration, and it is off by default. To enable it, follow the steps in Batch ingest.
Search by policy store release, policy_mismatch_flag and reserved_member_flag use columns added in this release. An existing VM install must add them once; see Upgrading an Existing Deployment.
Breaking change for RUNTIME_EFFECT producers
trace.event.effect_status is now required on every RUNTIME_EFFECT record. Lock has no tolerance for a missing value. A producer that sent only outcome gets 400 invalid_assertion with reason missing:trace.event.effect_status, and nothing is stored. Before you upgrade Lock, update each RUNTIME_EFFECT producer to sign one of planned, dispatched, confirmed, failed or reverted. A confirmed record also needs result_digest. Records that Lock stored earlier are not changed and stay retrievable.
Receipt entries now have terminal states and an ingest deadline. See Receipt chain. No schema change is needed for this.
Time. Lock uses received_at for receipt order and treats signed_at as a producer claim. Keep node clocks synchronized.
Batch ingest#
Batch ingest is an opt-in write mode for bulk requests and single submissions. It does not change the API: request and response shapes, status codes, error ids, reasons, receipt fields and flags are the same, and no schema change is needed.
What it changes. With batchIngestEnabled set to true, POST /audit/trace/bulk and the gRPC TraceService/SubmitBulkRecords method work like this:
- Lock verifies every element and plans its correlation (chain position, flags) without holding any lock. An element that fails is reported on its own, as before. A later element sees the earlier valid elements of the same request as already stored, so a chain of many records in one request gets the same flags as if the records had arrived one by one.
Within one bulk request Lock reads each producer key (producer, kid) from the database once and reuses it for the other elements that name the same key. The validity window (valid_from, valid_until, revocation) is still checked for every element, and every error id and reason is the same as without the reuse. Only a key that verified an element is reused: a missing, revoked, expired or not yet valid key, or a failed read, is looked up again by the next element that names it. Nothing is kept after the request ends, so a key revoked while a bulk request is being verified is seen by the next request.
2. Lock reserves the next N receipt positions of the evidence domain for the N records that are new, and writes the N receipt entries and the N records in one database transaction.
3. After the commit, Lock updates the derived flags of neighboring records as it does for a single submission.
Single submissions (POST /audit/trace, SubmitRecord) go through the same steps as a batch of one element: one transaction writes the receipt entry and the record, instead of the three commits of the per-record path. The response, the errors and the stored rows are the same as on the per-record path.
Thread pools. Two optional settings move work of the batch path off the request thread. Both apply only while batchIngestEnabled is true, both default to 0 (no pool, the request thread does the work), and both are clamped to 64.
batchVerifyThreads: the number of threads that verify the elements of one bulk request in parallel (parse, validation, producer key lookup, signature). The results keep request order, and an element that fails verification fails on its own as before. The threads share the producer key reads of the request (see step 1 above): concurrent elements that name the same key wait for one read, so the pool adds signature work, not key lookups.postCommitCorrelationThreads: the number of threads that update the derived flags of neighboring records (coverage_gap_flag,chain_link_failure_flag,equivocation_flagon records stored earlier) after a batch commit. The response no longer waits for these updates. A read right after the response can therefore show flags before the update, on a neighbor and, when a neighbor arrived from another request at the same time, on the submitted record itself (see Flags explained). Updates still queued when Lock stops get 10 seconds to finish; after that they are lost and Lock logs aWARNline with their number. Nothing repairs a lost update later, so leave this at0where the neighbor flags must be correct immediately.
Each pool queues up to 1024 tasks. When the queue is full, the request thread does the work itself, so a busy pool slows requests down instead of dropping work. A changed thread count takes effect with the next request.
Requirements. The store must support atomic batch inserts. The ORM provides this on the SQL backend (PostgreSQL and MySQL) only; see Atomic batch insert and multi-get by DN. On another backend, or when the selected crypto provider is not available, Lock keeps the per-record path for every request. Keep maxBulkRecords at 500 or below while batch ingest is on: one transaction writes a receipt entry and a record per element, and the ORM accepts at most 1000 rows in one batch. A larger request is not written as one transaction; Lock logs a WARN line with reason batch_too_large and ingests its elements one by one.
Receipt chain. Batch rows are inserted as COMMITTED in the same transaction as the records, so a batch never leaves a PENDING entry. The chain, the receipt hash input and the meaning of receipt_sequence are unchanged. A request that stores N new records in one successful batch transaction gets N consecutive sequence numbers in request order. A replayed record keeps its original number, and a rejected element gets none. When the request falls back to per-record ingest, other writers can take positions between its records. Another node can write entries between two requests, so the numbers of two requests need not be adjacent.
Multi-node. The protocol is still compare-and-swap on the unique receipt position. If another node took a position first, the transaction fails, nothing is stored, and Lock retries the request from a fresh head. A record identity that another request stored in the meantime with the same content is reported as a replay and dropped from the retry. If it was stored with different content, later elements of the request may have been planned against it, so the request falls back to per-record ingest, which reports that element as a record_conflict. After the commit Lock reads such a record again, so the replay reports its flags as they are after the request's own records were stored and correlated: the stored state at the time of the response. In this case Lock correlates the request's records in the request thread before the read, also when postCommitCorrelationThreads is set. Corrections that other requests still have queued on their own pools are not waited for. Both steps are best-effort. If the correlation fails, the replay shows the stored flags without its corrections. If the second read fails, the replay keeps the flags read at the collision, which can predate the corrections from this request's records. Nodes running the old code, nodes with the flag off and nodes with the flag on can write the same evidence domain at the same time, because all of them compete for the same unique position.
Three settings control the retries of the batch path. They apply only while batchIngestEnabled is true; the per-record path keeps receiptAllocationRetryLimit (default 8), retries without a wait and is not affected.
rangeAllocationRetryLimit(default32): the number of attempts, the first one included, before the batch gives up its range and falls back to per-record ingest. A value of0or less means the default. A batch has a larger budget than a single claim because it holds its positions for a whole transaction, so under load from several nodes it loses the race more often.rangeAllocationBackoffMs(default5) andrangeAllocationMaxBackoffMs(default250): before retry k (k = 1 for the first retry) Lock waits a random time between 0 andmin(rangeAllocationBackoffMs * 2^(k-1), rangeAllocationMaxBackoffMs)milliseconds. Nodes that collided then retry at different times instead of colliding again in lockstep. Lock releases the evidence domain's in-process lock during the wait, so other requests of the same node can commit, and reads the head and the stored identities again after it.0for the base disables the wait: the retry starts at once with the lock still held. A negative base, and a cap of0or less, mean the default. In the 4-node measurement the backoff cut the retries, and so the rolled-back database transactions, several times over, but it did not raise throughput and lowered the throughput of single submissions with few nodes. Consider0when latency matters more than the database load of the retries.
The total wait of one request is at most (rangeAllocationRetryLimit - 1) * rangeAllocationMaxBackoffMs. With the defaults the windows add up to 6565 ms, under 8 seconds, on top of the attempts themselves. If the thread is interrupted during the wait (for example while Lock stops), Lock stores nothing more for that range, keeps the interrupt and fails closed: the elements without an outcome get 500 storage_failure with reason receipt_allocation_interrupted, and the batch does not fall back to the per-record path. A WARN line TRACE batch ingest interrupted, failing pending elements without fallback: size=..., pending=..., reason=... records it.
Failure behavior. If the batch transaction fails for any reason other than a position or identity collision, or if the collisions use up rangeAllocationRetryLimit attempts, nothing was stored by that attempt. Lock logs one WARN line with the batch size, the number of unfinished elements and the failure type, then processes the unfinished elements one by one through the per-record path, together with the elements that were dropped from a retry because another request stored them. Each element then gets the result of the per-record path, including its own errors, so one failing row does not fail its neighbors.
Enable and roll back.
- Lock reads
batchIngestEnabledon every request. Lock checks for a changed dynamic configuration every 30 seconds, so a change takes effect within about that time and needs no restart. Each node reloads on its own timer, so for a short time different nodes can use different modes. - Roll back by setting
batchIngestEnabledtofalse. Rows written by the batch path are ordinaryCOMMITTEDreceipt entries and records. They need no cleanup, and a replay of a record written by either path is answered the same way. - Recommended order: upgrade all nodes, enable the flag on one node, compare the rate of
500 storage_failureresponses with reasonreceipt_allocation_exhaustedand the ingest throughput against the other nodes, then enable it everywhere.
Sizing. The figures below help choose the settings and the client concurrency. They were measured on 2026-11-14 (batch ingest on) and 2026-11-16 (batch ingest off) over HTTPS on one test VM with 8 vCPUs and 8 GB of RAM. Lock (Jetty 12, Java heap limit 634 MB, behind Apache) and PostgreSQL 16 ran on that VM, and the load client ran on another host. The load was signed AUTHORIZATION_DECISION records for one evidence domain, one producer per request, with batchIngestEnabled=true unless noted and every other setting at its default. Before each pool setting the domain was preloaded with 200 000 records and the database statistics were refreshed. Each row is two runs of 20 000 records; each session began with one warm-up run, not shown. These are measurements on one test VM, not guarantees: measure on your own hardware before you rely on them.
Bulk requests of 100 records, 8 producer chains:
batchVerifyThreads |
postCommitCorrelationThreads |
Client connections | Records/s | Request p50, ms | Request p95, ms | Records reported with coverage_gap_flag (of 20 000) |
|---|---|---|---|---|---|---|
0 |
0 |
1 | 2209-2218 | 44 | 49-50 | 0 |
0 |
0 |
4 | 6264-6862 | 55-57 | 75-104 | 0 |
0 |
0 |
8 | 7103-7161 | 109-110 | 120-130 | 1-2 |
0 |
0 |
16 | 5986-6250 | 241-243 | 321-357 | 189 |
4 |
2 |
1 | 4074-4142 | 23 | 28 | 0 |
4 |
2 |
4 | 6377-6805 | 54-58 | 67-75 | 0 |
4 |
2 |
8 | 6675-6722 | 112-116 | 132-162 | 3-10 |
4 |
2 |
16 | 6639-6767 | 233-238 | 245-250 | 190-191 |
8 |
4 |
1 | 4154-4224 | 22 | 27-28 | 0 |
8 |
4 |
4 | 6926-7175 | 55-56 | 62-67 | 0 |
8 |
4 |
8 | 6930-7016 | 109-110 | 131-141 | 4 |
8 |
4 |
16 | 6280-6733 | 234-237 | 248-411 | 189-190 |
Other request shapes with batchVerifyThreads=8, postCommitCorrelationThreads=4 and 4 client connections, measured in one later session. The single-submission row comes from an earlier session without the preload, with 4/2 and 8 connections:
| Requests | Producer chains | Records/s | Request p50, ms | Request p95, ms | Records reported with coverage_gap_flag (of 20 000) |
|---|---|---|---|---|---|
| Bulk, 100 records | 8 | 6274-6991 | 55 | 64-129 | 0 |
| Bulk, 50 records | 8 | 5939-6009 | 32-33 | 39-40 | 0 |
| Bulk, 100 records | 1 | 5768-6743 | 57-60 | 66-142 | 199 |
| Bulk, 100 records | 32 | 6179-6182 | 55-56 | 88-96 | 0 |
Single (POST /audit/trace) |
8 | 414-425 | 16 | 28-29 | 42-58 |
With batch ingest off (batchIngestEnabled=false, the per-record path), 8 producer chains, measured the same way on 2026-11-16. A control run with batch ingest on (8/4, 4 connections, bulk requests of 100) gave 6714-6852 records/s that day, so the two days are comparable:
| Requests | Client connections | Records/s | Request p50, ms | Request p95, ms | Records reported with coverage_gap_flag (of 20 000) |
|---|---|---|---|---|---|
| Bulk, 100 records | 1 | 161-165 | 605-618 | 638-676 | 0 |
| Bulk, 100 records | 4 | 372-373 | 1071-1072 | 1115 | 0 |
| Bulk, 100 records | 8 | 365-368 | 2145-2180 | 2278-2350 | 15-17 |
| Bulk, 100 records | 16 | 354-372 | 4283-4503 | 4505-4982 | 183-186 |
Single (POST /audit/trace) |
8 | 329-330 | 22 | 34 | 5-6 |
With batch ingest off the rate stops at about 370 records/s from 4 connections on, and more connections only add latency. The limit is the database's write-ahead log: every record needs about three commits, and each commit waits for a log flush (top waits LWLock:WALWrite and IO:WALSync). Lock used less than one core.
Choosing the settings:
- Batch ingest. Turn on
batchIngestEnabledfor producers that send bulk requests. On the test VM, bulk requests of 100 records were about 13 times faster with one connection and no pools (2209-2218 against 161-165 records/s), about 25 times faster with one connection and pools, and 16-20 times faster from 4 connections on (5986-7175 against 354-373 records/s), at a small fraction of the latency. Single submissions gained less: 414-425 against 329-330 records/s, measured on different days and, with batch ingest on, without the preload. - Request size. Send bulk requests of 100 records, the default
maxBulkRecords. Bulk requests of 50 records reached 85-96 % of the rate of bulk requests of 100 in the same session, and single submissions about 6-7 %. - Client connections. From 4 parallel connections per Lock node and evidence domain on, every setting reached about 6000-7200 records/s, and 4 connections gave the lowest latency at that rate. More connections raised the latency much more than the rate: about 235-245 ms p50 at 16 connections.
- Thread pools. The pools pay off when producers use only one or two connections. With one connection,
batchVerifyThreads=4andpostCommitCorrelationThreads=2nearly doubled the rate and halved the latency;8/4added little more. With 4 or more connections all three settings reached the same rate, so the pools are optional there.postCommitCorrelationThreadshas a cost: the flags of neighboring records are updated after the response, and updates still queued when Lock stops are lost (see Thread pools above).batchVerifyThreadswithoutpostCommitCorrelationThreadswas not measured. - Requests of one chain. Send the requests of one producer chain one after another. When several requests of one chain are in flight, for example with more connections than chains or with one chain over 4 connections, records arrive out of order. Their responses then report
coverage_gap_flag, as the last column shows; Lock corrects the stored flag after the missing record arrives. - Where the limit is. With batch ingest on, CPU was not the limit. With 4 or more connections Lock used 1.3-2.3 cores, PostgreSQL 0.4-0.7 core and the whole VM at most 3.3 of its 8 cores. The limit is the per-domain receipt lock of a Lock node. One batch at a time holds it while Lock reads the receipt head and writes the batch transaction, about 14 ms per batch of 100 at this rate, and the other requests for that domain wait. More CPU does not raise the rate of one domain.
- Going further. Each evidence domain has its own lock, so separate domains should add up, but this was not measured. Lock nodes that write the same evidence domain compete for its receipt positions and retry. In tests with several nodes on one domain, four nodes were not reliably faster than two. Several domains or several nodes over HTTP were not measured.
- Rule of thumb. On comparable hardware, plan for about 6000-7000 records per second per evidence domain and Lock node, with bulk requests of 100 and about 4 client connections.
- Logs. Lock also checks every request's access token with jans-auth. On the test VM the jans-auth request log grew by more than 10 GB during the 2026-11-14 runs and filled the disk; on 2026-11-16 it grew by about 7.6 GB for about 64 000 requests. Size the log disks and log levels of both services for the request rate.
Diagnosing. Set the Lock log level to DEBUG to see one line per request that uses the batch path (a single submission shows size=1):
TRACE batch ingest: size=100, accepted=98, replays=1, rejected=1, attempts=1, fallback=false
The line has counts only. size is the number of elements, attempts is the number of transactions that Lock tried for the range, and fallback=true means the per-record path finished the request. The values above are an example of the format, not a measurement.
attemptsabove 1 means another node (or a concurrent writer) took a receipt position or a record identity first. A high value in most requests is contention between nodes on the same evidence domain. If it limits your throughput, this is the signal that a different allocation scheme is needed. ADEBUGlinereceipt range contended: domain=..., firstSeq=..., size=..., attempt=..., backoffMs=...shows each lost attempt and the wait before the next one (0after the last attempt or when the backoff is disabled).- A
WARNlineTRACE batch ingest falls back to per-record ingestmeans the transaction failed for another reason. Read thereasonvalue, and the stack trace in theDEBUGline that follows, to find the cause. - To check on PostgreSQL that bulk requests commit less often, sample write transactions and WAL flushes before and after you enable the flag, with the same load.
txid_snapshot_xmax(txid_current_snapshot())advances once per write transaction.pg_stat_wal.wal_synccounts WAL flushes.pg_stat_activityshowsIO:WALSyncandLWLock:WALWritewait events while commits wait for the flush. Do not usepg_stat_database.xact_commitas a count of commits per record: it also counts every read-only statement that runs in its own transaction.
Deferred features#
The current release does not implement:
- a quarantine store for rejected records: rejected records are not stored;
- Lock-side management through the config-api Lock plugin and the Jans CLI/TUI;
- settlement assessments or Lock-signed settlement receipts;
- execution completeness or assurance profiles;
- lineage checkpoints,
chain_linkvalidation, or claims of complete producer history; - an administrative operation that gives a record with a
VOIDreceipt a new receipt position, so the identity of a late insert stays blocked until an operator acts; - signed receipt checkpoints or external transparency anchoring;
- Evidence Packet export;
- delegation non-expansion analysis and authorization-transition reconstruction;
- user-intent and approval records;
- operation-binding commitments, including proof-of-possession binding;
- capability resolution (deriving a
capability_idfrom the signed action and resource type) and a capability index. The signed facts stay in the stored assertion, so the index can be added later without re-emitting records; - the Producer-Key Authorization Statement and per-measurement-point key scoping (Lock enforces
authorized_event_kindsagainst its live registry); - the WIMSE agent identifier profile, credential metadata on
tokens[]and theworkload_authenticationrecord; - token-claim enrichment, current-state resolution, and token-exchange lineage;
- attestation assessment or trust tiers;
- cross-domain correlation or import;
- Lock emitting TRACE records for access to its own endpoints;
- graph traversal beyond direct execution membership.
Later features are meant to be additive. They may assess, correlate, checkpoint, or package an accepted assertion, but they never rewrite it.