Cedarling TRACE#
With TRACE enabled, Cedarling signs a TRACE record for each authorization decision that carries a TRACE context and sends it to Lock Server. TRACE never changes the decision, and the authorization call never waits for Lock Server.
This page covers what Cedarling needs to emit records and how to find out why records do not arrive. The properties are listed in TRACE properties. How an application passes the TRACE context is described in Authorize options and TRACE context. What Lock Server does with the records is in Lock Server TRACE.
Requirements#
CEDARLING_LOCKset toenabled.- A
.cjarpolicy store: a local.cjarfile, a.cjarURL, Lock Server or a policy store URI. A directory or YAML policy store cannot be used, because each record carries the SHA-256 of the.cjarCedarling loaded. - A signing key:
CEDARLING_TRACE_SIGNING_KEYorCEDARLING_TRACE_SIGNING_KEY_PEM. - A native build. TRACE is not supported on WASM.
- Auth Server and Lock Server set up as below.
If a requirement is not met or a TRACE property is invalid, Cedarling does not start, and the error names the property.
Auth Server and Lock Server setup#
Scope. Lock Server accepts records only with an access token that has the https://jans.io/oauth/lock/trace.write scope; log.write is not enough. With TRACE enabled, Cedarling asks for this scope. When Cedarling registers through DCR (see Lock Server integration), add the scope to the software roles of the SSA and to the scope_list of the interception script that adds the Cedarling scopes. A token set with CEDARLING_LOCK_ACCESS_TOKEN_JWT must already carry the scope.
Producer key. At startup Cedarling logs its producer id, kid and key thumbprint. Register the key in Lock Server under them, see Register a key.
Lock Server traceConfiguration (see Lock Server TRACE configuration):
| Setting | Set it to | Without it |
|---|---|---|
enabled |
true |
no trace_bulk_endpoint is published, so no records are sent |
chainRegistrationMode |
FIRST_OBSERVED |
every record is refused with chain_not_registered |
defaultEvidenceDomainId |
a domain id, for example default |
every request is refused with HTTP 403 client_not_bound |
defaultAllowedProducerIds |
a list that contains CEDARLING_TRACE_PRODUCER_ID |
every record is refused with producer_not_allowed |
Cedarling starts a new producer chain at every start, and Lock Server registers it from its first record only in FIRST_OBSERVED mode. Cedarling also registers a new client through DCR at every start, so the default evidence domain and producer list are simpler than a per-client clientDomainBindings entry.
To see why Lock Server refused a record, set errorReasonEnabled: true in Lock Server and restart it. Cedarling then logs the reason with the error.
How records are sent#
- The authorization call puts the record in a queue and returns. The queue holds
CEDARLING_LOCK_LOG_CHANNEL_CAPACITYrecords. When it is full, for example while Lock Server is down, the record is not emitted (not_emitted_queue_full). - A background emitter sends what the queue holds as bulk requests, up to
CEDARLING_TRACE_BULK_MAX_RECORDSrecords andCEDARLING_TRACE_BULK_MAX_BYTESbytes each, as soon as it can.CEDARLING_LOCK_LOG_INTERVALdoes not apply. - A request that may succeed later (no answer, HTTP
5xx,429,408, or the matching gRPC status) is retried up toCEDARLING_LOCK_LOG_MAX_RETRIEStimes, waiting 10 seconds, doubling up to 5 minutes. Then its records are dropped. - Any other refusal is final: the records are dropped and logged at
ERROR. The message names the setting to change where there is one, see Troubleshooting. - While Lock Server keeps failing, the emitter waits between requests the same way, so Lock Server gets a request every few minutes rather than one per decision. Each wait is logged at
WARN. - On shut down, the emitter sends what it holds once, without retries. What Lock Server does not take is reported in one
WARNand counted intrace.dropped.
Troubleshooting#
| Symptom | Cause | What to change |
|---|---|---|
WARN "Lock Server does not publish trace_bulk_endpoint", status not_emitted_no_endpoint |
TRACE is disabled in Lock Server | enable traceConfiguration.enabled, or set CEDARLING_TRACE_ENDPOINT |
WARN that the endpoint "must use https", status not_emitted_no_endpoint |
the endpoint is plain http to a host other than localhost |
use https in CEDARLING_TRACE_ENDPOINT or in the Lock Server baseEndpoint |
HTTP 403 client_not_bound |
no evidence domain for the Cedarling client | defaultEvidenceDomainId |
HTTP 403 "Invalid token or insufficient scopes" |
the token has no trace.write scope |
add the scope, see Auth Server and Lock Server setup |
HTTP 401 after one renewal, or at once with CEDARLING_LOCK_ACCESS_TOKEN_JWT |
the token is invalid or expired; a token set by hand cannot be renewed | check the token or use DCR |
records refused with chain_not_registered |
Lock Server does not register chains from their first record | chainRegistrationMode: FIRST_OBSERVED |
records refused with producer_not_allowed |
the producer id is not allowed | add CEDARLING_TRACE_PRODUCER_ID to defaultAllowedProducerIds |
records refused with unknown_producer_key |
the key is not registered in Lock Server | register the key Cedarling logged at startup |
invalid_request max_bulk_records_exceeded or bulk_body_too_large |
a request is larger than Lock Server accepts | lower the setting the ERROR names: CEDARLING_TRACE_BULK_MAX_RECORDS or CEDARLING_TRACE_BULK_MAX_BYTES |
invalid_request without a reason |
Lock Server hides the reason | set errorReasonEnabled: true in Lock Server and read the next ERROR |
gRPC UNIMPLEMENTED |
Lock Server or its proxy has no TRACE gRPC route | update Lock Server and its proxy routes (see Lock Server gRPC transport), or set CEDARLING_TRACE_TRANSPORT to rest |
gRPC RESOURCE_EXHAUSTED |
a request over the gRPC message limit, or a server or proxy that slows Cedarling down | lower CEDARLING_TRACE_BULK_MAX_BYTES |
ERROR that Lock Server's answer cannot be read |
the answer is larger than CEDARLING_HTTP_MAX_RESPONSE_SIZE_BYTES |
raise CEDARLING_HTTP_MAX_RESPONSE_SIZE_BYTES. Lock Server has processed the records, so they are not sent again |
many not_emitted_queue_full |
Lock Server is down or slower than the decision rate | check Lock Server; raise CEDARLING_LOCK_LOG_CHANNEL_CAPACITY to ride out short outages |
| records refused with another error id | see the Lock Server error catalog |
Record status and metrics#
With TRACE enabled, every decision log entry has a trace_record_status field:
| Status | Meaning |
|---|---|
emitted |
The record was queued for the emitter. Whether Lock Server accepted it shows in the emitter logs and the trace.accepted metric. |
not_emitted_missing_context |
The call had no TRACE context, or a required context field was empty. |
not_emitted_over_limit |
A value is longer, or a list larger, than Lock Server accepts. Cedarling does not truncate signed values. The first one is logged at WARN with the field. |
not_emitted_invalid_field |
A value Lock Server rejects, such as a JWT issuer that is not https, or a .cjar whose metadata.json has no policy_store.id or policy_store.version. The first one is logged at WARN with the field. |
not_emitted_queue_full |
The queue to the emitter is full. |
not_emitted_stopped |
The emitter has stopped: Cedarling was shut down, or the emitter failed, which is logged at ERROR. |
not_emitted_no_endpoint |
There is no TRACE endpoint to send to. |
When TRACE is disabled, the field is absent.
With metrics enabled, operational_stats counts each status as trace.<status> (for example trace.emitted) and adds:
trace.accepted: records Lock Server accepted.trace.rejected: records Lock Server refused for good.trace.dropped: records given up: after their retries, on shut down, with a request that could not be sent (for example a redirect Cedarling refuses to follow), or with a producer chain Lock Server lost.trace.sign_failed: records that could not be signed.trace.chains_opened: producer chains started again because Lock Server lost or did not accept the previous one.
A counter appears only after it is first incremented.
Tokens in a record#
A record lists the tokens an authorize_multi_issuer decision used, by issuer, token type and jti, or a fingerprint of the token when it has no jti. A token reported in dropped_tokens is not listed. A decision that used more than 32 tokens gets no record (not_emitted_over_limit).
- A JWT is listed with its
iss. Lock Server accepts only anhttpsissuer, orhttpwith hostlocalhost, so a decision that uses a JWT with another issuer gets no record (not_emitted_invalid_field). - A token from a custom token processor is listed with the issuer
urn:jans:cedarling:custom-issuer:<issuer_id>, where<issuer_id>is the lowercasedissuerof its token type in the policy store.