Identity Reconciler

The declarative control plane that keeps Keycloak, the Canton participant, and Kubernetes converged on one reviewed identity manifest — continuously, with verified writes and per-principal drift reporting.

Why it exists

On Canton, a token proves only identity. Authority lives in the Canton user record — a party plus actAs/readAs rights keyed by the token's sub — and the correctness of the whole system depends on three stores agreeing: Keycloak (clients, service accounts), the participant (parties, users, rights), and Kubernetes (secrets, endpoints). Any of the three can change independently: environments get wiped, Keycloak reimports and every sub changes, participants restart.

The incident that motivated it. Our rights matrix originally lived inside ~500 lines of one-shot provisioning shell. Nine of eleven agents ran for 14 hours missing a single readAs grant — roughly 560 errors/minute, and no alert, because the intended state existed nowhere a machine could check. The reconciler replaces that model entirely: identity is declared as data and converged every cycle.

The model

Desired state lives in one manifest per environment (k8s/<env>/identity.yaml, schema medici.loan/identity-v1alpha1), delivered to the reconciler as a Flux-tracked ConfigMap. The reconciler converges it every 60 seconds, or immediately when the manifest changes.

apiVersion: medici.loan/identity-v1alpha1
env: dev
parties:
  - alias: operator
  - alias: price-feed
    policy: read-only          # lint: may never appear in any actAs
  - alias: settlement-reader
    hint: medici-settlement-reader   # optional — party-ID hint sent to Canton;
    policy: read-only                # falls back to alias when omitted
    displayName: Settlement Reader (observer-only)
principals:
  - name: oracle-agent
    kind: service-agent        # one of 5 kinds
    client: { id: oracle-svc, secretRef: oracle-svc-client-secret }
    party:  { alias: oracle, allocate: true }
    rights:
      actAs:  [oracle]
      readAs: [oracle, price-feed]

Each parties[] entry has an alias (the manifest-internal key other sections reference — never sent to Canton) and three optional fields: hint (the party-ID hint sent to Canton on allocation; falls back to alias when omitted, so this is backward compatible), displayName (an operator-facing label, informational only), and policy: read-only (see Lint rules below). Manifest loading is strict — any field with no matching schema entry is a load error naming the field, not a silent drop.

Each cycle: ensure Keycloak clients and service accounts (capturing each service account's sub) → ensure Canton party → user (keyed by that sub) → rights → verify. Every step is idempotent; a principal that fails is recorded and the cycle continues; the retry mechanism is the next cycle. The binary defaults to dry-run — live mutation is an explicit deployment argument.

Verified writes

Canton offers no usable rights read-back (the endpoint is masked as security-sensitive), returns grant deltas that are legitimately empty on re-grant, can reject inside HTTP 200 bodies, and accepts grants on non-existent parties. The reconciler compensates:

Quirk	Compensation
Grant on unknown party "succeeds"	Every alias is resolved to its full party fingerprint before any grant
Errors inside HTTP 200 bodies	Every response body is parsed for embedded Canton errors before being trusted
Delta-only grant responses	Non-empty deltas are checked against the manifest's expected rights set
No rights read-back	Agents independently hard-fail at boot via `POST /api/v1/auth/verify`; synthetic per-principal probes (mint a real token, one benign read) are the designed end-state

Lint rules

Five rules run on every cycle, and the live manifest is loaded and linted in CI:

A policy: read-only party may never appear in any actAs (protects the public price-feed party).
readAsAnyParty requires an explicit allowlist.
Every alias referenced by a principal must be declared under parties.
Every service-agent must declare both a client and a party.
A party's effective hint (its declared hint, or its own alias when none is set) must not collide with another party's effective hint — two entries resolving to the same Canton party-ID hint would allocate, or silently share, one party.

Drift and status

Each cycle updates per-principal drift state — drift means "this principal could not be converged (or probed) in the last cycle" — and writes the identity-reconciler-status ConfigMap:

kubectl get configmap identity-reconciler-status -n medici-dev-app \
  -o jsonpath='{.data.status\.json}' | jq .
# { "lastCycle": "...", "created": 0, "updated": 0,
#   "planned": [], "errors": [],
#   "principals": [ { "name": "oracle-agent", "converged": true, ... } ] }

Alert rules cover sustained drift (IdentityDrift), probe failure rates, and stalled reconciliation (no completed cycle in 10 minutes). A wiped dev environment reconverges to all-green in roughly 4–6 minutes with no manual steps — that wipe test was the cutover acceptance gate.

What it owns — and doesn't

Owned (converged every cycle)	Not owned
Keycloak clients, service accounts, bootstrap users, the `canton_party` user-profile declaration; Canton parties, users, rights grants; the status ConfigMap; derived client secrets and their rotation (manifest `rotate:` field)	The bootstrap Keycloak realm (Flux/Helm-owned); the `medici-party-config` party-ID cache (deploy scripts own it — reconciler reads only); dynamic per-user principals (onboarding service); SOPS trust anchors; external CIP-103 parties (key material, not manifest state)

Day-2 operations

Add a principal / change rights — edit k8s/<env>/identity.yaml, open a PR, push. Flux updates the ConfigMap; the reconciler converges on its next cycle. Never grant rights by hand.
Check convergence — read the status ConfigMap (above) or the reconciler's reconcile cycle complete … errors:0 log line. Trust drift state, not "the job ran".
After an environment wipe — do nothing. Verify all-green within ~6 minutes.

Don't provision out-of-band. Manual grants, hand-created Keycloak clients, and edited rights are exactly the drift the reconciler exists to eliminate — they will either be converged away or sit outside the reviewed manifest where no alert covers them. The manifest is the only interface.

Medici Protocol

Identity Reconciler

Why it exists

The model

Verified writes

Lint rules

Drift and status

What it owns — and doesn't

Day-2 operations

Further reading