When Keyring is down

The SDK verifies from a policy set it keeps in memory. When the control plane is unreachable, that set goes stale, and the question is what a request does then. onUnavailable answers it, globally or per route.

maxStalenessMs is 60 seconds by default.

What the default admits, exactly

The cache is complete for the project, not a partial memo, so a key absent from a fresh complete snapshot is a key that does not exist, and that miss is a 401 in every mode. The fail-open branch is reached only when the store cannot prove the miss, and what it admits then is a function of the store's state:

The key format's checksum is CRC-32, a public algorithm, so producing a well-formed key costs an attacker nothing. Read the last four rows before "Auth-failed" as the radius: in those states, on the default mode, an unknown key is admitted with tenant: null, no scopes and no scope check, until the next successful poll. The window is not a fixed length: it stays open for as long as the outage lasts, and closes on the first successful poll after recovery.

A 401 on the policy poll is not one more state — it overrides the others

Every row above is built for a control plane that is slow or unreachable. A 401 on the policy poll says the opposite: this node's own credential no longer authenticates, most often because GitHub secret scanning revoked a leaked krsk_live_ key and it was this deployment's own KEYRING_SECRET_KEY, and a refused credential is not evidence that any other key is fine to admit. So from the first poll that comes back 401 until a poll succeeds, a miss is refused (401; 503 under closed, as before) regardless of maxStalenessMs and of onUnavailable, whichever row the store would otherwise be in. A poll that fails any other way (a timeout, a 5xx) leaves the state as it is. This is fail static, not refuse-all: a hit still verifies from the store exactly as it always does, which also means a revocation issued after the first refused poll does not reach this node until a poll succeeds. Rotating the secret key is what ends it.

A cold node's boot refusal on a 400/403/404 enters the same state. 401 is not the only status a boot-time misconfiguration produces: a krsk_test_ key naming env: 'live' gets 403, a wrong KEYRING_PROJECT_ID gets 404, and a malformed request gets 400. A 403 never clears on its own. A 404 or 400 usually does not either -- but can also be the status a refused tenant widening produces (a tenant_id this workspace does not own fails the whole snapshot the same way), and that cause is not a misconfiguration: the SDK's own rollback drops the offending id and the very next poll succeeds, with no operator action. The status code alone cannot tell the two apart, which is why PolicyBootRefusedError's message names both rather than asserting one. Before this, a cold node -- one that has never loaded a snapshot -- left in "Never loaded" on one of these statuses admitted any well-formed key for as long as the condition lasted, which for a genuine misconfiguration is the life of the deploy. So the same override now applies, scoped to the cold refusal only: a warm node's ordinary failed poll on one of these statuses is unchanged, because it has a known key to fall back on and a control plane that starts refusing everything after having served correctly is a different failure from a boot that never authenticated at all. 429 and every 5xx are excluded on purpose -- they are transient, and a cold node keeps admitting any well-formed key while the control plane is merely slow or unreachable, same as before. onError receives a PolicyBootRefusedError rather than a PolicyAuthFailedError for this cause, because the likely reason and the fix differ; stats().policyCache.authFailed and authFailedRefusingAll read the same either way.

Two cases sit outside the state. A node that starts or restarts is in its own row above until its first poll comes back: with no disk snapshot, or with one older than maxStalenessMs, it admits any well-formed key for that first round trip. And a node built with store has no poller, so it never enters the state.

stats().policyCache.authFailed reports the state, distinct from ordinary staleness. onError receives a PolicyAuthFailedError or a PolicyBootRefusedError at most once a minute: on the refused poll that enters the state, unless one already went out in the minute before (so a credential flapping between 401 and success does not re-announce every time), and again while the state continues. That is on top of the raw PolicyRequestError every failed poll already reports.

A node that has never loaded a snapshot at all and gets a 401, or one of these 400/403/404 statuses, refuses every key, because it has no known key to fall back on; drain it the way you drain scopeTooLarge:

app.get('/readyz', (_req, res) => {
  const { authFailedRefusingAll, scopeTooLarge } =
    keyring.stats().policyCache ?? {};
  res
    .status(
      authFailedRefusingAll === true || scopeTooLarge === true ? 503 : 200,
    )
    .end();
});

This does not change the rate limiter or idempotency: both authenticate with the same credential, so a revoked key 401s them too, but each keeps its own existing onUnavailable behaviour for that failure (see below).

What to set

• Money routes: closed. A route declaring scopes is deliberately not scope-checked on a fail-open decision, so scopes alone protects nothing there.
• Internal routes that must never admit a stranger: stale-then-closed. Known keys keep working through an outage; unknown ones do not.
• Serverless, or any fresh-process-per-request platform: stale-then-closed on the client, because every invocation starts in the "never loaded" row.
• A long-lived server with mostly-read routes: the default, which is what makes an outage of the control plane invisible to your customers.

req.keyring.degraded is true on every decision that came from stale or incomplete policy, and verified is false when there was no policy record at all, so a handler can be stricter than its route.

The counters and the idempotency store

These are separate questions with separate defaults: rate limits fail open, idempotency fails closed. Rate limits and Idempotency explain the asymmetry.