Docs
Kiwi Distro
Version Current

Troubleshooting

Common problems

TLS, registration, routing, slow responses and settings problems in Kiwi Distro, with causes and fixes.

Distro Current 3 min read Updated

Note

Kiwi Distro is currently in private preview. Interfaces and behaviour described here can change before general availability. Contact Kiwify Labs for access.

A call returns 503#

Cause: There is no capable worker for the service, or the backing store timed out.

Fix: Check that a worker for that service is registered and healthy (kiwi distro list-workers --service-id <id>). If you used tag filters or a version pin, check that a worker matches them. A worker that stops sending heartbeats is eventually considered dead.

A call returns 403#

Cause: Authorisation denied it. The response deliberately contains no policy detail.

Fix: Check the caller-side grants (get-client-allowed-services, get-client-groups) for a client, or the target's allowed callers and method restrictions (get-allowed-callers, get-method-restrictions) for a worker-originated call. See Control access.

TLS errors when a worker or client connects#

  • Stale CA cache. If the middleware's CA was rotated, delete the cached CA files (~/.kiwi/ca/*.pem, and %LOCALAPPDATA%\kiwi\ca\*.pem for the CLI) and retry. After removing the cluster's database volumes the CA changes too.
  • Python 3.13 or newer fails validation. An older middleware build issues certificates without an authority key identifier, which strict validation rejects. Use a middleware built with the Vault PKI fix.
  • For a quick test only, ignore_ssl=true skips verification. Never use it in production.

A worker never receives requests#

  • Eager mode: the load balancer's idle or read timeout must be longer than long_poll_wait (default 25 s), or the hanging poll is cut off.
  • Lazy mode: the middleware must be able to reach the worker's registered ip:port (the wake-up probe port). Check firewalls and the address the worker advertised.
  • Confirm the method is registered by the worker (methods in registration) and that its name matches the call (the worker sees the bare method name).

InvalidOperationException reading settings#

Application settings (including service secrets) exist only after the worker registers. Reading them in a constructor, at import time or in the start-up initialiser throws. Read them inside the handler, or in .NET await IWorkerApplicationSettings.WhenReady.

Responses are fast, then suddenly 10x slower#

This is the queueing cliff: above a concurrency about equal to the number of registered workers for a service, requests wait for a free worker. Add workers to match peak in-flight requests. See Capacity planning.

Targeting a specific worker does not guarantee that worker answers#

For requests that do not need a lock, target = <worker-id> is not exclusive: another worker of the same service may answer first and win the result. The targeted worker still receives and processes the request, so check its log and kiwi distro list-feedback --worker-id <id>.

A script will not run#

  • The version must be authorised (authorize-script-version); new uploads are pending.
  • A worker must advertise the script's executor, platform, and any required interactive desktop or browser runtime. See Register and run scripts.
  • Script execution is not yet available in the Node.js, Python and Java worker SDKs; use a .NET worker.

A SideCar upload is rejected with 400#

The mapping document must be valid JSON with a non-empty version, a mappings[] array, and a unique source.method_name per mapping using only letters, digits and underscore.

A cache adapter is missing at start-up#

Client, service-group and client-group caches have in-memory and Redis implementations only. A middleware configured with Memcached for these fails fast at start-up. Switch to Redis or in-memory.