Concept
Performance and resource limits
How Kiwi Fabric keeps cross-source queries fast, and the guards that stop runaway queries.
How queries stay fast#
Parallel fan-out. Every source involved in a query is fetched concurrently, not one after another.
Predicate and projection pushdown. Each handler receives only the filters and columns relevant to it, translated into that source's native query form where possible. Pushdown is a performance hint: the engine re-applies the whole WHERE clause itself, so results are the same whether or not a source pushes down.
Co-location batching. Tables that live in the same physical database are fetched together in one round trip using a join, instead of several separate fetches.
Semi-join reduction. When a query filters one source by keys derived from another, Kiwi Fabric pushes an IN (...) list to the second source in batches. This stops cross-source joins from silently degrading into "pull everything and join locally" as data grows. It is tunable per provider:
| Setting | Controls |
|---|---|
| Batch size | How many keys go into each pushed IN list |
| Maximum iterations | How many batches may be issued |
| Parallel batches | How many batches run at once |
| Fallback policy | If the key set is too large to push down: abort, or fall back to an unfiltered fetch |
Semi-join settings are managed in the Web UI and through the REST API (/api/admin/semijoin-config/{providerKey}).
Arrow Flight for analytical workloads. Large result sets stream as native Apache Arrow record batches with no JSON serialisation.
Resource guards#
Guards protect the engine and the source systems from runaway queries. They are enforced live and can be changed without a restart (except the REST rate limit):
| Guard | Effect |
|---|---|
| Query timeout | Aborts queries that run too long |
| Maximum result rows | Rejects a query or a single handler fetch that returns more rows. It is checked at each handler fetch and again on the final merged result, so one runaway table cannot exhaust memory before the overall cap is checked |
| Maximum concurrent queries | A global cap shared by REST, PostgreSQL wire and Arrow Flight |
| REST rate limit | Requests per minute on the REST API; read at start-up |
See Configure resource guards. When a guard rejects a request, the span in telemetry records which guard fired and why.
Sizing note#
Kiwi Fabric is a single-process engine suited to federating operational and mid-scale analytical systems with transactional write guarantees. It is not a replacement for a distributed lakehouse engine on very large analytical datasets.