Architecture overview

Kiwi Fabric is one core engine, KiwiFabricEngine, with three thin network adapters on top: REST, the PostgreSQL wire protocol, and Arrow Flight SQL over gRPC. All three call the same code for planning, execution, security and telemetry, so behaviour does not depend on which protocol a client chose.

The team's guiding idea is "conductor, not musician": the engine orchestrates, routes and manages state, while the actual read and write logic for each backend lives in a small, focused handler.

The read path#

SQL query
   |  parsed with Microsoft's ScriptDom T-SQL parser
   v
Query planner: resolves each table to its registered handler and builds
               a per-source extraction plan (pushed-down predicates, projected columns)
   v
Handler dispatcher: fans out to every required handler in parallel
   |-- relational handlers (SQL Server, PostgreSQL, Oracle, ...)
   |-- NoSQL handlers (MongoDB, Cassandra, Redis, ...)
   |-- REST / SOAP handlers
   '-- file, cloud and vector-store handlers
   v
DuckDB, the in-process finishing engine: receives each handler's result as an
        in-memory table and re-executes the original query for joins, aggregation and sorting
   v
Result: rows plus per-source telemetry (fetch time, rows returned, predicates pushed down)

Correctness over performance. Predicate pushdown only decides how much data a source sends back. DuckDB always re-applies the full WHERE clause, so a handler that ignores pushdown entirely still produces correct results.

The write path#

A saga is a list of steps with declared dependencies. The orchestrator sorts them topologically and runs independent steps in parallel waves. Each write handler returns an opaque snapshot token that is stored durably and contains everything needed to reverse that specific write. On a failure, every completed step is compensated in strict reverse order. Details are in Sagas and compensation.

Handlers#

Each registered capability resolves to a typed handler (for example a table handler or a stored-procedure read handler). A handler declares which pushdown operations it supports and, if it can write, which write modes it supports. Custom sources can be added by implementing the handler interface.

Relational drivers are loaded dynamically per data-source type, so you only ship support for the systems you use.

Embedding the engine#

The engine can also be used as a library for in-process access with no network hop:

var engine = new KiwiFabricEngine()
    .Register(new OrdersPostgresHandler(connStr))
    .Register(new CustomersMongoHandler(mongoClient));

var result = await engine.ExecuteAsync(
    "SELECT o.id, c.name FROM orders o JOIN customers c ON o.customer_id = c.id");

Write-capable handlers are registered separately (engine.RegisterWrite(...)), so the engine knows which sources may take part in sagas.

Deployment model#

One self-hosted process per customer (a silo): there is no shared connection pool or execution context between tenants. See Deployment and persistence.