Architecture of Nuclia’s Hybrid AI Search Database: NucliaDB

Previously published on Nuclia.com. Nuclia is now Progress Agentic RAG.

NucliaDB is the open source distributed database engine designed for the Nuclia AI Search platform to efficiently scale to diverse workloads and datasets.

More than your typical vector database, NucliaDB is a robust, hybrid search database designed for diverse needs around unstructured data storage, indexing, retrieval and search and an important part of Nuclia’s full stack cloud offering for AI Search.

In this post, I will give you an overview of our architecture and how we scale NucliaDB at Nuclia.

Data storage architecture

NucliaDB is more than just a typical vector database. We need to support storage for a range of different types of data while also supporting multiple types of indexes.

To accomplish this, we built the data storage architecture around a 3-tiered data layer approach.

Tier 3: Blob Data

This is where we store files and extracted data. NucliaDB is compatible with both Amazon S3 and Google Cloud Storage (GCS) services.

Tier 2: Key-Value Storage

Our key-value storage layer is our primary resource, field and metadata storage layer.

All resource and field metadata are stored here.

For this, we utilize TiKV, a highly scalable, fault tolerant, Atomicity Consistency Isolation Durability (ACID) key-value database.

Tier 1: Index data

Our sharded index storage layer is our most complex storage layer.

Each shard is a package of our indexes that power our Hybrid Search.

Knowledge boxes, Index Data and Sharding

With Nuclia, the top-level data organization container is called a “Knowledge Box.”

A Knowledge Box is a collection of files, videos, audio and any other type of unstructured data that you want to be able to perform semantic searches on or get generative answers from.

Each Knowledge Box can then have many "shards" where its index data is managed.

Dynamic Sharding

From a data architecture perspective, we’ve built our architecture around scaling vector searches inside a Knowledge Box. Vector indexes can utilize a lot of disk and memory, so it is important to have an effective sharding strategy.

Additionally, to better dynamically scale our database, we wanted a sharding strategy that didn’t require knowing the correct size of the data ahead of time .

For example, in sharding strategies that databases like ElasticSearch use, you need to know the number of shards your dataset should use ahead of time. Strategies like this can work fine for time series data but do not work for other use cases.

Dynamic sharding allows us to add index data shards to a Knowledge Box as it grows. We manage the size of our shards mostly through controlling the number of vectors that are stored in the index. When we hit our threshold, we create a new shard.

Shard Storage and Index Node Architecture

The core part of our database is our Index Node component. An Index Node manages the disk, and reads and writes on that disk for a set of Shard Replicas. A Shard Replica is an instance of a shard.

As we want to be fault tolerant and dynamically scale reads, our shards can have many replicas. Each Shard Replica integrates four separate indexes into a unified, transportable replica storage object. Our replicas are then spread across many Index Nodes.

Cloud-Native + Standalone

We designed NucliaDB to be cloud-native and dynamically scalable.

NucliaDB is deployed as a service architecture. Search, reads, writes, indexing, processing and ingestion are all dynamically scalable.

We utilize NATS as our message bus and GRPC for service-to-service communication.

Standalone NucliaDB

NucliaDB Standalone is a packaging of NucliaDB with all batteries included and a single process to run. It allows customers to utilize the power of Nuclia on-premises, while still having complete ownership of their own data.

What is unique about NucliaDB is that while it is cloud-native, using Neural Autonomic Transport System (NATS) and Google Remote Procedure Call (gRPC) to communicate between systems to provide a robust scaling cloud system, NucliaDB can also be run in this simple standalone mode.

Python + Rust

We are able to deliver a standalone user experience partly due to how we can leverage Rust’s PyO3 python bindings integration. By using Python and Rust together, we are able to develop more quickly while keeping the important part of our database extremely fast.

Rust’s PyO3 bindings are an excellent way to glue Python and Rust together, and have allowed Nuclia to integrate service layers directly into Python. So, components that normally communicate over gRPC can be integrated through PyO3 bindings.

Putting It All Together

With a tiered data architecture and cloud-native service design, we are able to scale Nuclia’s cloud service.

Nuclia is more than just a database. It is a full stack solution with text/docs/video/audio processing, inference, semantic search, generative answers, REST API, Desktop App and a UI in a single solution.