Noria

Noria is a new streaming data-flow system designed to act as a fast storage backend for read-heavy web applications. Based on Jon Gjengset's PhD Thesis and presented at OSDI'18, Noria acts like a database but precomputes and caches relational query results for blazingly fast reads. It automatically keeps cached results up-to-date as the underlying data changes, using partially-stateful data-flow to reduce memory overhead and support dynamic, runtime data-flow, and query changes.

Key Features

  • High Performance: Precomputes and caches query results for fast reads.
  • Automatic Updates: Keeps cached results up-to-date with underlying data changes.
  • Flexible: Supports dynamic, runtime data-flow and query changes.
  • MySQL Compatibility: Includes a MySQL adapter for easy integration with existing applications.
  • Scalable: Designed to scale with the workload by distributing data and queries across nodes.

Running Noria

Noria follows a server-client model where many clients connect to a potentially distributed server. The server, noria-server, must be started before clients can connect. Noria uses Apache ZooKeeper for server location announcements, so ZooKeeper must be running.

Building Noria

To build noria-server, you need nightly Rust. This can be managed automatically using rustup. Install the necessary dependencies and build noria-server:

Dependencies

  • clang
  • libclang-dev
  • libssl-dev
  • liblz4-dev
  • build-essential

Build Command

cargo build --release --bin noria-server

Starting Noria

Ensure ZooKeeper is running and start noria-server:

cargo r --release --bin noria-server -- --deployment myapp --no-reuse --address 172.16.0.19 --shards 0

myapp is a deployment name, and multiple noria-server instances can share the workload in a single deployment.

Interacting with Noria

Rust Bindings

The noria crate provides native Rust bindings for interacting with noria-server. See the documentation for detailed usage instructions. Example programs can be found in the Noria repository.

MySQL Adapter

Noria includes a MySQL adapter that allows applications to use standard MySQL queries. Once the adapter is running, point your application to localhost:3306 to send queries to Noria.

CLI and Web UI

You can manually inspect data stored in Noria using any MySQL client (e.g., mysql CLI) or Noria's web interface. Noria also has an advanced web UI for a more comprehensive view of the data-flow graph and REST API endpoints.

Noria Development

Noria is a large project with many sub-crates and external tools. Key sub-crates include:

  • noria/: Contains everything needed for external programs to communicate with Noria.
  • applications/: Contains various Noria benchmarks.
  • server/src/: The Noria server, including data-flow operator execution, scheduling, and RPC handling.

Testing and Documentation

Run tests with:

cargo test

Build and open the documentation with:

cargo doc --open

Similar Projects

Rustfinity.com

Links

  1. Home
  2. Learn Rust
  3. Get Started
  4. Practice Rust
  5. Challenges
  6. Tutorials
  7. Blog
  8. Open source
  9. Learn Gleam

Socials

  1. GitHub
  2. X

Legal

  1. Privacy Policy
  2. Terms of Service