Crate sql[][src]

SQL-dataflow translation.

There are two main parts of the SQL–dataflow translation process:

Details

The purification step is, to our knowledge, unique to Materialize. In other SQL databases, there is no concept of purifying a statement before planning it. The reason for this difference is that in Materialize SQL statements can depend on external state: local files, Confluent Schema Registries, etc.

Presently only CREATE SOURCE statements can depend on external state, though this could change in the future. Consider, for example:

CREATE SOURCE ... FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY 'http://csr:8081'

The shape of the created source is dependent on the Avro schema that is stored in the schema registry running at csr:8081.

This is problematic, because we need planning to be a pure function of its input. Why?

Purification is the escape hatch. It is a transformation from SQL AST to SQL AST that "inlines" any external state. For example, we purify the schema above by fetching the schema from the schema registry and inlining it.

CREATE SOURCE ... FORMAT AVRO USING SCHEMA '{"name": "foo", "fields": [...]}'

Importantly, purification runs without access to the catalog. That means it can run in its own Tokio task so that it does not block any other SQL commands on the server.

Modules

ast

SQL abstract syntax tree.

catalog

Catalog abstraction layer.

func

TBD: Currently, sql::func handles matching arguments to their respective built-in functions (for most built-in functions, at least).

kafka_util

Provides parsing and convenience functions for working with Kafka from the sql package.

names

Structured name types for SQL objects.

normalize

SQL normalization routines.

parse

SQL parsing.

plan

SQL planning.

pure

SQL purification.