Iceberg

Crunchy Bridge for Analytics has support for reading Iceberg tables. That means you can use PostgreSQL to query and import data from Iceberg tables in your S3 account, with data and metadata automatically cached on a local NVMe drive.

To add an existing Iceberg table to PostgreSQL, you can create a crunchy_lake_analytics table using the URL of the latest .metadata.json file ("snapshot") as the path. The URL can be obtained via an Iceberg catalog. When creating the table you do not need to specify columns, since they will be inferred from the Iceberg schema.

create foreign table orders ()
server crunchy_lake_analytics
options (path 's3://my-iceberg-bucket/se/orders/metadata/00001-c786274c-a887-4beb-9b33-48bf04c3f487.metadata.json');

Copy

If the metadata file name does not end in .metadata.json, you can also specify the format explicitly, e.g.

create foreign table orders ()
server crunchy_lake_analytics
options (path 's3://my-iceberg-bucket/se/orders/metadata/v1.json', format 'iceberg');

Copy

If you need to switch to a newer snapshot, you can either use ALTER FOREIGN TABLE to set the path option or drop & recreate the table if you also need to update the schema.

Finally, you can import Iceberg directly into PostgreSQL tables for further processing.

CREATE TEMP TABLE orders_tmp ()
WITH (load_from = 's3://my-iceberg-bucket/se/orders/metadata/00001-c786274c-a887-4beb-9b33-48bf04c3f487.metadata.json');

-- or
COPY orders_tmp FROM 's3://my-iceberg-bucket/se/orders/metadata/00001-c786274c-a887-4beb-9b33-48bf04c3f487.metadata.json' WITH (format 'iceberg');
Copy

With the initial Iceberg support in Bridge for Analytics you can easily build a dashboard that directly queries Iceberg, or materialized views that are created directly from Iceberg tables.