Skip to main content
If you have some parquet files containing geometry data serialized in well known formats such as EWKB, WKB, WKT or GeoJSON, you can load these parquet files into a Havasu table without scanning these data files. We’ll use Google open buildings parquet files retrieved from here as an example. The file we gonna migrate looks like the this. You can see that there is a geometry column encoded in EWKB format.

Steps

Step 1: Create a external table for your data

We define geometry column as BINARY since it is a binary column in the parquet file. We’ll change it in-place to a geometry column after migrating it to Havasu.
The LOCATION clause specifies the path to the parquet files. Specify the path to a single file or a directory containing multiple files (globs are not permitted).

Step 2: Migrate the table

After migration, we can see that there is a Havasu table named org_catalog.test_db.open_building_aswan. The Havasu table was created using the original parquet files as data files.
However, geometry field is still a binary field. We’ll convert it to a geometry field in the next step.

Step 3: Set geometry fields

We can set the field storing geometry as a geometry field. This won’t rewrite the data files and will only affect how Havasu interpret configured fields.
Now the geometry field is geometry, and we can apply spatial functions provided by WherobotsDB to this column.

Step 4: (OPTIONAL) Create spatial index

We can run CREATE SPATIAL INDEX FOR statement to reorder the rows by spatial proximity to achieve better performance. We can optionally set a target file size for better query performance. Usually hundreds of megabytes per file is a good choice.
The rewritten files will be stored in the data path of the iceberg table, and it is usually not mixed up with your original data.

Step 5: (OPTIONAL) Allow expiring the original files

If everything is OK, we may want Havasu to take control over all the data files, including the original data file. If Havasu don’t need to use the original data file (for example, the entire table was rewritten), then we can allow Havasu to expire the original data file. We need to set some table properties to explicitly enable this.
If we don’t do this step, the Havasu table we migrated will be a snapshot table, and won’t allow us to expire snapshots, since it may accidentally delete your original parquet files.