Connect to AWS Glue Catalog

​

Before you start

​

Cloud Connections

​

Connect to the AWS Glue catalog

1

Open the Add Amazon Glue Catalog wizard

In Wherobots Cloud, open Data Catalogs.Start adding a catalog using either the Data Catalogs explorer or the Add Catalog button:

• Data Catalogs explorer
• Add Catalog button

With no catalog selected, the details pane shows Add data cards. Select the Amazon Glue card.

Click Add Catalog, then under Foreign Catalog choose Amazon Glue Catalog.

Either method opens the Add Amazon Glue Catalog wizard.

2

Enter the catalog details

On the Details step, enter:

• Name: The name of the catalog in Wherobots.
  • This affects the Fully Qualified Name (FQN) used to reference this catalog in Wherobots: CATALOG_NAME.DATABASE_NAME.TABLE_NAME
  • A name can contain any character. But if it includes anything other than letters, numbers, or underscores — such as a space, dash, or period — you must wrap the name in backticks wherever you reference it in the FQN. For example, a catalog named My-Catalog is referenced as `My-Catalog`.database.table.
  For simplicity and to avoid quoting issues, it’s recommended to use only alphanumeric characters or underscores.
• AWS Region: The region your Glue catalog is in.
• S3 Path: The bucket (and optional prefix) that holds your Glue table data.
• Access level: The level of access Wherobots has to the catalog.
  • Choose Read-only to allow Wherobots to read tables in your Glue catalog, but not create or modify them.
  • Choose Read-write to allow Wherobots to read, create, and modify tables in your Glue catalog.
• Click Continue.

3

Choose a Cloud Connection

On the Connection step, pick an existing Cloud Connection or create a new one, then click Continue.

Create a new Cloud Connection

Creating a connection takes two steps inside the Create Cloud Connection dialog.

1

Enter the connection details

• Connection Name — a label that identifies the trust relationship in Wherobots.
• AWS Account ID — the 12-digit AWS account you’re connecting to.

Click Create Connection.

2

Grant access in AWS

Wherobots generates a pre-filled CloudFormation stack that creates the connection’s IAM role. Click Open in AWS Console to launch it in a new tab, or Download Template to run it yourself.

On the AWS Quick create stack page, everything is pre-filled. Scroll to the bottom, select I acknowledge that AWS CloudFormation might create IAM resources, and click Create stack.

Back in Wherobots, click Done to finish creating the connection.

4

Deploy the catalog stack

On the Deploy step, launch the pre-filled CloudFormation stack (named wherobots-glue-readonly-<name> or wherobots-glue-readwrite-<name>). It attaches Glue and S3 access to your Cloud Connection.

1. Click Open in AWS Console (or Download Template to manually paste it into the AWS CloudFormation console).
2. On the AWS Quick create stack page, select I acknowledge that AWS CloudFormation might create IAM resources, and click Create stack.
3. Return to Wherobots and click Create catalog.

5

Verify and finish

After the stack completes, Wherobots verifies it can access your Glue catalog. Once verified, the catalog appears in Data Catalogs and is ready to use.

IAM changes can take a few minutes to propagate across AWS. If verification fails immediately, wait a moment and retry.

​

Next Steps

​

Geometry and geography columns

CREATE TABLE GlueDemo.SKG.places (
  id BIGINT,
  name STRING,
  geom GEOMETRY(4326)
)
USING iceberg
LOCATION 's3://your-bucket/iceberg-warehouse/SKG/places';

• Set a table location. When the Glue database has no default location, a CREATE TABLE without LOCATION fails with Location information cannot be null while creating an iceberg table.
• Grant Lake Formation permission on the location. If AWS Lake Formation manages the location, the IAM role of the catalog’s Cloud Connection needs Lake Formation permission on it. Otherwise the create fails with Insufficient Lake Formation permission(s).
• The table stays at format-version 3. A table created or replaced with a GEOMETRY(<srid>) or GEOGRAPHY(<srid>) column is format-version 3 and can’t go back to format-version 2. Amazon Athena can’t read format-version 3 tables, and Amazon Redshift can’t read Iceberg V3 geometry columns. Every Wherobots runtime can read them.

• For notebooks, add it to the Spark configuration of an Environment Preset.
• For Job Runs, pass it in sparkConfigs when you create the run.

​

Known Limitations

• Glue REST API doesn’t support staged creates, i.e. no CREATE TABLE foo AS SELECT ...
  • Glue can’t create a new table from a query result. These all fail with Malformed request: Stage create is not supported:
    • CREATE TABLE ... AS SELECT (CTAS)
    • CREATE OR REPLACE TABLE ... AS SELECT, when the table doesn’t exist yet
    • The DataFrame equivalents .writeTo(...).create() and .writeTo(...).createOrReplace(), when the table doesn’t exist yet
    • df.write.mode("overwrite").saveAsTable(...), when the table doesn’t exist yet
    Setting an explicit LOCATION doesn’t help — the request is rejected before any data files are written.
  • Once the table exists, replacing it works normally: REPLACE TABLE ... AS SELECT, CREATE OR REPLACE TABLE ... AS SELECT, .replace(), and .createOrReplace() all succeed, including when the new query changes the schema. Use one of these when the query’s columns no longer line up with the table’s, since they rebuild the table; INSERT OVERWRITE below keeps the existing schema and fails if the columns don’t match.
  • Watch for a query that returns a geometry column with no declared CRS, such as one computed with an ST_ function. Glue stores that column as binary, so replacing a table with such a query turns its geometry column, including a GEOMETRY(<srid>) column, into a binary column that ST_ functions can’t read. If the table already has the same columns with that column as binary, the replacement can fail instead, with Malformed request: Cannot set last added schema: no schema has been added. To keep a geometry column, declare it as GEOMETRY(<srid>) in CREATE TABLE and load it with INSERT INTO, as described in Geometry and geography columns.
  • To create a table, declare the schema first, then load it with INSERT OVERWRITE:
    Create a Glue table in place of CTAS

    CREATE TABLE IF NOT EXISTS GlueDemo.SKG.edges_california (
      id BIGINT,
      name STRING
    )
    USING iceberg
    LOCATION 's3://your-bucket/iceberg-warehouse/SKG/edges_california';
    
    INSERT OVERWRITE GlueDemo.SKG.edges_california
    SELECT id, name FROM wherobots.spatial_kg.edges_california;
    
    Written this way the pair is safe to re-run: the create is a no-op once the table exists, and the overwrite replaces whatever an earlier attempt left behind. LOCATION is required. Without it, the create fails with Malformed request: Location information cannot be null while creating an iceberg table. For a worked example, see Query AWS Glue Catalog in a Notebook.
  • To replace the contents of a table that already exists, use INSERT OVERWRITE for an atomic swap:
    Replace the contents of an existing Glue table

    INSERT OVERWRITE GlueDemo.SKG.edges_california
    SELECT id, name FROM wherobots.spatial_kg.edges_california;
    
    The query’s columns must match the table’s schema. To change the schema, use REPLACE TABLE ... AS SELECT instead. This replaces the whole table under the default spark.sql.sources.partitionOverwriteMode of STATIC. On a partitioned table in a session that sets it to DYNAMIC, INSERT OVERWRITE replaces only the partitions the query produces and leaves the rest in place.
• Glue REST API doesn’t support reading or writing views
• Glue REST API doesn’t support ALTER TABLE … RENAME TO
• A new Glue table is format-version 2 unless it has a GEOMETRY(<srid>) or GEOGRAPHY(<srid>) column or sets 'format-version' = '3'. Format-version 3 tables with geometry, geography, or variant columns can be read and written, and merge-on-read DELETE, UPDATE, and MERGE write deletion vectors on them; see Geometry and geography columns. Tables with timestamp_ns columns can’t be read.