> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wherobots.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Query AWS Glue Catalog in a Notebook
> Start a new runtime and query AWS Glue catalog tables from a Wherobots notebook.
After you connect your AWS Glue catalog in Data Hub, you can reference its tables from a notebook using the catalog name you chose.
## Before you start
The following requirements must be met before you can connect to a Glue catalog in a Wherobots Jupyter Notebook.
* An **Admin** or **User** Wherobots account within a **Professional**, **Innovation**, or **Enterprise** Organization Edition. For more information, see [Organization Editions](/get-started/organization-management/organization-editions).
Only Admins can *create* Glue catalog connections but a User can query the buckets bound to those Connections. If you are not an Admin, ask your Admin to create a Cloud Connection for you.
* An existing Glue catalog connection in Data Hub, or permission to create one. For more information on creating a Glue catalog connection, see [Connect to AWS Glue Catalog](/get-started/initial-storage/aws/connect-to-glue-catalog).
In Wherobots Cloud, you must start a **new** runtime after creating the catalog. Notebooks only see catalogs that existed when the runtime started.
A runtime only sees storage integrations that existed when it started, so a newer integration won't be available until you **destroy** the existing runtime and start a new one.
## Adapt this guide with AI
This section provides instructions for using AI tools to adapt this guide to your own environment.
The following prompts are designed for use with **Claude Code, Codex, or the Wherobots VS Code Extension** with the Wherobots MCP server enabled.
For optimal results, ensure that the Wherobots MCP server and/or VS Code extension are installed and active before execution. For more information, see [Get Started with Agentic Development in Wherobots](/develop/agentic-tools).
1. Click the v button at the top-right corner of this page (next to Copy page) to open the contextual menu for AI tools.
2. Load this page into your AI tool of choice (VS Code, Claude Code, Codex, etc.). This works for any Wherobots documentation page.
3. Open the dropdown to do either of the following:
* **Open in Claude Code** or **Open in Codex**: Start a conversation preloaded telling the model to read this page, [then add your own prompt](/get-started/initial-storage/aws/query-glue-catalog-in-a-notebook#notebook-generation-prompt) that includes a scenario or question specific to your environment.
### Example prompt
Success of the following prompts **requires** that you completed the [Connect to AWS Glue Catalog](/get-started/initial-storage/aws/connect-to-glue-catalog) guide and have a Glue catalog connection in [**Data Hub**](https://cloud.wherobots.com/data-hub).
Once the page is loaded into an LLM or AI tool, include your values in a prompt like the following to get a runnable notebook for your Glue catalog:
#### Notebook generation prompt
The following prompt can be used to generate a runnable notebook. Make sure to adapt this prompt's variables and goals to your own values.
```text wrap theme={"system"}
My variable values are the following:
`CATALOG_NAME` = ""
`DB_NAME` = ""
`TABLE_NAME` = ""
`SOURCE_FILE` = "s3:////.csv"
`WAREHOUSE` = "s3:////"
Using the Glue catalog example on this page, fill in the placeholder variables and return a complete `.ipynb` file that I can run in Wherobots Cloud.
The notebook should read the source file, , and write the result to a new Iceberg table in Glue.
```
## Example: Querying a Glue-backed Iceberg table in a notebook
This section details an end-to-end example of connecting, transforming, writing, and validating data in a Glue-backed Iceberg table from a notebook.
At a high level, the following example code does 3 main things:
* Configures a Sedona/Spark session to use AWS Glue as an Iceberg catalog, authenticating through your Wherobots Cloud Connection IAM role.
* Runs a small ETL flow on a Comma-separated values (CSV) file in S3: reading data, normalizing column names, trimming text, removing duplicates, and adding a synthetic foot traffic score plus ingestion metadata.
* Writes the result to an Iceberg table in Glue, then verifies the write by showing table metadata, file formats, and sample rows.
### Set your variables and create a Sedona session
The following code imports libraries that are used in the example, sets variables for your Glue catalog and S3 locations, and creates a Sedona session that can read and write Iceberg tables in Glue.
Swap the placeholder values for your own before running the cell.
The catalog name you chose when you connected Glue in Data Hub.
Click on the catalog you created in the [Connect to Glue](https://cloud.wherobots.com/data-hub) step. Copy the catalog name from the **Data Hub** interface.
The S3 path to the file you want to load.
`s3://BUCKET/SUBFOLDER/file.csv` would be an example of a valid path to a CSV file.
The database (namespace) you want to create.
This is a name you can choose at the time of running this notebook if the database doesn't already exist in your Glue catalog.
The table you want to create.
This is a name you can choose at the time of running this notebook if the table doesn't already exist in your Glue catalog.
```python wrap title="Set variables and create a Sedona session" theme={"system"}
import re
from sedona.spark import *
from pyspark.sql import functions as F
# ----------------------------------------------------------------------
# Customer configuration
# ----------------------------------------------------------------------
CATALOG_NAME = "YOUR_CATALOG_NAME" # Glue catalog name from Data Hub
DB_NAME = "YOUR_DB_NAME" # Database (namespace) name
TABLE_NAME = "YOUR_TABLE_NAME"
SOURCE_FILE = "s3://YOUR_BUCKET/YOUR_FILE.csv"
# S3 prefix where the Iceberg table data and metadata will be stored.
# This must be a prefix/folder, not the CSV file.
WAREHOUSE = "s3://YOUR_BUCKET/SUBDIRECTORY/"
OUT_TABLE = f"{CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}"
TABLE_LOCATION = f"{WAREHOUSE.rstrip('/')}/{DB_NAME}/{TABLE_NAME}/"
# ----------------------------------------------------------------------
# Create Sedona / Spark session and select the namespace
# ----------------------------------------------------------------------
# Your Glue catalog is registered by Wherobots when you connect it in Data
# Hub and start a new runtime, so no catalog config is needed here. Use the
# fully qualified CATALOG_NAME.DB_NAME so statements route to the Glue catalog.
config = SedonaContext.builder().getOrCreate()
sedona = SedonaContext.create(config)
sedona.sql(f"CREATE NAMESPACE IF NOT EXISTS {CATALOG_NAME}.{DB_NAME}")
sedona.sql(f"USE {CATALOG_NAME}.{DB_NAME}")
```
Once you've connected your Glue catalog in Data Hub, Wherobots handles the Iceberg, authentication, and warehouse configuration for you. A standard `SedonaContext` can query the catalog like any other Wherobots catalog — reference its tables as `CATALOG_NAME.DATABASE_NAME.TABLE_NAME`.
### Basic ETL and synthetic scoring
The following code reads the source file, cleans it, adds a couple of derived columns, and writes the result to a new Iceberg table in your Glue catalog.
1. **Infers the file format** from the `SOURCE_FILE` extension, then reads the file as CSV with a header row and an inferred schema.
2. **Normalizes column names** with `normalize_col_name` (lowercase, non-alphanumeric characters replaced with `_`) and `make_unique_col_names` (adds a numeric suffix when two columns normalize to the same name) so the names are safe for Iceberg and Glue.
3. **Cleans the rows** by trimming whitespace from string columns and dropping exact duplicate rows.
4. **Adds a `synthetic_foot_traffic_score`** — a deterministic value from 1 to 100 derived from a hash of each row's contents. This is placeholder analytics; replace it with your own logic.
5. **Adds lineage columns** `_source_file` and `_ingested_at` so each row records where it came from and when it was loaded.
6. **Creates the namespace** if it doesn't already exist, then **writes the Iceberg table** with `createOrReplace`.
7. **Verifies the result** by printing the table definition, describing the table, checking the physical data file format, and selecting a few rows.
```python wrap title="Normalize, clean, and score the source data" theme={"system"}
# --------------------------------------------
# Read source CSV
# --------------------------------------------
raw_df = (
sedona.read
.option("header", True)
.option("inferSchema", True)
.csv(SOURCE_FILE)
)
# --------------------------------------------
# Basic ETL: normalize column names, trim text,
# and drop duplicates
# --------------------------------------------
def normalize_col_name(name: str) -> str:
cleaned = re.sub(r"[^A-Za-z0-9_]", "_", name.strip().lower())
cleaned = re.sub(r"_+", "_", cleaned).strip("_")
return cleaned or "col"
def make_unique_col_names(cols):
seen = {}
output = []
for col in cols:
base = normalize_col_name(col)
count = seen.get(base, 0)
if count == 0:
output.append(base)
else:
output.append(f"{base}_{count + 1}")
seen[base] = count + 1
return output
renamed_df = raw_df.toDF(*make_unique_col_names(raw_df.columns))
string_cols = {
col_name
for col_name, dtype_name in renamed_df.dtypes
if dtype_name == "string"
}
clean_df = (
renamed_df
.select(
*[
F.trim(F.col(c)).alias(c) if c in string_cols else F.col(c)
for c in renamed_df.columns
]
)
.dropDuplicates()
)
# --------------------------------------------
# Add synthetic foot traffic score
# --------------------------------------------
score_cols = [
F.coalesce(F.col(c).cast("string"), F.lit(""))
for c in clean_df.columns
]
scored_df = (
clean_df
.withColumn(
"synthetic_foot_traffic_score",
(
F.pmod(F.xxhash64(*score_cols), F.lit(100))
+ F.lit(1)
).cast("int")
)
.withColumn("_source_file", F.lit(SOURCE_FILE))
.withColumn("_ingested_at", F.current_timestamp())
)
# --------------------------------------------
# Create Iceberg table, then insert data
# --------------------------------------------
def quote_identifier(name: str) -> str:
return f"`{name.replace('`', '``')}`"
columns_sql = ",\n ".join(
f"{quote_identifier(field.name)} {field.dataType.simpleString()}"
for field in scored_df.schema.fields
)
select_sql = ",\n ".join(
quote_identifier(field.name)
for field in scored_df.schema.fields
)
scored_df.createOrReplaceTempView("scored_source_data")
sedona.sql(f"DROP TABLE IF EXISTS {OUT_TABLE}")
sedona.sql(f"""
CREATE TABLE {OUT_TABLE} (
{columns_sql}
)
USING iceberg
LOCATION '{TABLE_LOCATION}'
TBLPROPERTIES (
'format-version' = '2'
)
""")
sedona.sql(f"""
INSERT INTO {OUT_TABLE}
SELECT
{select_sql}
FROM scored_source_data
""")
# --------------------------------------------
# Verify table
# --------------------------------------------
sedona.sql(f"SELECT * FROM {OUT_TABLE} LIMIT 10").show(truncate=False)
sedona.sql(f"""
SELECT
file_format,
COUNT(*) AS file_count
FROM {OUT_TABLE}.files
GROUP BY file_format
""").show(truncate=False)
```
### Full example code
You can copy the full example code below. Swap the placeholder values for your own before running the cell. For more information on the variables, see [Set your variables and create a Sedona session](#set-your-variables-and-create-a-sedona-session).
```python wrap title="Set variables and create a Sedona session" theme={"system"}
import re
from sedona.spark import *
from pyspark.sql import functions as F
# ---------------------------------------
# Customer configuration
# ---------------------------------------
CATALOG_NAME = "YOUR_CATALOG_NAME" # Glue catalog name from Data Hub
DB_NAME = "YOUR_DB_NAME" # Database (namespace) name
TABLE_NAME = "YOUR_TABLE_NAME"
SOURCE_FILE = "s3://YOUR_BUCKET/YOUR_FILE.csv"
# S3 prefix where the Iceberg table data and metadata will be stored.
# This must be a prefix/folder, not the CSV file.
WAREHOUSE = "s3://YOUR_BUCKET/SUBDIRECTORY/"
OUT_TABLE = f"{CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}"
TABLE_LOCATION = f"{WAREHOUSE.rstrip('/')}/{DB_NAME}/{TABLE_NAME}/"
# --------------------------------------
# Create Sedona / Spark session and
# select the namespace
# --------------------------------------
# Your Glue catalog is registered by Wherobots when you connect it in Data
# Hub and start a new runtime, so no catalog config is needed here. Use the
# fully qualified CATALOG_NAME.DB_NAME so statements route to the Glue catalog.
config = SedonaContext.builder().getOrCreate()
sedona = SedonaContext.create(config)
sedona.sql(f"CREATE NAMESPACE IF NOT EXISTS {CATALOG_NAME}.{DB_NAME}")
sedona.sql(f"USE {CATALOG_NAME}.{DB_NAME}")
# --------------------------------------------
# Read source CSV
# --------------------------------------------
raw_df = (
sedona.read
.option("header", True)
.option("inferSchema", True)
.csv(SOURCE_FILE)
)
# --------------------------------------------
# Basic ETL: normalize column names, trim text,
# and drop duplicates
# --------------------------------------------
def normalize_col_name(name: str) -> str:
cleaned = re.sub(r"[^A-Za-z0-9_]", "_", name.strip().lower())
cleaned = re.sub(r"_+", "_", cleaned).strip("_")
return cleaned or "col"
def make_unique_col_names(cols):
seen = {}
output = []
for col in cols:
base = normalize_col_name(col)
count = seen.get(base, 0)
if count == 0:
output.append(base)
else:
output.append(f"{base}_{count + 1}")
seen[base] = count + 1
return output
renamed_df = raw_df.toDF(*make_unique_col_names(raw_df.columns))
string_cols = {
col_name
for col_name, dtype_name in renamed_df.dtypes
if dtype_name == "string"
}
clean_df = (
renamed_df
.select(
*[
F.trim(F.col(c)).alias(c) if c in string_cols else F.col(c)
for c in renamed_df.columns
]
)
.dropDuplicates()
)
# --------------------------------------------
# Add synthetic foot traffic score
# --------------------------------------------
score_cols = [
F.coalesce(F.col(c).cast("string"), F.lit(""))
for c in clean_df.columns
]
scored_df = (
clean_df
.withColumn(
"synthetic_foot_traffic_score",
(
F.pmod(F.xxhash64(*score_cols), F.lit(100))
+ F.lit(1)
).cast("int")
)
.withColumn("_source_file", F.lit(SOURCE_FILE))
.withColumn("_ingested_at", F.current_timestamp())
)
# --------------------------------------------
# Create Iceberg table, then insert data
# --------------------------------------------
def quote_identifier(name: str) -> str:
return f"`{name.replace('`', '``')}`"
columns_sql = ",\n ".join(
f"{quote_identifier(field.name)} {field.dataType.simpleString()}"
for field in scored_df.schema.fields
)
select_sql = ",\n ".join(
quote_identifier(field.name)
for field in scored_df.schema.fields
)
scored_df.createOrReplaceTempView("scored_source_data")
sedona.sql(f"DROP TABLE IF EXISTS {OUT_TABLE}")
sedona.sql(f"""
CREATE TABLE {OUT_TABLE} (
{columns_sql}
)
USING iceberg
LOCATION '{TABLE_LOCATION}'
TBLPROPERTIES (
'format-version' = '2'
)
""")
sedona.sql(f"""
INSERT INTO {OUT_TABLE}
SELECT
{select_sql}
FROM scored_source_data
""")
# --------------------------------------------
# Verify table
# --------------------------------------------
sedona.sql(f"SELECT * FROM {OUT_TABLE} LIMIT 10").show(truncate=False)
sedona.sql(f"""
SELECT
file_format,
COUNT(*) AS file_count
FROM {OUT_TABLE}.files
GROUP BY file_format
""").show(truncate=False)
```