Bring Your Own Spark (AWS EMR)

Bring Your Own Spark (BYOS) lets you run WherobotsDB on a Spark cluster you own and operate. WherobotsDB is a drop-in replacement for Apache Sedona: you install its JAR on your existing cluster, and your existing code, SQL, and configuration continue to work unchanged. Your data never leaves your AWS account. Wherobots does not run, manage, or connect to your cluster — the only outbound traffic is a lightweight authenticated call from the JAR to Wherobots Cloud for license metering. This guide covers Amazon EMR. Only EMR on EC2 deployments are supported; EMR Serverless and EMR on EKS are not. It is validated on EMR 7.8.0 (Spark 3.5.4, Scala 2.12, Amazon Corretto JDK 17), and every command below targets that release. WherobotsDB is also validated on emr-spark-8.0.0 — see Version compatibility for what changes.

BYOS is distinct from Bring Your Own Cloud (BYOC). Under BYOC, Wherobots runs its own managed compute inside your AWS account. Under BYOS, you run your own Spark cluster and Wherobots supplies only the engine.For Databricks clusters, see Bring Your Own Spark (Databricks) instead.

For existing Apache Sedona users

If you already run Apache Sedona on Amazon EMR, moving to WherobotsDB takes two changes:

Replace the JAR. Swap the public Apache Sedona JAR from Maven Central for the WherobotsDB EMR JAR that Wherobots supplies.
Old JAR New JAR
sedona-spark-shaded-*.jar sedona-emr-spark-shaded-*.jar
The WherobotsDB build adds the enterprise functions, Wherobots Cloud authentication, and the metering client.
Add your Wherobots API key. See Give Secret Manager Access and Create cluster.

Old JAR	New JAR
`sedona-spark-shaded-*.jar`	`sedona-emr-spark-shaded-*.jar`

Everything else in this guide — the IAM roles, bucket layout, and EMR Studio setup — is standard EMR configuration you may already have in place.

Before you start

The following requirements must be met within both your Wherobots Organization and AWS account before you begin:

Wherobots Requirements

An Admin account within a Professional, Innovation, or Enterprise Edition Organization.
Wherobots Organization members with the User role can use a cluster an Admin has already set up, but cannot create the API key this deployment requires. See Organization Roles.
Community Edition is not supported. See Organization Editions or Upgrade Organization.
The BYOS feature enabled on your Organization. Contact Wherobots to enable it. BYOS unlocks both halves of this deployment: read access to the WherobotsDB release JARs, and the runtime authentication and usage-reporting endpoints the JAR calls. Without it, those endpoints return 403 and cluster initialization fails.
A Wherobots API key. You create this in Give Secret Manager Access.

AWS Requirements

An AWS account.
Permission to create IAM roles, instance profiles, and inline policies in that account. This guide creates three roles — an EMR service role, an EC2 instance role, and an EMR Studio service role — plus one instance profile. AdministratorAccess is sufficient but broader than required. For the specific actions involved, see Actions defined by AWS Identity and Access Management and Actions, resources, and condition keys for Amazon EMR in the AWS documentation.
Permission to create an S3 bucket, an EMR cluster, EC2 security groups, an EMR Studio, and a Secrets Manager secret.
iam:PassRole on the three roles this guide creates. create-cluster passes both the service role and the instance profile, and create-studio passes the Studio service role, so an identity without iam:PassRole on WherobotsEMRServiceRole, WherobotsEMRInstanceRole, and WherobotsEMRStudioServiceRole receives AccessDenied before either resource is created. This is a permission on the operator running these commands, distinct from the iam:PassRole the service role itself is granted in Create Service Role.
An Amazon EMR 7.x or later release. This guide targets emr-7.8.0. EMR 6.x is not supported — see Version compatibility.
A VPC and subnet for the cluster. The EMR Studio you create later must use the same subnet as the cluster.
Outbound HTTPS (port 443) from the cluster to Wherobots Cloud, for license metering. A cluster in a private subnet needs a NAT gateway or an equivalent egress path; without it, SedonaContext.create fails to authenticate.
The WherobotsDB release bundle from Wherobots, containing the shaded JARs, the geotools-wrapper JAR, and the wherobots-sedona Python wheels. These are not published to Maven Central or PyPI.

Once BYOS is enabled for your Organization, a read-only WherobotsDB Releases storage source appears automatically in your Wherobots Cloud storage list. Go to Storage, open that source, and download the JARs and wheel from the release folder. If the source does not appear, BYOS is not yet enabled — contact Wherobots.

Version compatibility

WherobotsDB JARs are compiled against JDK 17, so Amazon EMR 7.x or later is required. Choose the JAR that matches your EMR release:

EMR release	Spark	Scala	JDK	PySpark Python	JAR artifact
`emr-7.8.0` (this guide)	3.5	2.12	17 (Corretto)	`python3` (3.9)	`sedona-emr-spark-shaded-3.5_2.12-VERSION.jar`
Other 7.x releases	3.5	2.12	17 (Corretto)	`python3` (3.9)	`sedona-emr-spark-shaded-3.5_2.12-VERSION.jar`
`emr-spark-8.0.0`	4.0	2.13	17 (Corretto)	`/usr/bin/python3.11`	`sedona-emr-spark-shaded-4.0_2.13-VERSION.jar`
6.x	3.3	2.12	8 (Corretto)	—	Not supported. The JDK 17 JAR will not load. Contact Wherobots if you must remain on EMR 6.x

A Spark or Scala mismatch surfaces at runtime as NoSuchMethodError or ClassNotFoundException. Every command in this guide targets emr-7.8.0. To deploy on emr-spark-8.0.0 instead, make all four substitutions below. They must move together — a release label that disagrees with the JAR produces the NoSuchMethodError described above.

	`emr-7.8.0` (this guide)	`emr-spark-8.0.0`
`--release-label` in `create-cluster`	`emr-7.8.0`	`emr-spark-8.0.0`
JAR	`sedona-emr-spark-shaded-3.5_2.12-VERSION.jar`	`sedona-emr-spark-shaded-4.0_2.13-VERSION.jar`
Wheel Python tag	`cp39`	`cp311`
`PY_BIN` in the bootstrap script	`/usr/bin/python3`	`/usr/bin/python3.11`

WherobotsDB is versioned independently of Apache Sedona. Each WherobotsDB release is API-compatible with a specific Apache Sedona version, and the GeoTools wrapper version follows the Sedona version — so VERSION and GEOTOOLS_VERSION are different values. For example:

WherobotsDB (`VERSION`)	Compatible Apache Sedona	GeoTools wrapper (`GEOTOOLS_VERSION`)
`1.28.0`	`1.9.0`	`1.9.0-33.5`

Always use the exact filenames Wherobots delivered to you.

Install WherobotsDB on an EMR cluster

Installing WherobotsDB involves staging its artifacts in Amazon S3, granting your cluster access to them, and launching an EMR cluster that loads them at startup. Complete the sections in the order listed below; later sections depend on values produced by earlier ones.

#	Section	Purpose
1	Prepare your command line	Install and authenticate the AWS CLI, and collect the account, network, and version values used throughout.
2	Configure your S3 bucket	Create the bucket and prefixes that hold the JARs, wheel, bootstrap script, application scripts, and cluster logs.
3	Configuration JSON	Define the Spark properties that load the WherobotsDB JARs and register the spatial SQL extensions.
4	Bootstrap script	Create and upload the script Amazon EMR runs on every node to install WherobotsDB.
5	WHL file	Upload the `wherobots-sedona` Python wheel that matches your cluster’s Python version and CPU architecture.
6	Create IAM roles	Create the trust policies for the three roles this deployment requires.
7	Create Service Role	Create the role Amazon EMR uses to provision and manage cluster instances.
8	Give Secret Manager Access	Store your Wherobots API key in AWS Secrets Manager and define the policy granting the cluster access to it.
9	Create Instance Profile	Create the role the cluster’s instances use to read your bucket, read your secret, and write logs.
10	Create security groups	Create the engine and workspace security groups that EMR Studio requires. Must be done before launching the cluster.
11	Create cluster	Launch the cluster with the configuration, bootstrap action, security groups, and key-configuration step in place.
12	Create EMR Studio	Create the Studio that hosts browser-based notebooks.
13	Launch notebook and attach cluster	Create a Workspace and connect it to your cluster.
14	Test in EMR Studio Notebook	Verify that WherobotsDB is installed, licensed, and running across the cluster.

Sections 1 through 12 are performed with the AWS CLI. Sections 13 and 14 are performed in the AWS Console and the notebook, because creating a Workspace has no CLI equivalent.

Prepare your command line

All aws commands in this guide are run from a terminal on your local machine, not on the cluster. To prepare your environment, complete the following steps:

Install the AWS CLI

Install AWS CLI v2 by following Installing or updating the latest version of the AWS CLI in the AWS documentation.

aws --version

Confirm your credentials

aws sts get-caller-identity

The command returns your account ID and the identity in use. If it returns ExpiredToken or Unable to locate credentials, re-authenticate with aws sso login or aws configure before continuing.

Set a default region

None of the commands in this guide pass --region, so your profile must have one set. Confirm it:

aws configure get region

If that prints nothing, set it before continuing — a profile with valid credentials but no region fails at create-cluster, the EC2 lookups, and the S3 calls:

aws configure set region REGION_IDENTIFIER

Collect the required values

Commands in this guide use uppercase placeholders. Gather the following values before you begin:

Placeholder	Where to find it
`YOUR_BUCKET_NAME`	The S3 bucket you create in the next section
`REGION_IDENTIFIER`	Your AWS region, for example `us-west-2`
`ACCOUNT_ID`	The `Account` field from `aws sts get-caller-identity`
`VPC_ID`, `SUBNET_ID`	`aws ec2 describe-subnets --query 'Subnets[].{Subnet:SubnetId,Vpc:VpcId,AZ:AvailabilityZone}' --output table`
`YOUR_KEY_PAIR`	`aws ec2 describe-key-pairs --query 'KeyPairs[].KeyName' --output table`
`VERSION`	The WherobotsDB release version, taken from the JAR and wheel filenames Wherobots supplied
`GEOTOOLS_VERSION`	The GeoTools wrapper version, taken from the `geotools-wrapper-.jar` filename. This is not* the same as `VERSION` — see Version compatibility

Several steps use cat > filename <<'EOF' … EOF to create a file in the current working directory. Run every command in this guide from that same directory.

Some commands pass a filename prefixed with file://, for example --policy-document file://ec2-trust.json. The prefix instructs the AWS CLI to read the argument from a file rather than interpret it as a literal value, and it is required. Omitting it causes the command to fail, because the CLI treats the filename itself as the policy document.The path after file:// is relative to your current directory. If you keep these files elsewhere, supply the full path instead, for example file:///Users/name/emr-setup/ec2-trust.json.

This guide uses the following Amazon EMR terms:

Term	Definition
Primary node	The coordinating node of the cluster. Spark drivers and notebook kernels run here.
Bootstrap action	A script Amazon EMR runs on every node during cluster creation, before Spark is installed. It cannot be added or modified after launch.
Step	A job submitted to a cluster. Steps run after Spark is installed, unlike bootstrap actions.
Instance profile	The IAM role used by the cluster’s EC2 instances. This role requires access to your S3 bucket and your Wherobots API key.
Service role	The IAM role Amazon EMR uses to provision and manage those instances. It does not require access to your data.

File destinations

This guide works with two kinds of files: the artifacts Wherobots supplies, and the files you author yourself. Five objects are uploaded to Amazon S3 and read by the cluster at runtime. The remaining files you author are never uploaded — the AWS CLI reads them from disk and submits their contents to Amazon EMR and IAM, which retain them as part of the cluster definition and the roles’ policies.

File	Destination	Read by
`sedona-emr-spark-shaded-.jar`, `geotools-wrapper-.jar`	`s3://YOUR_BUCKET_NAME/wherobots/jars/`	The bootstrap script, which copies them to `/jars` on every node
`wherobots_sedona-*.whl`	`s3://YOUR_BUCKET_NAME/wherobots/wheels/`	The bootstrap script, which installs it on every node
`bootstrap-wherobotsdb.sh`	`s3://YOUR_BUCKET_NAME/wherobots/bootstrap/`	Amazon EMR, on every node during cluster creation
`configure_key.py`	`s3://YOUR_BUCKET_NAME/apps/`	Spark, as the cluster’s first step
`configurations.json`	Retained in the cluster definition	Spark, as `spark-defaults` on every node
`ec2-trust.json`, `emr-trust.json`, `studio-trust.json`	Retained as IAM trust policies	IAM, whenever the role is assumed
`emr-service-role-policy.json`, `instance-profile-policy.json`, `studio-policy.json`	Retained as IAM permissions policies	IAM, on every request the role makes

Filenames must match exactly between Amazon S3 and the configuration block of the bootstrap script. A mismatch causes cluster creation to fail with NoSuchKey, or produces a cluster that starts without the Python client installed.

Values in configurations.json become part of the cluster definition and remain visible in the EMR console under Clusters → your cluster → Configurations, in the output of aws emr describe-cluster, and in the provisioning logs written to your log URI. Do not place your Wherobots API key in this file. This guide supplies the key through the configure_key.py step, which keeps it out of the cluster definition.

Configure your S3 bucket

In AWS, create a bucket with the following subdirectories:

YOUR_BUCKET_NAME/
├── apps/
├── emr-logs/
├── studio/
└── wherobots/
  ├── bootstrap/
  ├── jars/
  └── wheels/

AWS S3 Console
AWS CLI

Create the S3 bucket

In the AWS Console navigate to S3 → Create bucket. Enter YOUR_BUCKET_NAME, choose the region, enable default encryption and versioning as required, then click Create bucket.

Create the folders (prefixes)

Open the bucket in the Console, click Create folder, and add the following folders: apps, emr-logs, studio, and wherobots. Inside wherobots, create bootstrap, jars, and wheels.

When you are done, the wherobots prefix contains exactly three folders: bootstrap/, jars/, and wheels/.

# Create the bucket (replace region as needed)
aws s3api create-bucket --bucket YOUR_BUCKET_NAME --region REGION_IDENTIFIER \
  --create-bucket-configuration LocationConstraint=REGION_IDENTIFIER

# Create prefixes (S3 "folders")
aws s3api put-object --bucket YOUR_BUCKET_NAME --key apps/
aws s3api put-object --bucket YOUR_BUCKET_NAME --key emr-logs/
aws s3api put-object --bucket YOUR_BUCKET_NAME --key studio/
aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/
aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/bootstrap/
aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/jars/
aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/wheels/

# Verify contents
aws s3 ls s3://YOUR_BUCKET_NAME/ --recursive

In us-east-1, omit --create-bucket-configuration entirely. That region rejects an explicit LocationConstraint with InvalidLocationConstraint.

Upload the WherobotsDB JARs supplied by Wherobots to the wherobots/jars/ prefix:

aws s3 cp sedona-emr-spark-shaded-3.5_2.12-VERSION.jar s3://YOUR_BUCKET_NAME/wherobots/jars/
aws s3 cp geotools-wrapper-GEOTOOLS_VERSION.jar        s3://YOUR_BUCKET_NAME/wherobots/jars/

Use the sedona-emr-spark-shaded-* JAR. The sedona-dbr-spark-shaded-* JAR in the same bundle is built for Databricks Runtime and will not work on Amazon EMR, and the public sedona-spark-shaded-* JAR on Maven Central is the open-source Apache Sedona build without the WherobotsDB enterprise features.

Match the JAR to the Spark and Scala version of your EMR release, as listed in Version compatibility. A mismatch surfaces at runtime as NoSuchMethodError or ClassNotFoundException.

Configuration JSON

This file defines the Spark properties that load the WherobotsDB JARs and register the spatial SQL functions. It is not uploaded to Amazon S3. Instead, create-cluster reads it from your local disk and submits its contents to Amazon EMR, which retains them as part of the cluster definition. To create the configuration file, complete the following steps:

Create the file on your local machine

In the working directory you established in Prepare your command line, create a file named configurations.json using a text editor, and paste in the following:

[
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.jars": "local:///jars/sedona-emr-spark-shaded-3.5_2.12-VERSION.jar,local:///jars/geotools-wrapper-GEOTOOLS_VERSION.jar",
      "spark.yarn.dist.jars": "/jars/sedona-emr-spark-shaded-3.5_2.12-VERSION.jar,/jars/geotools-wrapper-GEOTOOLS_VERSION.jar",
      "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
      "spark.kryo.registrator": "org.apache.sedona.core.serde.SedonaKryoRegistrator",
      "spark.sedona.enableParserExtension": "false",
      "spark.sql.extensions": "org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions",
      "spark.redaction.regex": "(?i)secret|password|token|api\\.key"
    }
  }
]

Replace the JAR filenames

Substitute VERSION and GEOTOOLS_VERSION in both spark.jars and spark.yarn.dist.jars with the versions of the JARs you uploaded to wherobots/jars/. The two properties must list the same two JARs, in the same order.

Property	Purpose
`spark.jars`	Adds the JARs to the Spark driver classpath. The `local:///` scheme means “already present at this path on every node,” which the bootstrap action guarantees
`spark.yarn.dist.jars`	Distributes the JARs to YARN containers (executors, and the driver in cluster mode)
`spark.serializer`, `spark.kryo.registrator`	Configure Kryo serialization for geometry types
`spark.sedona.enableParserExtension`	Must remain `false`
`spark.sql.extensions`	Registers the `ST_*` SQL functions
`spark.redaction.regex`	Scrubs the API key from the Spark UI and event logs

spark.jars is required and is not interchangeable with spark.yarn.dist.jars. The latter distributes the JARs to YARN containers but does not add them to the driver classpath, so Spark cannot load the classes named in spark.sql.extensions. Spark logs a warning, continues without the extensions, and every ST_* call then fails with UNRESOLVED_ROUTINE on a cluster that otherwise appears healthy. Set both properties.

spark.sql.extensions is a comma-separated list, and the value you set here replaces any value already in effect. If your cluster configuration registers other extensions — Apache Iceberg’s, for example — append the two Sedona entries to your existing list instead of pasting this block verbatim. Stock Amazon EMR sets no spark.sql.extensions default of its own, so on an unmodified cluster the value above is safe as-is.

Validate the JSON

python3 -m json.tool configurations.json

The command prints the formatted file if it is valid. Malformed JSON causes create-cluster to fail immediately with a parse error.

After the cluster is created, you can confirm these settings applied in the EMR console under Clusters → your cluster → Configurations, or by running aws emr describe-cluster --cluster-id CLUSTER_ID --query 'Cluster.Configurations'.

Bootstrap script

A bootstrap action is a script that Amazon EMR runs on every node of the cluster during creation, before Spark is installed. This script copies the WherobotsDB JARs from Amazon S3 onto each node and installs the wherobots-sedona Python client. The API key is not handled here. Bootstrap actions run before Spark exists, and a shell export in a bootstrap script exits before any Spark process starts, so the variable never reaches them. The key is supplied by a cluster step instead — see Create cluster. You do not run this script yourself. You create it on your local machine, upload it to Amazon S3, and reference its S3 path in the --bootstrap-actions argument of Create cluster. Amazon EMR retrieves and runs it on each node. To create and upload the bootstrap script, complete the following steps:

Create the file on your local machine

In the working directory you established in Prepare your command line, create a file named bootstrap-wherobotsdb.sh using a text editor, and paste in the script below.

#!/bin/bash
set -euo pipefail

# ---- Configuration: edit this section --------------------------------------
S3_JARS="s3://YOUR_BUCKET_NAME/wherobots/jars"
S3_WHEELS="s3://YOUR_BUCKET_NAME/wherobots/wheels"
JARS_DIR="/jars"

SEDONA_JAR="sedona-emr-spark-shaded-3.5_2.12-VERSION.jar"
GEOTOOLS_JAR="geotools-wrapper-GEOTOOLS_VERSION.jar"
SEDONA_WHEEL="wherobots_sedona-VERSION-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl"

# The interpreter PySpark runs. On EMR 7.8.0 this is the system python3
# (3.9); on emr-spark-8.0.0 set it to /usr/bin/python3.11 instead.
PY_BIN="/usr/bin/python3"

# Set to "false" only for Scala- or SQL-only clusters that do not need the
# Python client. The notebook tests will not work without it.
REQUIRE_PYTHON_CLIENT="true"
# ---------------------------------------------------------------------------

echo "[wherobots-bootstrap] Copying WherobotsDB JARs to ${JARS_DIR} ..."
sudo mkdir -p "${JARS_DIR}"
# Each file is copied by exact name, so s3:GetObject alone is sufficient.
sudo aws s3 cp "${S3_JARS}/${SEDONA_JAR}"   "${JARS_DIR}/${SEDONA_JAR}"
sudo aws s3 cp "${S3_JARS}/${GEOTOOLS_JAR}" "${JARS_DIR}/${GEOTOOLS_JAR}"

echo "[wherobots-bootstrap] Verifying downloaded JARs:"
ls -la "${JARS_DIR}/" | grep -E "sedona-emr|geotools-wrapper" || {
  echo "[wherobots-bootstrap] ERROR: expected JARs not found in ${JARS_DIR}" >&2
  exit 1
}

# ---- Python client ---------------------------------------------------------
# INTERPRETER MATTERS: the wheel must be installed into the interpreter that
# PySpark runs, which is not always the one `python3` resolves to. On EMR 7.8.0
# they are the same; on emr-spark-8.0.0 Spark uses /usr/bin/python3.11 while
# the shell's `python3` is 3.9, so installing with plain `python3 -m pip` puts
# the client where Spark never looks. PY_BIN is set in the block above.
[ -x "${PY_BIN}" ] || {
  echo "[wherobots-bootstrap] ERROR: PY_BIN=${PY_BIN} is not executable on this node." >&2
  exit 1
}

WHEEL_LOCAL="/tmp/${SEDONA_WHEEL}"
if aws s3 cp "${S3_WHEELS}/${SEDONA_WHEEL}" "${WHEEL_LOCAL}"; then
  echo "[wherobots-bootstrap] Installing ${SEDONA_WHEEL} into ${PY_BIN} ..."
  sudo "${PY_BIN}" -m pip install --quiet "${WHEEL_LOCAL}"
  sudo "${PY_BIN}" -c 'import sedona; print("[wherobots-bootstrap] sedona installed:", sedona.__file__)'
elif [ "${REQUIRE_PYTHON_CLIENT}" = "true" ]; then
  # Fail the cluster now, with the cause in the bootstrap log, rather than
  # letting the notebook tests die later with ModuleNotFoundError.
  echo "[wherobots-bootstrap] ERROR: could not fetch ${SEDONA_WHEEL} from ${S3_WHEELS}" >&2
  echo "[wherobots-bootstrap] Stage the wheel for this node's Python and architecture, or set REQUIRE_PYTHON_CLIENT=false." >&2
  exit 1
else
  echo "[wherobots-bootstrap] Python client not installed (REQUIRE_PYTHON_CLIENT=false)."
fi

echo "[wherobots-bootstrap] Done."

Edit the configuration block

Replace the values in the Configuration: edit this section block at the top of the file. Leave the rest of the script unchanged.

Variable	Set to
`S3_JARS`, `S3_WHEELS`	Your bucket name in place of `YOUR_BUCKET_NAME`
`SEDONA_JAR`, `GEOTOOLS_JAR`	The exact JAR filenames you uploaded to `wherobots/jars/`
`SEDONA_WHEEL`	The exact wheel filename you will upload in WHL file
`PY_BIN`	`/usr/bin/python3` on EMR 7.8.0. Change it only if you are deploying on `emr-spark-8.0.0`, where it must be `/usr/bin/python3.11`
`REQUIRE_PYTHON_CLIENT`	Leave as `true` unless this is a Scala- or SQL-only cluster

If you add your own pip install lines, put them after every aws command in the script. The AWS CLI on Amazon Linux is rpm-managed and shares the system Python, so installing packages such as pandas or geopandas upgrades python-dateutil and jmespath out from under it. Any aws command that runs afterwards fails with ModuleNotFoundError: No module named 'dateutil'.

Install packages with sudo "${PY_BIN}" -m pip rather than hard-coding an interpreter. On EMR 7.8.0 PY_BIN and python3 are the same binary, but on emr-spark-8.0.0 they are not — using ${PY_BIN} keeps the script correct on both.

Upload the script to Amazon S3

Run the following in your terminal, from the directory containing the file:

aws s3 cp bootstrap-wherobotsdb.sh \
  s3://YOUR_BUCKET_NAME/wherobots/bootstrap/bootstrap-wherobotsdb.sh

Verify the upload

aws s3 ls s3://YOUR_BUCKET_NAME/wherobots/bootstrap/

The file must appear with a non-zero size. Amazon EMR reads this object at cluster-creation time, so re-upload it after any edit — clusters do not pick up local changes.

WHL file

The wherobots-sedona Python client is distributed as prebuilt wheels in the WherobotsDB release bundle, alongside the JARs. It is not published to PyPI. You still install it with pip — you just point pip at the wheel file rather than at a package name. pip install wherobots-sedona cannot resolve, because there is no index to resolve it from; pip install /path/to/wherobots_sedona-....whl works normally, which is what the bootstrap script does. The wheel filename encodes both a Python version tag and a CPU architecture, and both must match your cluster. Pick the Python tag from your EMR release:

EMR release	PySpark interpreter	Python tag
`emr-7.8.0` (this guide)	`/usr/bin/python3` (3.9)	`cp39`
Other 7.x releases	`/usr/bin/python3` (3.9)	`cp39`
`emr-spark-8.0.0`	`/usr/bin/python3.11`	`cp311`

Pick the architecture from your instance type:

Instance family	Architecture	Architecture tag
`r8g`, `m7g`, `c7g`	Graviton / ARM64	`aarch64`
`m5`, `m6i`, `c5`	Intel / AMD	`x86_64`

The release bundle also contains macosx_* and win_* wheels. Those are for local development — cluster nodes always take a manylinux wheel.

The Python tag must match the interpreter PySpark uses, not the one python3 resolves to in your shell. On EMR 7.8.0 those are the same binary, so cp39 is correct. They diverge on emr-spark-8.0.0, where Spark uses Python 3.11 while the shell’s python3 is still 3.9 — a wheel built for the wrong Python version installs without complaint and is never importable from Spark. Installing the wrong architecture fails outright with not a supported wheel on this platform.

To upload the wheel, complete the following steps:

Confirm your cluster's Python tag and architecture

Check your EMR release and the instance type you plan to launch against the tables above. The examples in this guide use emr-7.8.0 on r8g.xlarge, which is Graviton, so they use the cp39 / aarch64 wheel.

Create the wheels prefix

In your terminal, run:

aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/wheels/

Upload the wheel

Run this from the directory containing the wheel files Wherobots supplied:

aws s3 cp wherobots_sedona-VERSION-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl \
  s3://YOUR_BUCKET_NAME/wherobots/wheels/

Verify the upload

aws s3 ls s3://YOUR_BUCKET_NAME/wherobots/wheels/

The filename listed must match SEDONA_WHEEL in the bootstrap script exactly.

Create IAM roles

This deployment uses three IAM roles. Confusing the first two is the most common setup mistake:

Role	Used by	Needs access to your bucket and API key?
Service role	Amazon EMR itself, to create and manage EC2 machines	No
EC2 instance profile	The machines inside the cluster, which run the bootstrap script and the key step	Yes
Studio service role	EMR Studio, to store notebook files and attach Workspaces to clusters	Bucket only

Each role needs a trust policy naming the AWS service allowed to use it. In your terminal, create all three files first. They are not uploaded to S3 — the aws iam commands read them from disk and submit their contents to IAM, which stores them on the role:

cat > ec2-trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

cat > emr-trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"elasticmapreduce.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

cat > studio-trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"elasticmapreduce.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

Create Service Role

To create the role Amazon EMR uses to manage your cluster, complete the following steps in your terminal:

Create the role

aws iam create-role --role-name WherobotsEMRServiceRole \
  --assume-role-policy-document file://emr-trust.json

Attach the AWS managed policy

aws iam attach-role-policy --role-name WherobotsEMRServiceRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEMRServicePolicy_v2

AmazonEMRServicePolicy_v2 alone is not sufficient for this setup. It scopes its EC2 permissions to resources carrying the for-use-with-amazon-emr-managed-policies tag, and it does not grant iam:PassRole for a custom instance profile name such as WherobotsEMRInstanceRole. The inline policy you add in the next two steps supplies both: explicit access to your subnet and security groups, and iam:PassRole for the instance profile.

Look up your security group IDs

aws ec2 describe-security-groups \
  --filters Name=vpc-id,Values=VPC_ID \
            Name=group-name,Values=ElasticMapReduce-master,ElasticMapReduce-slave \
  --query 'SecurityGroups[].{Name:GroupName,Id:GroupId}' --output table

Record both group IDs. You substitute them into the policy in the next step as SECURITY_GROUP_ID_MASTER and SECURITY_GROUP_ID_SLAVE.

If the command returns no rows, Amazon EMR has not yet created its managed groups in this VPC. This is normal before your first cluster launch. Take the “no managed groups yet” path:

In the policy file in the next step, remove the two security-group ARNs from the CreateInNetwork statement, keeping its subnet ARN, and delete the entire ManageSecurityGroups statement. Its Resource array contains only those two ARNs, so removing them individually would leave an empty array, which put-role-policy rejects as invalid.
In Create cluster, omit EmrManagedMasterSecurityGroup= and EmrManagedSlaveSecurityGroup= from --ec2-attributes. Amazon EMR then creates its default managed groups during launch. The CreateDefaultSecurityGroupInVPC statement is what permits this.
Afterwards, re-run the lookup above, add the two ARNs back to CreateInNetwork, restore the ManageSecurityGroups statement, and re-apply the policy with put-role-policy.

Create the policy file

This policy grants the service role access to your specific subnet, security groups, and VPC, and allows it to pass the instance profile to EC2.Substitute SUBNET_ID, SECURITY_GROUP_ID_MASTER, SECURITY_GROUP_ID_SLAVE, VPC_ID, and ACCOUNT_ID, then run:

cat > emr-service-role-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CreateInNetwork",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:RunInstances",
        "ec2:CreateFleet",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateLaunchTemplateVersion"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:subnet/SUBNET_ID",
        "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_MASTER",
        "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_SLAVE"
      ]
    },
    {
      "Sid": "ManageSecurityGroups",
      "Effect": "Allow",
      "Action": [
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_MASTER",
        "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_SLAVE"
      ]
    },
    {
      "Sid": "CreateDefaultSecurityGroupInVPC",
      "Effect": "Allow",
      "Action": ["ec2:CreateSecurityGroup"],
      "Resource": ["arn:aws:ec2:*:*:vpc/VPC_ID"]
    },
    {
      "Sid": "PassRoleForEC2",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::ACCOUNT_ID:role/WherobotsEMRInstanceRole",
      "Condition": {
        "StringLike": {"iam:PassedToService": "ec2.amazonaws.com"}
      }
    }
  ]
}
EOF

aws iam put-role-policy --role-name WherobotsEMRServiceRole \
  --policy-name wherobots-emr-network-access \
  --policy-document file://emr-service-role-policy.json

Verify both policies are attached

aws iam list-attached-role-policies --role-name WherobotsEMRServiceRole
aws iam list-role-policies --role-name WherobotsEMRServiceRole

The first command must list AmazonEMRServicePolicy_v2. The second must list wherobots-emr-network-access.

Give Secret Manager Access

To store your Wherobots API key in AWS Secrets Manager and grant the cluster access to it, complete the following steps:

Create a Wherobots API key

Copy the key before leaving the page. It is shown only once.

Store the key as a secret

In your terminal, run:Passing the key as a command-line argument writes it into your shell history and exposes it in the local process list. Read it into a 0600 file instead, hand that file to the CLI, then delete it:

umask 077
read -rs -p "Wherobots API key: " KEY && printf '%s' "$KEY" > api-key.txt && unset KEY

aws secretsmanager create-secret --name wherobots/api-key \
  --secret-string file://api-key.txt

rm -f api-key.txt

read -rs keeps the key off the screen, printf '%s' writes it without a trailing newline, and umask 077 makes the file readable only by you.Store the key as plaintext — no JSON wrapper and no trailing newline. The key step reads the value verbatim, so anything extra ends up inside the key.

Note the secret's ARN

aws secretsmanager describe-secret --secret-id wherobots/api-key --query ARN --output text

The ARN ends in six random characters, for example ...:secret:wherobots/api-key-Ab3xY9.

Create the access policy file

This policy grants the instance profile role — not the service role — permission to read the secret, read your bucket, and write cluster logs. It is applied to the role in Create Instance Profile.Substitute YOUR_BUCKET_NAME, REGION_IDENTIFIER, and ACCOUNT_ID, then run:

cat > instance-profile-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload",
                 "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
    },
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:REGION_IDENTIFIER:ACCOUNT_ID:secret:wherobots/api-key-*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::elasticmapreduce", "arn:aws:s3:::elasticmapreduce/*",
                   "arn:aws:s3:::*.elasticmapreduce/*"]
    }
  ]
}
EOF

Keep the trailing -* on the secret ARN. Secrets Manager appends six random characters to every secret ARN, so a policy pinned to the literal ARN breaks if the secret is ever recreated.

Without the s3:PutObject statement, Amazon EMR cannot write to the log URI. A cluster that fails then reports Unknown Error. with no logs available for diagnosis.

Create Instance Profile

An instance profile is a wrapper that lets EC2 machines use an IAM role. The role by itself is not enough — EC2 can only assume a role that is wrapped in an instance profile of the same name. To create it, complete the following steps in your terminal:

Create the role

aws iam create-role --role-name WherobotsEMRInstanceRole \
  --assume-role-policy-document file://ec2-trust.json

Attach the access policy

aws iam put-role-policy --role-name WherobotsEMRInstanceRole \
  --policy-name wherobots-emr-access \
  --policy-document file://instance-profile-policy.json

Wrap the role in an instance profile

aws iam create-instance-profile --instance-profile-name WherobotsEMRInstanceRole

aws iam add-role-to-instance-profile \
  --instance-profile-name WherobotsEMRInstanceRole \
  --role-name WherobotsEMRInstanceRole

Verify the grants

aws iam list-role-policies --role-name WherobotsEMRInstanceRole
aws iam get-instance-profile --instance-profile-name WherobotsEMRInstanceRole

The second command must list WherobotsEMRInstanceRole under Roles. If it returns an empty list, the previous step did not complete.

Create security groups

EMR Studio reaches the cluster over port 18888, the Jupyter Enterprise Gateway. That requires two security groups: an engine group attached to the cluster’s primary node, and a workspace group attached to the Workspace.

Create these groups before launching the cluster. The engine group is attached to the primary node at launch through AdditionalMasterSecurityGroups and cannot be added to a running cluster. If you launch first, your only options are to recreate the cluster or to add an inbound rule for TCP 18888 to the cluster’s existing primary security group.

In your terminal, create both groups and connect them:

ENG=$(aws ec2 create-security-group --group-name emr-studio-engine-sg \
  --description "EMR Studio engine" --vpc-id VPC_ID --query GroupId --output text)

WS=$(aws ec2 create-security-group --group-name emr-studio-workspace-sg \
  --description "EMR Studio workspace" --vpc-id VPC_ID --query GroupId --output text)

aws ec2 authorize-security-group-ingress --group-id $ENG \
  --ip-permissions "IpProtocol=tcp,FromPort=18888,ToPort=18888,UserIdGroupPairs=[{GroupId=$WS}]"

aws ec2 authorize-security-group-egress --group-id $WS \
  --ip-permissions "IpProtocol=tcp,FromPort=18888,ToPort=18888,UserIdGroupPairs=[{GroupId=$ENG}]"

aws ec2 create-tags --resources $ENG $WS \
  --tags Key=for-use-with-amazon-emr-managed-policies,Value=true

echo "engine=$ENG workspace=$WS"

Record both IDs. The next section uses the engine group as ENGINE_SECURITY_GROUP_ID, and Create EMR Studio uses both.

The tag on the engine security group is required. Because the group is attached to the primary node at launch, the EMR service role needs ec2:RunInstances on it — which AmazonEMRServicePolicy_v2 grants only for tagged security groups. The alternative is to add its ARN to the CreateInNetwork statement of the service role policy.

$ENG and $WS are shell variables that exist only in the terminal session that created them. If you open a new terminal before finishing this guide, re-read the IDs with

aws ec2 describe-security-groups --filters Name=group-name,Values=emr-studio-engine-sg,emr-studio-workspace-sg --query 'SecurityGroups[].{Name:GroupName,Id:GroupId}' --output table

Create cluster

The following steps create an EMR cluster that runs WherobotsDB. You can also use an existing EMR cluster, but it must meet the requirements outlined in the Before you start section, and it must already carry the engine security group on its primary node. The API key reaches Spark through a step that runs immediately after the cluster starts. It reads the secret on the primary node and writes spark.wherobots.api.key into Spark’s own configuration file, so every later session picks it up. This cannot be done in the bootstrap script, because Spark is not installed yet when bootstrap actions run. In your terminal, save the following as configure_key.py:

"""Write the Wherobots API key into Spark's config on the primary node.

Runs as an EMR step, i.e. AFTER Spark is installed. This cannot live in the
bootstrap action, because /etc/spark/conf does not exist at bootstrap time.
"""
import subprocess
import sys

import boto3

CONF = "/etc/spark/conf/spark-defaults.conf"
SECRET_ID = "wherobots/api-key"
REGION = "REGION_IDENTIFIER"
PROP = "spark.wherobots.api.key"


def already_set(prop):
    with open(CONF) as fh:
        return any(line.strip().startswith(prop) for line in fh)


def append(line):
    subprocess.run(["sudo", "tee", "-a", CONF], input=(line + "\n").encode(),
                   check=True, stdout=subprocess.DEVNULL)


key = boto3.client("secretsmanager", region_name=REGION).get_secret_value(
    SecretId=SECRET_ID)["SecretString"].strip()
if not key:
    sys.exit("FATAL: secret is empty")
print(f"fetched key, length={len(key)} (value not logged)")

if already_set(PROP):
    print(f"{PROP} already present; leaving it unchanged")
else:
    append(f"{PROP} {key}")
print("DONE")

Substitute REGION_IDENTIFIER with your actual region before uploading. Unlike the IAM policy files, this script is not read by the AWS CLI on your machine — it is uploaded verbatim and executed on the cluster’s primary node, where a literal REGION_IDENTIFIER makes the boto3 client fail immediately. Because the step runs with ActionOnFailure=TERMINATE_CLUSTER, that failure tears down the cluster you just launched.

Substitute the region, then upload it to the apps/ prefix, where the cluster’s first step reads it from:

sed -i.bak "s/REGION_IDENTIFIER/$(aws configure get region)/" configure_key.py && rm -f configure_key.py.bak

grep '^REGION' configure_key.py   # confirm it now shows a real region

aws s3 cp configure_key.py s3://YOUR_BUCKET_NAME/apps/configure_key.py

spark.redaction.regex is set in Configuration JSON rather than here, because it is not a secret and should apply from the moment the cluster starts. Spark’s default value, (?i)secret|password|token, does not match spark.wherobots.api.key; without extending it the key appears in the Spark UI Environment tab and in the event logs written to S3.

This step writes to /etc/spark/conf/spark-defaults.conf on the primary node only. That covers EMR Studio notebook kernels and any pyspark or spark-submit --deploy-mode client session on the primary node. Jobs submitted with --deploy-mode cluster run their driver in a YARN container on a core node, which does not see this file — pass the key to those jobs explicitly with --conf spark.wherobots.api.key=... or as the WHEROBOTS_API_KEY environment variable.

To create the cluster, complete the following steps in your terminal:

Run create-cluster

aws emr create-cluster \
  --name "wherobotsdb-emr" \
  --release-label emr-7.8.0 \
  --applications Name=Spark Name=Livy Name=JupyterEnterpriseGateway \
  --log-uri s3://YOUR_BUCKET_NAME/emr-logs/ \
  --tags for-use-with-amazon-emr-managed-policies=true \
  --configurations file://configurations.json \
  --bootstrap-actions Path=s3://YOUR_BUCKET_NAME/wherobots/bootstrap/bootstrap-wherobotsdb.sh,Name=wherobots-install \
  --ec2-attributes InstanceProfile=WherobotsEMRInstanceRole,SubnetId=SUBNET_ID,KeyName=YOUR_KEY_PAIR,EmrManagedMasterSecurityGroup=SECURITY_GROUP_ID_MASTER,EmrManagedSlaveSecurityGroup=SECURITY_GROUP_ID_SLAVE,AdditionalMasterSecurityGroups=ENGINE_SECURITY_GROUP_ID \
  --service-role WherobotsEMRServiceRole \
  --instance-groups \
      InstanceGroupType=MASTER,InstanceCount=1,InstanceType=r8g.xlarge \
      InstanceGroupType=CORE,InstanceCount=1,InstanceType=r8g.xlarge \
  --steps '[{"Type":"Spark","Name":"configure-wherobots-key","ActionOnFailure":"TERMINATE_CLUSTER","Args":["--deploy-mode","client","s3://YOUR_BUCKET_NAME/apps/configure_key.py"]}]'

Substitute ENGINE_SECURITY_GROUP_ID with the engine group ID from Create security groups. This is what lets EMR Studio reach the cluster, and it cannot be added after launch.

ActionOnFailure is set to TERMINATE_CLUSTER deliberately. With CONTINUE, a failed key step leaves a cluster that reaches WAITING and looks healthy, but every spatial query fails later with an authentication error. Failing at launch surfaces the cause while the step logs are the obvious place to look.

The command returns a ClusterId. Use it as CLUSTER_ID in the steps that follow.

--tags for-use-with-amazon-emr-managed-policies=true is required. The EMR console adds this cluster tag automatically, but the CLI does not. EMR propagates cluster tags onto the EC2 instances it launches, and AmazonEMRServicePolicy_v2 authorizes ec2:RunInstances on instance/* only when that request tag is present. Omitting it fails cluster creation with VALIDATION_ERROR: Service role ... has insufficient EC2 permissions — a message that does not mention tags, and produces no logs because no instance ever launches.

Make sure the logs S3 path has a trailing slash.

Wait for the cluster to be ready

aws emr describe-cluster --cluster-id CLUSTER_ID \
  --query 'Cluster.{State:Status.State,Dns:MasterPublicDnsName}'

The state moves through STARTING, BOOTSTRAPPING, and RUNNING before reaching WAITING, which takes roughly five to ten minutes.

Confirm the bootstrap installed the Python client

Find the instance ID, then read the bootstrap log:

aws s3 ls s3://YOUR_BUCKET_NAME/emr-logs/CLUSTER_ID/node/

aws s3 cp s3://YOUR_BUCKET_NAME/emr-logs/CLUSTER_ID/node/INSTANCE_ID/bootstrap-actions/1/stdout.gz - \
  | gunzip | grep wherobots-bootstrap

Expected output includes:

[wherobots-bootstrap] Copying WherobotsDB JARs to /jars ...
[wherobots-bootstrap] Installing wherobots_sedona-... into /usr/bin/python3 ...
[wherobots-bootstrap] sedona installed: /usr/local/lib/python3.9/site-packages/sedona/__init__.py
[wherobots-bootstrap] Done.

Logs are archived to S3 on a five-minute cycle, so they may not appear immediately after the cluster is ready.

EMR Studio Notebook

EMR Studio is the browser-based notebook environment for EMR. A Studio is the container; a Workspace is an individual notebook environment inside it that attaches to a cluster.

Create EMR Studio

A Studio needs its own service role, which EMR Studio uses to store notebook files in your bucket and to attach Workspaces to clusters. To create the Studio, complete the following steps in your terminal:

Create the Studio service role

This role uses the studio-trust.json trust policy you created in Create IAM roles:

aws iam create-role --role-name WherobotsEMRStudioServiceRole \
  --assume-role-policy-document file://studio-trust.json

Create and attach the Studio policy

Substitute YOUR_BUCKET_NAME, then run:

cat > studio-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "StudioNotebookStorage",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:GetEncryptionConfiguration",
                 "s3:ListBucket", "s3:DeleteObject"],
      "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*"]
    },
    {
      "Sid": "DescribeClustersForWorkspaceAttach",
      "Effect": "Allow",
      "Action": ["elasticmapreduce:ListInstances",
                 "elasticmapreduce:DescribeCluster",
                 "elasticmapreduce:ListSteps"],
      "Resource": "*"
    },
    {
      "Sid": "NetworkDiscovery",
      "Effect": "Allow",
      "Action": ["ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups",
                 "ec2:DescribeSubnets", "ec2:DescribeVpcs",
                 "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface",
                 "ec2:CreateNetworkInterfacePermission"],
      "Resource": "*"
    }
  ]
}
EOF

aws iam put-role-policy --role-name WherobotsEMRStudioServiceRole \
  --policy-name wherobots-emr-studio-access \
  --policy-document file://studio-policy.json

Create the Studio

Substitute the engine and workspace security group IDs from Create security groups if you are in a new terminal session, then run:

aws emr create-studio \
  --name wherobotsdb-emr-studio \
  --auth-mode IAM \
  --vpc-id VPC_ID \
  --subnet-ids SUBNET_ID \
  --service-role WherobotsEMRStudioServiceRole \
  --workspace-security-group-id $WS \
  --engine-security-group-id $ENG \
  --default-s3-location s3://YOUR_BUCKET_NAME/studio/

The command returns a StudioId and a Url. Save the URL — it is how you open the Studio.

The Studio’s subnet must be the same one the cluster runs in, or Workspaces will not be able to reach it.

Security groups cannot be added to a Studio after it is created. If you see the message “You haven’t set up the VPC, subnets, or security groups in the Studio for this Workspace”, the Studio was created without them and you need to create a new one.

Launch notebook and attach cluster

Creating a Workspace has no CLI equivalent. To create one, complete the following steps in the AWS Console:

Open the Studio

Go to the Url returned by create-studio, or in the AWS Console navigate to EMR → EMR Studio, then select your Studio.

Create a Workspace

Select Workspaces → Create Workspace. Enter a name and leave the subnet at its default, which comes from the Studio.

Attach the cluster

In the Workspace, open the cluster panel on the left sidebar and attach it to the cluster you created. You can also select the cluster while creating the Workspace.

Launch and choose a kernel

Open the Workspace, create a new notebook, and select the PySpark kernel.

A Workspace in Ready status has its cluster and security group fields locked. Stop the Workspace first if you need to change which cluster it attaches to, then start it again.

Test in EMR Studio Notebook

These commands run in a notebook in your browser, not in your terminal. Nothing is installed in the notebook session and no API key is entered, because the bootstrap action and the configure-wherobots-key step have already supplied both. To open the notebook, complete the following steps:

Open EMR Studio

In the AWS Console, navigate to EMR → EMR Studio, then select the Studio you created in Create EMR Studio. You can also open the Url returned by aws emr create-studio.To retrieve the URL again, run:

aws emr list-studios --query 'Studios[].{Name:Name,Url:Url}' --output table

Open your Workspace

Select Workspaces, then select the Workspace you attached to your cluster. Its status must be Ready. If it is Stopped, select Start and wait for it to become Ready.

Confirm the cluster is attached

In the Workspace, open the cluster panel in the left sidebar and confirm your cluster is listed as attached. If it is not, attach it before continuing. See Launch notebook and attach cluster.

Create a PySpark notebook

Create a new notebook and select the PySpark kernel. The kernel must be PySpark; the other kernels do not connect to the Spark cluster.

Then run the following cells in order:

Confirm the client is installed against the interpreter Spark uses

import sys, sedona

print(sys.executable)   # /usr/bin/python3
print(sedona.__file__)  # .../python3.9/site-packages/sedona/__init__.py

If the import fails with No module named 'sedona', the wheel did not install. Check the bootstrap log for WARN: 'sedona' is NOT importable.

Initialize Sedona

from sedona.spark import *

sedona = SedonaContext.create(spark)

Run the validation query

sedona.sql("""
    SELECT
        ST_AsText(ST_Point(1.0, 2.0)) AS point,
        ST_Distance(ST_Point(0.0, 0.0), ST_Point(3.0, 4.0)) AS distance,
        ST_Area(ST_GeomFromText('POLYGON((0 0, 0 5, 5 5, 5 0, 0 0))')) AS area
""").show()

+-----------+--------+----+
|      point|distance|area|
+-----------+--------+----+
|POINT (1 2)|     5.0|25.0|
+-----------+--------+----+

Because spark.sql.extensions is set, ST_* functions are also available through plain spark.sql(...) without calling SedonaContext.create first. The later cells rely on this.

Confirm geometries reach Python

This exercises the wheel rather than the JAR alone — the returned value is a Shapely object, not text.

spark.sql("SELECT ST_Point(1.0, 2.0) AS geom").collect()

[Row(geom=<POINT (1 2)>)]

Confirm spatial functions run on executors

This requires the wheel to be present on every node, not just the primary.

df = spark.sql("SELECT ST_Point(CAST(id AS DOUBLE), CAST(id AS DOUBLE)) AS geom FROM range(0, 1000)")
df.rdd.map(lambda r: r.geom.wkt).take(2)

['POINT (0 0)', 'POINT (1 1)']

If a query fails with Wherobots Cloud credentials are required, the session did not pick up spark.wherobots.api.key. Confirm the configure-wherobots-key step completed, then restart the notebook kernel — spark-defaults.conf is read only when a session is created.

How Wherobots authentication works

WherobotsDB validates your API key at startup and uses it for usage-based billing. Both operations call Wherobots Cloud endpoints that are part of the BYOS feature:

At startup, the JAR authenticates against POST /apps/authenticate, which returns a workload ID.
While the workload runs, the JAR periodically reports usage heartbeats to POST /apps/report/{workload_id}.

If BYOS is not enabled for your Organization, both endpoints return 403 and initialization fails. The same feature grants access to the release JARs. See Before you start. You can supply the key in either of two ways:

Method	Notes
`spark.wherobots.api.key`	What this guide configures, through the `configure-wherobots-key` step. Works for notebook kernels and client-mode sessions on the primary node.
`WHEROBOTS_API_KEY` environment variable	The least visible option, since environment variables are not recorded in Spark’s properties tables or event logs. Export it in the shell before running `pyspark` or `spark-submit`. An `export` in the bootstrap script does not work, and it cannot reach EMR Studio or Livy sessions whose driver launches inside YARN.

For details on creating and rotating keys, see the Wherobots API Keys documentation.

Feature differences from Wherobots Cloud

BYOS provides the WherobotsDB engine on your cluster. Features of the managed Wherobots Cloud platform are not included:

The Wherobots managed catalogs — wherobots, wherobots_open_data, wherobots_pro_data, and the tutorial examples_temp catalog — are not available. Notebooks or jobs that reference them must be pointed at your own catalogs or read public data directly.
The EMR JAR does not bundle Apache Iceberg. Add your own Iceberg runtime and catalog configuration if you need Iceberg tables, appending its extension to spark.sql.extensions rather than replacing the Sedona entries.
Runtime conveniences that Wherobots Cloud preconfigures — the Spark checkpoint directory, visualization packages such as SedonaKepler, and pre-installed Python dependencies — must be set up on your cluster yourself. The bootstrap script covers the essentials.

Troubleshooting

ClassNotFoundException, UNRESOLVED_ROUTINE, ST_* functions not recognized, or 'JavaPackage' object is not callable in PySpark

Confirm both spark.jars and spark.yarn.dist.jars are set as in Configuration JSON, and that the filenames exactly match the files in /jars. A missing spark.jars is the classic cause when running pyspark or spark-submit in client mode: the extensions silently fail to load on the driver.
Check the bootstrap log at <log-uri>/<cluster-id>/node/<instance-id>/bootstrap-actions/1/ and confirm /jars was populated on every node.
If you set classpaths directly, append to Amazon EMR’s defaults rather than replacing them. Overwriting spark.driver.extraClassPath or spark.executor.extraClassPath at the cluster level removes EMRFS from the classpath and breaks S3 access. Prefer the spark.jars approach.

ModuleNotFoundError: No module named 'sedona' — the Python client is not installed into the interpreter PySpark uses. Check the bootstrap log for the wheel-install section, and confirm PY_BIN matches your EMR release: /usr/bin/python3 on EMR 7.8.0, /usr/bin/python3.11 on emr-spark-8.0.0. If you are installing by hand, pass pip the wheel’s path — pip install wherobots-sedona by name cannot resolve, since the package is not on PyPI. ModuleNotFoundError: No module named 'pandas' — wheels up to and including 1.28.0 did not declare their pandas dependency. When deploying those versions, add a sudo "${PY_BIN}" -m pip install pandas line to the bootstrap script, after every aws command. Bootstrap fails with Cannot uninstall setuptools ... RECORD file not found — pip is trying to replace the rpm-owned setuptools on Amazon Linux. Wheels up to and including 1.28.0 pinned setuptools>=69, which triggers this. Add --ignore-installed to the bootstrap’s pip install line when deploying those versions. Later releases carry no setuptools pin. UnsupportedClassVersionError — the cluster is running JDK 8. Use an EMR 7.x or later release, which ships JDK 17. See Version compatibility. NoSuchMethodError at query time — a Spark or Scala mismatch between the JAR and the EMR release, for example the 4.0_2.13 JAR on emr-7.8.0, which is Spark 3.5 and Scala 2.12. See Version compatibility. Bootstrap fails with S3 AccessDenied — the EMR EC2 instance profile is missing s3:GetObject on your staging prefix or on the bootstrap script itself. If you modified the bootstrap to use --recursive or sync, it additionally needs s3:ListBucket on the bucket. Cluster creation fails with VALIDATION_ERROR: Service role ... has insufficient EC2 permissions, and no logs are written — the for-use-with-amazon-emr-managed-policies=true cluster tag is missing. See Create cluster. No instance ever launches, so there is nothing to log. A failed cluster reports Unknown Error. with no logs — the instance profile cannot write to the log URI. Confirm the s3:PutObject statement from Give Secret Manager Access is present, and that --log-uri ends in a trailing slash. AccessDenied reading public S3 datasets — Amazon EMR’s S3 filesystem signs every request with the instance profile, and cross-account access requires an identity-policy allow even for publicly readable buckets. Either grant the instance profile read access to those buckets, or read them anonymously with s3a:// URIs plus

spark.hadoop.fs.s3a.bucket.<bucket-name>.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider

. Authentication errors from SedonaContext.create — confirm spark.wherobots.api.key or WHEROBOTS_API_KEY is set, that the key has not expired, that BYOS is enabled for your Organization, and that your security groups and network policy allow outbound HTTPS to Wherobots Cloud.

Support

For issues or questions, contact Wherobots support.

Agentic Development

Notebook Management

RasterFlow

SQL API & SDKs

Storage Management

Other deployment options

Orchestration

Monitoring

Bring Your Own Spark (AWS EMR)

For existing Apache Sedona users

Before you start

Version compatibility

Install WherobotsDB on an EMR cluster

Prepare your command line

File destinations

Configure your S3 bucket

Configuration JSON

Bootstrap script

WHL file

Create IAM roles

Create Service Role

Give Secret Manager Access

Create Instance Profile

Create security groups

Create cluster

EMR Studio Notebook

Create EMR Studio

Launch notebook and attach cluster

Test in EMR Studio Notebook

How Wherobots authentication works

Feature differences from Wherobots Cloud

Troubleshooting

Support

​For existing Apache Sedona users

​Before you start

​Version compatibility

​Install WherobotsDB on an EMR cluster

​Prepare your command line

​File destinations

​Configure your S3 bucket

​Configuration JSON

​Bootstrap script

​WHL file

​Create IAM roles

​Create Service Role

​Give Secret Manager Access

​Create Instance Profile

​Create security groups

​Create cluster

​EMR Studio Notebook

​Create EMR Studio

​Launch notebook and attach cluster

​Test in EMR Studio Notebook

​How Wherobots authentication works

​Feature differences from Wherobots Cloud

​Troubleshooting

​Support

For existing Apache Sedona users

Before you start

Version compatibility

Install WherobotsDB on an EMR cluster

Prepare your command line

File destinations

Configure your S3 bucket

Configuration JSON

Bootstrap script

WHL file

Create IAM roles

Create Service Role

Give Secret Manager Access

Create Instance Profile

Create security groups

Create cluster

EMR Studio Notebook

Create EMR Studio

Launch notebook and attach cluster

Test in EMR Studio Notebook

How Wherobots authentication works

Feature differences from Wherobots Cloud

Troubleshooting

Support