> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wherobots.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Bring Your Own Spark (AWS EMR)

> Install and run the WherobotsDB JAR on your own Amazon EMR cluster as a drop-in replacement for Apache Sedona, so your data never leaves your AWS account.

**Bring Your Own Spark (BYOS)** lets you run WherobotsDB on a Spark cluster you own and operate. WherobotsDB is a drop-in replacement for Apache Sedona: you install its JAR on your existing cluster, and your existing code, SQL, and configuration continue to work unchanged.

Your data never leaves your AWS account. Wherobots does not run, manage, or connect to your cluster — the only outbound traffic is a lightweight authenticated call from the JAR to Wherobots Cloud for license metering.

This guide covers Amazon EMR. Only **EMR on EC2** deployments are supported; EMR Serverless and EMR on EKS are not.

It is validated on **EMR 7.8.0** (Spark 3.5.4, Scala 2.12, Amazon Corretto JDK 17), and every command below targets that release. WherobotsDB is also validated on `emr-spark-8.0.0` — see [Version compatibility](#version-compatibility) for what changes.

<Note>
  BYOS is distinct from [Bring Your Own Cloud (BYOC)](/develop/storage-management/byoc). Under BYOC, Wherobots runs its own managed compute inside your AWS account. Under BYOS, you run your own Spark cluster and Wherobots supplies only the engine.

  For Databricks clusters, see [Bring Your Own Spark (Databricks)](/develop/deployment-options/byos-databricks) instead.
</Note>

## For existing Apache Sedona users

If you already run Apache Sedona on Amazon EMR, moving to WherobotsDB takes two changes:

1. **Replace the JAR.** Swap the public Apache Sedona JAR from Maven Central for the WherobotsDB EMR JAR that Wherobots supplies.

   | Old JAR                     | New JAR                         |
   | --------------------------- | ------------------------------- |
   | `sedona-spark-shaded-*.jar` | `sedona-emr-spark-shaded-*.jar` |

   The WherobotsDB build adds the enterprise functions, Wherobots Cloud authentication, and the metering client.

2. **Add your Wherobots API key.** See [Give Secret Manager Access](#give-secret-manager-access) and [Create cluster](#create-cluster).

Everything else in this guide — the IAM roles, bucket layout, and EMR Studio setup — is standard EMR configuration you may already have in place.

## Before you start

The following requirements must be met within both your Wherobots Organization and AWS account before you begin:

<AccordionGroup cols={2}>
  <Accordion title="Wherobots Requirements" icon="cloud">
    * An **Admin** account within a **Professional**, **Innovation**, or **Enterprise** Edition Organization.
          <Note>
            Wherobots Organization members with the **User** role can use a cluster an Admin has already set up, but cannot create the API key this deployment requires. See [Organization Roles](/get-started/organization-info/organization-roles/).
          </Note>
    * Community Edition is not supported. See [Organization Editions](/get-started/organization-management/organization-editions) or [Upgrade Organization](/get-started/upgrade-organization/).
    * The **BYOS feature enabled on your Organization**. Contact Wherobots to enable it. BYOS unlocks both halves of this deployment: read access to the WherobotsDB release JARs, and the runtime authentication and usage-reporting endpoints the JAR calls. Without it, those endpoints return `403` and cluster initialization fails.
    * A **Wherobots API key**. You create this in [Give Secret Manager Access](#give-secret-manager-access).
  </Accordion>

  <Accordion title="AWS Requirements" icon="aws">
    * An AWS account.
    * Permission to create IAM roles, instance profiles, and inline policies in that account. This guide creates three roles — an EMR service role, an EC2 instance role, and an EMR Studio service role — plus one instance profile. `AdministratorAccess` is sufficient but broader than required. For the specific actions involved, see [Actions defined by AWS Identity and Access Management](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsidentityandaccessmanagementiam.html#awsidentityandaccessmanagementiam-actions-as-permissions) and [Actions, resources, and condition keys for Amazon EMR](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonelasticmapreduce.html) in the AWS documentation.
    * Permission to create an S3 bucket, an EMR cluster, EC2 security groups, an EMR Studio, and a Secrets Manager secret.
    * **`iam:PassRole`** on the three roles this guide creates. `create-cluster` passes both the service role and the instance profile, and `create-studio` passes the Studio service role, so an identity without `iam:PassRole` on `WherobotsEMRServiceRole`, `WherobotsEMRInstanceRole`, and `WherobotsEMRStudioServiceRole` receives `AccessDenied` before either resource is created. This is a permission on the operator running these commands, distinct from the `iam:PassRole` the service role itself is granted in [Create Service Role](#create-service-role).
    * An **Amazon EMR 7.x or later** release. This guide targets `emr-7.8.0`. EMR 6.x is not supported — see [Version compatibility](#version-compatibility).
    * A VPC and subnet for the cluster. The EMR Studio you create later must use the same subnet as the cluster.
    * **Outbound HTTPS (port 443) from the cluster to Wherobots Cloud**, for license metering. A cluster in a private subnet needs a NAT gateway or an equivalent egress path; without it, `SedonaContext.create` fails to authenticate.
    * The WherobotsDB release bundle from Wherobots, containing the shaded JARs, the `geotools-wrapper` JAR, and the `wherobots-sedona` Python wheels. These are not published to Maven Central or PyPI.
  </Accordion>
</AccordionGroup>

<Note>
  Once BYOS is enabled for your Organization, a read-only **WherobotsDB Releases** storage source appears automatically in your Wherobots Cloud storage list. Go to **Storage**, open that source, and download the JARs and wheel from the release folder. If the source does not appear, BYOS is not yet enabled — contact Wherobots.
</Note>

## Version compatibility

WherobotsDB JARs are compiled against **JDK 17**, so Amazon EMR 7.x or later is required. Choose the JAR that matches your EMR release:

| EMR release                  | Spark | Scala | JDK           | PySpark Python        | JAR artifact                                                                                 |
| ---------------------------- | ----- | ----- | ------------- | --------------------- | -------------------------------------------------------------------------------------------- |
| **`emr-7.8.0`** (this guide) | 3.5   | 2.12  | 17 (Corretto) | `python3` (3.9)       | `sedona-emr-spark-shaded-3.5_2.12-VERSION.jar`                                               |
| Other 7.x releases           | 3.5   | 2.12  | 17 (Corretto) | `python3` (3.9)       | `sedona-emr-spark-shaded-3.5_2.12-VERSION.jar`                                               |
| `emr-spark-8.0.0`            | 4.0   | 2.13  | 17 (Corretto) | `/usr/bin/python3.11` | `sedona-emr-spark-shaded-4.0_2.13-VERSION.jar`                                               |
| 6.x                          | 3.3   | 2.12  | 8 (Corretto)  | —                     | Not supported. The JDK 17 JAR will not load. Contact Wherobots if you must remain on EMR 6.x |

A Spark or Scala mismatch surfaces at runtime as `NoSuchMethodError` or `ClassNotFoundException`.

Every command in this guide targets **`emr-7.8.0`**. To deploy on `emr-spark-8.0.0` instead, make all four substitutions below. They must move together — a release label that disagrees with the JAR produces the `NoSuchMethodError` described above.

|                                       | `emr-7.8.0` (this guide)                       | `emr-spark-8.0.0`                              |
| ------------------------------------- | ---------------------------------------------- | ---------------------------------------------- |
| `--release-label` in `create-cluster` | `emr-7.8.0`                                    | `emr-spark-8.0.0`                              |
| JAR                                   | `sedona-emr-spark-shaded-3.5_2.12-VERSION.jar` | `sedona-emr-spark-shaded-4.0_2.13-VERSION.jar` |
| Wheel Python tag                      | `cp39`                                         | `cp311`                                        |
| `PY_BIN` in the bootstrap script      | `/usr/bin/python3`                             | `/usr/bin/python3.11`                          |

WherobotsDB is versioned independently of Apache Sedona. Each WherobotsDB release is API-compatible with a specific Apache Sedona version, and the GeoTools wrapper version follows the Sedona version — so `VERSION` and `GEOTOOLS_VERSION` are different values. For example:

| WherobotsDB (`VERSION`) | Compatible Apache Sedona | GeoTools wrapper (`GEOTOOLS_VERSION`) |
| ----------------------- | ------------------------ | ------------------------------------- |
| `1.28.0`                | `1.9.0`                  | `1.9.0-33.5`                          |

Always use the exact filenames Wherobots delivered to you.

## Install WherobotsDB on an EMR cluster

Installing WherobotsDB involves staging its artifacts in Amazon S3, granting your cluster access to them, and launching an EMR cluster that loads them at startup. Complete the sections in the order listed below; later sections depend on values produced by earlier ones.

| #  | Section                                                                   | Purpose                                                                                                                  |
| -- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| 1  | [Prepare your command line](#prepare-your-command-line)                   | Install and authenticate the AWS CLI, and collect the account, network, and version values used throughout.              |
| 2  | [Configure your S3 bucket](#configure-your-s3-bucket)                     | Create the bucket and prefixes that hold the JARs, wheel, bootstrap script, application scripts, and cluster logs.       |
| 3  | [Configuration JSON](#configuration-json)                                 | Define the Spark properties that load the WherobotsDB JARs and register the spatial SQL extensions.                      |
| 4  | [Bootstrap script](#bootstrap-script)                                     | Create and upload the script Amazon EMR runs on every node to install WherobotsDB.                                       |
| 5  | [WHL file](#whl-file)                                                     | Upload the `wherobots-sedona` Python wheel that matches your cluster's Python version and CPU architecture.              |
| 6  | [Create IAM roles](#create-iam-roles)                                     | Create the trust policies for the three roles this deployment requires.                                                  |
| 7  | [Create Service Role](#create-service-role)                               | Create the role Amazon EMR uses to provision and manage cluster instances.                                               |
| 8  | [Give Secret Manager Access](#give-secret-manager-access)                 | Store your Wherobots API key in AWS Secrets Manager and define the policy granting the cluster access to it.             |
| 9  | [Create Instance Profile](#create-instance-profile)                       | Create the role the cluster's instances use to read your bucket, read your secret, and write logs.                       |
| 10 | [Create security groups](#create-security-groups)                         | Create the engine and workspace security groups that EMR Studio requires. Must be done **before** launching the cluster. |
| 11 | [Create cluster](#create-cluster)                                         | Launch the cluster with the configuration, bootstrap action, security groups, and key-configuration step in place.       |
| 12 | [Create EMR Studio](#create-emr-studio)                                   | Create the Studio that hosts browser-based notebooks.                                                                    |
| 13 | [Launch notebook and attach cluster](#launch-notebook-and-attach-cluster) | Create a Workspace and connect it to your cluster.                                                                       |
| 14 | [Test in EMR Studio Notebook](#test-in-emr-studio-notebook)               | Verify that WherobotsDB is installed, licensed, and running across the cluster.                                          |

Sections 1 through 12 are performed with the AWS CLI. Sections 13 and 14 are performed in the AWS Console and the notebook, because creating a Workspace has no CLI equivalent.

### Prepare your command line

All `aws` commands in this guide are run from a terminal on your local machine, not on the cluster.

To prepare your environment, complete the following steps:

<Steps>
  <Step title="Install the AWS CLI">
    Install AWS CLI v2 by following [Installing or updating the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) in the AWS documentation.

    ```bash theme={"system"}
    aws --version
    ```
  </Step>

  <Step title="Confirm your credentials">
    ```bash theme={"system"}
    aws sts get-caller-identity
    ```

    The command returns your account ID and the identity in use. If it returns `ExpiredToken` or `Unable to locate credentials`, re-authenticate with `aws sso login` or `aws configure` before continuing.
  </Step>

  <Step title="Set a default region">
    None of the commands in this guide pass `--region`, so your profile must have one set. Confirm it:

    ```bash theme={"system"}
    aws configure get region
    ```

    If that prints nothing, set it before continuing — a profile with valid credentials but no region fails at `create-cluster`, the EC2 lookups, and the S3 calls:

    ```bash theme={"system"}
    aws configure set region REGION_IDENTIFIER
    ```
  </Step>

  <Step title="Collect the required values">
    Commands in this guide use uppercase placeholders. Gather the following values before you begin:

    | Placeholder           | Where to find it                                                                                                                                                            |
    | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `YOUR_BUCKET_NAME`    | The S3 bucket you create in the next section                                                                                                                                |
    | `REGION_IDENTIFIER`   | Your AWS region, for example `us-west-2`                                                                                                                                    |
    | `ACCOUNT_ID`          | The `Account` field from `aws sts get-caller-identity`                                                                                                                      |
    | `VPC_ID`, `SUBNET_ID` | `aws ec2 describe-subnets --query 'Subnets[].{Subnet:SubnetId,Vpc:VpcId,AZ:AvailabilityZone}' --output table`                                                               |
    | `YOUR_KEY_PAIR`       | `aws ec2 describe-key-pairs --query 'KeyPairs[].KeyName' --output table`                                                                                                    |
    | `VERSION`             | The WherobotsDB release version, taken from the JAR and wheel filenames Wherobots supplied                                                                                  |
    | `GEOTOOLS_VERSION`    | The GeoTools wrapper version, taken from the `geotools-wrapper-*.jar` filename. This is **not** the same as `VERSION` — see [Version compatibility](#version-compatibility) |
  </Step>
</Steps>

Several steps use `cat > filename <<'EOF' … EOF` to create a file in the current working directory. Run every command in this guide from that same directory.

<Note>
  Some commands pass a filename prefixed with `file://`, for example `--policy-document file://ec2-trust.json`. The prefix instructs the AWS CLI to read the argument from a file rather than interpret it as a literal value, and it is required. Omitting it causes the command to fail, because the CLI treats the filename itself as the policy document.

  The path after `file://` is relative to your current directory. If you keep these files elsewhere, supply the full path instead, for example `file:///Users/name/emr-setup/ec2-trust.json`.
</Note>

<Note>
  This guide uses the following Amazon EMR terms:

  | Term                 | Definition                                                                                                                              |
  | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
  | **Primary node**     | The coordinating node of the cluster. Spark drivers and notebook kernels run here.                                                      |
  | **Bootstrap action** | A script Amazon EMR runs on every node during cluster creation, before Spark is installed. It cannot be added or modified after launch. |
  | **Step**             | A job submitted to a cluster. Steps run after Spark is installed, unlike bootstrap actions.                                             |
  | **Instance profile** | The IAM role used by the cluster's EC2 instances. This role requires access to your S3 bucket and your Wherobots API key.               |
  | **Service role**     | The IAM role Amazon EMR uses to provision and manage those instances. It does not require access to your data.                          |
</Note>

### File destinations

This guide works with two kinds of files: the artifacts Wherobots supplies, and the files you author yourself.

Five objects are uploaded to Amazon S3 and read by the cluster at runtime. The remaining files you author are never uploaded — the AWS CLI reads them from disk and submits their contents to Amazon EMR and IAM, which retain them as part of the cluster definition and the roles' policies.

| File                                                                                 | Destination                                  | Read by                                                          |
| ------------------------------------------------------------------------------------ | -------------------------------------------- | ---------------------------------------------------------------- |
| `sedona-emr-spark-shaded-*.jar`, `geotools-wrapper-*.jar`                            | `s3://YOUR_BUCKET_NAME/wherobots/jars/`      | The bootstrap script, which copies them to `/jars` on every node |
| `wherobots_sedona-*.whl`                                                             | `s3://YOUR_BUCKET_NAME/wherobots/wheels/`    | The bootstrap script, which installs it on every node            |
| `bootstrap-wherobotsdb.sh`                                                           | `s3://YOUR_BUCKET_NAME/wherobots/bootstrap/` | Amazon EMR, on every node during cluster creation                |
| `configure_key.py`                                                                   | `s3://YOUR_BUCKET_NAME/apps/`                | Spark, as the cluster's first step                               |
| `configurations.json`                                                                | Retained in the cluster definition           | Spark, as `spark-defaults` on every node                         |
| `ec2-trust.json`, `emr-trust.json`, `studio-trust.json`                              | Retained as IAM trust policies               | IAM, whenever the role is assumed                                |
| `emr-service-role-policy.json`, `instance-profile-policy.json`, `studio-policy.json` | Retained as IAM permissions policies         | IAM, on every request the role makes                             |

<Warning>
  Filenames must match exactly between Amazon S3 and the configuration block of the bootstrap script. A mismatch causes cluster creation to fail with `NoSuchKey`, or produces a cluster that starts without the Python client installed.
</Warning>

<Warning>
  Values in `configurations.json` become part of the cluster definition and remain visible in the EMR console under **Clusters → your cluster → Configurations**, in the output of `aws emr describe-cluster`, and in the provisioning logs written to your log URI. Do not place your Wherobots API key in this file. This guide supplies the key through the [`configure_key.py` step](#create-cluster), which keeps it out of the cluster definition.
</Warning>

### Configure your S3 bucket

In AWS, create a bucket with the following subdirectories:

```text theme={"system"}
YOUR_BUCKET_NAME/
├── apps/
├── emr-logs/
├── studio/
└── wherobots/
  ├── bootstrap/
  ├── jars/
  └── wheels/
```

<Tabs>
  <Tab title="AWS S3 Console">
    <Steps>
      <Step title="Create the S3 bucket">
        * In the AWS Console navigate to **S3 → Create bucket**. Enter `YOUR_BUCKET_NAME`, choose the region, enable default encryption and versioning as required, then click **Create bucket**.
      </Step>

      <Step title="Create the folders (prefixes)">
        * Open the bucket in the Console, click **Create folder**, and add the following folders: `apps`, `emr-logs`, `studio`, and `wherobots`. Inside `wherobots`, create `bootstrap`, `jars`, and `wheels`.

        When you are done, the `wherobots` prefix contains exactly three folders: `bootstrap/`, `jars/`, and `wheels/`.
      </Step>
    </Steps>
  </Tab>

  <Tab title="AWS CLI">
    ```bash wrap theme={"system"}
    # Create the bucket (replace region as needed)
    aws s3api create-bucket --bucket YOUR_BUCKET_NAME --region REGION_IDENTIFIER \
      --create-bucket-configuration LocationConstraint=REGION_IDENTIFIER

    # Create prefixes (S3 "folders")
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key apps/
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key emr-logs/
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key studio/
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/bootstrap/
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/jars/
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/wheels/

    # Verify contents
    aws s3 ls s3://YOUR_BUCKET_NAME/ --recursive
    ```

    <Note>
      In `us-east-1`, omit `--create-bucket-configuration` entirely. That region rejects an explicit `LocationConstraint` with `InvalidLocationConstraint`.
    </Note>
  </Tab>
</Tabs>

Upload the WherobotsDB JARs supplied by Wherobots to the `wherobots/jars/` prefix:

```bash wrap theme={"system"}
aws s3 cp sedona-emr-spark-shaded-3.5_2.12-VERSION.jar s3://YOUR_BUCKET_NAME/wherobots/jars/
aws s3 cp geotools-wrapper-GEOTOOLS_VERSION.jar        s3://YOUR_BUCKET_NAME/wherobots/jars/
```

<Warning>
  Use the `sedona-emr-spark-shaded-*` JAR. The `sedona-dbr-spark-shaded-*` JAR in the same bundle is built for Databricks Runtime and will not work on Amazon EMR, and the public `sedona-spark-shaded-*` JAR on Maven Central is the open-source Apache Sedona build without the WherobotsDB enterprise features.
</Warning>

<Warning>
  Match the JAR to the Spark and Scala version of your EMR release, as listed in [Version compatibility](#version-compatibility). A mismatch surfaces at runtime as `NoSuchMethodError` or `ClassNotFoundException`.
</Warning>

### Configuration JSON

This file defines the Spark properties that load the WherobotsDB JARs and register the spatial SQL functions. It is not uploaded to Amazon S3. Instead, `create-cluster` reads it from your local disk and submits its contents to Amazon EMR, which retains them as part of the cluster definition.

To create the configuration file, complete the following steps:

<Steps>
  <Step title="Create the file on your local machine">
    In the working directory you established in [Prepare your command line](#prepare-your-command-line), create a file named `configurations.json` using a text editor, and paste in the following:

    ```json theme={"system"}
    [
      {
        "Classification": "spark-defaults",
        "Properties": {
          "spark.jars": "local:///jars/sedona-emr-spark-shaded-3.5_2.12-VERSION.jar,local:///jars/geotools-wrapper-GEOTOOLS_VERSION.jar",
          "spark.yarn.dist.jars": "/jars/sedona-emr-spark-shaded-3.5_2.12-VERSION.jar,/jars/geotools-wrapper-GEOTOOLS_VERSION.jar",
          "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
          "spark.kryo.registrator": "org.apache.sedona.core.serde.SedonaKryoRegistrator",
          "spark.sedona.enableParserExtension": "false",
          "spark.sql.extensions": "org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions",
          "spark.redaction.regex": "(?i)secret|password|token|api\\.key"
        }
      }
    ]
    ```
  </Step>

  <Step title="Replace the JAR filenames">
    Substitute `VERSION` and `GEOTOOLS_VERSION` in both `spark.jars` and `spark.yarn.dist.jars` with the versions of the JARs you uploaded to `wherobots/jars/`. The two properties must list the same two JARs, in the same order.

    | Property                                     | Purpose                                                                                                                                                       |
    | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `spark.jars`                                 | Adds the JARs to the Spark driver classpath. The `local:///` scheme means "already present at this path on every node," which the bootstrap action guarantees |
    | `spark.yarn.dist.jars`                       | Distributes the JARs to YARN containers (executors, and the driver in cluster mode)                                                                           |
    | `spark.serializer`, `spark.kryo.registrator` | Configure Kryo serialization for geometry types                                                                                                               |
    | `spark.sedona.enableParserExtension`         | Must remain `false`                                                                                                                                           |
    | `spark.sql.extensions`                       | Registers the `ST_*` SQL functions                                                                                                                            |
    | `spark.redaction.regex`                      | Scrubs the API key from the Spark UI and event logs                                                                                                           |

    <Warning>
      `spark.jars` is required and is **not** interchangeable with `spark.yarn.dist.jars`. The latter distributes the JARs to YARN containers but does not add them to the driver classpath, so Spark cannot load the classes named in `spark.sql.extensions`. Spark logs a warning, continues without the extensions, and every `ST_*` call then fails with `UNRESOLVED_ROUTINE` on a cluster that otherwise appears healthy. Set both properties.
    </Warning>

    <Warning>
      `spark.sql.extensions` is a comma-separated list, and the value you set here **replaces** any value already in effect. If your cluster configuration registers other extensions — Apache Iceberg's, for example — append the two Sedona entries to your existing list instead of pasting this block verbatim. Stock Amazon EMR sets no `spark.sql.extensions` default of its own, so on an unmodified cluster the value above is safe as-is.
    </Warning>
  </Step>

  <Step title="Validate the JSON">
    ```bash wrap theme={"system"}
    python3 -m json.tool configurations.json
    ```

    The command prints the formatted file if it is valid. Malformed JSON causes `create-cluster` to fail immediately with a parse error.
  </Step>
</Steps>

<Note>
  After the cluster is created, you can confirm these settings applied in the EMR console under **Clusters → your cluster → Configurations**, or by running `aws emr describe-cluster --cluster-id CLUSTER_ID --query 'Cluster.Configurations'`.
</Note>

### Bootstrap script

A bootstrap action is a script that Amazon EMR runs on every node of the cluster during creation, before Spark is installed. This script copies the WherobotsDB JARs from Amazon S3 onto each node and installs the `wherobots-sedona` Python client.

The API key is **not** handled here. Bootstrap actions run before Spark exists, and a shell `export` in a bootstrap script exits before any Spark process starts, so the variable never reaches them. The key is supplied by a cluster step instead — see [Create cluster](#create-cluster).

You do not run this script yourself. You create it on your local machine, upload it to Amazon S3, and reference its S3 path in the `--bootstrap-actions` argument of [Create cluster](#create-cluster). Amazon EMR retrieves and runs it on each node.

To create and upload the bootstrap script, complete the following steps:

<Steps>
  <Step title="Create the file on your local machine">
    In the working directory you established in [Prepare your command line](#prepare-your-command-line), create a file named `bootstrap-wherobotsdb.sh` using a text editor, and paste in the script below.

    ```bash wrap theme={"system"}
    #!/bin/bash
    set -euo pipefail

    # ---- Configuration: edit this section --------------------------------------
    S3_JARS="s3://YOUR_BUCKET_NAME/wherobots/jars"
    S3_WHEELS="s3://YOUR_BUCKET_NAME/wherobots/wheels"
    JARS_DIR="/jars"

    SEDONA_JAR="sedona-emr-spark-shaded-3.5_2.12-VERSION.jar"
    GEOTOOLS_JAR="geotools-wrapper-GEOTOOLS_VERSION.jar"
    SEDONA_WHEEL="wherobots_sedona-VERSION-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl"

    # The interpreter PySpark runs. On EMR 7.8.0 this is the system python3
    # (3.9); on emr-spark-8.0.0 set it to /usr/bin/python3.11 instead.
    PY_BIN="/usr/bin/python3"

    # Set to "false" only for Scala- or SQL-only clusters that do not need the
    # Python client. The notebook tests will not work without it.
    REQUIRE_PYTHON_CLIENT="true"
    # ---------------------------------------------------------------------------

    echo "[wherobots-bootstrap] Copying WherobotsDB JARs to ${JARS_DIR} ..."
    sudo mkdir -p "${JARS_DIR}"
    # Each file is copied by exact name, so s3:GetObject alone is sufficient.
    sudo aws s3 cp "${S3_JARS}/${SEDONA_JAR}"   "${JARS_DIR}/${SEDONA_JAR}"
    sudo aws s3 cp "${S3_JARS}/${GEOTOOLS_JAR}" "${JARS_DIR}/${GEOTOOLS_JAR}"

    echo "[wherobots-bootstrap] Verifying downloaded JARs:"
    ls -la "${JARS_DIR}/" | grep -E "sedona-emr|geotools-wrapper" || {
      echo "[wherobots-bootstrap] ERROR: expected JARs not found in ${JARS_DIR}" >&2
      exit 1
    }

    # ---- Python client ---------------------------------------------------------
    # INTERPRETER MATTERS: the wheel must be installed into the interpreter that
    # PySpark runs, which is not always the one `python3` resolves to. On EMR 7.8.0
    # they are the same; on emr-spark-8.0.0 Spark uses /usr/bin/python3.11 while
    # the shell's `python3` is 3.9, so installing with plain `python3 -m pip` puts
    # the client where Spark never looks. PY_BIN is set in the block above.
    [ -x "${PY_BIN}" ] || {
      echo "[wherobots-bootstrap] ERROR: PY_BIN=${PY_BIN} is not executable on this node." >&2
      exit 1
    }

    WHEEL_LOCAL="/tmp/${SEDONA_WHEEL}"
    if aws s3 cp "${S3_WHEELS}/${SEDONA_WHEEL}" "${WHEEL_LOCAL}"; then
      echo "[wherobots-bootstrap] Installing ${SEDONA_WHEEL} into ${PY_BIN} ..."
      sudo "${PY_BIN}" -m pip install --quiet "${WHEEL_LOCAL}"
      sudo "${PY_BIN}" -c 'import sedona; print("[wherobots-bootstrap] sedona installed:", sedona.__file__)'
    elif [ "${REQUIRE_PYTHON_CLIENT}" = "true" ]; then
      # Fail the cluster now, with the cause in the bootstrap log, rather than
      # letting the notebook tests die later with ModuleNotFoundError.
      echo "[wherobots-bootstrap] ERROR: could not fetch ${SEDONA_WHEEL} from ${S3_WHEELS}" >&2
      echo "[wherobots-bootstrap] Stage the wheel for this node's Python and architecture, or set REQUIRE_PYTHON_CLIENT=false." >&2
      exit 1
    else
      echo "[wherobots-bootstrap] Python client not installed (REQUIRE_PYTHON_CLIENT=false)."
    fi

    echo "[wherobots-bootstrap] Done."
    ```
  </Step>

  <Step title="Edit the configuration block">
    Replace the values in the `Configuration: edit this section` block at the top of the file. Leave the rest of the script unchanged.

    | Variable                     | Set to                                                                                                                            |
    | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
    | `S3_JARS`, `S3_WHEELS`       | Your bucket name in place of `YOUR_BUCKET_NAME`                                                                                   |
    | `SEDONA_JAR`, `GEOTOOLS_JAR` | The exact JAR filenames you uploaded to `wherobots/jars/`                                                                         |
    | `SEDONA_WHEEL`               | The exact wheel filename you will upload in [WHL file](#whl-file)                                                                 |
    | `PY_BIN`                     | `/usr/bin/python3` on EMR 7.8.0. Change it only if you are deploying on `emr-spark-8.0.0`, where it must be `/usr/bin/python3.11` |
    | `REQUIRE_PYTHON_CLIENT`      | Leave as `true` unless this is a Scala- or SQL-only cluster                                                                       |

    <Warning>
      If you add your own `pip install` lines, put them **after** every `aws` command in the script. The AWS CLI on Amazon Linux is rpm-managed and shares the system Python, so installing packages such as `pandas` or `geopandas` upgrades `python-dateutil` and `jmespath` out from under it. Any `aws` command that runs afterwards fails with `ModuleNotFoundError: No module named 'dateutil'`.
    </Warning>

    <Note>
      Install packages with `sudo "${PY_BIN}" -m pip` rather than hard-coding an interpreter. On EMR 7.8.0 `PY_BIN` and `python3` are the same binary, but on `emr-spark-8.0.0` they are not — using `${PY_BIN}` keeps the script correct on both.
    </Note>
  </Step>

  <Step title="Upload the script to Amazon S3">
    Run the following in your terminal, from the directory containing the file:

    ```bash wrap theme={"system"}
    aws s3 cp bootstrap-wherobotsdb.sh \
      s3://YOUR_BUCKET_NAME/wherobots/bootstrap/bootstrap-wherobotsdb.sh
    ```
  </Step>

  <Step title="Verify the upload">
    ```bash wrap theme={"system"}
    aws s3 ls s3://YOUR_BUCKET_NAME/wherobots/bootstrap/
    ```

    The file must appear with a non-zero size. Amazon EMR reads this object at cluster-creation time, so re-upload it after any edit — clusters do not pick up local changes.
  </Step>
</Steps>

### WHL file

The `wherobots-sedona` Python client is distributed as prebuilt wheels in the WherobotsDB release bundle, alongside the JARs. It is not published to PyPI.

You still install it with pip — you just point pip at the wheel file rather than at a package name. `pip install wherobots-sedona` cannot resolve, because there is no index to resolve it from; `pip install /path/to/wherobots_sedona-....whl` works normally, which is what the bootstrap script does.

The wheel filename encodes both a Python version tag and a CPU architecture, and both must match your cluster.

Pick the Python tag from your EMR release:

| EMR release                  | PySpark interpreter      | Python tag |
| ---------------------------- | ------------------------ | ---------- |
| **`emr-7.8.0`** (this guide) | `/usr/bin/python3` (3.9) | `cp39`     |
| Other 7.x releases           | `/usr/bin/python3` (3.9) | `cp39`     |
| `emr-spark-8.0.0`            | `/usr/bin/python3.11`    | `cp311`    |

Pick the architecture from your instance type:

| Instance family     | Architecture     | Architecture tag |
| ------------------- | ---------------- | ---------------- |
| `r8g`, `m7g`, `c7g` | Graviton / ARM64 | `aarch64`        |
| `m5`, `m6i`, `c5`   | Intel / AMD      | `x86_64`         |

The release bundle also contains `macosx_*` and `win_*` wheels. Those are for local development — cluster nodes always take a `manylinux` wheel.

<Warning>
  The Python tag must match the interpreter **PySpark** uses, not the one `python3` resolves to in your shell. On EMR 7.8.0 those are the same binary, so `cp39` is correct. They diverge on `emr-spark-8.0.0`, where Spark uses Python 3.11 while the shell's `python3` is still 3.9 — a wheel built for the wrong Python version installs without complaint and is never importable from Spark. Installing the wrong architecture fails outright with `not a supported wheel on this platform`.
</Warning>

To upload the wheel, complete the following steps:

<Steps>
  <Step title="Confirm your cluster's Python tag and architecture">
    Check your EMR release and the instance type you plan to launch against the tables above. The examples in this guide use `emr-7.8.0` on `r8g.xlarge`, which is Graviton, so they use the `cp39` / `aarch64` wheel.
  </Step>

  <Step title="Create the wheels prefix">
    In your terminal, run:

    ```bash wrap theme={"system"}
    aws s3api put-object --bucket YOUR_BUCKET_NAME --key wherobots/wheels/
    ```
  </Step>

  <Step title="Upload the wheel">
    Run this from the directory containing the wheel files Wherobots supplied:

    ```bash wrap theme={"system"}
    aws s3 cp wherobots_sedona-VERSION-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl \
      s3://YOUR_BUCKET_NAME/wherobots/wheels/
    ```
  </Step>

  <Step title="Verify the upload">
    ```bash wrap theme={"system"}
    aws s3 ls s3://YOUR_BUCKET_NAME/wherobots/wheels/
    ```

    The filename listed must match `SEDONA_WHEEL` in the bootstrap script exactly.
  </Step>
</Steps>

### Create IAM roles

This deployment uses three IAM roles. Confusing the first two is the most common setup mistake:

| Role                     | Used by                                                                          | Needs access to your bucket and API key? |
| ------------------------ | -------------------------------------------------------------------------------- | ---------------------------------------- |
| **Service role**         | Amazon EMR itself, to create and manage EC2 machines                             | No                                       |
| **EC2 instance profile** | The machines inside the cluster, which run the bootstrap script and the key step | **Yes**                                  |
| **Studio service role**  | EMR Studio, to store notebook files and attach Workspaces to clusters            | Bucket only                              |

Each role needs a *trust policy* naming the AWS service allowed to use it. In your terminal, create all three files first. They are not uploaded to S3 — the `aws iam` commands read them from disk and submit their contents to IAM, which stores them on the role:

```bash wrap theme={"system"}
cat > ec2-trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

cat > emr-trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"elasticmapreduce.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

cat > studio-trust.json <<'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
 "Principal":{"Service":"elasticmapreduce.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF
```

### Create Service Role

To create the role Amazon EMR uses to manage your cluster, complete the following steps in your terminal:

<Steps>
  <Step title="Create the role">
    ```bash wrap theme={"system"}
    aws iam create-role --role-name WherobotsEMRServiceRole \
      --assume-role-policy-document file://emr-trust.json
    ```
  </Step>

  <Step title="Attach the AWS managed policy">
    ```bash wrap theme={"system"}
    aws iam attach-role-policy --role-name WherobotsEMRServiceRole \
      --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEMRServicePolicy_v2
    ```

    `AmazonEMRServicePolicy_v2` alone is not sufficient for this setup. It scopes its EC2 permissions to resources carrying the `for-use-with-amazon-emr-managed-policies` tag, and it does not grant `iam:PassRole` for a custom instance profile name such as `WherobotsEMRInstanceRole`. The inline policy you add in the next two steps supplies both: explicit access to your subnet and security groups, and `iam:PassRole` for the instance profile.
  </Step>

  <Step title="Look up your security group IDs">
    ```bash wrap theme={"system"}
    aws ec2 describe-security-groups \
      --filters Name=vpc-id,Values=VPC_ID \
                Name=group-name,Values=ElasticMapReduce-master,ElasticMapReduce-slave \
      --query 'SecurityGroups[].{Name:GroupName,Id:GroupId}' --output table
    ```

    Record both group IDs. You substitute them into the policy in the next step as `SECURITY_GROUP_ID_MASTER` and `SECURITY_GROUP_ID_SLAVE`.

    <Note>
      If the command returns no rows, Amazon EMR has not yet created its managed groups in this VPC. This is normal before your first cluster launch. Take the "no managed groups yet" path:

      1. In the policy file in the next step, remove the two `security-group` ARNs from the `CreateInNetwork` statement, keeping its subnet ARN, and **delete the entire `ManageSecurityGroups` statement**. Its `Resource` array contains only those two ARNs, so removing them individually would leave an empty array, which `put-role-policy` rejects as invalid.
      2. In [Create cluster](#create-cluster), omit `EmrManagedMasterSecurityGroup=` and `EmrManagedSlaveSecurityGroup=` from `--ec2-attributes`. Amazon EMR then creates its default managed groups during launch. The `CreateDefaultSecurityGroupInVPC` statement is what permits this.
      3. Afterwards, re-run the lookup above, add the two ARNs back to `CreateInNetwork`, restore the `ManageSecurityGroups` statement, and re-apply the policy with `put-role-policy`.
    </Note>
  </Step>

  <Step title="Create the policy file">
    This policy grants the service role access to your specific subnet, security groups, and VPC, and allows it to pass the instance profile to EC2.

    Substitute `SUBNET_ID`, `SECURITY_GROUP_ID_MASTER`, `SECURITY_GROUP_ID_SLAVE`, `VPC_ID`, and `ACCOUNT_ID`, then run:

    ```bash wrap theme={"system"}
    cat > emr-service-role-policy.json <<'EOF'
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "CreateInNetwork",
          "Effect": "Allow",
          "Action": [
            "ec2:CreateNetworkInterface",
            "ec2:RunInstances",
            "ec2:CreateFleet",
            "ec2:CreateLaunchTemplate",
            "ec2:CreateLaunchTemplateVersion"
          ],
          "Resource": [
            "arn:aws:ec2:*:*:subnet/SUBNET_ID",
            "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_MASTER",
            "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_SLAVE"
          ]
        },
        {
          "Sid": "ManageSecurityGroups",
          "Effect": "Allow",
          "Action": [
            "ec2:AuthorizeSecurityGroupEgress",
            "ec2:AuthorizeSecurityGroupIngress",
            "ec2:RevokeSecurityGroupEgress",
            "ec2:RevokeSecurityGroupIngress"
          ],
          "Resource": [
            "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_MASTER",
            "arn:aws:ec2:*:*:security-group/SECURITY_GROUP_ID_SLAVE"
          ]
        },
        {
          "Sid": "CreateDefaultSecurityGroupInVPC",
          "Effect": "Allow",
          "Action": ["ec2:CreateSecurityGroup"],
          "Resource": ["arn:aws:ec2:*:*:vpc/VPC_ID"]
        },
        {
          "Sid": "PassRoleForEC2",
          "Effect": "Allow",
          "Action": "iam:PassRole",
          "Resource": "arn:aws:iam::ACCOUNT_ID:role/WherobotsEMRInstanceRole",
          "Condition": {
            "StringLike": {"iam:PassedToService": "ec2.amazonaws.com"}
          }
        }
      ]
    }
    EOF

    aws iam put-role-policy --role-name WherobotsEMRServiceRole \
      --policy-name wherobots-emr-network-access \
      --policy-document file://emr-service-role-policy.json
    ```
  </Step>

  <Step title="Verify both policies are attached">
    ```bash wrap theme={"system"}
    aws iam list-attached-role-policies --role-name WherobotsEMRServiceRole
    aws iam list-role-policies --role-name WherobotsEMRServiceRole
    ```

    The first command must list `AmazonEMRServicePolicy_v2`. The second must list `wherobots-emr-network-access`.
  </Step>
</Steps>

### Give Secret Manager Access

To store your Wherobots API key in AWS Secrets Manager and grant the cluster access to it, complete the following steps:

<Steps>
  <Step title="Create a Wherobots API key">
    Log in to [Wherobots Cloud](https://cloud.wherobots.com/), go to **Settings → API Keys**, and click **Create new key**.

    <Warning>
      Copy the key before leaving the page. It is shown only once.
    </Warning>
  </Step>

  <Step title="Store the key as a secret">
    In your terminal, run:

    Passing the key as a command-line argument writes it into your shell history and exposes it in the local process list. Read it into a `0600` file instead, hand that file to the CLI, then delete it:

    ```bash wrap theme={"system"}
    umask 077
    read -rs -p "Wherobots API key: " KEY && printf '%s' "$KEY" > api-key.txt && unset KEY

    aws secretsmanager create-secret --name wherobots/api-key \
      --secret-string file://api-key.txt

    rm -f api-key.txt
    ```

    `read -rs` keeps the key off the screen, `printf '%s'` writes it without a trailing newline, and `umask 077` makes the file readable only by you.

    Store the key as plaintext — no JSON wrapper and no trailing newline. The key step reads the value verbatim, so anything extra ends up inside the key.
  </Step>

  <Step title="Note the secret's ARN">
    ```bash wrap theme={"system"}
    aws secretsmanager describe-secret --secret-id wherobots/api-key --query ARN --output text
    ```

    The ARN ends in six random characters, for example `...:secret:wherobots/api-key-Ab3xY9`.
  </Step>

  <Step title="Create the access policy file">
    This policy grants the **instance profile role** — not the service role — permission to read the secret, read your bucket, and write cluster logs. It is applied to the role in [Create Instance Profile](#create-instance-profile).

    Substitute `YOUR_BUCKET_NAME`, `REGION_IDENTIFIER`, and `ACCOUNT_ID`, then run:

    ```bash wrap theme={"system"}
    cat > instance-profile-policy.json <<'EOF'
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": ["s3:GetObject", "s3:ListBucket"],
          "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*"]
        },
        {
          "Effect": "Allow",
          "Action": ["s3:PutObject", "s3:AbortMultipartUpload",
                     "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts"],
          "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
        },
        {
          "Effect": "Allow",
          "Action": ["secretsmanager:GetSecretValue"],
          "Resource": "arn:aws:secretsmanager:REGION_IDENTIFIER:ACCOUNT_ID:secret:wherobots/api-key-*"
        },
        {
          "Effect": "Allow",
          "Action": ["s3:GetObject", "s3:ListBucket"],
          "Resource": ["arn:aws:s3:::elasticmapreduce", "arn:aws:s3:::elasticmapreduce/*",
                       "arn:aws:s3:::*.elasticmapreduce/*"]
        }
      ]
    }
    EOF
    ```

    <Warning>
      Keep the trailing `-*` on the secret ARN. Secrets Manager appends six random characters to every secret ARN, so a policy pinned to the literal ARN breaks if the secret is ever recreated.
    </Warning>

    <Warning>
      Without the `s3:PutObject` statement, Amazon EMR cannot write to the log URI. A cluster that fails then reports `Unknown Error.` with no logs available for diagnosis.
    </Warning>
  </Step>
</Steps>

### Create Instance Profile

An *instance profile* is a wrapper that lets EC2 machines use an IAM role. The role by itself is not enough — EC2 can only assume a role that is wrapped in an instance profile of the same name.

To create it, complete the following steps in your terminal:

<Steps>
  <Step title="Create the role">
    ```bash wrap theme={"system"}
    aws iam create-role --role-name WherobotsEMRInstanceRole \
      --assume-role-policy-document file://ec2-trust.json
    ```
  </Step>

  <Step title="Attach the access policy">
    ```bash wrap theme={"system"}
    aws iam put-role-policy --role-name WherobotsEMRInstanceRole \
      --policy-name wherobots-emr-access \
      --policy-document file://instance-profile-policy.json
    ```
  </Step>

  <Step title="Wrap the role in an instance profile">
    ```bash wrap theme={"system"}
    aws iam create-instance-profile --instance-profile-name WherobotsEMRInstanceRole

    aws iam add-role-to-instance-profile \
      --instance-profile-name WherobotsEMRInstanceRole \
      --role-name WherobotsEMRInstanceRole
    ```
  </Step>

  <Step title="Verify the grants">
    ```bash wrap theme={"system"}
    aws iam list-role-policies --role-name WherobotsEMRInstanceRole
    aws iam get-instance-profile --instance-profile-name WherobotsEMRInstanceRole
    ```

    The second command must list `WherobotsEMRInstanceRole` under `Roles`. If it returns an empty list, the previous step did not complete.
  </Step>
</Steps>

### Create security groups

EMR Studio reaches the cluster over port `18888`, the Jupyter Enterprise Gateway. That requires two security groups: an **engine** group attached to the cluster's primary node, and a **workspace** group attached to the Workspace.

<Warning>
  Create these groups **before** launching the cluster. The engine group is attached to the primary node at launch through `AdditionalMasterSecurityGroups` and cannot be added to a running cluster. If you launch first, your only options are to recreate the cluster or to add an inbound rule for TCP `18888` to the cluster's existing primary security group.
</Warning>

In your terminal, create both groups and connect them:

```bash wrap theme={"system"}
ENG=$(aws ec2 create-security-group --group-name emr-studio-engine-sg \
  --description "EMR Studio engine" --vpc-id VPC_ID --query GroupId --output text)

WS=$(aws ec2 create-security-group --group-name emr-studio-workspace-sg \
  --description "EMR Studio workspace" --vpc-id VPC_ID --query GroupId --output text)

aws ec2 authorize-security-group-ingress --group-id $ENG \
  --ip-permissions "IpProtocol=tcp,FromPort=18888,ToPort=18888,UserIdGroupPairs=[{GroupId=$WS}]"

aws ec2 authorize-security-group-egress --group-id $WS \
  --ip-permissions "IpProtocol=tcp,FromPort=18888,ToPort=18888,UserIdGroupPairs=[{GroupId=$ENG}]"

aws ec2 create-tags --resources $ENG $WS \
  --tags Key=for-use-with-amazon-emr-managed-policies,Value=true

echo "engine=$ENG workspace=$WS"
```

Record both IDs. The next section uses the engine group as `ENGINE_SECURITY_GROUP_ID`, and [Create EMR Studio](#create-emr-studio) uses both.

<Note>
  The tag on the engine security group is required. Because the group is attached to the primary node at launch, the EMR service role needs `ec2:RunInstances` on it — which `AmazonEMRServicePolicy_v2` grants only for tagged security groups. The alternative is to add its ARN to the `CreateInNetwork` statement of the service role policy.
</Note>

<Note>
  `$ENG` and `$WS` are shell variables that exist only in the terminal session that created them. If you open a new terminal before finishing this guide, re-read the IDs with `aws ec2 describe-security-groups --filters Name=group-name,Values=emr-studio-engine-sg,emr-studio-workspace-sg --query 'SecurityGroups[].{Name:GroupName,Id:GroupId}' --output table`.
</Note>

### Create cluster

The following steps create an EMR cluster that runs WherobotsDB. You can also use an existing EMR cluster, but it must meet the requirements outlined in the [Before you start](#before-you-start) section, and it must already carry the engine security group on its primary node.

The API key reaches Spark through a *step* that runs immediately after the cluster starts. It reads the secret on the primary node and writes `spark.wherobots.api.key` into Spark's own configuration file, so every later session picks it up. This cannot be done in the bootstrap script, because Spark is not installed yet when bootstrap actions run.

In your terminal, save the following as `configure_key.py`:

```python theme={"system"}
"""Write the Wherobots API key into Spark's config on the primary node.

Runs as an EMR step, i.e. AFTER Spark is installed. This cannot live in the
bootstrap action, because /etc/spark/conf does not exist at bootstrap time.
"""
import subprocess
import sys

import boto3

CONF = "/etc/spark/conf/spark-defaults.conf"
SECRET_ID = "wherobots/api-key"
REGION = "REGION_IDENTIFIER"
PROP = "spark.wherobots.api.key"


def already_set(prop):
    with open(CONF) as fh:
        return any(line.strip().startswith(prop) for line in fh)


def append(line):
    subprocess.run(["sudo", "tee", "-a", CONF], input=(line + "\n").encode(),
                   check=True, stdout=subprocess.DEVNULL)


key = boto3.client("secretsmanager", region_name=REGION).get_secret_value(
    SecretId=SECRET_ID)["SecretString"].strip()
if not key:
    sys.exit("FATAL: secret is empty")
print(f"fetched key, length={len(key)} (value not logged)")

if already_set(PROP):
    print(f"{PROP} already present; leaving it unchanged")
else:
    append(f"{PROP} {key}")
print("DONE")
```

<Warning>
  Substitute `REGION_IDENTIFIER` with your actual region before uploading. Unlike the IAM policy files, this script is not read by the AWS CLI on your machine — it is uploaded verbatim and executed on the cluster's primary node, where a literal `REGION_IDENTIFIER` makes the boto3 client fail immediately. Because the step runs with `ActionOnFailure=TERMINATE_CLUSTER`, that failure tears down the cluster you just launched.
</Warning>

Substitute the region, then upload it to the `apps/` prefix, where the cluster's first step reads it from:

```bash wrap theme={"system"}
sed -i.bak "s/REGION_IDENTIFIER/$(aws configure get region)/" configure_key.py && rm -f configure_key.py.bak

grep '^REGION' configure_key.py   # confirm it now shows a real region

aws s3 cp configure_key.py s3://YOUR_BUCKET_NAME/apps/configure_key.py
```

<Note>
  `spark.redaction.regex` is set in [Configuration JSON](#configuration-json) rather than here, because it is not a secret and should apply from the moment the cluster starts. Spark's default value, `(?i)secret|password|token`, does not match `spark.wherobots.api.key`; without extending it the key appears in the Spark UI Environment tab and in the event logs written to S3.
</Note>

<Warning>
  This step writes to `/etc/spark/conf/spark-defaults.conf` on the **primary node only**. That covers EMR Studio notebook kernels and any `pyspark` or `spark-submit --deploy-mode client` session on the primary node. Jobs submitted with `--deploy-mode cluster` run their driver in a YARN container on a core node, which does not see this file — pass the key to those jobs explicitly with `--conf spark.wherobots.api.key=...` or as the `WHEROBOTS_API_KEY` environment variable.
</Warning>

To create the cluster, complete the following steps in your terminal:

<Steps>
  <Step title="Run create-cluster">
    ```bash wrap theme={"system"}
    aws emr create-cluster \
      --name "wherobotsdb-emr" \
      --release-label emr-7.8.0 \
      --applications Name=Spark Name=Livy Name=JupyterEnterpriseGateway \
      --log-uri s3://YOUR_BUCKET_NAME/emr-logs/ \
      --tags for-use-with-amazon-emr-managed-policies=true \
      --configurations file://configurations.json \
      --bootstrap-actions Path=s3://YOUR_BUCKET_NAME/wherobots/bootstrap/bootstrap-wherobotsdb.sh,Name=wherobots-install \
      --ec2-attributes InstanceProfile=WherobotsEMRInstanceRole,SubnetId=SUBNET_ID,KeyName=YOUR_KEY_PAIR,EmrManagedMasterSecurityGroup=SECURITY_GROUP_ID_MASTER,EmrManagedSlaveSecurityGroup=SECURITY_GROUP_ID_SLAVE,AdditionalMasterSecurityGroups=ENGINE_SECURITY_GROUP_ID \
      --service-role WherobotsEMRServiceRole \
      --instance-groups \
          InstanceGroupType=MASTER,InstanceCount=1,InstanceType=r8g.xlarge \
          InstanceGroupType=CORE,InstanceCount=1,InstanceType=r8g.xlarge \
      --steps '[{"Type":"Spark","Name":"configure-wherobots-key","ActionOnFailure":"TERMINATE_CLUSTER","Args":["--deploy-mode","client","s3://YOUR_BUCKET_NAME/apps/configure_key.py"]}]'
    ```

    Substitute `ENGINE_SECURITY_GROUP_ID` with the engine group ID from [Create security groups](#create-security-groups). This is what lets EMR Studio reach the cluster, and it cannot be added after launch.

    <Warning>
      `ActionOnFailure` is set to `TERMINATE_CLUSTER` deliberately. With `CONTINUE`, a failed key step leaves a cluster that reaches `WAITING` and looks healthy, but every spatial query fails later with an authentication error. Failing at launch surfaces the cause while the step logs are the obvious place to look.
    </Warning>

    The command returns a `ClusterId`. Use it as `CLUSTER_ID` in the steps that follow.

    <Warning>
      `--tags for-use-with-amazon-emr-managed-policies=true` is required. The EMR console adds this cluster tag automatically, but the CLI does not. EMR propagates cluster tags onto the EC2 instances it launches, and `AmazonEMRServicePolicy_v2` authorizes `ec2:RunInstances` on `instance/*` only when that request tag is present. Omitting it fails cluster creation with `VALIDATION_ERROR: Service role ... has insufficient EC2 permissions` — a message that does not mention tags, and produces no logs because no instance ever launches.
    </Warning>

    <Tip>
      Make sure the logs S3 path has a trailing slash.
    </Tip>
  </Step>

  <Step title="Wait for the cluster to be ready">
    ```bash wrap theme={"system"}
    aws emr describe-cluster --cluster-id CLUSTER_ID \
      --query 'Cluster.{State:Status.State,Dns:MasterPublicDnsName}'
    ```

    The state moves through `STARTING`, `BOOTSTRAPPING`, and `RUNNING` before reaching `WAITING`, which takes roughly five to ten minutes.
  </Step>

  <Step title="Confirm the bootstrap installed the Python client">
    Find the instance ID, then read the bootstrap log:

    ```bash wrap theme={"system"}
    aws s3 ls s3://YOUR_BUCKET_NAME/emr-logs/CLUSTER_ID/node/

    aws s3 cp s3://YOUR_BUCKET_NAME/emr-logs/CLUSTER_ID/node/INSTANCE_ID/bootstrap-actions/1/stdout.gz - \
      | gunzip | grep wherobots-bootstrap
    ```

    Expected output includes:

    ```text theme={"system"}
    [wherobots-bootstrap] Copying WherobotsDB JARs to /jars ...
    [wherobots-bootstrap] Installing wherobots_sedona-... into /usr/bin/python3 ...
    [wherobots-bootstrap] sedona installed: /usr/local/lib/python3.9/site-packages/sedona/__init__.py
    [wherobots-bootstrap] Done.
    ```

    <Note>
      Logs are archived to S3 on a five-minute cycle, so they may not appear immediately after the cluster is ready.
    </Note>
  </Step>
</Steps>

## EMR Studio Notebook

EMR Studio is the browser-based notebook environment for EMR. A **Studio** is the container; a **Workspace** is an individual notebook environment inside it that attaches to a cluster.

### Create EMR Studio

A Studio needs its own service role, which EMR Studio uses to store notebook files in your bucket and to attach Workspaces to clusters. To create the Studio, complete the following steps in your terminal:

<Steps>
  <Step title="Create the Studio service role">
    This role uses the `studio-trust.json` trust policy you created in [Create IAM roles](#create-iam-roles):

    ```bash wrap theme={"system"}
    aws iam create-role --role-name WherobotsEMRStudioServiceRole \
      --assume-role-policy-document file://studio-trust.json
    ```
  </Step>

  <Step title="Create and attach the Studio policy">
    Substitute `YOUR_BUCKET_NAME`, then run:

    ```bash wrap theme={"system"}
    cat > studio-policy.json <<'EOF'
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "StudioNotebookStorage",
          "Effect": "Allow",
          "Action": ["s3:PutObject", "s3:GetObject", "s3:GetEncryptionConfiguration",
                     "s3:ListBucket", "s3:DeleteObject"],
          "Resource": ["arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*"]
        },
        {
          "Sid": "DescribeClustersForWorkspaceAttach",
          "Effect": "Allow",
          "Action": ["elasticmapreduce:ListInstances",
                     "elasticmapreduce:DescribeCluster",
                     "elasticmapreduce:ListSteps"],
          "Resource": "*"
        },
        {
          "Sid": "NetworkDiscovery",
          "Effect": "Allow",
          "Action": ["ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups",
                     "ec2:DescribeSubnets", "ec2:DescribeVpcs",
                     "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface",
                     "ec2:CreateNetworkInterfacePermission"],
          "Resource": "*"
        }
      ]
    }
    EOF

    aws iam put-role-policy --role-name WherobotsEMRStudioServiceRole \
      --policy-name wherobots-emr-studio-access \
      --policy-document file://studio-policy.json
    ```
  </Step>

  <Step title="Create the Studio">
    Substitute the engine and workspace security group IDs from [Create security groups](#create-security-groups) if you are in a new terminal session, then run:

    ```bash wrap theme={"system"}
    aws emr create-studio \
      --name wherobotsdb-emr-studio \
      --auth-mode IAM \
      --vpc-id VPC_ID \
      --subnet-ids SUBNET_ID \
      --service-role WherobotsEMRStudioServiceRole \
      --workspace-security-group-id $WS \
      --engine-security-group-id $ENG \
      --default-s3-location s3://YOUR_BUCKET_NAME/studio/
    ```

    The command returns a `StudioId` and a `Url`. Save the URL — it is how you open the Studio.

    <Warning>
      The Studio's subnet must be the same one the cluster runs in, or Workspaces will not be able to reach it.
    </Warning>

    <Warning>
      Security groups cannot be added to a Studio after it is created. If you see the message *"You haven't set up the VPC, subnets, or security groups in the Studio for this Workspace"*, the Studio was created without them and you need to create a new one.
    </Warning>
  </Step>
</Steps>

### Launch notebook and attach cluster

Creating a Workspace has no CLI equivalent. To create one, complete the following steps in the AWS Console:

<Steps>
  <Step title="Open the Studio">
    Go to the `Url` returned by `create-studio`, or in the AWS Console navigate to **EMR → EMR Studio**, then select your Studio.
  </Step>

  <Step title="Create a Workspace">
    Select **Workspaces → Create Workspace**. Enter a name and leave the subnet at its default, which comes from the Studio.
  </Step>

  <Step title="Attach the cluster">
    In the Workspace, open the cluster panel on the left sidebar and attach it to the cluster you created. You can also select the cluster while creating the Workspace.
  </Step>

  <Step title="Launch and choose a kernel">
    Open the Workspace, create a new notebook, and select the **PySpark** kernel.
  </Step>
</Steps>

<Note>
  A Workspace in `Ready` status has its cluster and security group fields locked. Stop the Workspace first if you need to change which cluster it attaches to, then start it again.
</Note>

## Test in EMR Studio Notebook

These commands run in a notebook in your browser, not in your terminal. Nothing is installed in the notebook session and no API key is entered, because the bootstrap action and the `configure-wherobots-key` step have already supplied both.

To open the notebook, complete the following steps:

<Steps>
  <Step title="Open EMR Studio">
    In the AWS Console, navigate to **EMR → EMR Studio**, then select the Studio you created in [Create EMR Studio](#create-emr-studio). You can also open the `Url` returned by `aws emr create-studio`.

    To retrieve the URL again, run:

    ```bash wrap theme={"system"}
    aws emr list-studios --query 'Studios[].{Name:Name,Url:Url}' --output table
    ```
  </Step>

  <Step title="Open your Workspace">
    Select **Workspaces**, then select the Workspace you attached to your cluster. Its status must be `Ready`. If it is `Stopped`, select **Start** and wait for it to become `Ready`.
  </Step>

  <Step title="Confirm the cluster is attached">
    In the Workspace, open the cluster panel in the left sidebar and confirm your cluster is listed as attached. If it is not, attach it before continuing. See [Launch notebook and attach cluster](#launch-notebook-and-attach-cluster).
  </Step>

  <Step title="Create a PySpark notebook">
    Create a new notebook and select the **PySpark** kernel. The kernel must be PySpark; the other kernels do not connect to the Spark cluster.
  </Step>
</Steps>

Then run the following cells in order:

<Steps>
  <Step title="Confirm the client is installed against the interpreter Spark uses">
    ```python wrap theme={"system"}
    import sys, sedona

    print(sys.executable)   # /usr/bin/python3
    print(sedona.__file__)  # .../python3.9/site-packages/sedona/__init__.py
    ```

    If the import fails with `No module named 'sedona'`, the wheel did not install. Check the bootstrap log for `WARN: 'sedona' is NOT importable`.
  </Step>

  <Step title="Initialize Sedona">
    ```python wrap theme={"system"}
    from sedona.spark import *

    sedona = SedonaContext.create(spark)
    ```
  </Step>

  <Step title="Run the validation query">
    ```python wrap theme={"system"}
    sedona.sql("""
        SELECT
            ST_AsText(ST_Point(1.0, 2.0)) AS point,
            ST_Distance(ST_Point(0.0, 0.0), ST_Point(3.0, 4.0)) AS distance,
            ST_Area(ST_GeomFromText('POLYGON((0 0, 0 5, 5 5, 5 0, 0 0))')) AS area
    """).show()
    ```

    ```text theme={"system"}
    +-----------+--------+----+
    |      point|distance|area|
    +-----------+--------+----+
    |POINT (1 2)|     5.0|25.0|
    +-----------+--------+----+
    ```

    <Note>
      Because `spark.sql.extensions` is set, `ST_*` functions are also available through plain `spark.sql(...)` without calling `SedonaContext.create` first. The later cells rely on this.
    </Note>
  </Step>

  <Step title="Confirm geometries reach Python">
    This exercises the wheel rather than the JAR alone — the returned value is a Shapely object, not text.

    ```python wrap theme={"system"}
    spark.sql("SELECT ST_Point(1.0, 2.0) AS geom").collect()
    ```

    ```text theme={"system"}
    [Row(geom=<POINT (1 2)>)]
    ```
  </Step>

  <Step title="Confirm spatial functions run on executors">
    This requires the wheel to be present on every node, not just the primary.

    ```python wrap theme={"system"}
    df = spark.sql("SELECT ST_Point(CAST(id AS DOUBLE), CAST(id AS DOUBLE)) AS geom FROM range(0, 1000)")
    df.rdd.map(lambda r: r.geom.wkt).take(2)
    ```

    ```text theme={"system"}
    ['POINT (0 0)', 'POINT (1 1)']
    ```
  </Step>
</Steps>

<Note>
  If a query fails with `Wherobots Cloud credentials are required`, the session did not pick up `spark.wherobots.api.key`. Confirm the `configure-wherobots-key` step completed, then restart the notebook kernel — `spark-defaults.conf` is read only when a session is created.
</Note>

## How Wherobots authentication works

WherobotsDB validates your API key at startup and uses it for usage-based billing. Both operations call Wherobots Cloud endpoints that are part of the BYOS feature:

1. At startup, the JAR authenticates against `POST /apps/authenticate`, which returns a workload ID.
2. While the workload runs, the JAR periodically reports usage heartbeats to `POST /apps/report/{workload_id}`.

If BYOS is not enabled for your Organization, both endpoints return `403` and initialization fails. The same feature grants access to the release JARs. See [Before you start](#before-you-start).

You can supply the key in either of two ways:

| Method                                   | Notes                                                                                                                                                                                                                                                                                                                           |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spark.wherobots.api.key`                | What this guide configures, through the `configure-wherobots-key` step. Works for notebook kernels and client-mode sessions on the primary node.                                                                                                                                                                                |
| `WHEROBOTS_API_KEY` environment variable | The least visible option, since environment variables are not recorded in Spark's properties tables or event logs. Export it in the shell before running `pyspark` or `spark-submit`. An `export` in the bootstrap script does **not** work, and it cannot reach EMR Studio or Livy sessions whose driver launches inside YARN. |

For details on creating and rotating keys, see the [Wherobots API Keys documentation](/get-started/wherobots-cloud/api-keys).

## Feature differences from Wherobots Cloud

BYOS provides the WherobotsDB engine on your cluster. Features of the managed Wherobots Cloud platform are not included:

* The **Wherobots managed catalogs** — `wherobots`, `wherobots_open_data`, `wherobots_pro_data`, and the tutorial `examples_temp` catalog — are not available. Notebooks or jobs that reference them must be pointed at your own catalogs or read public data directly.
* The EMR JAR **does not bundle Apache Iceberg**. Add your own Iceberg runtime and catalog configuration if you need Iceberg tables, appending its extension to `spark.sql.extensions` rather than replacing the Sedona entries.
* Runtime conveniences that Wherobots Cloud preconfigures — the Spark checkpoint directory, visualization packages such as SedonaKepler, and pre-installed Python dependencies — must be set up on your cluster yourself. The bootstrap script covers the essentials.

## Troubleshooting

**`ClassNotFoundException`, `UNRESOLVED_ROUTINE`, `ST_*` functions not recognized, or `'JavaPackage' object is not callable` in PySpark**

* Confirm **both** `spark.jars` and `spark.yarn.dist.jars` are set as in [Configuration JSON](#configuration-json), and that the filenames exactly match the files in `/jars`. A missing `spark.jars` is the classic cause when running `pyspark` or `spark-submit` in client mode: the extensions silently fail to load on the driver.
* Check the bootstrap log at `<log-uri>/<cluster-id>/node/<instance-id>/bootstrap-actions/1/` and confirm `/jars` was populated on every node.
* If you set classpaths directly, **append** to Amazon EMR's defaults rather than replacing them. Overwriting `spark.driver.extraClassPath` or `spark.executor.extraClassPath` at the cluster level removes EMRFS from the classpath and breaks S3 access. Prefer the `spark.jars` approach.

**`ModuleNotFoundError: No module named 'sedona'`** — the Python client is not installed into the interpreter PySpark uses. Check the bootstrap log for the wheel-install section, and confirm `PY_BIN` matches your EMR release: `/usr/bin/python3` on EMR 7.8.0, `/usr/bin/python3.11` on `emr-spark-8.0.0`. If you are installing by hand, pass pip the wheel's path — `pip install wherobots-sedona` by name cannot resolve, since the package is not on PyPI.

**`ModuleNotFoundError: No module named 'pandas'`** — wheels up to and including `1.28.0` did not declare their pandas dependency. When deploying those versions, add a `sudo "${PY_BIN}" -m pip install pandas` line to the bootstrap script, after every `aws` command.

**Bootstrap fails with `Cannot uninstall setuptools ... RECORD file not found`** — pip is trying to replace the rpm-owned setuptools on Amazon Linux. Wheels up to and including `1.28.0` pinned `setuptools>=69`, which triggers this. Add `--ignore-installed` to the bootstrap's pip install line when deploying those versions. Later releases carry no setuptools pin.

**`UnsupportedClassVersionError`** — the cluster is running JDK 8. Use an EMR 7.x or later release, which ships JDK 17. See [Version compatibility](#version-compatibility).

**`NoSuchMethodError` at query time** — a Spark or Scala mismatch between the JAR and the EMR release, for example the `4.0_2.13` JAR on `emr-7.8.0`, which is Spark 3.5 and Scala 2.12. See [Version compatibility](#version-compatibility).

**Bootstrap fails with S3 `AccessDenied`** — the EMR EC2 instance profile is missing `s3:GetObject` on your staging prefix or on the bootstrap script itself. If you modified the bootstrap to use `--recursive` or `sync`, it additionally needs `s3:ListBucket` on the bucket.

**Cluster creation fails with `VALIDATION_ERROR: Service role ... has insufficient EC2 permissions`, and no logs are written** — the `for-use-with-amazon-emr-managed-policies=true` cluster tag is missing. See [Create cluster](#create-cluster). No instance ever launches, so there is nothing to log.

**A failed cluster reports `Unknown Error.` with no logs** — the instance profile cannot write to the log URI. Confirm the `s3:PutObject` statement from [Give Secret Manager Access](#give-secret-manager-access) is present, and that `--log-uri` ends in a trailing slash.

**`AccessDenied` reading public S3 datasets** — Amazon EMR's S3 filesystem signs every request with the instance profile, and cross-account access requires an identity-policy allow even for publicly readable buckets. Either grant the instance profile read access to those buckets, or read them anonymously with `s3a://` URIs plus `spark.hadoop.fs.s3a.bucket.<bucket-name>.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider`.

**Authentication errors from `SedonaContext.create`** — confirm `spark.wherobots.api.key` or `WHEROBOTS_API_KEY` is set, that the key has not expired, that BYOS is enabled for your Organization, and that your security groups and network policy allow outbound HTTPS to Wherobots Cloud.

## Support

For issues or questions, contact Wherobots support.