Skip to main content
Bring Your Own Spark (BYOS) lets you run WherobotsDB on a Spark cluster you own and operate. WherobotsDB is a drop-in replacement for Apache Sedona: you install its JAR on your existing cluster, and your existing code, SQL, and configuration continue to work unchanged. Your data never leaves your Databricks workspace. Wherobots does not run, manage, or connect to your cluster — the only outbound traffic is a lightweight authenticated call from the JAR to Wherobots Cloud for license metering. This guide covers Databricks. The WherobotsDB DBR JAR is fully compatible with Databricks Runtime, including Photon-enabled clusters.
BYOS is distinct from Bring Your Own Cloud (BYOC). Under BYOC, Wherobots runs its own managed compute inside your cloud account. Under BYOS, you run your own Spark cluster and Wherobots supplies only the engine.For Amazon EMR clusters, see Bring Your Own Spark (AWS EMR) instead.

For existing Apache Sedona users

If you already run Apache Sedona on Databricks, migrating to WherobotsDB takes two required changes, plus one optional improvement.
1

Replace the JAR

Swap the public Apache Sedona JAR for the WherobotsDB DBR JAR that Wherobots supplies, and update your init script to copy the new file.The WherobotsDB build adds the enterprise functions, Wherobots Cloud authentication, and the metering client.
2

Add your Wherobots API key

Add one Spark configuration entry to your cluster:
See How Wherobots authentication works.
3

Optionally switch the Python client

wherobots-sedona provides the fullest Python support for the WherobotsDB JARs, but apache-sedona continues to work. See Install the Python client.
All existing code, SQL queries, and cluster configuration continue to work. No code changes are required.

Before you start

The following requirements must be met within both your Wherobots Organization and your Databricks workspace:
  • An Admin account within a Professional, Innovation, or Enterprise Edition Organization.
    Wherobots Organization members with the User role can use a cluster an Admin has already set up, but cannot create the API key this deployment requires. See Organization Roles.
  • Community Edition is not supported. See Organization Editions or Upgrade Organization.
  • The BYOS feature enabled on your Organization. Contact Wherobots to enable it. BYOS unlocks both halves of this deployment: read access to the WherobotsDB release artifacts, and the runtime authentication and usage-reporting endpoints the JAR calls. Without it, those endpoints return 403 and cluster initialization fails.
  • A Wherobots API key. See How Wherobots authentication works.
  • A Databricks workspace with admin permissions.
  • Permission to create clusters, add init scripts, and install cluster libraries.
  • A supported Databricks Runtime — see Version compatibility.
  • A Unity Catalog volume if you plan to use Shared access mode clusters. Init scripts and JARs for Shared clusters must live in a volume, not in the workspace file system.
  • Outbound HTTPS (port 443) from the cluster to Wherobots Cloud, for license metering. A workspace with restricted egress needs Wherobots Cloud allowed; without it, SedonaContext.create fails to authenticate.
  • The WherobotsDB release artifacts from Wherobots: the sedona-dbr-spark-shaded-* JAR, the geotools-wrapper JAR, and the wherobots_sedona Python wheel. These are not published to Maven Central or PyPI.
Once BYOS is enabled for your Organization, a read-only WherobotsDB Releases storage source appears automatically in your Wherobots Cloud storage list. Go to Storage, open that source, and download the JARs and wheel from the release folder. If the source does not appear, BYOS is not yet enabled — contact Wherobots.

Version compatibility

Choose the JAR and Python wheel that match your Databricks Runtime. Wherobots recommends Long-Term Support (LTS) runtimes for production stability. A Spark or Scala mismatch surfaces at runtime as NoSuchMethodError or ClassNotFoundException.
DBR 16.4 LTS ships in two variants, one built against Scala 2.12 and one against Scala 2.13. They are separate runtime selections in the Databricks UI, and they need different JARs. Confirm which one your cluster uses before choosing — the release bundle contains both.
DBR 14.3 requires a JDK change. WherobotsDB JARs are compiled against JDK 17, and DBR 14.3 defaults to JDK 8. The cluster will fail with UnsupportedClassVersionError unless you switch it: add the Spark environment variable JNAME=zulu17-ca-amd64 under Advanced options → Spark → Environment variables. DBR 15.0 and later run on JDK 17 by default, so this applies only to 14.3.
Use the sedona-dbr-spark-shaded-* JAR. The sedona-emr-spark-shaded-* JAR in the same bundle is built for Amazon EMR and other vanilla Spark environments and will not work on Databricks Runtime, and the public sedona-spark-shaded-* JAR on Maven Central is the open-source Apache Sedona build without the WherobotsDB enterprise features.
WherobotsDB is versioned independently of Apache Sedona. Each WherobotsDB release is API-compatible with a specific Apache Sedona version, and the GeoTools wrapper version follows the Sedona version — so VERSION and GEOTOOLS_VERSION are different values. For example: Always use the exact filenames Wherobots delivered to you.

Install WherobotsDB on a Databricks cluster

Installing WherobotsDB involves staging its artifacts where the cluster can read them, adding an init script that copies the JARs into place at startup, and configuring the cluster to load them. Complete the sections in order.

Choose a storage location

Databricks offers two places to keep the JARs and init script. Your cluster’s access mode determines which one you can use.
Shared access mode clusters cannot read init scripts or JARs from the workspace file system. If you plan to use Shared clusters — or expect to later — stage everything in a Unity Catalog volume from the start. Moving an init script after the fact means editing every cluster that references it.
This guide uses /Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots as the example volume path. Substitute your own throughout.

Stage the JARs

1

Create the volume

In a Databricks notebook or the SQL editor, run:
You need CREATE VOLUME on the schema. See Create and manage volumes in the Databricks documentation.
2

Create the jars subdirectory

3

Upload both JARs

Using the Databricks CLI from your local machine:
You can also upload through the Catalog Explorer UI: open the volume, then select Upload to this volume.
4

Verify

Both JARs must appear with a non-zero size.

Create the init script

An init script runs on every node of the cluster during startup, before Spark begins accepting work. This one copies the WherobotsDB JARs into /databricks/jars, the directory Databricks adds to the Spark classpath. Write the script to the same place you staged the JARs in Stage the JARs. Use the tab matching that choice.
1

Write the script

Run this cell in a notebook, substituting your volume path in both the cat > destination and JARS_SRC.
2

Confirm it was written

The placeholders inside the heredoc are not substituted for you. <<'EOF' is quoted, so the shell writes the script verbatim — including a literal YOUR_CATALOG if you forget to replace it. Edit the cell before running it.
The init script fails the cluster if the JARs are missing, because set -euo pipefail makes the failed cp fatal. That is deliberate: a cluster that starts without the JARs looks healthy and fails every spatial query later.

Configure the cluster

1

Create the cluster

Go to Compute → Create compute, then choose:
  • A Databricks Runtime from Version compatibility.
  • An access mode consistent with your storage choice — Shared requires a Unity Catalog volume.
  • Either the Standard or Photon runtime engine. Both are supported.
2

Add the Spark configuration

Under Advanced options → Spark → Spark config, add these lines:
spark.sql.extensions must be a single line, and the value you set here replaces any value already in effect. If your cluster registers other extensions, append the two Sedona entries to your existing list rather than pasting this line verbatim.
Do not paste the API key itself into the Spark config. Values there are visible to anyone who can view the cluster. Use the {{secrets/SCOPE/KEY}} syntax shown above — Databricks resolves it at startup and redacts it from the UI. See How Wherobots authentication works.
3

Set the JDK on DBR 14.3

Only required on DBR 14.3. Under Advanced options → Spark → Environment variables, add:
Skip this step on DBR 15.0 and later, which already run JDK 17.
4

Attach the init script

Under Advanced options → Init scripts, add the path you created:Select the matching source type (Volumes or Workspace) in the dropdown beside the path.

Install the Python client

The wherobots-sedona Python client ships as prebuilt wheels in the WherobotsDB release bundle.
wherobots-sedona is not published to PyPI. Choosing PyPI as the library source and entering the package name fails to resolve, because there is no index to resolve it from. Install it from the wheel file instead, as below — the Python whl source does this for you, and a manual pip install /Volumes/.../wherobots_sedona-....whl works the same way.
The wheels contain compiled components, so there is no universal py3-none-any build. Each filename encodes a Python version tag and a platform tag, and both must match your cluster:
Ignore the macosx_* and win_* wheels in the bundle. Those are for local development, not cluster nodes. Stage the wheel alongside the JARs, using the same storage you chose in Choose a storage location. The examples below are for DBR 17.3 LTS on x86_64 workers — substitute the tags for your cluster.
1

Upload the wheel

2

Install it as a cluster library

On the cluster page, open the Libraries tab and select Install new → Python whl → Volumes, then enter the full wheel path:
Install at the cluster level, not with a notebook-scoped %pip. Spatial functions that run on executors need the client present on every node.
A wheel built for the wrong Python version fails to install with not a supported wheel on this platform, as does the wrong architecture. If you change the cluster’s Databricks Runtime later, revisit the wheel — DBR 14.3 uses Python 3.10 while 16.4 and 17.3 use 3.12.
apache-sedona from PyPI also works and can be installed as an ordinary PyPI library. Use the version matching your JAR’s Apache Sedona compatibility, from the table in Version compatibility. wherobots-sedona is preferred because it tracks the WherobotsDB JARs directly.

Validate the installation

Start the cluster, attach a notebook to it, and run the following cells in order.
1

Confirm the JARs were copied

If this returns nothing, the init script did not run or could not read the JARs. Check the cluster’s init script logs — see Troubleshooting.
2

Initialize Sedona

An authentication error here means the API key was not picked up. See How Wherobots authentication works.
3

Run the validation query

Because spark.sql.extensions is set, ST_* functions are also available through plain spark.sql(...) and in SQL cells without calling SedonaContext.create first.
4

Confirm geometries reach Python

This exercises the Python client rather than the JAR alone — the returned value is a Shapely object, not text.
5

Confirm spatial functions run on executors

This requires the client to be installed on every node, not just the driver.

Scala

WherobotsDB works the same way from Scala:

How Wherobots authentication works

WherobotsDB validates your API key at startup and uses it for usage-based billing. Both operations call Wherobots Cloud endpoints that are part of the BYOS feature:
  1. At startup, the JAR authenticates against POST /apps/authenticate, which returns a workload ID.
  2. While the workload runs, the JAR periodically reports usage heartbeats to POST /apps/report/{workload_id}.
If BYOS is not enabled for your Organization, both endpoints return 403 and initialization fails. The same feature grants access to the release artifacts. See Before you start.

Create an API key

1

Open the API Keys page

Log in to Wherobots Cloud and go to Settings → API Keys.
2

Create the key

Select Create new key, name it, and choose an expiration date. The default is one year. Select Submit.
3

Copy it immediately

The key is shown only once. Copy it before leaving the page.
For details on managing and rotating keys, see the Wherobots API Keys documentation.

Supply the key to the cluster

Databricks secrets are the recommended method, and the only one that keeps the key out of the cluster configuration.
1

Store the key in a secret scope

Using the Databricks CLI:
2

Reference it from the Spark config

Databricks resolves the reference when the cluster starts and displays [REDACTED] in the UI.
The key can also be supplied as the WHEROBOTS_API_KEY environment variable under Advanced options → Spark → Environment variables, which accepts the same {{secrets/...}} syntax.
Do not paste a raw API key into either the Spark config or an environment variable. Both are readable by anyone with view access to the cluster, and both are captured in cluster event logs.
If authentication fails, you will see an error when calling SedonaContext.create(spark). Confirm the key is correct and unexpired, that BYOS is enabled for your Organization, and that the cluster has outbound HTTPS to Wherobots Cloud.

Best practices

Runtime selection
  • Use an LTS runtime — 16.4 LTS or 17.3 LTS — for production.
  • Both Standard and Photon engines are supported.
  • Match the JAR’s Scala version to the runtime’s Scala version.
Cluster configuration
  • For large datasets, raise spark.driver.memory to accommodate global index building.
  • Raise spark.driver.maxResultSize — for example to 2g or 5g — if you hit out-of-memory errors when collecting results.
Storage
  • Use Unity Catalog volumes for Shared clusters, and for anything you expect to share across clusters.
  • Use workspace files only for single-user clusters.

Feature differences from Wherobots Cloud

BYOS provides the WherobotsDB engine on your cluster. Features of the managed Wherobots Cloud platform are not included:
  • The Wherobots managed catalogswherobots, wherobots_open_data, wherobots_pro_data, and the tutorial examples_temp catalog — are not available. Notebooks or jobs that reference them must be pointed at your own catalogs, including Unity Catalog, or read public data directly.
  • Runtime conveniences that Wherobots Cloud preconfigures — the Spark checkpoint directory, visualization packages such as SedonaKepler, and pre-installed Python dependencies — must be set up on your cluster yourself.

Troubleshooting

ClassNotFoundException, UNRESOLVED_ROUTINE, ST_* functions not recognized, or 'JavaPackage' object is not callable in PySpark
  • Confirm the JARs reached the cluster: ls -la /databricks/jars/sedona-dbr*.jar. If they are absent, the init script did not run or could not read the source path.
  • Check the init script output. Databricks writes it to the cluster log destination under init_scripts/, or to Driver logs on the cluster page.
  • Confirm the init script path and its source type — Volumes or Workspace — match where you actually stored it.
  • On Shared access mode clusters, confirm the init script and JARs are in a Unity Catalog volume. Shared clusters cannot read them from the workspace file system.
UnsupportedClassVersionError — the cluster is running JDK 8. On DBR 14.3, set the Spark environment variable JNAME=zulu17-ca-amd64. DBR 15.0 and later use JDK 17 by default. See Version compatibility. NoSuchMethodError at query time — a Spark or Scala mismatch between the JAR and the runtime. DBR 14.3 and 16.4 LTS use Scala 2.12; DBR 17.3 LTS uses Scala 2.13. See Version compatibility. ModuleNotFoundError: No module named 'sedona' — the Python client is not installed, or was installed on the notebook rather than the cluster. Install the wheel as a cluster library so it reaches every node. If you are installing by hand, give pip the wheel’s path — wherobots-sedona by name cannot resolve, since the package ships only in the release bundle and not on PyPI. Spatial functions work on the driver but fail inside rdd.map or a UDF — the Python client is installed on the driver only. Reinstall it as a cluster library and restart the cluster. Authentication errors from SedonaContext.create — confirm spark.wherobots.api.key resolves. A common cause is a secret scope or key name that does not match the {{secrets/SCOPE/KEY}} reference, which leaves the property set to the literal placeholder text. Also confirm the key has not expired, that BYOS is enabled for your Organization, and that the cluster has outbound HTTPS to Wherobots Cloud.

Appendix: Supported CRS transformations

WherobotsDB performs coordinate reference system transformations through ST_Transform, which accepts three CRS formats for both source and target.

Authority codes

WherobotsDB uses spatialreference.org as its CRS database, which covers several authorities:
Browse the available codes at spatialreference.org or epsg.io.

PROJ strings

Use PROJ.4 format for custom projections.

OGC WKT

Use OGC Well-Known Text v1 for a full CRS definition:

Behavior worth knowing

  • Coordinate order is lon/lat. If your data is stored lat/lon, apply ST_FlipCoordinates first.
  • Transformations run in strict mode. An invalid transformation raises an error rather than returning an approximation.
  • The source CRS can be inferred. If you omit it, WherobotsDB reads the geometry’s SRID, which you can set with ST_SetSRID:
For the full function reference, see the CRS transformation documentation.

Support

For issues or questions, contact Wherobots support.