Bring Your Own Spark (Databricks)

This guide installs WherobotsDB on Databricks with Bring Your Own Spark (BYOS). Your data never leaves your Databricks workspace. For what BYOS is and what the Databricks and EMR deployments share, see Bring Your Own Spark (BYOS). The WherobotsDB DBR JAR is fully compatible with Databricks Runtime, including Photon-enabled clusters.

For Amazon EMR clusters, see Bring Your Own Spark (AWS EMR) instead.

For existing Apache Sedona users

If you already run Apache Sedona on Databricks, migrating to WherobotsDB takes two required changes, plus one optional improvement.

Replace the JAR

Swap the public Apache Sedona JAR for the WherobotsDB DBR JAR that Wherobots supplies, and update your init script to copy the new file.

Old JAR	New JAR
`sedona-spark-shaded-3.5_2.12-*.jar`	`sedona-dbr-spark-shaded-3.5_2.12-*.jar`
`sedona-spark-shaded-4.0_2.13-*.jar`	`sedona-dbr-spark-shaded-4.0_2.13-*.jar`

The WherobotsDB build adds the enterprise functions, Wherobots Cloud authentication, and the metering client.

Add your Wherobots API key

Add one Spark configuration entry to your cluster:

spark.wherobots.api.key {{secrets/YOUR_SCOPE/wherobots-api-key}}

See How Wherobots authentication works.

Optionally switch the Python client

wherobots-sedona provides the fullest Python support for the WherobotsDB JARs, but apache-sedona continues to work. See Install the Python client.

All existing code, SQL queries, and cluster configuration continue to work. No code changes are required.

Before you start

The following requirements must be met within both your Wherobots Organization and your Databricks workspace:

Wherobots Requirements

The Wherobots requirements for BYOS: an account in a Professional, Innovation, or Enterprise Edition Organization, the BYOS feature enabled on your Organization, and a Wherobots API key.

Databricks Requirements

A Databricks workspace with admin permissions.
Permission to create clusters, add init scripts, and install cluster libraries.
A supported Databricks Runtime — see Version compatibility.
A Unity Catalog volume if you plan to use Shared access mode clusters. Init scripts and JARs for Shared clusters must live in a volume, not in the workspace file system.
Outbound HTTPS (port 443) from the cluster to Wherobots Cloud, for license metering. A workspace with restricted egress needs Wherobots Cloud allowed; without it, SedonaContext.create fails to authenticate.
The WherobotsDB release artifacts from Wherobots: the sedona-dbr-spark-shaded-* JAR, the geotools-wrapper JAR, and the wherobots_sedona Python wheel.

Version compatibility

Choose the JAR and Python wheel that match your Databricks Runtime. Wherobots recommends Long-Term Support (LTS) runtimes for production stability.

Databricks Runtime	Spark	Scala	Python	JDK	JAR artifact	Wheel tag
14.3 LTS	3.5.0	2.12	3.10	8 by default — see the warning below	`sedona-dbr-spark-shaded-3.5_2.12-VERSION.jar`	`cp310`
16.4 LTS (Scala 2.12)	3.5.2	2.12	3.12	17	`sedona-dbr-spark-shaded-3.5_2.12-VERSION.jar`	`cp312`
16.4 LTS (Scala 2.13)	3.5.2	2.13	3.12	17	`sedona-dbr-spark-shaded-3.5_2.13-VERSION.jar`	`cp312`
17.3 LTS	4.0.0	2.13	3.12	17	`sedona-dbr-spark-shaded-4.0_2.13-VERSION.jar`	`cp312`

A Spark or Scala mismatch surfaces at runtime as NoSuchMethodError or ClassNotFoundException.

DBR 16.4 LTS ships in two variants, one built against Scala 2.12 and one against Scala 2.13. They are separate runtime selections in the Databricks UI, and they need different JARs. Confirm which one your cluster uses before choosing — the release bundle contains both.

DBR 14.3 requires a JDK change. WherobotsDB JARs are compiled against JDK 17, and DBR 14.3 defaults to JDK 8. The cluster will fail with UnsupportedClassVersionError unless you switch it: add the Spark environment variable JNAME=zulu17-ca-amd64 under Advanced options → Spark → Environment variables. DBR 15.0 and later run on JDK 17 by default, so this applies only to 14.3.

Use the sedona-dbr-spark-shaded-* JAR. The sedona-emr-spark-shaded-* JAR in the same bundle is built for Amazon EMR and other vanilla Spark environments and will not work on Databricks Runtime, and the public sedona-spark-shaded-* JAR on Maven Central is the open-source Apache Sedona build without the WherobotsDB enterprise features.

VERSION and GEOTOOLS_VERSION are different values. For how WherobotsDB versions map to Apache Sedona and GeoTools wrapper versions, see Versions. Always use the exact filenames Wherobots delivered to you.

Install WherobotsDB on a Databricks cluster

Installing WherobotsDB involves staging its artifacts where the cluster can read them, adding an init script that copies the JARs into place at startup, and configuring the cluster to load them. Complete the sections in order.

#	Section	Purpose
1	Choose a storage location	Decide between Unity Catalog volumes and the workspace file system, based on your cluster access mode.
2	Stage the JARs	Upload the WherobotsDB and GeoTools JARs.
3	Create the init script	Write the script that copies the JARs into the cluster’s JAR directory at startup.
4	Configure the cluster	Set the Spark properties, attach the init script, and set the JDK on DBR 14.3.
5	Install the Python client	Add `wherobots-sedona` as a cluster library.
6	Validate the installation	Confirm WherobotsDB is installed, licensed, and running.

Choose a storage location

Databricks offers two places to keep the JARs and init script. Your cluster’s access mode determines which one you can use.

Storage	Path form	Use when
Unity Catalog volume	`/Volumes/CATALOG/SCHEMA/NAME`	Required for Shared access mode clusters. Works for all access modes.
Workspace files	`/Workspace/Shared/...`	Single user access mode clusters only.

Shared access mode clusters cannot read init scripts or JARs from the workspace file system. If you plan to use Shared clusters — or expect to later — stage everything in a Unity Catalog volume from the start. Moving an init script after the fact means editing every cluster that references it.

This guide uses /Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots as the example volume path. Substitute your own throughout.

Stage the JARs

Unity Catalog volume
Workspace files

Create the volume

In a Databricks notebook or the SQL editor, run:

CREATE VOLUME IF NOT EXISTS YOUR_CATALOG.YOUR_SCHEMA.wherobots;

You need CREATE VOLUME on the schema. See Create and manage volumes in the Databricks documentation.

Create the jars subdirectory

dbutils.fs.mkdirs("/Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/jars")

Upload both JARs

Using the Databricks CLI from your local machine:

VOLUME=/Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/jars

databricks fs cp sedona-dbr-spark-shaded-3.5_2.12-VERSION.jar dbfs:$VOLUME/
databricks fs cp geotools-wrapper-GEOTOOLS_VERSION.jar        dbfs:$VOLUME/

You can also upload through the Catalog Explorer UI: open the volume, then select Upload to this volume.

Verify

display(dbutils.fs.ls("/Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/jars"))

Both JARs must appear with a non-zero size.

Create the directory

In a Databricks notebook:

%sh
mkdir -p /Workspace/Shared/wherobots/jars

Upload both JARs

In the Databricks UI, navigate to Workspace → Shared → wherobots → jars, then select Import and upload both JAR files.

Verify

%sh
ls -la /Workspace/Shared/wherobots/jars/

This path works only for Single user access mode clusters. See Choose a storage location.

Create the init script

An init script runs on every node of the cluster during startup, before Spark begins accepting work. This one copies the WherobotsDB JARs into /databricks/jars, the directory Databricks adds to the Spark classpath. Write the script to the same place you staged the JARs in Stage the JARs. Use the tab matching that choice.

Unity Catalog volume
Workspace files

Write the script

Run this cell in a notebook, substituting your volume path in both the cat > destination and JARS_SRC.

%sh
mkdir -p /Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/init

cat > /Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/init/wherobots-init.sh <<'EOF'
#!/bin/bash
set -euo pipefail

JARS_SRC="/Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/jars"

echo "[wherobots-init] Copying WherobotsDB JARs from ${JARS_SRC} ..."
cp "${JARS_SRC}"/*.jar /databricks/jars/

echo "[wherobots-init] Verifying:"
ls -la /databricks/jars/sedona-dbr*.jar /databricks/jars/geotools-wrapper*.jar

echo "[wherobots-init] Done."
EOF

Confirm it was written

%sh
cat /Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/init/wherobots-init.sh

Write the script

Run this cell in a notebook, substituting your path in both the cat > destination and JARS_SRC if you did not use the default.

%sh
mkdir -p /Workspace/Shared/wherobots/init

cat > /Workspace/Shared/wherobots/init/wherobots-init.sh <<'EOF'
#!/bin/bash
set -euo pipefail

JARS_SRC="/Workspace/Shared/wherobots/jars"

echo "[wherobots-init] Copying WherobotsDB JARs from ${JARS_SRC} ..."
cp "${JARS_SRC}"/*.jar /databricks/jars/

echo "[wherobots-init] Verifying:"
ls -la /databricks/jars/sedona-dbr*.jar /databricks/jars/geotools-wrapper*.jar

echo "[wherobots-init] Done."
EOF

Confirm it was written

%sh
cat /Workspace/Shared/wherobots/init/wherobots-init.sh

This path works only for Single user access mode clusters. See Choose a storage location.

The placeholders inside the heredoc are not substituted for you. <<'EOF' is quoted, so the shell writes the script verbatim — including a literal YOUR_CATALOG if you forget to replace it. Edit the cell before running it.

The init script fails the cluster if the JARs are missing, because set -euo pipefail makes the failed cp fatal. That is deliberate: a cluster that starts without the JARs looks healthy and fails every spatial query later.

Configure the cluster

Create the cluster

Go to Compute → Create compute, then choose:

A Databricks Runtime from Version compatibility.
An access mode consistent with your storage choice — Shared requires a Unity Catalog volume.
Either the Standard or Photon runtime engine. Both are supported.

Add the Spark configuration

Under Advanced options → Spark → Spark config, add these lines:

spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
spark.sedona.enableParserExtension false
spark.sql.extensions org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
spark.wherobots.api.key {{secrets/YOUR_SCOPE/wherobots-api-key}}

Property	Purpose
`spark.serializer`, `spark.kryo.registrator`, `spark.sedona.enableParserExtension`, `spark.sql.extensions`	Shared by every BYOS deployment. See Spark properties
`spark.wherobots.api.key`	Your Wherobots API key, read from a Databricks secret

For the optional spatial join parameters, see WherobotsDB Spark Parameters.

spark.sql.extensions must be a single line, and the value you set here replaces any value already in effect. If your cluster registers other extensions, append the two Sedona entries to your existing list rather than pasting this line verbatim.

Do not paste the API key itself into the Spark config. Values there are visible to anyone who can view the cluster. Use the {{secrets/SCOPE/KEY}} syntax shown above — Databricks resolves it at startup and redacts it from the UI. See How Wherobots authentication works.

Set the JDK on DBR 14.3

Only required on DBR 14.3. Under Advanced options → Spark → Environment variables, add:

JNAME=zulu17-ca-amd64

Skip this step on DBR 15.0 and later, which already run JDK 17.

Attach the init script

Under Advanced options → Init scripts, add the path you created:

Storage	Path to enter
Unity Catalog volume	`/Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/init/wherobots-init.sh`
Workspace files	`/Workspace/Shared/wherobots/init/wherobots-init.sh`

Select the matching source type (Volumes or Workspace) in the dropdown beside the path.

Install the Python client

The wherobots-sedona Python client ships as prebuilt wheels in the WherobotsDB release bundle.

wherobots-sedona is not published to PyPI. Choosing PyPI as the library source and entering the package name fails to resolve, because there is no index to resolve it from. Install it from the wheel file instead, as below — the Python whl source does this for you, and a manual pip install /Volumes/.../wherobots_sedona-....whl works the same way.

The wheels contain compiled components, so there is no universal py3-none-any build. Each filename encodes a Python version tag and a platform tag, and both must match your cluster:

wherobots_sedona-VERSION-PYTAG-PYTAG-PLATFORM.whl

Component	Value for Databricks
`PYTAG`	From the Wheel tag column in Version compatibility — `cp310` on DBR 14.3, `cp312` on 16.4 and 17.3
`PLATFORM`	`manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64` on Intel or AMD workers, or the `aarch64` equivalent on Graviton workers

Ignore the macosx_* and win_* wheels in the bundle. Those are for local development, not cluster nodes. Stage the wheel alongside the JARs, using the same storage you chose in Choose a storage location. The examples below are for DBR 17.3 LTS on x86_64 workers — substitute the tags for your cluster.

Unity Catalog volume
Workspace files

Upload the wheel

VOLUME=/Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/wheels
databricks fs mkdirs dbfs:$VOLUME

databricks fs cp \
  wherobots_sedona-VERSION-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl \
  dbfs:$VOLUME/

Install it as a cluster library

On the cluster page, open the Libraries tab and select Install new → Python whl → Volumes, then enter the full wheel path:

/Volumes/YOUR_CATALOG/YOUR_SCHEMA/wherobots/wheels/wherobots_sedona-VERSION-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

Upload the wheel

In the Databricks UI, navigate to Workspace → Shared → wherobots, create a wheels folder, then select Import and upload the wheel file.

Install it as a cluster library

On the cluster page, open the Libraries tab and select Install new → Python whl → Workspace, then enter the full wheel path:

/Workspace/Shared/wherobots/wheels/wherobots_sedona-VERSION-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

This path works only for Single user access mode clusters, the same constraint that applies to the JARs and the init script.

Install at the cluster level, not with a notebook-scoped %pip. Spatial functions that run on executors need the client present on every node.

A wheel built for the wrong Python version fails to install with not a supported wheel on this platform, as does the wrong architecture. If you change the cluster’s Databricks Runtime later, revisit the wheel — DBR 14.3 uses Python 3.10 while 16.4 and 17.3 use 3.12.

apache-sedona from PyPI also works and can be installed as an ordinary PyPI library. Use the version matching your JAR’s Apache Sedona compatibility, from the table in Version compatibility. wherobots-sedona is preferred because it tracks the WherobotsDB JARs directly.

Validate the installation

Start the cluster, attach a notebook to it, and run the following cells in order.

Confirm the JARs were copied

%sh
ls -la /databricks/jars/sedona-dbr*.jar /databricks/jars/geotools-wrapper*.jar

If this returns nothing, the init script did not run or could not read the JARs. Check the cluster’s init script logs — see Troubleshooting.

Initialize Sedona

from sedona.spark import *

sedona = SedonaContext.create(spark)

An authentication error here means the API key was not picked up. See How Wherobots authentication works.

Run the validation query

sedona.sql("""
    SELECT
        ST_AsText(ST_Point(1.0, 2.0)) AS point,
        ST_Distance(ST_Point(0.0, 0.0), ST_Point(3.0, 4.0)) AS distance,
        ST_Area(ST_GeomFromText('POLYGON((0 0, 0 5, 5 5, 5 0, 0 0))')) AS area
""").show()

+-----------+--------+----+
|      point|distance|area|
+-----------+--------+----+
|POINT (1 2)|     5.0|25.0|
+-----------+--------+----+

Because spark.sql.extensions is set, ST_* functions are also available through plain spark.sql(...) and in SQL cells without calling SedonaContext.create first.

Confirm geometries reach Python

This exercises the Python client rather than the JAR alone — the returned value is a Shapely object, not text.

spark.sql("SELECT ST_Point(1.0, 2.0) AS geom").collect()

[Row(geom=<POINT (1 2)>)]

Confirm spatial functions run on executors

This requires the client to be installed on every node, not just the driver.

df = spark.sql("SELECT ST_Point(CAST(id AS DOUBLE), CAST(id AS DOUBLE)) AS geom FROM range(0, 1000)")
df.rdd.map(lambda r: r.geom.wkt).take(2)

['POINT (0 0)', 'POINT (1 1)']

Scala

WherobotsDB works the same way from Scala:

import org.apache.sedona.spark.SedonaContext

val sedona = SedonaContext.create(spark)
sedona.sql("SELECT ST_AsText(ST_Point(1.0, 2.0)) AS point").show()

How Wherobots authentication works

WherobotsDB validates your API key at startup against Wherobots Cloud endpoints that are part of the BYOS feature, and uses it for usage-based billing. See How Wherobots authentication works in the BYOS overview, and Create an API key if you don’t have one yet.

Supply the key to the cluster

Databricks secrets are the recommended method, and the only one that keeps the key out of the cluster configuration.

Store the key in a secret scope

Using the Databricks CLI:

databricks secrets create-scope YOUR_SCOPE

databricks secrets put-secret YOUR_SCOPE wherobots-api-key \
  --string-value "YOUR_WHEROBOTS_API_KEY"

Reference it from the Spark config

spark.wherobots.api.key {{secrets/YOUR_SCOPE/wherobots-api-key}}

Databricks resolves the reference when the cluster starts and displays [REDACTED] in the UI.

The key can also be supplied as the WHEROBOTS_API_KEY environment variable under Advanced options → Spark → Environment variables, which accepts the same {{secrets/...}} syntax.

Do not paste a raw API key into either the Spark config or an environment variable. Both are readable by anyone with view access to the cluster, and both are captured in cluster event logs.

If authentication fails, you will see an error when calling SedonaContext.create(spark). Confirm the key is correct and unexpired, that BYOS is enabled for your Organization, and that the cluster has outbound HTTPS to Wherobots Cloud.

Best practices

Runtime selection

Use an LTS runtime — 16.4 LTS or 17.3 LTS — for production.
Both Standard and Photon engines are supported.
Match the JAR’s Scala version to the runtime’s Scala version.

Cluster configuration

For large datasets, raise spark.driver.memory to accommodate global index building.
Raise spark.driver.maxResultSize — for example to 2g or 5g — if you hit out-of-memory errors when collecting results.

Storage

Use Unity Catalog volumes for Shared clusters, and for anything you expect to share across clusters.
Use workspace files only for single-user clusters.

Feature differences from Wherobots Cloud

The feature differences that apply to every BYOS deployment also apply on Databricks. On Databricks, the catalogs you point notebooks and jobs at can include Unity Catalog.

Troubleshooting

ClassNotFoundException, UNRESOLVED_ROUTINE, ST_* functions not recognized, or 'JavaPackage' object is not callable in PySpark

Confirm the JARs reached the cluster: ls -la /databricks/jars/sedona-dbr*.jar. If they are absent, the init script did not run or could not read the source path.
Check the init script output. Databricks writes it to the cluster log destination under init_scripts/, or to Driver logs on the cluster page.
Confirm the init script path and its source type — Volumes or Workspace — match where you actually stored it.
On Shared access mode clusters, confirm the init script and JARs are in a Unity Catalog volume. Shared clusters cannot read them from the workspace file system.

UnsupportedClassVersionError — the cluster is running JDK 8. On DBR 14.3, set the Spark environment variable JNAME=zulu17-ca-amd64. DBR 15.0 and later use JDK 17 by default. See Version compatibility. NoSuchMethodError at query time — a Spark or Scala mismatch between the JAR and the runtime. DBR 14.3 LTS uses Scala 2.12, DBR 16.4 LTS comes in Scala 2.12 and Scala 2.13 variants, and DBR 17.3 LTS uses Scala 2.13. See Version compatibility. ModuleNotFoundError: No module named 'sedona' — the Python client is not installed, or was installed on the notebook rather than the cluster. Install the wheel as a cluster library so it reaches every node. If you are installing by hand, give pip the wheel’s path — wherobots-sedona by name cannot resolve, since the package ships only in the release bundle and not on PyPI. Spatial functions work on the driver but fail inside rdd.map or a UDF — the Python client is installed on the driver only. Reinstall it as a cluster library and restart the cluster. Authentication errors from SedonaContext.create — confirm spark.wherobots.api.key resolves. A common cause is a secret scope or key name that does not match the {{secrets/SCOPE/KEY}} reference, which leaves the property set to the literal placeholder text. Also confirm the key has not expired, that BYOS is enabled for your Organization, and that the cluster has outbound HTTPS to Wherobots Cloud.

Support

For issues or questions, contact Wherobots support.

Start here

Notebooks

Editors and AI assistants

VS Code extension

MCP server and agent skills

dbt

Apache Airflow

Storage management

Task schedules

Deployment and monitoring

Bring Your Own Spark (Databricks)

For existing Apache Sedona users

Before you start

Version compatibility

Install WherobotsDB on a Databricks cluster

Choose a storage location

Stage the JARs

Create the init script

Configure the cluster

Install the Python client

Validate the installation

Scala

How Wherobots authentication works

Supply the key to the cluster

Best practices

Feature differences from Wherobots Cloud

Troubleshooting

Support

​For existing Apache Sedona users

​Before you start

​Version compatibility

​Install WherobotsDB on a Databricks cluster

​Choose a storage location

​Stage the JARs

​Create the init script

​Configure the cluster

​Install the Python client

​Validate the installation

​Scala

​How Wherobots authentication works

​Supply the key to the cluster

​Best practices

​Feature differences from Wherobots Cloud

​Troubleshooting

​Support

For existing Apache Sedona users

Before you start

Version compatibility

Install WherobotsDB on a Databricks cluster

Choose a storage location

Stage the JARs

Create the init script

Configure the cluster

Install the Python client

Validate the installation

Scala

How Wherobots authentication works

Supply the key to the cluster

Best practices

Feature differences from Wherobots Cloud

Troubleshooting

Support