WherobotsDB Spark Parameters

The spatial join parameters on this page work in Wherobots Cloud and on Bring Your Own Spark (BYOS) clusters. In a Wherobots notebook, change one at runtime with sedona.conf.set, as in step 3 below. On a BYOS cluster, you also set the WherobotsDB Spark properties yourself; for the properties every BYOS cluster needs, see Spark properties.

Usage

To change the value of a parameter, do the following:

On a BYOS cluster, set the parameter through SparkConf:

val config = SedonaContext.builder().
      config("spark.serializer","org.apache.spark.serializer.KryoSerializer").
      config("spark.kryo.registrator", "org.apache.sedona.core.serde.SedonaKryoRegistrator").
      config("sedona.global.index","true")
      .getOrCreate()

Verify your configuration settings:

val sedonaConf = new SedonaConf(sedona.conf)
println(sedonaConf)

Set the WherobotsDB configuration parameter at runtime:
sedona.conf.set("sedona.global.index","false")

Spatial join parameters

WherobotsDB features an advanced spatial join algorithm, which does not require tuning to achieve good performance. Advanced spatial join analyzes both joined datasets and tunes the spatial join parameters automatically. It is enabled by default, and it ignores sedona.global.index, sedona.global.indextype, sedona.join.indexbuildside, and sedona.join.spatitionside. To tune those manually, set sedona.join.advanced to false.

Parameter	Default	Possible values	Description
`sedona.join.advanced`	`true`	`true`, `false`	Use the advanced spatial join algorithm.
`sedona.global.index`	`true`	`true`, `false`	Use a spatial index. Currently only supported in SQL range join and SQL distance join. Applies only when `sedona.join.advanced` is `false`.
`sedona.global.indextype`	`rtree`	`rtree`, `quadtree`	Spatial index type. Applies only when `sedona.global.index` is `true` and `sedona.join.advanced` is `false`.
`sedona.join.autoBroadcastJoinThreshold`	Same as `spark.sql.autoBroadcastJoinThreshold`	Any integer with a byte suffix, such as `10MB` or `512KB`	Maximum size in bytes of a table that is broadcast to all worker nodes in a join. Set it to `-1` to disable automatic broadcasting.
`sedona.join.gridtype`	`kdbtree`	`quadtree`, `kdbtree`	Spatial partitioning grid type for join queries.
`spark.sedona.join.knn.includeTieBreakers`	`false`	`true`, `false`	Include all ties in KNN join results, which can return more than k results.

Advanced join parameters

These parameters are for advanced users. Change them only if you know how they affect your join.

Parameter	Default	Possible values	Description
`sedona.join.indexbuildside`	`left`	`left`, `right`	The side WherobotsDB builds spatial indexes on. Applies only when `sedona.join.advanced` is `false`.
`sedona.join.numpartition`	`-1`	Any integer	Number of partitions for both sides of a join. With `-1`, the advanced spatial join algorithm tunes it from the size of both datasets; without advanced spatial join, `-1` uses the existing partitions of the dominant side.
`sedona.join.spatitionside`	`left`	`left`, `right`	The dominant side in the spatial partitioning stage. Applies only when `sedona.join.advanced` is `false`.
`sedona.join.optimizationmode`	`nonequi`	`all`, `none`, `nonequi`	How WherobotsDB optimizes spatial join SQL queries: `all` always optimizes them, even equi-joins; `none` disables the optimization; `nonequi` optimizes spatial joins that are not equi-joins.

Raster parameters

These parameters control the cache WherobotsDB uses when it reads out-DB rasters. Set them when you create the Spark session; they can’t be changed after the session is created. For how to tune them, see Tune out-DB raster reads.

Parameter	Default	Description
`spark.wherobots.raster.outdb.pool.size`	`100`	Number of out-DB rasters in each executor core’s cache. Set it to `0` to disable the cache.
`spark.wherobots.raster.outdb.cache.dir`	`/data/outdb-cache`	Directory for storing cached raster data.
`spark.wherobots.raster.outdb.readahead`	`64k`	The minimum number of bytes to read on each read of remote storage. A larger value may improve the performance of reading large rasters, but may hurt it if only a small portion of the raster file is needed.

When you read a large number of out-DB rasters from S3, spark.hadoop.fs.s3a.connection.maximum also matters: it limits the number of concurrent connections to S3. Wherobots runtimes may hold many concurrent connections because of out-DB raster pooling, so it defaults to 2000.

​Usage

​Spatial join parameters

​Advanced join parameters

​Raster parameters

Usage

Spatial join parameters

Advanced join parameters

Raster parameters