ArcGIS Feature Service Reader

The arcgis data source reads ArcGIS Feature Service layers into a WherobotsDB DataFrame. It also reads Map Service layers that support the query operation. An ArcGIS Feature Service exposes vector features (points, lines, polygons, and their attributes) over the ArcGIS REST /query API. WherobotsDB reads a layer directly over HTTP, with no intermediate export to a file, and pushes filtering, column pruning, and limits down to the server so only the data you need is transferred.

Usage

Point the reader at a layer URL, a service URL ending in /<layerIndex>. This example reads the public U.S. Wind Turbine Database layer hosted on ArcGIS Online:

df = sedona.read.format("arcgis").load(
    "https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/US_Wind_Turbine_Database/FeatureServer/0"
)
df.printSchema()

The schema contains one column per attribute field in the layer, followed by a geometry column that carries the layer’s spatial reference (SRID). See Data types for how ArcGIS field types map to Spark types. Filter, project, and limit the DataFrame as usual. The reader turns these operations into REST query parameters so that the server does the work:

df = (
    sedona.read.format("arcgis")
    .option("outSR", 4326)
    .load("https://services.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/US_Wind_Turbine_Database/FeatureServer/0")
)
df.filter(df["t_state"] == "CA").select("FID", "t_state", "p_name", "p_year", "p_cap", "geometry").orderBy("FID").show(3)

+---+-------+------+------+-----+--------------------+
|FID|t_state|p_name|p_year|p_cap|            geometry|
+---+-------+------+------+-----+--------------------+
|  1|     CA|Alta X|  2013|136.8|POINT (-118.23992...|
|  2|     CA|Alta X|  2013|136.8|POINT (-118.25105...|
|  3|     CA|Alta X|  2013|136.8|POINT (-118.25404...|
+---+-------+------+------+-----+--------------------+
only showing top 3 rows

Once the data is in a DataFrame, every WherobotsDB spatial SQL function is available:

df.createOrReplaceTempView("turbines")
sedona.sql("""
    SELECT t_state, COUNT(*) AS turbines
    FROM turbines
    WHERE ST_Intersects(geometry, ST_GeomFromWKT('POLYGON ((-125 32, -114 32, -114 42, -125 42, -125 32))'))
    GROUP BY t_state
    ORDER BY turbines DESC
""").show()

+-------+--------+
|t_state|turbines|
+-------+--------+
|     CA|    5510|
|     AZ|     131|
|     NV|      68|
+-------+--------+

The example counts turbines rather than summing p_cap: that column repeats the parent project’s capacity on every turbine row and holds -9999 where the capacity is unknown, so a plain SUM overcounts and can turn negative.

Service root URLs

If the URL points at a service root (no trailing /<layerIndex>), tell the reader which layer to read with the layer option:

df = (
    sedona.read.format("arcgis")
    .option("layer", 0)
    .load("https://services.arcgis.com/<org>/arcgis/rest/services/<name>/FeatureServer")
)

The layer option is ignored when the URL already ends with a layer index.

Authentication

Public services, such as most ArcGIS Online sample layers, need no credentials. For secured services, supply a pre-issued token generated in ArcGIS Pro, ArcGIS Enterprise Portal, or ArcGIS Online. The token can be set three ways. The first non-empty value wins:

Precedence	Source	Key
1	DataFrame reader option	`auth.token`
2	Spark configuration	`spark.wherobots.arcgis.token`
3	Environment variable	`WHEROBOTS_ARCGIS_TOKEN`

Python
Scala
Java

df = (
    sedona.read.format("arcgis")
    .option("auth.token", "<pre-issued-token>")
    .load("https://<host>/arcgis/rest/services/<name>/FeatureServer/0")
)

val df = sedona.read.format("arcgis")
  .option("auth.token", "<pre-issued-token>")
  .load("https://<host>/arcgis/rest/services/<name>/FeatureServer/0")

Dataset<Row> df = sedona.read().format("arcgis")
    .option("auth.token", "<pre-issued-token>")
    .load("https://<host>/arcgis/rest/services/<name>/FeatureServer/0");

The token is redacted from Spark plans, logs, and exception messages.
Set spark.wherobots.arcgis.token at session creation or job submission. Changing it on a running session with spark.conf.set() has no effect: the reader keeps using the value the session was created with, or, if none was set then, the environment variable or no token at all. In interactive sessions, prefer the per-read auth.token option.
If the service rejects the token (HTTP 401, 498, or 499), the job fails immediately with a clear message rather than falling back to an unauthenticated read.
Tokens are not refreshed during a job. Issue a token with enough lifetime for your workload.
Only pre-issued tokens are supported. The reader does not mint tokens from a username and password, an API key, or OAuth 2.0.

Reading from multiple servers with different tokens

The token is resolved for each load() call, so two layers from different servers can carry their own tokens in the same job. Use the per-read auth.token option for this. The Spark configuration and the environment variable hold a single value and cannot carry two different tokens.

df_a = (
    sedona.read.format("arcgis")
    .option("auth.token", TOKEN_A)
    .load("https://serverA/arcgis/rest/services/<name>/FeatureServer/0")
)
df_b = (
    sedona.read.format("arcgis")
    .option("auth.token", TOKEN_B)
    .load("https://serverB/arcgis/rest/services/<name>/FeatureServer/3")
)
joined = df_a.join(df_b, "id")

Options

Option	Description	Default
`layer`	Layer index to read. Required when the load path is a service root. Ignored when the URL already ends with `/<layerIndex>`.	—
`outSR`	Output spatial reference (WKID). When set, geometries are reprojected by the server and the `geometry` column is tagged with this SRID.	The layer’s native spatial reference
`resultRecordCount`	Page size for each REST request.	The layer’s `maxRecordCount`
`auth.token`	Pre-issued ArcGIS token. See Authentication.	—

Data types

Attribute columns appear in the order the layer declares them. The geometry column is always last.

ArcGIS field type	Spark type
`esriFieldTypeOID`	`long`
`esriFieldTypeSmallInteger`	`short`
`esriFieldTypeInteger`	`integer`
`esriFieldTypeBigInteger`	`long`
`esriFieldTypeSingle`	`float`
`esriFieldTypeDouble`	`double`
`esriFieldTypeString`	`string`
`esriFieldTypeDate`	`timestamp`
`esriFieldTypeBoolean`	`boolean`
`esriFieldTypeDateOnly`, `esriFieldTypeTimeOnly`, `esriFieldTypeTimestampOffset`	`string` (ISO 8601 text, preserved as sent by the server)
`esriFieldTypeGUID`, `esriFieldTypeGlobalID`, `esriFieldTypeXML`	`string`
Any other type	`string`

Esri JSON geometries become WherobotsDB geometries as follows:

Layer geometry type	Geometry
`esriGeometryPoint`	`POINT`
`esriGeometryMultipoint`	`MULTIPOINT`
`esriGeometryPolyline`	`LINESTRING` for a single path, `MULTILINESTRING` for multiple paths
`esriGeometryPolygon`	`POLYGON` for a single outer ring, `MULTIPOLYGON` for multiple outer rings

Z values are kept when the layer has them. M values are dropped.

Pushdown and parallelism

The reader offloads work to the ArcGIS REST /query endpoint so that rows and columns your query discards are never transferred:

Filter pushdown. Supported attribute predicates become the REST where= clause: =, <>, <, <=, >, >=, IN, IS NULL, IS NOT NULL, AND, OR, NOT, and the LIKE-style startsWith, endsWith, and contains. Predicates that cannot be expressed in ArcGIS SQL are still evaluated by Spark, so results are always correct.
Column pruning. Only the selected attribute columns are requested through outFields=. When the geometry column is not selected, the reader sends returnGeometry=false, which is a large saving on polygon layers.
Limit pushdown. LIMIT n is sent as resultRecordCount and capped on the client, so a small query reads a single small page instead of the whole layer.

For large layers, the reader splits the read into parallel object ID ranges. The driver asks the service for the IDs that match the pushed filters with a returnIdsOnly query, sorts them, and chunks them into contiguous ranges. Partitioning applies when the matching IDs span more than one page. The number of partitions is the larger of 8 and the number of pages needed, capped at Spark’s default parallelism. A query with a pushed LIMIT reads in a single partition. If the service does not support the returnIdsOnly query, returns no IDs, or the probe fails for any reason other than authentication, the reader falls back to a single partition that pages through the results with resultOffset. A layer without an object ID field is therefore still readable, just not in parallel. An authentication failure during the probe fails the job. Transient server errors (HTTP 5xx) are retried with a bounded backoff. Authentication failures are not retried.

Spatial predicates such as ST_Intersects are WherobotsDB SQL functions rather than Spark data source filters, so Spark evaluates them after the scan. They are not pushed to the server.

Limitations

Read-only. The reader does not write to Feature Services.
Only pre-issued token authentication is supported. Username and password, API keys, and OAuth 2.0 are not supported.
Tokens are not refreshed during a job.
Spatial predicates are not pushed down to the server.
Parallel reads need a layer that answers returnIdsOnly queries with an object ID field. Other layers are read in a single partition. Paging uses resultOffset, so the service must support pagination.
Esri curve geometries (curveRings, curvePaths, circular arcs, Bézier curves) are not supported. Reading a feature that contains them raises an error. Densify or linearize such layers on the ArcGIS side before reading them.

Load from ArcGIS Feature Services in the vector data loading tutorial
STAC Reader

Wherobots Cloud REST API

WherobotsDB

Vector tiles (PMTiles)

Havasu (Iceberg) table management

GeoStats and Map Matching

Rasterflow

ArcGIS Feature Service Reader

Usage

Service root URLs

Authentication

Reading from multiple servers with different tokens

Options

Data types

Pushdown and parallelism

Limitations

​Usage

​Service root URLs

​Authentication

​Reading from multiple servers with different tokens

​Options

​Data types

​Pushdown and parallelism

​Limitations

​Related

Usage

Service root URLs

Authentication

Reading from multiple servers with different tokens

Options

Data types

Pushdown and parallelism

Limitations

Related