Raster Loaders - Wherobots

WherobotsDB raster loaders are available in Scala, Java and Python and have the same APIs.

Load raster files with the raster data source. It reads GeoTIFF and Arc Info ASCII Grid (.asc) files as out-of-database (out-db) rasters, splits each file into tiles as it reads it, and returns one tile per row in a Raster column. For NetCDF files, see Load other raster formats.

Loading raster using the raster loader

The raster data source reads raster files as out-db rasters, then splits that raster data into smaller tiles.

Python
Scala
Java

rawDf = sedona.read.format("raster").load("/FILE-PATH/*.tif")
rawDf.createOrReplaceTempView("rawdf")
rawDf.show()

var rawDf = sedona.read.format("raster").load("/FILE-PATH/*.tif")
rawDf.createOrReplaceTempView("rawdf")
rawDf.show()

Dataset<Row> rawDf = sedona.read().format("raster").load("/FILE-PATH/*.tif")
rawDf.createOrReplaceTempView("rawdf")
rawDf.show()

The output will look like this:

+--------------------+---+---+--------+
|                rast|  x|  y|    name|
+--------------------+---+---+--------+
|OutDbGridCoverage...|  0|  0|file.tif|
|OutDbGridCoverage...|  1|  0|file.tif|
|OutDbGridCoverage...|  2|  0|file.tif|
...

The output contains the following columns:

rast: The raster data in Raster format. This is an out-db raster tile that references to the original raster data file.
x: The 0-based x-coordinate of the tile. This column only presents when retile is not disabled.
y: The 0-based y-coordinate of the tile. This column only presents when retile is not disabled.
name: The name of the source file.

The size of the tile is determined by the internal tiling scheme of the raster data. Using the Cloud Optimized GeoTIFF (COG) format for raster data is recommended since doing so usually organizes the pixel data as square tiles. You can also disable automatic tiling using option("retile", "false"), or specify the tile size manually using options such as option("tileWidth", "256") and option("tileHeight", "256"). The options for the raster data source are as follows:

retile: Enables tiling. Default is true. Set it to false to load each file as one raster.
tileWidth: The width of the tile. If not specified, the size of internal tiles will be used.
tileHeight: The height of the tile. If not specified, will use tileWidth if tileWidth is explicitly set, otherwise the size of internal tiles will be used.
padWithNoData: Pad the right and bottom of the tile with NODATA values if the tile is smaller than the specified tile size. Default is false.
autoRescale: Whether to rescale the pixel values using the per-band scale and offset stored in the GeoTIFF file. Default is false, so tiles hold the raw digital numbers (DNs) as stored in the file. The STAC reader, RS_FromGeoTiff, and RS_FromPath rescale by default instead. See Band values, scale, and offset.

If the internal tiles of a file are too large, too thin, or empty, the raster data source throws an error such as Internal tile shape of <path> is too thin (64 x 4). Files stored in strips of rows rather than in tiles are a common cause. You can disable automatic tiling using option("retile", "false"), or specify the tile size manually to work around this issue. A better solution is to translate the raster data into COG format using gdal_translate or other tools.

The raster data source also works with Spark generic source file options, such as option("pathGlobFilter", "*.tif*") and option("recursiveFileLookup", "true"). For instance, you can load all the .tif files recursively in a directory using

sedona.read.format("raster").option("recursiveFileLookup", "true").option("pathGlobFilter", "*.tif*").load(path_to_raster_data_folder)

The raster data source uniquely handles paths ending in / by performing a recursive search for .tif and .tiff files. This is equivalent to omitting / from the path and setting option("recursiveFileLookup", "true") and option("pathGlobFilter", "*.{tif,tiff,TIF,TIFF}"). To load .asc files, pass their paths or a glob such as /FILE-PATH/*.asc. To load specific files, pass a list of paths:

sedona.read.format("raster").load([
    "s3://my-bucket/scenes/scene_1.tif",
    "s3://my-bucket/scenes/scene_2.tif",
])

The raster data source lists files through the Hadoop file system, so it can’t read http:// or https:// URLs. For a file in Amazon S3, such as a STAC asset whose href is an https:// URL, use its s3:// path. The DataFrame loaded by the raster data source will be automatically repartitioned by default, this is for evenly distributing the workload of processing raster tiles to the entire cluster. The number of partitions is proportional to the number of executor CPU cores in the cluster. You can disable auto repartitioning by setting the Spark session configuration spark.sedona.raster.load.autoRepartition to false. If you want to manually specify the number of partitions, you can set the Spark session configuration spark.sedona.raster.load.numPartitions to the desired number of partitions.

Band values, scale, and offset

A GeoTIFF can store a per-band scale and offset in its GDAL metadata, and the physical value of a pixel is its stored digital number (DN) multiplied by the scale plus the offset. STAC catalogs expose the same two values as scale and offset in the raster:bands extension. The raster data source keeps the raw DNs by default. With option("autoRescale", "true"), it applies the scale and offset while reading, and every SQL function that reads pixels from the raster, such as RS_Value, RS_BandAsArray, RS_MapAlgebra, RS_ZonalStats, and RS_AsInDB, returns values in physical units:

scenes = sedona.read.format("raster").option("autoRescale", "true").load("s3://my-bucket/sentinel-2/*.tif")

Rescaled bands are returned as doubles: RS_BandPixelType reports REAL_64BITS even when the file stores 16-bit integers.
Nodata pixels become NaN, and RS_BandNoDataValue reports NaN, so a valid pixel whose rescaled value equals the file’s nodata value stays valid.
Files without scale and offset metadata are returned unchanged, in their native pixel type.

For example, Sentinel-2 Collection 1 Level-2A COGs (the sentinel-2-c1-l2a collection on Earth Search) store surface reflectance as 16-bit DNs with scale 0.0001 and offset -0.1. With rescaling on, a vegetated near-infrared pixel stored as DN 3345 reads as reflectance 0.2345, because 3345 * 0.0001 - 0.1 = 0.2345. Pixels stored as the nodata value 0 come out as NaN, and DN 1000 is a valid reflectance of 0.0. Because the offset is negative, very dark pixels (DN below 1000) come out slightly below 0. Filter them before computing ratios such as NDVI.

Do not apply a data provider’s DN-to-reflectance formula, such as (DN - 1000) / 10000 for Sentinel-2 processing baseline 04.00 and later, on top of rescaled values. The reader has already applied the file’s scale and offset. Applying the formula a second time silently turns every reflectance into roughly -0.1: normalized-difference indices such as NDVI, NBR, and NDWI collapse to roughly 0 because the numerator cancels, and any other formula returns a plausible-looking but wrong value. No error is raised.

Loaders that rescale by default

Not every entry point uses the same default:

Loader	Rescales by default	How to change it
The `raster` data source	No	`.option("autoRescale", "true")` on the data source
Out-db rasters generated by the STAC reader	Yes	`.option("raster.reader.auto-rescale", "false")` on the reader
`RS_FromGeoTiff(content)`	Yes	`RS_FromGeoTiff(content, false)`
`RS_FromPath(path)`	Yes	`RS_FromPath(path, 'raster.reader.auto-rescale=false')`

Python raster UDFs see the same band values as the SQL functions on the same raster, including NaN for rescaled nodata pixels. For an out-db raster, raster.as_numpy(), raster.as_numpy_masked(), and raster.as_rasterio() read the source file in Python and follow the same setting as SQL: with rescaling on, they return physical units as float64, and as_rasterio() applies the scale and offset through a GDAL VRT so that windowed reads stay windowed; with rescaling off, they return the raw DNs. Do not apply the scale and offset again in the UDF.WherobotsDB 1.40.1 and earlier don’t turn rescaled nodata pixels into NaN. Depending on the function, a nodata pixel reads as the file’s nodata value or as that value rescaled, so SQL and Python can return different values for it, and a valid pixel whose rescaled value equals the file’s nodata value is treated as nodata. On those versions, turn rescaling off, mask the nodata pixels, and then apply the scale and offset yourself.

Load other raster formats

For NetCDF files, load the file content with Spark’s binaryFile data source, then create rasters with RS_FromNetCDF. Arc Info ASCII Grid (.asc) files can be loaded the same way with RS_FromArcInfoAsciiGrid. Both constructors return in-db rasters, which hold the whole file in memory, while the raster data source reads .asc files as out-db rasters.

var df = sedona.read.format("binaryFile").load("/some/path/*.asc")
df = df.withColumn("raster", f.expr("RS_FromArcInfoAsciiGrid(content)"))

RS_FromGeoTiff creates in-db rasters from GeoTIFF content the same way:

var df = sedona.read.format("binaryFile").load("/FILE-PATH/*.tiff")
df = df.withColumn("raster", f.expr("RS_FromGeoTiff(content)"))

To store the pixels of GeoTIFF files in a table, you can also load the files with the raster data source and convert the tiles to in-db rasters with RS_AsInDB.

Raster loading functions

The following functions can be used to create raster objects from various file formats. Each function has its own dedicated reference page:

Function	Description
RS_FromGeoTiff	Create a raster from GeoTiff binary data
RS_FromArcInfoAsciiGrid	Create a raster from Arc Info ASCII Grid binary data
RS_FromNetCDF	Create a raster from NetCDF binary data
RS_FromPath	Create an out-db raster from a file path. To load GeoTIFF files, use the `raster` data source instead
RS_MakeEmptyRaster	Create an empty raster with specified dimensions
RS_MakeRaster	Create a raster from an array of pixel values
RS_AsInDB	Convert an out-db raster to an in-db raster
RS_BandPath	Get the file path of an out-db raster
RS_NetCDFInfo	Get variable information from a NetCDF file

​Loading raster using the raster loader

​Band values, scale, and offset

​Loaders that rescale by default

​Load other raster formats

​Raster loading functions

Loading raster using the raster loader

Band values, scale, and offset

Loaders that rescale by default

Load other raster formats

Raster loading functions