Use RasterFlow with a Storage Integration

Public Preview

RasterFlow reads and writes directly to S3-compatible locations. Inputs — an Area of Interest, a GDAL Raster Tile Index (GTI), a source Zarr store — are read from the URI you pass in, and every RasterFlow task takes a storage_uri input that controls where its outputs and intermediate data land. By default, that destination is your Organization’s Wherobots Managed Storage. Point storage_uri at a Storage Integration instead when you want RasterFlow output to live in your own S3 bucket, under your own retention and access rules.

Where RasterFlow writes

Destination	How you get it	Retention	Use it for
Wherobots Managed Storage, task prefix	Default. Used when you omit `storage_uri`. Each run writes under a task-specific prefix.	Persistent.	Exploration, and results you read from a Wherobots notebook or open on the map.
A Managed Storage path you choose	Pass a managed storage prefix as `storage_uri`.	Persistent.	Work products you want to keep in a predictable place inside Wherobots and share with your Organization.
Your own S3 bucket	Pass an `s3://` prefix backed by a storage integration as `storage_uri`.	Yours — governed by your bucket’s lifecycle and IAM policies.	Production pipelines, data that has to stay in your account, outputs you read outside Wherobots, handoff to other systems.

Reading outputs outside Wherobots. A Wherobots notebook can read Managed Storage directly. To read a task’s outputs from your own machine or systems, see Download a task run’s results, or write them to your own bucket with storage_uri.

Before you start

Wherobots requirements

A Wherobots account. RasterFlow is available to every Organization with nothing to enable.
Outside a Wherobots notebook or job, a Wherobots API key, set as WHEROBOTS_API_KEY.
For the Python examples, Python 3.11 or later with wherobots-python-sdk. The API examples need only curl.

Storage requirements

A storage integration that is already configured through the normal Wherobots setup flow. See S3 Storage Integration for the guided CloudFormation wizard.
Read & Write access on the integration if you intend to use the bucket as a RasterFlow destination. A Read integration can supply inputs only.

Runtime Restart Required After Data IntegrationTo use new storage integrations or catalogs in your notebooks, destroy the existing runtime and start a new one. Notebooks can only access storage integrations or catalogs that were created before the runtime started.

How RasterFlow authenticates to storage

You do not supply an IAM role, an access key, or a credential file to RasterFlow. Access is resolved per path, at the moment the task needs it:

Scoped, path-aware credentials

RasterFlow matches the S3 path it is about to read or write against the storage scopes configured for your Organization, then mints short-lived AWS credentials scoped to that path and to the access mode it needs — read for inputs, read-write for destinations. Credentials are cached and refreshed before they expire, so long-running mosaic and inference tasks keep working without intervention.

Anonymous access to public S3

Publicly readable S3 URIs are read anonymously — no storage integration and no Cloud Connection are involved. This is what lets you point gti_uri or aoi at an open dataset while writing results to your own private bucket.

No storage credentials in your code

Your Wherobots identity authorizes the task run (your API key, or your sign-in inside a Wherobots notebook or job), and RasterFlow resolves storage access on the server. Your code never holds AWS credentials, and the same code runs unchanged from a notebook, a script, or a scheduler.

Credential resolution is automatic, but the integration itself is not. RasterFlow can only reach buckets that an Admin has already added as a storage integration — it will not prompt you to create one.

Set a destination bucket

The examples on this page run RasterFlow through the Tasks API, in two tabs. Python uses the Wherobots Python SDK, and API calls the REST API with curl. Outside a Wherobots notebook or job, both need a Wherobots API key in WHEROBOTS_API_KEY. Each example starts a task run; to wait for it and read its results, see Get started and Run RasterFlow Tasks.

Pass storage_uri as an s3:// URI prefix. RasterFlow writes its outputs and intermediate data under that prefix.

# pip install wherobots-python-sdk
import wherobots

wb = wherobots.Wherobots()  # outside Wherobots, reads WHEROBOTS_API_KEY

# Replace `<stint>` with the bucket name of your storage integration.
run = wb.tasks.run("build_mosaics", {
    "datasets": ["naip_30cm"],
    "aoi": "s3://wherobots-examples/rasterflow/aois/collegepark.parquet",
    "start": "2023-01-01T00:00:00Z",
    "end": "2023-12-31T00:00:00Z",
    "target_crs": "EPSG:3857",
    "storage_uri": "s3://<stint>/rasterflow/mosaics/",
}, region="aws-us-west-2")
# When the run succeeds, results["mosaic_index_uri"] is the mosaic index under your prefix.

# Replace `<stint>` with the bucket name of your storage integration.
curl -X POST "https://api.cloud.wherobots.com/tasks/build_mosaics/runs" \
  -H "X-API-Key: $WHEROBOTS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "region": "aws-us-west-2",
    "name": "NAIP mosaic in my bucket",
    "inputs": {
      "datasets": ["naip_30cm"],
      "aoi": "s3://wherobots-examples/rasterflow/aois/collegepark.parquet",
      "start": "2023-01-01T00:00:00Z",
      "end": "2023-12-31T00:00:00Z",
      "target_crs": "EPSG:3857",
      "storage_uri": "s3://<stint>/rasterflow/mosaics/"
    }
  }'

# When the run succeeds, results.mosaic_index_uri is the mosaic index under your prefix.

The store URIs in the returned mosaic index point into the prefix you supplied, for example:

s3://<stint>/rasterflow/mosaics/marion.zarr

Every RasterFlow task accepts storage_uri: build_mosaics, build_gti_mosaics, run_mosaic_inference, run_cog_inference, the recipe and geometry inference tasks, run_mosaics_change, vectorize_mosaic, and build_zarr_multiscales. Get a task returns each task’s full input schema.

build_mosaics and build_gti_mosaics also accept an id input, which creates a deterministic output prefix under storage_uri instead of one named after the task run. Use it when you want a predictable, re-runnable output path.

Use different buckets for input and output

storage_uri is set per task run and applies only to that run’s output, so inputs and outputs are independent. A built-in dataset, a public tile index, or a store in a bucket you only read from can each be an input while everything you produce goes to your own bucket. That holds across the steps of a pipeline too. Each task reads the URI returned by the previous run and writes wherever you send it. This example follows the ChesapeakeRSC recipe: it filters the NAIP tile index for native 1-meter imagery from 2017 and uses a 1-meter output grid. Choose a date range with coverage for your AOI.

# 1. Mosaic into the working bucket.
wb.tasks.run("build_gti_mosaics", {
    "gti_uri": "s3://wherobots-examples/rasterflow/indexes/naip_index.parquet",
    "bands": ["r", "g", "b", "ir"],
    "location_field": "url",
    "time_column": "year",
    "sort_field": "time",
    "query": "res == 1.0 and year == 2017",
    "requester_pays": True,
    "resampling": "nearest",
    "nodata": 0.0,
    "aoi": "s3://wherobots-examples/rasterflow/aois/collegepark.parquet",
    "target_crs": "EPSG:32618",
    "resolution": 1.0,
    "storage_uri": "s3://<stint>/rasterflow/mosaics/",
}, region="aws-us-west-2")

# 2. When the mosaic run succeeds, run road detection on a store from the location column
#    of its results["mosaic_index_uri"], writing the predictions to a separate prefix.
wb.tasks.run("run_mosaic_inference", {
    "mosaic_uri": "s3://<stint>/rasterflow/mosaics/<path>.zarr",
    "model_path": "https://huggingface.co/wherobots/chesapeakersc-pt2/resolve/b1729ac4be74040f66d0caf5c0f83d0477d9e091/chesapeakersc-ep.pt2",
    "actor": "semantic_segmentation_pytorch",
    "features": ["r", "g", "b", "ir"],
    "labels": ["background", "road"],
    "patch_size": 512,
    "clip_size": 64,
    "device": "cuda",
    "max_batch_size": 128,
    "merge_mode": "weighted_average",
    "storage_uri": "s3://<stint>/rasterflow/predictions/",
}, region="aws-us-west-2")

# 3. When the prediction run succeeds, vectorize the index in its results["mosaic_index_uri"]
#    into a bucket your downstream consumers already read. Passing the index vectorizes every store it lists.
wb.tasks.run("vectorize_mosaic", {
    "mosaic_uri": "s3://<stint>/rasterflow/predictions/<path>/mosaic_index.parquet",
    "features": ["road"],
    "threshold": 0.5,
    "stats": True,
    "storage_uri": "s3://<stint>-analytics/roads/",
}, region="aws-us-west-2")

# 1. Mosaic into the working bucket.
curl -X POST "https://api.cloud.wherobots.com/tasks/build_gti_mosaics/runs" \
  -H "X-API-Key: $WHEROBOTS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "region": "aws-us-west-2",
    "name": "NAIP mosaic",
    "inputs": {
      "gti_uri": "s3://wherobots-examples/rasterflow/indexes/naip_index.parquet",
      "bands": ["r", "g", "b", "ir"],
      "location_field": "url",
      "time_column": "year",
      "sort_field": "time",
      "query": "res == 1.0 and year == 2017",
      "requester_pays": true,
      "resampling": "nearest",
      "nodata": 0.0,
      "aoi": "s3://wherobots-examples/rasterflow/aois/collegepark.parquet",
      "target_crs": "EPSG:32618",
      "resolution": 1.0,
      "storage_uri": "s3://<stint>/rasterflow/mosaics/"
    }
  }'

# 2. When the mosaic run succeeds, run road detection on a store from the location column
#    of its results.mosaic_index_uri, writing the predictions to a separate prefix.
curl -X POST "https://api.cloud.wherobots.com/tasks/run_mosaic_inference/runs" \
  -H "X-API-Key: $WHEROBOTS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "region": "aws-us-west-2",
    "name": "Road predictions",
    "inputs": {
      "mosaic_uri": "s3://<stint>/rasterflow/mosaics/<path>.zarr",
      "model_path": "https://huggingface.co/wherobots/chesapeakersc-pt2/resolve/b1729ac4be74040f66d0caf5c0f83d0477d9e091/chesapeakersc-ep.pt2",
      "actor": "semantic_segmentation_pytorch",
      "features": ["r", "g", "b", "ir"],
      "labels": ["background", "road"],
      "patch_size": 512,
      "clip_size": 64,
      "device": "cuda",
      "max_batch_size": 128,
      "merge_mode": "weighted_average",
      "storage_uri": "s3://<stint>/rasterflow/predictions/"
    }
  }'

# 3. When the prediction run succeeds, vectorize the index in its results.mosaic_index_uri
#    into a bucket your downstream consumers already read.
curl -X POST "https://api.cloud.wherobots.com/tasks/vectorize_mosaic/runs" \
  -H "X-API-Key: $WHEROBOTS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "region": "aws-us-west-2",
    "name": "Road polygons",
    "inputs": {
      "mosaic_uri": "s3://<stint>/rasterflow/predictions/<path>/mosaic_index.parquet",
      "features": ["road"],
      "threshold": 0.5,
      "stats": true,
      "storage_uri": "s3://<stint>-analytics/roads/"
    }
  }'

Every bucket you reference — whether as an input URI or as a storage_uri destination — needs to be either publicly readable or covered by a storage integration. Reading from one integrated bucket and writing to another is supported; reading from a bucket that has no integration is not.

Supported formats

RasterFlow reads and writes standard open formats at these locations. Supported examples include:

Zarr

Mosaics, model predictions, and change-detection outputs are written as Zarr stores. build_zarr_multiscales reads a Zarr store and writes an optimized one.

COG

Cloud Optimized GeoTIFFs are read as mosaic inputs through a GDAL Raster Tile Index with the build_gti_mosaics task.

GeoParquet

Areas of Interest and tile indexes are read as GeoParquet. Mosaic index tables and vectorize_mosaic results are written as GeoParquet.

Parquet

Non-spatial tabular inputs and outputs alongside your raster products.

RasterFlow writes mosaics, predictions, and vectorized results as files at these locations and doesn’t add them to a catalog. To query vectorized results as a table, load the GeoParquet output of the vectorize_mosaic task into an Iceberg table with WherobotsDB, for example in a Managed Catalog on an integrated bucket.

Enable map access to outputs in your bucket

Wherobots-GL — the renderer behind the Wherobots Cloud Map page and the notebook Map widget — loads outputs by URL, in the browser. Outputs in Wherobots Managed Storage work with no extra setup. Outputs in your own bucket have to be reachable with the credentials in play and allow cross-origin browser requests (CORS).

Before you visualize

Confirm that:

The output path points at the intended bucket and AWS region.
The output objects exist and the identity used for visualization can read them.
The bucket’s CORS policy allows requests from the Wherobots Cloud origins.
The policy permits GET and HEAD requests.
The policy allows the Range request header and exposes the ETag and Content-Range response headers. Wherobots-GL uses partial reads to stream raster data efficiently rather than downloading whole stores.

Setting up CORS has a ready-to-paste Amazon S3 policy that satisfies all of these, including the Wherobots Cloud origins and the full set of exposed headers.

Validate access

Two different things have to be true, and they are checked separately:

Integration access

Go to Organization Settings > Storage and use … > Verify Access. Wherobots checks read access, and write access for a Read & Write integration.This confirms RasterFlow can reach the bucket. It exercises the storage integration, not the browser.

Browser CORS access

Verify Access does not test the cross-origin path, so a bucket that verifies cleanly can still fail to render. Confirm CORS by opening the output URL on the Map page — a CORS failure shows up as a blank layer and a blocked request in your browser’s developer console.

Resolve both before sharing an output URL with someone else.

Troubleshooting visualization

CORS error or blank layer

Cause: The browser is blocked from reading the output.Solution:

Update the bucket’s CORS policy using Setting up CORS, then retry.
Confirm the policy includes the Wherobots Cloud origins, GET and HEAD methods, Range as an allowed request header, and ETag and Content-Range among the exposed response headers.

AWS region error

Cause: The output location and the bucket’s region do not match, or the bucket is configured for a different region than expected.Solution:

Verify the output location, bucket name, and AWS region.
Re-run the task with a corrected storage_uri, or republish the output to a bucket in the expected region.

Access denied or object not found

Cause: The output is not where you think it is, or the identity used for visualization cannot read it.Solution:

Print the output URI from the run’s results rather than reconstructing the path by hand.
Confirm the identity used for visualization has read access to that prefix.
Do not put long-lived cloud access keys into browser-based configuration. Use short-lived credentials or scoped access instead — in a notebook, pass them with Map(..., aws_credentials={...}) as shown in Visualize RasterFlow Outputs.

Security guidance

Grant only the minimum permissions needed to read the output objects. Keep write and administrative permissions out of anything browser-accessible, and scope temporary credentials to the relevant bucket and prefix.

Download task run results

To download a task’s output to your machine with only your API key, request short-lived, read-only AWS credentials for its URI, then copy it. This works for any RasterFlow task, whether the output is a single file, such as a mosaic index, or a directory, such as vector results.

Python

# pip install wherobots-python-sdk s3fs
from pathlib import Path

import s3fs
import wherobots
from wherobots import FilesAPI, WherobotsConfig

wb = wherobots.Wherobots()  # outside Wherobots, reads WHEROBOTS_API_KEY

run = wb.task_runs.get("<task run ID>")  # a run that SUCCEEDED
(output_uri,) = run.results.values()  # each RasterFlow task has one output
if output_uri is None:
    raise SystemExit("The run found no features, so there is nothing to download.")

creds = FilesAPI.from_config(WherobotsConfig.from_env()).get_storage_credentials(output_uri)
fs = s3fs.S3FileSystem(
    key=creds.access_key_id,
    secret=creds.secret_access_key,
    token=creds.session_token,
    client_kwargs={"region_name": creds.region},  # required: the credentials are region-scoped
)
output_dir = Path("rasterflow-output")
output_dir.mkdir(parents=True, exist_ok=True)
output_path = output_dir / output_uri.rstrip("/").rsplit("/", 1)[-1]
fs.get(output_uri.rstrip("/"), str(output_path), recursive=True)

The credentials cover only that output’s prefix and expire after about an hour. Request new ones for each output.
Zarr stores (output_uri from Build Multiscales) can be large. To read part of one instead of copying all of it, pass the same fs to your reader, such as xarray.open_zarr(fs.get_mapper(output_uri)).
With the REST API, POST https://api.cloud.wherobots.com/storage/credentials with {"path": "<output URI>", "access_mode": "read"} returns the same credentials.

Troubleshooting

A task run fails on a bucket you can see in Wherobots Cloud

Cause: The integration exists but does not grant the access the task needs. The run’s error_message (errorMessage in the REST API) names the path it could not reach.Solution:

Confirm the integration is Read & Write if you are using the bucket as a storage_uri destination. A Read integration cannot receive output.
If you read the outputs from a notebook, start a new runtime. Notebooks can only reach integrations that existed when the runtime started.
Re-verify the integration from Organization Settings > Storage using … > Verify Access.

You can't read a task's outputs outside Wherobots

Cause: The task ran without storage_uri, so its output is in Wherobots Managed Storage, which your own AWS credentials can’t read.Solution:

Read the output from a Wherobots notebook, or open it on the Map page.
Download the existing output with temporary storage credentials. For future runs, set storage_uri to a bucket you can read if you want the outputs there.

An input URI cannot be read

Cause: The bucket is private and has no storage integration, so RasterFlow falls back to anonymous access and is denied.Solution:

Add a storage integration for the bucket, or copy the input into a bucket that already has one.
Verify the URI itself, including any .zarr or .parquet suffix, by printing it before the call.

Bucket path is rejected

Cause: The bucket name contains periods, which storage integrations do not support.Solution:

Use a bucket whose name has no periods — s3://my-bucket rather than s3://my.bucket.name. See Limitations.

Usage and best practices

Do
Don't

Choose where to store outputs: You can download Managed Storage results with temporary credentials. Set storage_uri when you want future outputs in your own bucket.
Separate prefixes by stage: Distinct prefixes for mosaics, predictions, and vectors make outputs easy to find and easy to expire independently.
Read public imagery directly: Anonymous access means there is no reason to copy an open dataset into your bucket before mosaicking it.
Parameterize the destination: Reading storage_uri from configuration keeps one script usable across environments.
Set CORS once, up front: Add the CORS policy when you add the storage integration, so outputs are viewable on the map the first time you open one.

Don’t hardcode credentials: RasterFlow mints scoped, path-aware storage credentials itself, and the SDK reads your API key from WHEROBOTS_API_KEY. Keys in a script are unnecessary and unsafe.
Don’t expect a new integration in a running notebook: Start a new runtime after an Admin adds one.
Don’t rely on the default task prefix in production: It’s named after each run, so downstream systems can’t find outputs without the run’s results. Set storage_uri, and id where the task offers it.
Don’t read Verify Access as a visualization check: It confirms the storage integration works. It does not test the browser’s cross-origin path.

Next steps

Set up S3 Storage Integration

Create the integration that grants Wherobots access to your bucket.

Run a task

See the full request and response for starting a RasterFlow task run.

Build Multiscales

Add overview levels to a Zarr store so it can be visualized interactively.

API reference

For detailed API documentation, see:

Run a task - POST /tasks/{taskName}/runs
Get a task run - state, results, and errors
Wherobots Python SDK - wb.tasks and wb.task_runs

​Where RasterFlow writes

​Before you start

​How RasterFlow authenticates to storage

​Set a destination bucket

​Use different buckets for input and output

​Supported formats

​Enable map access to outputs in your bucket

​Before you visualize

​Validate access

​Troubleshooting visualization

​Security guidance

​Download task run results

​Troubleshooting

​Usage and best practices

​Next steps

Set up S3 Storage Integration

Run a task

Build Multiscales

​API reference

Where RasterFlow writes

Before you start

How RasterFlow authenticates to storage

Set a destination bucket

Use different buckets for input and output

Supported formats

Enable map access to outputs in your bucket

Before you visualize

Validate access

Troubleshooting visualization

Security guidance

Download task run results

Troubleshooting

Usage and best practices

Next steps

API reference