Use RasterFlow with a Storage Integration

Public Preview

RasterFlow Overview

Learn about RasterFlow’s key features and capabilities

S3 Storage Integration

Set up the integration that grants Wherobots access to your bucket.

Reference

Browse the RasterFlow API documentation

Run as a Job

Submit RasterFlow workflows as automated Job Runs.

RasterFlow reads and writes directly to S3-compatible locations. Inputs — an Area of Interest, a GDAL Raster Tile Index (GTI), a source Zarr store — are read from the URI you pass in, and every workflow method takes a bucket parameter that controls where its outputs and intermediate data land. By default, that destination is a temporary user storage location. Point bucket at a Storage Integration instead when you want RasterFlow output to live in your own S3 bucket, under your own retention and access rules.

Where RasterFlow writes

Destination	How you get it	Retention	Use it for
Temporary user storage	Default. Used when `bucket` is `None`.	Objects are removed on a 7-day lifecycle.	Exploration, one-off experiments, intermediate results you don’t need to keep.
Wherobots Managed Storage	Pass the managed storage prefix as `bucket`.	Persistent.	Work products you want to keep inside Wherobots and share with your Organization.
Your own S3 bucket	Pass an `s3://` prefix backed by a storage integration as `bucket`.	Yours — governed by your bucket’s lifecycle and IAM policies.	Production pipelines, data that has to stay in your account, handoff to other systems.

Temporary storage is not a place to keep results. Objects written to the default location are removed on a 7-day lifecycle. Set bucket on any workflow whose output you need beyond that window.

Before you start

Wherobots requirements

Access to RasterFlow in your Organization.
A Wherobots notebook, a VS Code Extension workspace, or a Job Run with rasterflow_remote available.

Storage requirements

A storage integration that is already configured through the normal Wherobots setup flow. See S3 Storage Integration for the guided CloudFormation wizard.
Read & Write access on the integration if you intend to use the bucket as a RasterFlow destination. A Read integration can supply inputs only.

Runtime Restart Required After Data IntegrationTo use new storage integrations or catalogs in your notebooks, you must start a new runtime. Notebooks can only access storage integrations or catalogs that were created before the runtime started.

How RasterFlow authenticates to storage

You do not supply an IAM role, an access key, or a credential file to RasterFlow. Access is resolved per path, at the moment the workflow needs it:

Scoped, path-aware credentials

RasterFlow matches the S3 path it is about to read or write against the storage scopes configured for your Organization, then mints short-lived AWS credentials scoped to that path and to the access mode it needs — read for inputs, read-write for destinations. Credentials are cached and refreshed before they expire, so long-running mosaic and inference workflows keep working without intervention.

Anonymous access to public S3

Publicly readable S3 URIs are read anonymously — no storage integration and no Cloud Connection are involved. This is what lets you point gti or aoi at an open dataset while writing results to your own private bucket.

No credentials in your code

Because credentials are resolved from your authenticated Wherobots session, RasterFlow scripts stay free of secrets. The same script runs unchanged in a notebook and in a scheduled Job Run.

Credential resolution is automatic, but the integration itself is not. RasterFlow can only reach buckets that an Admin has already added as a storage integration — it will not prompt you to create one.

Set a destination bucket

Pass bucket as an s3:// URI prefix. RasterFlow writes its outputs and intermediate data under that prefix.

from datetime import datetime

from rasterflow_remote import RasterflowClient
from rasterflow_remote.data_models import DatasetEnum

rf_client = RasterflowClient()

# Replace `<stint>` with the bucket name of your storage integration.
mosaic_index = rf_client.build_mosaics(
    datasets=[DatasetEnum.NAIP_30CM],
    aoi=aoi_uri,
    start=datetime(2023, 1, 1),
    end=datetime(2024, 1, 1),
    target_crs=3857,
    bucket="s3://<stint>/rasterflow/mosaics/",
)

print(mosaic_index.first_row_mosaic)

The returned URI points into the prefix you supplied, for example:

s3://<stint>/rasterflow/mosaics/marion.zarr

bucket is accepted by every RasterflowClient workflow method — build_mosaics, build_gti_mosaics, predict_mosaic and its recipe and geometry variants, run_mosaics_change, vectorize_mosaic, and build_zarr_multiscales.

build_gti_mosaics also accepts an id argument, which creates a deterministic output prefix under bucket instead of one named after the workflow execution. Use it when you want a predictable, re-runnable output path.

Use different buckets for input and output

bucket is set per call and applies only to that workflow’s output, so inputs and outputs are independent. A built-in dataset, a public tile index, or a store in a bucket you only read from can each be an input while everything you produce goes to your own bucket. That holds across the steps of a pipeline too. Each call reads the URI returned by the previous step and writes wherever you send it:

from rasterflow_remote.data_models import ModelRecipes, VectorizeMethodEnum

# 1. Mosaic into the working bucket.
mosaic_index = rf_client.build_mosaics(
    datasets=[DatasetEnum.NAIP_30CM],
    aoi=aoi_uri,
    start=datetime(2023, 1, 1),
    end=datetime(2024, 1, 1),
    target_crs=3857,
    bucket="s3://<stint>/rasterflow/mosaics/",
)

# 2. Predict, reading the mosaic and writing to a separate prefix.
predictions = rf_client.predict_mosaic_recipe(
    aoi=aoi_uri,
    start=datetime(2023, 1, 1),
    end=datetime(2024, 1, 1),
    model_recipe=ModelRecipes.FTW,
    target_crs="EPSG:3857",
    bucket="s3://<stint>/rasterflow/predictions/",
)

# 3. Vectorize into a bucket your downstream consumers already read.
vectors = rf_client.vectorize_mosaic(
    mosaic=predictions.first_row_mosaic,
    features=["field_boundaries"],
    threshold=0.5,
    vectorize_method=VectorizeMethodEnum.SEMANTIC_SEGMENTATION_RASTERIO,
    vectorize_config={"stats": True, "medial_skeletonize": False},
    bucket="s3://<stint>-analytics/fields/",
)

Every bucket you reference — whether as an input URI or as a bucket destination — needs to be either publicly readable or covered by a storage integration. Reading from one integrated bucket and writing to another is supported; reading from a bucket that has no integration is not.

Supported formats

RasterFlow reads and writes standard open formats at these locations. Supported examples include:

Zarr

Mosaics, model predictions, and change-detection outputs are written as Zarr stores. build_zarr_multiscales reads a Zarr store and writes an optimized one.

COG

Cloud Optimized GeoTIFFs are read as mosaic inputs through a GDAL Raster Tile Index with build_gti_mosaics.

GeoParquet

Areas of Interest and tile indexes are read as GeoParquet. Mosaic index tables and vectorize_mosaic results are written as GeoParquet.

Parquet

Non-spatial tabular inputs and outputs alongside your raster products.

Iceberg

Tables in an integrated bucket, reachable from Wherobots through a Managed Catalog.

Enable map access to outputs in your bucket

Wherobots-GL — the renderer behind the Wherobots Cloud Map page and the notebook Map widget — loads outputs by URL, in the browser. Outputs in Wherobots Managed Storage work with no extra setup. Outputs in your own bucket have to be reachable with the credentials in play and allow cross-origin browser requests (CORS).

Before you visualize

Confirm that:

The output path points at the intended bucket and AWS region.
The output objects exist and the identity used for visualization can read them.
The bucket’s CORS policy allows requests from the Wherobots Cloud origins.
The policy permits GET and HEAD requests.
The policy allows the Range request header and exposes the ETag and Content-Range response headers. Wherobots-GL uses partial reads to stream raster data efficiently rather than downloading whole stores.

Setting up CORS has a ready-to-paste Amazon S3 policy that satisfies all of these, including the Wherobots Cloud origins and the full set of exposed headers.

Validate access

Two different things have to be true, and they are checked separately:

Integration access

Go to Organization Settings > Storage and use … > Verify Access. Wherobots checks read access, and write access for a Read & Write integration.This confirms RasterFlow can reach the bucket. It exercises the storage integration, not the browser.

Browser CORS access

Verify Access does not test the cross-origin path, so a bucket that verifies cleanly can still fail to render. Confirm CORS by opening the output URL on the Map page — a CORS failure shows up as a blank layer and a blocked request in your browser’s developer console.

Resolve both before sharing an output URL with someone else.

Troubleshooting visualization

CORS error or blank layer

Cause: The browser is blocked from reading the output.Solution:

Update the bucket’s CORS policy using Setting up CORS, then retry.
Confirm the policy includes the Wherobots Cloud origins, GET and HEAD methods, Range as an allowed request header, and ETag and Content-Range among the exposed response headers.

AWS region error

Cause: The output location and the bucket’s region do not match, or the bucket is configured for a different region than expected.Solution:

Verify the output location, bucket name, and AWS region.
Re-run the workflow with a corrected bucket, or republish the output to a bucket in the expected region.

Access denied or object not found

Cause: The output is not where you think it is, or the identity used for visualization cannot read it.Solution:

Print the URI the workflow returned rather than reconstructing the path by hand.
Confirm the identity used for visualization has read access to that prefix.
Do not put long-lived cloud access keys into browser-based configuration. Use short-lived credentials or scoped access instead — in a notebook, pass them with Map(..., aws_credentials={...}) as shown in Visualize RasterFlow Outputs.

Security guidance

Grant only the minimum permissions needed to read the output objects. Keep write and administrative permissions out of anything browser-accessible, and scope temporary credentials to the relevant bucket and prefix.

Run a RasterFlow Job against your bucket

Job Runs accept the same input and output options as the examples above, including the bucket argument, which makes storage integration the natural choice for scheduled work — a Job’s output outlives the 7-day temporary storage window and lands where your downstream systems already look. Read the destination from the Job Run environment rather than hardcoding it, so the same script stays portable across buckets. See Run RasterFlow as a Job for the full job script and submission workflow.

Troubleshooting

WorkflowExecutionError on a bucket you can see in Wherobots Cloud

Cause: The integration exists but does not grant the access the workflow needs, or the runtime started before the integration was created.Solution:

Confirm the integration is Read & Write if you are using the bucket as a bucket destination. A Read integration cannot receive output.
Start a new runtime. Notebooks can only reach integrations that existed when the runtime started.
Re-verify the integration from Organization Settings > Storage using … > Verify Access.

Outputs from last week are gone

Cause: The workflow ran without a bucket argument, so its output went to temporary user storage and was removed on the 7-day lifecycle.Solution:

Re-run the workflow with bucket pointed at your storage integration or Managed Storage.
Audit long-lived scripts for calls that still omit bucket.

An input URI cannot be read

Cause: The bucket is private and has no storage integration, so RasterFlow falls back to anonymous access and is denied.Solution:

Add a storage integration for the bucket, or copy the input into a bucket that already has one.
Verify the URI itself, including any .zarr or .parquet suffix, by printing it before the call.

Bucket path is rejected

Cause: The bucket name contains periods, which storage integrations do not support.Solution:

Use a bucket whose name has no periods — s3://my-bucket rather than s3://my.bucket.name. See Limitations.

Usage and best practices

Do
Don't

Set bucket on anything you want to keep: The default destination is temporary and expires on a 7-day lifecycle.
Separate prefixes by stage: Distinct prefixes for mosaics, predictions, and vectors make outputs easy to find and easy to expire independently.
Read public imagery directly: Anonymous access means there is no reason to copy an open dataset into your bucket before mosaicking it.
Parameterize the destination in Job Runs: Reading the bucket from an environment variable keeps one script usable across environments.
Set CORS once, up front: Add the CORS policy when you add the storage integration, so outputs are viewable on the map the first time you open one.

Next steps

Set up S3 Storage Integration

Create the integration that grants Wherobots access to your bucket.

Run RasterFlow as a Job

Schedule RasterFlow workflows that write to your own storage.

Build Multiscales

Add overview levels to a Zarr store so it can be visualized interactively.

API reference

For detailed API documentation, see:

Client API Reference - RasterflowClient methods
Data Models Reference - Enums and configuration objects
Exceptions Reference - Error handling

Agentic Development

Notebook Management

RasterFlow

SQL API & SDKs

Storage Management

Other deployment options

Orchestration

Monitoring

Use RasterFlow with a Storage Integration

RasterFlow Overview

S3 Storage Integration

Reference

Run as a Job

Where RasterFlow writes

Before you start

How RasterFlow authenticates to storage

Set a destination bucket

Use different buckets for input and output

Supported formats

Enable map access to outputs in your bucket

Before you visualize

Validate access

Troubleshooting visualization

Security guidance

Run a RasterFlow Job against your bucket

Troubleshooting

Usage and best practices

Next steps

Set up S3 Storage Integration

Run RasterFlow as a Job

Build Multiscales

API reference

RasterFlow Overview

S3 Storage Integration

Reference

Run as a Job

​Where RasterFlow writes

​Before you start

​How RasterFlow authenticates to storage

​Set a destination bucket

​Use different buckets for input and output

​Supported formats

​Enable map access to outputs in your bucket

​Before you visualize

​Validate access

​Troubleshooting visualization

​Security guidance

​Run a RasterFlow Job against your bucket

​Troubleshooting

​Usage and best practices

​Next steps

Set up S3 Storage Integration

Run RasterFlow as a Job

Build Multiscales

​API reference

Where RasterFlow writes

Before you start

How RasterFlow authenticates to storage

Set a destination bucket

Use different buckets for input and output

Supported formats

Enable map access to outputs in your bucket

Before you visualize

Validate access

Troubleshooting visualization

Security guidance

Run a RasterFlow Job against your bucket

Troubleshooting

Usage and best practices

Next steps

API reference