> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wherobots.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Use RasterFlow with a Storage Integration

> Read and write RasterFlow mosaics, predictions, and vectorized results directly in your own S3 bucket using a Wherobots storage integration.

<Badge color="purple">Public Preview</Badge>

<br />

<br />

<CardGroup cols={2}>
  <Card title="RasterFlow Overview" icon="file-lines" href="/develop/rasterflow/">
    Learn about RasterFlow's key features and capabilities
  </Card>

  <Card title="S3 Storage Integration" icon="aws" href="/develop/storage-management/s3-storage-integration">
    Set up the integration that grants Wherobots access to your bucket.
  </Card>

  <Card title="Reference" icon="code" href="/reference/rasterflow/client">
    Browse the RasterFlow API documentation
  </Card>

  <Card title="Run as a Job" icon="bolt" href="/develop/rasterflow/rasterflow-jobs">
    Submit RasterFlow workflows as automated Job Runs.
  </Card>
</CardGroup>

RasterFlow reads and writes directly to S3-compatible locations. Inputs — an Area of Interest, a GDAL Raster Tile Index (GTI), a source Zarr store — are read from the URI you pass in, and every workflow method takes a `bucket` parameter that controls where its outputs and intermediate data land.

By default, that destination is a temporary user storage location. Point `bucket` at a [Storage Integration](/develop/storage-management/s3-storage-integration) instead when you want RasterFlow output to live in your own S3 bucket, under your own retention and access rules.

## Where RasterFlow writes

| Destination                                                                  | How you get it                                                                                                            | Retention                                                     | Use it for                                                                             |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| **Temporary user storage**                                                   | Default. Used when `bucket` is `None`.                                                                                    | Objects are removed on a 7-day lifecycle.                     | Exploration, one-off experiments, intermediate results you don't need to keep.         |
| **[Wherobots Managed Storage](/develop/storage-management/managed-storage)** | Pass the managed storage prefix as `bucket`.                                                                              | Persistent.                                                   | Work products you want to keep inside Wherobots and share with your Organization.      |
| **Your own S3 bucket**                                                       | Pass an `s3://` prefix backed by a [storage integration](/develop/storage-management/s3-storage-integration) as `bucket`. | Yours — governed by your bucket's lifecycle and IAM policies. | Production pipelines, data that has to stay in your account, handoff to other systems. |

<Warning>
  **Temporary storage is not a place to keep results.** Objects written to the default location are removed on a 7-day lifecycle. Set `bucket` on any workflow whose output you need beyond that window.
</Warning>

## Before you start

<AccordionGroup cols={2}>
  <Accordion title="Wherobots requirements" icon="cloud">
    * Access to RasterFlow in your Organization.
    * A Wherobots notebook, a [VS Code Extension](/develop/vscode-extension/notebooks) workspace, or a [Job Run](/develop/rasterflow/rasterflow-jobs) with `rasterflow_remote` available.
  </Accordion>

  <Accordion title="Storage requirements" icon="aws">
    * A storage integration that is **already configured** through the normal Wherobots setup flow. See [S3 Storage Integration](/develop/storage-management/s3-storage-integration) for the guided CloudFormation wizard.
    * **Read & Write** access on the integration if you intend to use the bucket as a RasterFlow destination. A **Read** integration can supply inputs only.

    <Info>
      **Runtime Restart Required After Data Integration**

      To use new storage integrations or catalogs in your notebooks, you must start a new runtime.
      Notebooks can only access storage integrations or catalogs that were created before the runtime started.
    </Info>
  </Accordion>
</AccordionGroup>

## How RasterFlow authenticates to storage

You do not supply an IAM role, an access key, or a credential file to RasterFlow. Access is resolved per path, at the moment the workflow needs it:

<AccordionGroup>
  <Accordion title="Scoped, path-aware credentials" icon="key">
    RasterFlow matches the S3 path it is about to read or write against the storage scopes configured for your Organization, then mints short-lived AWS credentials scoped to that path and to the access mode it needs — read for inputs, read-write for destinations. Credentials are cached and refreshed before they expire, so long-running mosaic and inference workflows keep working without intervention.
  </Accordion>

  <Accordion title="Anonymous access to public S3" icon="globe">
    Publicly readable S3 URIs are read anonymously — no storage integration and no Cloud Connection are involved. This is what lets you point `gti` or `aoi` at an open dataset while writing results to your own private bucket.
  </Accordion>

  <Accordion title="No credentials in your code" icon="shield-halved">
    Because credentials are resolved from your authenticated Wherobots session, RasterFlow scripts stay free of secrets. The same script runs unchanged in a notebook and in a scheduled [Job Run](/develop/rasterflow/rasterflow-jobs).
  </Accordion>
</AccordionGroup>

<Note>
  Credential resolution is automatic, but the integration itself is not. RasterFlow can only reach buckets that an Admin has already added as a storage integration — it will not prompt you to create one.
</Note>

## Set a destination bucket

Pass `bucket` as an `s3://` URI prefix. RasterFlow writes its outputs and intermediate data under that prefix.

```python theme={"system"}
from datetime import datetime

from rasterflow_remote import RasterflowClient
from rasterflow_remote.data_models import DatasetEnum

rf_client = RasterflowClient()

# Replace `<stint>` with the bucket name of your storage integration.
mosaic_index = rf_client.build_mosaics(
    datasets=[DatasetEnum.NAIP_30CM],
    aoi=aoi_uri,
    start=datetime(2023, 1, 1),
    end=datetime(2024, 1, 1),
    target_crs=3857,
    bucket="s3://<stint>/rasterflow/mosaics/",
)

print(mosaic_index.first_row_mosaic)
```

The returned URI points into the prefix you supplied, for example:

```
s3://<stint>/rasterflow/mosaics/marion.zarr
```

`bucket` is accepted by every `RasterflowClient` workflow method — [`build_mosaics`](/reference/rasterflow/client#build_mosaics), [`build_gti_mosaics`](/reference/rasterflow/client#build_gti_mosaics), [`predict_mosaic`](/reference/rasterflow/client#predict_mosaic) and its recipe and geometry variants, [`run_mosaics_change`](/reference/rasterflow/client#run_mosaics_change), [`vectorize_mosaic`](/reference/rasterflow/client#vectorize_mosaic), and [`build_zarr_multiscales`](/reference/rasterflow/client#build_zarr_multiscales).

<Tip>
  `build_gti_mosaics` also accepts an [`id`](/reference/rasterflow/client#build_gti_mosaics) argument, which creates a deterministic output prefix under `bucket` instead of one named after the workflow execution. Use it when you want a predictable, re-runnable output path.
</Tip>

## Use different buckets for input and output

`bucket` is set per call and applies only to that workflow's output, so inputs and outputs are independent. A built-in dataset, a public tile index, or a store in a bucket you only read from can each be an input while everything you produce goes to your own bucket.

That holds across the steps of a pipeline too. Each call reads the URI returned by the previous step and writes wherever you send it:

```python theme={"system"}
from rasterflow_remote.data_models import ModelRecipes, VectorizeMethodEnum

# 1. Mosaic into the working bucket.
mosaic_index = rf_client.build_mosaics(
    datasets=[DatasetEnum.NAIP_30CM],
    aoi=aoi_uri,
    start=datetime(2023, 1, 1),
    end=datetime(2024, 1, 1),
    target_crs=3857,
    bucket="s3://<stint>/rasterflow/mosaics/",
)

# 2. Predict, reading the mosaic and writing to a separate prefix.
predictions = rf_client.predict_mosaic_recipe(
    aoi=aoi_uri,
    start=datetime(2023, 1, 1),
    end=datetime(2024, 1, 1),
    model_recipe=ModelRecipes.FTW,
    target_crs="EPSG:3857",
    bucket="s3://<stint>/rasterflow/predictions/",
)

# 3. Vectorize into a bucket your downstream consumers already read.
vectors = rf_client.vectorize_mosaic(
    mosaic=predictions.first_row_mosaic,
    features=["field_boundaries"],
    threshold=0.5,
    vectorize_method=VectorizeMethodEnum.SEMANTIC_SEGMENTATION_RASTERIO,
    vectorize_config={"stats": True, "medial_skeletonize": False},
    bucket="s3://<stint>-analytics/fields/",
)
```

<Note>
  Every bucket you reference — whether as an input URI or as a `bucket` destination — needs to be either publicly readable or covered by a storage integration. Reading from one integrated bucket and writing to another is supported; reading from a bucket that has no integration is not.
</Note>

## Supported formats

RasterFlow reads and writes standard open formats at these locations. Supported examples include:

<AccordionGroup cols={3}>
  <Accordion title="Zarr" icon="cubes">
    Mosaics, model predictions, and change-detection outputs are written as Zarr stores. [`build_zarr_multiscales`](/develop/rasterflow/rasterflow-multiscales) reads a Zarr store and writes an optimized one.
  </Accordion>

  <Accordion title="COG" icon="image">
    Cloud Optimized GeoTIFFs are read as mosaic inputs through a GDAL Raster Tile Index with [`build_gti_mosaics`](/reference/rasterflow/client#build_gti_mosaics).
  </Accordion>

  <Accordion title="GeoParquet" icon="vector-polygon">
    Areas of Interest and tile indexes are read as GeoParquet. Mosaic index tables and [`vectorize_mosaic`](/reference/rasterflow/client#vectorize_mosaic) results are written as GeoParquet.
  </Accordion>

  <Accordion title="Parquet" icon="table">
    Non-spatial tabular inputs and outputs alongside your raster products.
  </Accordion>

  <Accordion title="Iceberg" icon="layer-group">
    Tables in an integrated bucket, reachable from Wherobots through a [Managed Catalog](/develop/storage-management/s3-storage-integration#managed-catalog).
  </Accordion>
</AccordionGroup>

## Enable map access to outputs in your bucket

[Wherobots-GL](/develop/rasterflow/rasterflow-visualization) — the renderer behind the Wherobots Cloud [**Map**](https://cloud.wherobots.com/map) page and the notebook `Map` widget — loads outputs **by URL, in the browser**. Outputs in Wherobots Managed Storage work with no extra setup. Outputs in your own bucket have to be reachable with the credentials in play *and* allow cross-origin browser requests (CORS).

### Before you visualize

Confirm that:

* The output path points at the intended bucket and AWS region.
* The output objects exist and the identity used for visualization can read them.
* The bucket's CORS policy allows requests from the Wherobots Cloud origins.
* The policy permits `GET` and `HEAD` requests.
* The policy allows the `Range` request header and exposes the `ETag` and `Content-Range` response headers. Wherobots-GL uses partial reads to stream raster data efficiently rather than downloading whole stores.

[Setting up CORS](/tutorials/wherobotsdb/tile-generation/cors) has a ready-to-paste Amazon S3 policy that satisfies all of these, including the Wherobots Cloud origins and the full set of exposed headers.

### Validate access

Two different things have to be true, and they are checked separately:

<AccordionGroup cols={2}>
  <Accordion title="Integration access" icon="circle-check">
    Go to [**Organization Settings** > **Storage**](https://cloud.wherobots.com/organization#storage) and use **... > Verify Access**. Wherobots checks read access, and write access for a **Read & Write** integration.

    This confirms RasterFlow can reach the bucket. It exercises the storage integration, not the browser.
  </Accordion>

  <Accordion title="Browser CORS access" icon="globe">
    Verify Access does not test the cross-origin path, so a bucket that verifies cleanly can still fail to render. Confirm CORS by opening the output URL on the [**Map**](https://cloud.wherobots.com/map) page — a CORS failure shows up as a blank layer and a blocked request in your browser's developer console.
  </Accordion>
</AccordionGroup>

Resolve both before sharing an output URL with someone else.

### Troubleshooting visualization

<AccordionGroup>
  <Accordion title="CORS error or blank layer" icon="circle-question">
    **Cause:** The browser is blocked from reading the output.

    **Solution:**

    * Update the bucket's CORS policy using [Setting up CORS](/tutorials/wherobotsdb/tile-generation/cors), then retry.
    * Confirm the policy includes the Wherobots Cloud origins, `GET` and `HEAD` methods, `Range` as an allowed request header, and `ETag` and `Content-Range` among the exposed response headers.
  </Accordion>

  <Accordion title="AWS region error" icon="circle-question">
    **Cause:** The output location and the bucket's region do not match, or the bucket is configured for a different region than expected.

    **Solution:**

    * Verify the output location, bucket name, and AWS region.
    * Re-run the workflow with a corrected `bucket`, or republish the output to a bucket in the expected region.
  </Accordion>

  <Accordion title="Access denied or object not found" icon="circle-question">
    **Cause:** The output is not where you think it is, or the identity used for visualization cannot read it.

    **Solution:**

    * Print the URI the workflow returned rather than reconstructing the path by hand.
    * Confirm the identity used for visualization has read access to that prefix.
    * Do not put long-lived cloud access keys into browser-based configuration. Use short-lived credentials or scoped access instead — in a notebook, pass them with `Map(..., aws_credentials={...})` as shown in [Visualize RasterFlow Outputs](/develop/rasterflow/rasterflow-visualization).
  </Accordion>
</AccordionGroup>

### Security guidance

Grant only the minimum permissions needed to read the output objects. Keep write and administrative permissions out of anything browser-accessible, and scope temporary credentials to the relevant bucket and prefix.

## Run a RasterFlow Job against your bucket

Job Runs accept the same input and output options as the examples above, including the `bucket` argument, which makes storage integration the natural choice for scheduled work — a Job's output outlives the 7-day temporary storage window and lands where your downstream systems already look. Read the destination from the Job Run environment rather than hardcoding it, so the same script stays portable across buckets.

See [Run RasterFlow as a Job](/develop/rasterflow/rasterflow-jobs) for the full job script and submission workflow.

## Troubleshooting

<AccordionGroup>
  <Accordion title="WorkflowExecutionError on a bucket you can see in Wherobots Cloud" icon="circle-question">
    **Cause:** The integration exists but does not grant the access the workflow needs, or the runtime started before the integration was created.

    **Solution:**

    * Confirm the integration is **Read & Write** if you are using the bucket as a `bucket` destination. A **Read** integration cannot receive output.
    * Start a new runtime. Notebooks can only reach integrations that existed when the runtime started.
    * Re-verify the integration from [**Organization Settings** > **Storage**](https://cloud.wherobots.com/organization#storage) using **... > Verify Access**.
  </Accordion>

  <Accordion title="Outputs from last week are gone" icon="circle-question">
    **Cause:** The workflow ran without a `bucket` argument, so its output went to temporary user storage and was removed on the 7-day lifecycle.

    **Solution:**

    * Re-run the workflow with `bucket` pointed at your storage integration or Managed Storage.
    * Audit long-lived scripts for calls that still omit `bucket`.
  </Accordion>

  <Accordion title="An input URI cannot be read" icon="circle-question">
    **Cause:** The bucket is private and has no storage integration, so RasterFlow falls back to anonymous access and is denied.

    **Solution:**

    * Add a storage integration for the bucket, or copy the input into a bucket that already has one.
    * Verify the URI itself, including any `.zarr` or `.parquet` suffix, by printing it before the call.
  </Accordion>

  <Accordion title="Bucket path is rejected" icon="circle-question">
    **Cause:** The bucket name contains periods, which storage integrations do not support.

    **Solution:**

    * Use a bucket whose name has no periods — `s3://my-bucket` rather than `s3://my.bucket.name`. See [Limitations](/develop/storage-management/s3-storage-integration#limitations).
  </Accordion>
</AccordionGroup>

## Usage and best practices

<Tabs>
  <Tab title="Do">
    * **Set `bucket` on anything you want to keep:** The default destination is temporary and expires on a 7-day lifecycle.
    * **Separate prefixes by stage:** Distinct prefixes for mosaics, predictions, and vectors make outputs easy to find and easy to expire independently.
    * **Read public imagery directly:** Anonymous access means there is no reason to copy an open dataset into your bucket before mosaicking it.
    * **Parameterize the destination in Job Runs:** Reading the bucket from an environment variable keeps one script usable across environments.
    * **Set CORS once, up front:** Add the CORS policy when you add the storage integration, so outputs are viewable on the map the first time you open one.
  </Tab>

  <Tab title="Don't">
    * **Don't hardcode credentials:** RasterFlow mints scoped, path-aware credentials itself. Access keys in a script are unnecessary and unsafe.
    * **Don't expect a new integration mid-session:** Start a new runtime after an Admin adds one.
    * **Don't point a production pipeline at temporary storage:** Intermediate outputs disappear, breaking any re-run that depends on them.
    * **Don't read Verify Access as a visualization check:** It confirms the storage integration works. It does not test the browser's cross-origin path.
  </Tab>
</Tabs>

## Next steps

<CardGroup cols={3}>
  <Card title="Set up S3 Storage Integration" icon="aws" href="/develop/storage-management/s3-storage-integration">
    Create the integration that grants Wherobots access to your bucket.
  </Card>

  <Card title="Run RasterFlow as a Job" icon="bolt" href="/develop/rasterflow/rasterflow-jobs">
    Schedule RasterFlow workflows that write to your own storage.
  </Card>

  <Card title="Build Multiscales" icon="layer-group" href="/develop/rasterflow/rasterflow-multiscales">
    Add overview levels to a Zarr store so it can be visualized interactively.
  </Card>
</CardGroup>

## API reference

For detailed API documentation, see:

* [Client API Reference](/reference/rasterflow/client) - `RasterflowClient` methods
* [Data Models Reference](/reference/rasterflow/data-models) - Enums and configuration objects
* [Exceptions Reference](/reference/rasterflow/exceptions) - Error handling