S3 Storage Integration

​

Benefits

• Ease of access to data: Integrating your S3 buckets allows Wherobots Organization members to seamlessly access and work with data stored in Amazon S3 buckets without having to manually transfer or duplicate the data.
• Self-service setup: Administrators can configure the integration themselves through a user interface.
• Secure authentication: Wherobots’ S3 integration supports secure authentication methods, including Amazon Web Services (AWS) access keys and AWS Identity and Access Management (IAM) role-based access.
• Control data access: Administrators can select specific buckets to be accessed through the integration, providing granular control over data access.
• Supports Requester Pays buckets: Wherobots’ S3 integration supports integrations with Amazon S3 Requester Pays buckets. For more information, see Using Requester Pays buckets for storage transfers and usage in the Amazon S3 documentation.

​

Before you start

​

Bucket types

​

Integrate a public or private bucket

​

Integration workflow

1. Wherobots Cloud: Start the integration and copy policy JSON (Steps 1-3)
2. AWS Console: Configure your IAM role with the copied policies
3. Wherobots Cloud: Submit and verify the integration (Step 4)

• Wherobots Cloud
• AWS Console

1

Start the integration

1. Log in to Wherobots Cloud
2. Click Storage
3. Click Create Storage Integration

2

Enter integration details

On the Add New Storage Integration page, enter the following:

• Name: Alphanumeric characters, spaces, special characters, or underscores (must include at least one letter)
• S3 Path: Your bucket path prefaced by s3:// (e.g., s3://my-bucket-name)
• Role ARN: Your IAM role’s Amazon Resource Name

Don’t have a Role ARN yet? You’ll get this from AWS. For now, keep this page open and switch to the AWS Console tab to create or find your IAM role.

3

Copy the policy JSON

Copy both the Sample Role Policy and Trust Relationship JSON from this page. You’ll need these in the next step.

Switch to the AWS Console tab above to configure your IAM role with these policies. Return here after completing the AWS configuration.

4

Submit and verify

After completing the IAM role configuration in AWS, return here to finish the integration.

1. Click Submit on the Create Storage Integration page
2. You’ll be taken to Organization Settings — scroll to Storage
3. Find your bucket and click … > Verify Access If unsuccessful, wait a few seconds and click Retry.

Before starting, make sure you have:

• The Sample Role Policy JSON (copied from Wherobots Cloud Step 3)
• The Trust Relationship JSON (copied from Wherobots Cloud Step 3)

If you don’t have these yet, switch to the Wherobots Cloud tab and complete Steps 1-3 first.

1

Create an IAM role

Already have an IAM role? Skip to the Add the Wherobots tag step.

1. Go to AWS IAM Roles.
2. Click Create role.
3. Under Service or use case, choose S3.
4. Click Next through Permissions Policy.
5. Enter a descriptive Role Name.
6. Click Create role.

2

Add the Wherobots tag

1. Click your newly created role.
2. Click the Tags tab, then Manage tags.
3. Click Add new tag. Enter the following:
   • Key: wherobotsOrgID
   • Value: Your Wherobots Organization ID (find this at Organization Settings).
4. Click Save changes.

3

Copy the Role ARN

1. Select your IAM role from the AWS IAM Roles list.
2. Copy the Role ARN from the Summary section.

You’ll paste this Role ARN into Wherobots Cloud in the Role ARN field (Step 2 in the Wherobots Cloud tab).

4

Add the inline policy

1. Under Permissions policies, click Add permissions > Create inline policy.
2. Click JSON and paste the Sample Role Policy from Wherobots Cloud.
3. Click Next, name the policy, and click Create Policy.

5

Configure the trust relationship

1. Click the Trust relationships tab.
2. Click Edit trust policy.
3. Select all and replace with the Trust Relationship JSON from Wherobots Cloud.
4. Click Update policy.
   AWS configuration is complete. Switch back to the Wherobots Cloud tab, paste the Role ARN into Step 2, then continue with Step 4: Submit and verify.

​

Manage storage integrations

​

View a specific integration’s contents

1

Open the Storage Source dropdown

Log in to Wherobots Cloud and click the Storage Source dropdown at the top of the screen.

2

Select your bucket

Select your desired bucket to view its contents.

​

Access integrated storage in a notebook

CATALOG_NAME.DATABASE_NAME.TABLE_NAME

# Replace `CATALOG_NAME`, `DATABASE_NAME`, and `TABLE_NAME` with your specific names.

from sedona.spark import *

config = SedonaContext.builder().getOrCreate()
sedona = SedonaContext.create(config)

# Access data using catalog.database.table format
df = sedona.table("CATALOG_NAME.DATABASE_NAME.TABLE_NAME")
df.printSchema()
df.show()

​

Managed Catalog

​

What is a Managed Catalog?

• Source of Truth: Wherobots becomes the authoritative source for all metadata, including schemas, table definitions, file locations, and partition information.
• Data Discovery: Wherobots actively scans the underlying storage (e.g., S3) to discover new data and automatically update the catalog.
• Lifecycle Management: Wherobots handles all metadata operations, such as creating, updating, and deleting tables. Changes in the underlying data are automatically synced to the catalog.
• Optimization: Because Wherobots has full control, it can build and manage advanced spatial indexes and perform other performance optimizations directly on the metadata.

​

Create a Managed Catalog from an S3 bucket

1

Open Data Hub

Log in to Wherobots Cloud and click Data Hub.

2

Add a new catalog

Click Add Catalog.

3

Configure the catalog

• Name: Alphanumeric characters, spaces, special characters, or underscores (must include at least one letter)
• Storage: Select a private bucket from the dropdown
• Path (Optional): Enter the sub-folder where you’d like to store this Managed Catalog

To use new storage integrations or catalogs in your notebooks, you must start a new runtime. Notebooks can only access storage integrations or catalogs that were created before the runtime started.

​

Limitations