Skip to content

Data Federation for Databricks Unity Catalog

Wherobots is excited to announce Data Federation for Databricks Unity Catalog.

Customers can now securely connect Wherobots to their governed datasets in Unity Catalog within our Wherobots' new Data Hub.

With this new capability, Paid Wherobots Organizations can connect their Unity Catalog tables to Wherobots high-performance compute and spatial ETL, without copying or moving data.

Data Federation enables teams building on lakehouse architectures to access high-performance and lower-cost ETL, data science, and AI solutions for their physical world data.

This integration simplifies how data teams can use Wherobots to accelerate innovation with data already in their lakehouse.

Databricks Unity Catalog Integration is limited to Paid Organizations

Only Professional and Enterprise Edition Organizations can use the Databricks Unity Catalog Integration.

Community Edition Organizations are limited to Wherobots-managed catalogs and open datasets.

For more information on Paid Organizations, see Wherobots Pricing.

If you have already decided on using a Paid Organization, see Create a new Organization or Upgrade Organization.

Get Started with a Paid Organization

To create a new Paid Organization or to upgrade an existing Organization, see the following instructions:

Benefits

The following are benefits of Wherobots' Data Federation for Databricks Unity Catalog integration:

  • Zero-Copy architecture: Read tables managed by Databricks Unity Catalog without moving or duplicating data.
  • Maintained governance: Databricks Workspace Admins can retain catalog- and table-level access control when reading their Databricks catalogs.
  • Secure federation: Connect securely using Databricks authentication credentials.
  • Accelerated innovation on the lakehouse: Take spatial ideas to market faster using Wherobots' 300+ spatial functions, raster inference, and compute for physical world data on your Unity Catalog data.

Key features

The following are key features of Wherobots' Data Federation for Databricks Unity Catalog integration:

  • Direct Unity Catalog connection: Establish a direct, live connection to your Databricks Unity Catalog from within the Wherobots Data Hub.
  • Full Iceberg Table support: Perform full read and write operations on Managed Iceberg tables in your Unity Catalog.
  • Delta Lake Table access: Read data directly from existing Delta Lake tables for use in Wherobots workloads.
  • Advanced Spatial analytics: Utilize connected data with Wherobots' extensive library of over 300 spatial functions and AI tools for advanced physical world analysis.

Get started

Go to the Wherobots Cloud Data Hub to connect your catalog.

Connect Your Catalog

For advanced spatial analysis, we recommend an ETL (Extract, Transform, Load) workflow:

  1. Extract: Read source data from Unity Catalog.
  2. Transform: Process the data in Wherobots.

  3. Load: Write new spatially-enriched data features to a new table in Unity Catalog.

    To maintain any GEOMETRY or RASTER column types, write to Wherobots-managed Storage.

    graph LR
    subgraph "Left (Extract)"
        A["<b>Databricks Unity Catalog</b><br>Read Data from a<br>Delta or Iceberg Table"]
    end

    subgraph "Middle (Transform)"
        B["<b>Wherobots</b><br>Perform Spatial Analysis in a<br>Notebook, Job Run, or SQL Session"]
    end

    subgraph "Right (Load)"
        direction LR
        C["<b>Option 1: Wherobots Catalog</b><br><b>Action:</b> Preserve GEOMETRY columns<br>and write the enriched table."]
        D["<b>Option 2: Databricks Unity Catalog</b><br><b>Action:</b> Generate simple features,<br>drop the GEOMETRY column,<br>and write the new feature table."]
    end

    A -- "Read" --> B
    B -- "Write" --> C
    B -- "Write" --> D

Important considerations

  • Read-only for Managed Delta Tables: Writing to Managed Delta tables is not supported. All Unity Catalog write operations from Wherobots should target External Delta tables.
  • Unity Catalog & Havasu Spatial Types: Because Databricks' implementation of Iceberg in Unity Catalog does not recognize Havasu's specific GEOMETRY or RASTER column types, you cannot use them directly in Unity Catalog tables. Therefore, all spatial analysis and transformations involving these data types must be performed within Wherobots.
  • Unity Catalog & Databricks GEO Types: While Databricks supports GEOMETRY and GEOGRAPHY data types, those types are implemented as a proprietary Delta Lake table feature. Using those data types in a Delta table prevents that table from being read by external systems, like Wherobots, which limits interoperability.
    • We recommend using columns with Well-Known Binary (WKB) types until this limitation is resolved by Databricks.
    • In contrast, Wherobots provides full, open support for GEOMETRY and RASTER data types directly within the Apache Iceberg format, ensuring reading and writing across different platforms.
    • Additionally, Wherobots can read Managed Delta Tables and write to External Delta Tables.
  • Naming Restriction: Unity Catalogs cannot be named wherobots, as this will cause a permanent naming conflict. Please rename any existing catalogs with this name in Databricks before connecting.
  • Unsupported Features: The current integration does not support Delta Sharing, views, or Iceberg table tags and branches.

Read the Documentation

For more detailed information and instructions, please refer to the following Wherobots documentation: