Spatial statistics

These functions take a Spark DataFrame with a geometry column and return a new DataFrame with the result columns added.

from sedona.spark.stats import dbscan, g_local, add_binary_distance_band_column

add_binary_distance_band_column

add_binary_distance_band_column(
    dataframe: DataFrame,
    threshold: float,
    include_zero_distance_neighbors: bool = True,
    include_self: bool = False,
    geometry: Optional[str] = None,
    use_spheroid: bool = False,
    saved_attributes: List[str] = None,
    result_name: str = 'weights',
) -> DataFrame

Annotates a dataframe with a weights column containing the other records within the threshold and their weight. Weights will always be 1.0. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided. The new column is named by result_name, ‘weights’ by default.

Parameters

DataFrame

required

DataFrame with geometry column

float

required

Distance threshold for considering neighbors

bool

default:"True"

whether to include neighbors that are 0 distance. Default is True

bool

default:"False"

whether to include self in the list of neighbors

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

List[str]

default:"None"

the attributes to save in the neighbor column. Default is all columns.

str

default:"'weights'"

the name of the resulting column. Default is ‘weights’.

Returns

DataFrame

The input DataFrame with a weight column added containing neighbors and their weights (always 1) added to each row.

add_distance_band_column

add_distance_band_column(
    dataframe: DataFrame,
    threshold: float,
    binary: bool = True,
    alpha: float = -1.0,
    include_zero_distance_neighbors: bool = False,
    include_self: bool = False,
    self_weight: float = 1.0,
    geometry: Optional[str] = None,
    use_spheroid: bool = False,
    saved_attributes: List[str] = None,
    result_name: str = 'weights',
) -> DataFrame

Annotates a dataframe with a weights column containing the other records within the threshold and their weight. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided. The new column is named by result_name, ‘weights’ by default.

Parameters

DataFrame

required

DataFrame with geometry column

float

required

Distance threshold for considering neighbors

bool

default:"True"

whether to use binary weights or inverse distance weights for neighbors (dist^alpha)

float

default:"-1.0"

alpha to use for inverse distance weights ignored when binary is true

bool

default:"False"

whether to include neighbors that are 0 distance. If 0 distance neighbors are included, binary is false, and alpha is negative, their weight is infinite (division by zero)

bool

default:"False"

whether to include self in the list of neighbors

float

default:"1.0"

the value to use for the self weight

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

List[str]

default:"None"

the attributes to save in the neighbor column. Default is all columns.

str

default:"'weights'"

the name of the resulting column. Default is ‘weights’.

Returns

DataFrame

The input DataFrame with a weight column added containing neighbors and their weights added to each row.

add_weighted_distance_band_column

add_weighted_distance_band_column(
    dataframe: DataFrame,
    threshold: float,
    alpha: float,
    include_zero_distance_neighbors: bool = True,
    include_self: bool = False,
    self_weight: float = 1.0,
    geometry: Optional[str] = None,
    use_spheroid: bool = False,
    saved_attributes: List[str] = None,
    result_name: str = 'weights',
) -> DataFrame

Annotates a dataframe with a weights column containing the other records within the threshold and their weight. Weights will be distance^alpha. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided. The new column is named by result_name, ‘weights’ by default.

Parameters

DataFrame

required

DataFrame with geometry column

float

required

Distance threshold for considering neighbors

float

required

exponent of the distance weights, computed as dist^alpha. Use a negative value, such as -1.0, for inverse distance weights

bool

default:"True"

whether to include neighbors that are 0 distance. Default is True. With a negative alpha, their weight is infinite (division by zero)

bool

default:"False"

whether to include self in the list of neighbors

float

default:"1.0"

the value to use for the self weight. Default is 1.0

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

List[str]

default:"None"

the attributes to save in the neighbor column. Default is all columns.

str

default:"'weights'"

the name of the resulting column. Default is ‘weights’.

Returns

DataFrame

The input DataFrame with a weight column added containing neighbors and their weights (dist^alpha) added to each row.

dbscan

dbscan(
    dataframe: DataFrame,
    epsilon: float,
    min_pts: int,
    geometry: Optional[str] = None,
    include_outliers: bool = True,
    use_sphere=False,
    is_core_column_name='isCore',
    cluster_column_name='cluster',
)

Annotates a dataframe with a cluster label for each data record using the DBSCAN algorithm. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided.

Parameters

DataFrame

required

spark dataframe containing the geometries

float

required

minimum distance parameter of DBSCAN algorithm

int

required

minimum number of points parameter of DBSCAN algorithm

Optional[str]

default:"None"

name of the geometry column

bool

default:"True"

whether to return outlier points. If True, outliers are returned with a cluster value of -1. Default is True

default:"False"

whether to use a cartesian or sphere distance calculation. Default is false

default:"'isCore'"

what the name of the column indicating if this is a core point should be. Default is “isCore”

default:"'cluster'"

what the name of the column indicating the cluster id should be. Default is “cluster”

Returns

A PySpark DataFrame containing the cluster label for each row

g_local

g_local(
    dataframe: DataFrame,
    x: str,
    weights: str = 'weights',
    permutations: int = 0,
    star: bool = False,
    island_weight: float = 0.0,
) -> DataFrame

Performs the Gi or Gi* statistic on the x column of the dataframe. Weights should be the neighbors of this row. The members of the weights should be comprised of structs containing a value column and a neighbor column. The neighbor column should be the contents of the neighbors with the same types as the parent row (minus neighbors). You can use sedona.spark.stats.weighting.add_distance_band_column to achieve this. To calculate the Gi* statistic, ensure the focal observation is in the neighbors array (pass include_self=True to the weighting function) and set star=True. Significance is calculated with a z score. Permutation tests are not yet implemented and thus island weight does nothing. The following columns will be added: G, EG (the expected G), VG (the variance of G), Z, and P (1 minus the standard normal CDF of |Z|).

Parameters

DataFrame

required

the dataframe to perform the G statistic on

str

required

The column name we want to perform hotspot analysis on

str

default:"'weights'"

The column name containing the neighbors array. The neighbor column should be the contents of the neighbors with the same types as the parent row (minus neighbors). You can use sedona.spark.stats.weighting.add_distance_band_column to achieve this.

int

default:"0"

Not used. Permutation tests are not supported yet. The number of permutations to use for the significance test.

bool

default:"False"

Whether the focal observation is in the neighbors array. If true this calculates Gi*, otherwise Gi

float

default:"0.0"

Not used. The weight for the simulated neighbor used for records without a neighbor in perm tests

Returns

DataFrame

A dataframe with the original columns plus the columns G, EG, VG, Z, P.

local_outlier_factor

local_outlier_factor(
    dataframe: DataFrame,
    k: int = 20,
    geometry: Optional[str] = None,
    approximate_knn: bool = False,
    handle_ties: bool = False,
    use_spheroid=False,
    result_column_name: str = 'lof',
)

Annotates a dataframe with a column containing the local outlier factor for each data record. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided.

Parameters

DataFrame

required

apache sedona idDataframe containing the point geometries

int

default:"20"

number of nearest neighbors that will be considered for the LOF calculation

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use approximate KNN. When false will use exact KNN join. Default is False

bool

default:"False"

whether to handle ties in the k-distance calculation. Default is false

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

str

default:"'lof'"

the name of the column containing the lof for each row. Default is “lof”

Returns

A PySpark DataFrame containing the lof for each row

Moran

Moran()

Global Moran’s I spatial autocorrelation test.

get_global()

Moran.get_global(
    df: DataFrame,
    two_tailed: bool = True,
    id_column: str = 'id',
    value_column: str = 'value',
) -> MoranResult

Computes the global Moran’s I of a value column. The dataframe needs an id column that uniquely identifies each row, a numeric value column, and a column named weights that lists each row’s neighbors and their weights, such as the one added by sedona.spark.stats.weighting.add_distance_band_column. Each neighbor must include the id and value columns, so pass them to the weighting function in saved_attributes.

Parameters

DataFrame

required

dataframe with the id, value, and weights columns

bool

default:"True"

whether to return a two-tailed p-value. Default is True

str

default:"'id'"

name of the column that identifies each row. Default is “id”

str

default:"'value'"

name of the numeric column to test. Default is “value”

Returns

MoranResult

A MoranResult with Moran’s I and its z-score and p-value under the normality assumption.

MoranResult

MoranResult(i: float, p_norm: float, z_norm: float)

Result of the global Moran’s I test.

Parameters

float

required

Moran’s I. Values near 1 mean that similar values are close to each other, values near -1 mean that neighbors tend to have dissimilar values, and values near 0 mean that the values are randomly distributed.

float

required

p-value of z_norm under the normality assumption. It is two-tailed unless Moran.get_global was called with two_tailed=False.

float

required

z-score of i under the normality assumption.

spatially_stratified_sample

spatially_stratified_sample(
    dataframe: DataFrame,
    fraction: float,
    partition_count: int,
    geometry: Optional[str] = None,
    seed: int = 42,
)

Spatially stratified sampling of a DataFrame containing spatial data.

Parameters

DataFrame

required

DataFrame containing spatial data to be sampled. Must contain a geometry column.

float

required

Sampling rate between 0 and 1

int

required

Number of partitions to divide the data into. If not a perfect square, the number of partitions in each dimension will be rounded to the nearest integer.

Optional[str]

default:"None"

Column containing the geometry data. Default is “geometry”

int

default:"42"

Seed for sampling the data

Returns

the input DataFrame sampled down to the specified rate

​add_binary_distance_band_column

​Parameters

​Returns

​add_distance_band_column

​Parameters

​Returns

​add_weighted_distance_band_column

​Parameters

​Returns

​dbscan

​Parameters

​Returns

​g_local

​Parameters

​Returns

​local_outlier_factor

​Parameters

​Returns

​Moran

​get_global()

​Parameters

​Returns

​MoranResult

​Parameters

​spatially_stratified_sample

​Parameters

​Returns

add_binary_distance_band_column

Parameters

Returns

add_distance_band_column

Parameters

Returns

add_weighted_distance_band_column

Parameters

Returns

dbscan

Parameters

Returns

g_local

Parameters

Returns

local_outlier_factor

Parameters

Returns

Moran

get_global()

Parameters

Returns

MoranResult

Parameters

spatially_stratified_sample

Parameters

Returns