Spatial statistics

These functions take a Spark DataFrame with a geometry column and return a new DataFrame with the result columns added.

from sedona.spark.stats import dbscan, g_local, add_binary_distance_band_column

add_binary_distance_band_column

add_binary_distance_band_column(
    dataframe: DataFrame,
    threshold: float,
    include_zero_distance_neighbors: bool = True,
    include_self: bool = False,
    geometry: Optional[str] = None,
    use_spheroid: bool = False,
    saved_attributes: List[str] = None,
    result_name: str = 'weights',
) -> DataFrame

Annotates a dataframe with a weights column containing the other records within the threshold and their weight. Weights will always be 1.0. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided. The new column will be named ‘cluster’.

Parameters

DataFrame

required

DataFrame with geometry column

float

required

Distance threshold for considering neighbors

bool

default:"True"

whether to include neighbors that are 0 distance. If 0 distance neighbors are included and binary is false, values are infinity as per the floating point spec (divide by 0)

bool

default:"False"

whether to include self in the list of neighbors

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

List[str]

default:"None"

the attributes to save in the neighbor column. Default is all columns.

str

default:"'weights'"

the name of the resulting column. Default is ‘weights’.

Returns

DataFrame

The input DataFrame with a weight column added containing neighbors and their weights (always 1) added to each row.

add_distance_band_column

add_distance_band_column(
    dataframe: DataFrame,
    threshold: float,
    binary: bool = True,
    alpha: float = -1.0,
    include_zero_distance_neighbors: bool = False,
    include_self: bool = False,
    self_weight: float = 1.0,
    geometry: Optional[str] = None,
    use_spheroid: bool = False,
    saved_attributes: List[str] = None,
    result_name: str = 'weights',
) -> DataFrame

Annotates a dataframe with a weights column containing the other records within the threshold and their weight. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided. The new column will be named ‘cluster’.

Parameters

DataFrame

required

DataFrame with geometry column

float

required

Distance threshold for considering neighbors

bool

default:"True"

whether to use binary weights or inverse distance weights for neighbors (dist^alpha)

float

default:"-1.0"

alpha to use for inverse distance weights ignored when binary is true

bool

default:"False"

whether to include neighbors that are 0 distance. If 0 distance neighbors are included and binary is false, values are infinity as per the floating point spec (divide by 0)

bool

default:"False"

whether to include self in the list of neighbors

float

default:"1.0"

the value to use for the self weight

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

List[str]

default:"None"

the attributes to save in the neighbor column. Default is all columns.

str

default:"'weights'"

the name of the resulting column. Default is ‘weights’.

Returns

DataFrame

The input DataFrame with a weight column added containing neighbors and their weights added to each row.

add_weighted_distance_band_column

add_weighted_distance_band_column(
    dataframe: DataFrame,
    threshold: float,
    alpha: float,
    include_zero_distance_neighbors: bool = True,
    include_self: bool = False,
    self_weight: float = 1.0,
    geometry: Optional[str] = None,
    use_spheroid: bool = False,
    saved_attributes: List[str] = None,
    result_name: str = 'weights',
) -> DataFrame

Annotates a dataframe with a weights column containing the other records within the threshold and their weight. Weights will be distance^alpha. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided. The new column will be named ‘cluster’.

Parameters

DataFrame

required

DataFrame with geometry column

float

required

Distance threshold for considering neighbors

float

required

alpha to use for inverse distance weights. Computation is dist^alpha. Default is -1.0

bool

default:"True"

whether to include neighbors that are 0 distance. If 0 distance neighbors are included and binary is false, values are infinity as per the floating point spec (divide by 0)

bool

default:"False"

whether to include self in the list of neighbors

float

default:"1.0"

the value to use for the self weight. Default is 1.0

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

List[str]

default:"None"

the attributes to save in the neighbor column. Default is all columns.

str

default:"'weights'"

the name of the resulting column. Default is ‘weights’.

Returns

DataFrame

The input DataFrame with a weight column added containing neighbors and their weights (always 1) added to each row.

dbscan

dbscan(
    dataframe: DataFrame,
    epsilon: float,
    min_pts: int,
    geometry: Optional[str] = None,
    include_outliers: bool = True,
    use_sphere=False,
    is_core_column_name='isCore',
    cluster_column_name='cluster',
)

Annotates a dataframe with a cluster label for each data record using the DBSCAN algorithm. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided.

Parameters

DataFrame

required

spark dataframe containing the geometries

float

required

minimum distance parameter of DBSCAN algorithm

int

required

minimum number of points parameter of DBSCAN algorithm

Optional[str]

default:"None"

name of the geometry column

bool

default:"True"

whether to return outlier points. If True, outliers are returned with a cluster value of -1. Default is False

default:"False"

whether to use a cartesian or sphere distance calculation. Default is false

default:"'isCore'"

what the name of the column indicating if this is a core point should be. Default is “isCore”

default:"'cluster'"

what the name of the column indicating the cluster id should be. Default is “cluster”

Returns

A PySpark DataFrame containing the cluster label for each row

g_local

g_local(
    dataframe: DataFrame,
    x: str,
    weights: str = 'weights',
    permutations: int = 0,
    star: bool = False,
    island_weight: float = 0.0,
) -> DataFrame

Performs the Gi or Gi* statistic on the x column of the dataframe. Weights should be the neighbors of this row. The members of the weights should be comprised of structs containing a value column and a neighbor column. The neighbor column should be the contents of the neighbors with the same types as the parent row (minus neighbors). You can use wherobots.weighing.add_distance_band_column to achieve this. To calculate the Gi* statistic, ensure the focal observation is in the neighbors array (i.e. the row is in the weights column) and star=true. Significance is calculated with a z score. Permutation tests are not yet implemented and thus island weight does nothing. The following columns will be added: G, E[G], V[G], Z, P.

Parameters

DataFrame

required

the dataframe to perform the G statistic on

str

required

The column name we want to perform hotspot analysis on

str

default:"'weights'"

The column name containing the neighbors array. The neighbor column should be the contents of the neighbors with the same types as the parent row (minus neighbors). You can use wherobots.weighing.add_distance_band_column to achieve this.

int

default:"0"

Not used. Permutation tests are not supported yet. The number of permutations to use for the significance test.

bool

default:"False"

Whether the focal observation is in the neighbors array. If true this calculates Gi*, otherwise Gi

float

default:"0.0"

Not used. The weight for the simulated neighbor used for records without a neighbor in perm tests

Returns

DataFrame

A dataframe with the original columns plus the columns G, E[G], V[G], Z, P.

local_outlier_factor

local_outlier_factor(
    dataframe: DataFrame,
    k: int = 20,
    geometry: Optional[str] = None,
    approximate_knn: bool = False,
    handle_ties: bool = False,
    use_spheroid=False,
    result_column_name: str = 'lof',
)

Annotates a dataframe with a column containing the local outlier factor for each data record. The dataframe should contain at least one GeometryType column. Rows must be unique. If one geometry column is present it will be used automatically. If two are present, the one named ‘geometry’ will be used. If more than one are present and neither is named ‘geometry’, the column name must be provided.

Parameters

DataFrame

required

apache sedona idDataframe containing the point geometries

int

default:"20"

number of nearest neighbors that will be considered for the LOF calculation

Optional[str]

default:"None"

name of the geometry column

bool

default:"False"

whether to use approximate KNN. When false will use exact KNN join. Default is False

bool

default:"False"

whether to handle ties in the k-distance calculation. Default is false

default:"False"

whether to use a cartesian or spheroidal distance calculation. Default is false

str

default:"'lof'"

the name of the column containing the lof for each row. Default is “lof”

Returns

A PySpark DataFrame containing the lof for each row

Moran

Moran()

get_global()

Moran.get_global(
    df: DataFrame,
    two_tailed: bool = True,
    id_column: str = 'id',
    value_column: str = 'value',
) -> MoranResult

Parameters

DataFrame

required

bool

default:"True"

str

default:"'id'"

str

default:"'value'"

MoranResult

MoranResult(i: float, p_norm: float, z_norm: float)

Parameters

float

required

float

required

float

required

spatially_stratified_sample

Import from sedona.stats.sampling.spatially_stratified_sampling.

spatially_stratified_sample(
    dataframe: DataFrame,
    fraction: float,
    partition_count: int,
    geometry: Optional[str] = None,
    seed: int = 42,
)

Spatially stratified sampling of a DataFrame containing spatial data.

Parameters

DataFrame

required

DataFrame containing spatial data to be sampled. Must contain a geometry column.

float

required

Sampling rate between 0 and 1

int

required

Number of partitions to divide the data into. If not a perfect square, the number of partitions in each dimension will be rounded to the nearest integer.

Optional[str]

default:"None"

Column containing the geometry data. Default is “geometry”

int

default:"42"

Seed for sampling the data

Returns

the input DataFrame sampled down to the specified rate

​add_binary_distance_band_column

​Parameters

​Returns

​add_distance_band_column

​Parameters

​Returns

​add_weighted_distance_band_column

​Parameters

​Returns

​dbscan

​Parameters

​Returns

​g_local

​Parameters

​Returns

​local_outlier_factor

​Parameters

​Returns

​Moran

​get_global()

​Parameters

​MoranResult

​Parameters

​spatially_stratified_sample

​Parameters

​Returns

add_binary_distance_band_column

Parameters

Returns

add_distance_band_column

Parameters

Returns

add_weighted_distance_band_column

Parameters

Returns

dbscan

Parameters

Returns

g_local

Parameters

Returns

local_outlier_factor

Parameters

Returns

Moran

get_global()

Parameters

MoranResult

Parameters

spatially_stratified_sample

Parameters

Returns