The following content is a read-only preview of an executable Jupyter notebook.To run this notebook interactively:

Go to Wherobots Cloud.
Start a runtime.
Open the notebook.
In the Jupyter Launcher:
1. Click File > Open Path.
2. Paste the following path to access this notebook: examples/Analyzing_Data/Local_Outlier_Factor.ipynb
3. Click Enter.

Local Outlier Factor (LOF) is a common algorithm for identifying data points that are inliers/outliers relative to their neighbors. The algorithm works by comparing how close an element is to its neighbors vs how close they are to their neighbors. The number of neighbors to use, k, is set by the user. Scores much less than one are inliers, scores much greater are outliers, and those near one are neither. This demo is derived from the scikit-learn Local Outlier Detection demo.

Define Sedona Context

from sedona.spark import *

config = SedonaContext.builder().getOrCreate()
sedona = SedonaContext.create(config)

Data Generation

We generate some data. Most of it is random, but some data is explicitly designed to be outliers

import numpy as np
import pyspark.sql.functions as f

from sedona.spark import *

np.random.seed(42)

X_inliers = 0.3 * np.random.randn(100, 2)
X_inliers = np.r_[X_inliers + 2, X_inliers - 2]
X_outliers = np.random.uniform(low=-4, high=4, size=(20, 2))
X = np.r_[X_inliers, X_outliers]

Generation LOF

We use the LOF implementation in Wherobots to generate this statistic on the data. We set k to 20.

df = sedona.createDataFrame(X).select(ST_MakePoint(f.col("_1"), f.col("_2")).alias("geometry"))
outliers_df = local_outlier_factor(df, 20)

outliers_df.show()

Visualization

We visualize the results with matplotlib, pulling the point coordinates out of the geometry column with ST_X and ST_Y. Some manipulations are made to the data to improve the clarity of the visualization.

import matplotlib.pyplot as plt

pdf = (outliers_df
       .withColumn("lof", f.col("lof") * 50)
       .selectExpr("ST_X(geometry) AS x", "ST_Y(geometry) AS y", "lof")
       .toPandas()
      )

fig, ax = plt.subplots(figsize=(10, 8))
# Circle size scales with the LOF score, so outliers stand out
ax.scatter(pdf["x"], pdf["y"], s=pdf["lof"], edgecolors="r", facecolors="none", label="Outlier Scores")
ax.scatter(pdf["x"], pdf["y"], s=1, color="k", label="Data points")
ax.set_title("LOF Scores")
ax.legend()
plt.show()

Use Cases

Getting Started

Spatial Queries

WherobotsDB

Data Connections

RasterFlow

Advanced Topics

GeoStats & Map Matching

Local Outlier Factor

Define Sedona Context

Data Generation

Generation LOF

Visualization

​Define Sedona Context

​Data Generation

​Generation LOF

​Visualization

Define Sedona Context

Data Generation

Generation LOF

Visualization