Apppublic

cooldragon12/borehole-outlier-detection-pipeline-api

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face

0likes

App README

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

FastAPI Backend - DBSCAN Outlier Detection

A FastAPI-based backend service for DBSCAN clustering-based outlier detection with RobustScaler preprocessing. Provides a 4-step pipeline for data analysis, preprocessing, detection, and results aggregation.

Quick Setup

1. Install Python Dependencies

bash

# Using pip
pip install -r requirements.txt

# Or with virtual environment (recommended)
python -m venv env
source env/bin/activate  # On Windows: env\Scripts\activate
pip install -r requirements.txt

2. Run the Server

bash

# Using Uvicorn
python -m uvicorn src.main:app --reload --port 8000

# Or directly with Python
python src.main.py

The API will be available at http://localhost:8000

3. Access Documentation

—Swagger UI (interactive): http://localhost:8000/docs
—ReDoc (static): http://localhost:8000/redoc

API Endpoints

DBSCAN Pipeline (4-Step Workflow)

Step 1: Analyze Data

POST /api/dbscan/analyze-data

Input: CSV file upload (multipart/form-data)

Response:

json

{
  "status": "success",
  "data_analysis": {
    "shape": [1000, 28],
    "columns": ["col1", "col2", ...],
    "dtypes": {"col1": "float64", ...},
    "missing_values": {"col1": 0, ...},
    "statistics": {...}
  }
}

Step 2: Preprocess Data

POST /api/dbscan/preprocess

Parameters:

—handle_missing (string): "drop", "mean", "median", "ffill", "bfill" (default: "drop")
—scale_method (string): "robust" (default, only supported method)

Response:

json

{
  "status": "success",
  "preprocessing_report": {
    "numeric_cols": [25 columns scaled with RobustScaler],
    "binary_cols": [3 columns preserved unscaled],
    "categorical_cols": [],
    "missing_values_count": 0
  }
}

Processing Details:

—Identifies numeric, binary, and categorical features
—Applies RobustScaler (median-centered, IQR-normalized) to numeric columns only
—Preserves binary features (0/1 values) unscaled
—Handles missing values using specified strategy

Step 3: Detect Outliers with DBSCAN

POST /api/dbscan/detect

Parameters (optional, uses optimal defaults):

—eps (float): Neighborhood radius (default: 5.193866452787452)
—min_samples (int): Minimum cluster density (default: 5)

Response:

json

{
  "status": "success",
  "detection_results": {
    "outlier_count": 67,
    "outlier_percentage": 6.7,
    "normal_count": 933
  }
}

Step 4: Get Complete Results

GET /api/dbscan/results

Response:

json

{
  "status": "success",
  "outlier_count": 67,
  "normal_count": 933,
  "outlier_indices": [5, 12, 45, ...],
  "outlier_records": [{...}, ...],
  "normal_preview": [{...}, ...],
  "detection_report": {...},
  "preprocessing_report": {...},
  "total_records": 1000
}

Utility Endpoints

Suggest Optimal EPS

POST /api/dbscan/suggest-eps

Parameters:

—k (int): k-distance curve parameter (default: 5)
—quantile (float): Quantile for eps selection (default: 0.9)

Response:

json

{
  "suggested_eps": 5.193866452787452,
  "k_distance_stats": {...},
  "optimal_config": {"eps": 5.194, "min_samples": 5}
}

Reset Pipeline State

GET /api/reset

Clears all stored data and resets the pipeline state.

DBSCAN Configuration (Optimal Parameters)

python

eps = 5.193866452787452          # From k-distance graph analysis
min_samples = 5                  # Minimum cluster density
metric = 'euclidean'             # Distance metric

Why these values?

—eps: Derived from k-distance quantiles (k=5, 90th percentile)
—min_samples: Minimum points in neighborhood for core point
—metric: Euclidean distance works well for scaled features

RobustScaler Preprocessing

Why RobustScaler?

—Uses median and IQR (Interquartile Range) instead of mean/std
—Robust to outliers in the data
—Better for skewed distributions

Formula:

X_scaled = (X - median) / IQR

Features:

—25 numeric columns are scaled
—3 binary columns (0/1 values) are preserved unscaled
—Categorical columns are excluded from scaling

Project Structure

backend/
├── src/
│   ├── main.py                          # Application entry point
│   ├── api/
│   │   ├── preprocess.py                # Generic preprocessing endpoints
│   │   ├── predict.py                   # DBSCAN pipeline endpoints
│   │   └── import_data.py               # Data import utilities
│   ├── services/
│   │   ├── preprocess_service.py        # RobustScaler preprocessing logic
│   │   └── dbscan_service.py            # DBSCAN detection logic
│   ├── core/
│   │   ├── config.py                    # Configuration (CORS, etc.)
│   │   ├── models.py                    # Pydantic request/response models
│   │   ├── security.py                  # Security utilities
│   │   └── middleware.py                # Custom middleware
│   └── __init__.py
├── Dockerfile                           # Docker configuration
├── requirements.txt                     # Python dependencies
└── README.md                            # This file

Example Workflow

Using cURL

bash

# Step 1: Upload and analyze data
curl -X POST "http://localhost:8000/api/dbscan/analyze-data" \
  -F "file=@data.csv"

# Step 2: Preprocess
curl -X POST "http://localhost:8000/api/dbscan/preprocess" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "handle_missing=drop&scale_method=robust"

# Step 3: Detect outliers
curl -X POST "http://localhost:8000/api/dbscan/detect"

# Step 4: Get results
curl -X GET "http://localhost:8000/api/dbscan/results"

Using Python Requests

python

import requests

BASE_URL = "http://localhost:8000/api"

# Step 1: Analyze
with open('data.csv', 'rb') as f:
    resp = requests.post(f"{BASE_URL}/dbscan/analyze-data", files={'file': f})
    print(resp.json())

# Step 2: Preprocess
resp = requests.post(
    f"{BASE_URL}/dbscan/preprocess",
    data={"handle_missing": "drop", "scale_method": "robust"}
)
print(resp.json())

# Step 3: Detect
resp = requests.post(f"{BASE_URL}/dbscan/detect")
print(resp.json())

# Step 4: Get results
resp = requests.get(f"{BASE_URL}/dbscan/results")
results = resp.json()
print(f"Outliers: {results['outlier_count']}")
print(f"Normal: {results['normal_count']}")

Data Requirements

Input CSV Format:

—First row contains column headers
—Numeric columns: Automatically detected and scaled with RobustScaler
—Binary columns: Automatically detected (values = {0, 1}) and preserved
—Missing values: Handled by specified strategy (default: drop)

Example Structure:

feature_1,feature_2,...,feature_25,binary_1,binary_2,binary_3
1.5,2.3,...,5.1,0,1,0
2.1,3.4,...,4.9,1,0,1
...

Environment Configuration

Create a .env file or set environment variables:

bash

FASTAPI_ENV=development
DEBUG=true
ORIGIN=http://localhost:3000

Dependencies

See requirements.txt for the complete list:

—fastapi - Web framework
—uvicorn - ASGI server
—pandas - Data manipulation
—numpy - Numerical computing
—scikit-learn - DBSCAN & RobustScaler
—pydantic - Data validation
—python-multipart - File upload handling

Troubleshooting

Port 8000 already in use

bash

# Use a different port
python -m uvicorn src.main:app --port 8001

# Or kill the process using port 8000
lsof -i :8000  # On Linux/Mac
netstat -ano | findstr :8000  # On Windows

Import errors

bash

# Ensure you're in a virtual environment
python -m venv env
source env/bin/activate  # Or env\Scripts\activate on Windows
pip install -r requirements.txt

CORS errors from frontend

—Check src/core/config.py for allow_origins settings
—Ensure frontend is included in CORS configuration
—Restart backend after config changes

Performance Notes

—Data size: Tested with 1000+ records
—DBSCAN complexity: O(n log n) with spatial indexing
—Preprocessing speed: <100ms for 1000 records
—Memory usage: ~50MB for 1000 records with 28 features

Docker Deployment

bash

# Build image
docker build -t outlier-detection-backend .

# Run container
docker run -p 8000:8000 outlier-detection-backend

# Or with Docker Compose (from root directory)
docker-compose up backend

References

—DBSCAN paper: https://en.wikipedia.org/wiki/DBSCAN
—RobustScaler: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html
—FastAPI docs: https://fastapi.tiangolo.com/

License

Part of a civil engineering thesis on automated outlier detection.