Apppublic

mlukac/xrf-explorer-dev

sourceHugging Faceupdated 10mo agoView on Hugging Face

0likes

App README

XRF Explorer

A data analysis and visualization platform for X-ray Fluorescence (XRF) spectroscopy data

Quick Start

Installation

bash

# Clone the repository
git clone <repository-url>
cd xrf-explorer

# Create virtual environment
python3 -m venv .venv
source ./.venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install as editable package
pip install -e .

Running the Application

bash

# Development mode with debugging
python scripts/serve.py --help  # See available options
python scripts/serve.py --verbose --enable-time-me

# Production mode  
python xrf_explorer/app.py

Basic Usage

1.Load XRF Data: Import your XRF measurement files
2.Apply Transforms: Use despiking, smoothing, and statistical transforms
3.Visualize Results: View processed data in interactive charts
4.Export Analysis: Save processed datasets and visualizations

Technical Specifications

System Requirements

—Python: >= 3.11
—Key Dependencies:
—Panel 1.3.4 (UI framework)
—Bokeh 3.3.4 (visualization engine)
—Pandas, NumPy (data processing)
—Scikit-learn (statistical analysis)

Architecture Principles

—Modular Design: Pluggable transforms and views
—Type Safety: Comprehensive type hints throughout
—Event-Driven: Reactive UI updates via Panel/Bokeh
—Extensible: Easy to add new transforms and visualizations

Development Tools

—Formatting: Black, isort
—Linting: Pylint
—Type Checking: MyPy
—Testing: Pytest

System Overview

XRF Explorer processes and visualizes geochemical measurement data through a modular pipeline architecture built with Panel/Bokeh.

mermaid

graph TB
    User["👤 Geologist/Scientist<br/>Analyzes XRF data"]
    XRF["🔬 XRF Explorer<br/>Data analysis and visualization platform"]
    Files["📁 XRF Data Files<br/>Raw measurement data"]
    
    User -->|Uses| XRF
    XRF -->|Reads| Files
    
    classDef userClass fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    classDef systemClass fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    classDef dataClass fill:#e8f5e8,stroke:#388e3c,stroke-width:2px
    
    class User userClass
    class XRF systemClass
    class Files dataClass

High-Level Architecture

The system follows a modular component architecture with clear separation of concerns:

mermaid

flowchart TD
    A1["📊 XRF Data Files"] --> B1{"🔍 XRF File Loader"}
    A2["📈 Curve Data Files<br/>(Mass Spec, Gamma Ray, Drilling)"] --> B2{"🔍 Curve File Loader"}
    
    B1 --> C1["✅ XRF Data Validation"]
    B2 --> C2["✅ Curve Data Validation"]
    
    C1 --> D1["⚙️ XRF Transform Pipeline"]
    C2 --> D2["⚙️ Curve Transform Pipeline"]
    
    D1 --> E1["📊 Statistics Transform<br/>(Mean, PCA, KMeans)"]
    D1 --> F1["📈 Correlation Analysis"]
    D1 --> G1["📊 KDE Transform"]
    
    D2 --> E2["🔧 Despiking Transform"]
    D2 --> F2["📈 Smoothing Transform"]
    
    E1 --> H1["📊 XRF Visualization Views<br/>(Element Data, Mean, Correlation, PCA)"]
    F1 --> H1
    G1 --> H1
    
    E2 --> H2["📈 Curve Visualization Views<br/>(Single Curve Views)"]
    F2 --> H2
    
    H1 --> I["🔄 Frame Merger & CDS Manager"]
    H2 --> I
    
    I --> J["📊 Combined Curve View"]
    I --> K["🖥️ Panel UI Dashboard"]
    
    style A1 fill:#e3f2fd
    style A2 fill:#f3e5f5
    style D1 fill:#fff3e0
    style D2 fill:#e8f5e8
    style I fill:#fce4ec
    style K fill:#f3e5f5

Core Architecture

Main Components

The application uses a handler-controller-view pattern with pluggable transforms:

mermaid

classDiagram
    class ApplicationHandler {
        <<handler>>
        -ctx: ApplicationContext
        -app_def: ApplicationDef
        -curve_registry: CurveRegistry
        -triggerer: Triggerer
        +setup_application()
        +coordinate_pipeline()
    }
    
    class param_Parameterized {
        <<framework>>
        +param: Parameters
        +watch()
        +trigger()
    }
    
    class CurveController {
        <<controller>>
        +curve_names: List
        +stacked_selector: Selector
        +curve_selector: Selector
        +curve_color: Color
        +handle_curve_selection()
    }
    
    class StatsController {
        <<controller>>
        -app: ApplicationHandlerProtocol
        -columns: List[str]
        +handle(event: XRFDatasetUpdated)
    }
    
    class SignalDespikingOperator {
        <<transform>>
        +signal: Array
        +threshold: Number
        +interp_radius: Number
        +disabled: Boolean
        +transform()
    }
    
    class SignalSmoothingOperator {
        <<transform>>
        +signal: Array
        +window_length: Integer
        +poly_order: Integer
        +disabled: Boolean
        +transform()
    }
    
    class MeanTransform {
        <<transform>>
        +output: DataFrame
        +transform(df: DataFrame): Self
    }
    
    param_Parameterized <|-- CurveController : extends
    param_Parameterized <|-- SignalDespikingOperator : extends
    param_Parameterized <|-- SignalSmoothingOperator : extends
    param_Parameterized <|-- MeanTransform : extends
    ApplicationHandler --> CurveController : uses
    ApplicationHandler --> StatsController : uses

Package Structure

mermaid

classDiagram
    namespace core {
        class Core {
            <<package>>
            Context management
            Type definitions
            Event system
        }
    }
    namespace controllers {
        class Controllers {
            <<package>>
            User interaction logic
            State coordination
        }
    }
    namespace handlers {
        class Handlers {
            <<package>>
            Business logic
            Event handling
        }
    }
    namespace transforms {
        class Transforms {
            <<package>>
            Data processing
            Statistical analysis
        }
    }
    namespace views {
        class Views {
            <<package>>
            Visualization components
            Chart generation
        }
    }
    namespace pipeline {
        class Pipeline {
            <<package>>
            Component assembly
            UI orchestration
        }
    }
    
    Controllers --> Core : depends on
    Controllers --> Handlers : coordinates
    Handlers --> Transforms : uses
    Handlers --> Views : updates
    Pipeline --> Controllers : assembles
    Pipeline --> Handlers : orchestrates

System Behavior

Data Processing Flow

Shows how user interactions flow through the system to update visualizations:

mermaid

sequenceDiagram
    participant User
    participant UI as Panel UI
    participant XRFController as XRF Controllers
    participant CurveController as Curve Controllers
    participant XRFHandler as XRF Handler
    participant CurveHandler as Curve Handler
    participant XRFTransform as XRF Transforms
    participant CurveTransform as Curve Transforms
    participant Views
    
    Note over User,Views: XRF Data Processing Flow
    User->>UI: Load XRF File
    UI->>XRFController: xrf_file_selected
    XRFController->>XRFHandler: process_xrf_file
    
    XRFHandler->>XRFTransform: apply_statistics
    XRFTransform-->>XRFHandler: mean, pca, kmeans
    
    XRFHandler->>XRFTransform: apply_correlation
    XRFTransform-->>XRFHandler: correlation_matrix
    
    XRFHandler->>Views: update_xrf_views
    Views-->>UI: render_element_data, mean_view, pca_views
    
    Note over User,Views: Curve Data Processing Flow
    User->>UI: Load Curve File (Mass Spec/Gamma Ray/Drilling)
    UI->>CurveController: curve_file_selected
    CurveController->>CurveHandler: process_curve_file
    
    CurveHandler->>CurveTransform: apply_despiking
    CurveTransform-->>CurveHandler: despiked_data
    
    CurveHandler->>CurveTransform: apply_smoothing
    CurveTransform-->>CurveHandler: smoothed_data
    
    CurveHandler->>Views: update_curve_views
    Views-->>UI: render_single_curve_view
    
    Note over User,Views: Combined Visualization
    Views->>Views: merge_all_data_sources
    Views-->>UI: render_combined_curve_view
    UI-->>User: Display Integrated Dashboard

Application State Management

Core application states and transitions:

mermaid

stateDiagram-v2
    [*] --> Initializing
    Initializing --> Ready: setup_complete
    
    Ready --> XRF_Loading: load_xrf_file
    Ready --> Curve_Loading: load_curve_file
    
    XRF_Loading --> XRF_Processing: xrf_validation_passed
    XRF_Loading --> Error: xrf_validation_failed
    
    Curve_Loading --> Curve_Processing: curve_validation_passed
    Curve_Loading --> Error: curve_validation_failed
    
    XRF_Processing --> XRF_Transforming: xrf_data_loaded
    XRF_Transforming --> XRF_Visualizing: xrf_transforms_applied
    XRF_Visualizing --> Ready: xrf_views_updated
    
    Curve_Processing --> Curve_Transforming: curve_data_loaded
    Curve_Transforming --> Curve_Visualizing: curve_transforms_applied
    Curve_Visualizing --> Ready: curve_views_updated
    
    XRF_Visualizing --> Data_Merging: both_datasets_available
    Curve_Visualizing --> Data_Merging: both_datasets_available
    
    Data_Merging --> Combined_View: merge_complete
    Combined_View --> Ready: combined_visualization_ready
    
    Error --> Ready: error_handled
    Ready --> [*]: application_shutdown
    
    note right of XRF_Transforming
        Apply statistics, PCA,
        correlation analysis
    end note
    
    note right of Curve_Transforming
        Apply despiking,
        smoothing operations
    end note
    
    note right of Data_Merging
        Frame merger combines
        all data sources
    end note

Transform System Architecture

Transform Pipeline Design

The transform system uses a pluggable architecture for extensible data processing:

mermaid

classDiagram
    class param_Parameterized {
        <<framework>>
        +param: Parameters
        +watch()
        +trigger()
    }
    
    class SignalDespikingOperator {
        <<transform>>
        +signal: Array
        +threshold: Number
        +interp_radius: Number  
        +disabled: Boolean
        +output: Array
        +transform()
        -_despike_signal(): Array
    }
    
    class SignalSmoothingOperator {
        <<transform>>
        +signal: Array
        +window_length: Integer
        +poly_order: Integer
        +disabled: Boolean
        +output: Array
        +transform()
        -_apply_savgol_filter(): Array
    }
    
    class MeanTransform {
        <<transform>>
        +output: DataFrame
        +transform(df: DataFrame): Self
    }
    
    class KMeansTransform {
        <<transform>>
        +n_clusters: Integer
        +output: DataFrame
        +transform(df: DataFrame): Self
    }
    
    class PCATransform {
        <<transform>>
        +n_components: Integer
        +output: DataFrame
        +transform(df: DataFrame): Self
    }
    
    class KernelDensityTransform {
        <<transform>>
        +bandwidth: Number
        +output: DataFrame
        +transform(df: DataFrame): Self
    }
    
    param_Parameterized <|-- SignalDespikingOperator : extends
    param_Parameterized <|-- SignalSmoothingOperator : extends
    param_Parameterized <|-- MeanTransform : extends
    param_Parameterized <|-- KMeansTransform : extends
    param_Parameterized <|-- PCATransform : extends
    param_Parameterized <|-- KernelDensityTransform : extends

Data Flow Through Transforms

mermaid

flowchart TB
    subgraph XRF ["🔬 XRF Data Processing Pipeline"]
        A1["📊 Raw XRF Data"] --> B1["🔍 XRF File Loading"]
        B1 --> C1["⚙️ XRF Processing"]
        
        C1 --> D1["📊 MeanTransform"]
        C1 --> E1["🎯 PCATransform"] 
        C1 --> F1["🔄 KMeansTransform"]
        C1 --> G1["📈 KernelDensityTransform"]
        
        D1 --> H1["📊 Statistical Analysis"]
        E1 --> H1
        F1 --> H1
        G1 --> H1
        
        H1 --> I1["📊 XRF Views<br/>(Element Data, Mean, Correlation, PCA)"]
    end
    
    subgraph CURVE ["📈 Curve Data Processing Pipeline"]
        A2["📈 Raw Curve Data<br/>(Mass Spec, Gamma Ray, Drilling)"] --> B2["🔍 Curve File Loading"]
        B2 --> C2["⚙️ Curve Processing"]
        
        C2 --> D2["🔧 SignalDespikingOperator"]
        C2 --> E2["📈 SignalSmoothingOperator"]
        
        D2 --> F2["🚫 Spike Removal"]
        E2 --> G2["📈 Savgol Smoothing"]
        
        F2 --> H2["📊 Processed Curve Data"]
        G2 --> H2
        
        H2 --> I2["📈 Single Curve Views"]
    end
    
    subgraph MERGE ["🔄 Data Integration"]
        I1 --> J["🔄 Frame Merger"]
        I2 --> J
        J --> K["💾 CDS Manager"]
        K --> L["📊 Combined Curve View"]
    end
    
    L --> M["🖥️ Panel UI Dashboard"]
    
    style XRF fill:#e3f2fd
    style CURVE fill:#e8f5e8
    style MERGE fill:#fce4ec
    style M fill:#f3e5f5

Factory & Builder System Architecture

Component Factory Pattern

The application uses a comprehensive factory system to manage complex component creation and dependency injection:

mermaid

classDiagram
    class ComponentsFactory {
        <<main-factory>>
        +create_all(app: ApplicationHandlerProtocol): AllComponents
        -Phase1: File setup
        -Phase2: Controllers  
        -Phase3: Slice components
        -Phase4: Curve components
        -Phase5: CDS components
        -Phase6: Views
        -Phase7: Helpers
    }
    
    class FileInputFactory {
        <<factory>>
        +create_all(app): Dict[DatasetType, FileInput]
    }
    
    class FileHandlerFactory {
        <<factory>>
        +create_all(app, file_inputs): FileHandlers
    }
    
    class DatasetComponentsFactory {
        <<factory>>
        +create(app, dataset_type, file_handler): DatasetComponents
    }
    
    class XRFControllersFactory {
        <<factory>>
        +create(app, xrf_file_handler): XRFControllers
    }
    
    class CDSComponentsFactory {
        <<factory>>
        +create(app, xrf_controllers, slice_components): CDSComponents
    }
    
    class ViewComponentsFactory {
        <<factory>>
        +create(app, controllers, cds_components): ViewComponents
    }
    
    class AnnotationComponentsFactory {
        <<factory>>
        +create(app): AnnotationComponents
    }
    
    ComponentsFactory --> FileInputFactory : uses
    ComponentsFactory --> FileHandlerFactory : uses
    ComponentsFactory --> DatasetComponentsFactory : uses
    ComponentsFactory --> XRFControllersFactory : uses
    ComponentsFactory --> CDSComponentsFactory : uses
    ComponentsFactory --> ViewComponentsFactory : uses
    ComponentsFactory --> AnnotationComponentsFactory : uses

Pipeline Builder System

The application uses specialized builders to construct event processing pipelines:

mermaid

classDiagram
    class AllPipelineBuilder {
        <<orchestrator>>
        +build_all_pipelines(app, components, verbose): void
    }
    
    class FileLoadingPipelineBuilder {
        <<builder>>
        +build_file_load_pipeline(app, components): void
    }
    
    class XRFPipelineBuilder {
        <<builder>>
        +build_xrf_load_pipeline(app, components): void
        +build_xrf_load_handled_pipeline(app, components): void
        +build_xrf_controller_pipelines(app, components): void
    }
    
    class CurvePipelineBuilder {
        <<builder>>
        +build_comb_ctrl_pipeline(app, components): void
        +build_update_viz_columns_pipeline(app, components): void
        +build_curve_dataset_pipelines(app, components, dataset_type): void
    }
    
    class SlicePipelineBuilder {
        <<builder>>
        +build_slice_button_pipeline(app, components): void
        +build_slice_step_pipeline(app, components): void
        +build_slice_fully_completed_pipeline(app, components): void
    }
    
    class AnnotationPipelineBuilder {
        <<builder>>
        +build_annotation_pipelines(app, components): void
    }
    
    class HandlerPipeline {
        <<infrastructure>>
        -event_type: EventType
        -handlers: List[HandlerSequence]
        +add(handler_sequence): HandlerPipeline
        +transform(event_transformer): HandlerPipeline
        +build(): void
    }
    
    class HandlerSequence {
        <<infrastructure>>
        -handlers: List[Handler]
        -event_filter: Optional[Filter]
        +__init__(handlers, event_filter, label): void
    }
    
    AllPipelineBuilder --> FileLoadingPipelineBuilder : coordinates
    AllPipelineBuilder --> XRFPipelineBuilder : coordinates
    AllPipelineBuilder --> CurvePipelineBuilder : coordinates
    AllPipelineBuilder --> SlicePipelineBuilder : coordinates
    AllPipelineBuilder --> AnnotationPipelineBuilder : coordinates
    
    XRFPipelineBuilder --> HandlerPipeline : creates
    CurvePipelineBuilder --> HandlerPipeline : creates
    SlicePipelineBuilder --> HandlerPipeline : creates
    HandlerPipeline --> HandlerSequence : contains

UI Builder Pattern

Panel UI components are constructed using a builder pattern for flexible layout assembly:

mermaid

classDiagram
    class PanelBuilder~T~ {
        <<abstract>>
        -app: Optional[ApplicationHandler]
        -view_type: Optional[ViewType]
        -component: Optional[T]
        +with_viz_switch(app, view_type): PanelBuilder[T]
        +build(): T
        #_build_component()*: T
    }
    
    class ColumnBuilder {
        <<concrete>>
        -components: List[Any]
        -sizing_mode: Optional[str]
        -aspect_ratio: Optional[float]
        -align: Optional[str]
        +add_builder(builder: PanelBuilder): ColumnBuilder
        +add_component(component): ColumnBuilder
        +sizing_mode(mode: str): ColumnBuilder
        +aspect_ratio(ratio: float): ColumnBuilder
        #_build_component(): pn.Column
    }
    
    class RowBuilder {
        <<concrete>>
        -components: List[Any]
        -sizing_mode: Optional[str]
        +add_builder(builder: PanelBuilder): RowBuilder
        +add_component(component): RowBuilder
        #_build_component(): pn.Row
    }
    
    class TabsBuilder {
        <<concrete>>
        -tabs: List[Tuple[str, Any]]
        +add_tab(name: str, component): TabsBuilder
        #_build_component(): pn.Tabs
    }
    
    PanelBuilder <|-- ColumnBuilder : extends
    PanelBuilder <|-- RowBuilder : extends
    PanelBuilder <|-- TabsBuilder : extends

Component Assembly Flow

Shows how the factory system creates the complete application in dependency-ordered phases:

mermaid

flowchart TD
    A["🚀 Application Start"] --> B["📁 Phase 1: File Setup"]
    B --> B1["FileInputFactory<br/>Creates file input widgets"]
    B --> B2["FileHandlerFactory<br/>Creates XRF & curve handlers"]
    
    B1 --> C["🎛️ Phase 2: Controllers"]
    B2 --> C
    C --> C1["XRFControllersFactory<br/>Element selectors, PCA, Stats"]
    C --> C2["CombCurveController<br/>Combined visualization control"]
    
    C1 --> D["✂️ Phase 3: Slice Components"]
    C2 --> D
    D --> D1["SliceComponentsFactory<br/>Range tools, slice handlers"]
    
    D1 --> E["📊 Phase 4: Curve Components"]
    E --> E1["DatasetComponentsFactory<br/>Mass spec, Gamma ray, Drilling"]
    
    E1 --> F["💾 Phase 5: CDS Components"]
    F --> F1["CDSComponentsFactory<br/>Column data sources, Frame merger"]
    
    F1 --> G["🎨 Phase 6: Views"]
    G --> G1["ViewComponentsFactory<br/>All visualization views"]
    
    G1 --> H["📝 Phase 7: Annotations"]
    H --> H1["AnnotationComponentsFactory<br/>Point annotation system"]
    
    H1 --> I["⚙️ Phase 8: Pipeline Assembly"]
    I --> I1["AllPipelineBuilder<br/>Event processing pipelines"]
    
    I1 --> J["✅ Application Ready"]
    
    style A fill:#e3f2fd
    style J fill:#e8f5e8
    style B fill:#fff3e0
    style C fill:#fff3e0
    style D fill:#fff3e0
    style E fill:#fff3e0
    style F fill:#fff3e0
    style G fill:#fff3e0
    style H fill:#fff3e0
    style I fill:#fff3e0

For detailed API documentation and development guides, see individual module docstrings and the `/docs` directory.