mlukac/xrf-explorer-dev
0
1---2app_file: app.py3pinned: false4title: XRF Explorer Dev5sdk: docker6emoji: 📊7colorFrom: green8colorTo: yellow9---10 11# XRF Explorer12 13> A data analysis and visualization platform for X-ray Fluorescence (XRF) spectroscopy data14 15## Quick Start16 17### Installation18 19```bash20# Clone the repository21git clone <repository-url>22cd xrf-explorer23 24# Create virtual environment25python3 -m venv .venv26source ./.venv/bin/activate # On Windows: .venv\Scripts\activate27 28# Install dependencies29pip install -r requirements.txt30 31# Install as editable package32pip install -e .33```34 35### Running the Application36 37```bash38# Development mode with debugging39python scripts/serve.py --help # See available options40python scripts/serve.py --verbose --enable-time-me41 42# Production mode 43python xrf_explorer/app.py44```45 46### Basic Usage47 481. **Load XRF Data**: Import your XRF measurement files492. **Apply Transforms**: Use despiking, smoothing, and statistical transforms503. **Visualize Results**: View processed data in interactive charts514. **Export Analysis**: Save processed datasets and visualizations52 53## Technical Specifications54 55### System Requirements56 57- **Python**: >= 3.1158- **Key Dependencies**: 59 - Panel 1.3.4 (UI framework)60 - Bokeh 3.3.4 (visualization engine)61 - Pandas, NumPy (data processing)62 - Scikit-learn (statistical analysis)63 64### Architecture Principles65 66- **Modular Design**: Pluggable transforms and views67- **Type Safety**: Comprehensive type hints throughout68- **Event-Driven**: Reactive UI updates via Panel/Bokeh69- **Extensible**: Easy to add new transforms and visualizations70 71### Development Tools72 73- **Formatting**: Black, isort74- **Linting**: Pylint 75- **Type Checking**: MyPy76- **Testing**: Pytest77 78---79## System Overview80 81XRF Explorer processes and visualizes geochemical measurement data through a modular pipeline architecture built with Panel/Bokeh.82 83```mermaid84graph TB85 User["👤 Geologist/Scientist<br/>Analyzes XRF data"]86 XRF["🔬 XRF Explorer<br/>Data analysis and visualization platform"]87 Files["📁 XRF Data Files<br/>Raw measurement data"]88 89 User -->|Uses| XRF90 XRF -->|Reads| Files91 92 classDef userClass fill:#e3f2fd,stroke:#1976d2,stroke-width:2px93 classDef systemClass fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px94 classDef dataClass fill:#e8f5e8,stroke:#388e3c,stroke-width:2px95 96 class User userClass97 class XRF systemClass98 class Files dataClass99```100 101## High-Level Architecture102 103The system follows a modular component architecture with clear separation of concerns:104 105```mermaid106flowchart TD107 A1["📊 XRF Data Files"] --> B1{"🔍 XRF File Loader"}108 A2["📈 Curve Data Files<br/>(Mass Spec, Gamma Ray, Drilling)"] --> B2{"🔍 Curve File Loader"}109 110 B1 --> C1["✅ XRF Data Validation"]111 B2 --> C2["✅ Curve Data Validation"]112 113 C1 --> D1["⚙️ XRF Transform Pipeline"]114 C2 --> D2["⚙️ Curve Transform Pipeline"]115 116 D1 --> E1["📊 Statistics Transform<br/>(Mean, PCA, KMeans)"]117 D1 --> F1["📈 Correlation Analysis"]118 D1 --> G1["📊 KDE Transform"]119 120 D2 --> E2["🔧 Despiking Transform"]121 D2 --> F2["📈 Smoothing Transform"]122 123 E1 --> H1["📊 XRF Visualization Views<br/>(Element Data, Mean, Correlation, PCA)"]124 F1 --> H1125 G1 --> H1126 127 E2 --> H2["📈 Curve Visualization Views<br/>(Single Curve Views)"]128 F2 --> H2129 130 H1 --> I["🔄 Frame Merger & CDS Manager"]131 H2 --> I132 133 I --> J["📊 Combined Curve View"]134 I --> K["🖥️ Panel UI Dashboard"]135 136 style A1 fill:#e3f2fd137 style A2 fill:#f3e5f5138 style D1 fill:#fff3e0139 style D2 fill:#e8f5e8140 style I fill:#fce4ec141 style K fill:#f3e5f5142```143 144## Core Architecture145 146### Main Components147 148The application uses a handler-controller-view pattern with pluggable transforms:149 150```mermaid151classDiagram152 class ApplicationHandler {153 <<handler>>154 -ctx: ApplicationContext155 -app_def: ApplicationDef156 -curve_registry: CurveRegistry157 -triggerer: Triggerer158 +setup_application()159 +coordinate_pipeline()160 }161 162 class param_Parameterized {163 <<framework>>164 +param: Parameters165 +watch()166 +trigger()167 }168 169 class CurveController {170 <<controller>>171 +curve_names: List172 +stacked_selector: Selector173 +curve_selector: Selector174 +curve_color: Color175 +handle_curve_selection()176 }177 178 class StatsController {179 <<controller>>180 -app: ApplicationHandlerProtocol181 -columns: List[str]182 +handle(event: XRFDatasetUpdated)183 }184 185 class SignalDespikingOperator {186 <<transform>>187 +signal: Array188 +threshold: Number189 +interp_radius: Number190 +disabled: Boolean191 +transform()192 }193 194 class SignalSmoothingOperator {195 <<transform>>196 +signal: Array197 +window_length: Integer198 +poly_order: Integer199 +disabled: Boolean200 +transform()201 }202 203 class MeanTransform {204 <<transform>>205 +output: DataFrame206 +transform(df: DataFrame): Self207 }208 209 param_Parameterized <|-- CurveController : extends210 param_Parameterized <|-- SignalDespikingOperator : extends211 param_Parameterized <|-- SignalSmoothingOperator : extends212 param_Parameterized <|-- MeanTransform : extends213 ApplicationHandler --> CurveController : uses214 ApplicationHandler --> StatsController : uses215```216 217### Package Structure218 219```mermaid220classDiagram221 namespace core {222 class Core {223 <<package>>224 Context management225 Type definitions226 Event system227 }228 }229 namespace controllers {230 class Controllers {231 <<package>>232 User interaction logic233 State coordination234 }235 }236 namespace handlers {237 class Handlers {238 <<package>>239 Business logic240 Event handling241 }242 }243 namespace transforms {244 class Transforms {245 <<package>>246 Data processing247 Statistical analysis248 }249 }250 namespace views {251 class Views {252 <<package>>253 Visualization components254 Chart generation255 }256 }257 namespace pipeline {258 class Pipeline {259 <<package>>260 Component assembly261 UI orchestration262 }263 }264 265 Controllers --> Core : depends on266 Controllers --> Handlers : coordinates267 Handlers --> Transforms : uses268 Handlers --> Views : updates269 Pipeline --> Controllers : assembles270 Pipeline --> Handlers : orchestrates271```272 273## System Behavior274 275### Data Processing Flow276 277Shows how user interactions flow through the system to update visualizations:278 279```mermaid280sequenceDiagram281 participant User282 participant UI as Panel UI283 participant XRFController as XRF Controllers284 participant CurveController as Curve Controllers285 participant XRFHandler as XRF Handler286 participant CurveHandler as Curve Handler287 participant XRFTransform as XRF Transforms288 participant CurveTransform as Curve Transforms289 participant Views290 291 Note over User,Views: XRF Data Processing Flow292 User->>UI: Load XRF File293 UI->>XRFController: xrf_file_selected294 XRFController->>XRFHandler: process_xrf_file295 296 XRFHandler->>XRFTransform: apply_statistics297 XRFTransform-->>XRFHandler: mean, pca, kmeans298 299 XRFHandler->>XRFTransform: apply_correlation300 XRFTransform-->>XRFHandler: correlation_matrix301 302 XRFHandler->>Views: update_xrf_views303 Views-->>UI: render_element_data, mean_view, pca_views304 305 Note over User,Views: Curve Data Processing Flow306 User->>UI: Load Curve File (Mass Spec/Gamma Ray/Drilling)307 UI->>CurveController: curve_file_selected308 CurveController->>CurveHandler: process_curve_file309 310 CurveHandler->>CurveTransform: apply_despiking311 CurveTransform-->>CurveHandler: despiked_data312 313 CurveHandler->>CurveTransform: apply_smoothing314 CurveTransform-->>CurveHandler: smoothed_data315 316 CurveHandler->>Views: update_curve_views317 Views-->>UI: render_single_curve_view318 319 Note over User,Views: Combined Visualization320 Views->>Views: merge_all_data_sources321 Views-->>UI: render_combined_curve_view322 UI-->>User: Display Integrated Dashboard323```324 325### Application State Management326 327Core application states and transitions:328 329```mermaid330stateDiagram-v2331 [*] --> Initializing332 Initializing --> Ready: setup_complete333 334 Ready --> XRF_Loading: load_xrf_file335 Ready --> Curve_Loading: load_curve_file336 337 XRF_Loading --> XRF_Processing: xrf_validation_passed338 XRF_Loading --> Error: xrf_validation_failed339 340 Curve_Loading --> Curve_Processing: curve_validation_passed341 Curve_Loading --> Error: curve_validation_failed342 343 XRF_Processing --> XRF_Transforming: xrf_data_loaded344 XRF_Transforming --> XRF_Visualizing: xrf_transforms_applied345 XRF_Visualizing --> Ready: xrf_views_updated346 347 Curve_Processing --> Curve_Transforming: curve_data_loaded348 Curve_Transforming --> Curve_Visualizing: curve_transforms_applied349 Curve_Visualizing --> Ready: curve_views_updated350 351 XRF_Visualizing --> Data_Merging: both_datasets_available352 Curve_Visualizing --> Data_Merging: both_datasets_available353 354 Data_Merging --> Combined_View: merge_complete355 Combined_View --> Ready: combined_visualization_ready356 357 Error --> Ready: error_handled358 Ready --> [*]: application_shutdown359 360 note right of XRF_Transforming361 Apply statistics, PCA,362 correlation analysis363 end note364 365 note right of Curve_Transforming366 Apply despiking,367 smoothing operations368 end note369 370 note right of Data_Merging371 Frame merger combines372 all data sources373 end note374```375 376## Transform System Architecture377 378### Transform Pipeline Design379 380The transform system uses a pluggable architecture for extensible data processing:381 382```mermaid383classDiagram384 class param_Parameterized {385 <<framework>>386 +param: Parameters387 +watch()388 +trigger()389 }390 391 class SignalDespikingOperator {392 <<transform>>393 +signal: Array394 +threshold: Number395 +interp_radius: Number 396 +disabled: Boolean397 +output: Array398 +transform()399 -_despike_signal(): Array400 }401 402 class SignalSmoothingOperator {403 <<transform>>404 +signal: Array405 +window_length: Integer406 +poly_order: Integer407 +disabled: Boolean408 +output: Array409 +transform()410 -_apply_savgol_filter(): Array411 }412 413 class MeanTransform {414 <<transform>>415 +output: DataFrame416 +transform(df: DataFrame): Self417 }418 419 class KMeansTransform {420 <<transform>>421 +n_clusters: Integer422 +output: DataFrame423 +transform(df: DataFrame): Self424 }425 426 class PCATransform {427 <<transform>>428 +n_components: Integer429 +output: DataFrame430 +transform(df: DataFrame): Self431 }432 433 class KernelDensityTransform {434 <<transform>>435 +bandwidth: Number436 +output: DataFrame437 +transform(df: DataFrame): Self438 }439 440 param_Parameterized <|-- SignalDespikingOperator : extends441 param_Parameterized <|-- SignalSmoothingOperator : extends442 param_Parameterized <|-- MeanTransform : extends443 param_Parameterized <|-- KMeansTransform : extends444 param_Parameterized <|-- PCATransform : extends445 param_Parameterized <|-- KernelDensityTransform : extends446```447 448### Data Flow Through Transforms449 450```mermaid451flowchart TB452 subgraph XRF ["🔬 XRF Data Processing Pipeline"]453 A1["📊 Raw XRF Data"] --> B1["🔍 XRF File Loading"]454 B1 --> C1["⚙️ XRF Processing"]455 456 C1 --> D1["📊 MeanTransform"]457 C1 --> E1["🎯 PCATransform"] 458 C1 --> F1["🔄 KMeansTransform"]459 C1 --> G1["📈 KernelDensityTransform"]460 461 D1 --> H1["📊 Statistical Analysis"]462 E1 --> H1463 F1 --> H1464 G1 --> H1465 466 H1 --> I1["📊 XRF Views<br/>(Element Data, Mean, Correlation, PCA)"]467 end468 469 subgraph CURVE ["📈 Curve Data Processing Pipeline"]470 A2["📈 Raw Curve Data<br/>(Mass Spec, Gamma Ray, Drilling)"] --> B2["🔍 Curve File Loading"]471 B2 --> C2["⚙️ Curve Processing"]472 473 C2 --> D2["🔧 SignalDespikingOperator"]474 C2 --> E2["📈 SignalSmoothingOperator"]475 476 D2 --> F2["🚫 Spike Removal"]477 E2 --> G2["📈 Savgol Smoothing"]478 479 F2 --> H2["📊 Processed Curve Data"]480 G2 --> H2481 482 H2 --> I2["📈 Single Curve Views"]483 end484 485 subgraph MERGE ["🔄 Data Integration"]486 I1 --> J["🔄 Frame Merger"]487 I2 --> J488 J --> K["💾 CDS Manager"]489 K --> L["📊 Combined Curve View"]490 end491 492 L --> M["🖥️ Panel UI Dashboard"]493 494 style XRF fill:#e3f2fd495 style CURVE fill:#e8f5e8496 style MERGE fill:#fce4ec497 style M fill:#f3e5f5498```499 500 501## Factory & Builder System Architecture502 503### Component Factory Pattern504 505The application uses a comprehensive factory system to manage complex component creation and dependency injection:506 507```mermaid508classDiagram509 class ComponentsFactory {510 <<main-factory>>511 +create_all(app: ApplicationHandlerProtocol): AllComponents512 -Phase1: File setup513 -Phase2: Controllers 514 -Phase3: Slice components515 -Phase4: Curve components516 -Phase5: CDS components517 -Phase6: Views518 -Phase7: Helpers519 }520 521 class FileInputFactory {522 <<factory>>523 +create_all(app): Dict[DatasetType, FileInput]524 }525 526 class FileHandlerFactory {527 <<factory>>528 +create_all(app, file_inputs): FileHandlers529 }530 531 class DatasetComponentsFactory {532 <<factory>>533 +create(app, dataset_type, file_handler): DatasetComponents534 }535 536 class XRFControllersFactory {537 <<factory>>538 +create(app, xrf_file_handler): XRFControllers539 }540 541 class CDSComponentsFactory {542 <<factory>>543 +create(app, xrf_controllers, slice_components): CDSComponents544 }545 546 class ViewComponentsFactory {547 <<factory>>548 +create(app, controllers, cds_components): ViewComponents549 }550 551 class AnnotationComponentsFactory {552 <<factory>>553 +create(app): AnnotationComponents554 }555 556 ComponentsFactory --> FileInputFactory : uses557 ComponentsFactory --> FileHandlerFactory : uses558 ComponentsFactory --> DatasetComponentsFactory : uses559 ComponentsFactory --> XRFControllersFactory : uses560 ComponentsFactory --> CDSComponentsFactory : uses561 ComponentsFactory --> ViewComponentsFactory : uses562 ComponentsFactory --> AnnotationComponentsFactory : uses563```564 565### Pipeline Builder System566 567The application uses specialized builders to construct event processing pipelines:568 569```mermaid570classDiagram571 class AllPipelineBuilder {572 <<orchestrator>>573 +build_all_pipelines(app, components, verbose): void574 }575 576 class FileLoadingPipelineBuilder {577 <<builder>>578 +build_file_load_pipeline(app, components): void579 }580 581 class XRFPipelineBuilder {582 <<builder>>583 +build_xrf_load_pipeline(app, components): void584 +build_xrf_load_handled_pipeline(app, components): void585 +build_xrf_controller_pipelines(app, components): void586 }587 588 class CurvePipelineBuilder {589 <<builder>>590 +build_comb_ctrl_pipeline(app, components): void591 +build_update_viz_columns_pipeline(app, components): void592 +build_curve_dataset_pipelines(app, components, dataset_type): void593 }594 595 class SlicePipelineBuilder {596 <<builder>>597 +build_slice_button_pipeline(app, components): void598 +build_slice_step_pipeline(app, components): void599 +build_slice_fully_completed_pipeline(app, components): void600 }601 602 class AnnotationPipelineBuilder {603 <<builder>>604 +build_annotation_pipelines(app, components): void605 }606 607 class HandlerPipeline {608 <<infrastructure>>609 -event_type: EventType610 -handlers: List[HandlerSequence]611 +add(handler_sequence): HandlerPipeline612 +transform(event_transformer): HandlerPipeline613 +build(): void614 }615 616 class HandlerSequence {617 <<infrastructure>>618 -handlers: List[Handler]619 -event_filter: Optional[Filter]620 +__init__(handlers, event_filter, label): void621 }622 623 AllPipelineBuilder --> FileLoadingPipelineBuilder : coordinates624 AllPipelineBuilder --> XRFPipelineBuilder : coordinates625 AllPipelineBuilder --> CurvePipelineBuilder : coordinates626 AllPipelineBuilder --> SlicePipelineBuilder : coordinates627 AllPipelineBuilder --> AnnotationPipelineBuilder : coordinates628 629 XRFPipelineBuilder --> HandlerPipeline : creates630 CurvePipelineBuilder --> HandlerPipeline : creates631 SlicePipelineBuilder --> HandlerPipeline : creates632 HandlerPipeline --> HandlerSequence : contains633```634 635### UI Builder Pattern636 637Panel UI components are constructed using a builder pattern for flexible layout assembly:638 639```mermaid640classDiagram641 class PanelBuilder~T~ {642 <<abstract>>643 -app: Optional[ApplicationHandler]644 -view_type: Optional[ViewType]645 -component: Optional[T]646 +with_viz_switch(app, view_type): PanelBuilder[T]647 +build(): T648 #_build_component()*: T649 }650 651 class ColumnBuilder {652 <<concrete>>653 -components: List[Any]654 -sizing_mode: Optional[str]655 -aspect_ratio: Optional[float]656 -align: Optional[str]657 +add_builder(builder: PanelBuilder): ColumnBuilder658 +add_component(component): ColumnBuilder659 +sizing_mode(mode: str): ColumnBuilder660 +aspect_ratio(ratio: float): ColumnBuilder661 #_build_component(): pn.Column662 }663 664 class RowBuilder {665 <<concrete>>666 -components: List[Any]667 -sizing_mode: Optional[str]668 +add_builder(builder: PanelBuilder): RowBuilder669 +add_component(component): RowBuilder670 #_build_component(): pn.Row671 }672 673 class TabsBuilder {674 <<concrete>>675 -tabs: List[Tuple[str, Any]]676 +add_tab(name: str, component): TabsBuilder677 #_build_component(): pn.Tabs678 }679 680 PanelBuilder <|-- ColumnBuilder : extends681 PanelBuilder <|-- RowBuilder : extends682 PanelBuilder <|-- TabsBuilder : extends683```684 685### Component Assembly Flow686 687Shows how the factory system creates the complete application in dependency-ordered phases:688 689```mermaid690flowchart TD691 A["🚀 Application Start"] --> B["📁 Phase 1: File Setup"]692 B --> B1["FileInputFactory<br/>Creates file input widgets"]693 B --> B2["FileHandlerFactory<br/>Creates XRF & curve handlers"]694 695 B1 --> C["🎛️ Phase 2: Controllers"]696 B2 --> C697 C --> C1["XRFControllersFactory<br/>Element selectors, PCA, Stats"]698 C --> C2["CombCurveController<br/>Combined visualization control"]699 700 C1 --> D["✂️ Phase 3: Slice Components"]701 C2 --> D702 D --> D1["SliceComponentsFactory<br/>Range tools, slice handlers"]703 704 D1 --> E["📊 Phase 4: Curve Components"]705 E --> E1["DatasetComponentsFactory<br/>Mass spec, Gamma ray, Drilling"]706 707 E1 --> F["💾 Phase 5: CDS Components"]708 F --> F1["CDSComponentsFactory<br/>Column data sources, Frame merger"]709 710 F1 --> G["🎨 Phase 6: Views"]711 G --> G1["ViewComponentsFactory<br/>All visualization views"]712 713 G1 --> H["📝 Phase 7: Annotations"]714 H --> H1["AnnotationComponentsFactory<br/>Point annotation system"]715 716 H1 --> I["⚙️ Phase 8: Pipeline Assembly"]717 I --> I1["AllPipelineBuilder<br/>Event processing pipelines"]718 719 I1 --> J["✅ Application Ready"]720 721 style A fill:#e3f2fd722 style J fill:#e8f5e8723 style B fill:#fff3e0724 style C fill:#fff3e0725 style D fill:#fff3e0726 style E fill:#fff3e0727 style F fill:#fff3e0728 style G fill:#fff3e0729 style H fill:#fff3e0730 style I fill:#fff3e0731```732 733*For detailed API documentation and development guides, see individual module docstrings and the `/docs` directory.* 734 735 736 737