eswar0474/FL_IDS

FL Enterprise - Federated Learning Intrusion Detection System

🚀 Enterprise-Grade FL-IDS Platform v3.1.0

AgisFL Enterprise is a cutting-edge Federated Learning Intrusion Detection System designed for enterprise-level cybersecurity operations. This platform combines the power of distributed machine learning with advanced security monitoring to provide robust, privacy-preserving threat detection across distributed networks.

✨ Key Features

🔒 Federated Learning Core

• —Distributed Model Training: Train ML models across multiple clients without sharing raw data
• —Privacy-Preserving: Differential privacy and secure aggregation protocols
• —Multi-Algorithm Support: FedAvg, FedProx, FedNova, and custom algorithms
• —Client Management: Geographic distribution, role-based access, and performance monitoring

🛡️ Advanced Security Engine

• —Real-time Threat Detection: Network packet analysis, behavioral monitoring, and anomaly detection
• —Multi-Source Intelligence: Integration with CrowdStrike, FireEye, and Recorded Future
• —Automated Response: Intelligent threat response and incident management
• —Compliance Ready: GDPR, HIPAA, SOX, and PCI-DSS compliance features

🧪 Research & Development Lab

• —Advanced FL Research: Cutting-edge federated learning algorithms and methodologies
• —Experimental Framework: A/B testing, model versioning, and rollback capabilities
• —Academic Integration: Publication tracking, patent management, and research collaboration
• —Multi-Modal Learning: Combining network traffic, logs, and behavioral data

📊 Enterprise Dashboard

• —Real-time Monitoring: System health, performance metrics, and security status
• —Advanced Analytics: ML predictions, anomaly detection, and performance trends
• —Client Insights: Geographic distribution, training progress, and model performance
• —Compliance Reporting: Automated compliance checks and audit trails

🖥️ Desktop Application

• —Pure Desktop Experience: Native Electron-based desktop application
• —Cross-Platform Support: Windows, macOS, and Linux compatibility
• —Offline Capability: Works without internet connection
• —Native Performance: Optimized for desktop usage

🆕 Latest Updates (v3.1.0)

🐛 Bug Fixes & Improvements

• —Fixed 100+ Critical Bugs: Resolved import errors, missing dependencies, and API inconsistencies
• —Enhanced Error Handling: Comprehensive error handling with fallback data
• —Improved Type Safety: Fixed TypeScript compilation issues and type mismatches
• —API Consistency: Standardized response formats across all endpoints

🚀 New Features

• —Real-time Data Integration: Live system metrics, network monitoring, and security alerts
• —Enhanced Dashboard: Comprehensive overview with real-time updates
• —Dataset Management: Upload, manage, and analyze datasets for FL training
• —Network Monitoring: Real-time packet analysis and threat detection
• —Security Center: Advanced threat management and response system
• —System Metrics: Detailed performance monitoring and optimization
• —Settings Management: Configurable application preferences

🎨 UI/UX Improvements

• —Dark Mode Support: Modern dark theme with light mode toggle
• —Responsive Design: Mobile-friendly interface with adaptive layouts
• —Real-time Updates: Live data refresh and status indicators
• —Professional Interface: Enterprise-grade design for production use

🔧 Technical Enhancements

• —Electron Integration: Full desktop application support
• —Real-time Monitoring: Integration with existing monitoring services
• —College Project Mode: Special environment for academic presentations
• —Performance Optimization: Improved response times and resource usage

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Frontend      │    │   Backend       │    │   FL Clients    │
│   (React +      │◄──►│   (FastAPI)     │◄──►│   (Distributed) │
│    Electron)    │    │                 │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   WebSocket     │    │   FL Engine     │    │   Local Models  │
│   Real-time     │    │   Core          │    │   Training      │
└─────────────────┘    └─────────────────┘    └─────────────────┘

🚀 Quick Start

Prerequisites

• —Python 3.10+
• —Node.js 16+
• —PostgreSQL (optional, SQLite for development)

Option 1: Universal Startup Script (Recommended)

# Run the universal startup script
START_AGISFL.bat

# Choose from available modes:
# 1. Quick Start (Core features, fastest startup)
# 2. Production Mode (Full enterprise features)
# 3. Development Mode (Hot reload enabled)
# 4. Test Mode (Run comprehensive tests)
# 5. Desktop Application (Pure desktop app with Electron)

Option 2: Manual Setup

Backend Setup

cd backend
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
python main.py

Frontend Setup

cd frontend
npm install
npm run build
npm start

Desktop Application

cd frontend
npm run electron:dev      # Development mode
npm run electron:build    # Build for distribution
npm run electron:pack     # Package for distribution

Production Deployment

# Build frontend
cd frontend && npm run build

# Build desktop application
npm run electron:build

# Start backend with production settings
cd backend
export ENVIRONMENT=production
export JWT_SECRET=your-secure-secret
python main.py

📡 API Endpoints

Core Endpoints

• —GET /api/dashboard - Comprehensive dashboard data with real-time metrics
• —GET /api/health - Health check and system status
• —GET /api/system/metrics - System performance metrics
• —GET /api/threats - Security threats and alerts
• —GET /api/security/metrics - Security performance metrics
• —GET /api/security/threats - Detailed threat information

FL-IDS Engine

• —GET /api/fl/strategies - Available FL strategies
• —GET /api/experiments - ML experiments and results
• —GET /api/research/enterprise/research-algorithms - Research algorithms
• —GET /api/fl/status - Federated Learning subsystem mode & metrics (simulation/full, rounds)

Network & Security

• —GET /api/network/stats - Network statistics and performance
• —GET /api/network/packets - Network packet analysis
• —GET /api/integrations/overview - System integrations status

Dataset Management

• —GET /api/datasets - Available datasets for training
• —POST /api/datasets/upload - Upload new datasets
• —DELETE /api/datasets/{id} - Remove datasets

📂 Datasets Overview

The datasets/ directory contains curated sample datasets to demonstrate federated intrusion detection workflows. For full details (schema guidance, quality scoring, privacy), see datasets/README.md.

Guidelines when adding new data:

1. 1.No raw PII (anonymize or aggregate first)
2. 2.Provide a concise schema & feature description
3. 3.Update (or create) datasets_metadata.json so the platform can index it
4. 4.Prefer columnar formats (Parquet) for large internal benchmarking to reduce load latency

Planned metrics (future enhancement):

• —agisfl_datasets_index_total – Index passes executed
• —agisfl_datasets_detected_total – Datasets discovered during scan
• —agisfl_dataset_load_seconds – Histogram of dataset load durations

Example minimal metadata entry (JSON):

{
    "name": "network_traffic_dataset",
    "file": "network_traffic_dataset.csv",
    "samples": 50000,
    "features": 25,
    "task": "intrusion_detection",
    "updated": "2025-08-15"
}

Settings & Configuration

• —GET /api/settings - Application settings
• —POST /api/settings - Update application settings

🧪 Testing

Run All Tests

# Comprehensive testing
python -m pytest tests/ -v

# Specific test categories
python -m pytest tests/test_app.py -v
python -m pytest tests/test_comprehensive.py -v
python -m pytest tests/test_production_ready.py -v

Test Coverage

• —Backend API: 100% endpoint coverage
• —Frontend Components: All pages and components tested
• —Integration: End-to-end functionality verification
• —Performance: Response time and resource usage validation

🔧 Configuration

Environment Variables

# College Project Mode
COLLEGE_PROJECT=true

# JWT Security
JWT_SECRET=your-secure-secret-key

# Environment
ENVIRONMENT=production|development|testing

College Project Mode

When COLLEGE_PROJECT=true, the system:

• —Enables enhanced real-time monitoring
• —Provides detailed FL training progress
• —Shows comprehensive security metrics
• —Optimizes for presentation and demonstration

🧾 Structured Logging

The backend emits newline-delimited JSON (NDJSON) log entries for key events (rate limiting, auth failures, dashboard generation, WebSocket lifecycle).

Sample log line:

{"ts":"2025-08-15T12:00:00.000000Z","level":"info","event":"dashboard_generated","version":"3.1.0","cache_ttl":1.0,"size":2489,"request_id":"6f12e52d"}

Fields:

• —ts: UTC timestamp
• —level: info|warning|error|debug
• —event: Event identifier (e.g. dashboardgenerated, wsconnected)
• —version: Application version
• —request_id: Correlates HTTP request lifecycle (added by middleware)
• —Additional dynamic fields (cachettl, size, clientip, etc.)

Enable pretty console output (optional):

SET PRETTY_LOGS=true   # Windows PowerShell: $Env:PRETTY_LOGS="true"

Forward logs to a file:

python main.py > logs/app.ndjson 2>&1

Integrating with tools:

• —Elastic / OpenSearch: Use filebeat to harvest NDJSON
• —Loki: promtail scrapeconfig with pipelinestages: json
• —jq filtering: jq 'select(.event=="ws_broadcast_failure")' logs/app.ndjson

Key events currently instrumented:

• —dashboardcachehit, dashboard_generated
• —ratelimitblocked, unauthorized_access (middleware)
• —wsconnected, wsdisconnected, wsbroadcastfailure, wsbroadcastsuppressed

Additional events:

• —ratelimitexceeded (per-request limiter)
• —ratelimitereviction (global limiter key eviction)
• —wsbackpressuredisconnect (WS dropped due to queued backlog)
• —wsinactivedisconnect (WS closed after inactivity timeout)
• —psutilsampletimeout (system metrics sampling exceeded 1s budget)

📈 Metrics (/metrics Endpoint)

Prometheus-style plaintext exposed at GET /metrics (guarded by ENABLEPROMMETRICS env flag). Each metric includes HELP/TYPE lines for automatic scraping.

Exported metrics:

• —agisflrequeststotal (counter) – Total HTTP requests processed
• —agisflrequestsinflight (gauge) – Current in‑flight requests
• —agisflratelimited_total (counter) – Requests rejected by rate limiter
• —agisflwsconnections_total (counter) – Cumulative accepted WebSocket connections
• —agisflwsactive (gauge) – Active WebSocket connections right now
• —agisfldashboardcache_hits (counter) – Cache hits for /api/dashboard 1‑second TTL cache
• —agisflflrounds_total (counter) – Federated learning rounds completed
• —agisflflsimulation_mode (gauge) – 1 when FL runs in simulation (no TF / forced), 0 in full mode

Example output:

# HELP agisfl_requests_total Total HTTP requests processed
# TYPE agisfl_requests_total counter
agisfl_requests_total 42
# HELP agisfl_requests_inflight Current in-flight HTTP requests
# TYPE agisfl_requests_inflight gauge
agisfl_requests_inflight 0
# HELP agisfl_rate_limited_total Requests rejected due to rate limiting
# TYPE agisfl_rate_limited_total counter
agisfl_rate_limited_total 3
# HELP agisfl_ws_connections_total Total WebSocket connections accepted
# TYPE agisfl_ws_connections_total counter
agisfl_ws_connections_total 5
# HELP agisfl_ws_active Active WebSocket connections
# TYPE agisfl_ws_active gauge
agisfl_ws_active 2
# HELP agisfl_dashboard_cache_hits Number of /api/dashboard cache hits
# TYPE agisfl_dashboard_cache_hits counter
agisfl_dashboard_cache_hits 18
# HELP agisfl_fl_rounds_total Total federated learning rounds completed
# TYPE agisfl_fl_rounds_total counter
agisfl_fl_rounds_total 4
# HELP agisfl_fl_simulation_mode 1 if FL running in simulation mode else 0
# TYPE agisfl_fl_simulation_mode gauge
agisfl_fl_simulation_mode 1

Scraping configuration snippet (Prometheus):

scrape_configs:
    - job_name: agisfl
        static_configs:
            - targets: ['localhost:8000']
        metrics_path: /metrics
        scheme: http

Disable metrics endpoint:

SET ENABLE_PROM_METRICS=false   # PowerShell: $Env:ENABLE_PROM_METRICS="false"

⚙️ Configuration (Pydantic)

Runtime config now powered by Pydantic BaseSettings (backend/config/app_config.py). Environment variables map automatically:

Backward compatibility: legacy inline dataclass removed in favor of config.app_config singleton.

Planned (extend as needed): datasetupload, modeltrainstart, modeltrain_complete Additional FL events/metrics added:

• —/api/fl/status endpoint for real-time FL mode & round metrics
• —agisflflrounds_total counter
• —agisflflsimulation_mode gauge

🌐 Network Capture Dependencies

Some advanced network monitoring features require packet capture libraries.

Windows:

1. 1.Install Npcap (https://nmap.org/npcap/) with support for WinPcap API compatibility
2. 2.Ensure "Install Npcap in WinPcap API-compatible Mode" is checked
3. 3.Restart the backend after installation

Linux (Debian/Ubuntu):

sudo apt update
sudo apt install -y libpcap-dev tcpdump

macOS:

brew install libpcap

Python packages (only if packet capture modules are enabled):

pip install scapy

Troubleshooting:

• —Permission denied capturing packets: run with elevated privileges or grant CAPNETRAW
• —No interfaces found: verify driver (Npcap/libpcap) installed
• —High CPU: reduce capture filter breadth or sampling frequency

📱 Desktop Application Features

Cross-Platform Support

• —Windows: Native Windows application with installer
• —macOS: macOS app bundle with proper signing
• —Linux: AppImage and package formats

Desktop-Specific Features

• —Native Menus: File, Edit, View, Window, Help menus
• —System Integration: Proper window management and notifications
• —Offline Operation: Works without internet connection
• —Performance: Optimized for desktop hardware

🚀 Deployment Options

1. Web Application

• —Traditional web-based interface
• —Accessible from any browser
• —Real-time updates via WebSocket
• —Mobile-responsive design

2. Desktop Application

• —Native desktop experience
• —Offline capability
• —System integration
• —Professional presentation

3. Hybrid Mode

• —Run both web and desktop simultaneously
• —Shared backend services
• —Consistent data across platforms

🔒 Security Features

Authentication & Authorization

• —JWT-based authentication
• —Role-based access control (RBAC)
• —Session management
• —Secure password policies

Data Protection

• —End-to-end encryption
• —Privacy-preserving FL algorithms
• —Secure data transmission
• —Compliance with security standards

📊 Performance Metrics

System Requirements

• —Minimum: 4GB RAM, 2 CPU cores, 10GB storage
• —Recommended: 8GB RAM, 4 CPU cores, 50GB storage
• —Production: 16GB RAM, 8 CPU cores, 100GB storage

Performance Benchmarks

• —API Response: <100ms average
• —Real-time Updates: <1 second latency
• —FL Training: Scalable to 100+ clients
• —Threat Detection: <50ms analysis time

🤝 Contributing

Development Setup

1. 1.Fork the repository
2. 2.Create a feature branch
3. 3.Make your changes
4. 4.Run tests to ensure quality
5. 5.Submit a pull request

Code Standards

• —Follow PEP 8 for Python code
• —Use TypeScript for frontend
• —Maintain test coverage above 90%
• —Document all public APIs

🙏 Acknowledgments

• —Federated Learning Community: For research and algorithms
• —Open Source Contributors: For various libraries and tools
• —Academic Partners: For research collaboration and validation

🛡️Instustion Detection System Using Federated Learning

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference