karthyks/text-embeddings-test

🔍 GitLab Log Embedding Analyzer

A modern, full-stack application for processing, cleaning, and analyzing GitLab CI/CD log files using advanced text embeddings and LLM-based filtering. Built with React TypeScript frontend and FastAPI Python backend.

🏗️ Architecture Overview

This project follows a clean, modular architecture:

• —🐍 Backend (`backend/`) - Python module with FastAPI server
• —⚛️ Frontend (`frontend/`) - React TypeScript application
• —🧪 Tests (`backend/tests/`) - Comprehensive test suite

✨ Key Features

🧹 Intelligent Log Processing

• —Two-Stage Pipeline: Basic cleaning + LLM-based error filtering
• —Smart Chunking: Configurable chunk sizes with overlap
• —GitLab CI/CD Support: Specialized parsing for GitLab logs
• —ANSI Code Removal: Clean text extraction from terminal logs

🧠 Advanced LLM Filtering

• —Local Gemma Models: Privacy-first AI processing (default)
• —Multiple Providers: OpenAI, Anthropic, Ollama support
• —Error Focus: Filters out noise, keeps only critical errors and sections
• —Section-Aware: Only preserves GitLab sections containing actual errors
• —Smart Condensation: Dramatically reduces log volume while preserving critical information

📊 Dual Embedding Models

• —🔥 Gemma Models: Advanced transformer-based embeddings (default)
• —⚡ Sentence Transformers: Fast, efficient embeddings (fallback)
• —🔍 Semantic Search: Natural language queries across processed logs
• —📈 Similarity Scoring: Ranked results with confidence scores
• —🔐 Token Management: Secure HuggingFace authentication via UI

🎨 Modern UI/UX

• —React + TypeScript: Type-safe frontend development
• —Responsive Design: Works on desktop and mobile
• —Real-time Updates: Live progress and status tracking
• —Interactive Results: Expandable chunks with syntax highlighting

🚀 Quick Start

Prerequisites

• —Python 3.8+ with virtual environment
• —Node.js 20.19+ and npm
• —🔐 HuggingFace Account with access token
• —🎯 Gemma Model Access (request at google/gemma-2b)

1. Setup Backend

# Clone and setup Python environment
git clone <repository>
cd text-embeddings-test
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

2. Setup Frontend

cd frontend
npm install
cd ..

3. Start Development Servers

# Option 1: Combined script (recommended)
python start_dev.py

# Option 2: Separate terminals
# Terminal 1: python -m uvicorn backend.api_server:app --reload
# Terminal 2: cd frontend && npm run dev

4. Configure HuggingFace Authentication

1. 1.Get Your Token: Visit HF Settings → Tokens
2. 2.Open the App: Navigate to http://localhost:5173
3. 3.Add Token: Click "Add Token" in the header bar
4. 4.Paste Token: Enter your hf_your_token_here
5. 5.Select Gemma: Choose Gemma models for advanced processing

5. Access Points

• —🎨 React UI: http://localhost:5173
• —🔧 API Backend: http://localhost:8000
• —📚 API Documentation: http://localhost:8000/docs

📖 Usage Workflow

1. 1.🔐 Configure Token - Add HuggingFace token in the header bar
2. 2.📁 Upload Log File - Drag & drop GitLab CI/CD log files
3. 3.⚙️ Configure Processing - Set chunk size, overlap, and LLM filtering options
4. 4.🧹 Process Logs - Two-stage pipeline: ANSI cleaning → LLM error filtering
5. 5.🧠 Generate Embeddings - Choose Gemma or Sentence Transformers
6. 6.🔍 Search & Analyze - Query logs with natural language
7. 7.📊 Visualize Results - Explore patterns and insights

🛠️ Development

Backend Development

# Start API server with auto-reload
uvicorn backend.api_server:app --reload --port 8000

# Run tests
python run_tests.py
# Or: python -m pytest backend/tests/

# View API documentation
open http://localhost:8000/docs

Frontend Development

cd frontend

# Start development server
npm run dev

# Build for production
npm run build

# Run linting
npm run lint

📁 Project Structure

text-embeddings-test/
├── backend/                   # 🐍 Python Backend Module
│   ├── tests/                # 🧪 Test Suite
│   │   ├── README.md         # Testing documentation
│   │   ├── test_embeddings.py
│   │   └── test_llm_integration.py
│   ├── README.md             # Backend documentation
│   ├── api_server.py         # FastAPI REST API
│   ├── log_processor.py      # Basic log processing
│   ├── llm_log_processor.py  # LLM-based filtering
│   └── embedding_generator.py # Text embeddings
├── frontend/                 # ⚛️ React Frontend
│   ├── README.md             # Frontend documentation
│   ├── src/components/       # React components
│   ├── src/services/         # API integration
│   └── src/types/           # TypeScript definitions
├── README.md                 # 📋 This file (project overview)
├── requirements.txt          # 📦 Python dependencies
├── pytest.ini              # 🧪 Test configuration
├── run_tests.py             # 🧪 Test runner
└── start_dev.py             # 🚀 Development server

🏗️ Technical Architecture

This project uses a modern, scalable architecture:

Key Benefits

• —✅ Modern Stack: React + FastAPI for optimal performance
• —✅ Type Safety: Full TypeScript support throughout
• —✅ Scalable: Modular architecture for easy expansion
• —✅ Fast: Client-side rendering with efficient API calls

🤝 Contributing

1. 1.Fork the repository
2. 2.Create a feature branch (git checkout -b feature/amazing-feature)
3. 3.Make changes in the appropriate module (backend/ or frontend/)
4. 4.Add tests for new functionality
5. 5.Update relevant README files
6. 6.Commit changes (git commit -m 'Add amazing feature')
7. 7.Push to branch (git push origin feature/amazing-feature)
8. 8.Open a Pull Request

📄 License

This project is licensed under the MIT License.

🙏 Acknowledgments

• —Hugging Face for Transformers, Gemma models, and Sentence Transformers
• —Google for the powerful Gemma language models
• —FastAPI for the excellent Python web framework
• —React and Vite for modern frontend tooling
• —OpenAI and Anthropic for additional LLM capabilities

Happy Log Analysis! 🚀

For detailed documentation:

• —Backend: See `backend/README.md`
• —Frontend: See `frontend/README.md`
• —Testing: See `backend/tests/README.md`