MuratcanKoylan/Marketing-Memory-Routing-8B

1# Synthetic Data Generation Pipeline2 3This directory contains the tools for generating and validating synthetic training data using Cohere's `command-a-reasoning-08-2025` model.4 5## Setup6 71.  **Install Dependencies**:8    ```bash9    python3 -m venv venv10    source venv/bin/activate11    pip install cohere python-dotenv tinker tinker-cookbook12    ```13 142.  **Environment Variables**:15    Ensure your `.env` file contains your Cohere API key:16    ```17    COHERE_API_KEY=your_api_key_here18    ```19 20## Usage21 22### 1. Generate Data23Use the `SyntheticDataPipeline` class to generate data batches.24 25```python26from synthetic_data.pipeline import SyntheticDataPipeline27 28pipeline = SyntheticDataPipeline()29# Generate 10 examples for a specific category30results = pipeline.run_batch(count=10, category="company.brand_core")31```32 33You can also run the sample generator script:34```bash35python3 synthetic_data/generate_sample.py36```37 38### 2. Validate Data39Run the validation script on any generated JSON or JSONL file to check compliance with the schema and distribution targets.40 41```bash42python3 synthetic_data/validate.py synthetic_data/sample_batch.json43```44 45The validator checks:46*   JSON structure and required fields47*   Category distribution48*   Multi-label frequency49*   Conversation length50*   Persistence and scope consistency51 52## Pipeline Components53 54*   `pipeline.py`: Core logic for 2-stage generation (Scenario -> Conversation) using Cohere.55*   `validate.py`: Quality assurance script implementing checks from `docs/synthetic_data.md`.56*   `test_pipeline.py`: Unit tests for the pipeline structure.57*   `generate_sample.py`: Helper script to produce a quick sample batch.58 59