yyouretoast/freightiq

FreightIQ

Freight carrier query routing engine built with LangGraph. Resolves discrete constraints (states, equipment, safety ratings) via sub-millisecond SQLite queries and qualitative operational capabilities via hybrid FTS5 BM25 + dense vector retrieval with neural Cross-Encoder re-ranking.

   

Live Demo: huggingface.co/spaces/yyouretoast/freightiq Repository: github.com/yyouretoast/freightiq

https://github.com/user-attachments/assets/87267c8e-72b0-4862-9c19-cc56a6c3b4f8

<p align="center"> <em>Demo: Multi-tool routing across SQLite structured queries, ChromaDB semantic search, NMFC freight calculation, and FMCSA safety verification.</em> <br> _{<em>If video does not play inline, <a href="https://github.com/user-attachments/assets/87267c8e-72b0-4862-9c19-cc56a6c3b4f8">click here to watch the direct demo recording</a> or try the <a href="https://huggingface.co/spaces/yyouretoast/freightiq">live interactive demo</a>.</em>} </p>

Table of Contents

• —Overview & Query Routing Rationale
• —Architecture
• —Tools
• —Retrieval Benchmarks
• —Engineering Trade-offs & Limitations
• —Guardrails & Reliability Controls
• —Representative Routing Examples
• —Verification Test Suite
• —Setup & Execution
• —Project Structure
• —Design Documents
• —License

Overview & Query Routing Rationale

Commercial freight inquiries fall into two fundamentally distinct retrieval classes:

• —Deterministic Relational Queries (e.g., "Find flatbed carriers in Ohio with a satisfactory safety rating"): Dense vector search approximates semantic closeness and frequently returns carriers in adjacent states or with missing certifications. These queries are routed to SQLite, where discrete constraints evaluate with 100% precision in under $1\text{ ms}$.

• —Qualitative Domain Queries (e.g., "Carriers specializing in perishable pharmaceutical cold chain with continuous monitoring"): Relational schemas cannot cleanly express nuanced operational capabilities, equipment phrasing, or special certifications. These queries are routed to a two-stage hybrid search pipeline (FTS5 BM25 + dense ChromaDB embeddings fused via RRF and re-ranked with a neural Cross-Encoder).

Orchestration is handled by a stateful LangGraph ReAct workflow supporting multiple LLM backends (Groq qwen/qwen3.8-27b with automatic fallback to qwen/qwen3.6-27b, OpenAI, and local Ollama) that dynamically routes incoming requests across specialized tools.

For detailed design rationale, see ADR-001: SQL vs. Vector Routing.

Architecture

flowchart TD
    subgraph UI ["User Interface"]
        User["User Query"] --> Streamlit["Streamlit (app.py)"]
    end

    subgraph Orchestrator ["LangGraph State Machine"]
        Streamlit --> Agent["Agent Node (agent/nodes.py)
• Groq qwen/qwen3.8-27b (fallback: qwen3.6-27b)
• Turn-scoped loop breaker
• 8-message context sliding window"]
        Agent --> Router{"Tool Call Required?"}
        Router -- "Yes" --> ToolNode["Tool Execution Node"]
        Router -- "No" --> Output["Final Response"]
    end

    subgraph Tools ["Tools (agent/tools.py)"]
        ToolNode --> T1["carrier_sql_query
• Read-only URI (?mode=ro)
• Subquery limit 25
• Automated zero-row relaxation"]
        ToolNode --> T2["carrier_semantic_search
• FTS5 BM25 + ChromaDB
• RRF fusion (k=60)
• Cross-Encoder re-ranker"]
        ToolNode --> T3["check_fmcsa_authority
• Live QCMobile REST API
• Local database fallback"]
        ToolNode --> T4["freight_class_calculator
• Volume & density formula
• NMFC exception table"]
        ToolNode --> T5["web_search
• Tavily API
• DuckDuckGo fallback"]
    end

    subgraph Data ["Storage & Backends"]
        T1 --> DB[("SQLite carriers.db (WAL)")]
        T2 --> FTS[("SQLite FTS5")]
        T2 --> Chroma[("ChromaDB Vector Store")]
        T2 --> CrossEnc["Cross-Encoder"]
        T3 --> SAFER["FMCSA SAFER Portal"]
        T4 --> Tables["NMFC Rules"]
        T5 --> Web["Web APIs"]
    end

    DB --> Agent
    CrossEnc --> Agent
    SAFER --> Agent
    Tables --> Agent
    Web --> Agent

Tools

1. carrier_sql_query

• —Queries data/carriers.db.
• —SQLite connection uses file:DB?mode=ro (read-only enforced at engine level).
• —Rejects statements that do not start with SELECT or WITH.
• —Enforces an upper bound by wrapping queries in SELECT * FROM (...) AS _bounded_carriers LIMIT 25.
• —Zero-row relaxation: If a multi-clause WHERE statement returns 0 rows, the tool drops the last constraint, runs a fallback query (LIMIT 5), and returns alternative candidates with notice.

2. carrier_semantic_search

• —Two-stage hybrid pipeline:
• —Lexical retrieval via SQLite FTS5 inverted index (BM25) with regex tokenization and stop-word filtering.
• —Dense vector retrieval via ChromaDB (all-MiniLM-L6-v2).
• —Candidate fusion via Reciprocal Rank Fusion ($k=60$) over the top 25 results from each source.
• —Neural re-ranking of the top 15 fused candidates via cross-encoder/ms-marco-MiniLM-L-6-v2 (falls back to dense cosine similarity if unavailable).

3. check_fmcsa_authority

• —Queries the FMCSA QCMobile JSON REST API (mobile.fmcsa.dot.gov/qc/services/carriers/) using a USDOT number, parameterized via FMCSA_WEB_KEY.
• —Evaluates legal entity registration, operating authority status (Active vs. Inactive/Revoked), and federal safety ratings (Satisfactory, Conditional, Unsatisfactory).
• —Enforces compliance safety gating: immediately flags carriers with unsatisfactory safety ratings as FAIL — DO NOT DISPATCH and conditional carriers with WARNING — Supervisory Review Required.
• —Local fallback: validates against internal database records if the public API times out, applying identical safety compliance gating. Directs brokers to SAFER for direct BMC-91X insurance filing checks.

4. freight_class_calculator

• —Calculates shipment volume (L * W * H / 1728) and density (Weight / Volume).
• —Maps density to standard NMFC tiers (Class 50 for $\ge 50$ lb/cu ft up to Class 500 for $< 1$ lb/cu ft).
• —Evaluates exception rules (e.g., insulation fixed at Class 150 regardless of density).

5. web_search

• —Retrieves spot market rates, fuel surcharges, and corridor updates.
• —Primary provider: Tavily Search API. Secondary fallback: DuckDuckGo (ddgs).

Retrieval Benchmarks

Evaluated against 500 commercial carrier profiles using 60 test queries in tests/evaluate_retrieval.py:

<p align="center"> <img src="docs/assets/retrieval_benchmark.png" alt="FreightIQ Multi-Strategy Retrieval Benchmark" width="100%"> </p>

Overall Metrics (60 Queries)

<p align="center"> <img src="docs/assets/retrievallatencytradeoff.png" alt="FreightIQ Retrieval Latency vs Accuracy Pareto Trade-Off" width="100%"> </p>

<details> <summary><strong>View Stratified Breakdown by Query Category (Click to expand)</strong></summary> <br>

<p align="center"> <img src="docs/assets/retrievalstratifiedcategories.png" alt="FreightIQ Stratified Retrieval Performance Across Query Categories" width="100%"> </p>

</details>

Key Findings

1. 1.Lexical Retrieval Impact: FTS5 BM25 retrieves exact domain tokens with sub-millisecond latency (0.22ms), outperforming dense vector search on structured constraint terms.
2. 2.Consensus Ranking: RRF ($k=60$) successfully balances lexical keyword recall with dense semantic breadth.
3. 3.Cross-Encoder Re-Ranking Impact: Cross-encoder re-ranking increases Overall Recall@1 from 0.300 to 0.700 (+133.3%) and MRR from 0.429 to 0.764 (+78.1%) across the 60 benchmark queries.
4. 4.Pareto Trade-Off Justification: As shown in Figure 2, deterministic relational queries are resolved in 0.31 ms with 0.967 MRR via SQLite, preventing unnecessary invocation of the ~500 ms neural cross-encoder pipeline.

Engineering Trade-offs & Limitations

• —Cross-Encoder Compute Latency (~500ms): Cross-encoder scoring over the top-15 candidate pool takes ~400–500ms on CPU (compared to 0.3ms for SQLite queries and 0.2ms for FTS5 BM25). For interactive use, this is within normal turn thresholds; batch retrieval workloads would require GPU acceleration or pre-filtering.
• —Multi-Constraint Semantic Falloff (0.550 Recall@1): When queries combine discrete attributes with free-form requirements (e.g., "California flatbed carriers specializing in semiconductors"), unranked dense and lexical search drop to 0.100–0.150 R@1, while the cross-encoder reaches 0.550 R@1 (0.750 Recall@5). This is why discrete constraints are routed to SQLite, reserving semantic search for unstructured descriptions (Figure 3).
• —Single-Vendor Sibling Failover: Intra-provider failover switches between qwen/qwen3.8-27b and qwen/qwen3.6-27b on Groq. While this protects against per-model rate limits and transient 503s with sub-second inference speeds and identical tool-binding semantics, an upstream platform outage or account-level quota exhaustion on Groq affects both siblings simultaneously. Production systems can configure alternative providers (e.g. OpenAI or local Ollama).
• —Single-Turn Single-Tool Principle (`parallel_tool_calls=False`): To prevent redundant API calls and keep token usage within the 950-token budget (Groq OTPM safety ceiling), the model is bound with parallel_tool_calls=False. For multi-part questions requiring multiple tools, the agent addresses the primary intent first and relies on follow-up user turns rather than parallel execution.
• —Synthetic Dataset: 500 fictional carrier profiles are deterministically generated to avoid real-carrier compliance or data-quality misrepresentation while preserving authentic freight domain complexity (TWIC badges, GDP cold chain, Moffett forklifts, RGN lowboys, Carrier Vector chillers).
• —Groq Free-Tier Token Budgets (200k TPD): Free-tier Groq API accounts enforce daily token limits. FreightIQ mitigates this via automatic sibling failover (qwen/qwen3.8-27b $\leftrightarrow$ qwen/qwen3.6-27b), tool output length bounding (8,000 characters), and turn-aligned 8-message context truncation.
• —SQLite Write Serialization: SQLite in WAL mode provides lock-free concurrent reads, but writes are serialized. High-volume multi-user writes in enterprise production would necessitate PostgreSQL.
• —FMCSA Public API Availability & Compliance Gating: The tool queries the FMCSA QCMobile JSON REST service. If external network timeouts occur, it falls back to local database records while strictly enforcing carrier safety ratings (rejecting unsatisfactory carriers). Direct BMC-91X insurance filing checks are redirected to SAFER.

Guardrails & Reliability Controls

Representative Routing Examples

Verification Test Suite

• —✅ System Integration (tests/verify_system.py): 6 / 6 Passed (All 5 domain tools + LangGraph ReAct loop)
• —✅ Agent Trajectory Audit (tests/evaluate_agent_trajectories.py): 20 / 20 Passed (Routing, prompt injections, safety bounds, loop breaker)
• —✅ Retrieval Benchmark (tests/evaluate_retrieval.py): 60 / 60 Evaluated (Empirical ground truth across 5 retrieval strategies)
• —✅ Concurrency Stress Test (tests/stress_test_concurrency.py): 15 / 15 Passed (Zero errors under concurrent SQLite WAL load)

<p align="center"> <img src="docs/assets/agenttrajectorymatrix.png" alt="FreightIQ Agent Routing Trajectory & Guardrail Compliance Matrix" width="100%"> </p>

Setup & Execution

1. Prerequisites

• —Python 3.10+ (tested on Python 3.11)
• —LLM API key (Groq, OpenAI, or local Ollama)

2. Installation

git clone https://github.com/yyouretoast/freightiq.git
cd freightiq

# Using venv
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt

3. Environment Variables

Create a .env file from the template:

cp .env.example .env

4. Database Seeding

python scripts/seed_db.py                    # Seed if not already populated
python scripts/seed_db.py --force            # Force re-seed SQLite database & ChromaDB vector index
python scripts/seed_db.py --regenerate-data  # Regenerate synthetic dataset profiles from scratch

Populates 500 carrier profiles in data/carriers.db (with real-time FTS5 triggers) and builds data/chroma_db.

5. Running the Application

streamlit run app.py

6. Running Tests

# Run integration tests
python -m tests.verify_system

# Run retrieval benchmark
python -m tests.evaluate_retrieval

# Run trajectory & guardrail audit
python -m tests.evaluate_agent_trajectories

# Run concurrency stress test
python -m tests.stress_test_concurrency

7. Observability & Tracing (LangSmith)

FreightIQ has native LangSmith distributed tracing pre-integrated via LangGraph. When LANGCHAIN_TRACING_V2=true and LANGCHAIN_API_KEY are present in your environment, execution traces are automatically streamed to LangSmith:

• —Complete LangGraph ReAct trajectories (agent $\leftrightarrow$ tool state loops).
• —Tool inputs, serialized outputs, and execution latencies.
• —Token counts, prompt formatting, and sibling model failover events.
• —Zero-code activation: runs directly via standard LangChain telemetry handlers.

Project Structure

freightiq/
├── .github/workflows/             # CI/CD & Automated Mirroring
│   ├── verify.yml                 # Test verification suite
│   └── sync_to_hf.yml             # Automated Hugging Face Spaces sync
├── agent/                         # Agent orchestration
│   ├── graph.py                   # LangGraph definition & conditional edges
│   ├── nodes.py                   # Reasoning node, guardrails, model failover
│   ├── state.py                   # AgentState schema
│   └── tools.py                   # 5 domain tools
├── docs/
│   └── adr/                       # Architecture Decision Records (ADR-001 to 004)
├── rag/                           # Data storage & retrieval
│   ├── generate_carriers.py       # 500-profile dataset generator
│   ├── setup_sqlite.py            # SQLite & FTS5 table initialization
│   ├── ingest_chroma.py           # ChromaDB dense vector indexing
│   ├── retriever.py               # Hybrid retriever (FTS5 BM25 + ChromaDB RRF)
│   ├── reranker.py                # Cross-Encoder with cosine fallback
│   └── utils.py                   # Text formatting & sanitization
├── scripts/
│   └── seed_db.py                 # Primary database seeder
├── tests/
│   ├── verify_system.py           # Integration smoke test
│   ├── evaluate_retrieval.py      # 60-query retrieval benchmark
│   ├── evaluate_agent_trajectories.py # 20-case trajectory & guardrail test
│   └── stress_test_concurrency.py # SQLite concurrency test
├── app.py                         # Streamlit UI
├── config.py                      # Global configuration
├── AGENTS.md                      # Operational guidelines for AI coding agents
├── DATA.md                        # Dataset schema & provenance
├── pyproject.toml                 # Package configuration
└── requirements.txt               # Dependencies

Design Documents

• —DATA.md: Relational schema, JSON array types, and data generation details.
• —ADR-001: SQL vs. Vector Routing: Rationale for dual-modality query separation.
• —ADR-002: Neural Cross-Encoder Re-Ranking: Re-ranking candidate pool design and fallbacks.
• —ADR-003: Hybrid FTS5 BM25 + Vector Fusion: Lexical-dense fusion mechanics.
• —ADR-004: Dual Web Search Fallbacks: Multi-tier search engine integration.

License

MIT License. See LICENSE for details.