firobeid/SP_DOW_NASDAQ_stocks__News_Headlines_labeled

Quantitative Textual Analysis: Classifier Selection & Routing Logic

Subject: Algorithmic Selection of NLP Models for Financial Signal Generation Methodology: Lopez de Prado’s Framework for False Discovery Control Metric Focus: Precision (Minimization of Type I Errors)

1. Executive Summary

This report evaluates the predictive utility of various NLP architectures for generating "Buy/No-Buy" signals. In accordance with quantitative finance principles, we prioritize Precision in the Inference Partition (Out-of-Sample) to minimize the cost of False Positives (entering losing trades).

Key Findings:

1. 1.The Generalist Champion: The `FinBERT_SIF_Retrival_System` achieves the highest global precision (62.41%) and demonstrates superior stability across the widest range of topics.
2. 2.The "Lookup" Benchmark: The `FinBERT_SIF_SVC_KNN` functions as a valid, high-performance benchmark. While its training score reflects the expected behavior of a Nearest Neighbor "lookup table" (memorization), its Inference performance (60.92%) is robust. However, it is consistently outperformed by specialized architectures in every specific sector.
3. 3.The Heterogeneity of Alpha: Financial news is not linguistically uniform. The inclusion of the Entertainment sector confirms that some topics are Lexical (keyword-driven), while others are Semantic (context-driven) or Structured (data-driven).
4. 4.Strategic Implication: No single model is optimal. A Meta-Labeling Routing System is required to direct headlines to the "Specialist" model best suited for that specific topic.

2. Global Performance Hierarchy (Inference Partition)

The following table ranks the models based on their ability to discriminate signal from noise in the Inference partition.

3. Sector-Specific "Tournaments" (Cluster Analysis)

Using a "Tournament" approach, we identified the dominant model for each sector. We observe three distinct linguistic clusters.

Cluster A: The "Lexical" Sectors (Entertainment & Earnings)

Characteristics: Binary, keyword-driven news (e.g., "Beat/Miss", "Blockbuster/Flop"). Deep context models often introduce noise by over-analyzing simple syntax.

• —Topic: Entertainment
• —Winner: `NonContextDependent_TFiDF`
• —Precision: 60.11%
• —Vs Benchmark (KNN): +2.68% (KNN: 57.43%)
• —Analysis: Entertainment news relies on "buzzwords." TFIDF captures this efficiently.
• —Topic: Earnings_Ratings
• —Winner: `NonContextDependent_TFiDF`
• —Precision: 65.54%
• —Vs Benchmark (KNN): +3.87% (KNN: 61.67%)
• —Analysis: Earnings are the most keyword-dense sector. TFIDF is the undisputed leader here.

Cluster B: The "Structured" Sectors (Banking & Manufacturing)

Characteristics: Quasi-tabular data (deal values, contract specs, unit orders). Models with tabular reasoning capabilities excel.

• —Topic: Investment_Banking
• —Winner: `TabICL_Classifier` (Tabular In-Context Learning)
• —Precision: 65.14%
• —Vs Benchmark (KNN): +5.62% (KNN: 59.52%)
• —Analysis: TabICL parses structured deal data (M&A values) significantly better than semantic embeddings.
• —Topic: Mechanical_Transportation
• —Winner: `TabICL_Classifier`
• —Precision: 67.54%
• —Vs Benchmark (KNN): +4.48% (KNN: 63.06%)
• —Analysis: A massive victory for TabICL. This sector is heavy on technical specifications, which TabICL handles superiorly.

Cluster C: The "Semantic" Sectors (Funds, Tech, Pharma)

Characteristics: Narrative-driven. "Sentiment" is hidden in the nuance of the language ("outlook," "headwinds"), requiring deep embedding vectors.

• —Topic: Financial_Funds
• —Winner: `FinBERT_SIF_Retrival_System`
• —Precision: 68.06%
• —Vs Benchmark (KNN): +2.39% (KNN: 65.67%)
• —Analysis: The highest alpha in the dataset. Fund news requires understanding "manager sentiment," which SIF embeddings capture perfectly.
• —Topic: Technology
• —Winner: `FinBERT_SIF_Retrival_System`
• —Precision: 64.22%
• —Vs Benchmark (KNN): +3.11% (KNN: 61.11%)
• —Analysis: Tech news is volatile. The Retrieval system stabilizes this volatility better than the KNN lookup.
• —Topic: Pharmaceutical
• —Winner: `FinBERT_SIF_Retrival_System`
• —Precision: 57.26%
• —Vs Benchmark (KNN): +0.85% (KNN: 56.41%)
• —Analysis: A tight race, but Retrieval edges out the benchmark.

4. Model Stability & Selection (Lopez de Prado Framework)

We apply strict criteria to reject models that show "False Discovery" characteristics.

1. 1.`FinBERT_SIF_SVC_KNN` (The Valid Benchmark):
2. 2.Status: Accepted as Benchmark.
3. 3.Analysis: The Training Precision of 99.98% is not "overfitting" in the traditional sense; it is the mathematical property of a $k=1$ Neighbor algorithm (perfect memory). Its Inference score (60.92%) is valid and stable. However, it is rejected as the Primary Model simply because it loses every specific sector tournament to a specialist.
4. 4.`Gemma-2b_Decoder` (The Beta Trap):
5. 5.Status: Rejected.
6. 6.Analysis: While Precision is acceptable (61.6%), the False Positive Rate is 0.987. It predicts "Buy" on 98.7% of negative news. It has zero discriminative power and maximizes risk.
7. 7.`L6_LM_LSTM` (The Liquidity Trap):
8. 8.Status: Rejected.
9. 9.Analysis: High Precision in Banking (90%) is negated by a Recall < 6%. It fails to generate enough signals to be economically viable.

5. Implementation: The Routing Logic Flow

To maximize the Deflated Sharpe Ratio, we implement a Meta-Model Router. This logic directs the incoming JSON payload to the specific classifier that has statistically proven dominance in that sector.

Logic Flow Diagram:

graph TD
    A[Incoming News Headline] --> B{Check 'Topic' Tag}
    
    %% Cluster A: Lexical Models
    B -- Entertainment --> C[Route to: TFIDF Classifier]
    B -- Earnings_Ratings --> C
    
    %% Cluster B: Tabular Models
    B -- Investment_Banking --> D[Route to: TabICL Classifier]
    B -- Mechanical_Transportation --> D
    
    %% Cluster C: Semantic Models (Default)
    B -- Financial_Funds --> E[Route to: FinBERT SIF Retrieval]
    B -- Technology --> E
    B -- Pharmaceutical --> E
    B -- OVERALL/Unknown --> E
    
    %% Execution
    C --> F{Prediction > Threshold?}
    D --> F
    E --> F
    
    F -- Yes --> G[Generate BUY Signal]
    F -- No --> H[Generate NO_BUY Signal]

Python Implementation Logic

def get_trading_signal(headline, topic):
    """
    Routes the headline to the specialist model based on Lopez de Prado's 
    Cluster Analysis of the Inference Partition.
    """
    
    # Cluster A: Lexical/Keyword Heavy (Use TFIDF)
    # Precision: ~60-65%
    if topic in ['Entertainment', 'Earnings_Ratings']:
        model = load_model('L7_8_NonContextDependent_TFiDF_Classifier')
        return model.predict(headline)

    # Cluster B: Structured/Tabular Data (Use TabICL)
    # Precision: ~65-67%
    elif topic in ['Investment_Banking', 'Mechanical_Transportation']:
        model = load_model('L7_8_TabICL_Classifier')
        return model.predict(headline)

    # Cluster C: Semantic/Narrative (Use FinBERT SIF Retrieval)
    # Precision: ~57-68%
    # Also acts as the 'Generalist' fallback for unknown topics
    else: 
        # Covers: Financial_Funds, Technology, Pharmaceutical, OVERALL
        model = load_model('L7_8_FinBERT_SIF_Retrival_System_Classifier')
        return model.predict(headline)