Modelpublic

rbhatia46/harrier-270m-lora-attn-downproj-qwen3-oracle

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes8downloads

Model Card

Harrier-270m Fine-tuned — Q2eliter8lr5e6downonly

Fine-tuned from microsoft/harrier-oss-v1-270m using LoRA on MS MARCO hard negatives mined with Qwen3-Reranker-8B as oracle scorer.

Description

Harrier-270m fine-tuned with LoRA on attention+down_proj layers. Hard negatives mined from MS MARCO 8.8M via Qwen3-Reranker-8B oracle scoring. Elite filter: negatives scored 0.01-0.05 (genuine confusion zone). Last-token pooling. 268M params.

Usage

python

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

model = AutoModel.from_pretrained('rbhatia46/harrier-270m-lora-attn-downproj-qwen3-oracle', torch_dtype=torch.bfloat16)
tok   = AutoTokenizer.from_pretrained('rbhatia46/harrier-270m-lora-attn-downproj-qwen3-oracle')

QUERY_INST = 'Given a web search query, retrieve relevant passages that answer the query'

def encode(texts, is_query=False):
    if is_query:
        texts = [f'Instruct: {QUERY_INST}\nQuery: {t}' for t in texts]
    enc = tok(texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
    with torch.no_grad():
        out = model(**enc)
    # Last token pooling
    seq_lens = enc['attention_mask'].sum(dim=1) - 1
    emb = out.last_hidden_state[torch.arange(len(seq_lens)), seq_lens]
    return F.normalize(emb.float(), p=2, dim=-1)

Training

—Base model: Harrier-270m (268M parameters)
—Training data: MS MARCO 8.8M corpus, hard negatives mined via Qwen3-Reranker-8B
—Adaptation: LoRA (r=8, alpha=16) on attention + down_proj layers
—Loss: InfoNCE with temperature=0.02