abdullah693/pakistan-legal-qa-llama70b

Pakistan Legal QA — Llama 3.3 70B LoRA

A LoRA fine-tune of meta-llama/Llama-3.3-70B-Instruct specialising in Pakistani law, trained on a curated dataset of 5,116 question–answer pairs grounded in primary statutory text.

What it does

Given a legal question, the model produces:

• —A short reasoning trace explaining the applicable provision
• —A direct answer in plain English
• —An exact statutory citation (e.g. PPC s.302, CONST art.184(3), QSO art.17)

Coverage

All answers are grounded in verbatim provision text extracted from official Pakistani statutes. A "hard slice" of 110 test questions covers Pakistani-distinctive doctrine (qisas/diyat, Qanun-e-Shahadat evidence rules, Muslim family law) where general-purpose models most commonly confuse Pakistani law with Indian or British law.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    quantization_config=bnb_config,
    device_map="auto"
)
model = PeftModel.from_pretrained(base, "abdullah693/pakistan-legal-qa-llama70b")
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")

question = "Under PPC s.302, what is the punishment for qatl-i-amd?"
inputs = tok.apply_chat_template(
    [{"role": "user", "content": question}],
    return_tensors="pt", return_dict=True, add_generation_prompt=True
).to(model.device)

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Output format

Reasoning: <short chain-of-thought grounded in the statutory text>
Answer: <direct answer>
Citation: <canonical citation e.g. PPC s.302>

Training details

Because the adapter targets all linear layers (including the MLP/FFN blocks where factual associations are stored), it can encode Pakistani legal knowledge rather than just output formatting. Eval loss was still declining at the end of epoch 2.

Dataset

The training data was built from licence-clean Pakistani legal corpora:

• —Constitution of Pakistan 1973 — full text, 305 provisions (LEGAL-UQA seed, MIT licence)
• —Pakistan Penal Code — 498 provisions (CC-BY)
• —Qanun-e-Shahadat Order 1984 — 165 provisions (CC-BY)
• —Muslim Family Laws Ordinance 1961 — 59 provisions (CC-BY)
• —Code of Civil Procedure — 143 provisions (CC-BY)

Each QA pair includes a verbatim provision excerpt as grounding context plus a chain-of-thought reasoning trace generated and verified by Claude.

Dataset: abdullah693/adaption-pakistan-law-qa-pairs

Limitations

• —Not legal advice. This model is for research and informational purposes only.
• —Coverage is limited to the five statutes above; it does not cover tax law, intellectual property, regulatory law, or case law.
• —The model may still hallucinate on questions outside its training distribution.
• —Urdu-language queries are not yet well-supported (English-only training data).

Licence

LoRA weights: CC-BY-4.0. Subject to Meta's Llama 3.3 Community Licence.