chennab28/model-functional

Universal Hugging Face Model Compatibility & Functional Verifier

A GitHub-ready Python agent and Streamlit dashboard that accepts a Hugging Face model URL or owner/model identifier, checks compatibility first, and only then performs a task-aware functional smoke test.

Version 2 organizes the full verification lifecycle as a guided, multipage product:

1. 1.Executive overview — evidence-backed KPIs, gates, risks and actions
2. 2.Verify model — guided Quick, Standard or Production workflow plus manual controls
3. 3.Evaluate quality — golden-set evaluation and decision scorecard
4. 4.Performance & portability — portability, benchmark and optimization
5. 5.Production readiness — security, vLLM, capacity, B70, diagnosis and CI/CD
6. 6.Runs & reports — persistent evidence, comparisons and included baseline
7. 7.Insights & help — usage analytics, curated AI/ML news and the annotated guide

The guided workflow always runs metadata-only compatibility first, pauses for explicit approval before downloading weights, enforces a configurable maximum model-size guardrail, and never converts missing production evidence into a pass. Status language is consistent across the app: Not started, Running, Passed, Review required, Failed and Blocked.

Reliability and deployment

• —runtime.txt pins Streamlit Cloud to Python 3.12.
• —GitHub Actions tests Python 3.12 and 3.13, compiles the package, runs unit/UI smoke tests, and executes a real metadata-only Qwen preflight.
• —The fresh-session AppTest protects against null-state and optional-dependency regressions.
• —RUN_STORE_PATH controls durable SQLite run history; use a persistent mounted path or replace the adapter with PostgreSQL for multi-replica production.
• —ANALYTICS_DB_PATH separately controls privacy-first site analytics storage.

Executive, analytics and news views

The Executive tab rolls up the other tests without inventing results. Unrun blocking gates remain NOT_CHECKED, and the decision remains HOLD. It shows readiness, quality, security, p95 latency, throughput, risks, gaps and prioritized owners/actions.

The Usage Analytics view stores anonymous Streamlit session counts, reruns and interaction-based duration in first-party SQLite. It does not collect IP addresses, persistent identity cookies or browser fingerprints. The default database under /tmp can be ephemeral and is not shared across app replicas. For durable single-instance storage set:

~~~bash export ANALYTICSDBPATH=/persistent-volume/modelverifieranalytics.db ~~~

The AI ML News tab reads a fixed allowlist of publisher RSS feeds, ranks for recency and source authority, removes duplicate titles, groups the result and caps it at 50 items. Projections are explicitly labeled as signal-based inferences. Allowlisting is a curated quality proxy, not a provable “top 1%.”

The Qualification workbench contains ten additional nested tabs without overcrowding the primary lifecycle navigation:

1. 1.Quality — golden-set exact, contains, regex and JSON-validity gates
2. 2.Scorecard — executive blocking/non-blocking qualification decision
3. 3.History — capture and compare session runs
4. 4.vLLM — securely probe an existing OpenAI-compatible endpoint
5. 5.Reproduce — manifest and downloadable evidence ZIP
6. 6.Diagnose — error classification and actionable remediation
7. 7.Security — license, revision, remote-code and artifact readiness screen
8. 8.Capacity — workload, cost and power planning estimates
9. 9.B70 — Intel Arc Pro B70 qualification checklist
10. 10.CI/CD — downloadable GitHub Actions qualification workflow

The Functional tab also annotates every key step with its Intel Arc Pro B70 difference: artifact handling, XPU runtime, device selection, precision, input parity, synchronization, quality comparison and evidence capture.

Why two phases?

A model should not be downloaded or executed blindly. The compatibility agent first inspects small Hub metadata and config.json, detects blockers, and records a decision. Large weights are loaded only when the model and host appear compatible.

Preflight checks

• —URL/repository identifier validation
• —Public, private, missing or gated access
• —Resolved immutable commit revision
• —Transformers config.json availability and parsing
• —Standard weight file availability and estimated size
• —Pipeline task and safe test-adapter support
• —Remote custom-code detection
• —Available disk, CPU RAM, CUDA and GPU memory
• —Estimated disk and CPU-memory needs
• —Actionable blocker remediation

HF tokens are held in memory and are never written to reports.

Automatically tested tasks

• —Text generation and text-to-text generation
• —Summarization and translation
• —Fill-mask
• —Text and token classification
• —Question answering
• —Feature extraction
• —Image classification
• —Audio classification and automatic speech recognition

Models such as diffusion pipelines, GGUF-only repositories, adapters, multimodal chat systems and custom research architectures can need a model-specific adapter. They are reported as unsupported—not incorrectly classified as broken.

Run locally

~~~bash python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate python -m pip install -r requirements.txt streamlit run app.py ~~~

Paste a model link into the sidebar, run preflight, review every check, and then run the functional test if enabled.

Command line

Preflight without downloading weights:

~~~bash python run_verification.py https://huggingface.co/google/flan-t5-small --preflight-only ~~~

Preflight plus real functional inference:

~~~bash python run_verification.py Qwen/Qwen2.5-0.5B-Instruct ~~~

Pinned revision and private/gated model:

~~~bash python runverification.py owner/model --revision COMMITSHA --token YOURREADTOKEN ~~~

Only use --trust-remote-code after reviewing repository code:

~~~bash python run_verification.py owner/custom-model --trust-remote-code ~~~

Test the package

~~~bash pytest -q python -m compileall -q app.py qwenverifier tests runverification.py ~~~

Important interpretation

• —READY_TO_RUN means the preflight found no known blocker.
• —FUNCTIONAL means the model loaded and returned a non-empty result for a small task-aware input.
• —Neither result proves production readiness.
• —Production qualification also needs representative accuracy, safety, reliability, concurrency, latency, throughput and cost testing.

Portability, benchmarking and optimization

The portability tab builds a backend matrix for CPU, NVIDIA CUDA, Intel XPU, Apple MPS, vLLM and OpenVINO. A detected backend is only readiness; portability is proven only after the same pinned revision and acceptance input run there.

The benchmark separates model-load time from warmed synchronized inference and reports min/mean/p50/p95/p99/max, standard deviation, coefficient of variation, requests/second, approximate tokens/second, error rate, accelerator peak memory when available and per-iteration evidence. TTFT, inter-token latency, power and tokens/joule remain explicitly unavailable in the non-streaming Transformers microbenchmark; use the vLLM endpoint/benchmark path for those serving metrics.

The optimization tab runs a controlled baseline-versus-static-batching experiment with identical model, revision, input and output-token cap. It reports throughput change and preserves an output fingerprint. The fingerprint is an initial guard, not a replacement for a task-specific golden set.

Every phase includes a per-step Intel Arc Pro B70 column. On B70:

• —use the Intel GPU driver and an XPU-enabled upstream PyTorch build;
• —require torch.xpu.is_available();
• —use xpu, not CUDA, as the device;
• —synchronize torch.xpu before and after timed regions;
• —validate BF16/FP16 before adopting lower precision;
• —increase batching within the card's 32 GB VRAM;
• —record driver, runtime, device, dtype, VRAM and quality evidence;
• —use the Intel-XPU vLLM build or container for vLLM serving tests.

Intel Extension for PyTorch is not required by this package; current Intel functionality is expected through upstream PyTorch.

References

• —Hugging Face Hub API
• —Transformers pipeline guide
• —Custom models and remote code

The package includes the real Qwen2.5-0.5B-Instruct evidence from the original verifier as an example baseline.