syzdekbr/item_review_psychometric_criteria

Item Review Tool

Automated multiple-choice item screening against Haladyna & Rodriguez (2004) item-writing guidelines, powered by Qwen2.5-72B-Instruct via the HuggingFace Inference API.

Usage

1. 1.Enter your HuggingFace token (free account; Inference API access required — enable it in your token settings)
2. 2.Upload an .xlsx file with columns: item_title, stem, option_a, option_b, option_c, option_d, key
3. 3.Optionally set a filter (All / Accept / Revise / Reject) before running
4. 4.Click Run review
5. 5.Download the full report with Export report (.xlsx)

Score interpretation

Haladyna & Rodriguez criteria evaluated

1. 1.Stem presents a single, clear problem
2. 2.Stem is positively worded (avoids NOT/EXCEPT)
3. 3.Stem contains no irrelevant/window-dressing material
4. 4.Stem is not written as fill-in-the-blank
5. 5.Options are homogeneous and parallel in form
6. 6.Distractors are all plausible (no give-aways)
7. 7.No "all of the above" / "none of the above"
8. 8.No grammatical or clang cues to correct answer
9. 9.Options avoid absolute terms (always, never)
10. 10.Correct answer is unambiguously best
11. 11.Cognitive level is application or analysis (not pure recall)
12. 12.Options are not overlapping or redundant

Important caveats

• —The tool flags structural and format flaws reliably. Content accuracy must still be verified by a subject matter expert.
• —AI-flagged items should have their rationale reviewed by a qualified reviewer before final disposition decisions are made.
• —Keep exported reports as part of your item development audit trail.

Local development

pip install -r requirements.txt
python app.py