SaiSyam2000/vedic-judge-comparison

Vedic Judge — Human Evaluation Interface

A Streamlit-based tool for human evaluation of AI-generated Vedic attribute content. Evaluators compare outputs from multiple AI models on the same mantra, scoring them on four Mimamsa-informed metrics. Results are saved to a private cloud dataset automatically.

Quick Start (for evaluators)

1. 1.Open the app link shared with you (Hugging Face Space).
2. 2.The interface loads with pre-configured data — no uploads needed.
3. 3.Follow the steps below.

Step-by-Step Guide

1. Enter Your Name

At the top of the page under Your Scoring Session, type your name in the Who Is Scoring? field. This name is attached to every score you submit.

2. Choose Filters (optional)

Both dropdowns display values in Title Case. Internal filtering is case-insensitive.

3. Start Scoring

Click Start My Scoring Cards. The app generates 10 randomly sampled cards (fixed seed = 42 so every evaluator sees the same set for the same filters).

4. Evaluate Each Card

Each card shows:

5. Score Using the Four Metrics

For each model, assign a score (1-5) on each metric:

Rule of Interpretation (Higher is better)

Is the passage correctly identified as a command or an explanatory statement, and are Mimamsa rules applied accordingly?

Three Perspectives Are Truly Different (Higher is better)

Are the three perspectives genuinely distinct, or is the same idea repeated with different labels?

Completeness of Meaning (Higher is better)

Does each perspective fully convey what the passage means from its own angle, or does something important go unsaid?

Hallucination (Lower is better)

Does the answer introduce unsupported or fabricated claims not grounded in the provided content?

6. Save and Navigate

Your scores are preserved in memory when navigating back and forth.

7. Completion

After all 10 cards are scored:

• —Scores are automatically uploaded to a private Hugging Face Dataset repository.
• —A Download my scores (Excel) button lets you save a local copy.
• —Click Start over to begin a new session.

Understanding the Interface Layout

+-------------------------------------------------------+
|  Your Scoring Session                                  |
|  [Name]   [Language filter]   [Attribute filter]       |
|  [Start My Scoring Cards]                              |
+-------------------------------------------------------+
|  (o) Human Judging    ( ) Scoring Reference            |
+-------------------------------------------------------+
|  Your Progress                                         |
|  Card: 1/10 | Mantra: 6.1.3.6 | Scripture: Aitareya   |
|              | Topic: Bhavartha | Language: English     |
+-------------------------------------------------------+
|  Rate each AI answer (metric summaries + ref links)    |
+-------------------------------------------------------+
|  Original Vedamantra (Read First)                      |
|  [source mantra text]                                  |
+-------------------------------------------------------+
|  Model Outputs (Aligned by Plane)                      |
|  Adhibhautika:  [Model-1 text]  [Model-2 text]        |
|  Adhidaivika:   [Model-1 text]  [Model-2 text]        |
|  Adhyatmika:    [Model-1 text]  [Model-2 text]        |
+-------------------------------------------------------+
|  Scoring Inputs                                        |
|  Model-1:  [Rule] [Perspectives] [Completeness] [Hall]|
|  Model-2:  [Rule] [Perspectives] [Completeness] [Hall]|
+-------------------------------------------------------+
|  [Save & Next]  [Skip]  [<- Go Back]                  |
+-------------------------------------------------------+

Key Design Decisions

Bias Prevention

Model names are hidden from evaluators. The UI shows only Model-1, Model-2, etc. The mapping is deterministic (alphabetical sort of real model names) and consistent across all evaluators. The saved Excel file contains both model (real name) and model_alias (what the evaluator saw) for audit purposes.

Scripture Identification

The first number in the Mantra reference (before the first dot) maps to a scripture name. For example, 6.1.3.6 starts with 6, which maps to Aitareya Brahmana.

Three Planes

Each model output contains three interpretive dimensions:

Score Persistence

Scores entered on a card are preserved in session memory. Navigating back to a previously scored card restores your earlier values.

Interpreting the Output Excel File

After completion, the downloaded/uploaded Excel file contains one row per (model, mantra, attribute, language) combination with these columns:

Scoring Reference Tab

Switch to the Scoring Reference tab (radio button at top) at any time to review full metric definitions. You can also click the Open reference buttons next to each metric in the judging view to jump directly to the relevant definition.

For Administrators

Data Files

The app loads Excel workbooks from output/ and input_data/ at runtime. These folders are not tracked in git (see .gitignore) so code pushes stay small and conflict-free.

Place or upload the required files on the Space via the Files tab (or copy them locally for development):

After uploading data on Hugging Face, do not run git add output/ or git add input_data/ from your machine — push code only (compare_scores_app.py, requirements.txt, etc.).

Cloud Save Configuration

Set these as Secrets in the Hugging Face Space settings:

• —HF_EVAL_TOKEN — a Hugging Face write-access token
• —HF_EVAL_REPO — dataset repo ID (e.g. SaiSyam2000/vedic-judge-evaluations)

Completed evaluations are automatically uploaded to Evaluations/ in the specified private dataset repository.

Theme

The app is locked to a light theme via .streamlit/config.toml for maximum text readability across all devices and browsers.