avihayamor/tripmatch-ai-plan-comparisons

TripMatch AI — Original vs Alternative Plan Comparisons

This dataset is Amit's professor-assigned extension of TripMatch AI. It compares the original daily plan with the richer alternative plan using an LLM judge.

The decision is generated by the LLM as strict JSON. Python is used only for orchestration, persistence, and JSON-schema validation; it does not calculate scores, choose a winner, or write explanations.

The generation jobs use vLLM structured outputs with the published comparison_schema.json, so the JSON format is constrained while each LLM is answering. Invalid judgment content is sent back to the same LLM for a full regeneration. Python may canonicalize only the three copied identity metadata strings (trip_id, judge model, and generator model) from the source row; it never creates or changes a score, winner, reason, confidence, or explanation.

Cross-model judging

• —Alternatives created by Gemma 3 are judged by Qwen 3.
• —Alternatives created by Qwen 3 are judged by Gemma 3.

This prevents a model from judging its own generated plan.

Evaluation criteria

The judge evaluates budget alignment, pace alignment, duration feasibility, companion suitability, practical organization, and destination specificity.

Expected final configuration

The full configuration will contain 10,000 rows. Important fields include daily_plan_original, daily_plan_alternative, created_by_model, judge_model, comparison_json, overall_winner, confidence, and json_valid.

Reproducibility files

• —comparison_schema.json: strict professor-facing JSON contract.
• —jobs/tripmatch_cross_model_judge_vllm.py: GPU LLM judging and checkpoints.
• —jobs/tripmatch_merge_comparisons.py: validation, Dataset Viewer publishing, summary metrics, and charts.

Source dataset: https://huggingface.co/datasets/avihayamor/tripmatch-ai-daily-plan-alternatives

<!-- FINALRESULTSSTART -->

Complete 10,000-row comparison results

Every original trip is compared with its richer alternative. The LLM judge generated the scores, reasons, winner, confidence, and explanation as schema-valid JSON; Python did not make the decision.

Overall LLM decisions: 9,960 alternative, 36 original, and 4 ties.

Dataset Viewer configurations: full, gemma_judge, and qwen_judge.

[image]

[image] <!-- FINALRESULTSEND -->

Download every comparison as JSON

All 10,000 schema-valid LLM judgments are also published as standalone JSON files for direct inspection and submission:

• —`exports/all_comparisons.json`: one JSON array containing all 10,000 comparison objects.
• —`exports/all_comparisons.jsonl`: JSON Lines format with one complete comparison object per line.
• —`exports/manifest.json`: row count, validation rate, file sizes, and SHA-256 checksums.

Every object follows `comparison_schema.json` and contains the trip ID, generator model, judge model, six criterion-level score pairs and explanations, overall winner, confidence, and overall explanation.