JiHyuk-Byun/3D-PAQA

3D-PAQA — Preference-Aligned 3D Quality Assessment

Preference-aligned perceptual quality labels for 240,636 Objaverse assets, rated on six perceptual criteria. The goal of 3D-PAQA is to move beyond synthetic-distortion 3D-QA benchmarks and provide human-preference-aligned quality scores for real, human-created 3D assets, at a scale usable for training and benchmarking automatic quality evaluators. Drawn from a 264,966-asset Objaverse corpus.

• —train.csv — 216,540 labeled assets
• —test.csv — 24,096 labeled assets

Companion model: **JiHyuk-Byun/3D-PAQA-evaluator** (PointTransformerV3 trained on these labels) · Training code: **github.com/JiHyuk-Byun/3D-PAQA**

How the labels were made

These are not raw human ratings. Collecting human MOS for hundreds of thousands of assets is not scalable, so the labels are generated by a multimodal large language model (MLLM), Qwen2-VL-72B, guided to align with human preference through a two-stage exemplar-anchored relative-ranking pipeline:

1. 1.Human-labeled exemplar anchors. A pool of 100 reference objects (10 categories × 10 objects) is labeled by humans and grouped into 5 quality levels (20 objects each). For each asset to annotate, one object is sampled from each level, yielding a set of 5 anchors that span the full quality spectrum.
2. 2.Relative Ranking (RR). The target asset (rendered to multi-view images) is presented to the MLLM alongside the 5 anchors, with criterion-specific instructions. Instead of scoring in the abstract, the MLLM ranks the target's position among the anchors per criterion; the score is derived by averaging the nearest anchors. This grounding makes outputs far more stable than direct absolute scoring.
3. 3.Averaging. The process is repeated 5 times with independently sampled anchor sets and averaged, reducing anchor-specific bias.

So human preference enters through (a) the human-labeled exemplar anchors injected into every prompt, and (b) a separate held-out human user study (~12k human responses on an Objaverse subset, all six criteria) that is used only to validate how well these MLLM annotations — and the evaluator trained on them — agree with people. The companion evaluator trained on these labels correlates strongly with that human study and, on the held-out subset, exceeds its teacher MLLM on every criterion.

Scope: 3D-PAQA labels cover human-created Objaverse assets. Applying it to AI-generated / text-to-3D outputs is a natural extension but is not validated by the labels here — the dataset does not (yet) contain generative-model outputs.

Schema

All scores are on roughly a 1–5 scale (higher = better) and are asset-level (they apply to the mesh; the .npy suffix refers to the processed point-cloud form below).

from datasets import load_dataset
ds = load_dataset("JiHyuk-Byun/3D-PAQA")
print(ds["train"][0])
# {'filename': '000-015/3876fb82991240c7bf4856c4bb4185b0.npy',
#  'geometry': 4.0, 'texture': 4.4, 'material': 4.8,
#  'plausibility': 3.2, 'artifacts': 3.6, 'preference': 2.8}

Normalization — labels are raw on purpose

The labels are shipped un-normalized (raw scores). This is intentional: the raw score is the supervision target, and the companion evaluator regresses these raw values. The six criteria are on different effective scales (train split):

If you want to normalize downstream (e.g. to compare criteria on one scale, or to balance a multi-task loss), use `criteria_stats.json` (per-criterion mean/std and a cdf_grid computed on the full train split):

import json
s = json.load(open("criteria_stats.json"))["stats"]
z = (x - s["material"]["mean"]) / s["material"]["std"]   # z-score

Note z-scoring is affine — it rescales but does not change each criterion's distribution shape (some criteria, e.g. artifacts, are skewed/peaked).

Getting the 3D assets (Objaverse provenance)

This repo distributes labels + processing code only. The underlying 3D meshes are from **Objaverse** and remain under their original per-asset licenses, so we do not redistribute them. Reproduce the assets yourself from the UID in filename:

pip install objaverse trimesh numpy
python process_objaverse.py --csv train.csv --out ./pc_npy

process_objaverse.py downloads each asset by its Objaverse UID and runs mesh2pc.py to produce the exact 11-feature point cloud the evaluator consumes: per point [coord(3), color(3), normal(3), metallic(1), roughness(1)]. This sampler is the data contract used at label/train time — keep it byte-identical.

License & attribution

• —Labels & code in this repo: CC-BY-4.0 (this work).
• —Source 3D assets: Objaverse, each under its own license — not included here; download via the script above and respect per-asset terms.

Citation

@misc{byun2026_3dpaqa,
  title  = {3D-PAQA: Towards Preference-Aligned 3D Quality Assessment},
  author = {Byun, JiHyuk},
  year   = {2026},
  note   = {https://huggingface.co/datasets/JiHyuk-Byun/3D-PAQA}
}