pragnyanramtha/avro-fastavro-duplicate-label-dual-view-poc

fastavro Duplicate Avro Label Field Creates Dual-View Training Label

Severity

Medium, 5.8/10.

Rationale: this is artifact-carried ML data/label manipulation through a normal fastavro.reader() embedded-schema load path. It is not arbitrary code execution, and the Avro schema is invalid because it contains duplicate field names, but current fastavro==1.12.1 accepts it and returns a different label than strict-schema or Apache Avro consumers.

Summary

This repository contains a benign Avro object-container file demonstrating a fastavro parser mismatch around duplicate field names.

The embedded Avro schema contains duplicate label fields:

{
  "type": "record",
  "name": "TrainingLabelRow",
  "fields": [
    {"name": "label", "type": "int"},
    {"name": "label", "type": "int"},
    {"name": "text", "type": "string"}
  ]
}

The single record encodes:

first label  = 0
second label = 1
text         = "benign sample"

Observed behavior:

fastavro embedded-schema read:      {"label": 1, "text": "benign sample"}
fastavro strict reader-schema read: {"label": 0, "text": "benign sample"}
Apache Avro Python 1.12.1:          rejects duplicate field name
Apache Avro Java 1.12.1:            rejects duplicate field name

No code execution payload is included. The PoC only demonstrates deterministic label-value manipulation / parser disagreement for an Avro data artifact.

Impact

• —Artifact-carried label flip in Avro ML data, feature-store, evaluation, or model-metadata workflows that trust embedded Avro schemas through fastavro.reader().
• —A fastavro preprocessing or training job can see label=1, while an auditor, validator, or downstream reader using a strict reader schema sees label=0.
• —Apache Avro Python and Apache Avro Java reject the artifact, creating a cross-reader disagreement instead of a clean universal rejection.
• —ModelScan 0.8.8 reports no issues and scans zero files, but Avro is not a supported ModelScan model format, so this is supporting context only and is not the reportable vulnerability.

Affected Versions Tested

• —Python 3.12.3
• —fastavro==1.12.1
• —avro==1.12.1
• —Apache Avro Java 1.12.1
• —ModelScan 0.8.8

Files

duplicate_label_dual_view.avro
  Primary Avro object-container PoC artifact.

verify_avro_duplicate_label_dual_view_poc.py
  End-to-end verifier. Reads the artifact with fastavro embedded schema,
  fastavro strict reader schema, Apache Avro Python, and ModelScan.

scripts/build_duplicate_label_dual_view.py
  Rebuilds the artifact byte-for-byte.

evidence/fresh_verify.json
  Fresh Python verification output from this staged repository.

evidence/apache_java_verify.txt
  Apache Avro Java 1.12.1 read-only rejection output.

evidence/hub_download_verify.json
  Verification after uploading to Hugging Face, downloading the public dataset
  repository, and running the verifier against the downloaded copy.

evidence/sha256.txt
  Hashes for uploaded files.

Reproduction

Install matching Python dependencies:

pip install "fastavro==1.12.1" "avro==1.12.1" "modelscan==0.8.8"

Run:

python verify_avro_duplicate_label_dual_view_poc.py

Expected output highlights:

{
  "artifact_sha256": "1fd67081a7a86b7973087015e1adf606aaaaa14e54c3da96a1bc5311b2c7ee09",
  "fastavro_embedded_schema_rows": [
    {"label": 1, "text": "benign sample"}
  ],
  "fastavro_strict_reader_schema_rows": [
    {"label": 0, "text": "benign sample"}
  ],
  "apache_avro_python": {
    "ok": false,
    "error_type": "SchemaParseException",
    "error": "Field name label already in use."
  },
  "verified": true
}

Optional Apache Avro Java cross-check:

mvn -q -f lab/java_avro_probe/pom.xml compile exec:java \
  -Dexec.mainClass=local.avroprobe.ReadOnlyProbe \
  -Dexec.args=duplicate_label_dual_view.avro

Expected Java result:

JAVA_DATAFILE_READ: AvroRuntimeException: Duplicate field label in record TrainingLabelRow...

Scanner Behavior

ModelScan 0.8.8 output for the artifact:

{
  "total_issues": 0,
  "scanned": {"total_scanned": 0},
  "skipped": {
    "total_skipped": 1,
    "skipped_files": [
      {"category": "SCAN_NOT_SUPPORTED", "source": "duplicate_label_dual_view.avro"}
    ]
  }
}

This is not claimed as a scanner bypass because Avro is unsupported by ModelScan in this environment. The reportable issue is fastavro accepting the invalid embedded schema and returning the later duplicate label value.

Root Cause

fastavro.reader() accepts an Avro object-container file whose embedded schema declares two fields with the same name. When decoding into a Python dictionary, the later duplicate field overwrites the earlier field under the same key.

When a caller supplies a strict reader schema with one label field, fastavro resolves the first writer field and returns label=0. When the caller trusts the embedded schema, fastavro returns the later duplicate value label=1.

Safety Notes

• —No arbitrary code execution is used.
• —No network callbacks, credential access, persistence, or destructive payloads are present.
• —The values are benign numeric labels chosen only to make the parser mismatch obvious.

Limitations

• —This is data/label output manipulation, not ACE.
• —The target is fastavro, not Apache Avro. Apache Avro Python and Java reject the artifact.
• —The schema is invalid under Avro’s duplicate-field-name rules; the issue is that fastavro accepts it on the embedded-schema datafile path.
• —This is strongest for ML pipelines, feature-store exports, dataset ingestion, or model-metadata workflows that load Avro OCF artifacts through fastavro.reader() without a strict reader schema.

Duplicate Checks

Local research did not find an obvious published advisory for this specific fastavro duplicate-field embedded-schema acceptance and dual-view label behavior. It is distinct from generic unsafe deserialization because the payload is a binary Avro object-container data artifact and the demonstrated impact is label-value disagreement across normal Avro reader paths.