jniimi/datafusion-acs

Data Fusion: ACS Housing

A small, public-domain dataset for statistical data fusion: two surveys share a block of covariates, each measures a different block of outcomes, and no respondent answers both. The task is to fill in, for each respondent, the block their survey did not ask.

This is the ACS-housing task of Cross-Block Conditioning in Deep Boltzmann Machines for Statistical Data Fusion. It is meant to be picked up in a few minutes: load it, see what fusion data look like, fit a baseline, and score it the way the paper does.

What fusion data look like

[image]

Each group of respondents answers one of the two surveys, and the block the other survey collected is unobserved for them; only the covariates are shared. In this dataset the two sources are a labor-and-income block ($YA$, the `ya columns) and a housing block ($Y_B$, the yb` columns), and the common variables are demographics ($X$, the `x` columns).

The train split is 80,000 households divided at random into two panels, as if a labor-force survey and a housing survey had each interviewed half of them. No training row observes both outcome blocks; the block a panel did not collect is null. The validation and test splits keep every block, only so that predictions can be scored.

Because $YA$ and $YB$ never appear together, any model of their joint distribution has to get the association between them from something other than paired examples, and any model that maps $X$ to outcomes has given up on it. That is what makes fusion different from ordinary supervised learning or from missing-value imputation with a few complete rows.

Quickstart

import numpy as np
from datasets import load_dataset
from sklearn.linear_model import LogisticRegression

ds = load_dataset("jniimi/datafusion-acs", "housing")
train = ds["train"].to_pandas().head(2000)   # the first n rows are the paper's n_train = n
test = ds["test"].to_pandas()

X  = [c for c in train if c.startswith("x_")]    # common block: demographics (30)
YA = [c for c in train if c.startswith("ya_")]   # outcome block A: labor and income (22)
YB = [c for c in train if c.startswith("yb_")]   # outcome block B: housing (17)

panel_a = train[train.source == "A"]   # observes X and Y_A; Y_B is null
panel_b = train[train.source == "B"]   # observes X and Y_B; Y_A is null

def fit_predict(panel, targets):
    return {y: LogisticRegression(max_iter=1000).fit(panel[X], panel[y].astype(int))
                                             .predict_proba(test[X])[:, 1] for y in targets}

pred = {**fit_predict(panel_a, YA), **fit_predict(panel_b, YB)}
acc = {y: np.mean((pred[y] > 0.5) == test[y]) for y in YA + YB}
print(f"combined accuracy, X-only: {100 * np.mean(list(acc.values())):.2f}%")   # 83.10%

This is the simplest fusion baseline: each outcome block is predicted from $X$ alone, with a model fitted on the panel that observed it.

Scoring

All variables are binary. The paper's headline metric is combined accuracy, the accuracy over all $22 + 17$ outcome columns at a threshold of 0.5 (equivalently, the block accuracies weighted by block width). Cross-entropy is a useful second metric.

A model can be scored under two conditioning sets:

• —$X$-only: both outcome blocks are hidden and predicted from $X$. Every method can do this.
• —cross-block: $YA$ is predicted from $X$ and $YB$, and $YB$ from $X$ and $YA$. This is the operational task (a panel-B household has its housing answers and needs its labor answers filled in), but it asks for $p(yA \mid x, yB)$, which the training data never show directly.

One rule keeps the evaluation honest: choose hyperparameters and checkpoints on $X$-only validation accuracy only. Real fusion data contain no row that observes both blocks, so whether cross-block conditioning helps can never be checked there. A method that tunes its cross-block predictions on complete validation rows is using information a practitioner would not have.

Reference results

Test combined accuracy [%] on the released split (seed 0, all 30 $X$ columns), from the paper. Baselines are tuned on $X$-only validation; "+ cross" means the imputer is also given the other outcome block at test time.

Two things stand out. Predicting from $X$ alone, a tuned logistic regression is hard to beat. And giving the imputers the other outcome block makes most of them worse: trained without a single paired row, they do not know how to use it. OBMP, a Deep Boltzmann Machine fine-tuned on the observed blocks, gains from it. The paper reports means over five seeds and the full grid of sample sizes and covariate widths; the pattern is the same there.

There is room to do better. With paired rows (which fusion never has), a logistic regression gains about 1.3 points from the other block at $n = 2{,}000$; OBMP recovers about a third of that without them.

Files and columns

• —housing/{train,validation,test}.parquet
• —serialno: the PUMS housing-unit serial number, so that other PUMS variables can be joined on.
• —source: "A" or "B" in train (which panel the row belongs to), null in validation and test.
• —x_*, ya_*, yb_*: binary indicators (nullable int), one-hot or quantile-binned from PUMS variables. A variable's reference level is the all-zero pattern.
• —housing/features.parquet: one row per indicator, with its block, source PUMS variable, level, a description and its prevalence. Read it with pd.read_parquet("hf://datasets/jniimi/datafusion-acs/housing/features.parquet").

Blocks. $X$ (30): age band, sex, race, Hispanic origin, marital status, education, citizenship, language at home, limited English, veteran status, household size, children present. $YA$ (22): employment status, class of worker, usual hours, weeks worked, wage tertiles, personal-income quartiles, and receipt of self-employment, retirement, Social Security and public-assistance income. $YB$ (17): tenure, household-income quartiles, SNAP receipt, vehicles, structure type, rent burden of at least 30%, no internet access.

Rows. One row per occupied housing unit in California: the householder, aged 18 or over, joined to the housing record; group quarters are excluded. Keeping one row per household means housemates, who share every housing variable, cannot fall on both sides of a split. Survey weights are not used; the task is prediction on the sample.

Reproducing the paper. The rows are the paper's seed-0 split, and train is ordered so that train[:n] is exactly the paper's training set for sample size $n$ with all 30 $X$ columns. For narrower common blocks the paper keeps the $k$ most prevalent $X$ columns (see prevalence in features.parquet).

Source and license

Derived from the 2024 American Community Survey 1-year Public Use Microdata Sample for California (US Census Bureau), a US federal work in the public domain. The derived files are released under CC0.

This dataset uses Census Bureau data but is not endorsed or certified by the Census Bureau. PUMS records are already anonymized by the Census Bureau; do not attempt to identify individuals or households, or to link the records to other data for that purpose.

Citation

@article{niimi2026crossblock,
  title   = {Cross-Block Conditioning in Deep {Boltzmann} Machines for Statistical Data Fusion},
  author  = {Niimi, Junichiro},
  journal = {arXiv preprint arXiv:2609.14934},
  year    = {2026}
}