astro-legacy-archive/cbi-released-products

Cosmic Background Imager released products

This repository contains the numerical products released for four generations of Cosmic Background Imager (CBI) analysis: the 2000 deep fields, the 2000 mosaic fields, the final 2002–2005 temperature and polarization analysis, and the final 2000–2005 total-intensity analysis. It contains the published band powers, their window functions, the same-band correlation blocks and full Fisher matrices distributed in CosmoMC .newdat files, and the correlation and inverse-Fisher matrices distributed with the earlier deep and mosaic products. These are statistical products, not calibrated visibilities or sky maps. LAMBDA's current CBI product table serves neither of those data classes.

[image]

Create an isolated fleet environment with python -m venv .venv && .venv/bin/pip install datasets huggingface_hub pyarrow numpy.

The 145 configurations are deliberately fine-grained. There are 128 scalar statistical tables using statistical-product 1.2.0 and 17 arrays using dense-matrix 1.0.0. A source file containing one table uses its filename stem. An archive member uses <source-file-stem>__<member-basename>. This retains the anomalous source stems deep.tar.gz, deep_windows.tar.gz, mosaic.tar.gz, and mosaic_windows.tar.gz: the downloaded filenames genuinely end in .tar.gz.tar, so applying the ordinary final-suffix stem rule leaves .tar.gz in the configuration. Full tar member paths remain in Parquet metadata even where the slash-free configuration uses the member basename.

The two .newdat files require one additional piece of source description because their matrix blocks are positional and carry no inline member token. Their identities are not archive inventions. Two pinned LAMBDA detailed-product documents call them “same-band correlations between band-powers”, “Fisher matrix for the entire spectrum”, and, for the polarization product, “Fisher matrix for the entire spectrum (all polarizations)”. Those phrases are copied verbatim into configuration names and dense columns. Each dense record embeds the authority document's URL, SHA-256 and exact HTML passage, while its source structure records the block position, source line span, polarization token and absence of axis names.

Reading a band-power table

The following reads one row from the final total-intensity TT product. The displayed result was run against the built CBI09__TT.parquet.

from datasets import load_dataset
row = load_dataset("astro-legacy-archive/cbi-released-products", "CBI09__TT", split="train[:1]")[0]
print(row["Column 1"], row["Column 2"], row["Column 3"], row["Column 6"], row["Column 7"])

1 5695.0 753.2 0.0 350.0

The numbered names are intentional. The numeric rows themselves are headerless, so the statistical-product rule retains positional Column 1, Column 2, and so on rather than turning documentation into a replacement header. The pinned LAMBDA descriptions remain attached as field descriptions. Integral band-number and likelihood-flag tokens remain int64; the remaining .newdat values remain float64. Window and older qb tables likewise retain an integer first column. No effective multipole, symmetric uncertainty, normalized window, bin label or spectrum classification has been added.

The standalone deep and mosaic configurations preserve their printed table labels. Their published cells such as 7193 +/-2478 remain single text values because splitting them would change the source table's shape. The mosaic even/odd boundaries and the older qb files' terminal -2Ln(L) records are retained in self-description rather than promoted to invented columns or rows. Every table keeps source row order.

Reconstructing a matrix

A dense matrix has one fixed-size-list Parquet column. One Parquet row is one source matrix row; flattening that column in row order and reshaping to the recorded source shape reconstructs the source array. This example reconstructs the final all-polarization Fisher matrix. Its output was run against the built Parquet.

import json
import pyarrow.parquet as pq
from huggingface_hub import hf_hub_download
path = hf_hub_download("astro-legacy-archive/cbi-released-products", "CBIpol_2.0_final__Fisher matrix for the entire spectrum (all polarizations).parquet", repo_type="dataset")
print(json.loads(pq.read_metadata(path).metadata[b"dense_matrix"])["array"]["identifier"])

Fisher matrix for the entire spectrum (all polarizations)
(51, 51) float64 935580.0 62091.0

All dense values are source float64 values in exact source serialization order. The earlier archive matrices use their verbatim leaf basenames, such as joint_3deep_std2_best_corr, as both the dense column identity and the final component of the configuration address. Matrix preambles are retained verbatim in metadata; they are not interpreted as axes. Correlation, inverse-Fisher and Fisher products have different conventions and scales, so consumers should use the attached source structure and cited analysis rather than assume that every square array is interchangeable.

Scope and provenance

The source census consists of ten pinned LAMBDA data files plus four pinned LAMBDA HTML documents: the deep-field, mosaic, total-intensity, and temperature/polarization interpretation pages. The mosaic page's covariance and window sections contain copied deep-field wording; those passages do not describe the mosaic members. The source archives contain 131 regular members. The two .newdat files contribute twelve configurations, and the two standalone text files contribute two more.

The preview uses only the five built .newdat band-power Parquets. It plots the exact published Column 2 values over the exact Column 6–Column 7 bin spans. For CBIpol it reads both supplied error-bar fields, Column 3 and Column 4; for CBI09 it uses the single documented Gaussian error bar. The plot does not derive effective multipoles or alter negative band powers.

No named or formal data licence was found on the checked CBI product or detailed-description pages. HEASARC states verbatim: “HEASARC materials are all available freely for your use.” LAMBDA asks users of the total-intensity release also to cite its original page and publication date, 2009-01-28.

We acknowledge the use of the Legacy Archive for Microwave Background Data Analysis (LAMBDA), part of the High Energy Astrophysics Science Archive Center (HEASARC). HEASARC/LAMBDA is a service of the Astrophysics Science Division at the NASA Goddard Space Flight Center.

Release papers: Mason et al. (2003), DOI, arXiv; Pearson et al. (2003), DOI, arXiv; Sievers et al. (2007), DOI, arXiv; and Sievers et al. (2009), DOI, arXiv. The invoked likelihood methods are Myers et al., DOI, arXiv, and Sievers et al., DOI, arXiv.

Errata and requests

To report an erratum, propose a correction, or ask a question about this dataset, open a Discussion or Pull Request in its Community tab; posting requires a free Hugging Face account. Dataset requests and archive-wide questions may be posted in the Community tab of any dataset. Confirmed corrections are made in the source conversion tools and published by rebuilding the dataset.