archiitecture/dynamic_proteins_promise_sf_cluster_continuation

Dynamic Protein Benchmarking MVP

This repository contains a first-pass Python package for benchmarking whether protein structure-generation models recover multiple experimentally observed conformations of the same protein.

The MVP targets unconditioned multistate recovery for BioEmu and Boltz across full-MSA, shallow-MSA, and no-MSA style conditions. The initial implementation emphasizes reproducible data structures, evaluation metrics, deterministic outputs, and a notebook UI for generating figures from a manifest and prediction metadata.

Repository layout

• —src/dynbench/ — installable Python package with inference adapters, curation helpers, structural metrics, analysis, plotting, and smoke-test workflow code.
• —configs/ — YAML configuration files for thresholds, model conditions, paths, and run defaults.
• —data/ — local data area. data/manifests/ contains editable MVP manifest templates; data/raw/ is reserved for downloaded or manually placed source structures.
• —results/ — deterministic output area for raw predictions, evaluation tables, and generated figures.
• —notebooks/ — front-facing notebook interface for users who want to plug in a protein or manifest and generate benchmark figures.
• —scripts/ — command-line entry points for ingestion, curation, smoke-test execution, evaluation, and plotting.
• —tests/ — pytest coverage for alignment, metrics, assignment, and dry-run workflow behavior.

Every folder includes its own README with details about expected inputs and outputs.

Quick start

Create the UV environment and install the package:

uv sync --group dev

Run tests:

uv run --group dev pytest

Run the legacy smoke-test workflow in dry-run mode:

uv run dynbench-smoke --config configs/default.yaml

Prepare the first MSA-depth experiment from the final shared manifest:

uv run dynbench-run-experiment \
  --config configs/experiments/one_protein.yaml \
  --prepare-colab

This writes:

• —results/one_protein_msa_mvp/run_table.csv
• —results/one_protein_msa_mvp/msa_metadata.csv
• —results/one_protein_msa_mvp/colab_transfers/remote_jobs.json
• —results/one_protein_msa_mvp/colab_transfers/dynbench_workspace.zip

Attach the VS Code Google Colab kernel to notebooks/colab_worker_scaffold.ipynb, upload the workspace ZIP to /content/dynbench_workspace.zip, and run the notebook. BioEmu/Boltz execute on the Colab GPU; result import, evaluation, and plotting run locally on the Mac CPU.

The Colab workspace archive intentionally includes only package source, configs, scripts, data/msas/, data/manifests/, and data/dynamic_protein_pairs.parquet when present. For metrics-only jobs, it also includes only the specific data/references/ files named in the job bundle. It does not copy data/raw/ or the full generated reference cache into Colab.

Build a manifest from the Protein State Router/Dynamic Conditioning handoff:

uv run dynbench-router-handoff \
  --output-manifest data/dynamic_protein_pairs.parquet \
  --reference-root data/references/router_dataset_v1

The documented handoff filename was router_dataset_v1.parquet. If that file is absent, the adapter falls back to the current Dynamic Conditioning initial-router file, protein_state_router_initial.parquet. In the current local dataset this produces 133 DynamicMPNN-positive protein pairs. Older handoff exports may contain the larger 493-row positive/negative mix.

Run the first basic variability workflow:

uv run dynbench-basic-variability \
  --manifest data/dynamic_protein_pairs.parquet \
  --experiment-id basic_variability \
  --n-proteins 4 \
  --samples-per-protein 5 \
  --prepare-colab

After running the generated Colab workspace and importing /content/dynbench_results.zip, evaluate and write user-facing graphs:

uv run dynbench-basic-variability \
  --manifest data/dynamic_protein_pairs.parquet \
  --experiment-id basic_variability \
  --n-proteins 4 \
  --samples-per-protein 5 \
  --evaluate \
  --plot

Graphs are written to outputs/basic_variability/graphs/, including a combined matplotlib subplot overview and per-protein RMSD scatter grid.

For the current 10-protein, 20-sample BioEmu/Boltz Colab smoke run:

uv run dynbench-basic-variability \
  --manifest data/dynamic_protein_pairs.parquet \
  --experiment-id router_10x20_bioemu_boltz \
  --n-proteins 10 \
  --samples-per-protein 20 \
  --model bioemu \
  --model boltz \
  --prepare-colab

This creates results/router_10x20_bioemu_boltz/colab_transfers/dynbench_workspace.zip. Run that workspace in notebooks/colab_worker_scaffold.ipynb with a VS Code Google Colab GPU kernel. The remote worker computes per-sample RMSD-to-state-A, RMSD-to-state-B, DME, radius of gyration, clash score, and state assignment in Colab, then deletes generated structure files before packaging results.

Generate plots from an evaluation table:

uv run dynbench-plot results/evaluations/sample_metrics.csv --out-dir results/figures

The older low-level VS Code + Google Colab GPU commands are still available:

uv run dynbench-colab-prepare-jobs \
  --manifest data/manifests/smoke_manifest.csv \
  --run-id smoke_colab_001 \
  --model bioemu \
  --protein-id smoke_adenylate_kinase

uv run dynbench-colab-pack-workspace \
  --job-bundle results/colab_transfers/remote_jobs.json \
  --output results/colab_transfers/dynbench_workspace.zip

Upload results/colab_transfers/dynbench_workspace.zip to a VS Code-attached Colab kernel and run notebooks/colab_worker_scaffold.ipynb. After downloading the result ZIP:

uv run dynbench-colab-import-results /path/to/dynbench_results.zip --force
uv run dynbench-evaluate --manifest data/manifests/smoke_manifest.csv
uv run dynbench-plot results/evaluations/sample_metrics.csv

The Colab notebook now hard-fails if nvidia-smi cannot see a GPU or if any BioEmu/Boltz job writes failed prediction metadata. Do not run GPU adapters on the Mac; local execution is for curation, import, evaluation, and plotting only.

Composite ProMiSE/SF-Cluster continuation dataset

Build the local-path-first composite tables with ProMiSE and SF-Cluster source directories. DynamicMPNN remains owned by the separate Dynamic Conditioning project and is accepted only as an exported parquet/CSV table:

uv run dynbench-composite-build \
  --promise-root /path/to/promise-bench \
  --sfcluster-root /path/to/sf-cluster \
  --dynamicmpnn /path/to/dynamicmpnn_export.parquet

Review data/composite/dataset_summary.json, overlap, ingestion errors, and manual-review outputs before publication. The notebook notebooks/composite_dataset_workflow.ipynb provides the interactive workflow. Single-reference SF-Cluster IDR cases are retained as validation records and are never treated as negatives by the builder.

GitHub publishing note

This checkout currently needs a configured origin remote before changes can be pushed from here. If the repo exists on GitHub, add the remote once:

git remote add origin git@github.com:<owner>/<repo>.git

After that, normal git push works from the current branch. The GitHub CLI (gh) is optional for pushing but required by the draft-PR helper workflow.

MVP status

Implemented now:

• —package-style project structure with UV configuration;
• —common inference adapter interface;
• —BioEmu and Boltz dry-run adapters with deterministic metadata;
• —serialized Colab job bundles and archive-based VS Code Colab transfer;
• —experiment YAML runner for one-protein, four-protein, and MVP-scale MSA-depth runs;
• —final parquet manifest adapter for dynamic_protein_pairs.parquet;
• —benchmark-owned MSA prep for full, shallow, and query-only conditions;
• —basic variability workflow for n proteins by m samples per protein;
• —Protein State Router/Dynamic Conditioning handoff adapter with DynamicMPNN .pt reference extraction and RCSB mmCIF fallback download;
• —metrics-only remote worker mode for Colab runs that keeps evaluation tables but drops generated structure files;
• —manifest schema and curation/QC helpers;
• —global RMSD, optional US-align TM-score, radius of gyration, clash score, and distance-map error;
• —optional residue-mask support for region-aware metrics;
• —ensemble summary statistics and state assignment;
• —plotting functions for RMSD, TM score, confidence/variance, and occupancy;
• —front-facing notebook UI;
• —smoke-test manifest and MVP manifest templates;
• —unit tests for rigid-transform invariant RMSD and workflow behavior.

Deferred/runtime-dependent:

• —end-to-end validation of exact production BioEmu/Boltz runtime flags in a live Colab GPU;
• —installing/provisioning the US-align binary on developer machines and Colab images;
• —manual biological annotation completion.

Data policy

Do not commit large raw structure files, model weights, or generated predictions unless intentionally creating a small fixture. Use data/raw/ and results/ as local working directories.