Modelpublic

fractal-agi/fdra-half-life-regularization

sourceHugging Faceupdated 8mo agoView on Hugging Face

0likes

Model Card

FDRA Half-Life Regularization V3: Bug-Fixed Implementation

Date: 2026-01-22 Repository: fractal-agi/fdra-half-life-regularization

Summary

This package contains the bug-fixed implementation of the half-life regularizer for FDRA oscillators, addressing critical issues discovered during code review.

The Problem

FDRA models trained at GPT-2 scale experience half-life collapse: all oscillator decay parameters converge to short values (τ ≈ 2-10 steps), losing the ability to maintain long-range context.

The Solution

A half-life regularizer that maintains a log-uniform distribution of τ ∈ [1, L] where L is the sequence length, ensuring some oscillators can attend to the full context.

Bug Fixes (V3)

Bug	Severity	Issue	Fix
`np.clip` argument order	CRITICAL	`np.clip(x, max, min)` clips everything to `min`	Swapped to `(min, max)`
Missing tau bounds	CRITICAL	Moment-matching created pathological τ<1	Added `compute_bounds_loss()`
Sigmoid overflow	Medium	`exp(-k*tau)` overflowed	Added `np.clip(x, -500, 500)`
Learning rate	Medium	lr=0.3 overshot valid λ range	Changed to lr=0.0001
Mean-only convergence	Medium	All τ converged to same value	Use log-uniform init directly

Results

Before Bug Fixes (V1-V2)

Regularized tau: [0.48, 6931.1]  ← PATHOLOGICAL
23/32 oscillators with τ < 1     ← WORSE than collapsed!
Basin width: 256 tokens

After Bug Fixes (V3)

Regularized tau: [1.0, 4096.0]   ← Proper log-uniform spread
3/32 oscillators with τ > 2048   ← Long-range coverage
Basin width: 1024 tokens         ← 4x improvement

Identity Reconstruction Comparison

Condition	Verdict	Basin Width	Notes
Collapsed (no regularization)	FAIL	0	Identity immediately lost
Regularized (log-uniform τ)	PARTIAL	1024 (25% of L)	Identity preserved to K=1024

Files

├── half_life_regularizer.py     # Core regularizer with bounds constraint
├── fdra_oscillators.py          # Oscillator bank implementation
├── identity_reconstruction_experiment_v2.py  # Fixed diagnostic experiment
├── identity_v2_*.json           # Raw experimental results
├── IDENTITY_V2_REPORT_*.md      # Generated reports
├── BUGFIX_REPORT.md             # Detailed bug analysis
└── IMPLICATIONS.md              # Research implications

Usage

python

from half_life_regularizer import HalfLifeRegularizer, HalfLifeRegularizerConfig

config = HalfLifeRegularizerConfig(
    sequence_length=4096,
    tau_min=1.0,
    tau_max=4096.0,
    lambda1=0.01,  # Log-uniform prior weight
    lambda2=0.01,  # Long-tail weight
    lambda3=0.1    # Bounds constraint weight (NEW!)
)

regularizer = HalfLifeRegularizer(config)

# During training:
loss, metrics = regularizer.compute(oscillator_lambdas)
total_loss = task_loss + loss

Citation

If you use this work, please cite:

@misc{fdra-half-life-regularization-2026,
  title={Half-Life Regularization for FDRA Oscillators: Preventing Decay Collapse},
  author={Fractal AGI},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/fractal-agi/fdra-half-life-regularization}
}

License

Apache 2.0