Ethahtz/state-tracking-gpt2small-hybrid_8lin_last

Random-init hybrid GPT-2-small / Gated DeltaNet (hybrid_8lin_last); layer_types=['full_attention', 'linear_attention', 'linear_attention', 'full_attention', 'linear_attention', 'linear_attention', 'full_attention', 'linear_attention', 'linear_attention', 'full_attention', 'linear_attention', 'linear_attention']

initial commit