Birchlabs/mosaicml-mpt-7b-chat-qlora

apply device-transfer patch from https://github.com/mosaicml/llm-foundry/pull/225/files

prefer NamedTuple

gradient checkpointing for multi-query attention

apply gradient checkpointing to Attention blocks

add support for AutoModelForCausalLM#from_pretrained()'s device_map='auto'. support gradient checkpointing, probably. add lots of type hints so I could understand what's going on. multiline long method signatures/calls (for easier comparison between checkpointed/non-checkpointed variants, and because these lines got even longer when I added type hints). make MPTForCausalLM#forward accept additional kwargs, since PeftModelForCausalLM#forward tries to send it an argument inputs_embeds=None, which it didn't like too much.

updt flash_attn_triton import (#12)

add flash_attn_triton.py (#9)

Update README.md

Update README.md

Remove print about meta init (#2)

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Create README.md

Add loss function and bool cast

Upload folder using huggingface_hub

initial commit