ThatOneShortGuy/MusicalFalcon

1---2language:3- en4license: apache-2.05library_name: peft6tags:7- music8datasets:9- ThatOneShortGuy/SongLyrics10base_model: OpenAssistant/falcon-7b-sft-mix-200011---12# Musical Falcon13 14[OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000) model fine tuned using PEFT on15[Song Lyrics](https://huggingface.co/datasets/ThatOneShortGuy/SongLyrics) to write lyrics to songs.16 17## Model Details18- **Finetuned from**: [OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000)19- **Model Type**: Causal decoder-only transformer language model20- **Language**: English (and limited capabilities in German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);21- **License**: Apache 2.022- **Contact**: Lol don't. This is just for fun.23 24## Usage25The basic basic format getting it in is:26```python27from peft import PeftModel, PeftConfig28from transformers import AutoModelForCausalLM29 30config = PeftConfig.from_pretrained("ThatOneShortGuy/MusicalFalcon")31model = AutoModelForCausalLM.from_pretrained("OpenAssistant/falcon-7b-sft-mix-2000")32model = PeftModel.from_pretrained(model, "ThatOneShortGuy/MusicalFalcon")33```34 35## Prompting36Considering this comes from [OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000), it uses the same structure.37Two special tokens are used to mark the beginning of user and assistant turns: `<|prompter|>` and `<|assistant|>`. Each turn ends with a `<|endoftext|>` token.38The training prompt used the structure: 39```40<|prompter|>Come up with the lyrics for a song from "{artist}" {"from " + year if year else ""} titled "{title}".<|endoftext|>41<|assistant|>Sure! Here are the lyrics:42{lyrics}43<|endoftext|>44```45 46However, it still seems to work just fine using:47```48<|prompter|>Write me a song titled "{title}"<|endoftext|><|assistant|>49```50or any anything of similar nature. Feel free to add a description of the song in there too.51 52## Examples53Prompt54```55<|prompter|>Write a song with the lyrics for a song from artist "Eminem" from 2018 titled "Suzy's Lemonade Stand"<|endoftext|>56<|assistant|>Sure! Here are the lyrics:57```58Result59```60[Intro]61I'm gonna make you famous, baby62 63[Verse 1]64It's been a long time since I had a lemonade stand65But I'm back in town and I'ma do it again66You know what? I've been on my own since I was born67So I'll be here all day long, so come on, let's go68I got some new tricks up my sleeve that I'd like to show69And I'mma get your name out of your mouth, so please don't blow70Let me see if I can get you to say my name71If not, then I'l just have to take it back to the drawing board72I gotta get this money, I's a little bit broke73I need some cash, I need some dough, I gotta get these clothes off74I wanna be a star, I wanna be the man75I want to be a superstar, I want to get my hands on them bands76```77 78 79## Training procedure80 81Trained on 1 epoch of all songs from [Song Lyrics](https://huggingface.co/datasets/ThatOneShortGuy/SongLyrics) with "Rap" or "Hip" in the genre title.82 83- GPU: RTX Titan84- load_in_8bit: True85 86Further trained on ~8 epochs of the top 10% of [Song Lyrics](https://huggingface.co/datasets/ThatOneShortGuy/SongLyrics)87 88- GPU: RTX Titan89- load_in_8bit: False90  - torch_dtype: torch.float1691 92- load_in_4bit: False93- llm_int8_threshold: 6.094- llm_int8_skip_modules: None95- llm_int8_enable_fp32_cpu_offload: True96- llm_int8_has_fp16_weight: False97- bnb_4bit_quant_type: fp498- bnb_4bit_use_double_quant: False99 100### Framework versions101 102 103- PEFT 0.5.0.dev0