CoolFace
Modelpublic

ThatOneShortGuy/MusicalFalcon

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face
0likes21downloads
Model Card

Musical Falcon

OpenAssistant/falcon-7b-sft-mix-2000 model fine tuned using PEFT on Song Lyrics to write lyrics to songs.

Model Details

  • —Finetuned from: OpenAssistant/falcon-7b-sft-mix-2000
  • —Model Type: Causal decoder-only transformer language model
  • —Language: English (and limited capabilities in German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);
  • —License: Apache 2.0
  • —Contact: Lol don't. This is just for fun.

Usage

The basic basic format getting it in is:

python
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM

config = PeftConfig.from_pretrained("ThatOneShortGuy/MusicalFalcon")
model = AutoModelForCausalLM.from_pretrained("OpenAssistant/falcon-7b-sft-mix-2000")
model = PeftModel.from_pretrained(model, "ThatOneShortGuy/MusicalFalcon")

Prompting

Considering this comes from OpenAssistant/falcon-7b-sft-mix-2000, it uses the same structure. Two special tokens are used to mark the beginning of user and assistant turns: <|prompter|> and <|assistant|>. Each turn ends with a <|endoftext|> token. The training prompt used the structure: 

<|prompter|>Come up with the lyrics for a song from "{artist}" {"from " + year if year else ""} titled "{title}".<|endoftext|>
<|assistant|>Sure! Here are the lyrics:
{lyrics}
<|endoftext|>

However, it still seems to work just fine using:

<|prompter|>Write me a song titled "{title}"<|endoftext|><|assistant|>

or any anything of similar nature. Feel free to add a description of the song in there too.

Examples

Prompt

<|prompter|>Write a song with the lyrics for a song from artist "Eminem" from 2018 titled "Suzy's Lemonade Stand"<|endoftext|>
<|assistant|>Sure! Here are the lyrics:

Result

[Intro]
I'm gonna make you famous, baby

[Verse 1]
It's been a long time since I had a lemonade stand
But I'm back in town and I'ma do it again
You know what? I've been on my own since I was born
So I'll be here all day long, so come on, let's go
I got some new tricks up my sleeve that I'd like to show
And I'mma get your name out of your mouth, so please don't blow
Let me see if I can get you to say my name
If not, then I'l just have to take it back to the drawing board
I gotta get this money, I's a little bit broke
I need some cash, I need some dough, I gotta get these clothes off
I wanna be a star, I wanna be the man
I want to be a superstar, I want to get my hands on them bands

Training procedure

Trained on 1 epoch of all songs from Song Lyrics with "Rap" or "Hip" in the genre title.

  • —GPU: RTX Titan
  • —loadin8bit: True

Further trained on ~8 epochs of the top 10% of Song Lyrics

  • —GPU: RTX Titan
  • —loadin8bit: False
  • —torch_dtype: torch.float16
  • —loadin4bit: False
  • —llmint8threshold: 6.0
  • —llmint8skip_modules: None
  • —llmint8enablefp32cpu_offload: True
  • —llmint8hasfp16weight: False
  • —bnb4bitquant_type: fp4
  • —bnb4bitusedoublequant: False

Framework versions

  • —PEFT 0.5.0.dev0