etang98/lyric-generator-mvp

Mistral-7B Fine-Tuned to Write Lyrics

This repository contains a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.1 that has been specialized to generate song lyrics.

Model Description

This model is a Minimum Viable Product (MVP) and proof-of-concept for a larger Lyrical Style Emulation Pipeline. While this specific model has been trained to emulate the unique lyrical style of Taylor Swift, the underlying end-to-end pipeline was designed to be able to fine-tune the model on the style of any musical artist. This MVP model serves as the first successful implementation of this pipeline.

Training Procedure

1. 1.Data Engineering: A robust data acquisition and processing pipeline was developed in Python to automate the scraping, cleaning, and structuring of lyrical data into a custom, model-ready format with semantic and structural tags. The pipeline also extracts metadata, such as a biography or commonly used moods/themes, about the desired artist.
2. 2.AI-Powered Data Labeling: A more powerful language model (Cohere) was leveraged for knowledge distillation to the smaller Mistral model. Cohere analyzed the processed lyrics to generate the stylistic "Lyrical Blueprints" that formed the core of the training dataset.
3. 3.Parameter-Efficient Fine-Tuning: The mistral-7B-instruct-v0.1 model was fine-tuned for 5 epochs on consumer hardware (8GB VRAM) by implementing QLoRA and 4-bit quantization. Early stopping was used to select the checkpoint with the lowest validation loss.

Bias, Risks, and Limitations

This model is an MVP and has significant limitations. The generated output may produce nonsensical, repetitive, or generic text. The model's understanding of style is based entirely on the lyrical data it was trained on and may not capture the full nuance of the artist. It may also reflect biases present in the training data.

How to Use

You can use the model directly with a pipeline from the transformers library. The model has been fine-tuned to internalize the artist's style, so you only need to provide a creative brief. <u>This method loads the model in full precision. Please read the next section for notes on hardware usage. </u>

from transformers import pipeline

generator = pipeline('text-generation', model='etang98/lyric-generator-mvp')

# Here is a simple template for a prompt 
prompt = """
[INST]
You are an expert lyricist emulating the style of Taylor Swift. Write a complete, original song based on the following creative brief.

**Creative Brief:**
- **Theme:** [Describe the song's topic, e.g., "A reflective song about memories in a small town"]
- **Moods:** [List several moods, e.g., "Nostalgia, warmth, a little bittersweet"]
- **Setting (Optional):** [Add any specific imagery, e.g., "An old coffee shop in the rain"]
- **Additional Requirements (Optional):** [Anything else that should be included in the song, e.g., "Mention the year 1996"]

Generate the full song lyrics now.
[/INST]
"""

result = generator(prompt, max_new_tokens=750, temperature=0.7)
print(result[0]['generated_text'])

Hardware Requirements

Warning: This is a 7-billion parameter model and requires significant computational resources to run. For users with limited hardware, you can load the model in a lower precision using the from_pretrained method.

<u>GPU Required:</u> Running this model on a CPU will be extremely slow. A CUDA-enabled GPU is strongly recommended for reasonable performance.

1. Loading the Model

You can choose one of the following snippets to load the model and tokenizer based on your hardware.

Full Precision (float16) - ~16 GB VRAM

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("etang98/lyric-generator-mvp")
model = AutoModelForCausalLM.from_pretrained("etang98/lyric-generator-mvp")

8-bit Precision - ~8-10 GB VRAM

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("etang98/lyric-generator-mvp")
model = AutoModelForCausalLM.from_pretrained(
    "etang98/lyric-generator-mvp",
    load_in_8bit=True
)

4-bit Precision - ~5-6 GB VRAM

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("etang98/lyric-generator-mvp")
model = AutoModelForCausalLM.from_pretrained(
    "etang98/lyric-generator-mvp",
    load_in_4bit=True
)

2. Generating Text/Using the Model

Once you have chosen how to load the model and tokenizer, you use the following snippet to manually tokenize your prompt, generate the output tokens, and decode them back into text.

prompt = """
[INST]
You are an expert lyricist emulating the style of Taylor Swift. Write a complete, original song based on the following creative brief.

**Creative Brief:**
- **Theme:** [Describe the song's topic, e.g., "A reflective song about memories in a small town"]
- **Moods:** [List several moods, e.g., "Nostalgia, warmth, a little bittersweet"]
- **Setting (Optional):** [Add any specific imagery, e.g., "An old coffee shop in the rain"]
- **Additional Requirements (Optional):** [Anything else that should be included in the song, e.g., "Mention the year 1996"]

Generate the full song lyrics now.
[/INST]
"""

# Tokenize the prompt
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate the output tokens
outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.7)

# Decode the tokens back to a string
decoded_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded_text)

<u>Training Configuration & Hyperparameters</u>

LoRA Configuration

• —Rank (`r`): 64
• —Alpha (`alpha`): 64
• —LoRA Dropout: 0.1

Training Hyperparameters

• —Learning Rate: 3e-5
• —Batch Size: 2
• —Gradient Accumulation Steps: 8
• —Effective Batch Size: 16 (2 * 8)
• —Weight Decay: 0.01
• —Warmup Ratio: 0.1
• —Early Stopping Patience: 2
• —Early Stopping Threshold: 0.001
• —Epochs: 6 (Stopped after 5 due to early stopping)