Modelpublic

momentarek1/Text-to-Face_Generation

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes

Model Card

🧑‍🎨 Text-to-Face Generation with BERT, GAN & CelebA

A deep learning project for text-to-face image generation, where natural-language descriptions are converted into realistic facial images using Sentence-BERT embeddings and a conditional Generative Adversarial Network (GAN).

The project contains two text-conditioned face generation models:

1.Baseline Text-to-Face Generator — a lightweight conditional generator.
2.Attention-based Text-to-Face GAN — an improved architecture using Self-Attention, Spectral Normalization, and a conditional Discriminator.

📌 Project Overview

The goal of this project is to generate a face image from a textual description.

For example, given:

"The female has high cheekbones. Her hair is black. She has arched eyebrows, a big nose and bushy eyebrows. She is young, smiling and wearing lipstick."

the model attempts to generate a corresponding face.

The overall pipeline is:

text

Text Description
       │
       ▼
Sentence-BERT
       │
       ▼
768-D Text Embedding
       │
       ▼
Text Conditioning
       │
       +
Random Noise
       │
       ▼
Generator
       │
       ▼
Generated Face

The project uses the CelebA dataset with natural-language descriptions associated with facial images.

🎯 Objectives

The project aims to explore:

—Text-to-image generation
—Conditional GANs
—Text embeddings
—Sentence-BERT
—Face generation
—GAN architecture design
—Self-Attention
—Spectral Normalization
—Conditional Discriminators
—Image-text matching
—Generative model training

🗂️ Dataset

The project uses the CelebA (CelebFaces Attributes Dataset).

CelebA contains large-scale celebrity face images together with facial attribute annotations.

The project additionally uses textual descriptions generated from the facial attributes.

Dataset sources

—Kaggle CelebA Dataset
—CUHK Multimedia Lab — CelebA

The project expects a dataset containing:

text

Face Images
      +
Text Descriptions
      +
CelebA Attribute Information

📝 Text Descriptions

Each image is associated with one or more natural-language descriptions.

Example:

text

The female has pretty high cheekbones and an oval face.
She has brown hair.
She has arched eyebrows and a pointy nose.
She is smiling, seems attractive and young.
She has rosy cheeks and heavy makeup.
She is wearing earrings and lipstick.

Another example:

text

He wears a 5 o'clock shadow.
His hair is brown and straight.
He has a slightly open mouth and a pointy nose.
He looks attractive and young and is smiling.
He is wearing a necktie.

These descriptions provide the semantic information used to condition the image generator.

🧠 Text Encoder

The project uses:

text

SentenceTransformer

with:

text

all-mpnet-base-v2

The model converts each textual description into a 768-dimensional semantic embedding.

python

SentenceTransformer("all-mpnet-base-v2")

The descriptions are split into sentences and each sentence is encoded individually.

The resulting sentence embeddings are then averaged:

text

Text
 │
 ├── Sentence 1 ──► Embedding
 ├── Sentence 2 ──► Embedding
 ├── Sentence 3 ──► Embedding
 └── Sentence N ──► Embedding
                    │
                    ▼
              Mean Embedding
                    │
                    ▼
                 768-D

This produces a fixed-size representation regardless of the number of sentences.

🔢 Text Embedding

The initial embedding size is:

text

The embedding is then reduced to:

text

using a projection layer.

text

Sentence-BERT
     │
     ▼
768 dimensions
     │
     ▼
Linear Layer
     │
     ▼
256 dimensions

🏗️ Model 1 — Baseline Text-to-Face Generator

The first model is a conditional image generator that combines:

—Random noise
—Text embeddings
—Transposed convolution layers
—Batch Normalization
—ReLU / LeakyReLU
—Tanh output

The model generates RGB face images.

Architecture

The input consists of:

text

Random Noise
100 dimensions

and:

text

Text Embedding
768 dimensions

The text embedding is projected to:

text

256 dimensions

The generator then concatenates the noise and text representation.

text

                Text Description
                       │
                       ▼
                Sentence-BERT
                       │
                       ▼
                    768-D
                       │
                       ▼
                 Projection
                       │
                       ▼
                    256-D
                       │
                       ├────────────┐
                       │            │
                       ▼            ▼
                 Random Noise     Text
                   100-D          256-D
                       │            │
                       └─────┬──────┘
                             ▼
                    Conditional Input
                             │
                             ▼
                    ConvTranspose2D
                             │
                             ▼
                          4 × 4
                             │
                             ▼
                          8 × 8
                             │
                             ▼
                         16 × 16
                             │
                             ▼
                         32 × 32
                             │
                             ▼
                         64 × 64
                             │
                             ▼
                       RGB Image

Generator Configuration

python

model = Generator(
    100,     # noise size
    128,     # feature size
    3,       # RGB channels
    768,     # embedding size
    256      # reduced embedding size
)

Main parameters

Parameter	Value
Noise Dimension	100
Initial Feature Size	128
Text Embedding	768
Reduced Embedding	256
Output Channels	3
Optimizer	Adam
Learning Rate	0.0002
Betas	(0.5, 0.5)

🖼️ Generated Image Resolution

The baseline generator progressively upsamples the feature representation:

text

The final output is an RGB face image.

💾 Pretrained Generator

A trained generator checkpoint can be loaded using:

python

model.load_state_dict(
    torch.load(
        "generator_50k.pth",
        map_location="cpu"
    )
)

model.eval()

This allows the model to generate faces directly from new text descriptions without retraining.

🧪 Baseline Inference

Example:

python

test_noise = torch.randn(
    size=(1, 100)
)

test_embeddings = sentence_encoder.convert_text_to_embeddings([
    "The female has pretty high cheekbones and an oval face. "
    "She has brown hair. She has arched eyebrows and a pointy nose. "
    "She is smiling, seems attractive, young, has rosy cheeks and heavy makeup. "
    "She is wearing earrings and lipstick."
])

test_image = model(
    test_noise,
    test_embeddings
)

The generated image is then visualized using torchvision.

🧠 Model 2 — Attention-Based Text-to-Face GAN

The second model is a more advanced conditional GAN architecture.

It contains:

—Conditional Generator
—Conditional Discriminator
—Sentence-BERT
—Self-Attention
—Spectral Normalization
—Batch Normalization
—Transposed Convolution
—Conditional image-text discrimination

The architecture is designed to improve image quality and capture long-range spatial relationships in facial features.

🏗️ Generator Architecture

The second generator receives:

text

Text Embedding
+
Random Noise

The text embedding is processed through:

text

The resulting representation is combined with the random noise through element-wise multiplication.

python

concat_input = torch.mul(
    noise,
    encoded_text
)

Generator Pipeline

text

                    Text
                     │
                     ▼
              Sentence-BERT
                     │
                     ▼
                   768-D
                     │
                     ▼
                  Linear
                     │
                     ▼
                   256-D
                     │
                     ▼
                  Linear
                     │
                     ▼
                   100-D
                     │
                     │
Random Noise ────────┤
                     ▼
              Element-wise
               Multiplication
                     │
                     ▼
                  1 × 1
                     │
                     ▼
               4 × 4 Feature
                     │
                     ▼
               8 × 8 Feature
                     │
                     ▼
              16 × 16 Feature
                     │
                     ▼
                Self-Attention
                     │
                     ▼
              32 × 32 Feature
                     │
                     ▼
                Self-Attention
                     │
                     ▼
              64 × 64 Feature
                     │
                     ▼
                Self-Attention
                     │
                     ▼
             128 × 128 Image

👁️ Self-Attention

The generator includes custom Self-Attention modules.

The attention mechanism allows the model to establish relationships between distant spatial locations.

This is useful for face generation because facial features are not completely independent.

For example:

text

Eyes
 │
 ├──────► Nose
 │
 ├──────► Mouth
 │
 └──────► Face Shape

Instead of only processing local convolutional features, Self-Attention allows the network to model broader spatial relationships.

🔬 Self-Attention Architecture

The module uses three projections:

text

Query
Key
Value

implemented using convolution layers:

python

self.query_conv
self.key_conv
self.value_conv

Attention scores are calculated using matrix multiplication:

text

Query × Key
     │
     ▼
 Softmax
     │
     ▼
Attention Map
     │
     ▼
Value

The attention output is then combined with the original feature map through a learnable parameter:

text

Output = γ × Attention + Input

🛡️ Spectral Normalization

The advanced architecture also implements Spectral Normalization for convolutional layers.

Spectral normalization constrains the magnitude of the network weights and helps stabilize GAN training.

It is applied to:

text

Generator
    │
    ├── ConvTranspose2D
    └── ConvTranspose2D

Discriminator
    │
    └── Conv2D

Conceptually:

text

Convolution
     │
     ▼
Spectral Normalization
     │
     ▼
Controlled Weight Magnitude
     │
     ▼
More Stable GAN Training

🕵️ Discriminator

The discriminator determines whether an image is:

text

Real
   or
Fake

but this model also receives the corresponding text description.

Therefore, it performs conditional discrimination.

Instead of asking only:

Is this image real?

the discriminator effectively evaluates:

Is this image realistic and consistent with the given text?

🔄 Conditional Discriminator

The architecture is:

text

                 Image
                   │
                   ▼
             CNN Encoder
                   │
                   ▼
            Image Features
                   │
                   │
Text ──► Sentence-BERT
                   │
                   ▼
             Text Encoder
                   │
                   ▼
            Text Features
                   │
             ┌─────┴─────┐
             │           │
             └─────┬─────┘
                   ▼
           Image + Text
              Features
                   │
                   ▼
             CNN Layers
                   │
                   ▼
          Real / Fake Score

🧩 Wrong Image Training

An important part of the discriminator training is the use of wrong image-text pairs.

For each text description:

text

Correct Pair

Text ─────────► Correct Face

and:

text

Incorrect Pair

Text ─────────► Different Face

The discriminator learns to distinguish between:

text

Real Image + Correct Text

and:

text

Wrong Image + Text
Fake Image + Text

This encourages the generated image to be semantically related to the description.

📊 Dataset Pipeline

The custom dataset returns:

python

true_image
true_text
wrong_image

The pipeline is:

text

CelebA
   │
   ├────────► Real Image
   │
   ├────────► Text Description
   │
   └────────► Random Wrong Image

Images are resized to:

text

128 × 128

and normalized using:

python

transforms.Normalize(
    mean=(0.5),
    std=(0.5)
)

⚙️ Training Configuration

The advanced model uses:

Parameter	Value
Epochs	20
Batch Size	16
Dataset Subset	20,000
Noise Size	100
Feature Size	64
Image Size	128 × 128
Channels	3
Text Embedding	768
Reduced Embedding	256
Generator LR	0.0002
Discriminator LR	0.0002
Attention	Enabled
Optimizer	Adam

⚔️ GAN Training

The training process alternates between the Generator and Discriminator.

text

                 Text
                  │
                  ▼
            Text Encoder
                  │
                  ▼
             Embeddings
                  │
                  ▼
              Generator
                  ▲
                  │
             Random Noise
                  │
                  ▼
             Fake Image
                  │
          ┌───────┴────────┐
          │                │
          ▼                ▼
      Generator        Discriminator
        Loss               │
                           │
              ┌────────────┼────────────┐
              │            │            │
              ▼            ▼            ▼
           Real Image   Wrong Image   Fake Image

🎯 Generator Objective

The generator attempts to fool the discriminator.

text

Text + Noise
     │
     ▼
Generator
     │
     ▼
Fake Face
     │
     ▼
Discriminator
     │
     ▼
Real?

The generator loss is calculated using:

python

nn.BCELoss()

and attempts to make the discriminator classify generated images as real.

🛡️ Discriminator Objective

The discriminator receives three types of examples:

1. Real Image + Correct Text

Expected:

text

2. Wrong Image + Text

Expected:

text

3. Generated Image + Text

Expected:

text

Therefore:

text

Discriminator Loss
      │
      ├── Real Loss
      ├── Wrong Pair Loss
      └── Fake Loss

📈 Experiment Tracking

The project uses:

Weights & Biases (W&B) for experiment tracking.

python

wandb.init(
    project="text-to-face",
    name="n-sagan"
)

The following metrics are tracked:

text

Generator Loss
Discriminator Loss
Generated Images

Generated images are logged after each configured epoch.

🖼️ Visualization

The project visualizes generated images using:

text

Matplotlib
+
Torchvision

Generated images can be arranged into grids:

python

torchvision.utils.make_grid(
    output,
    normalize=True
)

This makes it possible to monitor image quality during training.

🔬 Model Comparison

The project contains two approaches:

Feature	Baseline Generator	Attention GAN
Text Conditioning	✓	✓
Sentence-BERT	✓	✓
Random Noise	✓	✓
ConvTranspose	✓	✓
Conditional Generation	✓	✓
Discriminator	—	✓
Self-Attention	—	✓
Spectral Normalization	—	✓
Wrong Image Pairs	—	✓
W&B Tracking	—	✓
GAN Training	Partial	✓
Output	Face	Face

🔄 Overall Architecture

text

                         Text Description
                                │
                                ▼
                      Sentence-BERT
                       all-mpnet-base-v2
                                │
                                ▼
                         768-D Embedding
                                │
                    ┌───────────┴───────────┐
                    │                       │
                    ▼                       ▼
              Baseline Model          Attention GAN
                    │                       │
                    │                  Text Projection
                    │                       │
                    │                       ▼
                    │                  Random Noise
                    │                       │
                    │                       ▼
                    │                  Generator
                    │                       │
                    │              ┌────────┴────────┐
                    │              │                 │
                    │              ▼                 ▼
                    │        Self-Attention   Spectral Norm
                    │              │
                    │              ▼
                    │         Generated Face
                    │              │
                    │              ▼
                    │         Discriminator
                    │              ▲
                    │              │
                    │         Real / Wrong
                    │            Images
                    │
                    ▼
              Generated Face

🛠️ Technologies

Deep Learning

—PyTorch
—Torchvision
—PyTorch Neural Networks

Natural Language Processing

—Sentence Transformers
—all-mpnet-base-v2
—Text Embeddings

Computer Vision

—OpenCV
—PIL
—Torchvision
—Matplotlib

Machine Learning

—NumPy
—Pandas

GAN / Generative AI

—Conditional GAN
—Self-Attention
—Spectral Normalization
—Transposed Convolution

Experiment Tracking

—Weights & Biases

Dataset

—CelebA

📦 Installation

Clone the repository:

bash

git clone https://github.com/kad99kev/FGTD.git

Move into the project directory:

bash

cd FGTD

Install dependencies:

bash

pip install -r requirements.txt

Install the main deep-learning dependencies if required:

bash

pip install torch torchvision

Install Sentence Transformers:

bash

pip install sentence-transformers

Install experiment tracking:

bash

pip install wandb

🚀 Running the Project

1. Load the Text Encoder

python

from sentence_transformers import SentenceTransformer

sentence_encoder = SentenceTransformer(
    "all-mpnet-base-v2"
)

2. Generate an Image from Text

Prepare a text description:

python

text = [
    "The female has high cheekbones and black hair. "
    "She is young and smiling."
]

Convert it into an embedding:

text

Text
 ↓
Sentence-BERT
 ↓
768-D Embedding

Generate a random latent vector:

python

noise = torch.randn(
    1,
    100
)

Then pass both into the generator:

python

image = generator(
    noise,
    text_embeddings
)

🧪 Example Prompts

Female Face

text

The female has pretty high cheekbones and an oval face.
Her hair is black.
She has arched eyebrows and a pointy nose.
She is smiling and looks young.
She is wearing earrings and lipstick.

Male Face

text

The man is young and attractive.
He has brown straight hair and a pointy nose.
He is smiling and wearing a necktie.

Different Facial Attributes

text

The man has a double chin and high cheekbones.
He has black hair and big lips.
He looks young.

📁 Suggested Repository Structure

text

Text-to-Face-GAN/
│
├── README.md
├── requirements.txt
│
├── notebooks/
│   ├── baseline_text_to_face.ipynb
│   └── attention_text_to_face_gan.ipynb
│
├── models/
│   ├── generator.py
│   ├── discriminator.py
│   ├── attention.py
│   └── spectral_norm.py
│
├── text_encoder/
│   └── sentence_encoder.py
│
├── dataset/
│   ├── text_5_descr_celeba.csv
│   └── list_attr_celeba.csv
│
├── checkpoints/
│   └── generator_50k.pth
│
├── outputs/
│   └── generated_faces/
│
└── results/
    └── wandb/

🧠 Key Concepts Demonstrated

Natural Language Processing

—Sentence Embeddings
—Sentence-BERT
—Semantic Representation
—Text Conditioning

Computer Vision

—Face Generation
—Image Preprocessing
—Image Normalization
—Image Visualization

Deep Learning

—PyTorch
—CNNs
—Transposed Convolution
—Batch Normalization
—ReLU
—Tanh

Generative AI

—GANs
—Conditional GANs
—Text-to-Image Generation
—Latent Noise
—Generator / Discriminator Training

Advanced GAN Techniques

—Self-Attention
—Spectral Normalization
—Conditional Discrimination
—Wrong Image-Text Pairing

🔍 Important Design Decisions

Why Sentence-BERT?

Instead of treating text as individual words, Sentence-BERT provides a semantic representation of the entire description.

text

Natural Language
      ↓
Semantic Embedding
      ↓
768-D Vector
      ↓
Generator Conditioning

This allows descriptions containing multiple facial attributes to be represented in a compact vector.

Why Conditional GAN?

A normal GAN learns:

text

Random Noise → Image

This project instead learns:

text

Random Noise + Text → Image

Therefore, the generated image can be influenced by the supplied description.

Why Self-Attention?

Convolutional layers are excellent at learning local patterns, while Self-Attention helps the network model relationships between distant regions of an image.

This can be useful when generating coherent facial structures.

Why Spectral Normalization?

GAN training can be unstable.

Spectral Normalization helps constrain the network's weight matrices and can improve training stability, particularly in the discriminator.

📊 Expected Workflow

text

              CelebA Dataset
                    │
                    ▼
            Image + Caption
                    │
          ┌─────────┴─────────┐
          │                   │
          ▼                   ▼
       Image              Text
          │                   │
          │                   ▼
          │             Sentence-BERT
          │                   │
          │                   ▼
          │                768-D
          │                   │
          │                   ▼
          │              Projection
          │                   │
          │                   ▼
          │                Text Code
          │                   │
          └──────────┬────────┘
                     ▼
                  GAN
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
      Generator           Discriminator
          │                     │
          ▼                     │
      Fake Face ────────────────┤
                                │
      Real Face ────────────────┤
                                │
      Wrong Face ───────────────┤
                                ▼
                         Real / Fake

⚠️ Implementation Notes

The project is primarily an experimental research implementation.

Several aspects can be improved for a cleaner production/research implementation:

—Separate training and inference scripts.
—Add explicit validation/testing datasets.
—Save both Generator and Discriminator checkpoints.
—Add automatic checkpoint recovery.
—Track additional GAN metrics.
—Evaluate generated image quality quantitatively.
—Use a dedicated configuration file.
—Avoid hard-coded CUDA calls and use the configured device consistently.
—Add reproducible random seeds.
—Add automated experiment logging.

For example, instead of:

python

generator.cuda()

a more portable approach is:

python

generator.to(cfg.device)

This allows the project to run on either GPU or CPU.

🚀 Future Improvements

Possible extensions include:

—Increase image resolution to 256×256.
—Experiment with larger text encoders.
—Use CLIP-based text-image alignment.
—Add perceptual loss.
—Add text-image similarity metrics.
—Add FID evaluation.
—Add Inception Score.
—Improve caption diversity.
—Experiment with different GAN architectures.
—Compare Self-Attention vs standard convolution.
—Implement progressive image generation.
—Add mixed-precision training.
—Add distributed training.
—Improve dataset balancing.
—Add automated checkpointing.
—Build a web interface for text-to-face generation.

📚 Learning Outcomes

This project provides practical experience with:

—Text-to-image generation
—Conditional GANs
—PyTorch
—Sentence Transformers
—BERT-based embeddings
—CNN architectures
—Transposed convolution
—GAN optimization
—Generator/Discriminator training
—Self-Attention
—Spectral Normalization
—CelebA preprocessing
—Image-text conditioning
—Experiment tracking with W&B

🔑 Keywords

text

Text-to-Image
Text-to-Face
Face Generation
Generative AI
GAN
Conditional GAN
cGAN
PyTorch
Sentence-BERT
Sentence Transformers
all-mpnet-base-v2
CelebA
Computer Vision
Deep Learning
Self-Attention
Spectral Normalization
Image Generation
Natural Language Processing
Multimodal AI
Generative Models

📜 Disclaimer

This project is intended for educational, research, and experimental purposes.

Generated faces are synthetic outputs produced by a machine-learning model and should not be interpreted as photographs or evidence of real individuals.

👨‍💻 Project Summary

Text-to-Face Generation with BERT & Conditional GANs is a multimodal generative AI project that connects natural-language descriptions with facial image generation.

The project progresses from a baseline text-conditioned generator to an advanced GAN architecture incorporating Sentence-BERT embeddings, Self-Attention, Spectral Normalization, and a conditional Discriminator.

text

Text
 ↓
Sentence-BERT
 ↓
768-D Embedding
 ↓
Text Conditioning
 +
Random Noise
 ↓
Generator
 ↓
Generated Face
 ↓
Conditional Discriminator
 ↓
GAN Training

The project demonstrates how NLP and Computer Vision can be combined into a single generative AI pipeline for text-guided face synthesis.