LetsChurch/bible-embeddings

Bible Embeddings

A comprehensive tool for generating and evaluating Bible verse embeddings using various state-of-the-art embedding models. This project supports both commercial APIs (OpenAI, Google Gemini, Voyage AI) and open-source models (HuggingFace sentence-transformers) for semantic search across biblical texts.

Setup

This project is managed with uv. Make sure you have uv installed, then set up the project:

# Install dependencies
uv sync

# Install specific provider dependencies
uv sync --extra openai      # For OpenAI models
uv sync --extra gemini      # For Google Gemini models
uv sync --extra voyage      # For Voyage AI models
uv sync --extra bedrock     # For Amazon Bedrock models
uv sync --extra all         # Install all provider dependencies

API Keys

Set up environment variables for the providers you want to use:

# OpenAI
export OPENAI_API_KEY="your-openai-api-key"

# Google Gemini
export GOOGLE_API_KEY="your-google-api-key"

# Voyage AI
export VOYAGE_API_KEY="your-voyage-api-key"


# Amazon Bedrock (AWS credentials)
export AWS_ACCESS_KEY_ID="your-aws-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-aws-secret-access-key"
export AWS_REGION="us-east-1"  # Optional, defaults to us-east-1

# HuggingFace (for private models or increased rate limits)
export HF_TOKEN="your-huggingface-token"

Note: The HuggingFace token is optional for most public models but required for:

• —Private or gated models (like some variants of EmbeddingGemma)
• —Avoiding rate limits when downloading models
• —Models that require authentication

You can get a HuggingFace token from https://huggingface.co/settings/tokens.

AWS Bedrock Setup: To use Amazon Nova embedding models, you need:

1. 1.AWS Account with access to Amazon Bedrock
2. 2.IAM Permissions - Your AWS user/role needs the following permissions:
3. 3.bedrock:InvokeModel for the Nova embedding model
4. 4.Model Access - Request access to the Nova model in the Bedrock Console
5. 5.Navigate to: Bedrock → Model access → Manage model access
6. 6.Enable: Amazon Nova multimodal embeddings
7. 7.Credentials - Configure AWS credentials via:
8. 8.Environment variables (shown above), or
9. 9.AWS CLI (aws configure), or
10. 10.IAM role (if running on AWS infrastructure)
11. 11.Region - Nova is available in: us-east-1, us-west-2, and other regions (check AWS docs for latest)

Usage

The tool provides several modes of operation:

Generate Embeddings

Generate embeddings for Bible verses using your chosen model:

# Interactive mode - prompts for translation and model selection
uv run main.py embed

# Specify translation and model directly
uv run main.py embed --translation bsb --model "BAAI/bge-large-en"

# Skip verses that already have embeddings
uv run main.py embed --translation bsb --model "text-embedding-3-small" --skip-existing

# Customize batch size for processing
uv run main.py embed --translation bsb --model "sentence-transformers/all-MiniLM-L6-v2" --batch-size 50

# BGE-M3 dense mode (default - uses FlagEmbedding library
uv run main.py embed --translation bsb --model "BAAI/bge-m3"
uv run main.py embed --translation bsb --model "BAAI/bge-m3:dense"

# BGE-M3 hybrid mode (combines dense + sparse for hybrid search)
uv run main.py embed --translation bsb --model "BAAI/bge-m3:hybrid"

# Amazon Nova with different dimensions (uses Matryoshka Representation Learning)
uv run main.py embed --translation bsb --model "amazon.nova-2-multimodal-embeddings-v1:0:3072"  # 3072 dims (default)
uv run main.py embed --translation bsb --model "amazon.nova-2-multimodal-embeddings-v1:0:1024"  # 1024 dims
uv run main.py embed --translation bsb --model "amazon.nova-2-multimodal-embeddings-v1:0:256"   # 256 dims
uv run main.py embed --translation bsb --model "amazon.nova-2-multimodal-embeddings-v1:0:128"   # 128 dims

Search Verses

Search for Bible verses using natural language queries:

# Interactive search mode
uv run main.py query

# Disable HNSW optimization for exact brute-force search
uv run main.py query --no-hnsw

Batch Evaluation

Run batch queries for evaluation and benchmarking:

# Run batch queries from queries.yaml file
uv run main.py batch --translation bsb --model "text-embedding-3-large"

# Specify custom files and concurrency
uv run main.py batch --queries-file custom_queries.yaml --results-file results.csv --concurrency 10

# Disable HNSW optimization for exact results
uv run main.py batch --translation bsb --model "BAAI/bge-large-en" --no-hnsw

# BGE-M3 evaluation with different modes
uv run main.py batch --translation bsb --model "BAAI/bge-m3:dense"
uv run main.py batch --translation bsb --model "BAAI/bge-m3:hybrid"

Generate Reports

Generate markdown reports from evaluation results:

# Update README.md with latest results
uv run main.py report --results-file results.csv

# Generate custom report with limited query examples
uv run main.py report --results-file results.csv --output-file evaluation_report.md --max-queries 5

After embedding, you will see a directory structure with JSON files organized like this:

embeddings
├── huggingface
│   └── BAAI-bge-large-en
│       ├── Genesis
│       │   ├── 001.json
│       │   ├── 002.json
│       │   ├── 003.json
│       │   └── etc
│       ├── Exodus
│       │   ├── 001.json
│       │   ├── 002.json
│       │   ├── 003.json
│       │   └── etc
│       └── Leviticus
│           ├── 001.json
│           ├── 002.json
│           ├── 003.json
│           └── etc
├── openai
│   ├── text-embedding-3-large
│   ├── text-embedding-3-small
│   └── text-embedding-ada-002
├── google-gemini
│   ├── text-embedding-004
│   └── gemini-embedding-2-preview
├── voyage
│   └── voyage-3
└── amazon-bedrock
    ├── amazon.nova-2-multimodal-embeddings-v1:0:128
    ├── amazon.nova-2-multimodal-embeddings-v1:0:256
    ├── amazon.nova-2-multimodal-embeddings-v1:0:1024
    └── amazon.nova-2-multimodal-embeddings-v1:0:3072

Available Models

The tool supports embedding models from multiple providers:

Commercial APIs

• —OpenAI: text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002
• —Google Gemini: text-embedding-004, gemini-embedding-2-preview, gemini-embedding-2-preview:768, gemini-embedding-2-preview:1536, gemini-embedding-2-preview:3072
• —Voyage AI: voyage-3
• —Amazon Bedrock: amazon.nova-2-multimodal-embeddings-v1:0:128, amazon.nova-2-multimodal-embeddings-v1:0:256, amazon.nova-2-multimodal-embeddings-v1:0:1024, amazon.nova-2-multimodal-embeddings-v1:0:3072

Open Source (HuggingFace)

• —ModernBERT: answerdotai/ModernBERT-base, answerdotai/ModernBERT-large
• —BGE: BAAI/bge-large-en, BAAI/bge-base-en, BAAI/bge-small-en, BAAI/bge-m3 (use :dense or :hybrid suffix)
• —Google EmbeddingGemma: google/embeddinggemma-300m
• —IBM Granite: ibm-granite/granite-embedding-30m-english, ibm-granite/granite-embedding-125m-english, ibm-granite/granite-embedding-107m-multilingual, ibm-granite/granite-embedding-278m-multilingual
• —INF Retriever: infly/inf-retriever-v1
• —E5: intfloat/e5-large-v2, intfloat/e5-base-v2, intfloat/e5-small-v2
• —Jina: jinaai/jina-embeddings-v4
• —Nomic: nomic-ai/nomic-embed-text-v1.5
• —NVIDIA: nvidia/NV-Embed-V2
• —Perplexity: perplexity-ai/pplx-embed-v1-0.6b, perplexity-ai/pplx-embed-v1-4b, perplexity-ai/pplx-embed-context-v1-0.6b, perplexity-ai/pplx-embed-context-v1-4b
• —Qwen: Qwen/Qwen3-Embedding-0.6B, Qwen/Qwen3-Embedding-4B, Qwen/Qwen3-Embedding-8B
• —Sentence Transformers: sentence-transformers/all-MiniLM-L6-v2
• —Salesforce: Salesforce/SFR-Embedding-Mistral
• —Snowflake: Snowflake/snowflake-arctic-embed-l-v2.0
• —GTE: thenlper/gte-large, thenlper/gte-base, thenlper/gte-small

Features

• —Multiple Providers: Support for OpenAI, Google Gemini, Voyage AI, Amazon Bedrock, and 18+ HuggingFace models
• —Batch Processing: Efficient batch embedding generation with configurable batch sizes
• —FAISS Integration: Fast approximate nearest neighbor search using HNSW indexing
• —Evaluation Framework: Comprehensive batch query evaluation with accuracy scoring
• —Resume Capability: Skip already processed verses with --skip-existing
• —Flexible Output: JSON embeddings organized by provider/model/book/chapter
• —Interactive Search: Real-time verse search with natural language queries
• —Report Generation: Automated markdown report generation from evaluation results

File Structure

• —main.py - Main application with CLI interface
• —text/ - Bible text files in JSON format (e.g., bsb.json)
• —embeddings/ - Generated embeddings organized by provider/model
• —queries.yaml - Evaluation queries and expected results
• —results.csv - Batch evaluation results
• —pyproject.toml - Project dependencies and configuration

Model Performance

For high-level information about these models, see the MTEB Leaderboard.

• —answerdotai/ModernBERT-base
• —768 dimensions
• —BSB: 681.349 MB
• —answerdotai/ModernBERT-large
• —1024 dimensions
• —BSB: 907.059 MB
• —BAAI/bge-large-en
• —1024 dimensions
• —BSB: 899.769 MB
• —BAAI/bge-base-en
• —768 dimensions
• —BSB: 674.226 MB
• —BAAI/bge-small-en
• —384 dimensions
• —BSB: 338.351 MB
• —BAAI/bge-m3
• —1024 dimensions (dense mode)
• —Multi-functionality: supports dense retrieval and hybrid search
• —Multilingual support (100+ languages)
• —Max input tokens: 8,192
• —No instruction needed for queries
• —google/embeddinggemma-300m
• —768 dimensions (also supports 512, 256, 128 via MRL)
• —BSB: 645.719 MB
• —300M parameters, multilingual support (100+ languages)
• —ibm-granite/granite-embedding-30m-english
• —384 dimensions
• —BSB: 337.962 MB
• —ibm-granite/granite-embedding-125m-english
• —768 dimensions
• —BSB: 677.83 MB
• —ibm-granite/granite-embedding-107m-multilingual
• —384 dimensions
• —BSB: 334.367 MB
• —ibm-granite/granite-embedding-278m-multilingual
• —768 dimensions
• —BSB: 642.516 MB
• —infly/inf-retriever-v1
• —3584 dimensions
• —BSB: 3000 MB
• —7B parameters, optimized for Chinese & English retrieval
• —Max input tokens: 32,768
• —intfloat/e5-large-v2
• —1024 dimensions
• —BSB: 898.283 MB
• —intfloat/e5-base-v2
• —768 dimensions
• —BSB: 672.98 MB
• —intfloat/e5-small-v2
• —384 dimensions
• —BSB: 336.522 MB
• —jinaai/jina-embeddings-v4
• —2048 dimensions (supports Matryoshka dimensions: 128, 256, 512, 1024, 2048)
• —BSB: 2100 MB
• —3.75B parameters, multimodal multilingual retrieval
• —Max input tokens: 32,768
• —nvidia/NV-Embed-V2
• —1024 dimensions
• —BSB: 3465.70 MB
• —sentence-transformers/all-MiniLM-L6-v2
• —384 dimensions
• —BSB: 336.741 MB
• —Salesforce/SFR-Embedding-Mistral
• —4096 dimensions
• —BSB: 3632.75 MB
• —Snowflake/snowflake-arctic-embed-l-v2.0
• —1024 dimensions (supports compression to 128 bytes via MRL)
• —568M parameters, multilingual support (74 languages)
• —Max input tokens: 8,192
• —text-embedding-004
• —768 dimensions
• —BSB: 465.544 MB
• —gemini-embedding-2-preview (Gemini Embedding 2)
• —768, 1536, or 3072 dimensions (default: 3072)
• —Max input tokens: 8,192
• —text-embedding-3-large
• —3072 dimensions
• —BSB: 2717.202 MB
• —text-embedding-3-small
• —1536 dimensions
• —BSB: 1352.259 MB
• —text-embedding-ada-002
• —1536 dimensions
• —BSB: 1359.56 MB
• —amazon.nova-2-multimodal-embeddings-v1:0
• —Supports 4 dimensions via Matryoshka Representation Learning (MRL):
• —3072 dimensions (default) - BSB: ~2717 MB
• —1024 dimensions - BSB: ~906 MB
• —256 dimensions - BSB: ~226 MB
• —128 dimensions - BSB: ~113 MB
• —First multimodal embedding model supporting text, images, documents, video, and audio
• —Multilingual support (200 languages)
• —Max input tokens: 8,192
• —API-based via Amazon Bedrock
• —thenlper/gte-large
• —1024 dimensions
• —BSB: 899.583 MB
• —thenlper/gte-base
• —768 dimensions
• —BSB: 674.492 MB
• —thenlper/gte-small
• —384 dimensions
• —BSB: 338.211 MB
• —voyage-3
• —1024 dimensions
• —BSB: 899.583 MB
• —perplexity-ai/pplx-embed-v1-0.6b / perplexity-ai/pplx-embed-context-v1-0.6b
• —1024 dimensions
• —0.6B parameters, 32K context window
• —pplx-embed-v1 for queries/independent text; pplx-embed-context-v1 for RAG document chunks
• —perplexity-ai/pplx-embed-v1-4b / perplexity-ai/pplx-embed-context-v1-4b
• —2560 dimensions
• —4B parameters, 32K context window

Results

✅ denotes accurate result.

openai/text-embedding-3-large (BSB)

Accuracy: 89.3% (1426/1596 points across 532 queries)

... and 522 more queries

huggingface/infly-inf-retriever-v1 (BSB)

Accuracy: 84.2% (1344/1596 points across 532 queries)

... and 522 more queries

huggingface/Qwen-Qwen3-Embedding-8B (BSB)

Accuracy: 82.8% (1322/1596 points across 532 queries)

... and 522 more queries

openai/text-embedding-3-small (BSB)

Accuracy: 81.0% (1293/1596 points across 532 queries)

... and 522 more queries

google-gemini/text-embedding-004 (BSB)

Accuracy: 80.3% (1282/1596 points across 532 queries)

... and 522 more queries

openai/text-embedding-ada-002 (BSB)

Accuracy: 78.3% (1250/1596 points across 532 queries)

... and 522 more queries

huggingface/jinaai-jina-embeddings-v4 (BSB)

Accuracy: 74.5% (1189/1596 points across 532 queries)

... and 522 more queries

huggingface/perplexity-ai-pplx-embed-v1-0.6b (BSB)

Accuracy: 74.2% (1184/1596 points across 532 queries)

... and 522 more queries

huggingface/Qwen-Qwen3-Embedding-0.6B (BSB)

Accuracy: 73.9% (1179/1596 points across 532 queries)

... and 522 more queries

huggingface/thenlper-gte-large (BSB)

Accuracy: 73.1% (1166/1596 points across 532 queries)

... and 522 more queries

huggingface/thenlper-gte-base (BSB)

Accuracy: 72.5% (1157/1596 points across 532 queries)

... and 522 more queries

huggingface/Qwen-Qwen3-Embedding-4B (BSB)

Accuracy: 72.4% (1156/1596 points across 532 queries)

... and 522 more queries

huggingface/intfloat-e5-large-v2 (BSB)

Accuracy: 72.3% (1154/1596 points across 532 queries)

... and 522 more queries

huggingface/nomic-ai-nomic-embed-text-v1.5 (BSB)

Accuracy: 70.0% (1117/1596 points across 532 queries)

... and 522 more queries

huggingface/thenlper-gte-small (BSB)

Accuracy: 69.2% (1105/1596 points across 532 queries)

... and 522 more queries

huggingface/intfloat-e5-base-v2 (BSB)

Accuracy: 69.2% (1104/1596 points across 532 queries)

... and 522 more queries

huggingface/Salesforce-SFR-Embedding-Mistral (BSB)

Accuracy: 69.0% (1102/1596 points across 532 queries)

... and 522 more queries

huggingface/ibm-granite-granite-embedding-125m-english (BSB)

Accuracy: 68.5% (1093/1596 points across 532 queries)

... and 522 more queries

google-gemini/gemini-embedding-2-preview (BSB)

Accuracy: 68.1% (1087/1596 points across 532 queries)

... and 522 more queries

huggingface/perplexity-ai-pplx-embed-v1-4b (BSB)

Accuracy: 68.0% (1085/1596 points across 532 queries)

... and 522 more queries

huggingface/Snowflake-snowflake-arctic-embed-l-v2.0 (BSB)

Accuracy: 67.0% (1070/1596 points across 532 queries)

... and 522 more queries

voyage/voyage-3 (BSB)

Accuracy: 67.0% (1070/1596 points across 532 queries)

... and 522 more queries

huggingface/ibm-granite-granite-embedding-30m-english (BSB)

Accuracy: 65.4% (1043/1596 points across 532 queries)

... and 522 more queries

huggingface/ibm-granite-granite-embedding-278m-multilingual (BSB)

Accuracy: 64.7% (1033/1596 points across 532 queries)

... and 522 more queries

huggingface/perplexity-ai-pplx-embed-context-v1-4b (BSB)

Accuracy: 62.7% (1000/1596 points across 532 queries)

... and 522 more queries

huggingface/BAAI-bge-m3-dense (BSB)

Accuracy: 62.2% (992/1596 points across 532 queries)

... and 522 more queries

huggingface/BAAI-bge-m3-hybrid (BSB)

Accuracy: 62.2% (992/1596 points across 532 queries)

... and 522 more queries

huggingface/ibm-granite-granite-embedding-107m-multilingual (BSB)

Accuracy: 61.3% (978/1596 points across 532 queries)

... and 522 more queries

huggingface/intfloat-e5-small-v2 (BSB)

Accuracy: 61.0% (974/1596 points across 532 queries)

... and 522 more queries

huggingface/perplexity-ai-pplx-embed-context-v1-0.6b (BSB)

Accuracy: 60.7% (969/1596 points across 532 queries)

... and 522 more queries

huggingface/sentence-transformers-all-MiniLM-L6-v2 (BSB)

Accuracy: 59.6% (951/1596 points across 532 queries)

... and 522 more queries

huggingface/BAAI-bge-base-en (BSB)

Accuracy: 56.9% (908/1596 points across 532 queries)

... and 522 more queries

huggingface/BAAI-bge-large-en (BSB)

Accuracy: 47.4% (756/1596 points across 532 queries)

... and 522 more queries

huggingface/google-embeddinggemma-300m (BSB)

Accuracy: 46.4% (741/1596 points across 532 queries)

... and 522 more queries

huggingface/BAAI-bge-small-en (BSB)

Accuracy: 43.4% (693/1596 points across 532 queries)

... and 522 more queries

huggingface/answerdotai-ModernBERT-base (BSB)

Accuracy: 7.7% (123/1596 points across 532 queries)

... and 522 more queries

huggingface/answerdotai-ModernBERT-large (BSB)

Accuracy: 7.1% (113/1596 points across 532 queries)

... and 522 more queries

Legend

• —✅ Perfect Match - Expected result appears as #1 result (3 points)
• —⚠️ Good Match - Expected result appears as #2 result (2 points)
• —❌ Poor/No Match - Expected result appears as #3 result (1 point) or not in top 3 (0 points)