Corp-o-Rate-Community/entity-references

Entity References Database

A comprehensive entity database for organizations, people, roles, and locations with embedding-based semantic search. Built from authoritative sources (GLEIF, SEC, Companies House, Wikidata) for entity linking and named entity disambiguation.

Dataset Description

• —Repository: Corp-o-Rate-Community/entity-references
• —Source Code: corp-o-rate/corp-entity-db
• —Point of Contact: Corp-o-Rate-Community

Dataset Summary

This dataset provides fast lookup and qualification of named entities using vector similarity search. It stores records from authoritative global sources with embeddings generated by google/embeddinggemma-300m (768 dimensions), stored in USearch HNSW indexes for sub-millisecond approximate nearest neighbor search.

Key Features:

• —9.7M+ organization records from GLEIF, SEC Edgar, Companies House, and Wikidata
• —63M+ people records including executives, politicians, athletes, artists, and more
• —139K+ roles and 25K+ locations with hierarchical relationships
• —USearch HNSW indexes for sub-millisecond vector search (embeddings never stored in SQLite)
• —Canonical linking across sources (same entity from multiple sources linked)
• —Three-tier people search achieving 100% acc@1 on 280 queries across 12 person types

Supported Tasks

• —Entity Linking: Match extracted entity mentions to canonical database records
• —Named Entity Disambiguation: Distinguish between entities with similar names
• —Knowledge Base Population: Enrich extracted entities with identifiers and metadata

Languages

English (en)

Dataset Structure

Schema (v5)

The database uses SQLite with normalized FK references (INTEGER FKs replace TEXT enums). All embeddings exist only in USearch HNSW indexes, never in SQLite.

Organizations Table

People Table

Roles Table

Locations Table

USearch HNSW Indexes

All embeddings are generated on-the-fly during index building and stored only in versioned USearch HNSW index files:

The composite people index only includes people with org associations. Name, role, and org are embedded separately, independently L2-normalized, weighted (name=8, role=1, org=4), and concatenated into a 768-dim vector for AND-style matching.

Metadata Table

Contains schema_version = 5 for version detection.

Enum Lookup Tables

Data Splits

The lite version + USearch indexes is recommended for most use cases.

Dataset Creation

Source Data

Organizations

People

Embedding Model

Canonicalization

Records are linked across sources based on:

Organizations:

1. 1.Same LEI (globally unique)
2. 2.Same ticker symbol
3. 3.Same CIK
4. 4.Same normalized name + region

People:

1. 1.Same Wikidata QID
2. 2.Same normalized name + same organization
3. 3.Same normalized name + overlapping date ranges

Source priority: wikidata > secedgar > companieshouse

Usage

Installation

pip install corp-entity-db

Download

# Download lite version + USearch indexes (recommended)
corp-entity-db download

# Download full version + USearch indexes
corp-entity-db download --full

Storage location: ~/.cache/corp-extractor/entities-v5.db (or entities-v5-lite.db for lite)

Search

# Search organizations
corp-entity-db search "Microsoft"

# Search people (composite embedding + name fallback + identity fallback)
corp-entity-db search-people "Tim Cook" --role CEO --org Apple

# Search roles
corp-entity-db search-roles "CEO"

# Search locations
corp-entity-db search-locations "California"

# Check database status
corp-entity-db status

Python API

from corp_entity_db import OrganizationDatabase, CompanyEmbedder, get_database_path

# Search organizations
db = OrganizationDatabase(get_database_path())
embedder = CompanyEmbedder()
matches = db.search(embedder.embed("Microsoft"), top_k=10)
for record, score in matches:
    print(f"{record.name} ({record.entity_type}) - score: {score:.3f}")

# Search people (composite embeddings + name fallback + identity fallback)
from corp_entity_db import PersonDatabase, get_person_database
person_db = get_person_database()
query_emb = embedder.embed_composite_person("Tim Cook", role="CEO", org="Apple")
matches = person_db.search(
    query_emb, top_k=5, query_name="Tim Cook",
    embedder=embedder, query_role="CEO", query_org="Apple",
)

Technical Details

Vector Search Performance

Using USearch HNSW indexes with int8 quantization:

Similarity Thresholds

Building from Source

# Import data sources
corp-entity-db import-gleif --download
corp-entity-db import-sec --download
corp-entity-db import-companies-house
corp-entity-db import-people --all
corp-entity-db import-wikidata-dump --download --limit 50000

# Post-import: build USearch indexes, VACUUM
corp-entity-db post-import

# Link equivalent records
corp-entity-db canonicalize

# Create lite version for deployment
corp-entity-db create-lite ~/.cache/corp-extractor/entities-v5.db

Wikidata Dump Import (Recommended for Large Imports)

# Download and import from Wikidata dump (~100GB)
corp-entity-db import-wikidata-dump --download --limit 50000

# Import only people
corp-entity-db import-wikidata-dump --download --people --no-orgs

# Import only locations
corp-entity-db import-wikidata-dump --dump dump.json.bz2 --locations --no-people --no-orgs

Build extras (optional):

pip install "corp-entity-db[build]"  # Adds orjson + indexed_bzip2

The dump import uses a 3-thread parallel pipeline (reader → embedder → writer) and creates multiple records per person (one per position+org combination). Canonicalization runs automatically at the end. Supports .bz2, .zst, and .gz compressed dumps.

Considerations for Using the Data

Social Impact

This dataset enables entity linking for NLP applications. Users should be aware that:

• —Organization and people records may be incomplete or outdated
• —Historic people (deceased) are included with death_date field
• —Not all notable entities are covered

Biases

• —Coverage is weighted toward English-speaking countries (US, UK) due to source availability
• —Wikidata coverage depends on Wikipedia notability criteria
• —SEC and Companies House data is limited to their respective jurisdictions

Limitations

• —Embedding similarity is not perfect for entity disambiguation
• —Updates require re-importing from source data

License

Apache 2.0

Citation

If you use this dataset, please cite:

@dataset{entity_references_2025,
  title = {Entity References Database},
  author = {Corp-o-Rate-Community},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/Corp-o-Rate-Community/entity-references}
}

Dataset Card Authors

Corp-o-Rate-Community

Dataset Card Contact

Open an issue on the GitHub repository for questions or feedback.