Modelpublic

Chunjiang-Intelligence/Socrates-embedding

sourceHugging Faceopenrailupdated 8mo agoView on Hugging Face

11likes74downloads

Model Card

Model Card for Socrates-embedding

Note: This model was stopped during the training process after one epoch because we were unable to afford the cost of AutoDL. The current version shows the result after training with nil for 18,000 steps.

Model Details

Socrates-embedding is a lightweight, high-density text embedding model. Unlike contemporary models that rely on massive parameter counts to brute-force semantic understanding, Socrates-embedding leverages Low-Rank Decay (LoRD) to achieve high-quality vector representations with minimal computational overhead.

This model is part of the Chunjiang Intelligence edge-computing initiative, aiming to bring retrieval-augmented generation (RAG) and semantic search capabilities to consumer-grade hardware.

—Developed by: Chunjiang Intelligence
—Model Type: Dual-Encoder Transformer.
—Language: English, Japanese, German, Chinese

The model was evaluated on the AmazonCounterfactualClassification dataset across multiple languages.

Language	Accuracy
Japanese (ja)	54.83
German (de)	52.57
English (en)	49.70
English-Ext (en-ext)	49.15

To put the model's efficiency into perspective, we compare its single-task score on Japanese classification against the overall MTEB average scores of much larger models. (Our budget is insufficient to cover the bill for the GPU used for the ongoing tests.)

<img src="modelefficiencycomparison.png" width="800"> Figure 1: Our 83M model's score on a single challenging task rivals the average performance of models up to 85x larger.

Clustering performance was evaluated using the V-measure score (multiplied by 100) on the StackExchangeClustering task.

We compared Socrates-embedding against other popular lightweight models (<110M params).

Model	Parameters	Clustering Score (V-measure x 100)
Socrates-embedding	83M	8.92 🏆
`snowflake-arctic-embed-m`	109M	7.25
`KartonBERT-USE-base-v1`	104M	6.93
`jina-embedding-s-en-v1`	35M	6.64
`all-MiniLM-L6-v2`	23M	6.62

—Observation: Our model achieves the highest clustering score in its weight class, demonstrating a superior vector space structure compared to established baselines.

<img src="modelclusteringcomparison.png" width="800"> Figure 2: Leading clustering performance among lightweight embedding models.

Model Architecture

The model utilizes a custom Transformer Encoder architecture optimized for inference latency on Apple MPS and NVIDIA TensorRT backends.

Parameter	Value
`vocab_size`	50295
`hidden_size`	768
`embedding_dim`	512
`n_layer`	12
`n_head`	6
`n_kv_head`	2
`max_seq_len`	512
`pooling`	Mean

Model Size & Efficiency

Metric	Value
Total Parameters	83.23 M
Trainable Parameters	83.23 M
Model File Size	328.99 MB

Environmental Impact & Carbon Footprint

At Chunjiang Intelligence, we are committed to sustainable AGI development.

The training of Socrates-embedding was conducted with extreme energy discipline.

—Hardware: 1 $\times$ NVIDIA RTX 6000
—Training Duration: 1.2 Hours
—Compute Region: Zaozhuang, China
—Total Energy Consumption: ~0.36 kWh

Carbon Emissions

—Estimated CO₂ Emissions: 0.17 kg (equivalent to driving a Tesla for 0.8 miles).

Carbon Offset Strategy (Net Zero Achievement)

To strictly adhere to our carbon-neutral commitment, the following offset measures were implemented during the 1.2-hour training session:

—Optical Energy Conservation: All illumination devices within the laboratory (i.e., the bedroom) were deactivated. The researcher operated solely under the photon emission from the terminal display.
—Biological Metabolism Suppression: The lead researcher voluntarily reduced their respiratory frequency by approximately 15% during the backpropagation phase to minimize biological CO₂ exhalation.
—Thermal Regulation: No air conditioning was used; the ambient temperature was regulated solely by the waste heat generated by the GPU fan and the researcher's anxiety.

Based on these rigorous countermeasures, we certify this model as Carbon Negative by a margin of 0.02 grams.

Intended Use

—Semantic Search: Efficiently indexing personal knowledge bases on local devices.
—RAG Pipelines: Providing vector retrieval for Socrates-Nano generation.
—Edge Deployment: Running on mobile phones, Raspberry Pis, or browser-based WASM environments.

Out-of-Scope Use

—Planetary-Scale Indexing: Please do not use this model to index the entire internet; it's 86M parameters, have some mercy.
—Heating: This model is too efficient to generate significant heat during inference. If you are cold, please buy a heater or run LLaMA-70B instead.