Shuu12121/CodeEncoderDecoderModel-Ghost-large

1---2license: apache-2.03language:4- en5- multilingual6tags:7- code-to-docstring8- code-summarization9- code-documentation10- encoder-decoder11- code12- python13- java14- transformers15- huggingface16- modernbert17- gpt218base_model:19- Shuu12121/CodeModernBERT-Ghost20- openai-community/gpt2-large21pipeline_tag: text-generation22---23 24#　CodeEncoderDecoderModel-Ghost-large👻25 26A multilingual encoder-decoder model for generating **docstrings from code snippets**.  27It is based on a custom BERT-style encoder pretrained on source code (`CodeModernBERT-Ghost`) and a large-scale decoder model (`GPT2-large`).28 29## 🏗️ Model Architecture30 31- **Encoder:** [`Shuu12121/CodeModernBERT-Ghost`](https://huggingface.co/Shuu12121/CodeModernBERT-Ghost)  32- **Decoder:** [`openai-community/gpt2-large`](https://huggingface.co/openai-community/gpt2-large)  33- Connected via HuggingFace's `EncoderDecoderModel` with cross-attention.34 35## 🎯 Intended Use36 37- Generating docstrings (documentation comments) for functions or methods in multiple languages.38- Summarizing code for educational or review purposes.39- Assisting in automated documentation generation pipelines.40 41Supported languages (code input):42- Python43- Java44 45## 📦 How to Use46 47```python48from transformers import AutoTokenizer, EncoderDecoderModel49import torch50 51model = EncoderDecoderModel.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large").to("cuda")52encoder_tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large", subfolder="encoder_tokenizer")53decoder_tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large", subfolder="decoder_tokenizer")54 55if decoder_tokenizer.pad_token is None:56    decoder_tokenizer.pad_token = decoder_tokenizer.eos_token57 58code = '''59def greet(name):60    return f"Hello, {name}!"61'''62 63inputs = encoder_tokenizer(code, return_tensors="pt", truncation=True, padding=True, max_length=2048).to("cuda")64outputs = model.generate(65    input_ids=inputs.input_ids,66    attention_mask=inputs.attention_mask,67    max_length=256,68    num_beams=5,69    early_stopping=True,70    decoder_start_token_id=model.config.decoder_start_token_id,71    eos_token_id=model.config.eos_token_id,72    pad_token_id=model.config.pad_token_id,73    no_repeat_ngram_size=274)75 76docstring = decoder_tokenizer.decode(outputs[0], skip_special_tokens=True)77print(docstring)78```79 80## 🧪 Training Details81 82- **Task:** Code-to-docstring generation83- **Dataset:** [CodeXGLUE: Code-to-Text](https://github.com/microsoft/CodeXGLUE) – using subsets of Python, Java, JavaScript, Go, Ruby, PHP84- **Loss:** Cross-entropy loss over tokenized docstrings85- **Max input length:** 2048 (encoder), max output length: 256 (decoder)86- **Decoder modifications:** Adapted GPT2-large with padding and cross-attention87 88## ⚠️ Limitations & Risks89 901. **Generated documentation may be inaccurate, incomplete, or misleading**. Always review generated docstrings manually.912. **Formatting may not follow specific standards** (e.g., Google/Numpy style in Python or full Javadoc).923. **Limited context:** Only considers single-function input; lacks broader project-level understanding.934. **Language variance:** Performance may differ depending on the programming language due to data distribution.945. **⚠️ Decoder risks (GPT2-large):**  95   GPT-2 models are known to sometimes generate inappropriate, offensive, or biased outputs, depending on the prompt.  96   Although this model is fine-tuned on technical datasets (code-docstring pairs), due to inherited properties from `gpt2-large`, similar risks **may still be present** in edge cases. Please exercise caution, especially when using the model in public or educational settings.97 98## 📄 License99 100Apache-2.0  101Model weights and tokenizer artifacts are released under the same license. You are free to use, modify, and redistribute with attribution.