sagard21/python-code-explainer

1---2tags:3- autotrain4- summarization5language:6- en7widget:8- text: > 9    def preprocess(text: str) -> str:10        text = str(text)11        text = text.replace('\\n', ' ')12        tokenized_text = text.split(' ')13        preprocessed_text = " ".join([token for token in tokenized_text if token])14 15        return preprocessed_text16datasets:17- sagard21/autotrain-data-code-explainer18co2_eq_emissions:19  emissions: 5.39307904512897320license: mit21pipeline_tag: summarization22---23 24# Model Trained Using AutoTrain25 26- Problem type: Summarization27- Model ID: 274558134928- CO2 Emissions (in grams): 5.393129 30# Model Description31 32This model is an attempt to simplify code understanding by generating line by line explanation of a source code. This model was fine-tuned using the Salesforce/codet5-large model. Currently it is trained on a small subset of Python snippets.33 34# Model Usage35 36```py37from transformers import (38    AutoModelForSeq2SeqLM,39    AutoTokenizer,40    AutoConfig,41    pipeline,42)43 44model_name = "sagard21/python-code-explainer"45 46tokenizer = AutoTokenizer.from_pretrained(model_name, padding=True)47 48model = AutoModelForSeq2SeqLM.from_pretrained(model_name)49 50config = AutoConfig.from_pretrained(model_name)51 52model.eval()53 54pipe = pipeline("summarization", model=model_name, config=config, tokenizer=tokenizer)55 56raw_code = """57def preprocess(text: str) -> str:58    text = str(text)59    text = text.replace("\n", " ")60    tokenized_text = text.split(" ")61    preprocessed_text = " ".join([token for token in tokenized_text if token])62 63    return preprocessed_text64"""65 66print(pipe(raw_code)[0]["summary_text"])67 68```69 70## Validation Metrics71 72- Loss: 2.15673- Rouge1: 29.37574- Rouge2: 18.12875- RougeL: 25.44576- RougeLsum: 28.08477- Gen Len: 19.00078