Alawy21/News_analyzer_Qwen2.5_1.5B_Finetuning

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

Results

<img src="./results.png" alt="Sample Invoice" width="auto"/>

models

This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the News dataset. It achieves the following results on the evaluation set:

• —Loss: 0.2032

Model description

The primary objective of the Qwen2.5-1.5B-Instruct model, which has been fine-tuned, is to automatically extract and summarize critical information from Arabic text inputs, such as news articles, generating structured JSON-like outputs.

Training and evaluation data

Fine-Tuning Dataset: 2001 samples of Arabic technology-related text, used to adapt the model for structured extraction tasks.

Evaluation Dataset: 100 samples of Arabic sports-related text, used to assess performance on a different domain.

Average similarity scores on the evaluation data

• —This similarity measure is applied only to one field from the output JSON, namely the "News_title," and the data belongs to a different domain than the one the model was fine-tuned on.
• —Mean similarity: 0.6871507857739926
• —Note: The average similarity score is quite good, considering that the Qwen model was fine-tuned on a tech dataset. However, since all the test data here is related to sports, testing on data more similar to the tech domain would likely yield even better accuracy.

Training procedure

Since the dataset was not labeled, Llama 4 Scout was employed as a teacher model in a knowledge distillation framework to generate pseudo-labels or guide the training of Qwen2.5-1.5B-Instruct (the student model). Knowledge distillation transfers knowledge from a larger, more capable model (Llama 4 Scout) to a smaller, efficient model (Qwen2.5-1.5B-Instruct).

• —Role of Llama 4 Scout: Teacher Model: Llama 4 Scout, a powerful language model, was used to process the unlabeled 2001 technology samples and generate high-quality structured outputs (e.g., pseudo-labels for story titles, keywords, summaries, categories, and entities). Output Generation: For each input text, Llama 4 Scout produced: Story Title: A concise headline summarizing the main event. Keywords: Relevant terms extracted based on contextual understanding. Summary: A set of key sentences or abstractive summary points. Category: A predicted category (e.g., “technology” for training data). Entities: Identified entities with types (e.g., person, organization), using its advanced NER capabilities.

• —Tool: LLaMA-Factory used for streamlined fine-tuning, supporting LoRA (Low-Rank Adaptation).

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 0.0001
• —trainbatchsize: 1
• —evalbatchsize: 1
• —seed: 42
• —gradientaccumulationsteps: 4
• —totaltrainbatch_size: 4
• —optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
• —lrschedulertype: cosine
• —lrschedulerwarmup_ratio: 0.1
• —num_epochs: 3.0

Training results

Framework versions

• —PEFT 0.15.1
• —Transformers 4.51.3
• —Pytorch 2.6.0+cu124
• —Datasets 3.5.0
• —Tokenizers 0.21.1