complicat9d/mirea-tl-eda

RTU MIREA Telegram Channel EDA


The dataset contains 22 metrics, describing post user engagement, linguistic units features, readability, AI messages generation score, semantic topic label, and model labelling confidence for the whole available time period at the time of analysis (over 10k text messages).

Overview

The dataset is the result of EDA performed on the data from RTU MIREA Telegram channel, retreived via aiogram. The full code for the data processing pipeline can be found in the GitHub repository.

Preprocessing

The first step of the preprocessing is to get the data needed. Telegram via its API provides the following information about messages of a channel:

• —id (Telegram-specific message id)
• —date (when a message was sent)
• —views
• —forwards
• —replies
• —reactions

Having retreived the data, the messages has been cleaned (removed emojis, hashtags, trailing and extra spaces), then filtered to contain at least a word, and dict-like structures has been flattened out.

Engagement

To access user post engagement, the following metrics are used: views, forwards, replies and reactions. The metrics do not give the full picture of post user engagement by themselves, so an index has been proposed:

$$F_{1k} = \frac{\text{forwards}}{\text{views}} \times 1000$$

$$R_{1k} = \frac{\text{reactions}}{\text{views}} \times 1000$$

$$P_{1k} = \frac{\text{replies}}{\text{views}} \times 1000$$

$$EI = 0.25 \times F{1k} + 0.35 \times R{1k} + 0.40 \times P_{1k}$$

Since the distribution of views is highly skewed to the right and to preserve historicity of the index data with an addition of new elements, it has been decided to abstain from using any kind of normalizations. Multiplication by 1000 ensures interpretability of the index.

Linguistic Units Features

The analysis focused on syntactic complexity, discourse segmentation for further analysis of sentence burstiness, polysyllabic word frequency, and phonological complexity for message readability.

To battle a rather heterogeneous style of writing (inconsistent punctuation marks and indentations), razdel is used to find lengths of sentences, even though sometimes it is inconsistent. pyphen and pymorphy3 are used to find word syllables and word lemmas, and part of speech, respectively.

Readability and Grade Index

To access readability and grade needed to read a message, Fleisch-Kincaid for readability, Fleisch-Kincaid for grade level and LIX indexes adapted for Russian are used:

$$ \begin{aligned} &FKR = (206.835 - 1.52W - 65.14S){\text{clip}(0,100)} \\ &LIX = W + 100L \\ &FKG = (0.5W + 11.8S - 15.59){\text{clip}(0,100)} \\ \\ &\text{where:} \\ &\quad W = \text{words per sentence} \\ &\quad S = \text{syllables per word} \\ &\quad L = \text{long words ratio (words > 6 characters)} \end{aligned} $$

AI Generation Score

To access AI text generation perplexity and sentence burstiness are used. Pexplexity is calculated on rugpt3 large model, it is calculated using the following formula:

$$ \begin{aligned} &\text{Burstiness:} \quad B = \frac{\sigmaL}{\muL} \times 100\% \\ \\ &\text{where:} \\ &\quad \muL = \text{mean sentence length} \\ &\quad \sigmaL = \text{standard deviation of sentence lengths} \end{aligned} $$

Conversion to the index is made the following way:

$$ \begin{aligned} \text{Perplexity Norm:} & \quad p' = \frac{\ln(1+p) - \ln(1+p{\min})}{\ln(1+p{\max}) - \ln(1+p{\min})} \\ \text{Burstiness Norm:} & \quad b' = \frac{b - b{\min}}{b{\max} - b{\min}} \\ \text{AI Score:} & \quad S = 1 - (0.7p' + 0.3b') \end{aligned} $$

It is worth mentioning, since it is really hard to determine sentences boarders, burstiness may give slightly deviated values.

Additionally, perplexity scores are only applicable to autoregressive language models, as they measure how well a model predicts the next token in a sequence. Different model architectures (like encoder-only or encoder-decoder models) would require different evaluation metrics. Therefore, relying solely on perplexity from a single model (rugpt3 large) provides limited robustness and may not generalize well across different AI text generators. The combination with burstiness helps mitigate this limitation by incorporating structural patterns that are model-agnostic.

For determining whether a message may have been generated by an AI, a threshold of 75-percentile of the given ai_score distribution is used.

Topic Classification

To classify the topic of a message, a text classification model with 17 labels is used. It is trained on the labelled dataset of 1k messages from RTU MIREA Telegram channel. The confidence of a given topic is measured by a score value.

Analysis Visualisation

Post Lengths X Engagement Over Time

[image]

It is clear that with the time, posts have become daily and regular, and the sum of post lengths has a positive upwards trend. There is no clear correlation between engagement and post lengths. User post engagement does not show any clear trends; it is rather stagnating.

Linguistics Units Features

[image]

As can be inferred from the plot, the populations of all words are roughly described by nouns and full adjectives. Other parts of speech (verbs, prepositions, conjunctions, adverbials, etc.) do not contribute much to the description of posts semantic meaning.

[image]

In the plot, the top 200 words by frequency can be seen. Only nouns and full adjectives are displayed. The majority of words are domain-specific (РТУ, МИРЭА, студент, проект, Россия, etc.). The only one that may stand out is война (war), but as it is seen in bigrams, it is primarily used in the context of the Great Patriotic War, only supporting the premise of the importance of this event in Russia. Also, despite being launched in March 2025, word MAX also appears in the word cloud; this could be attributed to the footer texts.

[image]

This is the depiction of the top 200 words in bigrams. As it has been said, the biggest clusters of bigrams are primarily domain-specific.

[image]

The analysis of Zipf's curve shows that the corpus of the messages slightly deviates from a natural language. Type-token ratio of roughly 8% shows that the text is highly repetitive, which is one of the reasons why the line is concaved upwards, and hapax legomena percentage of 46% tells us that only half of the words in the corpus are used on a frequent basis. These anomalies could be explained by the domain specificity: Telegram messages, which clearly are not intended to show high vocabulary diversification, and RTU MIREA specific terms.

Readability and Grade Level Indexes

[image]

Due to inability to determine correct sentences boarders, rather short texts, for which the application of these indexes is not the intended purpose, and Russian language specifics, which could have not been accounted for, when reinvetning these indexes for it, the distributions of all indexes are quite skewed: every index shows that roughly 50% of messages are really hard to read and you need a professor level of a degree to understand them.

Text Classification

[image]

The analysis reveals clear patterns in audience engagement across content categories. Rating-related content achieves the highest engagement score of 6.62, indicating that evaluative metrics strongly resonate with the audience. The "other" category, representing approximately 8-9% of all messages that were difficult to classify, achieves the second-highest engagement at 6.01. This significant result suggests that these unlabeled communications contain elements that effectively capture audience interest, revealing a potential opportunity to analyze and formalize these successful but currently uncategorized content patterns.

General announcements and patriotism-themed messages follow with scores around 5.90, demonstrating consistent audience interest in official updates and national narratives. Social content, while being the most frequently communicated category with 1,394 instances, generates only moderate engagement at 5.25 per message. This suggests that social interactions serve as routine communication rather than peak engagement drivers. Educational and academic content falls into a lower engagement tier between 4.23 and 4.04, revealing that despite its institutional importance, such information is perceived as more routine by the audience.

The lowest engagement scores belong to competition announcements (3.59) and weather forecasts (3.04).

An inverse relationship emerges between engagement and content volume: categories with fewer messages often achieve higher per-instance engagement, while high-frequency communications tend toward moderate or lower scores.

AI Score

[image]

The data reveals distinct patterns in AI versus human content generation across different communication categories within institutional channels. International relations content shows the highest AI involvement at 38.3%, followed closely by patriotism-themed messages at 35.6%. This elevated AI usage suggests these topics may rely more on standardized language or require careful diplomatic phrasing that aligns with institutional messaging frameworks.

The middle range of AI participation includes digest summaries at 33.1%, scholarship information at 31.7%, and volunteering content at 27.9%. These categories represent information-dense communications where AI assistance likely helps structure complex information consistently. Science communications and admission-related content both show approximately 27% AI involvement, indicating moderate use of automated support for technical and procedural information.

Notably, the largest volume categories—social content with 1,394 messages and announcements with 1,383 messages—demonstrate relatively low AI percentages at 23.0% and 25.8% respectively. This suggests that interpersonal communications and official notifications retain stronger human authorship, possibly because they require more personal tone or immediate responsiveness.

The categories with the lowest AI involvement reveal interesting patterns. Rating-related content, despite achieving the highest engagement scores in previous analysis, shows only 15.9% AI generation, suggesting that evaluative content benefits from or requires human judgment. The "other" category, representing unclassified messages, maintains very low AI usage at 13.2%, indicating that ambiguous or hybrid communications resist easy automation.

Sports content shows minimal AI involvement at 13.0%, likely because athletic updates require timely, specific reporting that resists templating. Weather forecasts demonstrate the second-lowest AI percentage at 10.3%, possibly because these brief, data-driven messages need little augmentation. Cybersecurity communications show the absolute lowest AI usage at just 9.95%, which may indicate the need for expert human oversight in these specialized communications.

Contact

If you have any suggestions, please contact me through issues.