jacky5124/TLDR-TrendingTopics
TLDR: Trending Topics
What is this Space about?
TLDR: Trending Topics is an AI with grounding that summarizes relevant news of trending topics. If you like to read news to keep yourself updated about what is happening, but you find yourself too busy to read full news articles, then this Space is just for you! By simply selecting a country (currently either Canada or United States), you can instantly browse a wide range of news topics trending in that country within six hours, and for each trending topic, you can view an accurate and succinct summary based on the linked news articles given below the summary. Thanks to great understanding of natural language and flexible instruction following, Meta Llama 3.1 Instruct models make such task possible by grounding with data from reliable APIs and prompt engineering to generate output in desired format. With this Space, you can get to know all the trending news in just seconds and save time for what matter to you the most!
Architecture
Backend Repositories
Workflow
The scheduled jobs execute the following to fetch trending topics and generate summaries:
- Call Bing Search API for trending topics in either Canada or United States.
- Call Brave Search API for news articles as snippets with the trending topics as queries.
- Use Meta Llama 3.1 8B Instruct to filter the snippets to keep only the relevant ones.
- Use Meta Llama 3.1 8B Instruct again to summarize the relevant snippets for each trending topic.
- Cache the summaries as well as original links to the news articles for all trending topics.
This Space only accesses the results cached by the scheduled jobs. The results are available until the scheduled jobs run again, which overwrite the cache with new results.
Design Decision
The above workflow is run only periodically. Since there could be around 30 trending topics returned by Bing Search API, to cover every trending topic, Brave Search API must be called 30 times, each of which returns up to 5 news articles as snippets. However, Brave Search API sometimes could return news articles that are irrelevant to the given trending topic, so each returned news article must be checked for relevance, which means Meta Llama 3.1 8B Instruct is called for every pair of trending topic and its queried news article, and there are up to 150 such pairs. Finally, Meta Llama 3.1 8B Instruct is used to produce a summary for every trending topic, and this means another 30 calls. As a complete workflow is both computationally expensive and pricey, to accommodate any number of users, it is made to run by a pre-defined schedule to share the same results as much as possible. This strategy works because the only input of the process is just a country name, and users should see consistent results for the same country name at the same time. The only downside of this strategy is that users are not able to access real-time results, and the results could be up to 6 hours old.
The backend is hosted on Microsoft Azure. To save cost, everything except the cache is run on a serverless basis. Since the scheduled jobs are run only once per six hours, and the average time of a complete workflow is one minute, serverless approach maximizes resource utilization, and Functions on Azure provide the best solution to it. Similarly, Meta Llama 3.1 8B Instruct is also made available as serverless API rather than being provided by dedicated GPU instances, because the generated results are already accurate without fine-tuning the model, so that default weights on the API can be used as it is, and the model is only utilized once per six hours by the scheduled jobs. The cache service and Redis are containerized instances, which means they can be scaled up or down to handle any kind of user traffic with ease. Microsoft Azure provides all these features, and it is easy to monitor every part of the backend running on the cloud.