Sahil1694/Atlan
0
1�
2 �h� � 3 � � d dl Z d dlZd dlZd dlZd dlmZ d dlmZmZ ddgZ dZ4ddiZ d e d5e
fd �Zd e d6edz fd
�Zedk �r� ed� � e
� � Ze D ]"Z ee� � Ze� e� � �# ed ee� � � �� � g Z ee� � Z e ee� � d� � D ]_\ Z Z ede � de� de � d�� � ee � � Z e r(e d r e� e � � ed� � ej d� � �` ed ee� � � �� � e j! e j" �# e7� � d�� � e$e8dd � � � 5 Z% ej& ee%d d � � � ddd� � n # 1 swxY w Y ed!e9� d"�� � dS # e'$ rZ( ed#e(� �� � Y dZ([(dS dZ([(ww xY wdS )$� N)�
BeautifulSoup)�urljoin�urlparsezhttps://docs.atlan.com/z https://developer.atlan.com/zdata/knowledge_base.jsonz10User-AgentzAtlanSupportCopilotBot/1.0�base_url�returnc � � t d| � d�� � t � � } t j | t �� � }|� � � t
|j d� � }t | � � j }|�11 dd�� � D ]`}|d }t | |� � }t |� � j |k r.|� |�
d � � d12 � � �an1# t j $ r }t d | � d |� �� � Y d
}~nd
}~ww xY wt dt |� � � d�� � |S )zF13 Crawls a starting URL to find all unique, same-domain links.14 z!Crawling to discover links from: z...��headers� html.parser�aT)�hrefr
�#r u 🚨 Error crawling �: Nu15 ✅ Found z unique links.)�print�set�requests�get�HEADERS�raise_for_statusr �contentr �netloc�find_allr �add�split�RequestException�len) r � found_links�response�soup� base_domain�a_tagr
�full_url�es �UC:\Users\Sahil\OneDrive\Desktop\Atlan Project\backend\scripts\build_knowledge_base.py�
get_all_linksr% sh � � 16�17;�h�18;�19;�20;�<�<�<��%�%�K�6��<��'�:�:�:���!�!�#�#�#� �X� -�}�=�=�� �x�(�(�/� ��]�]�3�T�]�2�2� 8� 8�E���=�D� �x��.�.�H� ��!�!�(�K�7�7�������s� 3� 3�A� 6�7�7�7�� 8�� � $� 6� 6� 6�
�4�X�4�4��4�4�5�5�5�5�5�5�5�5�����6���� 21�227�s�;�'�'�237�247�257�8�8�8� �s �CC4 �4D"�D � D"�urlc � � t j | t �� � }|� � � t |j d� � }|� d� � p|� d� � }|r |� dd�� � }| |d�S t d | � d26�� � d S # t j $ r }t d | � d
|� �� � Y d }~d S d }~ww xY w)z�27 Scrapes the main textual content from a single URL.28 Returns a dictionary with the URL and its content, or None on failure.29 r r �main�article� T)� separator�strip)r& r u! ⚠️ No main content found for z , skipping.Nu 🚨 Error scraping r )30r r r r r r �find�get_textr r )r& r r � main_content�textr# s r$ �scrape_page_contentr1 7 s � �31��<��W�5�5�5���!�!�#�#�#� �X� -�}�=�=��32 �y�y��(�(�@�D�I�I�i�,@�,@� � � � �(�(�3�d�(�C�C�D� �4�0�0� 0�
�F�c�F�F�F� G� G� G��4�� � $� � � �
�/�S�/�/�A�/�/�0�0�0��t�t�t�t�t��������s �B B# �B# �#C�2C
�
C�__main__u- 🚀 Starting Knowledge Base Build Process...z033Total unique links to scrape across all sites: � z34--- Scraping URL �/r z ---r u# ✅ Content extracted successfully.u2 35✨ Scraping complete. Total pages with content: T)�exist_ok�wzutf-8)�encoding� F)�indent� ensure_asciiu& ✅ Knowledge base raw data saved to '�'u 🚨 Error saving data to file: ))�os�json�timer �bs4r � urllib.parser r �36START_URLS� SAVE_PATHr �strr r% �dictr1 �__name__r �all_site_linksr& �links�updater � scraped_data� total_links� enumerate�list�i�link� content_dict�append�sleep�makedirs�path�dirname�open�f�dump� Exceptionr# � � r$ �<module>r[ se �� � � � � � � � � � � � � ���� � � � � � � *� *� *� *� *� *� *� *� �"��37�
'� � �.� �� �C� �C� � � � �: �S� �T�D�[� � � � �B �z��� �E�389�:�:�:��S�U�U�N� � %� %�� �
�c�"�"��� � �e�$�$�$�$� �E�39S�c�c�.�>Q�>Q�40S�41S�T�T�T��L� �#�n�%�%�K� �9�T�T�.� 1� 1�1�5�5� � ���4�
��A�A�A�A� �A�A�t�A�A�A�B�B�B�*�*�4�0�0� � � 9�L�� 3� 9� � � � � -� -� -� �E�7� 8� 8� 8� ��42�1�
�
�
�
� 43�E�44S���L�@Q�@Q�45S�46S�T�T�T� 6��� �B�G�O�O�I�.�.��>�>�>�>�
�T�)�S�7�
3�
3�
3� E�q� �D�I�l�A�a�e� D� D� D� D� E� E� E� E� E� E� E� E� E� E� E���� E� E� E� E� ��C�y�C�C�C�D�D�D�D�D�� � 6� 6� 6�
��4��4�4�5�5�5�5�5�5�5�5�5�����6����M �s< �.8F# �&F�; F# �F � F# �F �F# �#G�(F<�<G