ddrg/super_eurlex

Super-EURLEX dataset containing legal documents from multiple languages. The datasets are build/scrapped from the EURLEX Website [https://eur-lex.europa.eu/homepage.html] With one split per language and sector, because the available features (metadata) differs for each sector. Therefore, each sample contains the content of a full legal document in up to 3 different formats. Those are raw HTML and cleaned HTML (if the HTML format was available on the EURLEX website during the scrapping process) and cleaned text. The cleaned text should be available for each sample and was extracted from HTML or PDF. 'Cleaned' HTML stands here for minor cleaning that was done to preserve to a large extent the necessary HTML information like table structures while removing unnecessary complexity which was introduced to the original documents due to actions like writing each sentence into a new object. Additionally, each sample contains metadata which was scrapped on the fly, this implies the following 2 things. First, not every sector contains the same metadata. Second, most metadata might be irrelevant for most use cases. In our minds the most interesting metadata is the celex-id which is used to identify the legal document at hand, but also contains a lot of information about the document see [https://eur-lex.europa.eu/content/tools/eur-lex-celex-infographic-A3.pdf] as well as eurovoc- concepts, which are labels that define the content of the documents. Eurovoc-Concepts are, for example, only available for the sectors 1, 2, 3, 4, 5, 6, 9, C, and E. The Naming of most metadata is kept like it was on the eurlex website, except for converting it to lower case and replacing whitespaces with '_'.