KUI71/ACE-Step
0
1"""2This file bundles language identification functions.3 4Modifications (fork): Copyright (c) 2021, Adrien Barbaresi.5 6Original code: Copyright (c) 2011 Marco Lui <saffsd@gmail.com>.7Based on research by Marco Lui and Tim Baldwin.8 9See LICENSE file for more info.10https://github.com/adbar/py3langid11 12Projects:13https://github.com/juntaosun/LangSegment14"""15 16import os17import re18import sys19import numpy as np20from collections import Counter21from collections import defaultdict22 23# import langid24# import py3langid as langid25# pip install py3langid==0.2.226 27# 启用语言预测概率归一化，概率预测的分数。因此，实现重新规范化 产生 0-1 范围内的输出。28# langid disables probability normalization by default. For command-line usages of , it can be enabled by passing the flag. 29# For probability normalization in library use, the user must instantiate their own . An example of such usage is as follows:30from py3langid.langid import LanguageIdentifier, MODEL_FILE31 32# Digital processing33try:from .utils.num import num2str34except ImportError:35 try:from utils.num import num2str36 except ImportError as e:37 raise e38 39# -----------------------------------40# 更新日志：新版本分词更加精准。41# Changelog: The new version of the word segmentation is more accurate.42# チェンジログ:新しいバージョンの単語セグメンテーションはより正確です。43# Changelog: 분할이라는 단어의 새로운 버전이 더 정확합니다.44# -----------------------------------45 46 47# Word segmentation function: 48# automatically identify and split the words (Chinese/English/Japanese/Korean) in the article or sentence according to different languages, 49# making it more suitable for TTS processing.50# This code is designed for front-end text multi-lingual mixed annotation distinction, multi-language mixed training and inference of various TTS projects.51# This processing result is mainly for (Chinese = zh, Japanese = ja, English = en, Korean = ko), and can actually support up to 97 different language mixing processing.52 53#===========================================================================================================54#分かち書き機能:文章や文章の中の例えば（中国語/英語/日本語/韓国語）を、異なる言語で自動的に認識して分割し、TTS処理により適したものにします。55#このコードは、さまざまなTTSプロジェクトのフロントエンドテキストの多言語混合注釈区別、多言語混合トレーニング、および推論のために特別に作成されています。56#===========================================================================================================57#(1)自動分詞:「韓国語では何を読むのですかあなたの体育の先生は誰ですか?今回の発表会では、iPhone 15シリーズの4機種が登場しました」58#（2）手动分词:“あなたの名前は<ja>佐々木ですか?<ja>ですか?”59#この処理結果は主に（中国語=ja、日本語=ja、英語=en、韓国語=ko）を対象としており、実際には最大97の異なる言語の混合処理をサポートできます。60#===========================================================================================================61 62#===========================================================================================================63# 단어 분할 기능: 기사 또는 문장에서 단어(중국어/영어/일본어/한국어)를 다른 언어에 따라 자동으로 식별하고 분할하여 TTS 처리에 더 적합합니다.64# 이 코드는 프런트 엔드 텍스트 다국어 혼합 주석 분화, 다국어 혼합 교육 및 다양한 TTS 프로젝트의 추론을 위해 설계되었습니다.65#===========================================================================================================66# (1) 자동 단어 분할: "한국어로 무엇을 읽습니까? 스포츠 씨? 이 컨퍼런스는 4개의 iPhone 15 시리즈 모델을 제공합니다."67# (2) 수동 참여: "이름이 <ja>Saki입니까? <ja>?"68# 이 처리 결과는 주로 (중국어 = zh, 일본어 = ja, 영어 = en, 한국어 = ko)를 위한 것이며 실제로 혼합 처리를 위해 최대 97개의 언어를 지원합니다.69#===========================================================================================================70 71# ===========================================================================================================72# 分词功能：将文章或句子里的例如（中/英/日/韩），按不同语言自动识别并拆分，让它更适合TTS处理。73# 本代码专为各种 TTS 项目的前端文本多语种混合标注区分，多语言混合训练和推理而编写。74# ===========================================================================================================75# （1）自动分词：“韩语中的오빠读什么呢？あなたの体育の先生は誰ですか? 此次发布会带来了四款iPhone 15系列机型”76# （2）手动分词：“你的名字叫<ja>佐々木？<ja>吗？”77# 本处理结果主要针对（中文=zh , 日文=ja , 英文=en , 韩语=ko）, 实际上可支持多达 97 种不同的语言混合处理。78# ===========================================================================================================79 80 81# 手动分词标签规范：<语言标签>文本内容</语言标签>82# 수동 단어 분할 태그 사양: <언어 태그> 텍스트 내용</언어 태그>83# Manual word segmentation tag specification: <language tags> text content </language tags>84# 手動分詞タグ仕様:<言語タグ>テキスト内容</言語タグ>85# ===========================================================================================================86# For manual word segmentation, labels need to appear in pairs, such as:87# 如需手动分词，标签需要成对出现，例如：“<ja>佐々木<ja>” 或者 “<ja>佐々木</ja>”88# 错误示范：“你的名字叫<ja>佐々木。” 此句子中出现的单个<ja>标签将被忽略，不会处理。89# Error demonstration: "Your name is <ja>佐々木。" Single <ja> tags that appear in this sentence will be ignored and will not be processed.90# ===========================================================================================================91 92 93# ===========================================================================================================94# 语音合成标记语言 SSML , 这里只支持它的标签（非 XML）Speech Synthesis Markup Language SSML, only its tags are supported here (not XML)95# 想支持更多的 SSML 标签？欢迎 PR！ Want to support more SSML tags? PRs are welcome!96# 说明：除了中文以外，它也可改造成支持多语种 SSML ，不仅仅是中文。97# Note: In addition to Chinese, it can also be modified to support multi-language SSML, not just Chinese.98# ===========================================================================================================99# 中文实现：Chinese implementation:100# 【SSML】<number>=中文大写数字读法（单字）101# 【SSML】<telephone>=数字转成中文电话号码大写汉字（单字）102# 【SSML】<currency>=按金额发音。103# 【SSML】<date>=按日期发音。支持 2024年08月24, 2024/8/24, 2024-08, 08-24, 24 等输入。104# ===========================================================================================================105class LangSSML:106 107 def __init__(self):108 # 纯数字109 self._zh_numerals_number = {110 '0': '零',111 '1': '一',112 '2': '二',113 '3': '三',114 '4': '四',115 '5': '五',116 '6': '六',117 '7': '七',118 '8': '八',119 '9': '九'120 }121 122 # 将2024/8/24, 2024-08, 08-24, 24 标准化“年月日”123 # Standardize 2024/8/24, 2024-08, 08-24, 24 to "year-month-day"124 def _format_chinese_data(self, date_str:str):125 # 处理日期格式126 input_date = date_str127 if date_str is None or date_str.strip() == "":return ""128 date_str = re.sub(r"[\/\._|年|月]","-",date_str)129 date_str = re.sub(r"日",r"",date_str)130 date_arrs = date_str.split(' ')131 if len(date_arrs) == 1 and ":" in date_arrs[0]:132 time_str = date_arrs[0]133 date_arrs = []134 else:135 time_str = date_arrs[1] if len(date_arrs) >=2 else ""136 def nonZero(num,cn,func=None):137 if func is not None:num=func(num)138 return f"{num}{cn}" if num is not None and num != "" and num != "0" else ""139 f_number = self.to_chinese_number140 f_currency = self.to_chinese_currency141 # year, month, day142 year_month_day = ""143 if len(date_arrs) > 0:144 year, month, day = "","",""145 parts = date_arrs[0].split('-')146 if len(parts) == 3: # 格式为 YYYY-MM-DD147 year, month, day = parts148 elif len(parts) == 2: # 格式为 MM-DD 或 YYYY-MM149 if len(parts[0]) == 4: # 年-月150 year, month = parts151 else:month, day = parts # 月-日152 elif len(parts[0]) > 0: # 仅有月-日或年153 if len(parts[0]) == 4:154 year = parts[0]155 else:day = parts[0]156 year,month,day = nonZero(year,"年",f_number),nonZero(month,"月",f_currency),nonZero(day,"日",f_currency)157 year_month_day = re.sub(r"([年|月|日])+",r"\1",f"{year}{month}{day}")158 # hours, minutes, seconds159 time_str = re.sub(r"[\/\.\-：_]",":",time_str)160 time_arrs = time_str.split(":")161 hours, minutes, seconds = "","",""162 if len(time_arrs) == 3: # H/M/S163 hours, minutes, seconds = time_arrs164 elif len(time_arrs) == 2:# H/M165 hours, minutes = time_arrs166 elif len(time_arrs[0]) > 0:hours = f'{time_arrs[0]}点' # H167 if len(time_arrs) > 1:168 hours, minutes, seconds = nonZero(hours,"点",f_currency),nonZero(minutes,"分",f_currency),nonZero(seconds,"秒",f_currency)169 hours_minutes_seconds = re.sub(r"([点|分|秒])+",r"\1",f"{hours}{minutes}{seconds}")170 output_date = f"{year_month_day}{hours_minutes_seconds}"171 return output_date172 173 # 【SSML】number=中文大写数字读法（单字）174 # Chinese Numbers(single word)175 def to_chinese_number(self, num:str):176 pattern = r'(\d+)'177 zh_numerals = self._zh_numerals_number178 arrs = re.split(pattern, num)179 output = ""180 for item in arrs:181 if re.match(pattern,item):182 output += ''.join(zh_numerals[digit] if digit in zh_numerals else "" for digit in str(item))183 else:output += item184 output = output.replace(".","点")185 return output186 187 # 【SSML】telephone=数字转成中文电话号码大写汉字（单字）188 # Convert numbers to Chinese phone numbers in uppercase Chinese characters(single word)189 def to_chinese_telephone(self, num:str):190 output = self.to_chinese_number(num.replace("+86","")) # zh +86191 output = output.replace("一","幺")192 return output193 194 # 【SSML】currency=按金额发音。195 # Digital processing from GPT_SoVITS num.py （thanks）196 def to_chinese_currency(self, num:str):197 pattern = r'(\d+)'198 arrs = re.split(pattern, num)199 output = ""200 for item in arrs:201 if re.match(pattern,item):202 output += num2str(item)203 else:output += item204 output = output.replace(".","点")205 return output206 207 # 【SSML】date=按日期发音。支持 2024年08月24, 2024/8/24, 2024-08, 08-24, 24 等输入。208 def to_chinese_date(self, num:str):209 chinese_date = self._format_chinese_data(num)210 return chinese_date211 212 213class LangSegment:214 215 def __init__(self):216 217 self.langid = LanguageIdentifier.from_pickled_model(MODEL_FILE, norm_probs=True)218 219 self._text_cache = None220 self._text_lasts = None221 self._text_langs = None222 self._lang_count = None223 self._lang_eos = None224 225 # 可自定义语言匹配标签：カスタマイズ可能な言語対応タグ:사용자 지정 가능한 언어 일치 태그:226 # Customizable language matching tags: These are supported，이 표현들은 모두 지지합니다227 # <zh>你好<zh> , <ja>佐々木</ja> , <en>OK<en> , <ko>오빠</ko> 这些写法均支持228 self.SYMBOLS_PATTERN = r'(<([a-zA-Z|-]*)>(.*?)<\/*[a-zA-Z|-]*>)'229 230 # 语言过滤组功能, 可以指定保留语言。不在过滤组中的语言将被清除。您可随心搭配TTS语音合成所支持的语言。231 # 언어 필터 그룹 기능을 사용하면 예약된 언어를 지정할 수 있습니다. 필터 그룹에 없는 언어는 지워집니다. TTS 텍스트에서 지원하는 언어를 원하는 대로 일치시킬 수 있습니다.232 # 言語フィルターグループ機能では、予約言語を指定できます。フィルターグループに含まれていない言語はクリアされます。TTS音声合成がサポートする言語を自由に組み合わせることができます。233 # The language filter group function allows you to specify reserved languages. 234 # Languages not in the filter group will be cleared. You can match the languages supported by TTS Text To Speech as you like.235 # 排名越前，优先级越高，The higher the ranking, the higher the priority，ランキングが上位になるほど、優先度が高くなります。236 237 # 系统默认过滤器。System default filter。(ISO 639-1 codes given)238 # ----------------------------------------------------------------------------------------------------------------------------------239 # "zh"中文=Chinese ,"en"英语=English ,"ja"日语=Japanese ,"ko"韩语=Korean ,"fr"法语=French ,"vi"越南语=Vietnamese , "ru"俄语=Russian240 # "th"泰语=Thai241 # ----------------------------------------------------------------------------------------------------------------------------------242 self.DEFAULT_FILTERS = ["zh", "ja", "ko", "en"]243 244 # 用户可自定义过滤器。User-defined filters245 self.Langfilters = self.DEFAULT_FILTERS[:] # 创建副本246 247 # 合并文本248 self.isLangMerge = True249 250 # 试验性支持：您可自定义添加："fr"法语 , "vi"越南语。Experimental: You can customize to add: "fr" French, "vi" Vietnamese.251 # 请使用API启用：self.setfilters(["zh", "en", "ja", "ko", "fr", "vi" , "ru" , "th"]) # 您可自定义添加，如："fr"法语 , "vi"越南语。252 253 # 预览版功能，自动启用或禁用，无需设置254 # Preview feature, automatically enabled or disabled, no settings required255 self.EnablePreview = False256 257 # 除此以外，它支持简写过滤器，只需按不同语种任意组合即可。258 # In addition to that, it supports abbreviation filters, allowing for any combination of different languages.259 # 示例：您可以任意指定多种组合，进行过滤260 # Example: You can specify any combination to filter261 262 # 中/日语言优先级阀值（评分范围为 0 ~ 1）:评分低于设定阀值 <0.89 时，启用 filters 中的优先级。\n263 # 중/일본어 우선 순위 임계값(점수 범위 0-1): 점수가 설정된 임계값 <0.89보다 낮을 때 필터에서 우선 순위를 활성화합니다.264 # 中国語/日本語の優先度しきい値（スコア範囲0〜1）:スコアが設定されたしきい値<0.89未満の場合、フィルターの優先度が有効になります。\n265 # Chinese and Japanese language priority threshold (score range is 0 ~ 1): The default threshold is 0.89. \n266 # Only the common characters between Chinese and Japanese are processed with confidence and priority. \n267 self.LangPriorityThreshold = 0.89268 269 # Langfilters = ["zh"] # 按中文识别270 # Langfilters = ["en"] # 按英文识别271 # Langfilters = ["ja"] # 按日文识别272 # Langfilters = ["ko"] # 按韩文识别273 # Langfilters = ["zh_ja"] # 中日混合识别274 # Langfilters = ["zh_en"] # 中英混合识别275 # Langfilters = ["ja_en"] # 日英混合识别276 # Langfilters = ["zh_ko"] # 中韩混合识别277 # Langfilters = ["ja_ko"] # 日韩混合识别278 # Langfilters = ["en_ko"] # 英韩混合识别279 # Langfilters = ["zh_ja_en"] # 中日英混合识别280 # Langfilters = ["zh_ja_en_ko"] # 中日英韩混合识别281 282 # 更多过滤组合，请您随意。。。For more filter combinations, please feel free to......283 # より多くのフィルターの組み合わせ、お気軽に。。。더 많은 필터 조합을 원하시면 자유롭게 해주세요. .....284 285 # 可选保留：支持中文数字拼音格式，更方便前端实现拼音音素修改和推理，默认关闭 False 。286 # 开启后 True ，括号内的数字拼音格式均保留，并识别输出为："zh"中文。287 self.keepPinyin = False 288 289 # DEFINITION290 self.PARSE_TAG = re.compile(r'(⑥\$*\d+[\d]{6,}⑥)')291 292 self.LangSSML = LangSSML()293 294 def _clears(self):295 self._text_cache = None296 self._text_lasts = None297 self._text_langs = None298 self._text_waits = None299 self._lang_count = None300 self._lang_eos = None301 302 def _is_english_word(self, word):303 return bool(re.match(r'^[a-zA-Z]+$', word))304 305 def _is_chinese(self, word):306 for char in word:307 if '\u4e00' <= char <= '\u9fff':308 return True309 return False310 311 def _is_japanese_kana(self, word):312 pattern = re.compile(r'[\u3040-\u309F\u30A0-\u30FF]+')313 matches = pattern.findall(word)314 return len(matches) > 0315 316 def _insert_english_uppercase(self, word):317 modified_text = re.sub(r'(?<!\b)([A-Z])', r' \1', word)318 modified_text = modified_text.strip('-')319 return modified_text + " "320 321 def _split_camel_case(self, word):322 return re.sub(r'(?<!^)(?=[A-Z])', ' ', word)323 324 def _statistics(self, language, text):325 # Language word statistics:326 # Chinese characters usually occupy double bytes327 if self._lang_count is None or not isinstance(self._lang_count, defaultdict):328 self._lang_count = defaultdict(int)329 lang_count = self._lang_count330 if not "|" in language:331 lang_count[language] += int(len(text)*2) if language == "zh" else len(text)332 self._lang_count = lang_count333 334 def _clear_text_number(self, text):335 if text == "\n":return text,False # Keep Line Breaks336 clear_text = re.sub(r'([^\w\s]+)','',re.sub(r'\n+','',text)).strip()337 is_number = len(re.sub(re.compile(r'(\d+)'),'',clear_text)) == 0338 return clear_text,is_number339 340 def _saveData(self, words,language:str,text:str,score:float,symbol=None):341 # Pre-detection342 clear_text , is_number = self._clear_text_number(text)343 # Merge the same language and save the results344 preData = words[-1] if len(words) > 0 else None345 if symbol is not None:pass346 elif preData is not None and preData["symbol"] is None:347 if len(clear_text) == 0:language = preData["lang"]348 elif is_number == True:language = preData["lang"]349 _ , pre_is_number = self._clear_text_number(preData["text"])350 if (preData["lang"] == language):351 self._statistics(preData["lang"],text)352 text = preData["text"] + text353 preData["text"] = text354 return preData355 elif pre_is_number == True:356 text = f'{preData["text"]}{text}'357 words.pop()358 elif is_number == True: 359 priority_language = self._get_filters_string()[:2]360 if priority_language in "ja-zh-en-ko-fr-vi":language = priority_language361 data = {"lang":language,"text": text,"score":score,"symbol":symbol}362 filters = self.Langfilters363 if filters is None or len(filters) == 0 or "?" in language or \364 language in filters or language in filters[0] or \365 filters[0] == "*" or filters[0] in "alls-mixs-autos":366 words.append(data)367 self._statistics(data["lang"],data["text"])368 return data369 370 def _addwords(self, words,language,text,score,symbol=None):371 if text == "\n":pass # Keep Line Breaks372 elif text is None or len(text.strip()) == 0:return True373 if language is None:language = ""374 language = language.lower()375 if language == 'en':text = self._insert_english_uppercase(text)376 # text = re.sub(r'[(（）)]', ',' , text) # Keep it.377 text_waits = self._text_waits378 ispre_waits = len(text_waits)>0379 preResult = text_waits.pop() if ispre_waits else None380 if preResult is None:preResult = words[-1] if len(words) > 0 else None381 if preResult and ("|" in preResult["lang"]): 382 pre_lang = preResult["lang"]383 if language in pre_lang:preResult["lang"] = language = language.split("|")[0]384 else:preResult["lang"]=pre_lang.split("|")[0]385 if ispre_waits:preResult = self._saveData(words,preResult["lang"],preResult["text"],preResult["score"],preResult["symbol"])386 pre_lang = preResult["lang"] if preResult else None387 if ("|" in language) and (pre_lang and not pre_lang in language and not "…" in language):language = language.split("|")[0]388 if "|" in language:self._text_waits.append({"lang":language,"text": text,"score":score,"symbol":symbol})389 else:self._saveData(words,language,text,score,symbol)390 return False391 392 def _get_prev_data(self, words):393 data = words[-1] if words and len(words) > 0 else None394 if data:return (data["lang"] , data["text"])395 return (None,"")396 397 def _match_ending(self, input , index):398 if input is None or len(input) == 0:return False,None399 input = re.sub(r'\s+', '', input)400 if len(input) == 0 or abs(index) > len(input):return False,None401 ending_pattern = re.compile(r'([「」“”‘’"\':：。.！!?．？])')402 return ending_pattern.match(input[index]),input[index]403 404 def _cleans_text(self, cleans_text):405 cleans_text = re.sub(r'(.*?)([^\w]+)', r'\1 ', cleans_text)406 cleans_text = re.sub(r'(.)\1+', r'\1', cleans_text)407 return cleans_text.strip()408 409 def _mean_processing(self, text:str):410 if text is None or (text.strip()) == "":return None , 0.0411 arrs = self._split_camel_case(text).split(" ")412 langs = []413 for t in arrs:414 if len(t.strip()) <= 3:continue415 language, score = self.langid.classify(t)416 langs.append({"lang":language})417 if len(langs) == 0:return None , 0.0418 return Counter([item['lang'] for item in langs]).most_common(1)[0][0],1.0419 420 def _lang_classify(self, cleans_text):421 language, score = self.langid.classify(cleans_text)422 # fix: Huggingface is np.float32423 if score is not None and isinstance(score, np.generic) and hasattr(score,"item"):424 score = score.item()425 score = round(score , 3)426 return language, score427 428 def _get_filters_string(self):429 filters = self.Langfilters430 return "-".join(filters).lower().strip() if filters is not None else ""431 432 def _parse_language(self, words , segment):433 LANG_JA = "ja"434 LANG_ZH = "zh"435 LANG_ZH_JA = f'{LANG_ZH}|{LANG_JA}'436 LANG_JA_ZH = f'{LANG_JA}|{LANG_ZH}'437 language = LANG_ZH438 regex_pattern = re.compile(r'([^\w\s]+)')439 lines = regex_pattern.split(segment)440 lines_max = len(lines)441 LANG_EOS =self._lang_eos442 for index, text in enumerate(lines):443 if len(text) == 0:continue444 EOS = index >= (lines_max - 1)445 nextId = index + 1446 nextText = lines[nextId] if not EOS else ""447 nextPunc = len(re.sub(regex_pattern,'',re.sub(r'\n+','',nextText)).strip()) == 0448 textPunc = len(re.sub(regex_pattern,'',re.sub(r'\n+','',text)).strip()) == 0449 if not EOS and (textPunc == True or ( len(nextText.strip()) >= 0 and nextPunc == True)):450 lines[nextId] = f'{text}{nextText}'451 continue452 number_tags = re.compile(r'(⑥\d{6,}⑥)')453 cleans_text = re.sub(number_tags, '' ,text)454 cleans_text = re.sub(r'\d+', '' ,cleans_text)455 cleans_text = self._cleans_text(cleans_text)456 # fix:Langid's recognition of short sentences is inaccurate, and it is spliced longer.457 if not EOS and len(cleans_text) <= 2:458 lines[nextId] = f'{text}{nextText}'459 continue460 language,score = self._lang_classify(cleans_text)461 prev_language , prev_text = self._get_prev_data(words)462 if language != LANG_ZH and all('\u4e00' <= c <= '\u9fff' for c in re.sub(r'\s','',cleans_text)):language,score = LANG_ZH,1463 if len(cleans_text) <= 5 and self._is_chinese(cleans_text):464 filters_string = self._get_filters_string()465 if score < self.LangPriorityThreshold and len(filters_string) > 0:466 index_ja , index_zh = filters_string.find(LANG_JA) , filters_string.find(LANG_ZH)467 if index_ja != -1 and index_ja < index_zh:language = LANG_JA468 elif index_zh != -1 and index_zh < index_ja:language = LANG_ZH469 if self._is_japanese_kana(cleans_text):language = LANG_JA470 elif len(cleans_text) > 2 and score > 0.90:pass471 elif EOS and LANG_EOS:language = LANG_ZH if len(cleans_text) <= 1 else language472 else:473 LANG_UNKNOWN = LANG_ZH_JA if language == LANG_ZH or (len(cleans_text) <=2 and prev_language == LANG_ZH) else LANG_JA_ZH474 match_end,match_char = self._match_ending(text, -1)475 referen = prev_language in LANG_UNKNOWN or LANG_UNKNOWN in prev_language if prev_language else False476 if match_char in "。.": language = prev_language if referen and len(words) > 0 else language477 else:language = f"{LANG_UNKNOWN}|…"478 text,*_ = re.subn(number_tags , self._restore_number , text )479 self._addwords(words,language,text,score)480 481 # ----------------------------------------------------------482 # 【SSML】中文数字处理：Chinese Number Processing (SSML support)483 # 这里默认都是中文，用于处理 SSML 中文标签。当然可以支持任意语言，例如：484 # The default here is Chinese, which is used to process SSML Chinese tags. Of course, any language can be supported, for example:485 # 中文电话号码：<telephone>1234567</telephone>486 # 中文数字号码：<number>1234567</number>487 def _process_symbol_SSML(self, words,data):488 tag , match = data489 language = SSML = match[1]490 text = match[2]491 score = 1.0492 if SSML == "telephone":493 # 中文-电话号码494 language = "zh"495 text = self.LangSSML.to_chinese_telephone(text)496 elif SSML == "number":497 # 中文-数字读法498 language = "zh"499 text = self.LangSSML.to_chinese_number(text)500 elif SSML == "currency":501 # 中文-按金额发音502 language = "zh"503 text = self.LangSSML.to_chinese_currency(text)504 elif SSML == "date":505 # 中文-按金额发音506 language = "zh"507 text = self.LangSSML.to_chinese_date(text)508 self._addwords(words,language,text,score,SSML)509 510 # ----------------------------------------------------------511 def _restore_number(self, matche):512 value = matche.group(0)513 text_cache = self._text_cache514 if value in text_cache:515 process , data = text_cache[value]516 tag , match = data517 value = match518 return value519 520 def _pattern_symbols(self, item , text):521 if text is None:return text522 tag , pattern , process = item523 matches = pattern.findall(text)524 if len(matches) == 1 and "".join(matches[0]) == text:525 return text526 for i , match in enumerate(matches):527 key = f"⑥{tag}{i:06d}⑥"528 text = re.sub(pattern , key , text , count=1)529 self._text_cache[key] = (process , (tag , match))530 return text531 532 def _process_symbol(self, words,data):533 tag , match = data534 language = match[1]535 text = match[2]536 score = 1.0537 filters = self._get_filters_string()538 if language not in filters:539 self._process_symbol_SSML(words,data)540 else:541 self._addwords(words,language,text,score,True)542 543 def _process_english(self, words,data):544 tag , match = data545 text = match[0]546 filters = self._get_filters_string()547 priority_language = filters[:2]548 # Preview feature, other language segmentation processing549 enablePreview = self.EnablePreview550 if enablePreview == True:551 # Experimental: Other language support552 regex_pattern = re.compile(r'(.*?[。.?？!！]+[\n]{,1})')553 lines = regex_pattern.split(text)554 for index , text in enumerate(lines):555 if len(text.strip()) == 0:continue556 cleans_text = self._cleans_text(text)557 language,score = self._lang_classify(cleans_text)558 if language not in filters:559 language,score = self._mean_processing(cleans_text)560 if language is None or score <= 0.0:continue561 elif language in filters:pass # pass562 elif score >= 0.95:continue # High score, but not in the filter, excluded.563 elif score <= 0.15 and filters[:2] == "fr":language = priority_language564 else:language = "en"565 self._addwords(words,language,text,score)566 else:567 # Default is English568 language, score = "en", 1.0569 self._addwords(words,language,text,score)570 571 def _process_Russian(self, words,data):572 tag , match = data573 text = match[0]574 language = "ru"575 score = 1.0576 self._addwords(words,language,text,score)577 578 def _process_Thai(self, words,data):579 tag , match = data580 text = match[0]581 language = "th"582 score = 1.0583 self._addwords(words,language,text,score)584 585 def _process_korean(self, words,data):586 tag , match = data587 text = match[0]588 language = "ko"589 score = 1.0590 self._addwords(words,language,text,score)591 592 def _process_quotes(self, words,data):593 tag , match = data594 text = "".join(match)595 childs = self.PARSE_TAG.findall(text)596 if len(childs) > 0:597 self._process_tags(words , text , False)598 else:599 cleans_text = self._cleans_text(match[1])600 if len(cleans_text) <= 5:601 self._parse_language(words,text)602 else:603 language,score = self._lang_classify(cleans_text)604 self._addwords(words,language,text,score)605 606 def _process_pinyin(self, words,data):607 tag , match = data608 text = match609 language = "zh"610 score = 1.0611 self._addwords(words,language,text,score)612 613 def _process_number(self, words,data): # "$0" process only614 """615 Numbers alone cannot accurately identify language.616 Because numbers are universal in all languages.617 So it won't be executed here, just for testing.618 """619 tag , match = data620 language = words[0]["lang"] if len(words) > 0 else "zh"621 text = match622 score = 0.0623 self._addwords(words,language,text,score)624 625 def _process_tags(self, words , text , root_tag):626 text_cache = self._text_cache627 segments = re.split(self.PARSE_TAG, text)628 segments_len = len(segments) - 1629 for index , text in enumerate(segments):630 if root_tag:self._lang_eos = index >= segments_len631 if self.PARSE_TAG.match(text):632 process , data = text_cache[text]633 if process:process(words , data)634 else:635 self._parse_language(words , text)636 return words637 638 def _merge_results(self, words):639 new_word = []640 for index , cur_data in enumerate(words):641 if "symbol" in cur_data:del cur_data["symbol"]642 if index == 0:new_word.append(cur_data)643 else:644 pre_data = new_word[-1]645 if cur_data["lang"] == pre_data["lang"]:646 pre_data["text"] = f'{pre_data["text"]}{cur_data["text"]}'647 else:new_word.append(cur_data)648 return new_word649 650 def _parse_symbols(self, text):651 TAG_NUM = "00" # "00" => default channels , "$0" => testing channel652 TAG_S1,TAG_S2,TAG_P1,TAG_P2,TAG_EN,TAG_KO,TAG_RU,TAG_TH = "$1" ,"$2" ,"$3" ,"$4" ,"$5" ,"$6" ,"$7","$8"653 TAG_BASE = re.compile(fr'(([【《（(“‘"\']*[LANGUAGE]+[\W\s]*)+)')654 # Get custom language filter655 filters = self.Langfilters656 filters = filters if filters is not None else ""657 # =======================================================================================================658 # Experimental: Other language support.Thử nghiệm: Hỗ trợ ngôn ngữ khác.Expérimental : prise en charge d’autres langues.659 # 相关语言字符如有缺失，熟悉相关语言的朋友，可以提交把缺失的发音符号补全。660 # If relevant language characters are missing, friends who are familiar with the relevant languages can submit a submission to complete the missing pronunciation symbols.661 # S'il manque des caractères linguistiques pertinents, les amis qui connaissent les langues concernées peuvent soumettre une soumission pour compléter les symboles de prononciation manquants.662 # Nếu thiếu ký tự ngôn ngữ liên quan, những người bạn quen thuộc với ngôn ngữ liên quan có thể gửi bài để hoàn thành các ký hiệu phát âm còn thiếu.663 # -------------------------------------------------------------------------------------------------------664 # Preview feature, other language support665 enablePreview = self.EnablePreview666 if "fr" in filters or \667 "vi" in filters:enablePreview = True668 self.EnablePreview = enablePreview669 # 实验性：法语字符支持。Prise en charge des caractères français670 RE_FR = "" if not enablePreview else "àáâãäåæçèéêëìíîïðñòóôõöùúûüýþÿ"671 # 实验性：越南语字符支持。Hỗ trợ ký tự tiếng Việt672 RE_VI = "" if not enablePreview else "đơưăáàảãạắằẳẵặấầẩẫậéèẻẽẹếềểễệíìỉĩịóòỏõọốồổỗộớờởỡợúùủũụứừửữựôâêơưỷỹ"673 # -------------------------------------------------------------------------------------------------------674 # Basic options:675 process_list = [676 ( TAG_S1 , re.compile(self.SYMBOLS_PATTERN) , self._process_symbol ), # Symbol Tag677 ( TAG_KO , re.compile(re.sub(r'LANGUAGE',f'\uac00-\ud7a3',TAG_BASE.pattern)) , self._process_korean ), # Korean words678 ( TAG_TH , re.compile(re.sub(r'LANGUAGE',f'\u0E00-\u0E7F',TAG_BASE.pattern)) , self._process_Thai ), # Thai words support.679 ( TAG_RU , re.compile(re.sub(r'LANGUAGE',f'А-Яа-яЁё',TAG_BASE.pattern)) , self._process_Russian ), # Russian words support.680 ( TAG_NUM , re.compile(r'(\W*\d+\W+\d*\W*\d*)') , self._process_number ), # Number words, Universal in all languages, Ignore it.681 ( TAG_EN , re.compile(re.sub(r'LANGUAGE',f'a-zA-Z{RE_FR}{RE_VI}',TAG_BASE.pattern)) , self._process_english ), # English words + Other language support.682 ( TAG_P1 , re.compile(r'(["\'])(.*?)(\1)') , self._process_quotes ), # Regular quotes683 ( TAG_P2 , re.compile(r'([\n]*[【《（(“‘])([^【《（(“‘’”)）》】]{3,})([’”)）》】][\W\s]*[\n]{,1})') , self._process_quotes ), # Special quotes, There are left and right.684 ]685 # Extended options: Default False686 if self.keepPinyin == True:process_list.insert(1 , 687 ( TAG_S2 , re.compile(r'([\(（{](?:\s*\w*\d\w*\s*)+[}）\)])') , self._process_pinyin ), # Chinese Pinyin Tag. 688 ) 689 # -------------------------------------------------------------------------------------------------------690 words = []691 lines = re.findall(r'.*\n*', re.sub(self.PARSE_TAG, '' ,text))692 for index , text in enumerate(lines):693 if len(text.strip()) == 0:continue694 self._lang_eos = False695 self._text_cache = {}696 for item in process_list:697 text = self._pattern_symbols(item , text)698 cur_word = self._process_tags([] , text , True)699 if len(cur_word) == 0:continue700 cur_data = cur_word[0] if len(cur_word) > 0 else None701 pre_data = words[-1] if len(words) > 0 else None702 if cur_data and pre_data and cur_data["lang"] == pre_data["lang"] \703 and cur_data["symbol"] == False and pre_data["symbol"] :704 cur_data["text"] = f'{pre_data["text"]}{cur_data["text"]}'705 words.pop()706 words += cur_word707 if self.isLangMerge == True:words = self._merge_results(words)708 lang_count = self._lang_count709 if lang_count and len(lang_count) > 0:710 lang_count = dict(sorted(lang_count.items(), key=lambda x: x[1], reverse=True))711 lang_count = list(lang_count.items())712 self._lang_count = lang_count713 return words714 715 def setfilters(self, filters):716 # 当过滤器更改时，清除缓存717 # 필터가 변경되면 캐시를 지웁니다.718 # フィルタが変更されると、キャッシュがクリアされます719 # When the filter changes, clear the cache720 if self.Langfilters != filters:721 self._clears()722 self.Langfilters = filters723 724 def getfilters(self):725 return self.Langfilters726 727 def setPriorityThreshold(self, threshold:float):728 self.LangPriorityThreshold = threshold729 730 def getPriorityThreshold(self):731 return self.LangPriorityThreshold732 733 def getCounts(self):734 lang_count = self._lang_count735 if lang_count is not None:return lang_count736 text_langs = self._text_langs737 if text_langs is None or len(text_langs) == 0:return [("zh",0)]738 lang_counts = defaultdict(int)739 for d in text_langs:lang_counts[d['lang']] += int(len(d['text'])*2) if d['lang'] == "zh" else len(d['text'])740 lang_counts = dict(sorted(lang_counts.items(), key=lambda x: x[1], reverse=True))741 lang_counts = list(lang_counts.items())742 self._lang_count = lang_counts743 return lang_counts744 745 def getTexts(self, text:str):746 if text is None or len(text.strip()) == 0:747 self._clears()748 return []749 # lasts750 text_langs = self._text_langs751 if self._text_lasts == text and text_langs is not None:return text_langs 752 # parse753 self._text_waits = []754 self._lang_count = None755 self._text_lasts = text756 text = self._parse_symbols(text)757 self._text_langs = text758 return text759 760 def classify(self, text:str):761 return self.getTexts(text)762 763def printList(langlist):764 """765 功能：打印数组结果766 기능: 어레이 결과 인쇄767 機能:配列結果を印刷768 Function: Print array results769 """770 print("\n===================【打印结果】===================")771 if langlist is None or len(langlist) == 0:772 print("无内容结果,No content result")773 return774 for line in langlist:775 print(line)776 pass 777 778 779 780def main():781 782 # -----------------------------------783 # 更新日志：新版本分词更加精准。784 # Changelog: The new version of the word segmentation is more accurate.785 # チェンジログ:新しいバージョンの単語セグメンテーションはより正確です。786 # Changelog: 분할이라는 단어의 새로운 버전이 더 정확합니다.787 # -----------------------------------788 789 # 输入示例1：（包含日文，中文）Input Example 1: (including Japanese, Chinese)790 # text = "“昨日は雨が降った，音楽、映画。。。”你今天学习日语了吗？春は桜の季節です。语种分词是语音合成必不可少的环节。言語分詞は音声合成に欠かせない環節である！"791 792 # 输入示例2：（包含日文，中文）Input Example 1: (including Japanese, Chinese)793 # text = "欢迎来玩。東京，は日本の首都です。欢迎来玩. 太好了!"794 795 # 输入示例3：（包含日文，中文）Input Example 1: (including Japanese, Chinese)796 # text = "明日、私たちは海辺にバカンスに行きます。你会说日语吗：“中国語、話せますか” 你的日语真好啊！"797 798 799 # 输入示例4：（包含日文，中文，韩语，英文）Input Example 4: (including Japanese, Chinese, Korean, English)800 # text = "你的名字叫<ja>佐々木？<ja>吗？韩语中的안녕 오빠读什么呢？あなたの体育の先生は誰ですか? 此次发布会带来了四款iPhone 15系列机型和三款Apple Watch等一系列新品，这次的iPad Air采用了LCD屏幕" 801 802 803 # 试验性支持："fr"法语 , "vi"越南语 , "ru"俄语 , "th"泰语。Experimental: Other language support.804 langsegment = LangSegment()805 langsegment.setfilters(["fr", "vi" , "ja", "zh", "ko", "en" , "ru" , "th"])806 text = """807我喜欢在雨天里听音乐。808I enjoy listening to music on rainy days.809雨の日に音楽を聴くのが好きです。810비 오는 날에 음악을 듣는 것을 즐깁니다。811J'aime écouter de la musique les jours de pluie.812Tôi thích nghe nhạc vào những ngày mưa.813Мне нравится слушать музыку в дождливую погоду.814ฉันชอบฟังเพลงในวันที่ฝนตก815"""816 817 818 819 # 进行分词：（接入TTS项目仅需一行代码调用）Segmentation: (Only one line of code is required to access the TTS project)820 langlist = langsegment.getTexts(text)821 printList(langlist)822 823 824 # 语种统计:Language statistics:825 print("\n===================【语种统计】===================")826 # 获取所有语种数组结果，根据内容字数降序排列827 # Get the array results in all languages, sorted in descending order according to the number of content words828 langCounts = langsegment.getCounts()829 print(langCounts , "\n")830 831 # 根据结果获取内容的主要语种 (语言，字数含标点)832 # Get the main language of content based on the results (language, word count including punctuation)833 lang , count = langCounts[0] 834 print(f"输入内容的主要语言为 = {lang} ，字数 = {count}")835 print("==================================================\n")836 837 838 # 分词输出：lang=语言，text=内容。Word output: lang = language, text = content839 # ===================【打印结果】===================840 # {'lang': 'zh', 'text': '你的名字叫'}841 # {'lang': 'ja', 'text': '佐々木？'}842 # {'lang': 'zh', 'text': '吗？韩语中的'}843 # {'lang': 'ko', 'text': '안녕 오빠'}844 # {'lang': 'zh', 'text': '读什么呢？'}845 # {'lang': 'ja', 'text': 'あなたの体育の先生は誰ですか?'}846 # {'lang': 'zh', 'text': ' 此次发布会带来了四款'}847 # {'lang': 'en', 'text': 'i Phone '}848 # {'lang': 'zh', 'text': '15系列机型和三款'}849 # {'lang': 'en', 'text': 'Apple Watch '}850 # {'lang': 'zh', 'text': '等一系列新品，这次的'}851 # {'lang': 'en', 'text': 'i Pad Air '}852 # {'lang': 'zh', 'text': '采用了'}853 # {'lang': 'en', 'text': 'L C D '}854 # {'lang': 'zh', 'text': '屏幕'}855 # ===================【语种统计】===================856 857 # ===================【语种统计】===================858 # [('zh', 51), ('ja', 19), ('en', 18), ('ko', 5)]859 860 # 输入内容的主要语言为 = zh ，字数 = 51861 # ==================================================862 # The main language of the input content is = zh, word count = 51863 864 865if __name__ == "__main__":866 main()867