mingoliminze/high_frequency_insight
0
1{2 "cells": [3 {4 "cell_type": "markdown",5 "id": "0bcb6459",6 "metadata": {},7 "source": [8 "# 1 提取因子"9 ]10 },11 {12 "cell_type": "markdown",13 "id": "1a9b8fc7",14 "metadata": {},15 "source": [16 " 生成JSON卡片，对于论文要求输出最主要的结论，对于书籍要求输出最全面的结论。"17 ]18 },19 {20 "cell_type": "code",21 "execution_count": 12,22 "id": "e3914353",23 "metadata": {},24 "outputs": [25 {26 "name": "stdout",27 "output_type": "stream",28 "text": [29 "Start Processing: C:/Users/Minze Li/OneDrive/MAFN/CITIC Intern/QuantPaper/20251227_Guojin_Alpha_LLM_Framework.pdf\n",30 "Uploading to Google AI Studio...\n",31 "Gemini is analyzing the math logic...\n",32 "\n",33 "[Success] Extraction Complete! Obtain 7 Factors:\n",34 "\n",35 "--------------------------------------------------\n",36 "[\n",37 " {\n",38 " \"factor_name\": \"Price_Momentum_Volume_Confirmation\",\n",39 " \"category\": \"Momentum\",\n",40 " \"sub_category\": \"Volume Weighted Momentum\",\n",41 " \"frequency\": \"Daily\",\n",42 " \"frequency_reasoning\": \"The formula uses daily 'close' and 'volume' data with short-term rolling windows (5 days), which is typical for daily frequency factor calculation.\",\n",43 " \"financial_logic\": \"This factor aims to identify strong upward trends by combining three signals: price acceleration (Slope of close), volume expansion (Slope of volume), and strong price-volume co-movement (Covariance/Variance as Beta). A high value indicates that a stock's price is rising rapidly, accompanied by increasing volume and a tight price-volume relationship, suggesting strong conviction from market participants. The final EMA smooths the signal.\",\n",44 " \"core_formula_latex\": \"EMA(Slope(close, 5) * Cov(close, volume, 5) / Var(close, 5) * Slope(volume, 5), 5)\",\n",45 " \"inputs_required\": [\n",46 " \"close\",\n",47 " \"volume\"\n",48 " ],\n",49 " \"parameters\": [\n",50 " 5,\n",51 " 5,\n",52 " 5,\n",53 " 5,\n",54 " 5\n",55 " ],\n",56 " \"calculation_type\": \"Rolling Window\",\n",57 " \"numpy_implementation_hint\": \"For Slope(x, 5), use a rolling window applying a linear regression of the data against np.arange(5). For Cov and Var, use pandas .rolling().cov() and .rolling().var(). For EMA, use pandas .ewm().mean(). Combine the intermediate series with element-wise multiplication.\"\n",58 " },\n",59 " {\n",60 " \"factor_name\": \"Breakout_Strength_With_Volume_Confirmation\",\n",61 " \"category\": \"Momentum\",\n",62 " \"sub_category\": \"Breakout Strategy\",\n",63 " \"frequency\": \"Daily\",\n",64 " \"frequency_reasoning\": \"The formula uses daily OHLCV data ('close', 'high', 'low', 'volume') over a 5-day lookback period, consistent with a daily trading frequency.\",\n",65 " \"financial_logic\": \"This factor measures the strength of a price breakout. The first part, (close - Max(high, 5)) / (Max(high, 5) - Min(low, 5)), quantifies the magnitude of the current close relative to the recent price range, with higher values indicating a stronger push above the recent high. This is multiplied by the moving average of volume, meaning the signal is amplified when the breakout is supported by significant trading activity, indicating strong investor consensus.\",\n",66 " \"core_formula_latex\": \"(close - Max(high, 5)) / (Max(high, 5) - Min(low, 5)) * EMA(volume, 5)\",\n",67 " \"inputs_required\": [\n",68 " \"close\",\n",69 " \"high\",\n",70 " \"low\",\n",71 " \"volume\"\n",72 " ],\n",73 " \"parameters\": [\n",74 " 5,\n",75 " 5,\n",76 " 5,\n",77 " 5\n",78 " ],\n",79 " \"calculation_type\": \"Rolling Window\",\n",80 " \"numpy_implementation_hint\": \"Use pandas rolling functions: `(df['close'] - df['high'].rolling(5).max()) / (df['high'].rolling(5).max() - df['low'].rolling(5).min())`. Multiply the result by `df['volume'].ewm(span=5, adjust=False).mean()`.\"\n",81 " },\n",82 " {\n",83 " \"factor_name\": \"Volume_Amplified_Volatility_Decoupling\",\n",84 " \"category\": \"Mean Reversion\",\n",85 " \"sub_category\": \"Volatility Signal\",\n",86 " \"frequency\": \"Daily\",\n",87 " \"frequency_reasoning\": \"The formula utilizes daily OHLCV data along with daily VWAP over multiple lookback windows (5, 10, 20 days), indicating a daily calculation frequency.\",\n",88 " \"financial_logic\": \"This factor identifies potential trend reversals. It measures recent volatility (5-day range) amplified by average trading volume. This is then normalized by the correlation between price and VWAP. A low correlation suggests price is moving away from its volume-weighted average, indicating a potential decoupling. When this decoupling occurs during a period of high, volume-amplified volatility, it signals that the current trend may be unsustainable and prone to reversal. The paper states a lower factor value implies a higher probability of reversal.\",\n",89 " \"core_formula_latex\": \"Mean(volume, 20) * (Max(high, 5) - Min(low, 5)) / (Corr(close, vwap, 10) + 2)\",\n",90 " \"inputs_required\": [\n",91 " \"volume\",\n",92 " \"high\",\n",93 " \"low\",\n",94 " \"close\",\n",95 " \"vwap\"\n",96 " ],\n",97 " \"parameters\": [\n",98 " 20,\n",99 " 5,\n",100 " 5,\n",101 " 10,\n",102 " 2\n",103 " ],\n",104 " \"calculation_type\": \"Rolling Window\",\n",105 " \"numpy_implementation_hint\": \"Use pandas rolling functions: `df['volume'].rolling(20).mean()`, `df['high'].rolling(5).max()`, `df['low'].rolling(5).min()`, and `df['close'].rolling(10).corr(df['vwap'])`. The `+ 2` is added to the correlation to prevent division by small numbers and to scale the result.\"\n",106 " },\n",107 " {\n",108 " \"factor_name\": \"Cash_Flow_Yield_And_Efficiency_Trend\",\n",109 " \"category\": \"Fundamental\",\n",110 " \"sub_category\": \"Value and Quality\",\n",111 " \"frequency\": \"Low-Frequency\",\n",112 " \"frequency_reasoning\": \"The inputs are fundamental accounting items (e.g., NET_CASH_FLOWS_OPER_ACT, CASH_RECP_SG_AND_RS) which are reported quarterly, making this a low-frequency factor.\",\n",113 " \"financial_logic\": \"This factor identifies undervalued companies with improving operational quality. It combines two components: 1) operating cash flow yield (Operating Cash Flow / Market Value) as a value metric, and 2) the 2-year average year-over-year trend in cash conversion efficiency (Cash from Sales / Cash Paid for Goods) as a quality metric. A high factor value suggests a company is generating strong cash flow relative to its price and is becoming more efficient at converting sales into cash, which may be overlooked by the market.\",\n",114 " \"core_formula_latex\": \"Div(Log(Mul(Div(NET_CASH_FLOWS_OPER_ACT, MarketValue), yoy_mean(Div(CASH_RECP_SG_AND_RS, CASH_PAY_GOODS_PURCH_SERV_REC), 2))), MarketValue)\",\n",115 " \"inputs_required\": [\n",116 " \"NET_CASH_FLOWS_OPER_ACT\",\n",117 " \"MarketValue\",\n",118 " \"CASH_RECP_SG_AND_RS\",\n",119 " \"CASH_PAY_GOODS_PURCH_SERV_REC\"\n",120 " ],\n",121 " \"parameters\": [\n",122 " 2\n",123 " ],\n",124 " \"calculation_type\": \"Cross-Sectional\",\n",125 " \"numpy_implementation_hint\": \"This requires a financial data provider. For `yoy_mean(X, 2)`, retrieve data for the current period (T), T-4 quarters, and T-8 quarters. Calculate two YoY changes and average them. The `Div`, `Mul`, `Log` operations are element-wise. Ensure proper alignment of fiscal periods.\"\n",126 " },\n",127 " {\n",128 " \"factor_name\": \"Profitability_to_Liability_Growth\",\n",129 " \"category\": \"Fundamental\",\n",130 " \"sub_category\": \"Growth and Quality\",\n",131 " \"frequency\": \"Low-Frequency\",\n",132 " \"frequency_reasoning\": \"The inputs are quarterly financial statement items (Net Profit and Total Current Liabilities), which defines it as a low-frequency factor.\",\n",133 " \"financial_logic\": \"This factor measures the year-over-year growth in a company's profitability relative to its short-term obligations. It calculates the ratio of net profit to total current liabilities and then computes its annual percentage change. A high positive value indicates that the company's ability to cover its short-term debts with profits is improving significantly, signaling enhanced financial health and operational efficiency that the market may be slow to price in.\",\n",134 " \"core_formula_latex\": \"yoy_chg(Div(Net_Profit_Excl_Min_Int_Inc, Tot_Cur_Liab), 1)\",\n",135 " \"inputs_required\": [\n",136 " \"Net_Profit_Excl_Min_Int_Inc\",\n",137 " \"Tot_Cur_Liab\"\n",138 " ],\n",139 " \"parameters\": [\n",140 " 1\n",141 " ],\n",142 " \"calculation_type\": \"Cross-Sectional\",\n",143 " \"numpy_implementation_hint\": \"Retrieve the required financial data for the current quarter (T) and the same quarter last year (T-4). Calculate the ratio `Ratio_T = Profit_T / Liability_T` and `Ratio_T-4 = Profit_T-4 / Liability_T-4`. The factor is `Ratio_T / Ratio_T-4 - 1`.\"\n",144 " },\n",145 " {\n",146 " \"factor_name\": \"Momentum_vs_Valuation_with_Flow_Acceleration\",\n",147 " \"category\": \"Momentum\",\n",148 " \"sub_category\": \"Relative Strength\",\n",149 " \"frequency\": \"Daily\",\n",150 " \"frequency_reasoning\": \"The formula relies on daily price and volume data (`close`, `volume`, `vwap`) with lookback windows ranging from 5 to 60 days.\",\n",151 " \"financial_logic\": \"This factor captures stocks with strong momentum that is confirmed by accelerating money flow. The first term, Log(Mean(Delta(close, 5), 10) / Med(close, 60)), measures short-term momentum relative to a longer-term median price, identifying stocks that are breaking out from their recent valuation anchor. The second term, (Corr(volume, vwap, 30) - Corr(volume, vwap, 10)), measures the change in price-volume correlation, with a positive value indicating that the link between volume and price has strengthened recently, suggesting an acceleration of capital flow into the stock.\",\n",152 " \"core_formula_latex\": \"Log(Mean(Delta(close, 5), 10) / Med(close, 60)) * (Corr(volume, vwap, 30) - Corr(volume, vwap, 10))\",\n",153 " \"inputs_required\": [\n",154 " \"close\",\n",155 " \"volume\",\n",156 " \"vwap\"\n",157 " ],\n",158 " \"parameters\": [\n",159 " 5,\n",160 " 10,\n",161 " 60,\n",162 " 30,\n",163 " 10\n",164 " ],\n",165 " \"calculation_type\": \"Rolling Window\",\n",166 " \"numpy_implementation_hint\": \"Calculate `df['close'].diff(5).rolling(10).mean() / df['close'].rolling(60).median()`. Calculate the difference between `df['volume'].rolling(30).corr(df['vwap'])` and `df['volume'].rolling(10).corr(df['vwap'])`. Multiply the log of the first term by the second term.\"\n",167 " },\n",168 " {\n",169 " \"factor_name\": \"Volatility_Normalized_Trend_Correlation\",\n",170 " \"category\": \"Momentum\",\n",171 " \"sub_category\": \"Trend Following\",\n",172 " \"frequency\": \"Daily\",\n",173 " \"frequency_reasoning\": \"The formula uses daily 'close' and 'volume' data with rolling windows of 10, 20, and 30 days, making it a daily frequency factor.\",\n",174 " \"financial_logic\": \"This factor identifies sustained trends by measuring the correlation between price and volume trends, normalized by their respective volatilities. It first calculates the 10-day slopes of price and volume to capture their immediate direction. It then computes the 20-day correlation between these two trend series. A high correlation indicates a persistent, volume-confirmed trend. Dividing by the product of price and volume standard deviations normalizes the signal, making it more sensitive to strong trend alignment during periods of stable volatility.\",\n",175 " \"core_formula_latex\": \"Corr(Slope(close, 10), Slope(volume, 10), 20) / (Std(close, 30) * Std(volume, 30))\",\n",176 " \"inputs_required\": [\n",177 " \"close\",\n",178 " \"volume\"\n",179 " ],\n",180 " \"parameters\": [\n",181 " 10,\n",182 " 10,\n",183 " 20,\n",184 " 30,\n",185 " 30\n",186 " ],\n",187 " \"calculation_type\": \"Rolling Window\",\n",188 " \"numpy_implementation_hint\": \"This involves nested rolling calculations. First, create two new series: rolling 10-day slope of close and rolling 10-day slope of volume. Then, calculate the 20-day rolling correlation between these two new slope series. Finally, divide this result by the product of the 30-day rolling std of close and the 30-day rolling std of volume.\"\n",189 " }\n",190 "]\n",191 "--------------------------------------------------\n"192 ]193 }194 ],195 "source": [196 "import os\n",197 "from google import genai\n",198 "import json\n",199 "from dotenv import load_dotenv \n",200 "\n",201 "# 加载 API Key\n",202 "load_dotenv()\n",203 "GOOGLE_API_KEY = os.getenv(\"GOOGLE_API_KEY\")\n",204 "\n",205 "if not GOOGLE_API_KEY:\n",206 " raise ValueError(\"GOOGLE_API_KEY not found.\")\n",207 "\n",208 "# 1. 配置模型\n",209 "gemini_client = genai.Client(api_key=GOOGLE_API_KEY)\n",210 "gemini_model = \"gemini-2.5-pro\"\n",211 "\n",212 "# 2. 核心 Prompt\n",213 "EXTRACT_PROMPT = \"\"\"\n",214 "Role: You are a Senior Quantitative Researcher.\n",215 "Task: Extract the CORE Alpha Factor mathematical logic from the document.\n",216 "\n",217 "**CRITICAL EXTRACTION SCOPE (Identify Document Type First):**\n",218 "TYPE A: RESEARCH PAPER\n",219 "- Goal: Extract ONLY the novel contribution.\n",220 "- Constraint: Ignore standard benchmarks (e.g., if extracting VPIN, ignore PIN).\n",221 "\n",222 "TYPE B: BOOK / HANDBOOK / SURVEY (Applies to this task)\n",223 "- Goal: Create a comprehensive library of indicators.\n",224 "- Constraint: **Extract ALL distinct indicators, moving averages, and filters defined in the text.**\n",225 "- Rule: Do NOT ignore well-known indicators (like Wilder MA) or variations (like Sine WMA).\n",226 "- Rule: Treat \"Digital Signal Processing (DSP) Filters\" and \"Smoothers\" (e.g., SuperSmoother, Ehlers Filters) as valid factors to be extracted.\n",227 "- Rule: Ensure coverage of ALL chapters, including \"Exotic\" or \"Advanced\" sections at the end of the text.\n",228 "\n",229 "**CRITICAL INSTRUCTION ON FREQUENCY INFERENCE:**\n",230 "You must infer the 'frequency' based on the input variables and logic:\n",231 "1. **High-Frequency/Tick:** If logic involves Order Book (Bid/Ask sizes, levels), Tick data, Order Flow, or Microstructure theories (e.g., VPIN, OFI).\n",232 "2. **Intraday:** If logic involves minute bars (e.g., 5-min Close), VWAP, or realized volatility from high-freq data.\n",233 "3. **Daily:** If logic uses standard OHLCV (Open/High/Low/Close/Volume) at a daily level, or fundamental data.\n",234 "\n",235 "Output Format: A JSON List of objects.\n",236 "\n",237 "JSON Schema for each object:\n",238 "1. \"factor_name\": Name of the factor.\n",239 "2. \"category\": \"Choose one: [Momentum, Volatility, Mean Reversion, Market Microstructure, Sentiment, Fundamental]\".\n",240 "3. \"sub_category\": \"Specific technique (e.g., Order Flow Imbalance, Moving Average Crossover)\".\n",241 "4. \"frequency\": \"Choose one: [High-Frequency/Tick, Intraday, Daily, Low-Frequency]\".\n",242 "5. \"frequency_reasoning\": \"Explain WHY you chose this frequency. (e.g., 'Uses Bid-Ask spread which requires L2 data')\",\n",243 "6. \"financial_logic\": A concise explanation of why this factor works (e.g., \"Captures order flow imbalance to predict short-term price pressure\").\n",244 "7. \"core_formula_latex\": The exact mathematical formula in LaTeX format.\n",245 "8. \"inputs_required\": List of data columns needed (e.g., [\"Bid1_Price\", \"Ask1_Price\", \"Bid1_Size\", \"Ask1_Size\"]).\n",246 "9. \"parameters\": List of numeric hyperparameters used inside functions (e.g., window size in rolling functions, decay rates). \n",247 " - CRITICAL RULE: Do NOT extract numbers that are part of a column name/variable name. \n",248 " - Example: For input \"adv50\", \"adv50\" is the Input Name. Do NOT extract 50 as a parameter.\n",249 " - Example: For \"delta(close, 5)\", 5 IS a parameter.\n",250 "10. \"calculation_type\": Choose one from [\"Rolling Window\", \"Recursive/Stateful\", \"Cross-Sectional\"].\n",251 " - Note: For HFT, usually it's \"Rolling Window\" or \"Recursive\".\n",252 "11. \"numpy_implementation_hint\": Crucial! Give a specific hint on how to implement this using Numpy.\n",253 " - Example: \"Use np.convolve for weighted moving average\" or \"Use (Bid_Size - Ask_Size) / (Bid_Size + Ask_Size)\".\n",254 "\n",255 "Constraint: \n",256 "If the document is a Research Paper: Extract ONLY the main proposed model (the contribution). Ignore benchmarks. \n",257 "If the document is a Book/Handbook: Extract ALL distinct technical indicators defined in the text, treating each as a primary tool.\n",258 "Specific Constraint for Flash Model: Do NOT paraphrase the formula. Transcription must be EXACT character-for-character matching the image. If uncertain, verify against the surrounding text definition.\n",259 "\"\"\"\n",260 "\n",261 "# 3. 执行\n",262 "def extract_factor_from_pdf(pdf_path):\n",263 " print(f\"Start Processing: {pdf_path}\")\n",264 " \n",265 " # Check file\n",266 " if not os.path.exists(pdf_path):\n",267 " print(f\"Error: File not found at {pdf_path}\")\n",268 " return\n",269 "\n",270 " try:\n",271 " # Upload to Gemini (Leveraging Vision Capabilities)\n",272 " print(\"Uploading to Google AI Studio...\")\n",273 " sample_file = gemini_client.files.upload(file=pdf_path)\n",274 " # Generate Content\n",275 " print(\"Gemini is analyzing the math logic...\")\n",276 " \n",277 " # Enforce JSON mode output if supported, or just ask nicely in prompt\n",278 " response = gemini_client.models.generate_content(\n",279 " model=gemini_model,\n",280 " contents=[sample_file, EXTRACT_PROMPT],\n",281 " config={\"response_mime_type\": \"application/json\"} \n",282 " )\n",283 " \n",284 " # Parse JSON just to verify validity\n",285 " factor_card = json.loads(response.text)\n",286 " \n",287 " # Output Handling\n",288 " print(f\"\\n[Success] Extraction Complete! Obtain {len(factor_card)} Factors:\\n\")\n",289 " print(\"-\" * 50)\n",290 " print(json.dumps(factor_card, indent=4, ensure_ascii=False))\n",291 " \n",292 " print(\"-\" * 50)\n",293 " # Save Factor\n",294 " with open(\"C:/Users/Minze Li/OneDrive/MAFN/CITIC Intern/CODE/data/raw_extracted_factors.json\", \"w\", encoding='utf-8') as f:\n",295 " json.dump(factor_card, f, indent=4, ensure_ascii=False)\n",296 "\n",297 " return\n",298 "\n",299 " except Exception as e:\n",300 " print(f\"\\n[Error]: {e}\")\n",301 "\n",302 "# 4. 主程序入口\n",303 "if __name__ == \"__main__\":\n",304 " target_pdf = \"C:/Users/Minze Li/OneDrive/MAFN/CITIC Intern/QuantPaper/20251227_Guojin_Alpha_LLM_Framework.pdf\"\n",305 " extract_factor_from_pdf(target_pdf)"306 ]307 },308 {309 "cell_type": "markdown",310 "id": "ce23888c",311 "metadata": {},312 "source": [313 "# 2 去重存库"314 ]315 },316 {317 "cell_type": "markdown",318 "id": "467ea840",319 "metadata": {},320 "source": [321 "参考 Alpha-GPT，基于 category/sub_category 字段进行分层文件存储。使用 Bert模型 embedding + 余弦相似度 进行逻辑语义去重。 \n",322 "\n",323 "注意：此处仅去除了逻辑描述高度相似的因子，并未基于因子回测数据去除统计相关性高的因子。"324 ]325 },326 {327 "cell_type": "code",328 "execution_count": 18,329 "id": "15491866",330 "metadata": {},331 "outputs": [332 {333 "name": "stdout",334 "output_type": "stream",335 "text": [336 "正在重建索引...\n",337 "开始处理: 24 个待分类因子\n",338 "\n",339 "[1/24] 入库: [Momentum/Moving_Average] General_Weighted_Moving_Average\n",340 "[2/24] 入库: [Momentum/Moving_Average] SMA\n",341 "[3/24] 入库: [Momentum/Moving_Average] LMA\n",342 "[4/24] 入库: [Momentum/Moving_Average] EMA\n",343 "[5/24] 入库: [Momentum/Moving_Average] Wilder_Moving_Average\n",344 "[6/24] 入库: [Momentum/Crossover_Strategy] P-MA\n",345 "[7/24] 入库: [Momentum/Crossover_Strategy] MA-C\n",346 "[8/24] 入库: [Momentum/Rate_of_Change] MOM\n",347 "[9/24] 入库: [Momentum/Moving_Average] MA-D\n",348 "[10/24] 入库: [Momentum/Moving_Average] HMA\n",349 "[11/24] 入库: [Momentum/Oscillator] MACD\n",350 "[12/24] 入库: [Momentum/Moving_Average] SWMA\n",351 "[13/24] 入库: [Momentum/Moving_Average] TMA\n",352 "[14/24] 入库: [Momentum/Moving_Average] DEMA\n",353 "[15/24] 入库: [Momentum/Moving_Average] TEMA\n",354 "[16/24] 入库: [Momentum/Moving_Average] GMA\n",355 "[17/24] 入库: [Momentum/Moving_Average] ALMA\n",356 "[18/24] 入库: [Momentum/Moving_Average] ZLEMA\n",357 "[19/24] 入库: [Momentum/Moving_Average] Tillson_T3_Moving_Average\n",358 "[20/24] 入库: [Momentum/Moving_Average] EPMA\n",359 "[21/24] 入库: [Momentum/DSP_Filter] Butterworth_Filter\n",360 "[22/24] 入库: [Momentum/DSP_Filter] SuperSmoother_Filter\n",361 "[23/24] 入库: [Momentum/DSP_Filter] Regularized_Filter\n",362 "[24/24] 入库: [Momentum/DSP_Filter] Instantaneous_Trendline_Filter\n",363 "处理完成！\n",364 "新增因子: 24\n",365 "跳过因子: 0\n"366 ]367 }368 ],369 "source": [370 "import os\n",371 "import json\n",372 "import numpy as np\n",373 "from sentence_transformers import SentenceTransformer\n",374 "from sklearn.metrics.pairwise import cosine_similarity\n",375 "\n",376 "# 配置\n",377 "RAW_FILE = \"data/raw_extracted_factors.json\" # 单个待分类文件\n",378 "LIB_DIR = \"data/alpha_lib\"\n",379 "SIMILARITY_THRESHOLD = 0.88 # 去重阈值\n",380 "\n",381 "class HierarchicalLibrarian:\n",382 " def __init__(self):\n",383 " self.model = SentenceTransformer('all-MiniLM-L6-v2')\n",384 " self.global_embeddings = [] \n",385 " self.global_metadata = [] # 存储现有因子的元数据\n",386 " \n",387 " # 加载现有库的向量（如果有）\n",388 " self._load_existing_state()\n",389 "\n",390 " def _load_existing_state(self):\n",391 " \"\"\"遍历 LIB_DIR 下所有 JSON 建立初始向量索引\"\"\"\n",392 " if not os.path.exists(LIB_DIR):\n",393 " return\n",394 " \n",395 " print(\"正在重建索引...\")\n",396 " for root, _, files in os.walk(LIB_DIR):\n",397 " for file in files:\n",398 " if file.endswith(\".json\"):\n",399 " filepath = os.path.join(root, file)\n",400 " try:\n",401 " with open(filepath, 'r', encoding='utf-8') as f:\n",402 " data = json.load(f)\n",403 " \n",404 " # 每个文件应该是单个因子字典\n",405 " if isinstance(data, dict):\n",406 " logic = data.get('financial_logic', '')\n",407 " if logic:\n",408 " self.global_metadata.append({\n",409 " \"path\": filepath,\n",410 " \"logic\": logic\n",411 " })\n",412 " except Exception as e:\n",413 " print(f\"读取文件失败 {filepath}: {e}\")\n",414 " \n",415 " if self.global_metadata:\n",416 " texts = [m['logic'] for m in self.global_metadata]\n",417 " self.global_embeddings = self.model.encode(texts)\n",418 " print(f\" -> 已索引 {len(self.global_embeddings)} 个现有因子\")\n",419 "\n",420 " def organize(self):\n",421 " \"\"\"读取未分类因子文件并分类存储\"\"\"\n",422 " if not os.path.exists(RAW_FILE):\n",423 " print(f\"未找到待分类文件: {RAW_FILE}\")\n",424 " return\n",425 " \n",426 " # 读取待分类因子列表\n",427 " try:\n",428 " with open(RAW_FILE, 'r', encoding='utf-8') as f:\n",429 " factors = json.load(f)\n",430 " \n",431 " # 确保 factors 是列表\n",432 " if isinstance(factors, dict):\n",433 " factors = [factors]\n",434 " elif not isinstance(factors, list):\n",435 " print(f\"文件格式错误，应为因子列表\")\n",436 " return\n",437 " \n",438 " print(f\"开始处理: {len(factors)} 个待分类因子\\n\")\n",439 " \n",440 " added_count = 0\n",441 " skipped_count = 0\n",442 " \n",443 " for i, factor in enumerate(factors, 1):\n",444 " if isinstance(factor, dict):\n",445 " result = self._process_single_factor(factor, i, len(factors))\n",446 " if result:\n",447 " added_count += 1\n",448 " else:\n",449 " skipped_count += 1\n",450 " \n",451 " print(f\"处理完成！\")\n",452 " print(f\"新增因子: {added_count}\")\n",453 " print(f\"跳过因子: {skipped_count}\")\n",454 " \n",455 " except Exception as e:\n",456 " print(f\"处理文件失败: {e}\")\n",457 "\n",458 " def _process_single_factor(self, factor, current_idx, total):\n",459 " \"\"\"处理单个因子, 用逻辑描述计算cosine similarity\"\"\"\n",460 " # 1. 提取逻辑描述\n",461 " logic = factor.get('financial_logic', '')\n",462 " if not logic:\n",463 " print(f\"[{current_idx}/{total}] 跳过：缺少 financial_logic\")\n",464 " return False\n",465 " \n",466 " # 2. 向量化\n",467 " new_vec = self.model.encode([logic])\n",468 " \n",469 " # 3. 语义查重\n",470 " if len(self.global_embeddings) > 0:\n",471 " sims = cosine_similarity(new_vec, self.global_embeddings)[0]\n",472 " max_sim = np.max(sims)\n",473 " if max_sim > SIMILARITY_THRESHOLD:\n",474 " name = factor.get('factor_name', 'Unknown')\n",475 " print(f\"[{current_idx}/{total}] 重复因子 '{name}' (相似度 {max_sim:.2f}) - 跳过\")\n",476 " return False\n",477 "\n",478 " # 4. 构建分层存储路径\n",479 " cat = factor.get('category', 'Uncategorized').replace(\" \", \"_\")\n",480 " sub_cat = factor.get('sub_category', 'General').replace(\" \", \"_\")\n",481 " \n",482 " raw_name = factor.get('factor_name', 'Unknown')\n",483 " name = self._clean_filename(raw_name)\n",484 " \n",485 " save_dir = os.path.join(LIB_DIR, cat, sub_cat)\n",486 " os.makedirs(save_dir, exist_ok=True)\n",487 " \n",488 " save_path = os.path.join(save_dir, f\"{name}.json\")\n",489 " \n",490 " # 如果文件已存在，添加序号避免覆盖\n",491 " counter = 1\n",492 " original_path = save_path\n",493 " while os.path.exists(save_path):\n",494 " save_path = original_path.replace('.json', f'_{counter}.json')\n",495 " counter += 1\n",496 " \n",497 " # 5. 写入文件\n",498 " with open(save_path, 'w', encoding='utf-8') as f:\n",499 " json.dump(factor, f, indent=4, ensure_ascii=False)\n",500 " \n",501 " # 6. 更新内存索引\n",502 " self.global_metadata.append({\"path\": save_path, \"logic\": logic})\n",503 " if len(self.global_embeddings) == 0:\n",504 " self.global_embeddings = new_vec\n",505 " else:\n",506 " self.global_embeddings = np.vstack([self.global_embeddings, new_vec])\n",507 " \n",508 " print(f\"[{current_idx}/{total}] 入库: [{cat}/{sub_cat}] {name}\")\n",509 " return True\n",510 "\n",511 " def _clean_filename(self, raw_name):\n",512 " \"\"\"清理文件名\"\"\"\n",513 " import re\n",514 " \n",515 " match = re.search(r'\\(([A-Z\\-]+)\\)', raw_name)\n",516 " if match:\n",517 " abbr = match.group(1)\n",518 " # 移除多余的破折号\n",519 " return abbr.replace('--', '-')\n",520 " \n",521 " # 如果没有缩写，清理完整名称\n",522 " # 移除括号及内容\n",523 " name = re.sub(r'\\([^)]*\\)', '', raw_name)\n",524 " # 替换特殊字符为下划线\n",525 " name = re.sub(r'[^\\w\\s-]', '_', name)\n",526 " # 替换空格和多个下划线/破折号为单个下划线\n",527 " name = re.sub(r'[\\s_-]+', '_', name)\n",528 " # 移除首尾下划线\n",529 " name = name.strip('_')\n",530 " # 限制长度（可选）\n",531 " if len(name) > 50:\n",532 " name = name[:50].rstrip('_')\n",533 " \n",534 " return name if name else 'Unknown'\n",535 "\n",536 "if __name__ == \"__main__\":\n",537 " librarian = HierarchicalLibrarian()\n",538 " librarian.organize()"539 ]540 },541 {542 "cell_type": "markdown",543 "id": "324f3c87",544 "metadata": {},545 "source": [546 "# 3 生成代码"547 ]548 },549 {550 "cell_type": "markdown",551 "id": "8a28c5f6",552 "metadata": {},553 "source": [554 "Self-correlation with feedback loop.\n",555 "\n",556 "读取因子定义JSON $\\rightarrow$ 通过LLM生成Python代码 $\\rightarrow$ 静态验证（检查是否有循环（高效性）、NaN处理（鲁棒性）、类型提示等） $\\rightarrow$ 运行时测试（8种边界情况） $\\rightarrow$ 智能重试（提供具体错误反馈） $\\rightarrow$ 输出高性能代码（纯numpy/scipy实现，完整的鲁棒性处理）。如果生成失败，保存错误日志供调试。\n",557 "\n",558 "可能三次重试次数不够。或者说三次重试失败后可以考虑重新生成一份新代码？\n",559 "\n",560 "EMA出了点问题。"561 ]562 },563 {564 "cell_type": "markdown",565 "id": "b9a32623",566 "metadata": {},567 "source": [568 "#### 3.1 Prompt"569 ]570 },571 {572 "cell_type": "code",573 "execution_count": null,574 "id": "5a069d4a",575 "metadata": {},576 "outputs": [],577 "source": [578 "\n",579 "# ============================================================================\n",580 "# ENHANCED PROMPTS\n",581 "# ============================================================================\n",582 "\n",583 "CODING_PROMPT = \"\"\"\n",584 "Role: You are a Lead Quantitative Developer at a High-Frequency Trading firm.\n",585 "Your job is to translate Factor Logic (JSON) into production-grade Python code for high-frequency trading.\n",586 "\n",587 "**CRITICAL CODING STANDARDS (NON-NEGOTIABLE):**\n",588 "\n",589 "1. **Pure Numpy/Vectorization:**\n",590 " - Core calculation MUST use numpy vectorization\n",591 " - **ABSOLUTELY NO explicit 'for' loops** in calculations\n",592 " - For recursive logic (EMA, RSI), use scipy.signal.lfilter, NOT loops\n",593 " - Do NOT use pandas.apply, pandas.iterrows, pandas.resample\n",594 "\n",595 "2. **Data Robustness (MANDATORY - WILL BE TESTED):**\n",596 " \n",597 " a) **Handle Missing Data Explicitly:**\n",598 " - Leading NaNs: Preserve in output, start calculation at first valid index\n",599 " - Trailing NaNs: Preserve in output\n",600 " - Middle NaNs: Choose one strategy and document it clearly:\n",601 " * Strategy A (IIR Filters): NaN propagates forward (for EMA, RSI, DEMA)\n",602 " * Strategy B (Rolling Windows): Skip NaNs, calculate on valid data only (for SMA, StdDev, Correlation)\n",603 " * Strategy C (Forward Fill): Use last valid value (USE SPARINGLY, document why)\n",604 " \n",605 " b) **Handle Edge Cases (MANDATORY):**\n",606 " - Empty array (len=0): Return empty array with dtype=float64\n",607 " - All-NaN array: Return all-NaN array with same shape\n",608 " - Single valid value: Must not crash, handle gracefully\n",609 " - Array with inf/-inf: Replace with NaN or handle explicitly\n",610 " \n",611 " c) **Implementation Pattern:**\n",612 "```python\n",613 " # Step 1: Initialize output\n",614 " result = np.full_like(data, np.nan, dtype=np.float64)\n",615 " \n",616 " # Step 2: Check for empty/all-NaN\n",617 " if len(data) == 0:\n",618 " return np.array([], dtype=np.float64)\n",619 " \n",620 " valid_mask = ~np.isnan(data)\n",621 " if not np.any(valid_mask):\n",622 " return result # All NaNs\n",623 " \n",624 " # Step 3: Find first valid index\n",625 " first_valid_idx = np.argmax(valid_mask)\n",626 " \n",627 " # Step 4: Perform calculation on valid data\n",628 " # [Your vectorized calculation here]\n",629 " \n",630 " # Step 5: Place results back\n",631 " result[first_valid_idx:] = calculated_values\n",632 "```\n",633 "\n",634 "3. **Numerical Stability:**\n",635 " - Division: Use np.divide with 'where' clause OR add epsilon (1e-10)\n",636 " - Example: `np.divide(a, b, out=np.full_like(a, np.nan), where=b!=0)`\n",637 " - Variance/Std: Use ddof=1, handle zero-variance cases\n",638 " - Logs: `np.log(np.maximum(x, 1e-10))` to avoid log(0)\n",639 "\n",640 "4. **Code Structure (MANDATORY FORMAT):**\n",641 "```python\n",642 " import numpy as np\n",643 " from scipy import signal # Only if needed for IIR filters\n",644 " import pandas as pd\n",645 " \n",646 " def _factor_name_numpy(data: np.ndarray, param1: float, param2: float) -> np.ndarray:\n",647 " '''\n",648 " Core vectorized calculation.\n",649 " \n",650 " Mathematical Formula:\n",651 " [Insert LaTeX or clear math description]\n",652 " \n",653 " NaN Handling Strategy:\n",654 " [Specify: Strategy A/B/C - explain choice]\n",655 " \n",656 " Edge Cases:\n",657 " - Empty input: Returns empty array\n",658 " - All-NaN input: Returns all-NaN array\n",659 " - Middle NaNs: [Describe behavior]\n",660 " \n",661 " Args:\n",662 " data: Input price/volume array (1D numpy array)\n",663 " param1: [Description, will be rounded if needed]\n",664 " param2: [Description]\n",665 " \n",666 " Returns:\n",667 " Calculated factor values (same shape as input, dtype=float64)\n",668 " \n",669 " Example:\n",670 " >>> data = np.array([100.0, 101.0, np.nan, 103.0, 104.0])\n",671 " >>> result = _factor_name_numpy(data, param1=10, param2=0.5)\n",672 " >>> # Expected output based on strategy\n",673 " '''\n",674 " # 1. Parameter sanitization (handle GA artifacts)\n",675 " window = int(round(param1)) # If param expects integer\n",676 " \n",677 " # 2. Input validation & edge cases\n",678 " if len(data) == 0:\n",679 " return np.array([], dtype=np.float64)\n",680 " \n",681 " result = np.full_like(data, np.nan, dtype=np.float64)\n",682 " \n",683 " valid_mask = ~np.isnan(data)\n",684 " if not np.any(valid_mask):\n",685 " return result\n",686 " \n",687 " first_valid_idx = np.argmax(valid_mask)\n",688 " \n",689 " # 3. Core calculation (pure numpy/scipy)\n",690 " # [YOUR VECTORIZED CODE HERE]\n",691 " \n",692 " # 4. Handle numerical stability\n",693 " # Use np.divide with where, or add epsilon\n",694 " \n",695 " # 5. Return results\n",696 " result[first_valid_idx:] = calculated_values\n",697 " return result\n",698 " \n",699 " def factor_name(df: pd.DataFrame, param1: float = 10, param2: float = 0.5) -> pd.Series:\n",700 " '''\n",701 " Pandas wrapper for integration with factor library.\n",702 " \n",703 " Args:\n",704 " df: DataFrame with required columns (e.g., 'close', 'volume')\n",705 " param1: [Description]\n",706 " param2: [Description]\n",707 " \n",708 " Returns:\n",709 " Series with factor values, indexed same as input DataFrame\n",710 " '''\n",711 " data = df['close'].values # Or other required column\n",712 " result = _factor_name_numpy(data, param1, param2)\n",713 " return pd.Series(result, index=df.index, name=f'factor_{param1}_{param2}')\n",714 "```\n",715 "\n",716 "5. **Type Hints:**\n",717 " - Use type hints for all function signatures\n",718 " - Example: `def func(data: np.ndarray, n: int) -> np.ndarray:`\n",719 "\n",720 "6. **Documentation Requirements:**\n",721 " - Docstring MUST include: formula, NaN strategy, edge cases, example\n",722 " - Inline comments for non-obvious logic\n",723 " - Document any epsilon values used (e.g., \"1e-10 prevents log(0)\")\n",724 "\n",725 "7. **Handling Genetic Algorithm Artifacts:**\n",726 " - Round float parameters to integers where logical (window sizes, delays)\n",727 " - Example: `delta(vwap, 1.29456)` → `window = int(round(1.29456))`\n",728 "\n",729 "8. **IIR Filter Initial State (CRITICAL FOR RECURSIVE INDICATORS):**\n",730 " \n",731 " For recursive indicators (EMA, RSI, MACD, etc.) using scipy.signal.lfilter:\n",732 " \n",733 " **Mathematical Foundation:**\n",734 " - lfilter uses Direct Form II Transposed: y[n] = b[0]*x[n] + z[n-1]\n",735 " - For EMA: b = [α], a = [1, -(1-α)], where α = 2/(span+1)\n",736 " - We want first output y[0] = x[0] (pandas ewm(adjust=False) behavior)\n",737 " \n",738 " **Derivation:**\n",739 "```\n",740 " y[0] = b[0]*x[0] + z[-1]\n",741 " x[0] = α*x[0] + z[-1] (we want y[0] = x[0])\n",742 " z[-1] = x[0]*(1 - α) (solve for z[-1])\n",743 "```\n",744 " \n",745 " **Correct Implementation Pattern:**\n",746 "```python\n",747 " # For EMA (single-pole IIR filter)\n",748 " alpha = 2.0 / (span + 1.0)\n",749 " b = np.array([alpha])\n",750 " a = np.array([1, -(1.0 - alpha)])\n",751 " \n",752 " # CORRECT initial state\n",753 " initial_value = valid_data[0]\n",754 " zi = np.array([initial_value * (1.0 - alpha)])\n",755 " \n",756 " # Apply filter\n",757 " ema_values = signal.lfilter(b, a, valid_data, zi=zi)[0]\n",758 "```\n",759 " \n",760 " **For Two-Pole Filters (e.g., SuperSmoother):**\n",761 "```python\n",762 " # Two-pole Butterworth: y[n] = c0*x[n] + c0*x[n-1] - c1*y[n-1] - c2*y[n-2]\n",763 " b = np.array([c0, c0, 0])\n",764 " a = np.array([1, c1, c2])\n",765 " \n",766 " # CORRECT: Use lfiltic for multi-pole filters\n",767 " initial_value = valid_data[0]\n",768 " zi = signal.lfiltic(\n",769 " b, a, \n",770 " y=[initial_value, initial_value], # Past outputs\n",771 " x=[initial_value, initial_value] # Past inputs\n",772 " )\n",773 " \n",774 " filtered_values = signal.lfilter(b, a, valid_data, zi=zi)[0]\n",775 "```\n",776 " \n",777 " **Common Mistakes to AVOID:**\n",778 "```python\n",779 " # WRONG: Direct use of x[0] for single-pole\n",780 " zi = np.array([valid_data[0]]) # This gives y[0] = (1+α)*x[0]\n",781 " \n",782 " # WRONG: Not using lfiltic for multi-pole\n",783 " zi = np.array([valid_data[0], valid_data[0]]) # Incorrect structure\n",784 " \n",785 " # WRONG: Not setting initial state at all\n",786 " ema_values = signal.lfilter(b, a, valid_data) # Defaults to zi=0\n",787 "```\n",788 " \n",789 " **Verification (MUST Include in Docstring Example):**\n",790 "```python\n",791 " # Test that first output equals first input\n",792 " data = np.array([100.0, 101.0, 102.0])\n",793 " result = ema_numpy(data, span=20)\n",794 " assert abs(result[0] - 100.0) < 1e-10, \"First EMA value must equal first price\"\n",795 "```\n",796 "\n",797 "9. **Output Requirements:**\n",798 " - Return ONLY executable Python code\n",799 " - No explanatory text before or after code\n",800 " - Code must be ready to save as .py file\n",801 "\n",802 "**TESTING CHECKLIST (Your code will be tested on these):**\n",803 "□ Empty array input\n",804 "□ All-NaN array input\n",805 "□ Leading NaNs (first 10 values are NaN)\n",806 "□ Middle NaNs (random NaN in middle)\n",807 "□ Single value array\n",808 "□ Array with inf/-inf values\n",809 "□ Division by zero cases\n",810 "□ Zero variance/std cases (if applicable)\n",811 "\"\"\"\n",812 "\n",813 "NAN_HANDLING_GUIDE = \"\"\"\n",814 "**NaN Handling Strategy Selection Guide:**\n",815 "\n",816 "**Strategy A - Propagation (for IIR/Recursive Filters):**\n",817 "Use when: Factor uses recursive calculation (EMA, RSI, MACD)\n",818 "Behavior: Middle NaN causes all subsequent values to be NaN\n",819 "Implementation: Natural behavior of scipy.signal.lfilter\n",820 "Example factors: EMA, DEMA, RSI, Bollinger Bands\n",821 "\n",822 "**Strategy B - Skip & Map (for Rolling Windows):**\n",823 "Use when: Factor uses rolling window (SMA, rolling std, correlation)\n",824 "Behavior: Calculate only on valid data, preserve NaN positions\n",825 "Implementation:\n",826 "```python\n",827 "valid_mask = ~np.isnan(data)\n",828 "valid_indices = np.where(valid_mask)[0]\n",829 "valid_data = data[valid_indices]\n",830 "# Calculate on valid_data\n",831 "# Map results back to original indices\n",832 "result[valid_indices[window-1:]] = calculated_values\n",833 "```\n",834 "Example factors: SMA, Rolling Std, Rolling Correlation, Momentum\n",835 "\n",836 "**Strategy C - Forward Fill (RARE - Document Why):**\n",837 "Use when: Missing data is truly negligible and interpolation is justified\n",838 "Behavior: Fill NaN with last valid value\n",839 "Implementation: Use pandas ffill() with clear documentation\n",840 "Example factors: Rarely justified - only for specific tick data scenarios\n",841 "\"\"\""842 ]843 },844 {845 "cell_type": "markdown",846 "id": "2a88081e",847 "metadata": {},848 "source": [849 "#### 3.2 Code"850 ]851 },852 {853 "cell_type": "code",854 "execution_count": null,855 "id": "ec9aded7",856 "metadata": {},857 "outputs": [858 {859 "name": "stdout",860 "output_type": "stream",861 "text": [862 "[2025-12-29 01:38:48] [INFO] ================================================================================\n",863 "[2025-12-29 01:38:48] [INFO] STARTING FACTOR CODE GENERATION FACTORY\n",864 "[2025-12-29 01:38:48] [INFO] Model: gemini-2.5-pro\n",865 "[2025-12-29 01:38:48] [INFO] Force regenerate: False\n",866 "[2025-12-29 01:38:48] [INFO] Test mode: False\n",867 "[2025-12-29 01:38:48] [INFO] ================================================================================\n",868 "[2025-12-29 01:38:48] [INFO] Found 24 factor definitions\n",869 "[2025-12-29 01:38:48] [INFO] \n",870 "[1/24] Processing: Double Moving Average Crossover (MA-C)\n",871 "[2025-12-29 01:38:48] [INFO] 🔄 Double Moving Average Crossover (MA-C): Attempt 1/3\n",872 "[2025-12-29 01:40:18] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",873 "[2025-12-29 01:40:18] [INFO] 🔄 Double Moving Average Crossover (MA-C): Attempt 2/3\n",874 "[2025-12-29 01:40:50] [INFO] Static validation passed\n"875 ]876 },877 {878 "name": "stderr",879 "output_type": "stream",880 "text": [881 "C:\\Users\\Minze Li\\AppData\\Local\\Temp\\ipykernel_34456\\472224630.py:397: UserWarning: A NumPy version >=1.23.5 and <2.3.0 is required for this version of SciPy (detected version 2.3.3)\n",882 " from scipy import signal\n"883 ]884 },885 {886 "name": "stdout",887 "output_type": "stream",888 "text": [889 "[2025-12-29 01:40:57] [INFO] Runtime tests passed\n",890 "[2025-12-29 01:40:57] [SUCCESS] SUCCESS: Double Moving Average Crossover (MA-C) (attempts: 3)\n",891 "[2025-12-29 01:40:57] [INFO] \n",892 "[2/24] Processing: Price and Moving Average Crossover (P-MA)\n",893 "[2025-12-29 01:40:57] [INFO] 🔄 Price and Moving Average Crossover (P-MA): Attempt 1/3\n",894 "[2025-12-29 01:41:33] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",895 "[2025-12-29 01:41:33] [INFO] 🔄 Price and Moving Average Crossover (P-MA): Attempt 2/3\n",896 "[2025-12-29 01:42:04] [INFO] Static validation passed\n",897 "[2025-12-29 01:42:04] [INFO] Runtime tests passed\n",898 "[2025-12-29 01:42:04] [SUCCESS] SUCCESS: Price and Moving Average Crossover (P-MA) (attempts: 3)\n",899 "[2025-12-29 01:42:04] [INFO] \n",900 "[3/24] Processing: Butterworth Filter\n",901 "[2025-12-29 01:42:04] [INFO] 🔄 Butterworth Filter: Attempt 1/3\n",902 "[2025-12-29 01:42:37] [INFO] Static validation passed\n",903 "[2025-12-29 01:42:37] [INFO] Runtime tests passed\n",904 "[2025-12-29 01:42:37] [SUCCESS] SUCCESS: Butterworth Filter (attempts: 3)\n",905 "[2025-12-29 01:42:37] [INFO] \n",906 "[4/24] Processing: Instantaneous Trendline Filter\n",907 "[2025-12-29 01:42:37] [INFO] 🔄 Instantaneous Trendline Filter: Attempt 1/3\n",908 "[2025-12-29 01:43:08] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",909 "[2025-12-29 01:43:08] [INFO] 🔄 Instantaneous Trendline Filter: Attempt 2/3\n",910 "[2025-12-29 01:44:09] [INFO] Static validation passed\n",911 "[2025-12-29 01:44:09] [INFO] Runtime tests passed\n",912 "[2025-12-29 01:44:09] [SUCCESS] SUCCESS: Instantaneous Trendline Filter (attempts: 3)\n",913 "[2025-12-29 01:44:09] [INFO] \n",914 "[5/24] Processing: Regularized Filter\n",915 "[2025-12-29 01:44:09] [INFO] 🔄 Regularized Filter: Attempt 1/3\n",916 "[2025-12-29 01:44:42] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",917 "[2025-12-29 01:44:42] [INFO] 🔄 Regularized Filter: Attempt 2/3\n",918 "[2025-12-29 01:45:05] [INFO] Static validation passed\n",919 "[2025-12-29 01:45:05] [INFO] Runtime tests passed\n",920 "[2025-12-29 01:45:05] [SUCCESS] SUCCESS: Regularized Filter (attempts: 3)\n",921 "[2025-12-29 01:45:05] [INFO] \n",922 "[6/24] Processing: SuperSmoother Filter\n",923 "[2025-12-29 01:45:05] [INFO] 🔄 SuperSmoother Filter: Attempt 1/3\n",924 "[2025-12-29 01:45:56] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",925 "[2025-12-29 01:45:56] [INFO] 🔄 SuperSmoother Filter: Attempt 2/3\n",926 "[2025-12-29 01:46:52] [INFO] Static validation passed\n",927 "[2025-12-29 01:46:52] [INFO] Runtime tests passed\n",928 "[2025-12-29 01:46:52] [SUCCESS] SUCCESS: SuperSmoother Filter (attempts: 3)\n",929 "[2025-12-29 01:46:52] [INFO] \n",930 "[7/24] Processing: Arnaud Legoux Moving Average (ALMA)\n",931 "[2025-12-29 01:46:52] [INFO] 🔄 Arnaud Legoux Moving Average (ALMA): Attempt 1/3\n",932 "[2025-12-29 01:47:32] [INFO] Static validation passed\n",933 "[2025-12-29 01:47:32] [INFO] Runtime tests passed\n",934 "[2025-12-29 01:47:32] [SUCCESS] SUCCESS: Arnaud Legoux Moving Average (ALMA) (attempts: 3)\n",935 "[2025-12-29 01:47:32] [INFO] \n",936 "[8/24] Processing: Double Exponential Moving Average (DEMA)\n",937 "[2025-12-29 01:47:32] [INFO] 🔄 Double Exponential Moving Average (DEMA): Attempt 1/3\n",938 "[2025-12-29 01:48:31] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",939 "[2025-12-29 01:48:31] [INFO] 🔄 Double Exponential Moving Average (DEMA): Attempt 2/3\n",940 "[2025-12-29 01:49:23] [INFO] Static validation passed\n",941 "[2025-12-29 01:49:23] [INFO] Runtime tests passed\n",942 "[2025-12-29 01:49:23] [SUCCESS] SUCCESS: Double Exponential Moving Average (DEMA) (attempts: 3)\n",943 "[2025-12-29 01:49:23] [INFO] \n",944 "[9/24] Processing: Exponential Moving Average (EMA)\n",945 "[2025-12-29 01:49:23] [INFO] 🔄 Exponential Moving Average (EMA): Attempt 1/3\n",946 "[2025-12-29 01:50:10] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",947 "[2025-12-29 01:50:10] [INFO] 🔄 Exponential Moving Average (EMA): Attempt 2/3\n",948 "[2025-12-29 01:50:58] [INFO] Static validation passed\n",949 "[2025-12-29 01:50:58] [INFO] Runtime tests passed\n",950 "[2025-12-29 01:50:58] [SUCCESS] SUCCESS: Exponential Moving Average (EMA) (attempts: 3)\n",951 "[2025-12-29 01:50:58] [INFO] \n",952 "[10/24] Processing: End Point Moving Average (EPMA)\n",953 "[2025-12-29 01:50:58] [INFO] 🔄 End Point Moving Average (EPMA): Attempt 1/3\n",954 "[2025-12-29 01:51:24] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",955 "[2025-12-29 01:51:24] [INFO] 🔄 End Point Moving Average (EPMA): Attempt 2/3\n",956 "[2025-12-29 01:52:40] [INFO] Static validation passed\n",957 "[2025-12-29 01:52:40] [INFO] Runtime tests passed\n",958 "[2025-12-29 01:52:40] [SUCCESS] SUCCESS: End Point Moving Average (EPMA) (attempts: 3)\n",959 "[2025-12-29 01:52:40] [INFO] \n",960 "[11/24] Processing: General Weighted Moving Average\n",961 "[2025-12-29 01:52:40] [INFO] 🔄 General Weighted Moving Average: Attempt 1/3\n",962 "[2025-12-29 01:53:36] [INFO] Static validation passed\n",963 "[2025-12-29 01:53:36] [INFO] Runtime tests failed: Test 'normal_data' crashed: shape mismatch: value array of shape (100,) could not be broadcast to indexing result of shape (91,); Test 'leading_nans' crashed: shape mismatch: value array of shape (90,) could not be broadcast to indexing result of shape (81,)\n",964 "[2025-12-29 01:53:36] [INFO] 🔄 General Weighted Moving Average: Attempt 2/3\n",965 "[2025-12-29 01:55:01] [INFO] Static validation failed: Missing NaN handling\n",966 "[2025-12-29 01:55:01] [INFO] 🔄 General Weighted Moving Average: Attempt 3/3\n",967 "[2025-12-29 01:56:19] [INFO] Static validation passed\n",968 "[2025-12-29 01:56:19] [INFO] Runtime tests failed: Test 'normal_data' crashed: object of type 'float' has no len(); Test 'leading_nans' crashed: object of type 'float' has no len(); Test 'middle_nans' crashed: object of type 'float' has no len()\n",969 "[2025-12-29 01:56:19] [ERROR] FAILED: General Weighted Moving Average - Failed after 3 attempts\n",970 "[2025-12-29 01:56:19] [ERROR] FAILED: General Weighted Moving Average - Generation failed after all retries\n",971 "[2025-12-29 01:56:19] [INFO] \n",972 "[12/24] Processing: Gaussian Moving Average (GMA)\n",973 "[2025-12-29 01:56:19] [INFO] 🔄 Gaussian Moving Average (GMA): Attempt 1/3\n",974 "[2025-12-29 01:56:52] [INFO] Static validation passed\n",975 "[2025-12-29 01:56:52] [INFO] Runtime tests passed\n",976 "[2025-12-29 01:56:52] [SUCCESS] SUCCESS: Gaussian Moving Average (GMA) (attempts: 3)\n",977 "[2025-12-29 01:56:52] [INFO] \n",978 "[13/24] Processing: Hull Moving Average (HMA)\n",979 "[2025-12-29 01:56:52] [INFO] 🔄 Hull Moving Average (HMA): Attempt 1/3\n",980 "[2025-12-29 01:57:27] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",981 "[2025-12-29 01:57:27] [INFO] 🔄 Hull Moving Average (HMA): Attempt 2/3\n",982 "[2025-12-29 01:58:12] [INFO] Static validation passed\n",983 "[2025-12-29 01:58:12] [INFO] Runtime tests passed\n",984 "[2025-12-29 01:58:12] [SUCCESS] SUCCESS: Hull Moving Average (HMA) (attempts: 3)\n",985 "[2025-12-29 01:58:12] [INFO] \n",986 "[14/24] Processing: Linear Moving Average (LMA)\n",987 "[2025-12-29 01:58:12] [INFO] 🔄 Linear Moving Average (LMA): Attempt 1/3\n",988 "[2025-12-29 01:59:12] [INFO] Static validation passed\n",989 "[2025-12-29 01:59:12] [INFO] Runtime tests passed\n",990 "[2025-12-29 01:59:12] [SUCCESS] SUCCESS: Linear Moving Average (LMA) (attempts: 3)\n",991 "[2025-12-29 01:59:12] [INFO] \n",992 "[15/24] Processing: Moving Average Direction (MA-D)\n",993 "[2025-12-29 01:59:12] [INFO] 🔄 Moving Average Direction (MA-D): Attempt 1/3\n",994 "[2025-12-29 01:59:45] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",995 "[2025-12-29 01:59:45] [INFO] 🔄 Moving Average Direction (MA-D): Attempt 2/3\n",996 "[2025-12-29 02:00:13] [INFO] Static validation passed\n",997 "[2025-12-29 02:00:13] [INFO] Runtime tests passed\n",998 "[2025-12-29 02:00:13] [SUCCESS] SUCCESS: Moving Average Direction (MA-D) (attempts: 3)\n",999 "[2025-12-29 02:00:13] [INFO] \n",1000 "[16/24] Processing: Simple Moving Average (SMA)\n",1001 "[2025-12-29 02:00:13] [INFO] 🔄 Simple Moving Average (SMA): Attempt 1/3\n",1002 "[2025-12-29 02:00:37] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1003 "[2025-12-29 02:00:37] [INFO] 🔄 Simple Moving Average (SMA): Attempt 2/3\n",1004 "[2025-12-29 02:01:01] [INFO] Static validation passed\n",1005 "[2025-12-29 02:01:01] [INFO] Runtime tests passed\n",1006 "[2025-12-29 02:01:01] [SUCCESS] SUCCESS: Simple Moving Average (SMA) (attempts: 3)\n",1007 "[2025-12-29 02:01:01] [INFO] \n",1008 "[17/24] Processing: Sine Weighted Moving Average (SWMA)\n",1009 "[2025-12-29 02:01:01] [INFO] 🔄 Sine Weighted Moving Average (SWMA): Attempt 1/3\n",1010 "[2025-12-29 02:01:29] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1011 "[2025-12-29 02:01:29] [INFO] 🔄 Sine Weighted Moving Average (SWMA): Attempt 2/3\n",1012 "[2025-12-29 02:02:00] [INFO] Static validation passed\n",1013 "[2025-12-29 02:02:00] [INFO] Runtime tests passed\n",1014 "[2025-12-29 02:02:00] [SUCCESS] SUCCESS: Sine Weighted Moving Average (SWMA) (attempts: 3)\n",1015 "[2025-12-29 02:02:00] [INFO] \n",1016 "[18/24] Processing: Triple Exponential Moving Average (TEMA)\n",1017 "[2025-12-29 02:02:00] [INFO] 🔄 Triple Exponential Moving Average (TEMA): Attempt 1/3\n",1018 "[2025-12-29 02:02:48] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1019 "[2025-12-29 02:02:48] [INFO] 🔄 Triple Exponential Moving Average (TEMA): Attempt 2/3\n",1020 "[2025-12-29 02:03:52] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1021 "[2025-12-29 02:03:52] [INFO] 🔄 Triple Exponential Moving Average (TEMA): Attempt 3/3\n",1022 "[2025-12-29 02:04:33] [INFO] Static validation passed\n",1023 "[2025-12-29 02:04:33] [INFO] Runtime tests passed\n",1024 "[2025-12-29 02:04:33] [SUCCESS] SUCCESS: Triple Exponential Moving Average (TEMA) (attempts: 3)\n",1025 "[2025-12-29 02:04:33] [INFO] \n",1026 "[19/24] Processing: Tillson T3 Moving Average\n",1027 "[2025-12-29 02:04:33] [INFO] 🔄 Tillson T3 Moving Average: Attempt 1/3\n",1028 "[2025-12-29 02:05:13] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1029 "[2025-12-29 02:05:13] [INFO] 🔄 Tillson T3 Moving Average: Attempt 2/3\n",1030 "[2025-12-29 02:06:14] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1031 "[2025-12-29 02:06:14] [INFO] 🔄 Tillson T3 Moving Average: Attempt 3/3\n",1032 "[2025-12-29 02:07:05] [INFO] Static validation passed\n",1033 "[2025-12-29 02:07:05] [INFO] Runtime tests passed\n",1034 "[2025-12-29 02:07:05] [SUCCESS] SUCCESS: Tillson T3 Moving Average (attempts: 3)\n",1035 "[2025-12-29 02:07:05] [INFO] \n",1036 "[20/24] Processing: Triangular Moving Average (TMA)\n",1037 "[2025-12-29 02:07:05] [INFO] 🔄 Triangular Moving Average (TMA): Attempt 1/3\n"1038 ]1039 },1040 {1041 "name": "stderr",1042 "output_type": "stream",1043 "text": [1044 "<string>:58: RuntimeWarning: invalid value encountered in subtract\n"1045 ]1046 },1047 {1048 "name": "stdout",1049 "output_type": "stream",1050 "text": [1051 "[2025-12-29 02:07:54] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1052 "[2025-12-29 02:07:54] [INFO] 🔄 Triangular Moving Average (TMA): Attempt 2/3\n",1053 "[2025-12-29 02:08:23] [INFO] Static validation passed\n",1054 "[2025-12-29 02:08:23] [INFO] Runtime tests passed\n",1055 "[2025-12-29 02:08:23] [SUCCESS] SUCCESS: Triangular Moving Average (TMA) (attempts: 3)\n",1056 "[2025-12-29 02:08:23] [INFO] \n",1057 "[21/24] Processing: Wilder Moving Average\n",1058 "[2025-12-29 02:08:23] [INFO] 🔄 Wilder Moving Average: Attempt 1/3\n",1059 "[2025-12-29 02:09:13] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1060 "[2025-12-29 02:09:13] [INFO] 🔄 Wilder Moving Average: Attempt 2/3\n",1061 "[2025-12-29 02:10:06] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1062 "[2025-12-29 02:10:06] [INFO] 🔄 Wilder Moving Average: Attempt 3/3\n",1063 "[2025-12-29 02:11:10] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1064 "[2025-12-29 02:11:10] [ERROR] FAILED: Wilder Moving Average - Failed after 3 attempts\n",1065 "[2025-12-29 02:11:10] [ERROR] FAILED: Wilder Moving Average - Generation failed after all retries\n",1066 "[2025-12-29 02:11:10] [INFO] \n",1067 "[22/24] Processing: Zero Lag Exponential Moving Average (ZLEMA)\n",1068 "[2025-12-29 02:11:10] [INFO] 🔄 Zero Lag Exponential Moving Average (ZLEMA): Attempt 1/3\n",1069 "[2025-12-29 02:12:06] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1070 "[2025-12-29 02:12:06] [INFO] 🔄 Zero Lag Exponential Moving Average (ZLEMA): Attempt 2/3\n",1071 "[2025-12-29 02:12:55] [INFO] Static validation passed\n",1072 "[2025-12-29 02:12:55] [INFO] Runtime tests passed\n",1073 "[2025-12-29 02:12:55] [SUCCESS] SUCCESS: Zero Lag Exponential Moving Average (ZLEMA) (attempts: 3)\n",1074 "[2025-12-29 02:12:55] [INFO] \n",1075 "[23/24] Processing: Moving Average Convergence/Divergence (MACD)\n",1076 "[2025-12-29 02:12:55] [INFO] 🔄 Moving Average Convergence/Divergence (MACD): Attempt 1/3\n",1077 "[2025-12-29 02:13:32] [INFO] Static validation failed: Unsafe division detected (use np.divide with where= or add epsilon)\n",1078 "[2025-12-29 02:13:32] [INFO] 🔄 Moving Average Convergence/Divergence (MACD): Attempt 2/3\n",1079 "[2025-12-29 02:14:49] [INFO] Static validation passed\n",1080 "[2025-12-29 02:14:49] [INFO] Runtime tests failed: Test 'normal_data': Expected some valid values, got all NaN\n",1081 "[2025-12-29 02:14:49] [INFO] 🔄 Moving Average Convergence/Divergence (MACD): Attempt 3/3\n",1082 "[2025-12-29 02:15:59] [INFO] Static validation passed\n",1083 "[2025-12-29 02:15:59] [INFO] Runtime tests failed: Test 'normal_data': Expected some valid values, got all NaN\n",1084 "[2025-12-29 02:15:59] [ERROR] FAILED: Moving Average Convergence/Divergence (MACD) - Failed after 3 attempts\n",1085 "[2025-12-29 02:15:59] [ERROR] FAILED: Moving Average Convergence/Divergence (MACD) - Generation failed after all retries\n",1086 "[2025-12-29 02:15:59] [INFO] \n",1087 "[24/24] Processing: Momentum (MOM)\n",1088 "[2025-12-29 02:15:59] [INFO] 🔄 Momentum (MOM): Attempt 1/3\n",1089 "[2025-12-29 02:17:00] [INFO] Static validation passed\n",1090 "[2025-12-29 02:17:00] [INFO] Runtime tests passed\n",1091 "[2025-12-29 02:17:00] [SUCCESS] SUCCESS: Momentum (MOM) (attempts: 3)\n",1092 "[2025-12-29 02:17:00] [INFO] ================================================================================\n",1093 "[2025-12-29 02:17:00] [INFO] GENERATION SUMMARY\n",1094 "[2025-12-29 02:17:00] [INFO] Total factors: 24\n",1095 "[2025-12-29 02:17:00] [SUCCESS] Success: 21\n",1096 "[2025-12-29 02:17:00] [ERROR] Failed: 6\n",1097 "[2025-12-29 02:17:00] [SKIP] Skipped: 0\n",1098 "[2025-12-29 02:17:00] [INFO] ================================================================================\n",1099 "\n",1100 "🎉 Generation complete!\n",1101 " Success rate: 21/24 (87.5%)\n"1102 ]1103 }1104 ],1105 "source": [1106 "import os\n",1107 "import json\n",1108 "import ast\n",1109 "import re\n",1110 "import traceback\n",1111 "from typing import Tuple, Optional, Dict, List\n",1112 "from datetime import datetime\n",1113 "from pathlib import Path\n",1114 "import numpy as np\n",1115 "import pandas as pd\n",1116 "from google import genai\n",1117 "from dotenv import load_dotenv\n",1118 "\n",1119 "# ============================================================================\n",1120 "# CONFIGURATION\n",1121 "# ============================================================================\n",1122 "\n",1123 "load_dotenv()\n",1124 "GOOGLE_API_KEY = os.getenv(\"GOOGLE_API_KEY\")\n",1125 "if not GOOGLE_API_KEY:\n",1126 " raise ValueError(\"GOOGLE_API_KEY not found in environment\")\n",1127 "\n",1128 "gemini_client = genai.Client(api_key=GOOGLE_API_KEY)\n",1129 "gemini_model = \"gemini-2.5-pro\"\n",1130 "\n",1131 "LIB_DIR = Path(\"data/alpha_lib\")\n",1132 "CODE_DIR = Path(\"data/alpha_code\")\n",1133 "LOG_DIR = Path(\"data/logs\")\n",1134 "FAILED_DIR = Path(\"data/failed_factors\")\n",1135 "\n",1136 "# 创建必要的目录\n",1137 "for dir_path in [CODE_DIR, LOG_DIR, FAILED_DIR]:\n",1138 " dir_path.mkdir(parents=True, exist_ok=True)\n",1139 "\n",1140 "\n",1141 "# ============================================================================\n",1142 "# CODE VALIDATOR (Enhanced)\n",1143 "# ============================================================================\n",1144 "\n",1145 "class EnhancedCodeValidator:\n",1146 " \"\"\"Comprehensive static code analysis for production standards\"\"\"\n",1147 " \n",1148 " @staticmethod\n",1149 " def check(code_str: str) -> Tuple[bool, str]:\n",1150 " \"\"\"\n",1151 " Validates code against all production standards.\n",1152 " \n",1153 " Returns:\n",1154 " (is_valid, error_message)\n",1155 " \"\"\"\n",1156 " # 1. Syntax check\n",1157 " try:\n",1158 " tree = ast.parse(code_str)\n",1159 " except SyntaxError as e:\n",1160 " return False, f\"Syntax Error: {e}\"\n",1161 " \n",1162 " # 2. Check for forbidden loops in core function\n",1163 " has_core_func = '_numpy(' in code_str or '_core(' in code_str\n",1164 " if has_core_func:\n",1165 " for node in ast.walk(tree):\n",1166 " if isinstance(node, (ast.For, ast.While)):\n",1167 " # Get context to see if it's in core function\n",1168 " return False, \"Forbidden: Explicit loop in core calculation function\"\n",1169 " \n",1170 " # 3. Check for forbidden pandas iteration\n",1171 " forbidden_patterns = [\n",1172 " (r'\\.apply\\s*\\(', \"pandas.apply()\"),\n",1173 " (r'\\.iterrows\\s*\\(', \"pandas.iterrows()\"),\n",1174 " (r'\\.resample\\s*\\(', \"pandas.resample()\"),\n",1175 " ]\n",1176 " \n",1177 " for pattern, name in forbidden_patterns:\n",1178 " if re.search(pattern, code_str):\n",1179 " return False, f\"Forbidden: {name} detected in code\"\n",1180 " \n",1181 " # 4. Check for required robustness patterns\n",1182 " required_patterns = [\n",1183 " (r'np\\.full_like|np\\.empty_like|np\\.zeros_like', \n",1184 " \"Missing output initialization (use np.full_like)\"),\n",1185 " (r'np\\.isnan|~np\\.isnan|pd\\.isna|pd\\.notna', \n",1186 " \"Missing NaN handling\"),\n",1187 " (r'if len\\(.*?\\)\\s*==\\s*0|if\\s+.*?\\.size\\s*==\\s*0', \n",1188 " \"Missing empty array check\"),\n",1189 " (r'valid_mask|first_valid', \n",1190 " \"Missing valid data detection logic\"),\n",1191 " ]\n",1192 " \n",1193 " for pattern, error_msg in required_patterns:\n",1194 " if not re.search(pattern, code_str):\n",1195 " return False, f\"{error_msg}\"\n",1196 " \n",1197 " # 5. Check for numpy usage\n",1198 " if 'import numpy' not in code_str and 'np.' not in code_str:\n",1199 " return False, \"No numpy usage detected\"\n",1200 " \n",