feat: 修复daily历史覆盖度缺口+增加覆盖度校验
- 新增 repair_daily_backfill.py:按daily_basic基准检测并补拉覆盖不足交易日 (解决2012-2013沪市+创业板整年缺失问题,断点续传日期级检测无法发现) - importer.py: 新增 check_daily_coverage() 覆盖度审计函数 - incremental_import.py: 增量导入后自动跑覆盖度校验并告警 - 数据批量导入.ipynb: 增加数据完整性检验单元格
This commit is contained in:
@@ -720,6 +720,68 @@
|
||||
"display(df3[['ts_code', 'name', 'industry', 'roe', 'roa', 'eps', 'debt_to_assets']])\n",
|
||||
"conn.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"## 数据完整性检验(覆盖度审计)\n",
|
||||
"\n",
|
||||
"> **背景**:2026-08-26 审计发现 daily 表 2012-2013 年存在覆盖度缺口(沪市+创业板整年缺失),\n",
|
||||
"> 而断点续传的\"日期级\"检测(`NOT EXISTS trade_date`)无法发现这种\"日期存在但覆盖不全\"的问题。\n",
|
||||
"> 本单元格做**覆盖度级**校验:以 `daily_basic` 当日股票数为基准,检查 `daily` 是否缺股。\n",
|
||||
">\n",
|
||||
"> 运行 `repair_daily_backfill.py` 可自动补拉覆盖不足的交易日。\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ============================================================\n",
|
||||
"# 数据完整性检验:daily vs daily_basic 每日覆盖度审计\n",
|
||||
"# 判据:daily 当日股票数 < daily_basic 当日股票数 × 95% → 覆盖不足\n",
|
||||
"# ============================================================\n",
|
||||
"import importlib\n",
|
||||
"import importer\n",
|
||||
"importlib.reload(importer)\n",
|
||||
"from importer import get_pg_connection, check_daily_coverage, check_table_summary\n",
|
||||
"\n",
|
||||
"conn = get_pg_connection()\n",
|
||||
"\n",
|
||||
"# 1) 各表整体概览(行数 / 股票数 / 日期范围)\n",
|
||||
"print('=' * 60)\n",
|
||||
"print('各表整体概览:')\n",
|
||||
"print('=' * 60)\n",
|
||||
"check_table_summary(conn=conn)\n",
|
||||
"\n",
|
||||
"# 2) 全历史覆盖度审计(默认容差 5%)\n",
|
||||
"print('\\n' + '=' * 60)\n",
|
||||
"print('覆盖度审计: daily vs daily_basic (容差 5%)')\n",
|
||||
"print('=' * 60)\n",
|
||||
"from datetime import date\n",
|
||||
"end_date = date.today().strftime('%Y-%m-%d')\n",
|
||||
"partial = check_daily_coverage(start_date='2010-01-01', end_date=end_date, conn=conn, tolerance=0.05)\n",
|
||||
"\n",
|
||||
"if partial:\n",
|
||||
" print(f'⚠ 发现 {len(partial)} 个交易日覆盖不足:')\n",
|
||||
" # 按年份汇总\n",
|
||||
" from collections import Counter\n",
|
||||
" years = Counter(p['trade_date'][:4] for p in partial)\n",
|
||||
" for y in sorted(years):\n",
|
||||
" print(f' {y} 年: {years[y]} 个交易日覆盖不足')\n",
|
||||
" print('\\n样例(前10条):')\n",
|
||||
" for p in partial[:10]:\n",
|
||||
" print(f\" {p['trade_date']}: daily={p['daily_cnt']} vs daily_basic={p['daily_basic_cnt']} ({p['coverage_pct']}%)\")\n",
|
||||
" print('\\n→ 修复方法: 运行 python3 repair_daily_backfill.py 补拉')\n",
|
||||
"else:\n",
|
||||
" print('✅ 覆盖度正常,无缺失交易日')\n",
|
||||
"\n",
|
||||
"conn.close()\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
|
||||
Reference in New Issue
Block a user