feat: 修复daily历史覆盖度缺口+增加覆盖度校验

- 新增 repair_daily_backfill.py:按daily_basic基准检测并补拉覆盖不足交易日
  (解决2012-2013沪市+创业板整年缺失问题,断点续传日期级检测无法发现)
- importer.py: 新增 check_daily_coverage() 覆盖度审计函数
- incremental_import.py: 增量导入后自动跑覆盖度校验并告警
- 数据批量导入.ipynb: 增加数据完整性检验单元格
This commit is contained in:
xiaoqiang
2026-08-26 10:47:32 +00:00
parent e07e9426c7
commit 84912ef99c
4 changed files with 326 additions and 0 deletions
@@ -720,6 +720,68 @@
"display(df3[['ts_code', 'name', 'industry', 'roe', 'roa', 'eps', 'debt_to_assets']])\n",
"conn.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 数据完整性检验(覆盖度审计)\n",
"\n",
"> **背景**:2026-08-26 审计发现 daily 表 2012-2013 年存在覆盖度缺口(沪市+创业板整年缺失),\n",
"> 而断点续传的\"日期级\"检测(`NOT EXISTS trade_date`)无法发现这种\"日期存在但覆盖不全\"的问题。\n",
"> 本单元格做**覆盖度级**校验:以 `daily_basic` 当日股票数为基准,检查 `daily` 是否缺股。\n",
">\n",
"> 运行 `repair_daily_backfill.py` 可自动补拉覆盖不足的交易日。\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ============================================================\n",
"# 数据完整性检验:daily vs daily_basic 每日覆盖度审计\n",
"# 判据:daily 当日股票数 < daily_basic 当日股票数 × 95% → 覆盖不足\n",
"# ============================================================\n",
"import importlib\n",
"import importer\n",
"importlib.reload(importer)\n",
"from importer import get_pg_connection, check_daily_coverage, check_table_summary\n",
"\n",
"conn = get_pg_connection()\n",
"\n",
"# 1) 各表整体概览(行数 / 股票数 / 日期范围)\n",
"print('=' * 60)\n",
"print('各表整体概览:')\n",
"print('=' * 60)\n",
"check_table_summary(conn=conn)\n",
"\n",
"# 2) 全历史覆盖度审计(默认容差 5%)\n",
"print('\\n' + '=' * 60)\n",
"print('覆盖度审计: daily vs daily_basic (容差 5%)')\n",
"print('=' * 60)\n",
"from datetime import date\n",
"end_date = date.today().strftime('%Y-%m-%d')\n",
"partial = check_daily_coverage(start_date='2010-01-01', end_date=end_date, conn=conn, tolerance=0.05)\n",
"\n",
"if partial:\n",
" print(f'⚠ 发现 {len(partial)} 个交易日覆盖不足:')\n",
" # 按年份汇总\n",
" from collections import Counter\n",
" years = Counter(p['trade_date'][:4] for p in partial)\n",
" for y in sorted(years):\n",
" print(f' {y} 年: {years[y]} 个交易日覆盖不足')\n",
" print('\\n样例(前10条):')\n",
" for p in partial[:10]:\n",
" print(f\" {p['trade_date']}: daily={p['daily_cnt']} vs daily_basic={p['daily_basic_cnt']} ({p['coverage_pct']}%)\")\n",
" print('\\n→ 修复方法: 运行 python3 repair_daily_backfill.py 补拉')\n",
"else:\n",
" print('✅ 覆盖度正常,无缺失交易日')\n",
"\n",
"conn.close()\n"
]
}
],
"metadata": {