fix:添加断点续传功能。

This commit is contained in:
2026-08-01 18:12:04 +08:00
parent cdf5582c90
commit 063f790650
2 changed files with 217 additions and 1 deletions
@@ -134,6 +134,9 @@
" get_stock_codes_from_db,\n",
" full_import,\n",
" batch_insert,\n",
" check_daily_progress,\n",
" check_table_summary,\n",
" resume_daily_by_year,\n",
" logger,\n",
")\n",
"from config import DB_CONFIG, TUSHARE_TOKEN, START_DATE, END_DATE\n",
@@ -345,6 +348,76 @@
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2.1 查看日线导入进度\n",
"\n",
"按年份统计 daily 表中已导入的记录数和独立股票数,用于确认导入到哪个年份了。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 查看每天的日线导入进度(按年份统计)\n",
"check_daily_progress()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 或者查看所有表的整体概览\n",
"check_table_summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2.2 断点续传 — 从中断处继续导入\n",
"\n",
"**方法一(推荐):** 使用 `resume_daily_by_year` 自动跳过已完成的年份,只导入缺失年份。\n",
"\n",
"**方法二:** 手动修改 `start_year` 参数重新调用 `import_daily_by_year`(因为 UPSERT 幂等,重复导入不会造成数据问题)。"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 方法一:自动检测已有年份,只导入缺失年份(推荐)\n",
"stock_list = get_stock_codes_from_db()\n",
"resume_daily_by_year(stock_list, start_year=2010, end_year=2025)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 方法二:手动指定断点年份重新调用 import_daily_by_year\n",
"# 例如假设 2010~2020 已完成,从 2021 年开始继续\n",
"# stock_list = get_stock_codes_from_db()\n",
"# import_daily_by_year(stock_list, start_year=2021, end_year=2025)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **提示:** 以上两个方法都可以安全使用。因为 daily 表使用 `ON CONFLICT (ts_code, trade_date) DO UPDATE`,重复导入已存在的数据不会产生重复记录。另外也可查看 `import_data.log` 文件获取最后一次成功的日志输出。"
]
},
{
"cell_type": "markdown",
"metadata": {},