fix:添加断点续传功能。
This commit is contained in:
@@ -134,6 +134,9 @@
|
||||
" get_stock_codes_from_db,\n",
|
||||
" full_import,\n",
|
||||
" batch_insert,\n",
|
||||
" check_daily_progress,\n",
|
||||
" check_table_summary,\n",
|
||||
" resume_daily_by_year,\n",
|
||||
" logger,\n",
|
||||
")\n",
|
||||
"from config import DB_CONFIG, TUSHARE_TOKEN, START_DATE, END_DATE\n",
|
||||
@@ -345,6 +348,76 @@
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.2.1 查看日线导入进度\n",
|
||||
"\n",
|
||||
"按年份统计 daily 表中已导入的记录数和独立股票数,用于确认导入到哪个年份了。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 查看每天的日线导入进度(按年份统计)\n",
|
||||
"check_daily_progress()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 或者查看所有表的整体概览\n",
|
||||
"check_table_summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 4.2.2 断点续传 — 从中断处继续导入\n",
|
||||
"\n",
|
||||
"**方法一(推荐):** 使用 `resume_daily_by_year` 自动跳过已完成的年份,只导入缺失年份。\n",
|
||||
"\n",
|
||||
"**方法二:** 手动修改 `start_year` 参数重新调用 `import_daily_by_year`(因为 UPSERT 幂等,重复导入不会造成数据问题)。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 方法一:自动检测已有年份,只导入缺失年份(推荐)\n",
|
||||
"stock_list = get_stock_codes_from_db()\n",
|
||||
"resume_daily_by_year(stock_list, start_year=2010, end_year=2025)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# 方法二:手动指定断点年份重新调用 import_daily_by_year\n",
|
||||
"# 例如假设 2010~2020 已完成,从 2021 年开始继续\n",
|
||||
"# stock_list = get_stock_codes_from_db()\n",
|
||||
"# import_daily_by_year(stock_list, start_year=2021, end_year=2025)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **提示:** 以上两个方法都可以安全使用。因为 daily 表使用 `ON CONFLICT (ts_code, trade_date) DO UPDATE`,重复导入已存在的数据不会产生重复记录。另外也可查看 `import_data.log` 文件获取最后一次成功的日志输出。"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
|
||||
Reference in New Issue
Block a user