shellway/quanxiel
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat:修改获取数据方式为按日期获取，而不是按股票循环，加快数据获取速度，增加按日期检查数据库缺失数据并补充的功能。

Browse Source
This commit is contained in:
shellway
2026-08-02 11:17:16 +08:00
parent 063f790650
commit e3089c25a6
2 changed files with 294 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files
quantitative_data/数据批量导入.ipynb
+71 -100
View File
@@ -42,16 +42,7 @@
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"工作目录: t:\\jupyter\\notebook\\quantitative_data\n",
"Python 版本: 3.10.2 (heads/master:d9999f5, Dec 16 2022, 16:20:32) [MSC v.1929 64 bit (AMD64)]\n"
]
}
],
"outputs": [],
"source": [
"import sys\n",
"import os\n",
@@ -64,28 +55,7 @@
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"pandas 1.5.0\n",
"python-dotenv 1.2.2\n",
"SQLAlchemy 1.3.24\n",
"tushare 1.2.89\n",
"vnpy-tushare 1.2.85.1\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"[notice] A new release of pip available: 22.2.2 -> 26.2\n",
"[notice] To update, run: python.exe -m pip install --upgrade pip\n"
]
}
],
"outputs": [],
"source": [
"# 检查依赖包\n",
"!pip list | findstr -i \"tushare pandas psycopg2-binary sqlalchemy python-dotenv\""
@@ -105,15 +75,7 @@
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"模块导入成功!\n"
]
}
],
"outputs": [],
"source": [
"# 导入核心模块\n",
"from importer import (\n",
@@ -122,7 +84,8 @@
" init_database,\n",
" import_stock_basic,\n",
" import_trade_cal,\n",
" import_daily_batch,\n",
" import_daily_for_stock,\n",
" import_daily_by_date,\n",
" import_daily_by_year,\n",
" import_daily_basic,\n",
" import_daily_basic_by_date,\n",
@@ -136,7 +99,8 @@
" batch_insert,\n",
" check_daily_progress,\n",
" check_table_summary,\n",
" resume_daily_by_year,\n",
" get_missing_daily_dates,\n",
" resume_daily_by_date,\n",
" logger,\n",
")\n",
"from config import DB_CONFIG, TUSHARE_TOKEN, START_DATE, END_DATE\n",
@@ -148,20 +112,7 @@
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"✗ 连接失败: connection to server at \"192.168.27.11\", port 5438 failed: fe_sendauth: no password supplied\n",
"\n",
"请检查:\n",
" 1. Docker 容器是否已启动: docker ps | findstr postgres\n",
" 2. 环境变量 (.env) 中的连接参数是否正确\n",
" 3. 防火墙是否开放 5438 端口\n"
]
}
],
"outputs": [],
"source": [
"# 测试数据库连接\n",
"try:\n",
@@ -309,28 +260,14 @@
"---\n",
"## Step 4: 导入日线行情 (核心表，最耗时)\n",
"\n",
"### 4.1 获取股票列表"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 获取所有需要导入的股票代码\n",
"stock_list = get_stock_codes_from_db()\n",
"print(f\"共 {len(stock_list)} 只股票需要导入日线行情\")\n",
"print(f\"前 10 只: {stock_list[:10]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2 按年批量导入 (推荐 - 断点续传友好)\n",
"> **新版改进：** 按交易日循环拉取全市场数据 `pro.daily(trade_date='20180810')`，\n",
"> API 调用从 ~80,000次 (5000只×16年) 降至 ~4,000次 (250交易日×16年)，速度提升约 **20倍**。\n",
"\n",
"数据量估算: 约5000只股票 × 250交易日/年 × 16年 ≈ 2000万条记录"
"### 4.1 按年批量导入 (推荐)\n",
"\n",
"数据量估算: ~4000个交易日，每个交易日约5000条记录 ≈ 2000万条记录\n",
"\n",
"> **注意：** 新版 `import_daily_by_year` 不再需要 `stock_list` 参数，内部自动从 `trade_cal` 获取交易日列表后逐日拉取全市场数据。"
]
},
{
@@ -339,12 +276,13 @@
"metadata": {},
"outputs": [],
"source": [
"# 按年份逐批导入日线行情\n",
"# 如果中断，可以修改年份范围从断点继续\n",
"# 按年份逐批导入日线行情 (按交易日循环拉取全市场数据)\n",
"# 不再需要 stock_list 参数 — 自动查询 trade_cal 获取交易日\n",
"# 如果中断，修改年份范围从断点继续即可 (UPSERT 幂等，不会重复)\n",
"import_daily_by_year(\n",
" stock_list,\n",
" start_year=2010,\n",
" end_year=2025,\n",
" sleep_interval=0.3, # API 频率控制，免费版建议 0.3~0.5\n",
")"
]
},
@@ -352,9 +290,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2.1 查看日线导入进度\n",
"### 4.1.1 查看日线导入进度\n",
"\n",
"按年份统计 daily 表中已导入的记录数和独立股票数，用于确认导入到哪个年份了。"
"按年份统计 daily 表中已导入的记录数和独立股票数，了解导入到哪个年份了。"
]
},
{
@@ -373,7 +311,7 @@
"metadata": {},
"outputs": [],
"source": [
"# 或者查看所有表的整体概览\n",
"# 查看所有表的整体概览\n",
"check_table_summary()"
]
},
@@ -381,11 +319,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.2.2 断点续传 — 从中断处继续导入\n",
"### 4.1.2 按缺失日期断点续传 (推荐)\n",
"\n",
"**方法一（推荐）：** 使用 `resume_daily_by_year` 自动跳过已完成的年份，只导入缺失年份。\n",
"新版 `resume_daily_by_date` 自动对比 `trade_cal` 和 `daily` 表，**只导入缺失日期的全市场数据**。\n",
"粒度精确到交易日级别，比旧的按年份续传更精细。\n",
"\n",
"**方法二：** 手动修改 `start_year` 参数重新调用 `import_daily_by_year`（因为 UPSERT 幂等，重复导入不会造成数据问题）。"
"**工作流程：**\n",
"1. 先调用 `get_missing_daily_dates()` 查看缺失的交易日列表\n",
"2. 调用 `resume_daily_by_date()` 仅补缺缺失的交易日"
]
},
{
@@ -394,9 +335,21 @@
"metadata": {},
"outputs": [],
"source": [
"# 方法一：自动检测已有年份，只导入缺失年份（推荐）\n",
"stock_list = get_stock_codes_from_db()\n",
"resume_daily_by_year(stock_list, start_year=2010, end_year=2025)"
"# 第1步：查看缺失的交易日 (仅查询，不导入)\n",
"missing = get_missing_daily_dates(\n",
" start_date=\"2010-01-01\",\n",
" end_date=\"2025-12-31\",\n",
")\n",
"print(f\"缺失交易日总数: {len(missing)}\")\n",
"if len(missing) <= 30:\n",
" print(f\"缺失日期: {missing}\")\n",
"else:\n",
" # 按年份汇总显示\n",
" from collections import Counter\n",
" year_counts = Counter(d[:4] for d in missing)\n",
" for y in sorted(year_counts):\n",
" print(f\" {y} 年: {year_counts[y]} 个缺失交易日\")\n",
" print(f\" (前10个缺失日期): {missing[:10]}\")"
]
},
{
@@ -405,24 +358,36 @@
"metadata": {},
"outputs": [],
"source": [
"# 方法二：手动指定断点年份重新调用 import_daily_by_year\n",
"# 例如假设 2010~2020 已完成，从 2021 年开始继续\n",
"# stock_list = get_stock_codes_from_db()\n",
"# import_daily_by_year(stock_list, start_year=2021, end_year=2025)"
"# 第2步：按缺失日期断点续传，自动补充缺失的交易日数据\n",
"# 例如之前中断了，这里只会导入尚未导入的交易日数据\n",
"resume_daily_by_date(\n",
" start_date=\"2010-01-01\",\n",
" end_date=\"2025-12-31\",\n",
" sleep_interval=0.3,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> **提示：** 以上两个方法都可以安全使用。因为 daily 表使用 `ON CONFLICT (ts_code, trade_date) DO UPDATE`，重复导入已存在的数据不会产生重复记录。另外也可查看 `import_data.log` 文件获取最后一次成功的日志输出。"
"> **备用方案：** 也可以手动指定断点年份重新调用 `import_daily_by_year`，因为 UPSERT 幂等，重复导入已存在的数据不会产生重复记录。\n",
">\n",
"> ```python\n",
"> # 例如假设 2010~2020 已完成，从 2021 年继续\n",
"> import_daily_by_year(start_year=2021, end_year=2025)\n",
"> ```\n",
">\n",
"> 也可查看 `import_data.log` 文件获取最后一次成功的日志输出。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4.3 单只股票导入 (补充/重试)"
"### 4.2 单只股票补充导入\n",
"\n",
"如果某只股票数据缺失，可以单独补导（保留原接口兼容）。"
]
},
{
@@ -433,8 +398,6 @@
"source": [
"# 导入单只股票日线行情 (用于补充导入或测试)\n",
"conn = get_pg_connection()\n",
"from importer import import_daily_for_stock\n",
"\n",
"n = import_daily_for_stock(\"000001.SZ\", \"2020-01-01\", \"2020-12-31\", conn)\n",
"print(f\"导入 000001.SZ 2020年数据: {n} 条\")\n",
"conn.close()"
@@ -517,6 +480,10 @@
"outputs": [],
"source": [
"# 导入复权因子 (用于前复权/后复权价格计算)\n",
"# 需要先获取股票列表\n",
"stock_list = get_stock_codes_from_db()\n",
"print(f\"共 {len(stock_list)} 只股票\")\n",
"\n",
"import_adj_factor_batch(\n",
" stock_list,\n",
" start_date=\"2010-01-01\",\n",
@@ -541,6 +508,10 @@
"outputs": [],
"source": [
"# 导入利润表、资产负债表、现金流量表、财务指标\n",
"# stock_list 从上一步已获取，或重新获取\n",
"if 'stock_list' not in dir():\n",
" stock_list = get_stock_codes_from_db()\n",
"\n",
"import_financial_statements(\n",
" stock_list,\n",
" start_date=\"2010-01-01\",\n",
@@ -613,7 +584,7 @@
"metadata": {},
"outputs": [],
"source": [
"# 一键全量导入 (需数小时~数十小时，请谨慎)\n",
"# 一键全量导入 (需数小时，请谨慎)\n",
"# full_import(\n",
"# start_date=\"2010-01-01\",\n",
"# end_date=\"2025-12-31\",\n",
@@ -768,4 +739,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
}
}