diff --git a/course/01-introduction.md b/course/01-introduction.md new file mode 100644 index 0000000..de76d32 --- /dev/null +++ b/course/01-introduction.md @@ -0,0 +1,23 @@ +# 01 · Introduction（课程总览） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Introduction（课程总览） +- **时长**：7 分 19 秒 ｜ **原视频**：https://youtu.be/EeSrQdab0s0 +- **本地视频**：[videos/01-introduction.mp4](videos/01-introduction.mp4) + +## 🎯 本集要点（中文导读） + +- **系列目标**：把算法交易的**完整开发流程**讲清楚——从想法生成 → 代码实现 → 回测优化 → 实盘部署。 +- **技术栈**：Python + **QuantConnect** 平台（网页自带 IDE，无需本地装软件，跟着敲代码即可）。 +- **数据**：QuantConnect 免费提供大量数据（美股/期权等，分钟级），支持导入自定义数据，省去自己抓数据的耗时与不稳定。 +- **代码隐私**：不主动公开则平台尊重算法 IP；不放心也可本地部署其开源引擎（Lean）。 +- **库支持**：内置数百个常用 Python 库（TensorFlow、NLTK 等），import 即用。 +- **前置要求**：Python 基础（变量/条件/循环/函数/列表·字典·集合/类）+ 基础交易概念（股票、做多/做空）；**无需算法交易经验**。本系列会用到 pandas / numpy / matplotlib。 +- **账户**：免费账户足够学习与回测；**实盘**需付费 live trading 节点。 +- **学习建议**：边看边动手，多改参数、多做小实验，效果远好于被动观看。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Hello and welcome back to the first video of a brand new video series. Over the past few months I've gotten countless requests to create a hands-on video course on algorithmic trading and that's exactly what this series is going to be. The purpose of this introductory video is to outline what you can expect to learn, the prerequisites for this course and which resources we will be using. That said, let's cut right to the chase and start by going over what you will learn in this video series. My first goal with this series is to bring the entire trading algorithm development process closer to you. We will cover everything from idea generation, code implementation, testing and optimizing to the actual live deployment of your algorithms. For this we will be using Python and the QuantConnect algorithmic trading platform. The QuantConnect platform has its own integrated development environment which allows you to create your algorithms right on their website. So this means that you won't have to install any additional software for you to be able to follow along. In this course we will cover all the major aspects of the QuantConnect API so that you can code your own algorithms on your own after watching this video series. To accomplish this there will be numerous example algorithms so that you can gain the best possible insights into what an actual trading algorithm looks like. We will go over everything from very basic beginner algorithms to advanced machine learning algorithms. I will of course make all of the code from these example bots available by linking to it in the description box of each video. My goal is to create at least one example algorithm for every asset class that QuantConnect supports. That means no matter if you want to create trading bots for stocks, bonds, forex, crypto, CFDs, options or futures this course and QuantConnect has you covered. But note that in the beginning we will focus on equities since I find this the best way to get started. But don't worry the concepts learned for stock trading bots are just as important for forex, crypto or any other kind of trading bot. When it comes to developing and testing your trading algorithms one of the biggest problems is getting the right data. One solution for this would be to scrape data from different websites. The problem with this is that it can be very time consuming and unreliable. Besides that you usually won't be able to find all the data you need easily accessible for free. Luckily QuantConnect allows you to use the data of a bunch of different data vendors for free. They have a huge collection of data at a resolution as low as a minute for all US stocks, options and other assets. Furthermore they have already announced that they have plans to add data for other international markets as well. It is also possible to import your own custom data which we will also cover in one of the lessons. Besides data another big concern of algorithmic trading developers is for them to keep the intellectual property and privacy of their algorithms. As long as you don't personally share your algorithms QuantConnect does respect your code's IP and privacy. If however you don't trust them it is also possible to set up their open source trading engine for local development. But since this series is mostly aimed at beginners we will just be using the website for the coding. For more advanced trading algorithms it can be very useful to use some of Python's libraries such as TensorFlow for machine learning, NLTK for sentiment analysis or some other library. QuantConnect supports hundreds of Python's most popular libraries and since we will be using QuantConnect's development environment you won't have to manually install any of these. Instead we can just import them with a simple import statement. As you hopefully can see QuantConnect is a very flexible platform that pretty much allows you to create whatever algorithm you want to. This flexibility also means that their API can be a little overwhelming to learn. But don't worry we will cover all of the major aspects of QuantConnect's API in this course on a step-by-step basis. To accomplish this this video series will have both theory and application oriented videos. For the coding part of this series I highly recommend coding along and trying out minor deviations, adjustments and experiments on your own. This will yield a much better learning experience than just passively watching these videos. For you to be able to code along make sure to create a free QuantConnect account. There's a link in the description box below. QuantConnect offers both free and premium accounts. For all the coding and bag testing you won't need a premium account so I will try to mainly focus on the features that can be used without paying. But note that if you want to live trade an algorithm you will have to pay for a live trading server node on QuantConnect since your algorithm has to be able to receive data and communicate with your broker on an ongoing basis. Besides that a premium account can decrease bag test times and give you more available memory for your algorithms. But like I said you can also follow along and use QuantConnect with a completely free account. Last but not least let me quickly present some prerequisites for this course. Here I firstly want to say that this course is aimed at beginners so I don't expect you to have any algorithmic trading experience for you to be able to gain value from this course. That said besides a free QuantConnect account it would be very useful to already have some Python experience. I will not be covering Python basics in this course. If you're new to programming I highly recommend first learning some basic programming principles and Python syntax. As a good Python foundation I'd recommend to be familiar with things such as variables, conditionals, control flow, loops, functions and basic data structures such as lists, sets and dictionaries. Besides that it could also be useful to have some experience with basic object oriented programming. If you have never heard of the concept of a class it would be a good idea to quickly freshen up that knowledge. That said it is totally okay if you aren't an expert programmer or only have very difficult coding experience. I will do my best to explain everything as simple as possible. Besides that we will also be using a few modules such as pandas, numpy and matplotlib among others. But if you aren't an expert at these don't worry. If the demand is high enough I could also create an introductory video series on these modules and more basic Python data analysis topics at some point in the future. So if that's something you want to see definitely let me know in the comment section below. Besides Python QuantConnect also supports C-sharp but since the majority of the community uses Python that's what I will focus on in this course. Next to these programming prerequisites it would also be very useful to have some prior knowledge of basic trading and finance topics. You don't need any real-world trading experience but it would be great if you know what a stock is, what going long and short is and generally have a basic understanding of the markets. If you don't you can check out some of the other videos on my channel and the articles on tradeoptionswithme.com. With that being said I hope you are as excited as I am to get started with this algorithmic trading series. In the next video we will get started with the course content by looking at the algorithm development process and how trading algorithms work. Thanks for watching and see you in the next video. diff --git a/course/02-episode-2.md b/course/02-episode-2.md new file mode 100644 index 0000000..e5a9520 --- /dev/null +++ b/course/02-episode-2.md @@ -0,0 +1,26 @@ +# 02 · How Trading Algorithms Work（算法原理与开发流程） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：How Trading Algorithms Work（算法原理与开发流程） +- **时长**：10 分 3 秒 ｜ **原视频**：https://youtu.be/K2fnO7wTUvM +- **本地视频**：[videos/02-episode-2.mp4](videos/02-episode-2.mp4) + +## 🎯 本集要点（中文导读） + +本集讲**交易算法的原理**与**开发流程**（概念层面）。 + +- **实盘算法的组成**：① 数据输入（价格/时间为主，可能还有基本面、新闻、订单数据）→ ② 算法分析数据做决策 → ③ 通过 API 把订单传给券商。QuantConnect 代管"取数、托管、券商通信"，你专注写逻辑。 +- **核心：先找 Edge（ALPHA）**。好的策略靠的是"超额收益 edge"，而不是先随便写个策略再硬找理由。Edge 是整件事的中心。 +- **盈利 ≠ 有 edge**：要跟**基准**（如标普 500）比，而且要**风险调整后**比；长期跑赢基准才算有 edge。 +- **找 alpha 的方法**：科学化地做假设检验（例：小盘股是否跑赢大盘股→用市值分组比较历史表现），并尽量找到**因果逻辑**解释。 +- **资源**：Quantpedia（大量策略/论文）、QuantConnect 策略库与论坛、平台内建研究环境（后续课讲）。 +- **alpha 会衰减**：越来越多人用就会失效，且策略有容量上限。 +- **完整开发流程**：研究找 edge → 编码实现 → **回测** → 优化 → **Paper trading** → 小资金实盘 → 上线后适度监控但**别频繁干预**。 +- ⚠️ **回测有偏误**（过拟合、未来函数/前视偏差、幸存者偏差），结果要打折看。 +- **要把概念"可量化"**：例如"上升趋势"对电脑无意义，需定义成"60 日均线 > 90 日均线"。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome to the second video of my algorithmic trading video course. In this video we will take a look at how trading algorithms work on a conceptual level. Besides that, you will learn what the process of developing a trading algorithm looks like. Before we start though, I quickly want to mention that the QuantConnect team has reached out to me after seeing some of my past videos and has offered to sponsor this video series, so big thanks to QuantConnect for sponsoring this series. As I said, let's start right away by quickly breaking down how trading algorithms work. For this, it is important to understand the key components involved in a live trading algorithm. When developing trading algorithms, you are basically trying to model and quantify reality so that your algorithms can make trading decisions based on this model of reality. To do so, your algorithm requires some kind of input or connection to reality. This is what data is for. Data can come in pretty much all shapes and sizes. Most commonly, a trading algorithm will require pricing and time data, and besides that it might also need fundamental data, certain market stats, news data, order data, and more. After receiving this data, the trading algorithm will then use and analyze this data to make trading decisions. These trading decisions are then passed on to your broker. This is typically done through a so-called API, but theoretically you could also manually execute the traits that your algorithm is recommending. That would be a systematic discretionary hybrid approach. Luckily, getting data, hosting your algorithms, and communicating with your broker are all automatically taken care of by QuantConnect, which is the platform we will be using for this series. That's why you won't have to worry about this stuff too much. Nonetheless, it is very important to understand these things. Next up, let's talk about the process of developing algorithmic trading strategies. In my opinion, this process is overlooked way too often. Especially newcomers think that all algorithmic trading consists of is writing some code for a trading bot, and then you can set and forget it. In reality, this process looks quite different, so let me walk you through it. First and foremost, it is important to understand that what distinguishes a good from a bad trading strategy is an edge. The goal of an algorithmic trader is to best possibly identify such an edge and then build a strategy around this edge. Note that the edge comes first. You can't just start with some random strategy and then try to add an edge. This edge is the centerpiece of the entire strategy. It is also important to understand that just because a trading strategy achieves profitability over a certain time frame does not mean that it has an edge. On the flip side, a strategy that is not profitable can very well have an edge. This is the case because in the world of Quantitative Finance, profitability is not the main benchmark that you use to compare your strategy's performance to. Instead, you always want to compare your strategy's performance to a relevant benchmark which usually is some market index in the given sector. If for instance, your strategy focuses on trading large cap US equities, a good benchmark would be the S&P500 index. So even if your strategy manages a yearly return of 5%, it might not have an actual edge since it's being outperformed by the S&P500. Note that, besides looking at the total returns, it is also very important to look at the risk and volatility associated with these returns. Risk adjusting your returns adds a great deal of comparability. If your strategy manages to generate excess returns over its respective benchmark over a significantly long enough period, it likely has an edge. This edge or excess returns are also known as ALFA. So the first step of developing trading algorithms is to find some form of ALFA. This is usually best done by testing different hypotheses. Testing your hypothesis is best done in a scientific manner using a systematic data-driven approach. Let me give you a quick example of how you could approach finding ALFA. Let's say your hypothesis is that generally speaking, small cap stocks tend to outperform large cap stocks. To test this, you first would need to clearly define what small cap and large cap stocks are. There are different metrics to measure this, but let's say we just use market capitalization. Then you could compare the historical performance of a large number of equities to their market cap. If you can find some kind of meaningful correlation, there might be a strategy that could profit from this correlation. Besides looking at the data, it is also best to try to find some logical explanation as to why your edge exists. Try putting it into cause and effect terms. For example, one possible explanation as to why small cap stocks might generally outperform large cap stocks would be that small cap stocks have much more room to grow than their larger counterparts. If you aren't sure how to perform such research, don't worry, QuantConnect offers a build-in research environment in their platform which we will be exploring in a later lesson. But besides doing your own data-driven research and tests, you can also take advantage of the countless scientific papers that have been published about various possible factors and correlations in the markets. A great website that allows you to read about thousands of different possible strategies is called Quantpedia. It has an archive of thousands of strategies, fundamental breakdowns, relevant papers and other useful data that can be accessed free of charge. Besides Quantpedia, other useful resources are QuantConnect strategy library and forum. Note that it is important to understand that alphas can change. Just because a strategy used to work does not mean that it still does all will in the future. In fact, strategies that work very well often tend to worsen with time since more and more traders will pick up on them. World strategies only have a limited capacity until they stop working. After you have found some alphas, it is finally time to build and code a strategy around this edge. Here it is best to draw out your strategy on a conceptual level before implementing it in actual code. Learning how to implement your strategies using Python is what the main focus of this video series will be on. This implementation part is probably where you have the greatest degree of freedom. Usually, the research part should already have restricted the possibilities to some extent. But there typically are many possibilities of how you could implement a strategy and take advantage of the found edge. Depending on your programming and data science skills, you could go for a very basic implementation, something intermediate or a more complex implementation, for instance using advanced machine learning concepts. Be aware, however, that coding a trading button or anything else for that matter is not nearly as seamless as it might seem in some video tutorials. In reality, you rarely just write down your code in one go and then everything works. Instead, there is a lot of trial and error testing, bug hunting, sanity checking and reading of the documentation forums involved. After you have successfully coded your trading bot, it is time to back test it. Back testing basically allows you to analyze how this strategy would have performed if you would have traded it in the past. This is a very powerful tool since you can gain great insights into the potential of your strategy without risking any money. Nonetheless, it is very important to understand that back test results are not as expressive as real world results. Back testing results can be influenced by a number of biases and unrealistic models. Some of these include overfitting, look ahead bias, survivorship bias among others, but more than these in a later video. For now, just remember to take back testing results with a grain of salt. After thoroughly analyzing your back test results, there are a few optimization procedures that you can apply to your algorithm to potentially improve its performance and versatility. Besides back testing, it is also recommendable to paper-trad your algorithm before funding it with any real money. This is a great last check to test if everything is working properly. If all of the previous steps have yielded positive results, it is finally time to deploy your algorithm and fund it with actual money. Here it is also best to start slowly. After deploying your algorithm, you should not just set and forget it. Instead, you should somewhat actively monitor it and double check if it's doing what it's supposed to do. However, what you should not do is constantly interfere and override the trades from your algorithm. If it's doing something it's not supposed to do, then that's a different story. But otherwise, you can't expect to get any representative results if you aren't giving your algorithm the chance to actually trade according to its programmed strategy. I hope that this walkthrough gave you some insight into what the process of developing trading algorithms looks like. Summed up, you should understand that one of the most important aspects of creating trading algorithms is first researching and finding an edge. Besides that, algorithmic trading is all about modeling and quantifying reality in the most efficient way. For this, it is important to be very concrete and concise. Everything needs to be clearly defined so that your algorithm can understand it. For example, buying case of an uptrend is not something that can be understood by a computer. The problem here is that an uptrend can be interpreted in many different ways. One possible simple definition of an uptrend would be that the 60-day moving average is above the 90-day moving average. Last but not least, you should always remember that coding your own bots takes a lot of trial and error, experimenting and looking at the documentation or forum posts for help. So please don't get frustrated when you can't just code your bots in one go. That said, I hope you enjoyed this video and learned something new. In the next video lesson, we will head over to the QuantConnect platform and start looking at some of QuantConnect's key concepts so that we can start writing our first trading bot very soon. If you are enjoying this free algorithmic trading video series, make sure to subscribe, turn on the notification bell and smash the like button. Thanks for watching. diff --git a/course/03-key-concepts.md b/course/03-key-concepts.md new file mode 100644 index 0000000..a57971e --- /dev/null +++ b/course/03-key-concepts.md @@ -0,0 +1,29 @@ +# 03 · Key Concepts（关键概念：时间处理·骨架·Symbol） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Key Concepts（关键概念：时间处理·骨架·Symbol） +- **时长**：10 分 5 秒 ｜ **原视频**：https://youtu.be/28kE5JIi4-0 +- **本地视频**：[videos/03-key-concepts.mp4](videos/03-key-concepts.mp4) + +## 🎯 本集要点（中文导读） + +首集进入 **QuantConnect API**，讲**时间处理**与**算法骨架**、**Symbol**。 + +- **Lean 引擎**：QuantConnect 基于开源 Lean。理解 Lean 如何处理时间至关重要。 +- **前视偏差（Look-ahead Bias）**：回测时若误用未来数据，回测会"看起来很好"，实盘必崩。示例：用"当日收盘价"做决策＝偷看未来。 +- **Lean 的时间规则**：数据是**流式**的；一根 bar **只能在其 end time 及以后**使用。用日线时，用周五收盘价做的决策，要到下一个 bar（周一开盘）才真正发出。 +- **数据形状**：tick（单一时间点）vs bar（有区间，含起止时间）；支持日/时/分/秒 bar，也可自定义 consolidated bar。同一算法可混用多种分辨率。 +- **time frontier（时间前沿）**：不能访问其之后的数据；默认时区为**纽约**。 +- **算法骨架**：所有算法继承 `QCAlgorithm`，提供大量辅助函数/属性： + - `Initialize()`：初始化（加数据、设回测参数、初始化变量等） + - `OnData(data)`：每来新数据（tick 或 bar 到达 end time）时触发——最常用 + - 还有 `OnEndOfDay`、`OnOrderEvent` 等 + - `self.Time`（当前时间）、`self.Securities`、`self.Portfolio` +- **Symbol 对象**（标识证券）：形如 `ticker+hash`，编码了 8 个属性：`Value`(ticker)、`SecurityType`、`Market`、`HasUnderlying`、`Date`、`OptionType`/`OptionRight` 等。 +- ⚠️ **别直接用 ticker**：ticker 会变、会被不同券商复用，产生歧义。应始终用 **Symbol 对象**：`self.AddEquity("AAPL").Symbol` 保存复用。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to the third video of this algorithmic trading video series. This video is the first video where we actually start looking at the QuantConnect API. The goal of this video is to present some of the most important key concepts as well as a basic trading algorithm skeleton inside of QuantConnect. With this knowledge, we can then finally start creating our very first basic trading algorithm in the next video lesson. Like last time, I want to start by quickly thanking the sponsor of this video series. QuantConnect has seen some of my algorithmic trading videos and they have offered to sponsor this series. So, thanks for the support. QuantConnect's trading engine is powered by the open source Lean engine. To truly understand how you can develop trading algorithms using QuantConnect and Lean, it is essential to understand how Lean handles time. For this, let's first quickly look at one of the biggest problems related to handling time inside of an algorithmic trading engine. Namely, look ahead bias. When testing your trading algorithm on historical data, you have to simulate the passing of time. If this is done incorrectly, your trading bot might try to access future data to make its trading decisions. Let me give you a very simple example of look ahead bias. Let's say you have a bot that only trades Apple. For its trading decisions, it compares the closing price of the current day to the current price. The problem with this is that the closing price is not yet actually known, so using that to make decisions is like seeing the future. I know this might seem like an obvious mistake, but in more complex algorithms, look ahead bias can be much, much harder to detect. The worst part about look ahead bias is that back testing will work just fine. Only when you switch to live trading, will your bot stop working since it can't actually access the future data anymore. The Lean engine uses streaming technology that allows you to access data from the past or present. This makes life a lot easier since you won't have to worry about look ahead bias anymore. However, it can also lead to some confusion, especially when you're just starting out, which is why I want to put special emphasis on this topic. One Connect has two shapes of data. On the one hand, they have tick data, which is data for a single point in time, and on the other hand, there is bar data, which has a period. Since bar data covers a period, it has both a start time and an end time. For tick data, start and end time are the same. The important thing to understand is that Lean only allows you to use the data of a bar at its end time, so that the start of the next bar or later. This restriction ensures that you don't accidentally use future data and thereby full prey to look ahead bias. This means that, if your algorithm uses the closing price of a given bar to make a trading decision, any orders sent out based on this decision are first sent out at the start of the next bar. So, if you're using daily data and you want to use Friday's closing price, your orders will first be ready at the start of the next bar, which will be on Saturday. But since markets usually are closed on Saturday, they would first be sent out at the open of the following Monday. So the Friday bar includes all ticks from Friday at 0 o'clock to Friday right before midnight. Besides daily bars, QuantConnect also has hourly, minutely and secondly bars. Besides that, you can also create your own custom consolidated bars for custom resolutions. You will learn how to do this in one of the later videos. The same algorithm can also use multiple different bar resolutions for different data. The current time is controlled by the so-called time frontier. To avoid look ahead bias, you cannot access anything behind this time frontier. Note that if you're using custom data, there are a couple of extra things you need to look out for, but more on that in a later lesson. The default time zone inside of a QuantConnect algorithm is New York's time zone, but this can be reconfigured if you want to. Now that you hopefully understand how time is being handled inside of Lean, let's move on to the skeleton of a QuantConnect algorithm. Every algorithm extends the QC algorithm class which provides you with hundreds of helper functions and properties. If for instance, you ever want to access the current time to check where the time frontier is at, you can do so with self.time inside of the QC algorithm class. Besides that, this class implements portfolio, securities, transactions and data subscription management. Let me quickly walk you through a few of the most commonly used properties just to give you a basic idea of this class. But if this all seems a little overwhelming, don't worry, in my opinion the best way to learn this is by creating plenty of example algorithms, which is what we will start doing in the next videos. Probably the most used method of the QC algorithm class is the initialised method. As the name implies, this method is used to initialise your algorithm. You can add what data you want, which models you want to use, edit back testing settings, initialise helper variables and much more. Next up, if a bunch of event handlers, I'd say the most commonly used one is the on data method. This is called every time the algorithm receives new data, so every time a bar reaches its end time or a tick event occurs. Besides that, there also are more specific event handlers such as on end of day or on order event, which are called at the end of each day and in case of an order event like an order being filled. Other important properties of the QC algorithm class include self.securities, which is an array of security objects and self.portfolio, which is used to access the holdings inside of your portfolio. Speaking of securities, let's now take a look at how QuantConnect handles accessing securities. For this, QuantConnect uses symbols that look something like this, a ticker followed by a hash code. This weird looking string encodes all the important information regarding a security in an easy to process fashion. To understand what exactly it encodes, let's take a closer look at the properties of a symbol object. But first, note that a symbol object can not only represent a stock, it can be anything from an equity, option, future, forex pair, cryptocurrency or a CFD. A symbol object has 8 main properties. The first is its value, which is basically the ticker of that security. More exactly, this is the ticker that you referenced when adding the data for this security. So for instance, the value of this symbol would be AAPL. Next up, a symbol object also has a security type, which can be an equity, option, future, forex pair, cryptocurrency or a CFD. Then there also is the market property, which can be used to determine on which market the security trades. For most equities, this would just be the US market. But for forex or cryptos, the same currency can be traded on different markets such as GDAX or Bitfinex. This property helps to distinguish between them. Besides that, the ultra-stable property symbol dot has underlying, which indicates whether this security is a derivative or not. For instance, has underlying would be true for options or futures, but falls for equities. In addition to that, we have symbol dot date, which saves the earliest listing date for equities and the expiration date for futures or options. For options, other relevant properties include symbol dot option type and symbol dot option right, which represent whether the option is an American or European option and whether it is a cool or a poor option. All these properties are encoded into the hash string after the ticker. So why can't we just save the ticker by itself? Usually, QuantConnect does allow you to use the ticker of a security directly to add and access data. However, this is not recommended since tickers do not hold nearly as much information as symbol objects. The problem is that the ticker of a stock can change and it can be reused by different brokerages. This can lead to a lot of unwanted ambiguity, which you definitely want to avoid. That's why you should always use a symbol object instead of a ticker. To create and save a symbol object, you can simply add dot symbol when adding data and save this object into a variable. So for example, let's say you have an algorithm that trades apple. Then you can add apple data by using self dot add equity aapl. You can then save the apple symbol object by appending dot symbol to add equity. You can then later use this save variable when sending out an order for apple instead of using the aapl ticker. But that's just to give you a short taste of a few helper functions that the QC algorithm class provides. In the next video, we will start exploring the API in a more detailed manner and we might even write our first simple trading bot. For now, you should just understand how lean handles time and more specifically you should always remember that events are transmitted at the end time of each bar. So at the end time of the last bar, the on data method of the QC algorithm class is called in which you can analyze the data from this bar and use it to make trading decisions. You can't access any data beyond the time frontier. Furthermore, you now hopefully understand how QuantConnect allows you to identify different securities, namely by using symbol objects. It is highly recommendable to always use the actual symbol object instead of the ticker for this. Like I said, I hope you enjoyed this video and learned something new. If you did, don't forget to subscribe, turn on the notification bell and smash the like button for more content like this. Thanks for watching. diff --git a/course/04-handling-data.md b/course/04-handling-data.md new file mode 100644 index 0000000..38127b5 --- /dev/null +++ b/course/04-handling-data.md @@ -0,0 +1,30 @@ +# 04 · Handling Data（处理数据 · 第一个算法） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Handling Data（处理数据 · 第一个算法） +- **时长**：18 分 1 秒 ｜ **原视频**：https://youtu.be/WQwyKfef80k +- **本地视频**：[videos/04-handling-data.mp4](videos/04-handling-data.mp4) + +## 🎯 本集要点（中文导读） + +**写第一个真正的算法**（实战入门）：买入并持有 SPY，涨/跌超 10% 就平仓，然后空仓 1 个月再循环。 + +- **入口**：QuantConnect → Lab → Create New Algorithm → `Exit Builder mode` 得到空白模板。 +- **Initialize()**： + - `SetStartDate` / `SetEndDate`（不设 end 则取最近）、`SetCash` + - `AddEquity("SPY", Resolution.Daily)` 添加数据（分辨率：tick/秒/分/时/日；一般不建议用 tick） + - `SetDataNormalizationMode(...)`：数据复权方式——默认 `SplitAndDividendAdjusted`，另有 `Raw`/`SplitAdjusted`/`TotalReturn`（决策权：期权等只支持 Raw） + - 保存 `Symbol`、`SetBenchmark(SPY)`（自动生成基准图表） + - 券商模型（默认即可，也可选 InteractiveBrokers 等）与账户类型（Cash / Margin，含 PDT 规则） +- **OnData(data)**：`data` 是 **Slice** 对象（含 `Bars`/`QuoteBars`/`OptionChains` 等，按 Symbol 索引）。 + - 取价三法：`data.Bars[self.spy].Close` / `data[self.spy].Close` / `self.Securities[symbol].Close` + - 用 `data.ContainsKey(symbol)` / `in` 判断数据是否就绪 +- **下单/平仓**：`self.MarketOrder(symbol, qty)`（负数=卖出）、`self.SetHoldings(symbol, 1.0)`（按比例，1.0=满仓）；`self.Liquidate()` 一键清仓。 +- **回测**：点 Build → Backtest，查看权益曲线、基准对比、统计指标、Logs / Orders 标签页。 +- ⚠️ 作者强调：这只是教学示例，"等 1 个月"等规则是随意的，**不是能实盘的好策略**。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to the fourth video of this algorithmic trading series using Python and the QuantConnect platform. So far, we've mainly had theory-based videos covering some of the key concepts of the algorithmic trading development process and the QuantConnect API. In this video, we will change things up a little by writing the very first actual trading algorithm of this series. While doing this, we will get to know the basics of the QuantConnect API in a practical manner. But before we head over to the QuantConnect platform, let me quickly thank the sponsor of this video series, namely QuantConnect themselves. I'm very glad that they offered to sponsor this series, since in my opinion that's an absolute win-win situation. So once again, thanks for the support. That said, let me quickly outline the idea behind the trading bot that we will implement in this video. Firstly, I have to say that this will be a very basic trading bot that will hopefully show different aspects of the QuantConnect API. So please don't expect to develop a trading bot that you would actually want to fund with real money. The idea behind this trading algorithm will be to buy and hold SPY, which is the ETF tracking the S&P 500 index. To make things a little more interesting, we will want to close our position as soon as it loses more than 10% or achieves a profit greater than 10%. After this, we stop investing for a predetermined period of time such as one month. Thereafter, we start this cycle all over again by investing in SPY again. With that out of the way, let's now head over to the QuantConnect platform. If you have not yet created a free QuantConnect account, there's a link in the description box below. Feel free to code along. On the QuantConnect website, you will want to navigate to the lab tab. There you will want to click on Create New Algorithm, which will bring you to the Strategy Builder view. The Strategy Builder can be very useful for more complex algorithms, which is why we will explore this feature later in this series. For now, we just want to start with a blank template algorithm by clicking on Exit Builder mode. This will bring you to a simple algorithm template with an algorithm class as well as an initialised and on-data method. The on-data method is an event handler that is being called every time your algorithm receives new data. More specifically, it is called every time a tick event occurs or a bar reaches its end time. For a more detailed breakdown of this, make sure to check out the previous video of this series. Before we start implementing the on-data method, let's first take care of the initialised method which is unsurprisingly used to initialise your algorithm. First and foremost, we will set the start and end date for backtesting. If you don't specify an end date, the most recent date will be chosen. Next up, we want to specify the starting cash balance for your algo. Once again, this is just for backtesting purposes. In real-life trading, this summer will be taken from your brokerage account balance. After that, we will add the data for the security that our algorithm will trade. Since we wanted to trade SPY, we will use Add Equity and the SPY ticker symbol. As a second argument, the Add Equity method takes a resolution. Here, the lowest resolution that you could specify is tick resolution which can be as low as a tick per millisecond. But note that such a low resolution will lead to a lot of data points which can be very hard to process efficiently. Furthermore, tick data is raw and unfiltered which can lead to some unwanted problems. So if you aren't completely sure of what you're doing, I wouldn't recommend going lower than Minute Resolution for now. But to be honest, you won't really need that low of a resolution for most use cases anyway. For this algorithm, we will just be using daily resolution. Since QuantConnect also supports a bunch of other asset classes besides equities, you can also add other securities. For instance, to add a Forex player, you could use Add Forex. For futures, you would use Add Futures for CFDs, and so on. In the next step, we will set the Data Normalization Mode for our SPY Equity data. QuantConnect supports four different data adjustment modes. The default data mode is split and dividend adjusted. This means that the data is smoothed out so that the stock splits don't look like certain huge price drops or spikes. This can be very useful since it makes the data easier to handle. Besides that, QuantConnect also supports raw data which isn't adjusted at all. Here dividends are paid in cash and the stock's price is not adjusted. Certain asset classes such as options only support raw data. To change the data mode, you can use the Set Data Normalization Mode method and specify the Data Normalization Mode. There also exists only split adjusted data and total return data which automatically accounts for reinvested dividends. Note that most of the time you won't have to explicitly change the Data Normalization Mode since you can just use the default mode, but nonetheless it is good to know how to do so. For this algorithm, I will just use Raw Data so that you can see how to change the Data Adjustment Mode. Next up, let's save the Symbol Object of SPY into new variable. If you aren't sure what Symbol Objects are or why they are important, I recommend watching the last video of this series. But in short, Symbol Objects hold more information than tickers which can help against unwanted ambiguity when referencing a given security. After saving this Symbol Object, we can set a benchmark for this algo with self.setbenchmark. Since we will be trading SPY, we will just use SPY as the benchmark. Here you usually want to choose some relevant market index in the trade sector. Setting a benchmark will automatically generate a chart for the benchmark for the specified period when backtesting your algorithm. This can help when analyzing the performance of your algo. QuantConnect also allows you to set different brokerage models so that your algorithm best possibly accounts for your broker's fee structure and account type. The default brokerage model usually is good enough, but you can also specify a brokerage model such as the Interactive Brokerage Model, Trade Here Brokerage Model, One-Dab brokerage, FXCM brokerage, Bitfenix and more. These different models will then adjust the fee structure to the chosen brokerage. Besides choosing a brokerage model, you can also set the account type to a cash or margin account. A cash account does not allow you to use leverage and it has a settlement period of three days for equities. Margin accounts however allow you to use leverage from two to four times your capital. You can also use the pattern day trading margin model to account for the PDT rule which allows you to pattern day trade only if you have more than $25,000 deposited in your account. For this algo we will just be using the Interactive Brokerage Model as well as a margin account, but note that this algorithm could also easily be used in a cash account. Usually in cash accounts you just have to be careful to account for the T plus three settlement rules. Last but not least we will quickly create three custom helper variables that we will need for this algorithm. The first is self.entry price which will track the entry price of our SPY position. The second one is self.period which we will set to a time frame of 31 days and lastly self.next entry time will track when we should re-enter our long SPY position. We will initialize this to the current time since we want to start investing right away. Now we are finally done with the initialize method which means we can move on to implementing the on data method. Remember this is called every time the end time of a bar is reached or a tick event occurs. So basically every time the algorithm receives new data while still accounting for look ahead bias. The on data method has one parameter which is called data. Let's quickly look at the theory behind this. The data parameter is a slice object that arrives at the event handler such as the on data method. It provides you with a couple of helpers to access the data in a structured manner. The properties of a slice object include option chains, trade bars, quote bars, spits, among others. These are dictionaries that are indexed by symbols. Let me now briefly break down the most important data type that you can access through such a slice object. All of these data types extend the base data class which provides you with symbol, value and time information. First and foremost you can access tick data. Tick data's most interesting property is the last price. But note that like I said earlier, tick data is rule and unfiltered and it might contain some bad ticks. So generally you should be cautious when using tick data. The next important type of data is trade bar data. Since this bar data it covers a period of time and is passed on to the event handlers at its end time. Trade bars are supported for equities, options and futures. It provides you with open high, low, close and volume information. Trade bars are built by consolidating trades from exchanges. Note that we will go over accessing trade bars in a minute. I just first want to quickly cover the most commonly used data types on a conceptual level. Besides trade bars, there are also quotes bars which are supported by all asset types, so also by Forex, crypto and CFDs. The main difference between quote bars and trade bars is that quote bars are built by consolidating bid and ask prices from exchanges. Quote bars offer open, high, low, close, bid, ask, last, bid size and last, ask, size properties. Here the bid and ask properties are bar objects themselves which once again include open, high, low and close info. The open, high, low and close properties of the quote bars are generated midpoints from the bid and ask bars. Other data types include delisting, simple change events, splits and dividends, but since these aren't used as often I won't cover them in too much detail for now. Now that you hopefully have a much better picture of the theory behind the shape of the data parameter of the on data method, let's start implementing the on data method. Firstly, we will want to save the current price of SBI into a variable named price. There are multiple ways of accomplishing this. The first would be to access the bars of the data slice object. Data.baras is a dictionary that we can index with self.spy, which is the symbol object for SBI. Since we want the most recent price, we will save the close price. Note that this will be the close price of the day before, since we don't yet know what today's closing price will be. Another way to accomplish the same would be to directly index the data variable. This will also return a trade bar object that we can use to find the last close price. A third way to achieve the same result would be to use the self.securities dictionary. We can then index self.securities by the symbol and save the closing price. When using the slice object to access data, it can sometimes be useful to first check if the requested data does already exist. If you just added the data or there's very little trading activity, your algorithm might not have data for a given symbol yet. To check if data exists, you can use .contains key or Python's in operator while specifying the requested symbol. Since SBI is very actively traded and we aren't dynamically adding data, this won't really make a difference for this algorithm. But in other cases, it can be very helpful to do so. Next up, let's finally implement the trade logic of this bot. Firstly, we want to check if our bot is already invested. We can do this with self.portfolio.invested which will return a Boolean. It is also possible to index self.portfolio with a symbol and then check if we are invested in this security specifically. But since we are only trading SBI, this won't be necessary here. Besides checking if we are invested, the portfolio property also allows you to check the available cash, remaining margin, total fees and other relevant information about your portfolio. If we aren't invested, we want to check if it's time to invest. As you hopefully can remember, this bot is supposed to buy and hold SBI until SBI drops or rises more than a certain amount. Thereafter, we will stay in cash for one month, where after we will buy and hold again. To account for this one month waiting time, we can use self.time to access the current time and then check if this is greater or equal to the next entry time. If this is fulfilled, we want to buy as much SBI as we can. We can buy SBI using self.market order, which unsurprisingly sends a market order for the specified symbol and quantity. Note that negative order sizes will be interpreted as cell orders. To calculate the order size, we divide the current cash in our portfolio by the current price of SBI. Since we want this to be an integer, we will cut off any decimal places by casting it to an end. An easy alternative to manually calculating the order size will be to use self.set holdings, which allows you to specify a certain percentage of your portfolio to allocate to a given security. So a one would mean that we would want to allocate 100% of our portfolio to SBI, which is exactly what we want to do. After this, we want to log that with just both SBI at the current price. Logging important actions can be very helpful when reviewing and debugging your algorithm. Instead of using self.log, you can also use self.debug, which will print things to the console, but more on this in the next video. Next up, we want to save the current price of SBI to the entry price variable, since we will need this for the exit condition. Note that this save price isn't actually the exact entry price, since a market order's fill price can very well deviate from the price of when the order was sent. But we will disregard that for now. Now, all that's left to do is implement the exit process and then we're good to go. For this, we will use an LF condition, which is reached when our portfolio currently is invested. Here, we check the condition whether the entry price either is 10% below or above the current price. In either case, we want to exit the SBI position. Once again, you could accomplish this with self.market order, but a far easier solution would be to use self.liquidate, which just liquidates all positions in your portfolio. Alternatively, you could also specify to only liquidate the holdings in a certain security by passing it simple as an argument. Here, we once again will log that we close our SPI position. Then, we will set the next entry time variable to the current time plus self.tor period, which we initialize to 31 days. This makes sure that for the next 30 days, we will stay in cash. With that, we've now successfully implemented our first simple trading algorithm. What there's left to do now is click on build and then on the back test button in the top right. This will then start a back test, which simulates how this bot would have traded over the specified back test time frame. Depending on the algorithm and time frame, back testing can take anything from a few seconds to hours. However, this algorithm should not take more than a few seconds. After the back test has finished up, a performance overview like this one will be generated. At the top, you can see an equity chart showing your strategy's performance. On this chart, you can clearly see the periods where our algorithm held a break from investing. Above this chart, there are some stats such as total profit, total fees, returns and more. In the top right corner, you can select which chart should be displayed. For us, the benchmark chart is probably one of the most interesting ones since it allows us to compare the performance of our bot to SPY. As you can see, the two seem to be highly correlated, which shouldn't be very surprising. This means that this bot would likely perform very poorly in bear markets. At the bottom, there are a bunch of other useful stats that can be used to analyze your strategy's performance. I will cover all of this more in-depth in a later lesson. At the bottom, you can also navigate to the logs tab, which shows you all the generated logs from your algorithm. Here, you can get a rough overview of when and at what prices we bought and sold. For a more detailed order overview, you can go to the orders tab. Here, you can see all the orders that your algorithm send as well as the field prices and other relevant info. Before I end this video, I want to emphasize that this is not a good trading strategy or a good trading bot. Waiting for one month after a 10% move is quite arbitrary and not really based on anything. The purpose of this bot is to give you an introduction to creating your first trading bot and not to make as much money possible. If you want to copy the code from this video, you can do so by using the link in the description box below. That said, I hope you enjoyed this video and learned something new. In the next video, we will start working on a more dynamic trading bot and we will cover topics such as saving and updating orders, using and creating indicators and much more. If you enjoyed the video, make sure to subscribe, turn on the notification bell and smash the like button for more content like this. Thanks for watching. diff --git a/course/05-trading-and-orders.md b/course/05-trading-and-orders.md new file mode 100644 index 0000000..0ac6e1d --- /dev/null +++ b/course/05-trading-and-orders.md @@ -0,0 +1,26 @@ +# 05 · Trading & Orders（下单、订单管理与调试） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Trading & Orders（下单、订单管理与调试） +- **时长**：28 分 46 秒 ｜ **原视频**：https://youtu.be/qkyvj5LIg0M +- **本地视频**：[videos/05-trading-and-orders.mp4](videos/05-trading-and-orders.mp4) + +## 🎯 本集要点（中文导读） + +本集深入 **下单（订单）与调试/日志**。 + +- **Order Ticket（订单票据）**：下单后返回的对象，用来追踪/查询/修改订单。因为订单是**异步**的（更新需请求、不保证成交），所以要靠 ticket 管理。常用属性：`Status`、`OrderId`、`Symbol`、`Quantity`/`QuantityFilled`、`AverageFillPrice`、`OrderType`。 +- **5 种订单类型**： + - **Market Order**（市价单）：`self.MarketOrder(symbol, qty)`；按最优价成交（买=ask、卖=bid，含点差）；默认**同步**（最多等 5 秒，可设 `self.Transactions.MarketOrderFillTimeout`），也可异步。 + - **Limit Order** 限价单、**Stop Market Order** 止损市价单、**Stop Limit Order** 止损限价单、**Market On Open/Close**（开盘/收盘市价单）等。 + - 更新/取消订单：先取 ticket，再 `Update`/`Cancel`（仅未成交时有效）。 +- **`self.Transactions`**：访问订单/成交历史。 +- **调试与日志**：`self.Log()`（日志，回测/实盘可见）、`self.Debug()`（控制台）；**Log 有容量上限**，别刷太多。还可用平台的 **Debug 模式**（打断点、加 Watch 变量、逐步执行）。 +- **自定义图表**：可绘图（后续课细讲）。 +- 本集用一个示例 bot 演示下单、更新、取消与调试流程。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome to the fifth video of my algorithmic trading video course using Python and the QuantConnect platform. If you're new to this series, I recommend first watching the previous videos before continuing with this one. In today's video lesson, we will continue our closer exploration of the QuantConnect API. More specifically, we will dive into how to deal with all kinds of trade orders as well as how to make use of the debugging and logging tools that QuantConnect provides. For this, we will once again be creating a relatively simple trading bot. To make sure you get the most out of your learning, make sure to code along. If you haven't created your free QuantConnect account yet, you can do so using the link in the description box below. But before we jump into the action and start actually coding this trading bot, let's first explore how QuantConnect handles trade orders on a conceptual level. For this, we first look at the concept of order tickets. An order ticket is the object you get after creating an order of any kind. It is used to track, access and edit the given order. The reason why you need an order ticket object instead of just an object for the order itself is that orders are asynchronous. This means that any potential updates first need to be requested and there is no guarantee that they will go through. If for example, you send a limit order for 100 shares of Apple and then decide that you want to console or update this order a few minutes later, you can only do so if the order has not yet been filled. Some of the most interesting properties of an order ticket object are its status, order ID, symbol, quantity, quantity, field, average, field price and order type. But more on some of these later. Now, let's first take a look at how to actually place orders and thereby create ordered tickets in the first place. One connector puts five different order types that we will go over one by one now. The first most simple order type is a standard market order. Market orders are filled at the first available price and they can be sent by using the self.market order method of the QC algorithm class. Self.market order takes two required arguments. The first being the symbol that you want to send a market order for and the second being the order quantity. So a market order for 50 shares of Tesla would look like this. Note that here it would usually be better to use a symbol object instead of a ticker symbol but in most cases both would work. Since market orders usually feel right away, you can't really update them after they are sent. To make things as realistic as possible, QuantConnect does account for the spread. This means buy orders are filled at the asking price and sell orders are filled at the bid price. By default, the market order method is synchronous which means that your algorithm will wait up to five seconds until the order is filled before moving on to the next line of code. It is also possible to customize this timeout delay with self.transactions.market order filled timeout. As a third optional argument, you can also specify whether the market order should be sent asynchronously. Passing true here will mean that your algorithm will not wait before moving on to the next line of code. This can be useful if you are trading large order sizes, sending many orders or trading in illiquid markets and don't want your algorithm to always wait for the entire market order to be filled. The market order method or any other order type method for that matter will return an order ticket object. To save the corresponding ticket, you simply have to save the output to a variable. The next order type is a limit order which allows you to specify a price at which you want your order to be filled at. Limit orders usually take longer to fill or might sometimes not even get filled at all, but on the flip side they allow you to get better entry and exit prices than market orders. You can send a limit order by using self.limit order which takes three arguments. The first two are the symbol and the quantity so the same as for market orders and the third argument is the limit price. Since limit orders can take longer to fill or might not be filled at all, it is possible to update them but more in updating orders in a few minutes. Next up, there are also stock market orders. This is basically a market order that is sent as soon as a predetermined price is reached. This price can be specified as the third argument. So for instance, this stock market order would mean that a market order to sell 200 shares of SPY would be sent as soon as SPY's price drops by 10%, assuming that the SPY close variable represents buy's latest closing price. Note that if prices are very volatile or gapping up or down, a stock market order might be executed at a much worse price than the specified trigger price. Besides stock market orders, the ults are stop limit orders which basically work the same way as stock market orders with the only difference being that a limit order instead of a market order will be sent. Here you can specify the limit price as a fourth argument. Last but not least, the ults are market on open and market on close orders. Like the name implies, these orders are filled at the open or close price of a given security. These orders must be submitted at least two minutes before the open or close respectively. Here you can once again only specify a symbol and a quantity. Note that if you are using daily data or another relatively low resolution, there's a high chance that your market orders will be turned into market on open orders. This is the case because data is submitted at its end time which usually is during market inactivity. So if you send out any market orders based on this info and their market is closed, they will automatically be converted into market on open orders. If you are confused about what I mean with data being submitted on its end time, I recommend watching the third video of this series where I covered the essential topic of understanding how time is being handled inside of the lean trading engine. You might be asking yourself why there aren't any more complex order types such as trading stop-loss orders or contingency orders. Even though QuantConnect does not have a direct helper method for these, you can still create more complex orders out of the aforementioned order types. Later in this video, I will show you an example of how you could go about creating a trailing stop-loss order. Besides the type of order, you can also specify the time-in-force of an order which basically adjusts the length of time that an order stays open until it is filled. Note that the time-in-force can only be adjusted for non-market orders since market orders usually are filled right away. There are three different settings for the time-in-force property. The default value is good till console which means that the order stays open until it is either filled or manually cancelled. The second possible value is day which means that the order will stay active until the end of the day. The final possible value is good till date which allows you to specify a date until the order should remain open. If this day is reached and the order has not yet been filled, it will be cancelled. To change the time-in-force of an order, you have to set the time-in-force value of the default order properties attribute. For instance, changing the time-in-force today would look like this. Next up, let's take a look at how we can go about updating and cancelling orders. Like I already mentioned earlier, depending on the order type, the things you can update can vary. All order types allow you to update the tag property. Tags can be used to mark a certain order ticket for later reference but it has no actual effect on your trades. This table shows which properties you can update for the various order types. Updating the stop price of a stop order while the price of the security rises is one way to create a trailing stop order. To actually update an order, you need to create an update order fields object and then specify the properties that you want to update. You can then pass this object into the update method on the given order tickets. I will show this in a more detailed manner later when we code the actual trading bot. Updating an order returns an order response object which you can use to find out whether the update was successful or not. To cancel an order, you simply code the cancel method on the given order ticket. Note however that you cannot do this for market orders since they are typically filled immediately. Next, let's talk about position sizing. When sending trade orders, you usually want to dynamically adjust their position sizing in relation to the available capital in your portfolio. For example, instead of always sending an order for 100 shares, you'd probably rather want to send an order for 10% of your portfolio's capital. Luckily, QuantConnect has some helper functions to make this easier as well. The first and most commonly used helper function is one that we already used in the last video, namely the sets-holding method. Set-holding allows you to specify what percentage of your portfolio you want to allocate to a given security. Then your algorithm automatically computes the adequate order size and sends out a market order to reach the specified allocation. As an optional third argument, you can pass a Boolean that specifies whether existing holdings should be liquidated before your portfolio capital is allocated. Alternatively, you can use set-holding to scale your holdings up or down to a desired level. For this, you need to pass a list of portfolio target objects like this. Doing so will send out market orders so that your holdings in SPY will be scaled down or up to 80% and those in IBM to 20% of your buying power. If you don't want to use market orders for your trading, you can't use the set-holding cell per function. However, there's still another way to intelligently calculate order sizes. For this, you can use self.calculate order quantity which takes two arguments. The first is a symbol and the second is a percentage. This will then return an integer which represents the number of shares of the specified security that would make up the specified percentage of your available buying power. So for instance, this is how you could send a limit order that allocates 30% of your buying power to Apple at its current market price. Sometimes it can be useful to have a buffer of cash in your account so that you don't suddenly run out of buying power. To ensure that you always have enough cash reserves, you can use self.settings.free portfolio value percentage and set it to a percentage of your portfolio that you want to keep in cash. An easy way to close positions is using self.liquidate. Without any argument, self.liquidate will simply liquidate all your holdings using market orders. Alternatively, you can pass a specific symbol to only close positions in this security. Before we finally move on to the actual coding part of this video, there are few more things I want to cover. One of them is the unorder event method. This is an event handler that is automatically called every time an order event such as a change in the status of an order occurs. This is a great way to check when your orders are being filled and potentially act upon this information as soon as you have it. All order status includes submitted, filled, partially filled, cancelled, invalid and others. We will cover this in more detail once we start coding in a few minutes. Besides accessing orders through order events, you can also do so through your algorithm's transaction manager. For example, self.transactions.getOpenOrders allows you to get all the open orders for a specified symbol. Furthermore, you can use self.transactions.getOrderByID to get the orders by their order ID. Another helper method that the transactions manager provides is CancelOpenOrders, which allows you to cancel all open orders for security or the entire portfolio. This can be very useful in a scenario where you would want to stop all trading activity. Last but not least, QuantConnect also has the option for you to select from various different fee and brokerage models to model trading costs as realistically as possible. You can also create your own custom slipper and other transaction cost models, but since this is a more advanced topic, this is something we might cover later in this series. That said, let us now finally start putting some of the things learned here into practice by writing some code. Just like last time, let me quickly present the trading strategy that we will implement on a theoretical level before we start actually writing the code. Once again, the strategy that we will be implementing is a very simple strategy and I would not recommend trading it with real money. Its main purpose is to show how orders are being handled inside of QuantConnect. The idea behind this strategy is to buy and hold a given stock or ETF. We then create a trading stop-loss order that trails the stock's price 5% below its price. This hopefully allows us to keep our losers relatively small while leaving the winners room to go up. If our stop-loss is hit, we exit the position and then we wait for one month before we start investing again. If we wouldn't wait for some time before investing again, the trailing stop would be quite useless since we will just immediately re-buy after closing the position. For this video, we will be using the ETF QQQ for our trading, but feel free to try out different stocks or other ETFs. That said, let's now head over to the QuantConnect platform and start implementing the code for this strategy. Out of the QuantConnect platform, we will create a new algorithm inside of the lab tab. Since we don't need the strategy builder, we will exit the builder mode and start with a blank template algorithm. As always, we will then start by implementing the initialize method. This is where we will set a few backtesting settings, add the necessary data and define some help for variables that we will need later. First off, we start by setting the start and end dates for our backtest. Here you can choose whatever time frame you want to. I'll just go for 2018 until 2021. After that, we set the start and cache balance for this backtest. Remember that this is only for backtesting. In real trading, this will be your account balance. Thereafter, we want to add the data for QQQQ, which is the security that this algorithm will be trading. For this, we can use the add equity method and pass the QQQ ticker symbol as well as a resolution. Here we will go with hourly resolution, but you could also use daily or another resolution. We then save the symbol object of QQQQ to the variable self.qqqq, which we will use to reference this security in our algorithm. Now all that's left to do in the initialize method is define and initialize a couple of helper variables. Here, we first create two variables that we will use to access the order tickets of our entry and exit orders. We call the first variable, entry ticket, and the second one, stop market ticket, and we initialize both the none. Then we create two more variables to track the field time of the entry and exit orders. This will be used to make sure that we wait 30 days before starting to invest again. Here we call the first variable, entry time, and the second, stop market field time. We initialize both for the earliest possible date since no orders have been filled yet. First but not least, we create the helper variable self.highestprize, which unsurprisingly keeps track of QQQ's highest price. We need this for our trailing stop loss later. With that, we are now done with the initialize method and can move on to the on-data method. This is called every time the algorithm receives new data. For a more detailed breakdown of how data flows inside of the QC algorithm class, make sure to check out some of the previous videos of this series. Before we write the actual Python code, let's sketch out what we want to do with a few comments. The first thing we want to do in the on-data method is check whether 30 days have passed since we closed the last position. Then we want to send an entry limit order for as many shares of QQQQ as we can buy. Since we will be using limit orders, there's no guarantee that they will be filled. If the limit order is not filled within one day, we will move up the limit price so that we will increase the chance of getting filled. The final thing that we want to do in the on-data method is move up the price of the trailing stop loss if necessary. You might have noticed that we don't actually send out any stop loss order anywhere in the on-data method. That's because we do this in the on-order event method. This event handler is called on every order event and this is where we want to send out the stop loss order. However, we only want to do this if the entry limit orders feel it. Last but not least, we want to save the current time in case our stop loss orders feel it. We need to do this so we correctly stop investing for 30 days after closing our position. That's basically what our algorithm will do, so all that's left to do now is code this out and then we're good to go. I will start with the on-data method. Waiting 30 days can be accomplished with a simple if statement. For this, we simply check if the difference between the current time and the stop market order feel time is less than 30 days. If it is, it's not time to invest again and we return. Otherwise we move on. For the rest of the method, we will need QQQ's price. As you hopefully can remember from last video, there are multiple ways to access QQQ's price. We will just index the self.securities dictionary and save the most recent price to the price variable. Then we want to send the limit order. However, we only want to do this if we aren't currently invested and there aren't already any other active orders for QQQ position. We can check this with portfolio.invested and transactions.getOpenOrders respectively. If there aren't any open orders, transactions.getOpenOrders will be an empty list and evaluate to false. If this condition is fulfilled, we first want to calculate the order quantity. For this we can use the helper function calculate order quantity. Here we pass self.qqq as the symbol at 90% as the allocation of our portfolio. This will then calculate the number of shares that we need to buy to achieve a 90% allocation of our portfolio to QQQQ. With this, we can then send a limit order with self.limitOrder. As an optional fourth argument, we can pass a tag for this order. For the sake of showing this, I will tag this order with entry order. Since we might want to update this order later, we will save its order ticket to the entry ticket variable. Furthermore, we will save the current time to the entry time variable. The next step is to move the limit price in case the limit order is not filled after one day. For this, we first check whether more than one day has passed and then we check whether the limit order has been filled or not. We can check this by accessing the status property of the entry ticket order ticket. If it has not been filled, we want to update the entry time to the current time as well as update the limit price. To update the limit price, we create an update order fields object. Then we set the limit price of this update order fields object to the current price of QQQQ. Finally, we can then update the limit order by passing the update order fields object to the update method on our entry ticket order ticket object. Now what that left to do in the on data method is update the trailing stop loss price when QQQQ's price reaches a new high. To accomplish this, we first must check if the stop market ticket object is not empty and that we are invested. This ensures that we currently have an active stop market order for our QQQQ position. If this is the case, we want to check if QQQ's most recent price is higher than our previously saved highest price. In that case, we first need to update the save.highest price variable to account for this new high. Then we once again create an update order fields object. This time however, we will set the stop price to a new value, namely to 5% below the most recent price which should be the same as QQQQ's highest price. Thereafter, we just have to update the stop market ticket and then we are already done with the on data method. Now all that left to do is add a few lines of code to the on order event method. Inside of the on order event method, we will first check if the status of the submitted order event equals filled. Other possible statuses include submitted, invalid, partially filled among others. But we are only interested in the cases where an order has been filled. That's why we return if this is not the case. After that, the first case we want to cover is the case in which the entry order has been filled. To check this, we first check that the entry ticket variable is not none and that the order ID of the current order event in fact equals the order ID of this entry ticket object. In that case, we know that our entry limit order has been filled. That's why we can now send a stop market order for our QQQ shares. For the quantity, we can just check the quantity of our entry ticket object since we want a stop market order for the entire position. Note that we pass a negative value here since this is a settled order. The third argument of the stop market order method is the stop price. We want this to be 5% below the average filled price of our entry order. After this, the only other case that we need to handle is the case in which the stop market order has been filled. To check this, we once again have to check whether the stop market order ticket object is non-empty and the order ID of the order event parameter is correct. In that case, we simply want to update the stop market order filled time variable to the current time. This is important since this ensures that we will wait 30 days starting now. Last but not least, we reset the self dot highest price variable to zero since we don't know if QQQ's price will be lower in a month from now. With that, we now have successfully implemented this trading bot. Now we can click on the build button and then on the back test button to test this bot over the specified time period. Since this is a relatively simple bot, the back testing should not take very long. When the back test is finished, you will see a performance overview like this one. I don't want to go over this in any detail since the point of this video is not to create a particularly well performing bot. I will cover how to use this report to analyze the performance of a strategy later in this series. For now, I just want to scroll down to the orders tab so that we can make sure that this bot is in fact doing what it's supposed to do. Even for relatively simple bots, it can be very helpful to look at the orders that are being sent to get a better understanding of what it's doing. This can be especially helpful when your bot is doing something it's not supposed to do and you're trying to find out why. Here we can see an overview of all orders that have been sent. If we click on the orders, we can see a more detailed breakdown of them. In the rightmost column, you can see the custom order tag that we submitted for the entry limit orders. As you can see, the first order here was a limit order for 584 shares of QQQ with a limit price of about $153.5. This limit order then got filled pretty much straight away. In hour after the limit order was sent, the stop plus order with an initial stop price of almost $146 was sent out. Then, we can see a bunch of updates of the stop price while QQQ's price increased until the stop order was filled on the 5th of February at a little over $157. The next entry order was first sent about a month after this field time, which is exactly how we want it to be. So judging from these few orders, the algorithm seems to be doing what it's supposed to do. Besides analyzing the orders to ensure you fully understand your algorithm, there are a few other debugging and logging tools that QQQ provides. One of the easiest ways to double check certain things is by using self.debog. This is similar to Python's print statement. If for instance, you want to make sure that the trading stop plus price is being updated correctly, you could use self.debog to print the updated stop price to the console. For this, you would then have to restart the backtest and look at the values printed to the console at the bottom of the page. Besides self.debog, there also is self.error which prints a red message to the console. Note that if the same values are being printed over and over again, QQQQ will limit the printing rate to prevent browser flooding. Finally, you can also use self.log to log certain actions. These logs can then be accessed when the backtest is finished at the bottom of the backtest report. These logs do come with a limited capacity however, so you can't generate megabytes of logs for every backtest. Another great way to improve the understanding of your algorithm is by creating custom charts and plotting certain values. But that's something you will learn in one of the next few videos. If all that doesn't help, QQQQ also has a debugging mode which you can access by clicking on debug on the right hand side of your code. Like in other ideas, you can set break points to the left of your code to incrementally step through your code. Besides break points, you can also add variables or other expressions to a watchlist. As an example, let me quickly set a breakpoint on this if condition and let's add self.highest price to the watchlist. If we now click on backtest, the algorithm will stop every time we reach this if condition. Furthermore, you can see how the highest price changes over time. Incrementally stepping through your code and analyzing certain variables can be very useful when looking for bugs. Note that besides watching simple variables, you can also track more complex expressions such as logical expressions, lists and much more. In my opinion, the best way to get familiar with the debugging mode is by just trying it out and playing around with it. That being said, this was quite the long video, so I hope it wasn't too overwhelming and you learned something new. If you thought a few things went over your head, don't worry, you can always rewatch parts of this video or even the entire thing. In the next video, we will take a look at how indicators work, how to create your own indicators and much more. If you enjoyed this video, definitely make sure to subscribe, turn on the notification bell and smash the like button for more content like this. Thanks for watching. diff --git a/course/06-indicators-history.md b/course/06-indicators-history.md new file mode 100644 index 0000000..965ac35 --- /dev/null +++ b/course/06-indicators-history.md @@ -0,0 +1,29 @@ +# 06 · Indicators & Historical Data（指标与历史数据） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Indicators & Historical Data（指标与历史数据） +- **时长**：28 分 32 秒 ｜ **原视频**：https://youtu.be/_BHi5Y2Aow4 +- **本地视频**：[videos/06-indicators-history.mp4](videos/06-indicators-history.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **历史数据请求（History）** 与 **指标（Indicators）**、自定义图表。 + +- **History 请求**：`self.History(symbol, period_or_barcount, resolution)` 返回 **Pandas DataFrame**。 + - 例：`self.History(self.spy, 30, Resolution.Daily)`；第二参可传"天数"或"bar 数量"，不传 resolution 则用该标的默认分辨率（否则分钟）。 + - 可传 **Symbol 列表** 一次取多个标的；也可对 universe 内所有标的取历史。 + - DataFrame 列：Equity 为 close/high/low/open/volume；Forex 默认 QuoteBar（含 bid/ask 各 OHL）；crypto/期权/期货默认同 Equity，可切换为 QuoteBar。 +- **指标（Indicators）**： + - QuantConnect 内置**大量现成指标**（SMA、RSI、MACD、Bollinger 等），可组合使用。 + - **快捷方式**：`self.SMA(symbol, period, resolution)` 一行创建并自动更新（`self.` 上的指标辅助方法）。 + - **自定义指标**：用 IndicatorBase 等自建；用 `self.RegisterIndicator(symbol, indicator, resolution)` 让指标自动接收数据更新（本集示例注册了自定义指标）。 + - 指标每次更新会触发 **Updated 事件**（可挂事件处理器）。 +- **自定义图表/Plot**：可在回测图中绘制自定义指标曲线（本集演示了自定义 plot）。 +- ⚠️ 注意 **warm-up（预热）**：新注册的指标需要积累数据才"ready"，故图表可能回测开始一段时间后才出现；用 history 预热可让曲线从头开始。 +- 提示：除非有特殊需求，多数情况直接用内置指标即可，不必自己写。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to the next video of this algorithmic trading video course. In this course you learn how to develop your own trading algorithms using Python and the QuantConnect algorithmic trading platform. If you are new to this series, I highly recommend going back and starting with the first few videos of this series. In this video you will learn how to perform history requests for different securities, how to use and create indicators inside of QuantConnect, how to create custom plots, and more. As always we will start with a brief theory based part before moving on to writing a simple example trading bot, displaying the learned concepts of this video. First and foremost we will cover how to perform history requests inside of QuantConnect. So far we have only learned how to access the most recent data that your algorithm is receiving. Sometimes you might need the data from a period of time such as the past 30 days. If for instance you want to compare the current price with that of a few days ago or take an average of the price, history requests can be very useful. Before we start getting into the specifics of how to perform history requests inside of your algorithm, let me first give you a rough outline of this feature. A history request can be performed with self.history which normally takes three arguments, namely a symbol, a period or bar count and a resolution. It is also possible to perform history requests without providing a symbol but more on that later. A history request like this one will return a Pandas data frame. If you aren't familiar with Pandas, a data frame basically just is a table that structures the data in an easy to access manner. For equities the return data frame will have a column for the close, high, low, open and volume for each of the past symbols. Since the default data type for forex pairs is quote bars, the data frame for them has a few more columns including close, high, low and open data for both bid and ask bars. For crypto options and futures, the default format is the same as for equities, however it is also possible to change the format to a quote bar format. After requesting this data, you can then access it like you would access any other Pandas data frame and use the data in your algorithm. I will show some examples of how to do this when we start coding later in this video. For now, let's quickly take a look at a few variations of the history request that you can use depending on the use case. Firstly, let's look at the second parameter of the history method. This can either be a period of time or a bar account, so simply passing an integer will be interpreted as a number of bars. For instance, this request would lead to a data frame with 5 rows, one for each of the past 5 one minute bars. If you don't provide a resolution as an argument, the resolution of the data for that symbol will be chosen. If no such resolution can be found, the default resolution is minutely. Instead of passing a bar account, you can also pass a period of time like this. This would get the daily trade bar data for the past 7 days of IBM. It is also possible to pass a list of symbol objects instead of just a single symbol if you want to request the historical data for multiple securities. Besides that, it is even possible to perform history requests for all securities in your algorithms universe. This however would no longer return a Pandas data frame but an array of slice objects instead. For more details and slice objects, I recommend watching the third video of this series. Next to using history requests to access historical data, you can also use them to compare the data points between different securities. Note however that performing many history requests can be quite inefficient. If for instance, your algorithm performs a history request for the past 30 days of pricing data every trading day, it is mostly requesting the same data over and over again since only one day of data actually changes. Due to this in certain cases there are better and more efficient alternatives to history requests. One such alternative is consolidating data or using a rolling window which we will learn more about in the next video. That said, let's now move on to indicators inside of QuantConnect. QuantConnect provides you with over 100 pre-built indicators that you can use with minimal effort. Besides these, you can also create your own indicators as well as combine the pre-existing indicators in various ways. Generally speaking indicators are a great way to gain some additional insights into the markets. This is a great way to complement the decision making of your algorithm. However, usually you would not want to base your entire strategy on one or two indicators and nothing else. If you aren't looking for a super specific indicator, chances are high that QuantConnect already has a shortcut method for your indicator. For an overview of all the available pre-built indicators, check out the link in the description box below. If your indicator is on that list, using it is very easy. Let me demonstrate this using the RSI indicator. The relative strength index indicator takes four arguments. The first is the symbol for which you want to create this indicator and the second is the period for the RSI indicator. Then we have the moving average type that will be used. The most common type of moving average is just a simple, unweighted moving average. Another common moving average is an exponential one which gives more weight to recent price swings. Besides that, there are a bunch of other types of moving averages that you can see in this table. The fourth argument of the RSI indicator helper method is the resolution. Generally, you can only pass resolutions that are equal or higher than the data resolution you set when adding the data. So for example, if you want an indicator with Minitley resolution, the data for that security has to be Minitley, secondly or tick resolution. After initializing an indicator, you usually can't just use it right away. This is the case because an indicator usually needs a certain number of data points to be ready to use. For instance, a simple moving average covering the past 30 days would first hit 30 days of data to calculate the 30 day moving average. There are two main ways to deal with the time it takes for an indicator to get ready. The first and simplest way is to set a warm up time for your algorithm. This can be accomplished with a set warm up method. To check whether your algorithm is currently warming up, you can use the flag self dot is warming up. Inside of your algorithm, you can use the isready property of indicators to check if an indicator is in fact ready. So before using an indicator, you can just use a simple if statement to check this attribute. The disadvantage of this method is that it does not work for more advanced algorithms that use a dynamic universe. Furthermore, you won't be able to use the indicator until enough time has passed. If you don't want to wait, you can also use history requests to pump data into the indicators. For this, you can iterate through the requested historical data and use the update method to update your indicator with the necessary data. I will show you how to do this in the coding part of this video. Once an indicator is ready to use, you can access its current value with current dot value. So if for instance, you want to access the current value of your RSI indicator, you could do so like this. The helper shortcut indicators are very easy to use since everything is done for you automatically. If you want more customizability, there also are other options. For example, most indicators also have the option for you to manually update them and decide which data should be used for this. This can be helpful if for instance you want to use the opening price instead of the closing price for the indicator updates or if you want to use custom time periods. For this, you can't use the helper shortcut methods to create an indicator. All of the pre-built indicators have a shortcut method and a written out version that does not come with automatic updates. To register the non-shortcut versions for automatic updates, you would have to use self dot register indicator. This takes three arguments, namely the symbol for which the indicator will be used, the instance of the indicator and the resolution of the data. Let me show you this with a relative strength index indicator. Instead of using self dot RSI, you would use self dot relative strength index. This takes the same arguments as self dot RSI except for the first symbol argument. To register this for daily updates, you would need this line of code. Self dot register indicator is also what you would use when creating your own custom indicators. I will show you how to do this when we get to the coding part of this video. Besides registering indicators for automatic updates, you can also just manually update them with the update method of the indicator. Last but not least, let me quickly show you how you can combine various indicators. QuantConnect has so-called indicator extensions that you can use to chain different indicators together. The simplest way to show you how this works is by demonstrating it on the identity indicator, which is an indicator that simply tracks the price of a security. Let's say you want to create a Paris trading bot that needs to compare the price spread between two different securities. You could track this spread with a minus indicator extension. This code would track the difference between Pepsi's and Coca-Cola's price in the delta indicator. Besides the minus indicator extension, there also is plus, times, max, min, average and others. This table shows all the possible indicator extensions. Hopefully the example with the identity indicator shows how and why these extensions can be useful. But note that you can also use these on all other indicators and even on custom created ones. So the possibilities of combining various indicators really are endless. That said, I hope this theory part gave you a good introduction to indicators and history requests. Let's now move on to the coding part of this video. Before we head over to the QuantConnect platform and start putting the learn things into practice, let me quickly outline the idea behind the trading bot that we will implement in this video. Just like the last few times, let me start with a short disclaimer. The bot that we will implement is very simple and its purpose is to show you how to apply the just presented concepts. It won't be a very good trading bot that you would actually want to trade with real money. For this video we will once again just be trading SPY, but definitely feel free to try out different securities. The strategy that we will implement will first try to identify whether SPY's price currently is an uptrend or in a downtrend. It will do this with the help of a simple moving average. If SPY's price is above its simple moving average, we consider it to be an uptrend. Otherwise we consider that to be a downtrend. Besides looking at the SMA indicator, we will also look at how close SPY's current price is to its 52-week high and low. If SPY is in an uptrend and close or above its 52-week high, we want to go long. Otherwise if SPY is in a downtrend and close to its 52-week low, we want to go short. In all other cases we don't want an active position. If SPY's price is in an uptrend and near its high we will buy as much as we can. We first exit this position if SPY's price crosses below its simple moving average or its price falls out of reach of its 52-week high. If we do this the other way around for a downtrend near SPY's low. The hypothesis behind our strategy is that combining multiple indicators such as a simple moving average for identifying a trend and a 52-week range breakout might yield better results than just using a single or no indicator at all. That said, let us now head over to the QuantConnect platform and start turning this idea into actual code. If you haven't already, make sure to use the link in the description box below to create your free QuantConnect account. Next to that link you will also find a link to copy all the code from this video. I do however recommend coding along and trying out things yourself since that usually leads to a better learning experience. Inside of QuantConnect we will create a new algorithm in the lab tab. Once again we won't be using the strategy builder for now which is why we click on exit and start with the blank template algorithm. As always we will start by implementing the initialised methods. First, we set the start and end date for backtesting. You can choose whatever values you want to hear. Thereafter I'll just leave the account balance to be 100,000 US dollars as it is right now. In real trading this will be taken from your broker account. Then, we will quickly add the data for SPY. Since I've already covered all these steps in previous videos I won't go over this in any detail here. For this algorithm we will be using the daily resolution. Next up we want to initialise a simple moving average indicator. For this we will use the sma helper shortcut method. First, we specify self.spy to be the security for this indicator. Then we specify 30 days for the length of this moving average and lastly we pass daily resolution as its resolution. Note that you can't use a lower resolution here since we added the SPY data as daily data as well. With this we are already done with the initialise method. I hope this shows you how easy it is to set up indicators with the help of these shortcut methods. After implementing this algorithm I will also show you how you could implement your own custom indicators by creating a custom implementation of a simple moving average indicator. But for now let's move on to the on data method. Instead of the on data method we will want to use the sma indicator for our decision making. The problem here is that it might not be ready yet since it first has enough data for a 30 day moving average after at least 30 days. That's why we have to check if it's ready before trying to use it. If it's not ready we just return. As I already mentioned earlier in this video it is also possible to use history requests to pump data into indicators so that you don't have to wait for them to become ready. Let me show you how to do this for this sma indicator. Since we only want to do this once in the beginning of our algorithm we will do this in the initialise method. The first thing we do for this is perform a history request using self.history. The past arguments are self.spy as the symbol, 30 as the bar account and daily resolution as the resolution. As you hopefully can remember this returns upon us data frame with a column for high, low, open, close and volume information for each of the past 30 bars. We will use the close price for the sma indicator which is why we index this data frame by the close column and save it into the closing prices variable. Next up we want to iterate over this series of time and closing price information. Since the requested data frame is also indexed by symbols we first want to lock onto the spy symbol. We can do this with .lock. Now what is left to do is update the sma indicator with the time and price info of each of these rows. Doing this makes sure that the sma indicator is ready and good to use right from the beginning. In theory this means that it would be unnecessary to check whether the indicator is ready but I leave it in for now nonetheless. The next thing we want to do in the on data method is save the high and low price of the past year. We will once again use a history request for this but before we do so I want to say that there are other much more efficient ways to accomplish the same result. To save the high and low we use the history request for daily resolution and the spy symbol. This time we pass a period of 365 days instead of a bar account. If we would specify 365 as a bar account instead of a time delta the history request would get data for the past 365 trading days. Since this excludes weekends that would be about one and a half years instead of only one year. So if you are creating an indicator or using the bar account and want to cover a time span of one year make sure to use 252 bars for 52 week time frame. As the low we save the minimum value of the low price column and as the high we save the maximum value of the high price column of this 365 day time frame. The reason why such a history request is so inefficient is that we are basically requesting data for the past 365 days every day. Here actually only the first and last day of the data changes so we are basically requesting the same data over and over again. The more efficient alternative would be to use a rolling window which we will learn in the next video. Another alternative would be to use the minimum and maximum indicators. If you want to practice your skills I'd recommend trying to replace this history request using min and max indicators after watching this video. For more details on these and other indicators check out the reference table on the indicators documentation page that I linked below. In the next step we will save SPY's most recent price to the price variable. For this we will access the securities array and index it with the SPY symbol object. After that we can now check if SPY's price is within 5% of its 365 day high and whether the price is above its simple moving average. To do so we have to access the indicators value which we can do with dot current dot value. If these two conditions are fulfilled we want to establish a long position. However we only want to do so if we don't already have such a position which is why we use self.prolio.is long to check this. If we don't have a long SPY position we use self.set holdings to allocate 100% of our buying power to SPY. Now we basically do the exact same thing for going short. So first we check if SPY's price is within 5% of its low as well as in the downtrend. Thereafter we check if we don't already have a short SPY position. If all these conditions are fulfilled we use set holdings to short as much of SPY as we can. In the case that none of the outro if and l if conditions are fulfilled we want to close any open SPY positions since we aren't near a low and in a downtrend and not in an outrend and near a high. We exit any potential open positions by calling self.liquidate. If there aren't any open positions self.liquidate does nothing. Last but not least we want to plot a few things so that we can better understand and analyze what our trading algorithm is doing. So let's use self.plot to plot the high, low and simple moving average value. Self.plot takes three arguments. The first is the name of the chart that we want the plot to be on. The second is the name of the actual plot and the final one is the actual data. We want to plot all our values on the same benchmark chart. The benchmark chart is an already existing chart with a plot of SPY's price on it. So with these three lines we will have a chart with plots for SPY's price, high, low and 30 day moving average all in one place. All that's left to do now is click on build and back test to view this chart as well as the performance of this algorithm over the specified time period. Once again we won't really look at the performance of this bot since that's not the point of this video. We'll do that later in the series when we create more interesting bots with better strategies. When the back test is finished we want to select the benchmark chart in the top right. We can then quickly enlarge in this chart so that we can take a better look at it. As you can see this chart now has four plots. The red plot is SPY's price over this time frame. The green one is the simple moving average and the other two are 365 day high and low. Here we can see that SPY seems to be making quite a few new highs over this time period while the lows stayed constant after the early 2020 dip. Judging from this chart this algorithm should only have established any short positions in the beginning of 2020 since there its price clearly was below the moving average and it was definitely near its low since it was making new lows. To verify that the algorithm was actually doing this let's scroll down and look at the orders that were sent during March of 2020. Here we can see that during March this algorithm exclusively shorted SPY which is exactly what we were expecting. Since SPY's price was very volatile during this time it had multiple days where the price moved more than 5% up from its low which is why the algorithm only had some of these short positions for very few days. Before I end this video let me quickly present one last thing namely how to create custom indicators. Like I mentioned earlier I will implement a custom simple moving average for this. This means that from the functionality this algorithm won't change but instead of using the help with short cut SMA method we will create our own version of the indicator. Hopefully this example will help you learn how to create your own indicators inside of QuantConnect. Firstly we will import DQ from collections. DQ is a double sided Q which allows you to append and remove elements from both sides of the collection. You can think of this like a normal Q but besides appending to the right you can also append to the left the same goes for removing elements. We will need this for the moving average since new price data is coming in and old data is going out. For a custom indicator we will now create our own indicator class. This class has to inherit from the Python indicator class and we have to implement both an initialize and update method. For the initialize method we will just use the standard Python init class method. This constructor will take a name and a period as arguments. In this method we will just initialize 4 helper variables. The first is the name of the indicator which will set to the past name. The second helper will be a variable saving the current time which we initialize with the earliest time possible. Then we have a variable that saves the actual value of our simple moving average indicator. Last but not least we create a variable in which we save a DQ object. The max length of this double sided Q will be the period that we passed. So if we want the 30 day moving average this Q will save up to 30 elements. Now that we have implemented the init method we just have to implement the update method so that the indicator actually can be used. The update method will simply take one input argument. We want this to be a trade bar object for SPY. The first thing we do in here is append the closing price of the past bar to the left of our Q. We want this Q to have the most recent values to the left and the older ones to the right. Then we save the end time of the past bar input to the self.time variable of this indicator class. After that it is time to calculate the actual value of the simple moving average. As you probably know this can be done by simply summing up the safe values and dividing by the total number of elements. We can use the sum method to sum all the elements in our Q and then divide by the length of this Q. Note that if the Q is not yet filled with all values we aren't actually taking the average of the period that we want. In other words this indicator is first ready after receiving at least as many data points as the max length of this Q. We don't actually have to manually set the Israeli attribute of the indicator though. Instead we just return true if it's ready and folds if not. We can do this in one line by returning this Boolean condition. With that we have now successfully implemented a custom version of the SMA indicator. To actually use it we will head back to the initialized method of RQC algorithm class. For this let me quickly comment out the code for the non-custom SMA indicator. After that we will first create an instance of our custom simple moving average class. The first argument of its constructor is just the name and the second is a period for which we will once again use 30. Now we have to register this instance to be an indicator for our algorithm. We can do this with self.register indicator which takes three arguments. First we specify for which security this indicator is. Then we specify what the actual indicator is and lastly we pass the resolution of the data that this indicator will be using. In our case this of course is daily resolution. This makes sure that our indicator will receive automatic daily updates with SPYTradeBarData. Now we are already done. We don't have to adjust anything inside of the on data method. So to double check if everything is working correctly let's build and back test one more time. After the back test is finished let's once again look at the benchmark chart. As you can see here our custom plots actually did not start until about a month after the back test start. This is because we waited for our custom indicator to get ready and did not pump any data into it right from the get go. Let's also why these results are slightly different but if we remove the history warmup for the shortcut helper sma the results should be the same. If you found this custom indicator implementation to be a little overwhelming don't worry too much. QuantConnect already provides you with lots and lots of pre-built indicators that you can use and combine in many different ways. So unless you want something very specific you likely won't have to create too many custom indicators yourself. That said I hope you enjoyed this video and learned some new things. If you want to clone this code there's a link in the description box below. In the next video we will look at consolidating data, scheduling events, rolling windows and more. If you liked this video and are enjoying this series make sure to smash the like button, subscribe and turn on the notification bell for more content like this. Thanks for watching. diff --git a/course/07-consolidators-rolling-windows.md b/course/07-consolidators-rolling-windows.md new file mode 100644 index 0000000..2539299 --- /dev/null +++ b/course/07-consolidators-rolling-windows.md @@ -0,0 +1,25 @@ +# 07 · Consolidators & Rolling Windows（合并器、滚动窗口、事件调度） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Consolidators & Rolling Windows（合并器、滚动窗口、事件调度） +- **时长**：22 分 40 秒 ｜ **原视频**：https://youtu.be/rQOn9iTchIg +- **本地视频**：[videos/07-consolidators-rolling-windows.mp4](videos/07-consolidators-rolling-windows.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **Rolling Window（滚动窗口）**、**Consolidator（数据合并器）**、**事件调度**。 + +- **Rolling Window（滚动窗口）**：定长的"数据数组"，自动"加最新、去最旧"。 + - 比反复用 History 请求近期数据**高效得多**（History 每次都重复取大量相同数据）。 + - 创建：`RollingWindow[数据类型](容量)`（如存 30 个收盘价，或 10 个 TradeBar）；通常在 `Initialize()` 里创建。 + - 添加：`.Add(x)`；访问：像列表，但**反向索引**——最新数据在**索引 0**，最旧在最后。 + - 常与**指标**搭配：把指标 `Updated` 事件里的值存进 rolling window（存 `IndicatorDataPoint`）。 +- **Consolidator（合并器）**：把细粒度数据合并成自定义周期的 bar（如把分钟数据合并成 4 小时 bar）。用 `Consolidate(...)` 注册；合并出的 bar 会触发事件。 +- **事件调度（Scheduling）**：可在指定时间/日期触发方法（如每月调仓），用于定时逻辑，而不是依赖每根 bar。 +- 本集用一个"**跳空缺口（gap）**"策略示例：检测近 30 天数据里的向下跳空 → 买入；向上跳空 → 做空；并用回测图**人工核对**订单发生日的行情，验证逻辑是否正确（交叉验证是很实用的排错手段）。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Hello and welcome back to the 7th video of this algorithmic trading course. If you are new to this course, I recommend going back to the first video and starting this series from the beginning. In this video we will cover rolling windows, consolidating data and scheduling events inside of QuantConnect. As in the past few videos, we will start with a theory based portion and end the video with an example Python bot that will show you how to apply the learned concepts to an actual bot. As I had said, let's not waste any more time and start by looking at rolling windows inside of QuantConnect. The rolling window simply is an array of data that allows you to keep track of data over a specified time frame. For instance, you could create a rolling window that keeps track of the closing prices of SPY over the past 30 days. From the things we learned in the previous video, we could also accomplish this with history requests. However, the problem with history requests is that they aren't very efficient if you're requesting recent data over and over again. For example, requesting price data for the past 30 days every day is very inefficient since you are basically requesting almost the same data over and over again. A far more efficient way to accomplish this is by using a rolling window since this just updates the most recent data point and removes the last data point from the array. Let me show you how you can create and use the rolling window inside of QuantConnect. You can create a rolling window object with the rolling window constructor. As an argument, you can specify how many data points this rolling window should be able to hold. Besides price data, rolling windows can also hold all other kinds of data. For this, you just have to specify the type within the square brackets. For example, a rolling window that can hold up to 10 trade bar objects can be created like this. Usually, you would want to create a rolling window in the initialized method of your algorithm or when new data for a given security is added. To add data to a rolling window, you can use the add method. For example, adding the close price of SPY to this close window rolling window would look like this. You can access the data in a rolling window in the same way as you would access the data in a normal Python list. Just note that rolling windows use reverse list access semantics. This means the most recent item in a rolling window is in the first spot and the least recently added item is in the last place. Like I already mentioned, rolling windows allow you to store all kinds of data. This includes indicator values. This is a particularly common use case for rolling windows, which is why I quickly want to cover it before moving on to consolidating data. These are the same values emit a so called updated event every time they receive an update. We can use this to our advantage by creating an event handler and attaching it to this updated event. This can be accomplished with this line of code. This line makes sure that our custom SMA updated event handler method is called every time the SMA indicator for SPY gets updated. To make sure that the correct data is saved into a rolling window, we create a rolling window for the type indicator data point like this. Inside of our custom SMA updated method, we then simply add the updated data to our rolling window. With that, we now have a rolling window that can be used to access the five most recent values of our SPY SMA indicator. To learn more about indicators, I recommend checking out the previous video of this series. Next up, let's take a look at how consolidating data inside of QuantConnect works. So far, we've only been able to use data in one of the existing resolutions. The existing resolutions cover tick, secondly, minutely, hourly and daily data. However, in some cases, you might want to use data that covers another period. For instance, you might want to create 5 minute bars, 30 minute bars, weekly bars or bars covering some other time frame. For this, we can consolidate the data from one of the existing resolutions to a custom resolution. For instance, you could consolidate 5 1 minute bars into 1 5 minute bar. You could then use your custom 5 minute bars for indicators, rolling windows or in any other way that you normally would use the 1 minute bars. So let's now look at exactly how we can accomplish this. By default, for all asset classes except for Forex, the bars are consolidated to trade bars. Since Forex only supports quote bars, the consolidated bars obviously also are quote bars. Consolidating bars can be done with self dot consolidate, which takes 3 arguments. The first argument is the symbol for which you want to consolidate data. The next is a period specifier, here you can go with a time delta, resolution or calendar object. The final argument is an event handler function that you need to create, more than that in a few seconds. Here you can see 3 ways of consolidating SPY trade bars. The first line specifies the time by using time delta. The second line uses resolution dot hour and the third line uses calendar dot weekly to consolidate the data to weekly trade bars. After specifying an event handler function as the third argument, you should create and implement such a function. The option event handler takes one argument, which is the resulting bar. Unless you're trading Forex, this will be a trade bar object. Note that you can choose whatever name you want to for this event handler as long as it's not taking already. Just make sure to specify the correct name as the third argument in the self dot consolidate function call. You can treat these event handlers like the on data method except that they are exclusively for the consolidated bars for the symbol that you specified in the consolidate helper function. In other words, this is where the resulting bars of the consolidation are sent. Depending on your algorithm, you could use them for your trading decision making only for logs or not at all. I already mentioned earlier that besides using these custom resolution bars for basic trading decisions, you can also use them for indicators. To demonstrate this, let me show you an example of a simple moving average that uses 10 7 minute bars for its calculations. To accomplish this, we can't use the shortcut indicator helper methods. Instead, we need to use the written out versions. So for a 10 bar simple moving average, we use the simple moving average constructor and pass 10 as the argument. Then we use self dot register indicator to register this indicator for automatic updates. As the third argument, we specify a time delta of 7 minutes. These three lines of code now allow you to access a 10 7 minute bar simple moving average for SPY through the self dot SMA variable. For more details on indicators, check out the previous video of this series. Next up, let's look at how we can combine consolidating data with rolling windows, since it can sometimes be useful to have a rolling window that stores data with the custom bar resolution. This is very simple to do. All you have to do is first create a standard rolling window object for your bar type. So a quote by rolling window for Forex and a trade by rolling window for other assets. Then you use the self dot consolidate helper function to consolidate the data and finally you use window dot add inside of the consolidated event handler to add the consolidated data to your rolling window. Note that if you don't want to do anything else in your consolidated event handler, you can even use a lambda function to directly implement this functionality in one line. Last but not least, let me quickly present to you how you can remove consolidated from your algorithm. You might be asking yourself why you'd want to remove a consolidated in the first place. Some algorithms will dynamically add consolidated. This is especially common if your algorithm has a dynamic universe of assets that it trades. In that case, you would usually create a consolidated once a new security enters your tradable universe and you would want to remove that consolidated once the security exits your universe. You would want your algorithm to do this since consolidateds take up computing power and space. So removing unnecessary consolidateds can boost the efficiency of your algorithm. To do so, you can use the remove consolidated method of the subscription manager like this. As arguments, you need to specify the symbol and your consolidated. You will learn more about dynamic universes in one of the next videos, so don't worry if my explanation for why this is useful went over your head. Before we move on to the coding part of this video, let me briefly demonstrate how you can schedule your own events. This is very easy to do so it won't take very long. Most of the code for the algorithms that we created thus far was inside of the undata event handler. Besides using such predefined event handlers, QuantConnect also allows you to create and schedule your own methods. You can do so with self.scheduled.on which takes three arguments. The first and second argument need to be a date and time rule respectively. You specify on which dates and at which time your scheduled event will take place. The final argument is the method that you want to schedule. Since these are quite simple, I'll just give you one or two examples. For all the rules, make sure to check out the documentation page. You could for example, schedule a method 10 minutes after SPY is open every day that SPY trades as follows. With this, the everyday aftermarket open method will automatically be called at the specified time. Scheduling a method on every Monday and Friday at 12 o'clock would look like this. If you only want your method to be called once at a specific date and time, the schedule.on functionality also allows you to do so. But like I said, for all the details on date and time rules, check out the documentation page that I linked below. With that, we are now done with the theory part of this video. Next up, let me quickly present to you what strategy we will be implementing for the second part of this video. In the past few videos, we've only coded bots that focus on longer timeframes such as multiple days or even months. In this video, we will switch things up a little and create a simple intraday trading bot. This bot will once again only trade in one security. For this video, I will use the SPY ETF, but like always, you can try this with whatever security you want to. The idea behind this bot is that every morning the bot will compare SPY's last closing price to the current market open price. In case of a gap up, the bot will open a short position and in the case of a gap down, the bot will open a long position. In other words, we are betting on a reversal of the overnight gap. For the market closes, the bot will always liquidate any open positions to take profits or cut losses. This means there should never be any overnight risk. As in the past videos, my goal is to demonstrate the just learned concept with this simple bot, so please don't expect it to perform particularly well. This is not yet an algorithm that you would actually want to trade real money with. That said, let's now head over to QuantConnect and start coding. As always, we create a new algorithm in the lab tab. Here, we will not use the strategy builder and instead click on exit to create a new blank template algorithm. The first thing we do here is implement the initialize method. The first few lines of code here are the same as in the previous videos, which is why I won't cover them in any detail. In short, we just set dates and a balance for back testing purposes. Thereafter, we add data for SPY and save SPY symbol to the self.symbol variable. As to resolution, we will choose immediately data. Next up, we will create a rolling window that stores trade bar objects. We will use this rolling window to compare the current price of SPY to its last closing price. For this, we don't need a long rolling window, which is why we initialize it to only hold up to two elements. After that, we create a consolidator for the Minidly SPY data. The objective of this consolidator is to consolidate SPY's Minidly bars to daily bars. As the third argument, Consolidate takes a consolidation event handler that we have to create. This is called every time a new daily bar is consolidated out of the Minidly bars. I will call this method custom bar handler and define it below. Note that it is important not to put brackets behind the specified method when using self.consolidate. Since we will use our rolling window to access the previous day's closing price, we want it to store daily trade bar objects. To add such a trade bar object, we can use the add method. That's also exactly what we will do inside of our custom bar handler. The bar parameter of this method is the consolidated daily bar which we want to add to our rolling window. Besides that, we don't really need to do anything else inside of this custom bar handler method. Next up, let's move on to the on data method and start writing some code here. The first thing that we want to do inside of the on data method is check whether our rolling window is ready. Rolling window is ready once it is filled up with data. Just like with indicators, we can use the isready attribute to check this. If it's not yet ready, we simply return. This return statement should only be reached in the beginning of our algorithm since thereafter the rolling window should always be filled with data. In the next step, we want to check the current time. We do this since we want our algorithm to trade right after the market open and not at any other time. To check this, we check if the hour equals 9 and the minute equals 31. We use 931 instead of 930 because bars are passed at their end time. This means the 1 minute 930 bar is first accessible at 931. If this is not the current time, we once again just return since we don't want this algorithm to make any opening trades at any other time. So in the remainder of the on data method, we can now safely assume that the market has just opened since we otherwise would not have surpassed this return statement. What we have left to do now is implement the trade logic. For this, we first check if there was a gap up in SPR's price. We do this by comparing the open price of the 931 minute bar with the close price of the daily bar of the previous day. This daily bar can be accessed through our rolling window. Since the most recent data is in the beginning of rolling windows, we index our rolling window with index 0. Since we are only interested in those days where there was an actual gap up, we multiply the previous day's closing price by 1.01. This will ensure that this condition is only fulfilled if SPR's price gaped up by 1% or more. Note that this 1% number is an absolutely arbitrary choice. When developing your own algorithms, you should try to put some research into finding meaningful values for parameters such as this one. You will learn more about research and optimization procedures later in this series. For now, just note that generally speaking, you should try to keep the number of arbitrary parameters in your algorithm relatively low. Often, it is better to use dynamic models instead of fixed parameters for values such as this one. You could, for instance, couple this parameter to the 30-day volatility of SPY so that your algorithm dynamically adjusts its behavior depending on the market conditions. But like I said, more in this in later videos. If this condition is fulfilled, we know that SPY's price gaped up by at least 1% since yesterday's close. In that case, we want to establish a short SPY position which we can achieve by passing minus 1 to set holdings. Now all that's left to do is implement the same logic for a gap down and going long. Here, we simply flip the gray through or equal sign the other way and multiply yesterday's closing price by 0.99 instead of 1.01. This condition will equal true if SPY's price gaped down by at least 1%. If that's the case, we establish a long position. For this, we once again used the self.set holdings helper method. So far, we've only implemented the logic for opening a long or short SPY position. All that's missing now is the part that closes this position before the market closes. For the exit logic, we will schedule a custom method that is called 15 minutes before the market closes. We can accomplish this with self.scheduled.on inside of the initialised method. As the day drool, we specify that our method should be called every day that SPY trades. As the time rule, we choose 15 minutes before the market closes. Note that this will account for days where the markets close early. On Christmas Eve for instance, our scheduled method will be called at 1245 instead of 1545 since the markets close early on that day. We will name the method that we want to schedule exit positions. Inside of this exit positions method, we will close any open positions. For this, we can use self.liquidate and optionally specify self.symbol as an argument. Without passing a symbol, the positions in all securities will be liquidated, but since this algorithm does not trade any other securities, this won't really make a difference. Now this exit positions method is called every day that SPY trades 15 minutes before its close. If we don't have any open positions, self.liquidate won't have an effect on your algorithm. With that, we are already done implementing this trading bot. I hope this example bot shows you how powerful QuantConnect and the Lean Trading Engine are. With only about 30 lines of code, we've just coded a fully functional trading bot. Granted, this is a very simple bot and you should not actually use it for trading, but in my opinion 30 lines of code is still very impressive for such a trading bot. Let's now build and back to this bot to check if it's actually doing what it's supposed to do. Once again, I won't look at the performance of this bot in any detail since the purpose of this bot is to show you different aspects of the API and not to perform particularly well. But a quick glance at the equity chart already tells you that this does not seem to be a very well performing strategy. Furthermore, since this bot trades quite actively and never holds a position for more than a day, it seems to generate relatively high fees. What I actually want to look at however is the overview of the orders that this bot sent. We can do this inside of the orders tab at the bottom of the page. From the first few orders, we can already see that this bot does in fact open its trades at 9.31am and close them 15 minutes before the market close, which is what we wanted to do. Let's now look at a daily chart of SPY over the same period as some of these trades and check if they're actually where your gap ups or gap downs on these days. For this, I will just use daily candlestick SPY chart from TradingView, but feel free to use whatever charting platform you want to. Note that if the prices inside of QuantConnect deviate from those shown on a chart, this likely is due to the default data normalization mode which is split and dividend adjusted. For more details on data normalization modes, check out the fourth video of this series. The first order that we sent was sent on the 6th of February 2018. Since this was a buy order, we would expect there to have been a gap down of at least 1% from the previous days close on this day. So let's check if that was the case on our SPY chart. As we can see, there was in fact quite a big gap down on this day, but later during the day, SPY's price did rise quite a bit, which likely led to a good profit on this particular day. The next trade was in the 9th of February 2018, and this was a short trade which means the SPY should have gapped up. If we look at the chart of SPY on this day and compare its price to the closing price of the day before, we can see that there actually was a gap up. So judging from these two days, everything seems to be working properly. Sometimes cross-referencing what your alginess is doing to other data can be a great way to find out if it's doing everything as planned, or if not, this can be a great way to potentially find out why it's not functioning properly. That said, if now hopefully learned how to consolidate data, what rolling windows are, and how to use them, and how to schedule events inside of QuantConnect. In the next video, we will look at how to create a dynamic universe of securities so that you aren't only limited to algorithms that only trade one or a few of the same securities over and over again. Besides that, we will learn how to use fundamental data for your decision making and much more. If you enjoyed this video and want to support this series, make sure to smash the like button, subscribe, and turn on the notification bell for more content like this. Thanks for watching. diff --git a/course/08-dynamic-universes.md b/course/08-dynamic-universes.md new file mode 100644 index 0000000..57805a9 --- /dev/null +++ b/course/08-dynamic-universes.md @@ -0,0 +1,23 @@ +# 08 · Dynamic Universes（动态证券池与基本面选股） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Dynamic Universes（动态证券池与基本面选股） +- **时长**：25 分 30 秒 ｜ **原视频**：https://youtu.be/HfGHS-HeDK8 +- **本地视频**：[videos/08-dynamic-universes.mp4](videos/08-dynamic-universes.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **动态 Universe（动态证券池）** 与 **基本面筛选**，重点解析 **幸存者偏差**。 + +- **Universe（universe 选择）**：让算法动态地、按市场条件决定"今天交易哪些标的"，例如：只交易接近 52 周新高的股票、高波动股票、盈利稳健的公司、或财报公布前后的股票等。 +- **幸存者偏差（Survivorship Bias）**：示例——"10 年前市值 >100 亿的股票买入持有"的回测，若只统计**今天还活着**的股票，结果会虚高（破产/被并购的公司被忽略）。需用"10 年前**当时存在**的全部股票"来筛选。 +- **选择偏差（Selection Bias）**：即使只交易单一标的也有此偏差——如"只做苹果"要警惕：表现好可能只是**苹果本身强**，而非策略有效。要问："如果苹果不再复制过去 10 年，策略还成立吗？" +- **QuantConnect 的 Universe 机制**能方便地规避这些偏差，并支持用**基本面数据（Fundamentals）**做筛选。 +- 本集实现一个**基本面选股**示例（按市值等因子选若干只股票，定期调仓）。 +- ⚠️ 作者提醒：示例里"取 10 只""只看流动性前 200"等**参数是随意的**，别当成好策略；这类练习的价值在于动手改进（换因子、加技术指标/风控）。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome to the 8th video of this algorithmic trading course. If you're new to this series, I highly recommend going back and starting with the first video of this series. So far, we've only created trading bots that trade in one specific security such as SPY. Besides that, we've learned how to add data for multiple different securities so that you could create algorithms that trade a handful of stocks or other securities. However, we've not yet learned how to create a dynamic universe of securities that changes based on the given market conditions. If for instance, you only want to trade stocks that are near 52-week high, stocks with high volatility, investing companies with solid earnings, only trade stocks around the earnings announcements or something else, you will need to set up a universe for this. That's why we'll cover how to set up a universe in this video. As always, we will first look at this on a conceptual level and thereafter we will end the video with an example bot demonstrating the learned concepts. Before we get into specifics however, let me first outline one of the main problems we look to address with Universe selection, namely survivorship bias. If you aren't using a dynamic universe of tradable instruments, you might be falling prey to this bias without even noticing it. In my opinion, survivorship bias is best understood with a brief example. Let's say you want to create a trading bot that buys and holds big US companies. To test if this strategy works, you look at all US stocks and you'll get their market cap 10 years ago. If the market cap was over 10 billion US dollars, you add the stock to your universe. To test if investing in large cap US stocks works, you now just have to test how this bot would have performed if it would have bought these stocks 10 years ago, right? Actually no. Testing this strategy like this would lead to a much better result than you would actually have achieved. The problem here is that in your selection process, you only consider those US stocks that still exist. So all the non-surviving stocks that for instance when bankrupt or merged aren't even considered. To fix this, you would have to look at all the US stocks that existed 10 years ago and then filter them by market cap. For some of you this might seem like an obvious mistake, but often things are a lot more subtle and harder to detect. The main takeaway here is to always be careful when you're confronted with the selection process. Make sure to think about if there are any invisible filters. Ask yourself whether new unknown data is introduced that at the time of the actual selection was not yet actually known. Even if you're creating a bot that only trades in one security, you can full-preat a selection bias since the choice of this security likely was not random. If for instance you decide to create a bot that only trades Apple, you have to keep in mind that Apple is one of the best performing stocks over the past decades. So here you have to ask yourself whether the performance generated from this bot can be attributed to its strategy or if it's just a result of choosing Apple as the security that is trades. Ask yourself would this strategy still work if Apple does not replicate its performance from the past 10 years? Luckily, QuantConnect's universe selection makes it quite easy to account for survivorship and selection bias, but especially if you're using custom data or a fixed universe you should definitely keep this in mind. Next up let's start looking at how you can implement a dynamic universe selection bot in QuantConnect. The idea here is that your algorithm receives a list of about 8000 daily stocks and then applies certain filters to this list and adds those that pass these filters to your universe. For this you would first start with the course filter that you can use to filter stocks by basic properties such as price, volume and the availability of data. You can then apply a fine filter to filter by other values such as fundamental company information, indicator values and more. Let's now look at the specific Python statements used to create a universe and the associated filters. To add a universe you can use the add universe method. This first takes a reference to your course filter method as the first argument and if you also want a fine filter you can optionally specify a reference to such a method as the second argument. In the next step you would want to implement your course and optionally the fine filter that is specified here. The main properties that you can filter by in the course filter are a stocks price, its dollar volume and whether or not it has fundamental data. Some tickers might have no or only limited fundamental data so if you're creating a fundamental investing bot you would want to filter out those stocks without fundamental data. Here is an example of a course filter that finds the 500 most liquid stocks that have a stock price above $10. The first line of this code sorts the security object inside of the course collection by dollar volume from highest to lowest. The next line saves the symbol for each of these stocks into a new list if the stock's price is above $10. Last but not least we will return the first 500 elements of this list. If you also implemented a fine filter this list would then be passed on to your fine filter where you could further filter out unwanted stocks. So let's look at an example of how we could implement a fine filter. As parameters a fine filter function takes a list of symbol objects that were passed on from the course filter. We can now filter this fine list for instance by looking at different fundamental stats of the associated companies. QuantConnect allows you to access hundreds of fundamental ratios, stats and other information that can be useful when analyzing stocks. For a reference table of all the supported fundamentals accessible through the QuantConnect API check out the link documentation page in the description box below. If for instance you want to find the 10 stocks out of the stocks in the fine list that have the lowest price to earnings ratio you could access this value with valuation ratios dot p e ratio. You can then sort the stocks by this ratio and return a list containing the symbols of the first 10 stocks. So after going through the just shown course and fine filters this algorithm would have a universe of 10 of the most liquid US stocks with a price over $10 and the lowest p e ratio. You can now iterate through this universe and potentially establish a position depending on the remaining implementation of your bot. You can access the securities currently in your universe with the self dot active securities. In comparison to the self dot securities array this collection only keeps track of the securities currently in your universe. If the securities removed from your active universe it will no longer be in this active securities array but it will still be accessible in the normal securities collection. This is important since you still might want to access this security for tracking purposes such as checking the fees spent or volume traded. You might remember that when adding individual securities you can change settings such as the data resolution, data mode, set leverage and more. Such things can of course also be customized when using a universe model. For this the firstly a global universe settings that you can change in the initialized method with self dot universe settings. For instance you can set the resolution for the added securities by setting self dot universe settings dot resolution to the desired resolution. Without specifying anything the default resolution is minutely. Other attributes such as leverage can be set in a similar way. Changing individual security properties is also possible. For this you need to set a security initializer which you should specify in the initialized method. Set security initializer takes a reference to a custom method as an argument. The method that you specify here will be called every time a new security is initialized which for example happens when it is added to your universe. So if you want the data normalization mode for your securities to be raw you could accomplish it like this. Inside of your custom security initializer you just set the data mode of the security that will be passed as an argument for the initialization. When a security enters or leaves the active universe a so called security changed event occurs. Like for all important events there is an event handler that is automatically called when this happens. This event handler is called onsecurities changed and its parameter is a security changed object which you can use to look at the added and removed securities. More specifically you can access the added securities with changes dot added securities and the removed ones with changes dot removed securities. One common use case of this onsecurities changed event handler is to initiate the exit process for the stocks that have left your active universe. Usually you only want to position in those securities that are currently in your universe. To close any potential positions in the removed securities you can iterate through this collection and liquid at any positions in these tickers. Before we move on to coding an example bot to demonstrate these features let's look at one last thing in the universe creation shortcuts. If you just want to use a simple dynamic universe without very specific requirements there are some shortcut helpers that QuantConnect provides. If for example you just want your universe to consist of the 50 US stocks with the highest dollar volume you can create such a universe in one line with self dot universe dot dollar volume dot top 50. For the bottom 50 you just have to replace top with bottom. Alternatively you can also filter by percentile. Here you can either pass a specific percentile or a range. This last line for instance would create a universe for stocks between the 70th and 80th dollar volume percentile. That said let me now present to you the main strategy idea behind the bot that we will implement in the remainder of this video. We base our strategy on a widely accepted and researched effect found in the markets, namely the so called size effect. Historically speaking small cap stocks have outperformed the large cap counterparts. This means that there is the potential to earn a premium without taking unpropotionally more risk when trading small cap stocks. There are many different explanations as to why this effect exists but there are two that are the most widely accepted. The first is that small cap stocks come with much worse liquidity than large cap stocks which makes them less suitable for institutional trading firms. The second reason is that smaller firms have much more space to grow. A smaller company could easily double triple or even 10 eggs its earnings within a few years. This is much much harder to do for a 100 billion dollar company. For a complete breakdown as well as links to various research papers about this size effect check out the Quantpedia entry that are linked below. We will try to implement a simple strategy that hopefully explores this size effect at least to some extent. However, we don't want our bot to invest in really bad micro cap junk stocks since these usually aren't very good investments. Instead, we want to focus on the 200 most liquid US stocks and then sort these by the market capitalization. We then invest in the 10 stocks with the lowest market cap out of these 200 stocks. For this we allocate an equal amount of our portfolio to each of these 10 securities. We rebalance the portfolio once a month. This means that every month we once again look at the most liquid US stocks sort them by market cap and if the lowest 10 have changed since last month we just our portfolio to account for this change. Note that this means that we aren't specifically investing in small cap stocks. Instead, we are investing in those stocks with the lowest market cap out of the 200 most liquid stocks ranked by the dollar trading volume. These likely still are very large cap stocks but they have lower market caps compared to other stocks on this top 200 most liquid stocks list. That said, let's now start writing some code. For this we will head over to the lab tab of the QuantConnect platform. If you haven't created your free account yet, let's link in the description box below. I highly recommend coding along. For now we won't be using the strategy builder. Instead, we just start with a blank template algorithm consisting only of the initialised and on-date methods. As always, we start by setting the start date and date and starting balance for our back test. Here, I just go with a two year time frame and a starting balance of $100,000 but feel free to try out different values. After that, I will initialise a couple of helper variables that we will need later for this algorithm. The first is Rebalance time which will help us implement the monthly rebalancing cycle. We initialise this to the earliest possible date since we want our bot to start trading right away. Then we create an empty set and save it to the active stocks variable. This will be used to keep track of the stocks in our universe. More on this later though. In the next step, we use the Add Universe method to create a dynamic universe. Here, we pass a reference to a course and fine filter which we still have to create. Before we implement these two filter methods, let me change the resolution of the securities that will be added to the universe. The default resolution is minutely but since this bot's average holding time is over a month, we won't need that lower resolution. Instead, we will use hourly resolution. This will also make back testing a lot faster. Last but not least, we create a variable called putForYourTarget which will be a list of putForYourTarget objects. With that, we are now done with the initialise method and can start implementing the course filter. As already mentioned earlier, in a course filter you can filter by price, volume and whether or not a stock has fundamental data. We can check this with dot has fundamental data which returns true if there was fundamental data available for the stock in the previous month. Note that this is not necessarily a guarantee that there will be fundamental data available now, so double checking if data is available for the specific fundamentals before using them can still be worthwhile. We will use this method to filter out the 200 most liquid stocks by dollar volume that are priced above $10. But before we implement the filtering, we first want to check if it's time to rebalance our portfolio. If it's not time to rebalance, it's pointless to even look for changes in the universe. To check if we should rebalance, we check if the current time is before the time saved in the rebalance time variable. If this is fulfilled, we return since we don't yet want to rebalance. If it's not fulfilled, we continue and set the rebalance time variable to one month from now so that we only rebalance once a month. After that, we can implement the filtering. After this, we first sort the securities in the course collection by their dollar volume. We order them in reverse since we're interested in those with the highest dollar volume. Thereafter, we return a list of symbols for the first 200 securities that are priced above $10 and that have fundamental data available. Note that the choice of 200 securities is quite arbitrary. You could easily bump this up to 300, 500 or even more. Just note that if you filter out too few securities, your algorithm might take a lot longer to backtest since it has to perform a lot of work. Generally, you should try to keep your active universe comparatively small. Creating a universe with thousands of securities will lead to a lot of performance problems and might require you to upgrade your server nodes RAM. Now the collection of securities that were returned from the course filter gets passed on to the fine filter which we will implement next. In here, we simply want to sort the equities by their market cap from lowest to highest. Out of these 200 most liquid stocks, we will then add the 10 stocks with the lowest market cap to our universe. Note that in the return statement, we filter out those stocks with the market cap of zero. We do this because some securities might not have an available value for their market cap. If that's the case, zero is the default replacement value but since that's obviously not the actual market cap, we don't want to include these stocks in our universe. Before we move on to the on data method, we will now implement the onsecurities changed event handler. This method is automatically called every time the universe of our algorithm gets updated. So in our case, this should happen once a month. The first thing we want to do in the onsecurities changed event handler is liquid at any open positions in those symbols that have been removed from our universe. To accomplish this, we can iterate through the changes.removesecurities collection. For each of these elements, we then call the liquidate function. Besides that, we want to remove them from the active stocks variable that keeps track of our universe. After that, we now iterate over the added securities which we can access through changes.addedsecurities. We then add each of these symbols to the active stocks variable. We should not try to send actual trade orders for the just added securities right in the onsecurities changed method. The reason for this is that the data for these securities first has to be added to the algorithm. This usually takes at least one iteration after the securities were added. You might be asking yourself why we are using our own self.active stocks variable and not just the self.active securities array since that should keep track of the active securities as well, right? The problem with the active securities array is that the removed security objects are not immediately removed from this collection. After liquidation, they are added to a pending removal list where they usually stay for at least one iteration. If we then open a position in that symbol before it is fully removed, it won't be removed from the active securities array. Therefore, we would somehow have to keep track whether this iteration has passed or not before opening a new position. Taking track of this would require more effort than just creating our own active stocks set. Next up, there's one last thing we do in the onsecurities changed method, namely create a list of portfolio target objects. For this algorithm, we'll go with an equal allocation to each of the equities in our universe. So we create a portfolio target object with a weight of 1 divided by the length of the active stock set which should be 10% for each symbol. Now we can move on to the on data method where we will use this list to set the portfolio holdings. The first thing we do in the on data method is check whether or not the universe has changed. We can do this by checking whether or not the portfolio target list is empty. If it's empty, nothing has changed and there's no reason to rebalance so we just return. Otherwise we want to rebalance our portfolio. However, we only want to do so if the algo has data for all the 10 securities in our active universe. This we can check by iterating through our active stock set and checking if each symbol is in the data parameter of this method. If this is not the case for one of the securities, we return since the data has not yet been added correctly. So in the remainder of this method, we now know that the data of all stocks has been added correctly and it's time to rebalance our portfolio. Now what that's left to do is pass the portfolio targets list to the set holdings method which will take care of the rebalancing for us. As a last thing, we now have to set the portfolio targets list to an empty list so that we don't try to rebalance the portfolio before the universe changes again. With that, we are now done and have successfully implemented our strategy. So let's now build and back to this bot. Depending on the specified time frame, this might take a few minutes. So let me just fast forward to when the back test is completed. As always, when the back test is finished up, you can see a performance report like this one. The first thing that we usually look at is the equity chart. Here we can see that the strategy seemed to perform quite well during 2019 but it had a big dip in the early 2020 crash. If we look at the benchmark chart and compare it to the equity chart, we can see that the two definite seem to be correlated. This should not be very surprising since this bot invested in 10 of the most liquid US stocks which likely are also holdings of the S&P 500 index. If we would increase the number of stocks, this correlation likely would increase even more. But be aware that over other time frames, the performance might deviate significantly. Note that I would not recommend actually funding such a bot with real money. One problem is that this bot still has quite a few totally arbitrary parameters. There's no good reason why we are investing in 10 instead of 12, 20, 30 or even 50 equities. Another example of an arbitrary parameter is that our course filter only looks at the 200 most liquid stocks. This strategy does not seem to be a very good choice if your goal is to explore the size factor. If your sum idea is about how to potentially improve this strategy or if you just want to play around with the code and take a closer look at this performance overview, you can clone the code from this video by using the link in the description box below. One possible extension is that you could look at other fundamentals instead of only looking at the market cap. Furthermore, you could also incorporate some technical indicators, risk management techniques and other possible improvements. Practicing by trying to code things like this yourself is probably the best way to get better at algorithmic trading and coding. I also have another video in which I implemented a more advanced fundamental investing bot. If you want to check that video out, there's a link in the description box below. Just know that there are some concepts in that video that have not yet been covered in this video series. One such example is the algorithm framework which is something we will cover in detail later in this series. So if you don't understand everything in that video yet, don't worry. In the next video, we will probably look at how to add external data such as your own custom data or data scraped from somewhere to QuantConnect. That said, I hope you enjoyed this video and learned a bunch of new things. If you are enjoying this series, make sure to smash the like button, subscribe and turn on the notification bell for more content like this. Thanks for watching. diff --git a/course/09-twitter-trading-bot.md b/course/09-twitter-trading-bot.md new file mode 100644 index 0000000..cf95819 --- /dev/null +++ b/course/09-twitter-trading-bot.md @@ -0,0 +1,24 @@ +# 09 · Twitter Trading Bot（自定义数据 · 推文情绪） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Twitter Trading Bot（自定义数据 · 推文情绪） +- **时长**：26 分 52 秒 ｜ **原视频**：https://youtu.be/X7XwkHsE-4Y +- **本地视频**：[videos/09-twitter-trading-bot.mp4](videos/09-twitter-trading-bot.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **自定义数据（Custom Data）**，示例：用**马斯克推文情绪**交易特斯拉。 + +- **为什么强大**：可引入社交媒体、天气、SEC 文件、新闻等任意外部数据。 +- **添加**：`self.AddData(数据类型, ticker, resolution)`（在 `Initialize()` 中；与 `AddEquity` 类似，只是更通用）。 +- **自定义数据类**：新建类继承 `PythonData`，**重写**两个方法： + - `GetSource()`：返回 `SubscriptionDataSource`（数据源 URL / 本地文件 / 等；传输方式有 RemoteFile、LocalFile 等）。 + - `Reader()`：解析你的数据格式（自定义，需手动定义结构）。 +- **本集示例 bot**：读取马斯克推文 → 用 `NLTK` 的 **VADER 情感强度分析器**打分（polarity）→ 正/负情绪驱动做多/做空特斯拉；用 `self.Plot` 把情绪分数可视化。 +- ⚠️ 局限：示例用的是**历史推文 CSV**；换时间框/实盘需要实时数据源并调整 Reader；且通用情感分析器对**推文语料**未优化，还有很大改进空间。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Hello and welcome to the ninth video of this algorithmic trading video course. As always I recommend starting with the first video if you're new to this course since you otherwise might not fully understand some of the covered topics. In this video we will cover how you can add custom data to QuantConnect and use it for the decision making of your algorithms. The ability to add custom data is very powerful since it opens so many doors. You could for instance add social media data, weather data, SEC filings, news data or whatever data you can find. Even though QuantConnect already offers a huge variety of different data, adding your own data gives you even more possibilities. To demonstrate this feature I will create a trading bot that trades Tesla stock based on simple sentiment analysis of Elon Musk's tweets. But before we get into coding this trading bot let me start by presenting the whole spectrum of options that QuantConnect provides when it comes to adding custom data. To add custom data you can use the add data helper method. You usually would do this in the initialised method of your algorithm since you only want to do this once in the beginning. Add data takes three arguments. The first one is the type of data that you want to add. Usually you would have to create a custom class for this and use this class as the type. You will see a concrete example of this in a few minutes. The second argument is a string which represents the ticker for that data. And the third argument is a resolution which specifies how often the custom data should be pulled. If for instance you specify a minutely resolution the algorithm will check for the new data once a minute. You might be able to see that the add data method is very similar to the add equity method. In fact they both fundamentally do the same thing. The add data method is just a little more general. Like I just said the first argument of the add data method is the type of the data that you want to add. For this you should create your own custom class. This class should extend the Python data class and it should override the reader and get source methods of that class. The get source method is used to get the source of that data and the reader method is used to read your custom data. Since the data is custom and not standardized you need to specify the structure of that data in the reader method. Let me present how to implement these methods with an example of custom weather data. For this let me start with the get source method which is actually very simple to implement. All you have to do is return a subscription data source object. To create such an object you need to pass two arguments to its constructor. The first is the actual source of your data and the second one is a type of subscription transport medium. As the source you usually would specify a URL such as a dropbox URL where you are uploaded your custom data as a CSV file. For the subscription transport medium you have four options. Probably the most common one is remote file which means that the algorithm expects one file with all the data in your remote location such as your dropbox. Alternatively you could also use local file if the data is accessible locally. There is a nice remote and local file that also are options REST and Streaming. Rest is used if each line of the custom data should be pulled individually and comes from a REST call. This might be the case if you have a subscription to a data vendor that provides you with custom data in real time. For our example we have custom weather data in a dropbox. The URL to this data is specified as the source and for the subscription transport medium would choose remote file. In some cases you might want the get source method to return different data depending on whether you are back testing your algorithm or actually live trading it. If for example you have a real time data subscription you can't use that for back testing and might instead want to use a CSV file for the historical data when you are back testing. For this you can use the isLive flag which is true when you are live trading and falls if you are just back testing. Now the data specified in the CSV file will get sent to the reader method line by line. The goal of the reader method is now to standardize this data so that your algorithm understands what it looks like and it can actually use it. Before I show you how you can implement this reader method let me first quickly show you what the data actually looks like. As you can see this data has 11 entries divided into 4 columns. Column A specifies the data for each row. The first four digits specify the year, the next to the month and the last to specify the day. Column B represents the max temperature in degrees tells us for that day. Column B represents the daily average and column C is the minimum temperature. I know this data does not look very interesting and I actually don't even know what location this weather data was taken from. However in my opinion this example is a great and simple way to understand how to use custom data in QuantConnect. Theoretically you could use weather data such as this one for trading in the agricultural space since this is heavily weather dependent. Here it might also be useful to take precipitation, the weather forecast and other variables into account. That said you now hopefully have a good understanding of how this custom weather data is structured. Remember that this data now gets fed to the reader method line by line. The first thing we want to do in the reader method is check whether a meaningful line of data was passed to it. We do this by checking if the line of data is not empty and that the first value is a digit. This ensures that we don't try to use the data from the header index and don't continue if there is no data left. So if this is not fulfilled we just return an empty object. Otherwise, we split the line at its commas and save the generated list to the data variable. Remember that we can split at commas since the file is a CSV file which stands for comma separated values. Then we create an object of the weather class. Since this class extends the Python data class there are three important properties that we should set, namely the symbol, value and time property. For the symbol property we should always use config.symbol. For the time we need to correctly format the values from the time column in the CSV file. We can do so with this date time helper method in which you can specify the format of the date and time. Here it is very important to understand that data in QuantConnect is passed on at its end time and usually the data and time in a custom file is the start time. That's why it's essential to add a time delta to the start time. If you don't do this your algorithm might access this data before it actually would be known. This is called lookahead bias and it can lead to very unrealistic results. That's why we add a time delta of 20 hours to the time of the weather data. If we would not do this the algorithm could use the weather data for a given day at the beginning of the day before it actually would have been known. Next up we set the value attribute of the weather object. For this we will take the daily average temperature which was saved into the second column of the weather data file. Note that the value attribute always is a decimal object. Besides that we also create two more properties for the max and min temperature that were saved in the first and third column. Last but not least we will turn this weather object. Since custom data can sometimes lead to some unexpected errors it usually is a good idea to add some exception handling to the reader method. I will show you a simple example of this when we get to the coding part of this video. You can now access this custom weather data like you would access other price data. Inside of the on data method you can index the data slice object with the symbol of the custom data and then access values either with the get property helper or the dot value attribute. Note that if this still seems a little overwhelming to you don't worry too much. We will go over another example step by step when we implement the Twitter trading bot in a few minutes. Besides adding custom data that you want to access on the go line by line, QuantConnect also allows you to add static data. If for instance you just want to add a file containing a list of tradable symbols for your universe or an AI training model file you can accomplish this with the self dot download. As an argument you can for example once again just pass a drop box download URL. If the file is a CSV file you can then use pandas to convert this file to pandas data frame. Note that if you want to use custom data from a data vendor such as Quantal, Tingo or Intriniou QuantConnect does offer direct helper methods for this. Here you only have to provide your API token and you are good to go. For the specifics on how to do this I recommend checking out the documentation page. That said let me now move on to the Elon Musk Twitter Tesla trading bot that we will code in this video. As always I will first give a rough outline as to how this strategy works on a conceptual level before starting to actually code it. The idea behind this bot is that we will use the tweets from Elon Musk to perform sentiment analysis and based on the results of this analysis we will then make a trade decision for Tesla stock. For this we will only consider those tweets that mention Tesla in some way. For these tweets we then use a sentiment analyzer to find out whether this tweet has a positive or negative connotation to it. If it is positive we establish a long Tesla position for that day. If it is negative we short Tesla for that day. If it is mostly neutral or slightly positive or slightly negative we don't do anything. For the sentiment analysis we will be using Python's natural language toolkit NLTK package. Note that if you aren't familiar with this module don't worry we will just be using a pre-trained sentiment and density analyzer but if you are familiar with it you could try using your own custom models for better analysis. Since QuantConnect does not directly offer tweets as data we will have to find a dataset containing Elon Musk's tweets and then import this custom dataset so that we can use it for back testing this strategy. Luckily I found a dataset containing Elon Musk's tweet from 2012 until 2017 for free on Kaggle.com. This means we can use this dataset to back test the strategy at least from 2012 until 2017. However, before we actually import this dataset I firstly want to perform some pre-processing on it since we otherwise might run into some unwanted problems. For the pre-processing I will import this dataset into a Jupyter notebook where we will use Pandas to import it into a data frame. Note that if you want to code along when I code this trading bot you won't have to do these steps since you can access the pre-processed file through the link in the description box below. The unprocessed dataset contains 5 columns including values such as retweets, the row number and the count. But we are actually only interested in two of these columns namely the tweet itself and its date and time. So firstly we want to remove the three unwanted columns. Next up I want to change the order since we want the most recent tweets to be at the bottom and the least recent ones at the top. Besides that we will remove all URLs from the tweets since they can lead to some unexpected formatting problems and that don't really add any value to our sentiment analysis. After that I exported the pre-processed data to a new csv file and uploaded it to Dropbox. Through this Dropbox URL you can now import the tweet data into QuantConnect where we will use it to test our strategy. If you have not yet created your free QuantConnect account you can do so using the link in the description box below. In QuantConnect we will head over to the lab tab where we will create a new algorithm. As always so far we won't be using the strategy builder tool. Instead we start with the blank template containing only the initialised and on data methods. The first thing we do in the initialised method is set the back testing time frame to the time covered by the mask tweet dataset. So we set the start time to November 2012 and the end time to the beginning of 2017. The starting balance I will just leave at $100,000. Before we move on let me input sentiment and density analyser from NLTK sentiment. We will use this sentiment analyser later to analyse the sentiment of the tweets but more on that later. Thereafter we want to add the data that this algorithm will use. Firstly we add tesla data which we can do with the usual add equity helper method. Since the tweets can be sent pretty much at any time of the day and we have specific time stems for them we will set the resolution for the tesla data to minutely data. Then we want to add our custom data. We can do so with the add data method which takes three arguments. The first is the type that this data will have. I will pass mask tweet for this which is a class that we will implement in a few seconds. As the second argument you need to specify a ticker symbol that you can use to access this data with. For this you can choose whatever name you want to. Just try to pick one that isn't already occupied by some stock or other asset. I will just go with mask tweets. Last but not least you need to specify resolution. This resolution will determine how often your algorithm follows your custom data source for potential new data. Here I will also go with minutely data. Since the tweet data actually has timestamps with up to second accuracy we could also go with second data but this would slow back testing down and in my opinion isn't that important for now. Note that just like for equities and other assets we can use dot symbol to save the symbol for this data to a variable. We will do this so that we can later use this symbol to request the data. For this you could also use the specified ticker but that can come with unwanted ambiguity problems. For details on this go watch the third video of this series. Next up let's start implementing the mask tweet custom data class. Like I mentioned in the theory part of this video this class has to extend the python data class and it has to implement the get source and reader methods. Let's start with the get source method but before we do so make sure to note that this class no longer is part of the QC algorithm class so you won't be able to access the usual helper methods such as self dot debug in here. Before we move on let me save an instance of the NLTK sentiment intensity analyzer into a class variable named self dot sia. In the reader method we will use this for the sentiment analysis. That said let's implement the get source method now. Here we simply want to set a drop box URL as the data source. Note that here it is very important to have dl equals 1 at the end of the URL. dl stands for download and we want the algorithm to download this data. If you would want to view this dsv file online you could do so by setting dl equal to 0 and opening the URL in your browser. However for the algorithm you will always have to set dl to 1 while using drop box URLs. Then all we have to do is return a subscription data source object. For this object we will just use the saved URL as the source and specify this to be a remote file. If you would want to live trade such a bot you would have to connect this bot with some kind of twitter API that sends you real time tweet updates. In that case you would use the REST or streaming subscription transport medium option. To differentiate between back testing and live trading mode you could then use the is live parameter of this method. But since we will just use this for back testing we won't have to worry about this for now. Next up let's move on to implementing the reader method which handles bringing the data into the right format so that your algorithm can actually use it in a meaningful way. Remember the reader method receives data from your remote file on a line by line basis. So the first thing we need to check in here is whether the line actually contains relevant data. If for instance we receive an empty line or the first index row we won't do anything which is why we return none. Otherwise we split the line by commas since the data comes from a comma separated file. The data variable now is a list containing two elements. The first element is the date and time of the tweet and the second element is the tweets content itself. Next we create an instance of the mask tweet class and save it to a new variable named tweet. The next step is to set the symbol time and value of this mask tweet object. So the symbol we can just use config.symbol. Config is one of the parameters of the reader method. As for the time we have to specify the format of the time column in our custom data. The format is year, month and day separated by a dash and hour, minute and second separated by a colon. Note that we add a time delta of one minute to this time since this will be the end time of this data point. Since tweets should be available pretty much right away we could also use a time delta of one second but since we are using minutely data anyways this likely won't make a big difference. Be aware that when using price data or other data covering a certain period to always correctly adjust the time to the end time of the data. Otherwise the results can be very unrealistic. After setting the time we convert the tweet which is saved into the second place of the data list to lowercase and save it to the content variable. Now we want to perform some simple sentiment analysis on the content of each tweet. However, we are only interested in those tweets that reference Tesla. Since the tweet was converted to lowercase we can check whether it contains the ticker symbol of Tesla or the written out version. If it does we use our sentiment intensity analyzer to calculate a polarity score for the tweet. This polarity score will return a dictionary with scores for negative, neutral, positive and compounded sentiment. This tweet says that Tesla just had its first profitable quarter thanks to awesome customers and hard work by a super dedicated team. If we would use the polarity scores method on this tweet it will return the following dictionary. As you can see according to the sentiment intensity analyzer this is the most positive tweet which does make sense. Note that the compounded sentiment is not just an average of the others. An example of a tweet with a negative sentiment would be this one. Not all good news Virginia DMV commissioner just denies Tesla a dealer license. This is interpreted to be a neutral to negative statement. But note that often the sentiment analysis isn't as accurate as in these two cases. What we want to do is use the sentiment to either establish a long or short Tesla position. We will only use the compounded score for this though. That's what we save it as the value of the tweet data object. In the case that Tesla was not referenced in the tweet we just assign it a score of zero which would be equivalent to a completely neutral tweet. Next up I also want the content of the tweet to be accessible in the data bar. To accomplish this we can simply create a property for the tweet object with whatever name you want to and save the content to it. We will use tweet as the name for this property. Earlier in the video I mentioned that sometimes that can occur some unexpected problems inside of the reader method since we are dealing with custom data. That's why it makes sense to add some form of exception handling here. So let's put a try statement around this code and if a value error is thrown we simply return none. After that all that's left to do is return the tweet object and then we are done with the reader method. Now we're almost done. We just have to quickly implement the on data method and the exit logic. The first thing we do in on data is check whether the data slice object contains any data for the musk tweet. We check this with a symbol object of that data. If that's the case we can save the score of the current tweet with data.value. To access the content of the tweet we can use .tweet which is the name that we gave this property in the reader method of the musk tweet class. For this algorithm we will establish a long position if the compounded sentiment score is above 0.5 and we establish a short position if the score is below negative 0.5. We can check this with a simple if-l-f condition and use set holdings to go long or short. For demonstration purposes I will also log the score and content of the tweets that meet one of these thresholds. Now all that's left to do is add a method that liquidates any open position before the market closes since we don't want to hold Tesla overnight based on one tweet. For this we can schedule an exit positions method in the initialized method. We schedule it to be cooled 15 minutes before the close on every day that Tesla trades. In the exit positions method we then simply have to cool self.liquidate to close all open positions. With that we are now done implementing this Twitter sentiment analysis trading bot. So let's now build and back test it. As you can see this strategy seemed to perform quite ok over this time frame. However, here it is important to note that it isn't a very long time frame and Tesla had a huge bull run over the past few years. If we scroll down to the stats overview we can see that this bot made 165 trades over this time frame which is a solid amount. If we go to the logs tab we can see an overview of the scores and content of those tweets that the algorithm used to actually base trades on. Some of them make sense such as the tweet on the 4th of December 2012 that says that Musk is happy to report that Tesla was narrowly cash flow positive and expects continued improvement. This had a polarity score of almost 0.9 and led to a long position. However, some of the other scores and tweets aren't as clear. If you want to look at these results in detail you can use the link in the description box below to clone this algorithm. Note that to test this strategy over another time frame or live traded you would need to find a data source that gives you access to Elon Musk tweets over the desired time frame or in real time. This data will likely be structured slightly differently than the data from this video. That means you would probably have to adjust the reader method to some extent to use that data. Also note that the sentiment analysis here is very simple and the pre-trained sentiment intensity analyzer likely isn't optimized for Twitter text data since this data usually has a different structure than other text data. So there certainly are many areas to improve upon. That said, I hope that this video gave you a good overview of how to import custom data into QuantConnect. I also hope that it shows you why this is a very powerful feature since it gives you so many more options as to what you can do. For some other examples of algorithms that use custom data check out the algos reference in the documentation page. With that being said, if you're enjoying this series make sure to smash the like button, subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/10-backtesting-performance.md b/course/10-backtesting-performance.md new file mode 100644 index 0000000..c160f7c --- /dev/null +++ b/course/10-backtesting-performance.md @@ -0,0 +1,24 @@ +# 10 · Backtesting & Performance Analysis（回测与绩效评估） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Backtesting & Performance Analysis（回测与绩效评估） +- **时长**：25 分 45 秒 ｜ **原视频**：https://youtu.be/oFvYbfDOJ5c +- **本地视频**：[videos/10-backtesting-performance.mp4](videos/10-backtesting-performance.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **策略评估** 与 **参数优化**——非常关键的一集。 + +- ❌ **只看总收益是错的**：相同收益下，波动/回撤小的策略更好。要看 **回撤、波动率、稳定性**。 +- ✅ **风险调整收益**：最常用 **夏普比率（Sharpe Ratio）** = (策略收益 − 无风险利率) / 超额收益标准差。**长期 >1 算不错**。还有 Sortino、Calmar、Treynor 等。 +- **其他统计**：胜率、平均盈利/亏损——**要结合起来看**（例：胜率 10% 但平均盈利是平均亏损的 20 倍，仍可能很有 edge）。 +- **始终用基准对照**（相关市场指数），别只比"是否盈利"。 +- **优化（Optimization）**：在 QuantConnect 里可批量回测不同参数（如 SMA 天数），按 Sharpe 等排序找相对最优。 + - ⚠️ **极易过拟合**：参数要**有逻辑意义**；优化本身会引入**前视偏差**（用这批数据挑最优参数）。务必留出样本外检验。 +- 建议关注指标：Sharpe、回撤、年化收益、交易次数等。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome to the 10th video lesson of my algorithmic trading course using Python and the QuantConnect platform. In this video lesson, we will discuss how you can analyze and evaluate the results of your trading algorithms. After that, I will also show you an optimization procedure that you can use inside of QuantConnect. As always, I will present this with the help of an example bot after going over it on a conceptual level. First and foremost, let me start by presenting how you should not measure the performance of your trading strategy. Even though this is what most beginners do, you should not just look at the total returns that your strategy generated over a certain time period and only rely on this number alone for your evaluation. This is not a good measure for performance, since not all returns are created equally. Even if you are presented with two strategies that achieve the exact same return over a given time period, one might be a lot more desirable than the other. The reason for this is that one strategy might achieve these returns with much lower volatility and a smaller drawdown than the other strategy. Besides returns, you should therefore always consider other metrics, such as the drawdown, volatility and stability of this strategy. Since it can be quite hard to look at all these metrics individually, you can risk adjust your returns to account for all these factors with one simple number. The most common way to risk adjust your returns is by using the so-called Sharpe Ratio. The Sharpe Ratio is calculated by subtracting the risk-free rate from your strategy's returns and then dividing this figure by the standard deviation of the excess returns of your strategy. The higher this number is, the better the risk adjusted returns of your strategy are. A strategy with the Sharpe Ratio above one over a significantly long enough time frame is considered relatively good. Since the Sharpe Ratio accounts for the volatility of your returns, you can use it to better compare the performance of different strategies. Next to the Sharpe Ratio, there are many other metrics that you can use to risk adjust your returns. Other examples include the Tray Neuro, the Calima and the Sotino Ratio. In addition to risk adjusting your returns, it also makes sense to look at other stats about your strategy so that you can better understand its performance characteristics. For instance, it can be very useful to look at things such as the win rate as well as the average win and loss of your strategy. Note that here it is important to always consider these stats in conjunction with each other, by themselves that don't really tell you much. A strategy with a win rate of 10% for example might sound like an undesirable strategy. However, if this strategy has an average win 20 times the size of its average loss, this strategy seems to have an edge and should be very profitable given enough time. Besides that, you should not use profitability as a benchmark to compare the performance too. Just because a strategy managed to generate a profit does not mean that it actually is a good strategy. Instead, you should always rely on some relevant market index as a benchmark to compare the performance of your strategy too. This benchmark should have a similar level of risk as your strategy and be relevant in the same market sector. For example, if you mainly trading large-cab US stocks, you could use the S&P 500 index as a benchmark. So even if your strategy does not manage to generate a net profit over a given time period, it might still outperform the S&P 500 index and thus be a good investment compared to similar alternatives. In other words, it is important to consider the current market conditions when analysing a strategy's performance. Is the traded sector currently going through a bear market or is it having a historic bull run? These are some very relevant questions to ask yourself. Note that if your strategy always trades at a 2x leverage, it might be taking on more risk than a simple buy and hold on leverage S&P position. So besides the market sector, make sure to account for the risk. Will to try to consider the directional exposure of your strategy and its correlation to other markets and strategies. Usually you don't want your strategy to have a too high correlation with another strategy that you're trading. Furthermore, if you don't want more exposure to a given sector, you should make sure that your strategy is not highly correlated to that sector since you otherwise just basically are increasing your exposure to that sector. In general, it is important to consider personal preferences. Just because a given strategy is good for me does not mean that it is a good fit for you. Everyone has their own preferences that are determined by their risk tolerance, account size, other investments, goals and more. In algorithmic trading, you usually back test your strategies over a long enough time frame to get the best possible idea about the potential of a strategy. However, when back testing a strategy and evaluating the performance from a back test, there are some things to look out for. Before we move on to coding and evaluating the back test of an example bot, let me quickly outline what to look out for when back testing. First of all, it is essential to understand that the performance shown on a back test is not nearly as significant as actual real life trading results. This is the case because back testing makes a bunch of assumptions about the executions, fees, data, timing and availability of your trades. These assumptions might very well be unrealistic and thus lead to unrealistic results. So always make sure to take back testing results with a grain of salt. If a strategy achieves an incredible result in a back test, always be wary and try to look deeper into it. Besides unrealistic executions, costs, margin or trading models, you might also have full prey to some biases that could lead to unrealistic behavior. Some examples of such biases include overfitting, look ahead bias and survivorship bias. We've already covered these in previous videos, but let me quickly outdone a few ways to avoid overfitting since it's such a common problem. Overfitting is the act of fitting your strategy too closely to the back test data. At some point, your algorithm won't actually make meaningful trading decisions, but instead, just make certain decisions because it knows the back test data too well already. The most common mistake, especially for newcomers, is to constantly adjust their algorithms and then back test to check whether or not the performance now is better than before. This is the easiest way to end with a useless overfit algorithm. In general, when developing an algo, you should not try to back test it too often unless you're purely doing so for back testing and development reasons. One of the most common ways to try to avoid overfitting is by dividing the available data into two subsets. So if your data covering 10 years, you might want to divide it into two five year intervals. You can then use one of the subsets when developing your algorithm and the second subset for the testing. So first when you're completely done with coding and developing the algo, you should use the test data set. You can then compare the performance over the development and test time frame. If it is significantly worse over the test time frame, the algorithm might be overfit to the development data sets. Another good practice is to keep the number of parameters in your algorithm relatively low. Parameter's are fixed numbers that you can set that affect the outcome. For example, if you have an algorithm that always liquidates its holdings after 10% loss, this 10% number would be a parameter. The reason why too many parameters can be harmful is that often these parameters are set quite arbitrarily. In the just mentioned example, for instance, the likelihood is no logical reason as to why the strategy does not exit after 8% or 12% or some other percentage instead of 10%. Instead of using many fixed parameters, it can often be useful to use more dynamic models. For cutting losses, you could for example use the recent volatility of the traded asset to determine the point after which to close the position. More dynamic models also allow your algorithm to adapt to changing market conditions and thus can lead to more versatile strategies. If you are using fixed parameters however, try to always come up with a reason for why the parameters value is what it is. This reason should never just be because the back test performance with this value is better than the performance with some other value. Another good way to find out whether your strategy's back test performance is realistic or not is by paper trading it. If you let it trade live but without real money, the resulting performance can no longer be affected by the previously mentioned biases. So if the paper trading performance is similar to the back test performance, the back test results might actually be trustworthy. Note however that here it is important to let it paper trade for long enough. Performance from a few weeks isn't very significant and should not be used to make important decisions. Even if everything looks good, you should still always start slowly when going live with an algorithm. Always start with small amounts and then scale up from there. You don't want to deal with technical issues associated with deployment if the algorithm has access to half of your net worth. After deployment you should still actively monitor your algorithm and double check if it's doing what it's supposed to do. This however does not mean that you should constantly interfere with it just because you don't agree with the given trade. If you do that, you can't actually evaluate its performance since you are not giving it the chance to trade its program strategy. That said, let's now move on to the coding part of this video. For this, we will first call a simple example bot and then look at the back test performance report for this strategy together. But first, let me quickly outline the idea behind this trading spot strategy. The strategy that we will be implementing is a very simple long-term investing strategy. What we will do is look at SPY's 30-day moving average. If SPY's price is above this average, we say that there is an uptrend and in that case we want to allocate 80% of our capital to SPY and 20% to B and D which is a bond market ETF. If SPY's price is in a downtrend, so below its 30-day moving average, we want to flip this allocation around and put 80% to B and D and 20% to SPY. We do this since bonds and equities at least once used to have a somewhat negative correlation and bonds can be seen as a safer and less risk investment than equities. That's why we switch to them in case of a downtrend. But speaking of the long term, bonds have had historically lower returns than equities which is why we still want most of our portfolio in SPY if it isn't an uptrend. If SPY's price does not cross its 30-day simple moving average for a month and thus we don't adjust our positions, we want to rebalance our portfolio back to an 80-20 allocation. With that, I hope you now understand the strategy that we will be implementing. So let's now head over to QuantConnect and start coding. If you haven't already, you can use the link in the description box to create your free QuantConnect account. I highly recommend coding along for the best learning experience, but you can also use another link in the description box to clone the code from this video. So let's now head over to the lab tab and click on Create New Algorithm. As always, we will then start off with a template algo consisting of the initialise and on data methods. For the backtesting time frame, I will go with 2018 until 2021. As usual, I will leave the starting cash balance at $100,000. Next up, I will add the two securities that we will want to trade. We can do this with add equity and save the symbol into class variables. Since this is a long-term investing bot, I will go with daily resolution here. Thereafter, we will add the simple moving average for the SPY data. For this, we can use the self.sma helper method. Then all we have to specify are the symbol, number of bars, and resolution. We save this into the self.sma variable, which we can later use to access the moving average indicator values. Last but not least, we will add two helper variables. The first is self.rebalance time, which we will use to keep track of rebalancing and self.op trend, which we will use to save the state of the market. We initialise this to true. Next up, let's move on to the on-data method in which we will implement the actual trade logic. The first thing we do in the on-data method is check whether our sma indicator is ready and that there is data available for SPY and B and D. For this, we can use the ezready attribute of the indicator. If this is not the case, we simply return since we aren't ready yet. Now, we move on to the actual trading decision making. Here, we firstly check whether SPY's current price is greater or equal to the 30-day sma value. If this is the case, we know that SPY is in an up trend. However, since we do not want to adjust our holdings if we already are invested and does not get time to rebalance, we check if it's time to rebalance or we previously have not been in an up trend. If one of these conditions is fulfilled, we will adjust our holdings with two set holding statements. First, we use set holdings to allocate 80% of our capital to SPY and then we do the same for B and D and 20%. Furthermore, we set the up trend variable to true and adjust the rebalance time to plus 30 days. If the out-to-if statement is not fulfilled, we know that SPY's price is below its 30-day moving average, which means that it is in a down trend. Here, we need to perform the same check as before, so we check whether it's time to rebalance or if we have previously been in an up trend. In either case, we adjust the holdings by allocating 20% to SPY and 80% to B and D. Once again, we use set holdings for this. Next up, we set the up trend variable to false and set the rebalance time to 30 days from now. This ensures that we rebalance these holdings only if the trend changes or 30 days pass by without a change in trend. Now, we already done implementing the decision making of this trading bot. Nonetheless, there is one small thing I want to add. I will use safe.plot to add the SPY simple moving average to the benchmark chart that plots SPY's price. This will help us see during which periods we had which asset allocation. In general, it can be very useful to create custom plots such as this one when analysing your algorithm since this can significantly help with the understanding. That said, let's now build and back test this bot. When the back test is finished up, you will see a performance report like this one. At the top, there's a bar with some of the most commonly used performance metrics. The leftmost value is the probabilistic sharp ratio. This represents the theoretical probability that the sharp ratio of your strategy is above one over the specified time frame. This takes into account the distribution of daily returns of your strategy. Besides that, you can see other measures such as the unrealized and realized returns, fees, the value of all your holdings and more. Below that, you can look at one of the most interesting charts, namely the equity chart. This shows the performance of your strategy over the chosen back test time frame. Here we can see that this strategy actually performed quite well. At the bottom of the equity chart, we also see a bar chart showing the distribution of the daily returns. On the right hand side, we can select to also display the benchmark chart which is a chart of SPY. Since we plotted the SMA onto this chart, we can also see it here. Even though this chart is good for a quick comparison, it could also quickly adjust the algorithm to buy and hold SPY over the same time frame, so let me quickly do that so that we can better compare the performance. All we have to do for this is add a set holdings and return statement to the beginning of the on data method. After that, we can then build and back test this strategy again. This would then generate the same report for a SPY buy and hold strategy. As you can see, the SPY buy and hold strategy achieved a return of almost 50% over this time frame, which actually is about 10% more than the other strategy. However, if we look at the probabilistic sharp ratio or the equity chart, we actually see that this higher return was achieved with a much higher volatility. This means if we use the sharp ratio to risk adjust the returns, the buy and hold strategy actually underperformed our custom dynamic allocation strategy. That said, let me now go back to the report of our strategy. Here we can scroll down to see a bunch of other performance metrics and stats that can be used to learn more about its performance. In the top right corner, we can see that the sharp ratio for this strategy over this time frame is slightly above 1. Other interesting stats, such as the number of trades, winning percentage, average win and loss can also be found here. Since covering all these metrics in detail will be quite boring and too much for this video, I highly recommend googling some of these and reading how they are calculated and how they can be used. Really understanding these stats can help you develop a much better understanding of your strategy's performance. By skimming this back test report, we can conclude that this strategy does seem to reduce the overall volatility compared to simple S&P buy and hold strategy. However, this reduction in volatility does come at a cost which is a reduction in net returns. Depending on your appetite for risk, the added stability might very well be worth this trade off. But here it is important to keep in mind that this reduction in returns might be even more significant over longer time frames. However, note that this is just based on a quick look at this one back test. It would be a very good idea to look at some other time frames to see the broader picture. Especially looking at the performance during other market conditions, such as the 2008 crisis can be a very good idea and a way to stress test and better understand the performance of your bot. By looking at the performance during the early 2020 crash and comparing it to the benchmark chart, we can already see that this strategy does seem to better handle down trending markets, which isn't very surprising. However, instead of analyzing this strategy in even more detail, I will now briefly show you QuantConnect's parameter optimization feature. If you want to analyze this strategy in more detail, you can clone the code and test it yourself using the link in the description box below. For the parameter optimization, we first have to decide which parameter or parameters we want to optimize. Since this strategy actually does not have that many parameters, we will use the number of bars of the simple moving average for the optimization. Currently, this parameter is set to 30 bars, but since this is not based on anything, we could try to look at the effect of changing this parameter on the performance. To do so, we first have to define it as a parameter. This can be accomplished with self dot get parameter and then the name that you want to give this parameter. We will call this sma underscore length and save it into the length variable. Then we will expand sidebar and scroll down to the algorithm parameter section. Here, we click on add new parameter and add a parameter with the name sma underscore length. After that, we click on this parameter and enter 30 as the initial value for this parameter. Now the algorithm uses the get parameter helper to get the value of this parameter from here. This is useful since when you have already deployed an algorithm to live trading, you would usually not want to edit the source code anymore. This feature then still allows you to change parameters without touching the actual code. However, in some cases, the get parameter method might not be able to get a value. To account for this case, we need to check whether or not it returned the value and if it did not, we used 30 as default value. This ensures that the length variable always is well defined. So now we only have to specify this length variable to be the second argument of the sma helper method. To optimize this parameter, we can now click on the optimize button next to the back test button. This will bring you to a menu where you can select what you want to optimize for. Here, you can maximize or minimize things such as the sharp ratio and your return or drawdown. We will go with maximizing the sharp ratio. Next up, we have to select which parameters we want to optimize. Since we only defined one parameter, we will use the sma underscore length parameter here. As the minimum value, I will go with 15 and as the maximum I will specify 90. Under additional settings, I will set the step size to 2. This means that this optimization will back test our strategy for every second number between 15 and 90 for the sma underscore length parameter. Optionally, we can also add some constraints for the sharp ratio, drawdown or annual return, but we won't do that here. In the final step, you now have to select an optimization node as well as the max number of nodes that you want to use for this optimization. Note that this optimization feature is not free of charge. Depending on the number of back test and complexity, this can cost anything from a few cents to hundreds of dollars. But an optimization like this one should not cost more than a couple of cents. I will just deploy the max number of nodes for this example since this will take the least amount of time. Deploying with the smaller number of nodes, let's confirm the optimization is working before scaling up the cluster. When the optimization has finished, you will see a screen like this one. At the top, you can see the equity chart of all the back tests. As you can see, the performance of these back tests does vary quite significantly, which means that the value for the time span of the spysma is important. Below this top chart, you can see three heat maps for the sharp ratio, compile return and drawdown. Beneath these charts, there's a table in which you can see all the exact values for the different parameter values. Here we can sort the back tests by different metrics. For example, if we sort them by sharp ratio, we see that the back test with the best sharp ratio is that with a 39 day simple moving average. This back test has a sharp ratio of over 1.2. By back testing so many times, the optimization does include some look ahead bias however. Besides the columns that you can see here, you can also add more metrics by clicking on the columns button to the right on the table. Here you can add other stats such as the win rate, average size of win or loss, number of trades, and more. For more details on a specific back test, you can click on the back test name. This will bring you to a standard in-depth performance report of that back test. Note that even though this optimization feature can be very useful, you should still be very careful when using it since it can easily lead to overfitting. Like I mentioned earlier, the parameter value should still make logical sense. That said, I hope that this video gave you a good overview of what to look out for when analyzing and evaluating the performance of a trading strategy. I hope you now understand that you should not only look at the returns and profitability of a strategy. Instead, always use some benchmark for reference and consider how the returns were achieved. If you have any questions or comments, don't hesitate to let me know in the comments section below. Otherwise, make sure to smash the like button, subscribe and turn on the notification bell for more videos like this one. Thanks for watching. diff --git a/course/11-forex-trading.md b/course/11-forex-trading.md new file mode 100644 index 0000000..cf078f6 --- /dev/null +++ b/course/11-forex-trading.md @@ -0,0 +1,22 @@ +# 11 · Forex Trading（外汇 · 均值回归） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Forex Trading（外汇 · 均值回归） +- **时长**：13 分 27 秒 ｜ **原视频**：https://youtu.be/CoIu-SXDRvY +- **本地视频**：[videos/11-forex-trading.mp4](videos/11-forex-trading.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **Forex（外汇）与 CFD** 算法，并演示 **自定义图表**。 + +- **数据添加**：`self.AddForex("EURUSD", resolution, market)`；市场可选 **Oanda**（71 组，GMT 时区）或 **FXCM**（39 组，东部时区）；可设杠杆（默认无）。 +- **CFD**：`self.AddCFD(...)`，数据来自 Oanda（51 个合约）；⚠️ 实盘 CFD 目前仅支持**非美国居民**。 +- **数据形态差异**：Forex **只有 QuoteBar，没有 TradeBar**（QuoteBar 由买卖报价合并；TradeBar 由真实成交合并）。 +- **示例 bot**：EURUSD 的**均值回归**策略——价偏离 20 日均线够多就反向入场，回归后平仓；用 **Bollinger Bands** 辅助判断；用 `self.Plot` 绘制价格、布林带、以及买/卖/平仓的**散点标记**（entry/exit）。 +- 工具：`self.Plot` 自定义图表可直观核对入场点；回测结果该示例约 +27%（同期 EURUSD 基本横盘）。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Hello and welcome back to another video of my Python and QuantConnect algorithmic trading course. If you're new to the series make sure to start from the beginning since you otherwise might not be able to follow along properly. In this video you will learn how to create trading algorithms for Forex and CFD trading. In the previous videos all the example trading bots exclusively traded equities. However, all the lessons learned in those videos are just as important for Forex trading. As that, I will show you how to create custom plots when we get to implementing the example Forex trading bot for this video. The main difference between trading equities and Forex inside of QuantConnect is that instead of adding equity data in the initialized method, you add data for the currency pairs or CFDs that you want to trade. This can be done with a simple add Forex statement. When adding currency pairs using the add Forex method, you would usually specify three arguments. The first is the currency pair itself. This could for example be EuroUSD. For a full list of the over 50 supported currency pairs, check out the link documentation page in the description box below. The second argument is the resolution, which can be anything from tick resolution to daily resolution. The third argument is where you specify the market that you want the data for that currency to come from. Currently, QuantConnect supports two markets, namely Oanda and FXCM. Oanda supports 71 pairs and FXCM supports 39 pairs. Note that the Oanda data uses the GMT time zone while FXCM uses Eastern Standard Time. It is also possible to set leverage as another parameter of the add Forex method. By default, there is no leverage though. One of the main differences between Forex and equity data is that Forex data only supports quote bars and no trade bars. The difference is that quote bars are built by consolidating bits and asks from exchanges while trade bars are built from actual trades. For more details on quote bars and trade bars, check out the fourth video of this series. Before we move on to implementing an actual example Forexbot, let me quickly show you how you can add data for CFDs. Since this is very similar to Forex, I will not create an actual trading bot for CFDs in this video. The CFD data also comes from Oanda, which currently supports 51 different contracts. However, note that live trading these contracts is only supported for non-US residents at the moment. To add CFD data, you can simply use the add CFD method, which works pretty much the same as the add Forex method. For a full list of the supported CFDs, check out the documentation page linked below. That said, let's now move on to implementing a Forex trading bot. As always, I will start with a brief theoretical breakdown of the bot that we will code. For this video, we will implement a simple mean reversion strategy on the EuroUSD currency pair. The idea is that we expect the price to stay somewhere around its 20 day mean most of the time. In other words, we do not expect it to deviate from its 20 day moving average too much. However, if it does deviate by more than a certain amount, we take the opposite position and try to profit from the price moving back to its mean. To measure the degree of deviation, we use the standard deviation. If the price moves above its 20 day moving average by more than two standard deviations, we take a short position and if it falls below its 20 day moving average by more than two standard deviations, we open a long position. In either case, we close the position once the price crosses its 20 day moving average. Luckily, we won't have to implement this logic manually since there already is ready to use Indicator that does exactly what I just described. This indicator is known as the Bollinger Bands Indicator. To better understand our algorithm, we will plot the Bollinger Bands Indicator as well as mark the points where the algorithm opened and closed positions on a custom chart. With that said, let us now head over to QuantConnect and start writing the actual code for this trading bot. If you haven't already, make sure to create your free QuantConnect account using the link in the description box below. For the best learning experience, I highly recommend coding along, but I will also post a link in the description box that allows you to clone the finished code from this video. Outside of QuantConnect, we will create a new algorithm in the lab tab. Then we will start off by implementing the initialise method. For this, we first set the backtest time frame to the last 6 years. Thereafter, we leave the starting cash balance to be $100,000 for now. Now, it's already time to add the data for the Forex pair that we want to trade. I will be using the EuroUSD pair for this example, but feel free to try this with other currency pairs as well. To add the data, we use Add Forex. As the first argument, we specify EuroUSD. As the second one, we pass Daily Resolution and as the third, we will use the FXCM market. Note that you could also use another resolution or the Oanda market instead. We then save the symbol of this pair into the self.pair class variable. After adding the data, we want to create a Bollinger band indicator for this currency pair. We can do this with a helper method self.bb which checks 3 arguments. First, we pass our pair so that the indicator knows for which data it is. Then we specify the period that it covers and last but not least, we define the number of standard deviations that the upper and lower bands will deviate from the moving average. The most commonly used parameter values here are 20 days and 2 standard deviations, which is what we will use here as well. Now, let us move on to implementing the onData method. The first thing we do in the onData method is check whether the Bollinger band indicator is ready to use already. If it's not, we just return since we can't use it yet. Then we save the current price of our currency pair into a local variable named price. We can access the price by indexing the current data slice. Next up, we want to decide between two different cases. In the case that we aren't already invested, we check whether the current price is below the lower Bollinger band. If it is, we want to buy as much of EuroUSD as we can. We can accomplish this with set holdings. If instead the price is above the upper 2 standard deviation Bollinger band, we want to establish a short position. To do so, we once again use set holdings. The other case would be that we are already invested. Here, we need to implement the exit logic of the strategy. For this, we first check if we currently have a long position open. If that's the case, we check if the price is crossed above the middle moving average line of the Bollinger band indicator. If it has, we just liquidate the position. If we do not have a long position, we must have a short position. In that case, we need to check if the price has dropped below the moving average. If it has, we once again use self.liquidate to close any open positions. Now we are already finished with implementing the trade logic of this bot. However, before we move on to backtesting this bot, let me show you how you can add and create some custom charts. I will create a chart that plus the price of the currency pair and all three lines of the Bollinger band indicator. Furthermore, I will add symbols to this chart to indicate where this algorithm bought and sold. This can be very helpful when trying to understand when the algorithm made which trades and why. For this, we first head back to the initialised method and create a chart with the name Trade Plot. We save this chart into the variable stock plot. Then we will add three series to this chart. By default, new series will be line plots. However, to mark the entrant exit points of the trades, we want a sketch plot instead. We use Add Series to add a series to the stock plot. To create the series, we use the Series Constructure and pass 5 arguments. The first is the name of the series. The second is the type. The third argument determines whether the label is for the x or y axis. The fourth argument is the color and the final one specifies the shape. For all the details of different types, colors and shapes of these charts, check out the documentation page that I linked below. In total, we add three series for Bias, Shorts and Liquidations. After that, we add the chart to the algorithm. Next up, we have to specify which data we want to plot in the on data method. First, we can add plots for the Bollinger bands and price of the currency pair. We do this with self.plot which I've used many times in the previous videos. Then we need to add the data for the entry and exit markings. For this, we also use self.plot but since we specified that to be three scatter plots in the initialised method, we can use them here. So when buying, we mark the chart with the entry symbol at the price of the entry. We do the same for shorting and when we liquidate any positions. With that, we can now click on Build and Back just to look at the results. As you can see, the results of this time frame seem to look relatively good. The strategy managed to achieve a return of almost 27% which is not bad, especially considering that the EuroUSD currency pair traded pretty much sideways over the same period. On the right hand side, we can select a viewer custom trade plot on which we plotted the price of our pair, the Bollinger bands and the entry and exit markings. With all these plots visible at once, the chart is quite full. However, it is also possible to deselect some of the plots to make it cleaner. If we for example, only select the price, entry and exit plots, you can clearly see when this bot board, shorted and close positions. The green triangle is marked by, the red one is marked short and the blue dot mark exits. If you want to analyze this backed test report in more detail, feel free to clone this algorithm using the link in the description box below. In the next video, I will go over how you can trade options within QuantConnect. That said, I hope you enjoyed this video. If you are a fan of this series, make sure to smash the like button, subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/12-options-trading.md b/course/12-options-trading.md new file mode 100644 index 0000000..48fbca0 --- /dev/null +++ b/course/12-options-trading.md @@ -0,0 +1,25 @@ +# 12 · Options Trading（期权入门） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Options Trading（期权入门） +- **时长**：15 分 28 秒 ｜ **原视频**：https://youtu.be/d3j2zYXKSUs +- **本地视频**：[videos/12-options-trading.mp4](videos/12-options-trading.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **期权（Options）算法**。 + +- **背景**：QuantConnect 支持约 4000 个标的的美股期权（CBOE）；**仅支持分钟分辨率**。期权按标的/类型/行权价/到期日划分，一个标的可有上百个合约 → 数据量大，**效率尤其重要**。 +- **方式一：AddOption + SetFilter**（本集采用） + - `self.AddOption(underlying)` 添加期权数据；`SetFilter(...)` 限定**行权价区间**与**到期区间**。 + - 在 `OnData` 里遍历 **OptionChain（期权链）**，按行权价、到期、**希腊字母（greeks）**、持仓量（open interest）、隐含波动率、价格等筛选合约。 +- **方式二：Option Chain Provider**（更高效） + - 只取某标的某日**可用合约列表**，筛选出目标合约后用 `self.AddOptionContract(...)` 单独添加——订阅数据少、回测更快。回测过慢时推荐改用此法。 +- **示例 bot**：一个简单的期权策略（标的创新高时入场买期权等）；`self.Plot` 可视化。 +- ⚠️ 作者强调：示例参数（到期时间、突破周期等）**很随意**、交易样本少（43 笔），**不可直接实盘**；且表现高度依赖标的本身。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Hello and welcome to the 12th video of my free algorithmic trading course. In this video we will look at how to create options trading algorithms inside of QuantConnect. Currently QuantConnect supports options for about 4000 symbols trading on the CBOE. Unlike for other asset types, QuantConnect only supports minutely resolution for options. Options are divided by underlying asset, option type, strike price and expiration date. This means there can be hundreds of different options for each underlying asset. That's why options trading algorithms usually have to go through a lot more data than simple equity bots which is why it is particularly important to pay special attention to efficiency. If you're not familiar with how options work, I highly recommend checking up my website on which I offer a free course breaking down everything you need to know about options. That said, let's now look at how you can add options data to your algorithm. For this you can use the add option method. Here you can specify for which underlying asset you want to add the option data for. After that you can use set filter to filter which options you want to add. Here you can specify which strike price range around the underlying price and which expiration range you are interested in. Typically you would add the data and apply the filter in the initialised method. To then actually use the option data and trade options you can then iterate through option chain objects in the on data method. You can then filter the option contract in these option chains by strike price, expiration, greeks, open interest, implied volatility, price and more. I will show this in more detail in a few minutes when we get to the coding part of this video. The problem with using add option and set filter is that it adds a lot of option data to your algorithm that you might not actually need if you only intend on trading one or two contracts. That's why QuantConnect also offers an alternative that allows you to only add the data for individual options to boost efficiency and back test speed. This method uses the so called option chain provider. The option chain provider allows you to get the list of available option contracts for one security on a specific date. You can then filter this list to find a particular contract that you are interested in. To then actually trade it you would need to first add its data which you can accomplish with self.add option contract. Typically algorithms using the option chain provider method are much more efficient since they don't subscribe to that much data. So if you find an option algorithm to take too long to back test you could try switching to an option chain provider implementation. In this video I will demonstrate the add option and set filter method by implementing a simple options trading bot. However, I also have another video in which I use the option chain provider that I highly recommend watching after this video. If the demand is there I can also create another video in which I implement a bot that trades more complex multi-legged option strategies such as iron condors, butterflies, strangles, straddles or something else. With that being said let me now quickly present to you the strategy that we will be implementing for this video. For simplicity's sake we will be implementing a simple bot that buys coal options if the underlying assets price breaks out of its one month high price. This means we will use an indicator to keep track of the underlying securities high price. If a new high price is reached we want to buy an add the money coal option. For this strategy we will allocate 5% of our available portfolio capital to the coal option. This 5% might not seem like that much but since you can easily lose your entire investment with options it is actually a very risky strategy. Since we do not want to exercise our options we close them a few days before expiration. That said let's now head over to QuantConnect and start writing some actual code for this trading bot. As always we will go to the lab tab and click on the Create New Algorithm button which will bring us to a sample template algorithm. If you haven't already make sure to create your free QuantConnect account using the link in the description box below. For this video I will choose the back test time frame to be 3 years between 2018 and 2021 but feel free to try out different periods. I will leave the starting cash balance at the default 100,000. Next up we can already add the data for the underlying security using the add equity method. Here I will use Microsoft as the underlying asset but you can of course try it out with different securities. As for the resolution of the data I will specify minutely data since options currently are only supported with minutely data. Furthermore options only support row price data which is why we set the data normalization mode for this equity to row. Thereafter we save Microsoft symbol to the self dot equity variable and set it to be the benchmark for this algorithm. Now we can add the data for the options themselves. In this video I will use the set filter approach but there's a link to a video using the option chain provider in the description box below that I highly recommend watching after this video. So to add the option data we first use the add option method and specify Microsoft as the underlying asset add minutely resolution. Next up we apply the filter to this option data. Here the first two arguments specify how many strike prices above and below the underlying assets price we want to consider. Since we only want to buy at the money cool options we don't need a very wide range here. The next two arguments determine the time span for the expiration dates that are being considered. We will consider options with an expiration date between 20 and 40 days from now. Note that if your filtering criteria is not very specific a lot of option data will need to be added to the algorithm which might make it very slow and inefficient. After that there's only one thing left to do in the initialised method namely add an indicator that keeps track of the high price of Microsoft over the past month. For this we can use the ready to use max help from indicator and specify our equity as the data source. Here we just use daily resolution since smaller intraday changes are not that important for the high price. Since we are interested in the high price we specify that we want to keep track of the high price as the last argument of this method. With that we are now done with the initialised method and can move on to implementing the on data method in which we will implement when to enter and exit positions. However, before we can actually move on to that part we need to check if our max indicator is ready to use. If it is not yet ready we just return. Otherwise, we move on and in the next step we want to check if you already have any open option positions. For this we create a list with any potential open option positions. To create this list we iterate through our portfolio objects and save those that are of the type option and invested. So if this list is not empty it means that we have an open option position. In that case we want to check if there still is enough time until expiration since we do not want to accidentally let it expire and thereby exercise the option. So if there are under 4 days left until expiration we simply liquidate the position. If there is more time left but we still have an open position we just return since we already are invested. In the case that we are not invested we first check whether the underlying price made a new high. We check this by comparing its price to the max indicator's value. If there is a new high we want to open a long call position. To accomplish this we iterate through any available option chains and pass the available option chains on to another method that we will call buy call. So next up let's actually create this custom buy call method. Like the name implies this method is supposed to find the right call option and buy it. For this we first need to filter out the correct option from the option chains that were passed onto this method. Here we first sort the available chains by expiration date and pick the furthest away expiration date. Note that this will always be at least 20 days and at most 40 days from now since we applied a 20 to 40 day filter in the initialised method. Next up we filter out only the call options with this expiration date as we are not interested in trading any pull options for this strategy. After doing that we now just have to sort these call options by the strike price and then we are ready to actually buy the option. We want the option with a strike price that is closest to the underlying price. That's why we sort them from closest to furthest away from the underlying price. If this list is not empty we save the first element of this call contracts list to the variable save.call. This will be the call option that we want to buy. However, before we can actually buy it we need to calculate the order quantity. For this we first divide the available portfolio value by the options price. Since an option controls 100 shares of the underlying asset we then divide this quantity by 100 and multiply it by 5%. We multiply it by 5% because we only want to invest 5% of our portfolio into one position. We round this value down to the nearest integer and then we can finally buy it. Buying the options can be accomplished with self.buy and then passing the option contract and the order quantity. Now we are done with implementing the trade logic for this bot. However, before I backtest it I quickly want to mention that you usually need to pay special attention to option exercising and assignment when developing options trading algorithms. Especially when you're shorting options assignment is a big topic. When buying options and letting them expire, QuantConnect automatically exercises them. This algorithm should never exercise any options since we closed them early enough. However, if you want to handle option exercise differently you can for example do so in the unorder event method. To check that an order event is an exercise you can check the order type. If for example you don't want any shares of the underlying asset in your portfolio you could use self.liquid to close all open positions and reset your portfolio. That said, let's now click on build and backtest to check out this strategy's performance over the specified time frame. If we look at the equity chart we can see that this strategy performed quite well over the specified time frame. However, here it is important to remember that this performance is heavily dependent on the performance of the underlying asset. If the underlying asset which here is Microsoft would not be performing so well this strategy would likely not either. So if you would go with a different underlying or time frame the performance might be much much worse. Besides that you can see that this strategy performs best when the underlying price rises fast. Due to the payoff profile of options this strategy would not be nearly as profitable if the underlying price would climb up slowly or more sideways for a while. However, this strategy does seem to handle market crashes and down trending markets relatively well. This is the case since it does only invest when the underlying price makes a new high which typically does not happen in a down trending market. So in such a market environment this bot just stays flat. Nonetheless I can't recommend actually trading this strategy with real money without any further development and testing. Firstly, a lot of the parameters such as the time to expiration, breakout period and more are chosen very arbitrarily. Furthermore, over this back test time frame that strategy only made 43 trades which isn't a very significant amount that you should use to draw many conclusions from. Nevertheless I hope that this bot gave you a good idea about how to trade options inside of QuantConnect. For another options trading example bot video in which I used the more efficient option chain provider I highly recommend checking out the video that I linked in the description box below. That said, make sure to smash the like button, subscribe and turn on the notification bell for more content like this. Thanks for watching. diff --git a/course/13-options-code-along.md b/course/13-options-code-along.md new file mode 100644 index 0000000..2d549dd --- /dev/null +++ b/course/13-options-code-along.md @@ -0,0 +1,22 @@ +# 13 · Options Code-Along（保护性看跌期权实战） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Options Code-Along（保护性看跌期权实战） +- **时长**：26 分 6 秒 ｜ **原视频**：https://youtu.be/Lq-Ri7YU5fU +- **本地视频**：[videos/13-options-code-along.mp4](videos/13-options-code-along.mp4) + +## 🎯 本集要点（中文导读） + +**完整代码实战**：给现有股票多头仓位买**保护性看跌期权（protective put）**。 + +- **策略目标**：降低组合波动、对冲下行风险，同时不浪费太多钱在"不必要的保护"上。 +- **为什么用 put**：负 delta（标的下跌时获利）、正 vega（隐含波动率上升时获利，而 IV 通常在下跌时上升）、风险有限而保留了上行空间。 +- **入场条件**：只在**隐含波动率相对较高**时才买 put（用过去半年的**标准化 IV**判断）——即"等市场已经跌了一些再买保险"，而不是一直买。 +- **结果解读**：作者演示如何用回测图表/指标/订单/日志评估；⚠️ 明确说明这是**教学示例，不保证盈利**。 +- 建议：逐行读代码、自己动手改，是熟悉 QuantConnect 期权 API 的好方法。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to another video. In this video we're going to develop and implement an options trading algorithm inside of the QuantConnect platform using Python. If you want to code along, make sure to first create a free QuantConnect account. There's a link in the description box below. Otherwise, I want to start with a short disclaimer. Name me that the main purpose of this video is to give you an overview of the trading algorithm development process, the QuantConnect platform and its API. Therefore, I'm not claiming that the strategy that we're going to implement is necessarily a very good or profitable strategy. So please keep this in mind when watching this video. With that being said, let's cut right to the chase and start developing a trading strategy. Before we start actually implementing anything, though, let me first present the strategy on a conceptual level so that you better understand what our code is supposed to do. The strategy that we're going to implement is not a very active strategy. The goal of the strategy is to reduce volatility and protect against downside moves in an existing long equity position. To accomplish this, we need an investment with an inverse correlation to our long stock position. For this, we will be using put options because of several reasons. First of all, they have a negative delta, which means that their profit from downward moves in the underlying asset. Secondly, they have a positive vega, which means that their profit from an increase in implied volatility. And this is good because implied volatility usually goes up when prices go down. In other words, implied volatility typically has an inverse correlation to the underlying asset. Last but not least, put options have limited risk while leaving a lot of upside potential. The problem with this strategy so far is that constantly buying put options as protection can be very expensive and negatively impact the overall performance. That's why we only want to buy put options if we suspect an upcoming market decline. We measure this by looking at the normalized implied volatility of the past half year. If implied volatility is relatively high, we buy a put option as protection. Usually, this will mean that the market has already dropped by a certain amount. But in the long run, we hope that buying protection a little later will be better than wasting money on unnecessary protection on a constant basis. Before we head over to the QuantConnect platform, let me quickly recap the strategy one more time. The goal of the strategy is to reduce downside risk and volatility of an existing long equity position without wasting too much money on unnecessary protection. We accomplished this by buying put options when volatility is relatively high. We used this high level of implied volatility as a sign for a potential upcoming market decline. With that being said, let's now head over to the QuantConnect platform and start writing some Python code. If you haven't done so already, make sure to quickly create a free account before continuing to watch this video. If you have done so, navigate to the lab tab and click on Create New Algorithm. Then a collection of template modules will pop up, but we won't be using any of them in this video. Instead, we click on Create Algorithm to create the new algorithm. For now, you should see an on data and initialize method. We will use both of these methods for our algorithm. But before we start implementing them, let's first import two things. Firstly, we will have to import time delta from date time, and then we will import everything from the QuantConnect.data.custom.cboe module. We will use this custom cboe data to get pricing data for the volatility index or VIX, which measures the implied volatility of the S&P 500 index. We will use this data to create the indicator that measures how high or low implied volatility is compared to recent history. But before we get to the volatility indicator, let's first implement our initialize method. We will first set the start and end date for our back tests. We will choose the first of October 2017 as a start date and the first October 2020 as an end date, but feel free to try out different periods if you want to. Next up, we will set a starting cash balance for the back tests. I will go with $100,000 for now. Then we will have to add the underlying security that we are going to trade. For this video, I will choose the ETF SPY as an underlying security since it tracks the S&P 500 index and has great options liquidity. We will add minutely data, since as of right now, the QuantConnect API only supports minutely data for options. Furthermore, only raw pricing data is supported with options, which means that the historical data does not adjust prices for stock splits or other similar events, but this shouldn't create any problems for us. Lastly, we save the symbol of our underlying asset into the variables self.symbol. Then we also want to add VIX pricing data, which we can do with the add data method. We save the symbol in self.vix so that we can later access the data. Then we initialize the volatility indicator to zero. We will later use this indicator as an entry signal for our options position. Besides that, we also have to initialize a few more variables. First, we initialize self.contract to an empty string and self.contract added to an empty set. self.contract added will keep track of which options contracts we have added so that we don't unnecessarily add options data multiple times. Next up, we will initialize most of our parameters. The first parameter is called self.days before exp and it represents how many days before expiration we will close our options position. We want to close the position before expiration so that we don't exercise any of our options. This means that two days before expiration we will close our long put position if we have one. I quote the next parameter self.dte and it represents our target time to expiration in days. So a value of 25 means that we aim to buy a put option with about 25 days until expiration. self.otm is the target percentage that our option should be out of the money. So 0.01 means that we want to buy a put option that is out of the money by 1%. Next up, we have self.lookbackiv which represents the number of days our volatility indicator looked into the past. So 150 means that it compares the current VIX level to that over the past 150 days. Self.iv level is a level of our volatility indicator above which we want to buy put options as a protection against the potential market crash. In other words, this is the signal line. Then we specify what percentage of our portfolio will go to the underlying asset. Since this strategy is a passive strategy, I will initialize this variable to 0.9 which means that about 90% of our portfolio will go to SPY. Last but not least, self.options.alloc specifies how many options contracts we buy compared to the number of shares of the underlying asset that we own. So a value of 90 here means that we buy one option for every 90 shares that we own. The standard option contract covers 100 shares of the underlying. So a value of 90 here means that when we buy put options as protection, we buy more than enough to cover the entire portfolio. The lower this value is, the more protection we buy. I know the initialized method can be a little boring, but we're almost done with it, so stick with me. We just quickly have to schedule two methods that we want to be called on a regular basis. First, we schedule our so called plotting function, which we use to create custom charts. We wanted to be called every market day that SPY trades. Or more specifically, we wanted to be called 30 minutes after every market open. We then scheduled the VIX rank method in the exact same way. This method will be used to calculate our volatility indicator value. Last but not least, we have to set a warmer period for our algorithm so that the volatility indicator has enough data to be used correctly right from the get-go. Now that we are done implementing the initialized method, let's create and implement the VIX rank method so that we can use our volatility indicator for entry signals. To create and update the volatility indicator, we first need to look at the past VIX price data. We can access VIX's price history by calling the history method. We then specify the symbol, time frame and resolution of the requested data. For the symbol, we use self.vix and for the time frame, we use the self.lookback.iv variable to get the price history of the past 150 days. Furthermore, we request daily data since we are only interested in the high price, low price and current price over this time frame. The history method then returns a data frame with a high, low, open, close price and volume for each day over the specified time. The next step is to actually calculate the value of our volatility indicator which we will call self.rank. To calculate this value, we have to normalize the current VIX level by looking at its high and low price. The exact formula that we are going to use is the current IV level minus the low price divided by the price range. In simple terms, the VIX rank method calculates how high or low the current VIX level is compared to recent price history. So a value of 1 would mean that the current VIX price is at a new high, whereas the value of 0 would mean that it is at a new low, and a value of 0.5 means that the current VIX price is right in the middle. Now we are finished implementing the VIX rank method, which means that we can move on to the on data method, which is called every time our algorithm receives new data. This should be pretty much every minute, since we request it minutely SPY data in the initialize method. The first thing we have to do is make sure that our algorithm is finished warming up. We can do this with a simple if statement. If it's not finished warming up, we just return and do nothing. Next up, we check whether we already are invested in the underlying asset. If we aren't, we use self.setHollings to buy shares until I put four years' holdings, and this asset are about the percentage of self.percentage. The next step is to check whether our volatility indicator self.rank is above the entry signal level, which we initialize to 0.5. If it is, it means that SPY's implied volatility is relatively high, and we therefore want to buy put options as protection. We do so by calling the self.buyput method, which is a method that we will implement in a few minutes. Now, there only is one thing left to implement in the on data method, namely checking whether we are too close to the expiration date. If we are, we want to close any existing put positions. Obviously, we only want to do this if we already are invested. That's why we first check whether self.contract is non-empty. If it is empty, we should be invested in a put option. We then check whether the expiration of this option is within two days of today. If it is, we want to close it so that it doesn't expire as exercised. To close the option, we use self.liquidate, there after we lock that we close the option and reset self.contract to an empty string. Now that we are done with the on data method, let's implement the buyput method, which will actually buy the put options. But before this method can buy any options, it first has to get the data for these options. So if you currently don't have a contract to trade, we first get it by calling options filter, which is another method that we have to implement here after. After saving this contract to self.contract, we return. We do this because adding new options data usually takes at least one iteration of your algorithm. If we already have a contract to trade, we aren't already invested in it and its data is ready, we want to buy this put option. This can be accomplished with a self.buy, which takes two arguments, the first is the symbol of the actual security, and the second one is the quantity. Since we want to buy the earlier specified put option, we use self.contract for the first argument. And for the quantity, we divide the number of shares of the underlying that we own by the self.options.alloc parameter, which we initialized earlier. But before we can use this value, we have to round it to the nearest integer. This method will now buy one put option for every 90 shares of the underlying security that we own. Note that the 90 shares is the value of self.options.alloc, which we initialized in the beginning. Next up, let's implement the options filter method, which we use to find out which option we want to trade and add the correct data for this option. The QuantConnect API has multiple ways to trade options. The easiest way to trade options is by using set filter and by iterating over option chain objects. The problem with this is that it is unnecessarily as huge amounts of options data, which makes the algorithm very inefficient. In fact, it makes it so inefficient that it can take many hours for you to back test your algorithm over any noteworthy length of time. That's why we won't be doing it this way. Instead, we will be using the so-called option chain provider, which allows you to manually add the data for only those options that you actually want to trade. Doing it this way allows you to back test strategies that would normally take hours in a couple of seconds or minutes. But one disadvantage of the option chain provider is that it currently does not allow you to use options, Greeks and implied volatility values. Luckily, we don't need them for this algorithm, but for many other options algorithms, we will want to use the Greeks to select option contracts to trade. So if that's your goal, the option chain provider is currently not the best way to do so. But that's a topic for another video. Now, let's implement the options filter method. We will first add the currently available options contracts for our underlying by using self.optionchainprovider.getOptioncontractList. We can now manually filter this contract list as strike price, expiration, option type, and option style. But before we do that, let's quickly save the current price of the underlying asset to self.underlying price. We can do this with self.securities.price. Now, we first filter out all the out-of-the-money-put options that expire close to our desired expiration date and save them into OTM puts. To filter out put options only, we check that the option write equals put for each option. To only select sufficiently out-of-the-money options, we make sure that the underlying price minus the strike price of each option is greater than the underlying price multiplied by the amount we want our option to be out of the money. This amount, self.otm, was initialized to 1% in the initialized method. Next up, we check that the number of days from now until the expiration date of each option is within plus or minus eight days of our desired expiration date. If the OTM puts list now isn't empty, we sort it in a way so that the options that are closest to our desired strike and desired expiration date come first. Thereafter, we select the first option of this sorted list. This is the option that we will want to trade. But before we can actually trade it, we must first add its data to the algorithm. We only do this however, if we haven't done so already. If we haven't, we use add option contract to subscribe the data for the specified contract. Then we return this contract so that the buy put method can buy it. If the OTM puts list is empty, we just return an empty string since there's no option that meets our criteria. Now, we are pretty much done with implementing the functionalities of this trading algorithm. Nevertheless, we want to add two more small things. Firstly, we implement the plotting method, which will create two charts that can help us understand our algorithm's performance. To create a custom plot, we use self dot plot, which takes three arguments. The first is the name of the chart that we want to create. And the second is the name of a specific plot on this chart. And the final argument is the actual data for this plot. We will create two custom charts with two plots each. The first will plot our IV indicator, as well as the entry signal line. And the second chart will plot the price of the underlying asset, as well as the strike price of any open put positions. To plot the strike price of an open put position, we must first check whether we are invested. We do this by creating lists of all option positions in our portfolio. If this list is not empty, we have an open put position. If that's the case, we use self dot plot to plot this option strike price. Last but not least, we will use one more method for one more tiny thing. The name of this method is an unorder event. And like the name implies, it is automatically called on every order event. All we want to do is use self dot log to log each order event. This is great for backtesting, debugging and live trading purposes, since it can give you a good overview of what order events are happening. With that being said, we have now successfully created our options trading algorithm. All that's left to do for us now is click the back test button in the top right. This will then back test our trading strategy over the time frame that was specified in the initialize method. Since I specified the time frame to be the last three years, the results will be those generated from this data. Please keep in mind that depending over which time frame you choose to back test this algorithm, the results might be significantly different and potentially much worse than over this time frame. When the back test has finished, a performance overview and report will be generated for you to analyze your strategy and its performance. I will select the data chart and voil chart since these are the custom charts that we created for this algorithm. Since we didn't use most of these other charts, I will deselect them. As you hopefully can see over this time, this strategy seemed to work pretty good. We can directly compare our strategy's performance with that of SPY by looking at the data chart below this equity chart. As you can see from the data chart, SPY also managed to generate a pretty solid return. However, the key difference here is that SPY achieved these returns with a pretty high degree of volatility. Over these three years, there were multiple times where SPY actually was down overall. Our strategy on the other hand was profitable pretty much the entire time. Especially the very steep crash of early 2020 was handled very well. Generally speaking, the put options protect us best against short and fast crashes. It doesn't perform nearly as well during long term bear markets such as the one in 2007 and 2008. Below the data chart, we can see the custom voil chart which displays our custom VIX indicator as well as the entry signal level. Below that chart, we can see an overview of many more metrics that can be helpful when analyzing and evaluating a trading strategy and its performance. Down here, you can also view all the orders that the algorithm places, access the logs, the code and much more. If you want to learn how to best evaluate a trading strategy with the help of these metrics, charts and more, I highly recommend checking out my latest video in which I talk about how to best evaluate your trading strategies. Otherwise, if you want to copy the code from this video and play around with this algorithm yourself, make sure to check out the link in the description box below to clone this algorithm. I highly encourage you to go through each line of code and try to really understand what is going on. If you want to, you can also try adding some improvements or just changing a few things here and there. In my opinion, that's one of the best ways to learn and get familiar with the new API. I know that the strategy presented in this video was relatively simple, but the QuantConnect options API can be confusing especially for newcomers. That's why I wanted to start with a relatively simple example. But if you want me to create a similar video in which I implement a more complex multi-legged option strategy or some other trading algorithm, definitely let me know in the comments section below. If you liked this video, you should also check out one of my other reason videos in which I implement a stock trading strategy in QuantConnect using Python. There's a link in the description box below. Otherwise, definitely make sure to like this video, subscribe to my channel and turn on the notification bell for more content like this. Thanks for watching. diff --git a/course/14-crypto-trading-bots.md b/course/14-crypto-trading-bots.md new file mode 100644 index 0000000..16683df --- /dev/null +++ b/course/14-crypto-trading-bots.md @@ -0,0 +1,23 @@ +# 14 · Crypto Trading Bots（加密货币 · RSI 动量） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Crypto Trading Bots（加密货币 · RSI 动量） +- **时长**：15 分 51 秒 ｜ **原视频**：https://youtu.be/yuZBBX47xK0 +- **本地视频**：[videos/14-crypto-trading-bots.mp4](videos/14-crypto-trading-bots.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **加密货币（Crypto）算法**。 + +- **支持的交易所**：Bitfinex（346 对）、Coinbase Pro（原 GDAX，9 对）、Binance；数据回溯到 2014（取决于币种）；**统一 UTC 时区**。 +- ⚠️ **必须按 Cashbook（现金簿）决策**：做多 `BTC-EUR` 需先持有 EUR，做多 `ETH-BTC` 需先持有 BTC。可用 `self.Portfolio.CashBook[...].Amount` 查某币持仓量。只想规避此问题就专做各币的 **USD 计价对**。 +- **添加数据**：`self.AddCrypto("BTCUSD", resolution, market)`（支持 tick~日线；market 选 Bitfinex/GDAX/Binance）。 +- **实盘**：需在受支持的交易所注册、生成 API token，并在 QuantConnect 实盘向导里配置。 +- **示例 bot**：**动量策略**——用 **RSI** 识别上行动量后买入，动量消退则平仓（做多若干加密对）。 +- ⚠️ 作者提醒：**加密回测极易"虚高"**（过去是大牛市）；且只用了"当前仍被支持的币"，仍存在**幸存者偏差**，换环境可能失效。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Hello and welcome back to another video of my Python algorithmic trading course. So far we have explored how you can create trading algorithms for equities, forex, CFDs and options inside of QuantConnect. In this video I will show you how you can create crypto trading bots. But note that if you are new to this series I recommend starting at the beginning since the previous video lessons are just as relevant to crypto trading as for the other asset types. That said, let's start with a brief theory part before moving on to coding an example crypto trading bot. QuantConnect currently supports crypto trading on Bitfinex, Coinbase Pro which was previously known as Gdex and Binance. Furthermore, Crocken will be supported soon. Bitfinex offers 346 cryptocurrency pairs whereas Coinbase Pro currently only offers 9 pairs. The data goes back all the way to 2014 but since some cryptocurrencies are newer the availability of data depends on the currency. For a full list of the supported pairs check out the link documentation page below. All crypto data uses the UTC time zone. One important thing to note when trading cryptocurrencies is that you must make trading decisions based on the cash book. The cash book keeps track of what kind of currency your portfolio currently holds. For example, if you want to buy the crypto pair Bitcoin Euro you would need to hold Euro in your portfolio. The same goes for a pair like Ethereum Bitcoin. To buy this pair you would need to own some Bitcoin since you are buying their Ethereum with Bitcoin. But note that you can also just buy crypto currencies against USD if you just focus on the crypto USD pairs. To get the amount of a certain coin in your portfolio you can access the amount property of the cash book of your portfolio. For example, this line would get the amount of Bitcoin that you own in your portfolio. Similar to the ad equity and ad for X methods that also exists in ad crypto method to add the data for crypto pair to your algorithm. As always, the first argument here simply is the ticker of the pair and the second argument is the resolution. Bitcoin Connect supports everything from tick, second, minute hourly to daily resolution for cryptos. As for Forex, the third argument is where you can specify the exchange that you want the data from. Here you can currently decide between BitFenix, GDAX and Binance. To live trade crypto currencies through QuantConnect you would have to sign up to one of the supported exchanges. You would then have to create an API token and specify this in the live trading setup wizard within QuantConnect. For more details on this however check out the documentation page. With that being said, let me now outline the strategy that we will implement in the coding part of this video. The strategy that we will implement is a simple momentum strategy where we try to identify crypto currencies with upward momentum and buy them until the momentum fades. To measure momentum we will use the relative strength index indicator also known as RSI in short. If the RSI for a given crypto crosses above a certain threshold we want to buy it and we first close the position if RSI drops below an exit threshold. The goal of developing and back testing algorithm trading strategies is to best possibly model reality so that the results obtained from analysis are actually meaningful. Therefore, you have to pay special attention to removing as many hidden biases as you can. Actually, one of the biggest challenges of back testing that just describes strategy is realistically modeling the universe selection process. One possible way to go about the universe selection would be to simply look at the currently most actively traded cryptos and then use them as the base universe. However, then the results would be heavily biased from survivorship bias since we are not considering currencies that no longer exist or used to be much more popular than they are today. In other words, we are applying a filter to only consider the surviving cryptos from today's perspective. So be very careful when developing your own bots and always ask yourself whether you are unknowingly applying such a filter. For this algorithm I will add the data for all crypto USD pairs that are supported by Bitfinex. The algorithm will then look at the 30 day average daily dollar volume and it only adds those cryptos to the current active universe if they have sufficiently high enough dollar volume. Only for these cryptos it then looks at the RSI for potential momentum trades. I hope this brief explanation makes it clear what the algorithm that we will code next is supposed to do. If you haven't already, I highly recommend creating a free QuantConnect account so that you can code along. Inside of QuantConnect we will create a new algorithm inside of the lab tab. As always we will start by implementing the Initialized method. Firstly, I will set the start and end date for the back test to the beginning of 2019 until 2021. Note that depending on which cryptocurrencies you want to trade the available data might be limited. After that I will keep the starting cash balance to be $100,000. Before we move on to creating some helper variables let me set the free portfolio value percentage to 5%. Doing this will ensure that always at least 5% of our portfolio will stay in cash which can be useful if you want to have a cash buffer for your portfolio. Now we will create four variables that will store some parameter values. The first is called Position Size USD and this is where we specify the amount we want to allocate to a single position. Here we will go with $5,000. The next two parameters will hold the Entry and Exit threshold for the relative strength indicators. We will use $70 as the entry threshold and $65 for the exit. This means if RSI rises above $70 we enter and we first exit if it falls below $65. Last but not least we specify the average daily dollar volume that we will use to filter out illiquid securities from our active universe. Here I will go with a value of $1 million. Note that all these values are chosen quite arbitrarily and you could definitely try out different ones if you want to. However, note that over optimizing them by backtesting and adjusting them over and over again is the quickest way to end up with a useless overfit algorithm. Next up we want to add the data for all available crypto USD pairs. For this I created a list with all these crypto pairs available on the bidfinex exchange. If you are coding along you can copy this list from the description box below. Note that there also is a link in the description box to copy my entire finished code but in my opinion coding along leads to a better learning experience. We now want to iterate through this list and add all the data for these tickers. However, since we want to access certain properties of all these tickers later it makes sense to create a custom class for this. So let's do that first. We will call this class pair and we will start by implementing its python init constructor. As arguments the constructor will take our algorithm, a ticker and a minimum volume which we will use to determine whether or not we want to invest in this pair. Firstly, we now want to add the data for the ticker. We can use the add crypto method to accomplish this. The first argument is the ticker, the second the resolution and the third is where we specify the bidfinex exchange. Next up we want to create a RSI indicator for this symbol. To create an RSI indicator we can use the RSI helper method and specify the symbol, period, type of moving average and resolution. I will go with the most common 40 day RSI but feel free to try out different values. In the next step we want to create an indicator that keeps track of the average daily dollar volume over the past 30 days. For this we will use the indicator extension that multiplies two indicators together. The two indicators that we want to multiply are a 30 day simple moving average of the cryptos volume and a 30 day SMA of its price. With that this volume variable should now keep track of the 30 day average daily dollar volume. Last but not least we save the past minimum volume argument into a class variable. Generally it is good practice to group things together that belong to one ticker by defining a class such as this one. Before we head back to the initialised method of the actual algorithm let me create a method with the name investable. As the name implies this method should return true if the crypto pair is investable. We define this to be the case if the average dollar volume exceeds the minimum volume threshold that we passed when initialising the pair object. With that being done let's now head back to the initialised method of the original algorithm class. Here we can now create a list saving all these custom pair objects by iterating through the lists of the tickers that we specified earlier. After that we set the bitcoin use D pair to be the benchmark of this algorithm. Furthermore we set the warm up time for this algorithm to be 30 days since the indicators that we use require this. After that we are now done with the niche loss method. This means all that's left to do now is implement the on data method where the actual trade decision making will take place. In the on data method we will iterate through our pairs lists. The first thing that we want to do is check if the RSI indicator for the current pair is ready. We do this by checking the isready attribute of the RSI. Next up we save the symbol as well as the current value of the RSI of the given pair into a local variable since we will need these multiple times in the next few lines. In the remainder of this for loop we want to differentiate between two cases. The first being that we already are invested in the current pair. We can test this by checking the invested property of our portfolio. If we are invested there are two conditions that would lead to us wanting to close the position. Firstly, the average volume of the pair might have fallen too low. We can check this with the investable helper method that we defined in the custom pair class. In that case we liquidate the position. Otherwise the RSI might have fallen below our exit threshold which we can check with a simple comparison. In this case we also use self.liquidate to close the position. After that we jump to the next iteration of the for loop since the rest is irrelevant for this case. So in the remainder of this loop we can assume that we do not have any active position in the given pair. Therefore we want to test if the RSI is above its entry threshold. However, we only want to do this if the volume of the pair is high enough which we once again test with the investable helper method. If it's not we continue to the next iteration. Otherwise we test if the RSI is above its entry threshold and that the remaining margin in our portfolio is above the desired position size. If both these conditions are fulfilled we open a position using the self.buy helper method. As the position size we need to divide the position size in USD by the price of the crypto pair. With that we are now done coding this crypto trading algorithm. So let's now click on build and backtest to look at its performance over the backtest time frame. As you can see this seems to be a quite volatile strategy which shouldn't be too surprising considering that it trades very volatile cryptocurrencies. After the end of 2020 this strategy pretty much only breaks even and most of the profits come after that. Here it is important to note that it is very easy to develop a crypto bot that performs incredibly well in backtesting since the crypto space has gone through one of the biggest bull markets ever. Just for reference you can look at the Bitcoin benchmark chart and see its incredible bull run. In general it's not hard to develop a profitable bullish strategy in a strong bull market. However, as soon as the market environment changes most of these strategies would not work anymore. So try to keep this in mind when developing a crypto trading bot. Besides that this strategy likely still falls prey to survivorship bias since we only consider cryptos that are currently supported by QuantConnect and Bitfinex. This list likely would have been different a few years ago. Another interesting chart to look at is the exposure chart which allows you to see the percentage of your portfolio allocated to various long or short positions. Here we can see that most of the time the exposure only was about 5 to 10% of our portfolio which equates to 1 to 2 positions since each position is worth about $5,000. Just once in a while the exposure jumps up to between 30 and 50% for a short period of time. If you want to check out this report and my code in more detail make sure to use the link in the description box below to clone the code. That said I hope you enjoyed this video and learned a lot. If you did make sure to smash the like button, subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/15-algorithm-framework.md b/course/15-algorithm-framework.md new file mode 100644 index 0000000..3d76dc9 --- /dev/null +++ b/course/15-algorithm-framework.md @@ -0,0 +1,27 @@ +# 15 · The Algorithm Framework（算法框架） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：The Algorithm Framework（算法框架） +- **时长**：34 分 14 秒 ｜ **原视频**：https://youtu.be/YQ3xwZuly8I +- **本地视频**：[videos/15-algorithm-framework.mp4](videos/15-algorithm-framework.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **Algorithm Framework（算法框架）**——用"职责分离"的类结构组织策略。 + +- 传统写法把逻辑都塞在 `OnData/Initialize` 里，**职责不清**；框架把算法拆成 **5~6 个模块**（可插拔、可复用）： + 1. **Universe Selection Model**：决定"考虑哪些标的"（选资产类别、剔除流动性差/无数据者、按主题等过滤）。 + 2. **Alpha Model**：你的 **edge** 所在——对标的给出 **alpha 分数**（方向 + 信心），即**交易信号**（还只是"建议"，不是订单）。 + 3. **Portfolio Construction Model**：把信号变成**目标组合**（每个信号分配多少资金；最简单的如等权）。 + 4. **Risk Management Model**：风控（限制敞口、止损等）。 + 5. **Execution Model**：把目标组合转成**实际订单**并执行。 + 6. **Transaction Cost Model**：计入交易成本（有时并入执行模型）。 +- **流程**：Universe → Alpha（信号）→ Portfolio Construction（定仓位）→ Risk（风控）→ Execution（下单）。 +- **示例 bot**：用框架实现一个"按板块/基本面选股、生成 Insights（价格预判）、做多"的策略；示例用 `Insight.Price(symbol, expiry, direction)` 生成 insight。 +- 优点：清晰、可替换各模块、易扩展；初期看着"绕"，但用熟后收益很大。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to the next video of my Python algorithmic trading course. So far we have only looked at one way of developing algorithms inside of QuantConnect, namely by creating different methods that handle different functions of the algorithm. Even though this is a simple and straightforward way to accomplish this, it does not really logically separate the different responsibilities of your strategy. Following a class-based approach with a clear separation of concerns instead can have many advantages. That's why QuantConnect offers the so-called algorithm framework which does exactly that. So in this video I will introduce the algorithm framework and its advantages. Furthermore, we will code a trading bot that uses this algorithm framework so that you can see it put into practice. With that being said, let's start breaking down what goes into a typical trading algorithm. Most trading algorithms are divided into six distinct components. The first component is the universe selection model. This is where your algorithm decides which assets it considers when looking for new trades. First, you will have to choose which asset class your strategy focuses on. Next up, you might want to implement further restrictions. At the very least you will want to filter out illiquid securities. Furthermore, you obviously only want to consider assets that you have data for. Besides that, your strategy might focus on a specific subset of the remaining assets. If for instance your strategy focuses on tech stocks, you will want to filter out all non-tech stocks. Alternatively, you might want to filter out too cheap or expensive securities or add other filters. After the universe selection model has filtered out those assets that don't meet all the required criteria, it can pass on the list of remaining securities to the alpha model. The alpha model is where your edge comes into play. This is where your algorithm assigns alpha scores to the securities that were passed on from the universe selection model. These alpha scores represent a directional assumption as well as a level of confidence in this prediction. In other words, this is where a trade signal is generated. But note that this trade signal is not directly a trade order. A trade signal can be thought of as a trade recommendation. Whether or not this recommendation is turned into a trade order is decided by the next model, namely the portfolio construction model. This is where the trade signals from the alpha model are turned into a potential portfolio. In other words, this is where it is decided how much capital is allocated to each trade signal. One of the easiest portfolio construction models is an equal weighted model that allocates the same amount of capital to each position. But sometimes you might want to allocate more to specific trades due to a higher level of confidence or some other reason. How this is decided depends on your implementation of this model. Next up is the transaction cost model. This is where you aggregate the accounts for transaction costs. Sometimes this model is integrated into the execution or portfolio construction model. The reason why this model is important is that sometimes it might not be worth it to open a new position due to high transaction costs. If you don't expect to make enough money to cover the transaction costs, you should not open the position. Besides the obvious transaction costs such as trade commissions, the transaction cost model also estimates other costs such as slippage and market impact costs. Depending on your order size and trade volume, these can have a significant impact to your bottom line which is why it is very important to consider them before sending out any trade orders. The model that is responsible for sending out the actual trade orders is the execution model. This model decides how a new position will be opened. It decides on how many trade orders your position is split up, what kind of trade orders are used, when these orders are sent out and more. Especially for bigger orders, it can be very important to split up the trade entry into multiple smaller steps. Furthermore, depending on the importance of the trade, you might want to use limit orders, market orders or some other type. Last but not least, let's talk about the risk model. This is the part of your trading algorithm that monitors your portfolio's risk and manages positions when necessary. A simple example of a risk model would be one that closes a position as soon as it is down more than a certain percentage, but obviously more sophisticated risk models can be much more complex. Note that, even though I presented all these models individually, they are all interdependent and very much rely on each other. All these models are very important. Even if an algorithm has a great alpha model, it can still perform terribly. Ten algorithms with the exact same alpha model can perform vastly different depending on the implementation of the other models. Now let me quickly go through each of these components for a simple example algorithm. This example algorithm bases its trades on the 50-day moving average and I would not recommend using it for any trading. Let's start with the Universe selection model. This algorithm only wants to trade liquid US financial stocks. That's why it orders all US financial stocks by the $80 volume and picks the top 10 as its trading universe. The alpha model then compares the current trading price of these 10 stocks to their 50-day moving average. If the current price is above the 50-day moving average, it sends a sales signal, otherwise it sends a buy signal. These signals are then picked up by the portfolio construction model which gives each of these 10 potential positions equal weights. Furthermore, it makes sure that not more than 75% of the available capital is allocated. The transaction cost model is very simple and assumes $5 per trade as commissions and simply disregards the cost of slippage and market impact. This means that the transaction cost model does not reject any trades due to two high transaction costs. The risk model only monitors the current percentage P&L of each position. If the profit of a position reaches 5% or a position loses more than 2%, the risk model sends a closed signal. First but not least, the execution model is called once a week on Monday at 10am. It then opens or closes positions with market orders depending on the information it received from the portfolio construction and risk model. I hope this brief example gives you an idea of what these models can look like. Of course, this is a very simple example. In reality, these models can get incredibly complex depending on the strategy and its implementation. But in the beginning, it is best to start out simple and move forward from there. You should now have a much better understanding of the design framework that more advanced trading algorithms usually follow. Before we move on to how you can use this algorithm framework inside of QuantConnect, let me first go over the advantages of using such a framework. First and foremost, this framework allows the coding principle of separation of concerns. This means that components of your algorithm that have their own functionalities will be separated into different classes. One of the main advantages of this is that the code is much easier modifiable and readable. If for example, you want to change how your algorithm handles the execution of your trades, you would only have to go and modify the code inside of the execution model while leaving the rest of your code untouched. Another advantage of dividing everything into its own module is that you can easily replace entire aspects of your strategy by using a different model. Besides developing your own custom models, you can also use community-created or other ready-to-use modules. If for example, you create a really promising risk model, you could just reuse the exact same model for different algorithms. Furthermore, QuantConnect offers a bunch of ready-made versions for pretty much all of the models that you can easily import into your code and use. I will demonstrate this later when we get to the coding part of this video. In general, using the algorithm framework allows you to focus on what you are good at. If for example, you don't like implementing the details of how trades are executed but instead prefer focusing on the alpha generation, you can just dedicate your time to the alpha model and use a pre-built execution model. That said, designing algorithms according to the algorithm framework does usually require a little more effort on the implementation side. Furthermore, it requires an understanding of class and object-oriented programming. That's why I recommend starting with basic classical algorithms, which is also exactly what we did in this series so far. However, once you get the grip of them, it can definitely be beneficial to try out the algorithm framework. Before we move on to actually coding a bot, let me quickly break down the details of how QuantConnect handles the algorithm framework. Firstly, let me quickly outline the workflow order in which QuantConnect executes these models. The first model that is executed is the Universe Selection model which will then filter out securities for the active universe. These securities will then be sent to the alpha model which generates trade signals based on the received data on these securities. These trade signals are then sent to the portfolio construction model which constructs the portfolio of investments out of them. However, before this portfolio is turned into actual trade orders, it is first sent to the risk model which makes sure that the theoretical new portfolio doesn't violate any risk requirements. Only after all that, the execution model will be called to actually turn the theoretical trades into actual real trade orders. Note that the alpha model does not generate any trade orders. It simply generates trade suggestions. Inside of QuantConnect, these trade suggestions are so-called inside objects. They are passed on to other models and the portfolio construction and execution model can then decide how to turn these insights into actual trade orders. An inside object has eight properties of which four are optional. The first argument of its constructor specifies the symbol for which the inside should be generated. The second one determines the time period that the inside should be valid. Depending on the strategy, this can be everything from a few seconds until multiple years. For example, if you want to base a trade on a foundational macroeconomic decision, you likely would go for a longer period than for a momentum intraday trading decision. The third argument is the type of insight. Here, you can either go with an insight for the price of the specified symbol or its volatility. In the vast majority of cases, you would want to emit a price insight. That's also why there is a direct helper method that allows you to skip this argument. The next argument is the direction of the insight. This can either be up, down or flat. The remaining arguments are optional, but depending on the implementation of the other models, you might want to add values for them as well. The first of these optional arguments is the magnitude of the insight. This is where you can specify how big you expect the upcoming move to be. Another parameter you can specify is the confidence in this insight which you can use to define how strong the signal is. Two more arguments are the source model and the weight of your insights. The source model is used to link an insight to a given alpha model. Since an algorithm can get insight from multiple alpha models, it would use this attribute to find out from which alpha model a given insight was emitted. The last weight argument can be used to state how much emphasis should be put on this insight compared to any other emitted insights. Here is an example of what an insight for AAPL could look like. This insight states that it expects Apple's price to rise 5% within the next 30 days with the confidence of 90%. Note that this is merely a trade suggestion and not an actual trade order. After the alpha model emits such an insight, the other models can decide whether to turn this into a trade order and if so, how to do so. Since you now hopefully understand how insights work, let me outline the structure of an alpha model inside of QuantConnect. To create an alpha model class in QuantConnect, you need to implement the i alpha model interface which consists of two mandatory class methods, namely an update method and an onsecurities changed method. The update method of an alpha model works very similar to the on data method since it updates the alpha model with new data and emits insights. To actually emit insights, you simply return your list of insights in this update method. To emit insights elsewhere, in your algorithm, you can use the emit insights helper method. As the name implies, the onsecurities changed method is where the alpha model handles changes in the active universe. Typically, you would handle positions that have left your active universe and apply the decision making logic to newly added securities. For helper class variables and initialization, you would use the standard Python init constructor. After this brief theoretical breakdown, I hope you now have a basic understanding of the algorithm framework inside of QuantConnect. To help facilitate this knowledge and to see it in action, let's create an actual trading algorithm that uses this algorithm framework. In the coding part of this video, we will implement an alpha model and a universe selection model. For the other model, such as the execution model and portfolio construction model, we will use some of QuantConnect's pre-built classes. However, before we actually start coding, let me quickly outline what strategy we will be implementing. The goal of the algorithm that we will implement will be to find high quality companies to invest in for the longer term. To identify quality stocks, we will use different quality criteria. Firstly, we will filter out stocks that have IPOed more than five years ago since we want the companies to have proven themselves by existing for some time. Next up, we want to diversify our holdings by investing in different sectors. For this algorithm, we will focus on the sector's financial services, real estate, healthcare, utilities and technology. But feel free to try out different variations. We then consider the factors price to earnings ratio, profit margin and return on equity. For each sector, we rank all companies in that given sector by these factors and establish a long position in the top 20% of the companies in each sector. Since the investment philosophy of this algorithm focuses on the long run, we only rebalance our portfolio on a quarterly basis. So summed up, this algorithm will rank US stocks in the previously mentioned sectors by some quality factor, once a quarter and invest in the top 20% of each sector. That said, you now hopefully understand what the bot is supposed to do, so let's now head over to QuantConnect and actually start writing some code. Inside of QuantConnect, we will go to the lab tab and create a new blank algorithm. As always, we will start by implementing the initialize method. But unlike in the previous videos, we will not use the on-data method since we will be using the algorithm framework instead. As usual, we will begin by setting the time frame and the starting cash balance for the back test. For this, I will just go with the year 2020 and $100,000. Note that this algorithm might have trouble with a too low balance since it wants to invest in a bunch of large cap US equities. Thereafter, we will create two helper variables, namely self.month, which we will use for keeping track of rebalancing times and self.nom course, which will be the number of stocks that will be considered in our course universe selection filter. I will initialize this variable to be 500. Then we set the resolution of the universe to daily, which we can do by accessing the universe settings property. Now, it's already time to add our universe by specifying the course and fine filters that we will use for this universe selection process. However, before we actually implement these filters, let me quickly implement a rebalancing function that will keep track of when to rebalance the universe and the portfolio for this algorithm. We will call this function is rebalance due and it will return the time to rebalance. As a parameter, it takes a time variable. Since we want to rebalance on a quarterly basis, we want to rebalance in the first, fourth, seventh and tenth months of every year. That's why we test whether the current month has passed or the current month is not one of these. If that's the case, we just return an empty object since it's not time to rebalance. If this is not the case, we set the self.month help variable to the current month and return the submitted time which will tell the other function that it is time to rebalance. With that, let's now move on to implementing the course universe selection filter. All we want to do here is check whether it's time to rebalance and if it is, we want to return a list of the 500 most liquid US stocks with a price above $5. To check whether it's time to rebalance, we use the just coded rebalancing check function. If it's not time, we just return the unchanged universe. Otherwise, we sort the list of available securities by dollar volume. Here, we also check whether the security has fundamental data and that its price is above $5. Thereafter, we return the symbols of the first 500 securities in this sorted list. This list of symbols will now be passed on to the fine universe selection filter which we will implement next. Here we want to find those securities that are in one of the desired sectors. For that, let's first quickly create a list containing the sector codes for financial securities, real estate, healthcare, utilities and technology. We can use Morningstar sector codes for this. Now, we once again create a list containing the symbols of the desired securities. However, we only add those symbols of the securities that I peeled more than 5 years ago and that are in one of the desired sectors. Furthermore, we test that there is a non-zero value for the fundamental factors, return on equity, net profit margin and price to earnings ratio. Last but not least, we return this filtered fine list. Next up, we will implement an alpha model for this algorithm which will take this filtered fine universe of stocks and rank them according to the just mentioned factors and generate buy insights for the top 20% of each sector. To add an alpha model, we will go back to the initialized function and use the helper method add alpha. Here we will add an instance of the alpha model fundamental factor alpha model. Note that this model does not exist yet, so what we will do is create a new file, name it alpha model dot pi and create the fundamental factor alpha model class inheriting from the alpha model class here. However, before we actually start implementing this class, let's first go back to the initialized method of the main file and import the alpha model file so that we can use this class. Furthermore, let's quickly finish the initialized method by adding the remaining models. First, we will add QuantConnect ready to use equal weighting portfolio construction model which will allocate an equal amount of capital to all of the insights it receives from an alpha model. As an argument, we passed the rebalancing function which it will use to find out when it is time to rebalance the portfolio. Next up, we set the risk management model to the null risk management model which basically does nothing. Alternatively, you could use one of QuantConnect's other pre-made risk models such as the trailing stop-wiz model or even implement your own as an exercise. Last but not least, we set the execution model to the immediate execution model which will always send out trade orders as soon as the portfolio construction model sends a trade signal. With that, we can now head back to the alpha model file and start implementing the alpha model of this algorithm which is where the actual trade decision making will take place. Just like for the main class, we will also use the initialised method for the alpha model. However, here we will just use Python's standard init constructor. In this constructor, we will create two class variables. One will be the rebalance time which we will use for rebalancing purposes and the other is a dictionary that will keep track of the securities in each sector. Here it is important to understand what the self dot sector's dictionary looks like since we will be using it a lot in the alpha model. The keys for this dictionary will be the sectors that we specified earlier. The value saved under such a key will be a Python set containing the stocks that are in the given sector. Besides an initialised method, an alpha model always has an update and onsecurities changed method. The update method is called every time there is new information for the alpha model and it should return a list of insights that will be passed on through the portfolio construction model. The unsecurities changed method is called ifsecurities are added or removed from the universe which is in our case should happen in about one secorture. We will start by implementing the unsecurities changed method. In this method, we will first of all iterate through the securities that were just removed from our universe. For each of these securities, we now want to remove those securities from the self dot sector's dictionary sets. To do so, we iterate over all sectors and check whether the given security is saved under that sector and if it is, we remove it. Next up, we basically do the exact opposite for those securities that were just added to the universe. We iterate over those and look at the sector that the given security trades in. If that sector is not yet in the self dot sector's dictionary, we add it to the dictionary with an empty set as the value. Then we add the security to the set that is saved under that sector. As you can see, the unsecurities changed method now simply takes care of keeping the self dot sector's dictionary up to date with the actual universe of stocks that are in our universe. Next, we will implement the update method to actually rank these stocks and send out buy insights for the top 20% of each sector. The first thing that we do in the update method is check whether it's time to rebalance. If it's not, we simply return an empty list since we do not want to send out any new insights. Otherwise, we set the rebalancing time to the end of the current quarter. Note that we have to use algorithm dot time to access the time since only the main QC algorithm class has this property. After that, we create an empty list and save it into the local insights variable. Now it's time to iterate over all the sectors in our self dot sectors dictionary and rank all the stocks in them. First, we sort the securities in that sector by return on equity, by profit margin and by price to earnings ratio. Since the higher return on equity and high profit margins are good, we set the reverse flag to be true for these two. On the other hand, we want to prioritize stocks with a low price to earnings ratio since we consider these to be relatively cheap, which is why we set the reverse to be false for the PE ratio. To calculate the scores, we create a dictionary named scores. We then iterate over the securities in the current e-considered sector and sum up the positions of that security in each of the sorted factor lists. So for example, if a stock would have the highest margins, highest return on equity and you would have the lowest PE ratio, it would be first place in all of them and receive the lowest and best score. Now that we have a score for each security in one sector, we want to find the best 20% of them. For this, we check how many elements there are in the scores dictionary so that we find out how many stocks are 20% in that sector. Note however, due to diversification purposes, we want this to be at least one. To say for example, there only would be two stocks in the current sector, the top 20% would be rounded down to be no stocks. But in that case, we would still want to invest in the first stock. With that, we return the scores dictionary into a key value pairs list and iterate over the 20% best ones. For each of these, we append an inside object to our insides list. Since we're just interested in price insights, we use insides.price. As arguments, we first specify the symbol of the security, then the expiration of the inside, namely the end of the current culture, and finally the direction in our case is up. We did all this in a for loop of all the sectors, which means that after this outer for loop is done, the insides list should have all the desired insides. This means all that's left to do is return this insides list. And with that, we're now done implementing this algorithm. So let's now build and backtest it to view its performance over the backtest time frame. As you can see, this bot managed to generate a positive return in the year 2020, however, not without quite the volatility during the early 2020 market decline. If we look at the benchmark chart of the S&P 500, we can see that this strategy seems to be highly correlated to the S&P 500. This should not be very surprising since they both invest in large-cap US equities. Since this bot trades a long-only strategy, it will be heavily affected by market downturns. If you want to analyze this performance report in more detail or add your own variations to this strategy, you can clone this code using the link in the description box below. Otherwise, I hope that this video gave you a good overview of the algorithm framework. I know it can be a little overwhelming at first, but once you get used to it, it has a multitude of benefits. That said, hopefully you are enjoying this video series. If you are, make sure to smash the like button, subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/16-data-driven-research.md b/course/16-data-driven-research.md new file mode 100644 index 0000000..1107a89 --- /dev/null +++ b/course/16-data-driven-research.md @@ -0,0 +1,23 @@ +# 16 · Data-Driven Research（研究环境 · QuantBook） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Data-Driven Research（研究环境 · QuantBook） +- **时长**：21 分 52 秒 ｜ **原视频**：https://youtu.be/XUib6Y3eePs +- **本地视频**：[videos/16-data-driven-research.mp4](videos/16-data-driven-research.mp4) + +## 🎯 本集要点（中文导读） + +本集讲 **Data-Driven Research（研究环境）**——写策略前的"研究/探索"环节。 + +- **Research Environment（研究环境）**：QuantConnect 内建 **Jupyter Notebook**，用于交互式探索数据、可视化、找相关性、建模型。 +- **QuantBook 类**：研究环境里不能用 `QCAlgorithm`，改用 `QuantBook`（是 QCAlgorithm 的包装）——可访问历史数据、consolidate、画图、建指标等；但**不能用事件驱动方法**（OnData、universe 事件处理器）。 +- **数据**：以 **Pandas DataFrame** 组织（带标签的表格）；可访问股票/期货/期权/外汇/加密的历史与另类数据。 +- **典型用法**：探索数据 → 找相关/因子 → 建 **机器学习/统计模型**（本集演示了一个**线性回归**示例）。 +- **把研究转成算法**：把 QuantBook 代码**直接导入/复制**到 QCAlgorithm（很多方法通用，通常把 `qb` 换成 `self`）。 +- ⚠️ 提醒：线性回归只能捕捉**线性关系**；示例数据量小，不代表样本多就一定更好。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to the next video of this Python algorithmic trading course. So far the focus of this video course was on the implementation side of the algorithmic development process. However, often you would want to perform certain research before actually coding out to strategy and backtesting it. During this research you could explore the available data, visualize it, analyze it for any potential correlations and even create advanced models to get a good idea of what kind of strategy you could use in the given sector. For this, QuantConnect has a built-in research environment that uses Jupyter Notebooks so that you can analyze data in an interactive manner. In this video we will create a new QuantConnect project and explore this research environment by going through some of the available features in an example research notebook. Typically the data inside of the research environment will be structured into Pandas Data frames. Pandas is a data analysis library for Python that basically allows you to structure data into tables and easily access and manipulate that data. These tables are so-called Pandas Data frames and you can think of them as labeled spread sheets. If you have never used Pandas before it might be a good idea to watch some introductory videos on how to use it since I will not cover all the Pandas basics in this video. That being said, before we head over to QuantConnect and look at the research environment in action let me quickly outline some of the theory relevant to the research notebooks. First and foremost, since you aren't coding actual trading algorithms in the research environment you can't use the QC algorithm class. Instead you have to use the so-called Quantbook class which is a wrapper around the QC algorithm class. This means that the Quantbook instance will allow you to access methods of the QC algorithm class inside of the research environment plus more. However, note that since you aren't actually back testing anything per se you can't use event driven methods such as on data or universe selection event handlers. But you can access all the historical data, consolidate it, visualize it with charts, create indicators and much more. QuantConnect allows you to access and analyze historical and alternative data for equities, functions, futures, forex and crypto. Another use case of the research environment is to develop machine learning or other advanced data science models that you can then use for your actual trading algorithms. After sufficiently analyzing the data in the research environment you would want to translate the Quantbook based research code into QC algorithm based code that can actually trade for you. For this you can actually import certain classes, models and methods from the research notebook directly into your QC algorithm. Furthermore, many methods can be directly copied into the algorithm since the Quantbook simply is a wrapper class. That being said, let us now actually go over to QuantConnect and create our first research notebook. To do so we have to create a new algorithm project like we always do. You can do this from the lab tab. However, instead of actually editing the main file of the project, we now click on the research notebook file in the sidebar. This will open the research environment including a few sample lines of code that we will remove for now. Note that it can be a good idea to code along and play around with the research environment yourself to better understand the concepts presented in this video. However, you can also clone the finished notebook of this video using a link that I'll put in the description box below. That said, the first thing we do here is import two libraries, namely NumPy S&P and Matplotlib.pyplot as PLT. We will use NumPy for calculation purposes and Pyplot for plotting. To execute a code block you can hold shift and press enter. If you're not yet familiar with Jupiter notebooks, you can click on the help button at the top for more useful shortcuts. Next up, we create an instance of the Quantbook class and save it to the variable QB. Since we now have a Quantbook instance, we can start adding data for the securities that we want to analyze. For this example notebook, we will focus on seven financial stocks, namely JP Morgan, Bank of America, Morgan Stanley, Schwab, Goldman Sachs, American Express, and Citigroup. To add these stocks to the Quantbook, we can use the usual add equity method. The same goes for other asset classes. We save the symbol object of each of these securities into the symbols list. Now we're interested in looking at the price to earnings ratio of the just added securities and its relationship to the price. So firstly, let's visualize how the PE ratio of these stocks has changed over the course of 2021. For this, we can use get fundamental and pass the symbols list, PE ratio, and a time frame as arguments to get the data. This method will return a Pandas data frame with a PE ratio over the specified time frame for each of our symbols. Note that in this data frame, you might notice that the symbols of some of the stocks deviate from the ticker that we specified earlier. For instance, the Bank of America symbol starts with NB. In contrast to tickers, QuantConnect symbols are constant and always refer to the underlying entity regardless of rebranding or name changes. However, to better understand the data, we will rename the columns to the actual current company names. Note that here it is important to respect the order of the columns in the data frame. To actually plot the PE ratios, we can use matplotlib. Before plotting it, however, I will quickly add some appearance parameters to increase the size of the chart and add some labels. As you can see, most PE ratios stayed relatively constant throughout the year. The biggest change was in America Express' PE ratio, which dropped from over 30 to under 20. Next, let's look at and compare the average PE ratio of these stocks throughout 2021. For this, we simply take the mean and then sort the values. Here you can see that Goldman Sachs had the lowest PE ratio closely followed by Citigroup. Schwab was the stock with the highest mean PE ratio of almost 30. Another interesting aspect to consider is the relationship between the returns of each stock in 2021 and the respective PE ratio. However, before we can look at that, let's first look at the returns that each of these stocks generated in 2021. To calculate these returns, we first need to look at the price data for each of these stocks. We can access the price history using the history method and passing the symbols list, time frame and resolution. This will return a multi-index pandas data frame with the open, close, high, low and volume history for each of these stocks. But since we are only interested in the close price, we only save that information into the history variable. We once again rename the columns for better readability and then output the first five columns of this new data frame using the head method. With this data frame, we can now easily plot the returns of these stocks using the percent change method. Since we always need two values to calculate the percentage change, we have to start at the second row of the data frame. Once again, we use matplotlib for the plotting. As you can see, there definitely seems to be some form of correlation between these stocks which shouldn't be too surprising since they're all in the same sector. Nonetheless, there is some deviation across their returns with Schwab being the best returning stock and Citigroup the worst one in 2021. Now we want to analyze the correlation between these 2021 yearly returns and the average PE ratio of these stocks. To calculate the correlation coefficient, we can use one of NumPy's methods and pass the last row of the returns data frame and the average PE ratio list as arguments. Here we can see that there seems to be a slight positive correlation between these two. In other words, according to this, granted very limited data, a higher average PE ratio led to higher returns for these stocks in 2021. A high PE ratio means that the company shares cost a lot compared to their earnings. There could be multiple reasons why this might be the case. We can also visualize this relationship with a scatterplot which is exactly what I will do next. Note that this is just to give you some examples of how you can use the research environment. These aren't very meaningful findings, especially since we are only considering a very insignificant number of stocks over one arbitrary time period. In the next step, let's take a look at how we can add data for more complex securities such as options. I will quickly demonstrate this using Bank of America as the underlying security for the options. Just like in standard QC algorithms, we can use Add Option and Set Filter to Add Options to the Quantbook. The Set Filter method takes the number of strikes that you want to consider above and below the current price as well as the time frame until expiration as arguments. Here I will go with a simple filter of a 10 point wide strike range and 20 to 50 days until expiration. To actually get historical data for the options over a certain time period, you'll have to use the GetOptionHistory method. You then have to pass the Request symbol as well as the time frame that you're interested in. Here it is important to understand that this will actually not directly return a data frame. Instead, it will return an Option History object. This Option History object has a bunch of helper methods that you can use to filter the option chains. For instance, you can use Get Strikes and Get Expiry dates to access the available strikes and expiration dates for the requested time frame. To access the actual historical data, you'll have to use the GetAllData function. This will then return a data frame containing information such as open, close, low, high and volume among others for all the available options. Now you can visualize and analyze this data to whatever extent you want to. However, instead of doing that now, let me rather show you some other useful aspects of the research environment. So let's instead look at how you can add, use and visualize indicators here. I will demonstrate this by plotting a BollingerBand indicator on Bank of America stock. But this works similar for other indicators. First, we need to create an instance of a BollingerBand object. We can do this with this constructor. As arguments, we pass the number of days that the moving average of the BollingerBands should take into account as well as how many standard deviations the outer bands should deviate from the mean price. Here I will go with a 30-day moving average and two standard deviations. Next, we get historical data for this indicator by applying it to Bank of America over the past 360 trading days. Note that here we specify that we want the BollingerBand to be created from the open price instead of the default closing price option. Last but not least, we use Matplotlib to plot this indicator at all its related data. Here you can see that besides the price and the upper, lower and middle band, we also plotted other metrics such as the band width. But since we only are interested in the actual indicator plots, we want to drop the columns referring to these other metrics. Thereafter, we create the plot again and get a less cluttered view of the actual indicator bands. Now you could further analyze the indicator data by for example putting it into relation with the underlying price. But once again, I will not do that here. Besides the rather simple analysis tool shown so far, the research environment also supports countless advanced Python data science and machine learning libraries. So it's very much possible to develop complex machine learning models and then import and use these in your actual algorithms. Building such a model is outside of the scope of this video. Nonetheless I want to give you a taste of the potential. That's why we will create a very simple linear regression model that attempts to predict stock prices based on the BollingerBand SMA. For this, we will use the last 60 days of 2021 of Bank America's Price and BollingerBand data to train and test this regression model. Furthermore, we will use scikit-learn which is a Python statistics library to create the linear regression model. To do this, we have to add the historical price data for Bank of America which we can once again do with the history method. Since we are only interested in the closing price, we can refactor this data frame and only save the daily closing prices into a list. The idea behind this very simple model will be that it will get the value of the middle band of the BollingerBand indicators the input and it will then try to predict the closing price purely based on that input. Note that the middle band basically just is a 30 day moving average of the daily opening prices. But to do this, we first need to divide our 60 days of data into training and test data. We will use the first 30 days as training data and the last 30 days to test our model. We can train a linear regression model using scikit's fit method. However, before we can do that, we need to bring the data into the right format which is exactly what we are doing here. After fitting the model to the data, we want to use the model to predict the last 30 days of the closing prices so that we can evaluate the model's accuracy. We can do this using the dot predict method. However, before that, we once again have to bring the data into the right format. To actually evaluate the accuracy, we will plot the training and testing data on a sketch plot. Since we want to be able to differentiate between the training and testing data, we will call the training data blue and the testing data green. Furthermore, we will plot the linear regression line in red. The x axis represents the middle band of the BollingerBand indicator while the y axis represents the closing price. Let me now quickly explain what this plot shows since you might not be very familiar with linear regression models. A linear regression model tries to find a linear relationship between two variables. In this case, the variables are the indicator value and the closing price. The red line represents this model. So what the model asserts is that the higher the middle band's value is, the higher the closing price of the stock will be for that day. To give you an example, it says that if the value of the middle band is 44, it predicts the closing price to be 46. If we look at the blue dots, we can actually see that they are somewhat aligned with the red line, which is because the model used these data points to create that line. However, if we look at the green dots that represent the testing data, we can see that the red line does a very bad job predicting these closing prices. This should not be too surprising since one simple moving average value is not enough to accurately predict the closing price of a stock. A stock can even have the exact same indicator value in the morning on different days and still close at very different prices. So unsurprisingly, that does not seem to be a significant linear relationship between the daily 30 day moving average values and daily closing prices. Also note that this is a very simple example. We only looked at 30 days of data for this model, which isn't exactly a lot. Nevertheless, that does not mean that using more data would lead to a more useful model in this specific case. In general, it is also important to understand that linear regression models can only be used to find linear relationships between variables. It cannot be used to find more complex relationships. Nevertheless, such models can be very useful in some scenarios. I hope that this brief example gave you a good overview of the potential that the research environment offers for building your own models and all the data that you can use to build such models. Besides the data presented in this sample notebook, you can also add data for a bunch of other asset types, alternative data and much more. For more details on this, I recommend checking out the docs. If you want to copy the sample notebook that I created in this video, you can clone this code by using a link in the description box below. For even more sample notebooks, make sure to check out the documentation pages as well as the community forum. Once you're done with your research, you can also easily convert the code from a research notebook into code supported by the QC algorithm. Some functions you can even directly copy or import into your algorithms. For the helper methods, you can often just replace the QB quantbook identifier with a self-pilot Python class identifier. With that being said, I really hope you enjoyed this video and learned a lot. If you did, make sure to smash the like button and subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/17-bitcoin-ml-bot.md b/course/17-bitcoin-ml-bot.md new file mode 100644 index 0000000..6aaa662 --- /dev/null +++ b/course/17-bitcoin-ml-bot.md @@ -0,0 +1,24 @@ +# 17 · Bitcoin ML Bot（用神经网络做比特币） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：Bitcoin ML Bot（用神经网络做比特币） +- **时长**：31 分 7 秒 ｜ **原视频**：https://youtu.be/waiBgdalmSE +- **本地视频**：[videos/17-bitcoin-ml-bot.mp4](videos/17-bitcoin-ml-bot.mp4) + +## 🎯 本集要点（中文导读） + +**用机器学习（神经网络）做比特币交易**（思路同样适用于其它标）。 + +- **思路**：用**神经网络**输入"过去 30 天的价格+成交量"，预测"下一根收盘价会高于还是低于当前收盘价"，据此**做多或做空**。 +- **神经网络简介**：输入层 → 若干隐藏层（每个神经元做加权计算 + 激活函数）→ 输出层；通过训练数据调整**权重**来学习。 +- **工具**：Python 的 **TensorFlow + Keras** 搭建网络。 +- ⚠️ 作者强调的关键点： + - 用 AI/神经网络时，最好把它当**辅助工具**（优化现有流程、作为策略补充），**不要只依赖网络预测**。 + - **严防过拟合**：本集用 2020–2022 训练，在这一区间回测"必然好看"（模型见过），不能代表真实表现。 + - 神经网络还可用于预测：波动率、长期趋势、成交量、与其他资产的相关性、指标值等。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome to another video. In this video you will learn how to create a cryptocurrency trading algorithm that uses machine learning techniques to make trading decisions. Even though I will use Bitcoin as the underlying asset in this video, note that you can also create such a bot for other cryptocurrencies, stocks, forex or whatever you want to trade. We will code this trading bot using Python and the QuantConnect platform. Note that you can follow along and do everything that I do in this video with a free QuantConnect account. If you haven't already, you can create your free account using the link in the description box below. If you are new to QuantConnect, don't worry. I will go over everything step by step in this video. However, if you want to become an algorithmic trader yourself and learn how to develop your own trading bots, you could check out my free algorithmic trading course on my channel. Before we will head over to QuantConnect and actually start coding, let me quickly outline what we will do in this video on a conceptual level. Like I already mentioned, the strategy that we will be implementing will make trading decisions based on artificial intelligence techniques. More specifically, we will create a neural network that takes the price and volume information of the last 30 days of Bitcoin as the input and then tries to predict whether the next closing price will be above or below the last closing price. Based on this prediction, our trading bot will then either buy or set a Bitcoin coin. If you're not familiar with neural networks, three blue one brown has a great introductory video on them. But in very simple terms, a neural network is a weighted graph that consists of a bunch of nodes that are called neurons. The first layer of a neural network is the input layer. And this is where you can pass what you want the network to use as an input. In our case, this would be where we pass the 30 day price and volume data. The last layer is the output layer, and we will use the value output here as our prediction for the direction. In between the input and output layer, they usually are a variable number of hidden layers. Each neuron applies a certain function to the input it receives, and then outputs their respective results based on a so-called activation function. Depending on where this neuron is, that output will then be used as the input of another neuron or as the final output. The connection between the neurons are labeled with certain weights that are used for the calculations in each neuron. Through certain techniques, you can train a neural network by feeding it a bunch of training data and then adjusting the weights based on the accuracy of its predictions. Depending on the problem, the predictions thereby become more and more accurate and sophisticated. For now, you can just think of this as a black box that spits out a certain output based on what you give it as an input. We will be creating our own neural network using the TensorFlow and Keras modules for Python. With that being said, let us now head over to QuantConnect and start actually writing some code. Note that for the best learning effect, I highly recommend coding along. However, I will also post a link in the description box below that allows you to clone the entire code from this video for you to play with after watching this video. Inside of QuantConnect, we will go to the lab tab and create a new algorithm from there. This will bring you to a screen like this with a bare-bones template for a new trading algorithm. Before we code out the actual algorithm though, we will first create our neural network in QuantConnect's research environment which uses Jupyter notebooks. The first thing we do in this research notebook is import all the required dependencies that we will use. This mainly includes some things from TensorFlow and Keras. You can run the code in a given code cell by clicking Shift and Enter. After we are done with our imports, let's save the start and end dates that we will use for the training and testing data for the neural network that we will create. For this video, I will use two years from 2020 until 2022 but feel free to try out different time periods. Now we will initialize an instance of the QuantBook which is QuantConnect's research environment class that gives you access to all of QuantConnect's research API methods. The first of these methods that we will use is AddCrypto which will add the data for Bitcoin over the chosen time period. To actually access the specific historical price data, we use the history method which will return a data frame consisting of Bitcoin's price data over the last two years. For our neural network, we will not use the raw price data though. Instead, we will use the percentage change of the open, high, low, close and volume data of every day since we just care about the changes in this data and not the actual absolute values. Before recording this video, I took a look at this data and saw that there are some problems in the volume column. Some of the values got assigned infinity as a value. Since we can't use this for any calculations, we will place the occurrence of infinity with a max value in the volume column. Now that we've cleaned up the data frame, we have to bring the data into the correct format so that we can actually use it to train a neural network. For this, we create some variables. The Features list is basically the input list and the labels list is the desired output. When we later actually use this neural network, we won't yet know what the desired output would be since we are trying to predict it. But when training it, we will use the labels list to improve its accuracy. So each element in the Features list should consist of the open, high, low, close and volume values for each of the last 30 days. An element from the labels list on the other hand will be one if the price was up on that date and zero if it was down. Before we can now actually use these lists for the neural network, we need to turn them into non-pay arrays since that is a required input format for the tensorflow method that we will use. Next up, let's divide this data into training and testing data. We will use this training data to actually train and improve the neural network and the testing data to test it. Here it is very important to not mix the two up. It is very easy to create a model that performs well on training data since it can just remember the desired output and overfit itself to that data. However, in that case, the model would not perform very well on new data that it has not seen before. And that's exactly what we will use the test data for, to make sure that it is not too overfit to the training data. So let's use 70% of the two-year data for training and the rest for testing. A common pitfall here is that the output of the training data is not evenly distributed. If for example, 80% of the days that we use to train the model with are up days, an easy way for the model to achieve an 80% accuracy is by just always predicting that a day will be an update. However, that is not a very intelligent guess and that won't work on other data. That's why we check the ratio of up to down days here. As you can see in the training data, almost 56% of the days are up days. Ideally, this would be as close to 50% as possible. So let's check if this improves if we consider the last part of the two-year time frame instead of the first part. As you can see here, this actually leads to a slightly more even distribution of about 53% updates. This is still not optimal, but I leave it for now. Now we can finally start building the actual neural network. For this, we will use Keras sequential model, which basically allows us to build a network layer by layer. We will start with a density-connected layer as the input layer for our neural network. For this layer, we will specify the shape of the training data as the input shape so that the network knows what input to expect. As I briefly mentioned earlier, each neuron in a neural network has an activation function that decides what a neuron should output. If you want the output to be binary, like in the case of a classification problem, a common activation function is a step or sigmoid function since it transforms the input to a value between 0 and 1. Another very common activation function is the real-you activation function which we will use here. Next up, we will create another dense layer with the same function. After that, we want to output our result in one last neuron, but for this, we first need to add a flat layer that flattens the input data. Then we can create the last layer with only one output. Since we here want a value between 0 and 1 to encode the predicted direction, we will use the sigmoid function. If you want a full tutorial on how to apply Python, TensorFlow and Keras for deep learning, you could check out a great tutorial series by CentDex that I linked in the description box below. Note that finding the optimal number and types of layers and connections for a specific neural network depends on the problem and is still a heavily researched problem. So here it often helps to play around a little to see what yields good results. After defining the neural net, we can now configure the model for training. Here we define a loss function, an optimizer, as well as the metric we want to monitor when training the network. The binary cross entropy loss function is a common choice for binary classification problems like this one. And the addm optimizer is probably one of the most used optimizers as of right now. Remember that when optimizing a model to always pay attention to what metric you're optimizing for, since otherwise your model might be trying to achieve something that wasn't actually a intended goal. Now we can finally fit our training data to this neural network to actually train it. Usually you go through the entire training data multiple times when fitting neural networks. Each of these cycles is called an epoch. The more epochs you run, the more the neural network will adjust itself to the training data. However, here it is important not to overfit the model by letting it run for too long. We will let it run for five epochs for now. While it runs, we can actually see how the accuracy and whatever other metrics we configured above change for each training cycle. As you can see here, the prediction accuracy for the training data seems to be steadily increasing for each epoch we let it run. Note that these values will be different every time you run the fitting since there are some probabilistic processes involved. So if you're coding along, keep in mind that the neural network you will get probably will not be the same as the one in this video. Now that the model is trained, we want to test its performance using the testing data that it has not yet seen. For this, we let it run an X test and then format the data so that we can plot it on a graph. On this graph, the blue line represents whether a given day actually was an update or not, while the orange line symbolizes the prediction of the network. If the orange line is above 0.5, we consider it to have predicted an update while below 0.5 means it thinks the day will end in the red. If we were to increase the number of epochs that we use to train our model, we would see the orange plot edge and closer and closer to the blue plot. But this would likely not improve the model's accuracy on the general data but just overfitted to the training data. Even though this graph gives us a good general picture, we can't actually use it to analyze its accuracy very well. So let's instead output the model's accuracy and error first on the training data and then on the testing data so that we can compare the values with each other. As you can see, the accuracy for the test data is lower than the accuracy for the training data. However, the difference isn't that big so I will just leave the neural network as it is. But as you can see, even with 30 days of price and volume data, this network does not manage to accurately predict Bitcoin's daily price direction much more than 50% of the time, which just shows how hard it is to do this. Like I said, I will leave it as it is right now for the rest of this video, but I highly recommend for you to code along or clone this notebook using the link below and play around with the network yourself to see if you can manage to improve it without overfitting it. Now we actually want to use this model that we just created inside of a QuantConnect algorithm to make trading decisions based on the predictions. However, to do so, we first save the model as an object into QuantConnect so called object store, which allows you to save models such as this one so that you can access and use them from wherever you want to. To accomplish this, we turn the model into a JSON object and then save it into the object store using the key Bitcoin price predictor. Let me now show you how we can access this model directly from the object store. I will show you how to do this in the research notebook, but we can then basically use the exact same code inside of the trading algorithm that we will create in a few minutes. To get the model from the object store, we first check where that exists. If it does, we read it in as a string by importing it using dot read. Thereafter, we turn it into a JSON object and then turn it into the neural network we want using Keras sequential class. To test if everything is working as intended, let us get the prediction for today's Bitcoin price direction. For that, we access the past 40 days of price history once again using the history method. To actually use this data to get the prediction from the model, we now just have to turn it into the right input format. So let's focus on the open, high, low, close and volume columns and get their percentage change values. Then we take the last 30 rows, add them to a list and wrap an umpire array around it. With that, we can now use this as the input data for our neural network and check whether the rounded output equals zero or one. Note that the output of the prediction is a double nested array, which is why we first have to index it. With these few lines of code, we can now get the prediction for any date that we want to using the model that we created. So let us now copy this code and use it to create an actual trading bot. For this, we go back to the main.py file. Here, we once again need to import the tensorflow.keros sequential object as well as the JSON library. The first thing we do after that is implement the initialize method, which is used to initialize our trading algorithm class. Here, we for instance set the backtest time frame for this algorithm. I will go with 2018 until 2020. Here it is important to keep in mind that if you use the year 2021 to backtest this algorithm, you might get quite biased results since this is the time frame we used to train the neural network with. After setting the backtest time frame, we can copy paste the code from the research notebook to import the actual model. Note that the quantbook class is a wrapper class around the quant connect algorithm class, which is why it is important to use self instead of the QB quantbook identifier for some of the methods. In the next step, we will set the brokerage model so that we can buy and sell Bitcoin however we want to. For this, we choose bid finics as the exchange and a margin account as the account type. Then we set the starting cash balance for this backtest. Here I will go with 100,000 USD but feel free to enter a different value here. Next up, we need to add the actual cryptocurrency to this algorithm so that we can trade it. We do this with add crypto. Quant connect supports data in many different resolutions, even including minutely resolution, but since we are only interested in the daily data, we will go with daily resolution. Last but not least, we set bitcoin to be the benchmark for this algorithm. This will allow us to later compare this strategy's performance versus just investing in Bitcoin. Now we are done with the initialize method. Next up we will implement a method that will allow us to get the prediction for the next day using the model we trained in the previous part of this video. For this, we can basically just copy the code from the research notebook that transforms the data into the right shape and once again replace the QB identifier with the Python self class identifier. So here the history method gets the past 40 days of data and then it is transformed into the right input format for the model. Then we check whether the model's output for that input is closer to one or zero and then return it down or up respectively. Note that since we will call this method every day in this algorithm, this is actually not a very efficient way to implement this. The history request will always have to fetch 40 days of data every day, even though we already fetched 39 of the same days on the day before. A more efficient way to accomplish this would be to use a rolling window that stores the intermediate data, but I will not be doing that here. If you want to learn how to do this and all the details of developing algorithms in Quant connect, you should check out my free algorithmic trading course. A possible variation of this algorithm would be to use the actual output values of the neural network instead of the rounded counterpart. You could then interpret values closer to one or closer to zero as more confident predictions than values closer to 0.5. You could then, for instance, reflect this confidence level by adjusting the position size, but we will just stick with the rounded values for now. All that's left to do now is implement the on data method, which is called every time the algorithm receives new data, which should be daily since we added data at a daily resolution. So every time this is called, we want to get the prediction of the model using the method that we just implemented. If that returns up, we want to set our holdings in Bitcoin to 100%, which is equivalent to buying as much of it as we can. Otherwise, if the model predicts down, we want a short Bitcoin. But due to the high risk nature of shorting and volatility in crypto, we only set this to 50% of our available capital. Alternatively, you could also decide to just go flat in this case. With that, we can now click on Build and Backtest to find out how this algorithm performs over the given time frame. This will start the backtest, which should be done in a few seconds to a minute. When the backtest is done, a performance report like this one will be generated that allows you to analyze the strategy's performance. If you scroll down, you can also find more performance metrics, as well as the option to view every order that was sent by the algorithms separately. This is great to check if the algorithm is actually doing what it's supposed to do. If we look at the equity chart at the top, we can see that this bot did not seem to perform that well over the chosen backtest time frame. However, if we look at the benchmark chart of Bitcoin's price, we can also see that Bitcoin itself didn't exactly perform that well during 2018 until 2020. Nonetheless, our algorithm's performance definitely could use some improvements. However, the point of this video was not to create the best possible trading strategy, but instead to show you how easy it is to develop your own trading algorithms, even if you want to use more complex machine learning and AI models. Once you've developed and thoroughly tested an algorithm that you are satisfied with, QuantConnect allows you to connect the algorithm to your broker platform so that it can actually trade with real money. Besides applying a neural network such as the one presented in this video to Bitcoin, QuantConnect also allows you to do the exact same for almost all assets. In fact, you could easily clone the code from this video and simply use add equity instead of add crypto in the research notebook, as well as the algorithm itself to try if this works better for some stock or market ETF. Just now that you should also just remove the brokerage model line in that case. In general, a neural network probably works best for more predictable stable assets, which are highly volatile cryptocurrency probably isn't. Here it is best to find a period that is as representative of that asset's general price behavior as possible. Finding such a period is a challenge in itself though. That being said, short term price changes such as daily changes are notoriously hard to predict across all asset classes since the data is very noisy. Therefore, it generally makes more sense to focus on more stable variables when using AI techniques such as neural networks. Besides using a neural network to predict the direction, there are many more use cases where you could apply such a technique. Possible examples include predicting the upcoming volatility, for instance by using the percentage change as the output, trying to predict the longer term trends, the volume, correlation to other assets, indicator values and more. But in general, it is a good idea to think about AI techniques such as neural networks more as a supplementary helping tool in your strategies instead of solely relying on the predictions of a neural network as your strategy. You can use them to improve and optimize existing processes and as additions to your strategy. Also, always make sure to pay attention not to overfit your models or to draw too many conclusions from the results that stem from the training data since the model already has seen this data. For example, in this video, we used the time frame 2020 up to 2022 to train the model. So the testing of our trading algorithm on this time frame will not necessarily yield representative results since the model knows this data already. That said, I really hope you enjoyed this video and learned a lot. To learn more about creating your own trading algorithms, check out my free algorithmic trading course in which you'll learn everything you need to know on how to develop your own trading algorithms including dozens of example trading bots and much more. If you enjoyed this video, make sure to smash the like button, subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/18-live-trading.md b/course/18-live-trading.md new file mode 100644 index 0000000..548a2ea --- /dev/null +++ b/course/18-live-trading.md @@ -0,0 +1,23 @@ +# 18 · How to Live Trade（实盘部署与 Paper Trading） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：How to Live Trade（实盘部署与 Paper Trading） +- **时长**：26 分 3 秒 ｜ **原视频**：https://youtu.be/-F3ITjfelrM +- **本地视频**：[videos/18-live-trading.mp4](videos/18-live-trading.mp4) + +## 🎯 本集要点（中文导读） + +**把算法部署到实盘（Live Trading）**，并演示 **Paper Trading**。 + +- **量化实盘选 QuantConnect 托管的好处**：同地机房服务器（数据更快、成交更快）、比自建更抗断电/断网。 +- **需要券商账户**：券商要支持算法交易（有 API）。QuantConnect 支持 10+ 家（Interactive Brokers、OANDA、Tradier、Binance 等）；作者推荐 **Interactive Brokers**（品种全、价优，支持 100+ 市场）。 +- **流程**：先充分回测 → **Paper trading（用实时数据的模拟盘）** → **小资金实盘** → 持续监控。 +- ⚠️ **回测 ≠ 实盘**：滑点、成交、佣金、做空券源、各类偏差都会让实盘偏离回测。 +- ⚠️ **风险**：算法依赖"数据源 + 与券商 API 的稳定连接"，任一中断算法就工作异常；要监控服务器/算法状态。 +- **不要"set and forget"**：尤其初期要主动盯着它是否按预期运行，但**也别频繁手动干预**。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to another video. In this video you will learn how to set up live algorithmic trading inside of QuantConnect by connecting your trading bot with your broker account. Since 2015, QuantConnect has hosted more than 200,000 live algorithms and over $20 billion in volume has been traded on their servers. Besides covering how to live trade with real money, I will also demonstrate how you can set up paper trading with live data in this video. If you are new to QuantConnect, I recommend first watching the first videos of my algorithmic trading course in which you will learn everything you need to know to start developing your own trading algorithms. If you do not want to host your algorithms using QuantConnect, you can still use their open source lean engine to develop and write your trading bots. However, then you would have to find your own hosting solutions. But in this video, we will focus on how to set up live trading using QuantConnect's hosting options since that is definitely the easiest option. Besides that, QuantConnect hosts the algorithms on co-located servers which means the servers receive data faster. This can lead to faster order executions and fills. In addition to that, it can better protect you against interference problems such as power outages or internet problems compared to hosting the algorithms on your own. That being said, before we head over to the QuantConnect platform and I develop a simple example bot and show you how you can deploy it to live trading, let me outline some general key points that you should be aware of when switching from backtesting to live trading. First and foremost, to be able to trade algorithm quickly, you need to have an account at a brokerage that supports algorithmic trading. This is usually the case if the broker offers an API for you to use and connect your algorithm with. As of the recording of this video, QuantConnect supports over 10 brokers including interactive brokers, Wanda, Tradia, Binance and more. However, they are continuously working on adding support for more brokers. You can check out the link in the description box to see which brokers are currently supported for which asset classes at the time you watch this. If you do not yet have a broker account with one of the supported brokers, in my opinion interactive brokers is a great choice since they are one of the biggest brokers worldwide. They offer stocks, options, futures, forex, bonds and funds on over 100 markets at competitive prices. To learn more and help out this channel, you can check out the referral link below. I will later demonstrate how to connect a broker platform using interactive brokers as an example, but this will be similar for any of the other supported platforms. Before deploying a trading algorithm to trade with real money, you will want to back test and analyze its performance thoroughly. However, here it is important to understand that back test results are not equivalent to live results under real world conditions. The reason for this is that when developing and back testing a strategy, you are basically trying to build the best possible model of reality. However, no matter how good this model is, it will still just be a model. That's why there is a difference in simulated back test and actual performance, even if they both cover the exact same time frame. Such differences can result from unrealistic filling conditions, fees, market impact, biases like overfitting or look ahead bias, differences in data, brokerage specific differences and more. The process of understanding this discrepancy is called reconciliation. And to analyze it, QuantConnect's live trading page displays the actual performance of your algorithm and a so called out-of-sample back test performance. The out-of-sample back test simply is a back test that was performed over the same period as that of the live trading performance results. Obviously, this can only be done after letting the algorithm run on live data for a while. The closer these two performance plots are to each other, the more realistic your back testing model of reality seems to be. This means that the performance resulting from back testing seems to be more indicative of actual performance. Besides looking at the chart for visual analysis, there are also different metrics you can use to measure the reconciliation. One such metric is the correlation between the returns generated by the back test and the actual returns. The closer to one this correlation is, the more correlated the returns are. Another metric that can be used to analyze the similarity of two sequences is the dynamic time warping distance or DTW in short. A DTW value close to zero means that the two sequences are more similar. For intuition, you can think of an annualized DTW value of 0.2, meaning that there is about a 20% deviation between the back test and actual performance over one year. Note that it is very hard to achieve a DTW value much lower than 20%. However, note that these two metrics are at the time of recording this not yet available for live trading. Instead, they are currently only available for alpha streams. For more details on these metrics, check out the documentation page below. Since back testing uses a theoretical model, it is always a good idea to first paper trade your algorithms on live data before deploying them with real money. QuantConnect offers multiple ways of doing this which I will demonstrate in a few minutes. Another important difference to understand between back testing and live trading lies in how your algorithm receives data. When back testing, you will usually use QuantConnect's data feed which gives you algorithm standardized data at a very specific point in time. However, when trading based on live data, you cannot always decide at what exact time you receive the data. Especially for custom or alternative data, you need to be aware that you can first use the data once it is available which might not always be at the same time. So far, we often checked if certain data was available at one specific point in time. This might work without any problems in back testing, but in reality, data might not be available at one specific point in time. Therefore, it often makes more sense to check for new data over span of time instead of at one particular time like you would in back testing. You can also use QuantConnect's data feed as the input for live trading. Alternatively, you can also use the data feed of one of the supported brokers like interactive brokers or trade a year. Just note that, to get access to real time data through IB, you will need to subscribe to one of their data feeds. Depending on where you get your data from, make sure to have a deep understanding of the data format, the related time zone and how and when your algorithm will receive this data. Last but not least, before we move on to actually deploying a live algorithm, let me outline the prerequisites for this. Besides having an algorithm that you would like to live trade, you will also need a live trading node to actually deploy your algorithm. This is the case, no matter if you want to trade with real money or just pay to trade your algorithm. Purchasing a subscription to a live trading server node will give you access to hosting your algorithm on one of QuantConnect's low latency co-located New York servers. QuantConnect currently offers four different live trading nodes that only differ in the amount of rum available. The cheapest node has half a gigabyte of RAM and costs $24 a month, while the most expensive node offers four gigabytes of RAM at a monthly cost of $96. Note that there is also a cheaper yearly subscription option. For most relatively simple algorithms, a cheaper node should be sufficient. But if you're planning on developing an algorithm with a huge universe or a memory intensive machine learning model, a more expensive node might be better suited. That being said, let me now quickly outline the strategy that we will use to demo the deployment process. Instead of writing an entirely new trading algorithm, we will actually use a slightly modified version of a strategy that we implemented in one of the previous videos. For a step-by-step walkthrough on how I created this bot, check out video 10 of my algorithmic trading course. There's a link in the description box below to that video. However, note that I made some slight code structure modifications to it, but the strategy itself basically stayed the same. But before we move on, let me still give you a quick high-level overview of the strategy that this bot follows. The algorithm follows a dynamic allocation strategy to QQQ and BND, which are two ETFs that track the top stocks on NASDAQ and the bond market respectively. Depending on the current trend of QQQQ, we either want a more aggressive allocation with 80% to QQQQ and 20% to the bond ETF or a more conservative 2080 allocation. The idea behind this is that bonds often are more stable than equities, especially during market downturns, which is why we switch to them in case of a downtrend. However, at the same time, equities have had larger historical returns, which is why we still want to capitalize on them. We identify the current trend using a 30 days simple moving average of QQQQ's price. If the average is below QQQ's price, we consider that to be an uptrend and vice versa. If the trend does not change for 30 days, we rebalance our portfolio back to the 8020 or 2080 allocation. That said, let's now head over to QQQQQ and take a quick look at this algorithm before we deploy it to live trading. The algorithm trades at an hourly resolution while the SMA is updated at a daily resolution. I didn't go for a higher resolution such as Minnadley since this would lead to a bunch of fast allocation changes while the price jumps above and below the SMA's value. In the initialised method, we set the capital for this algorithm to 100,000 US dollars. Keep in mind that this is just for backtesting purposes. When we live trade our algorithm, this number will be disregarded and the actual balance from your broker account will be used. Since I will be setting up live trading using an interactive broker's account, I will set the brokerage model for this algorithm to be an IB margin account. This will make sure the backtesting settings, including commissions, will be as realistic as possible for that account. QQQQQ also has the option for you to distinguish between backtesting and live trading mode using the live mode flag. So if you want your algorithm to treat certain scenarios differently in backtesting than an actual trading, you can do so with a simple if statement. A common use case for this would be if you are using custom data from a third party API, since in backtesting you will just use historical records while in live mode you want to perform actual API calls. That said, let me now build and backtest this algorithm so that we can get a brief first impression of this strategy's performance before I connect it with the IB account. As you can see over the time period specified in the initialised method, this algorithm seems to perform relatively well. But this isn't too surprising since QQQQ was generating good returns throughout this period. To get a good overview of when this algorithm is using which allocation, it plots QQQQ's SMA on the benchmark chart. Especially during live trading this can be a great way to determine whether the algorithm currently thinks that there is an uptrend or downtrend. Since we are using an hourly resolution, the benchmark chart will be updated with QQQ's price once an hour. The SMA value on the other hand will only be updated once a day since we do this at a daily resolution. If you want to analyze this backtest report in more detail or in general play around with this algorithm, there's a link in the description box below that allows you to clone this code. Next up, let me show you how you can set up live trading in QQQQQ. For this, I will first show how you can pay per trade directly through QQQQQQQ and thereafter how to connect your algorithm to interactive brokers. For either case, you need to click on the go live button at the top next to the backtest and optimize button. This will bring you to a new view which will look like this. Here, you first need to choose a brokerage that you want to use for their live trading. For this first demo, I will just go with QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ or for both. Notifications are a great option for you to gain insights into when your algorithm is doing something without you having to actively monitor it. Last but not least, you can choose whether you want your algorithm to automatically restart in case it runs into an error. Once you set everything up, you can click the deploy button and the algorithm will start running with real market data. Since it is still a paper trading, it won't trade any real money though, but this is nonetheless a great way to test whether your bot is set up correctly to handle live data. Once an algorithm is deployed to live trading, you can view its live performance through this page. This page has a similar structure to the backtest view with runtime statistics at the top, some charts and the ability to view orders and logs at the bottom. A new tab at the bottom is the holdings tab. This allows you to view your current positions, subscribe to new data for more securities and even place manual open and close orders for any of the subscribed securities. In case you want to stop your algorithm, you have two options. At the top, there are two buttons. The liquidate button liquids all your positions and the stop button simply stops your algorithm. Note that just clicking the stop button will leave all currently open positions open. This is especially relevant if you have an actual live broker account connected, so keep this in mind. Since you now hopefully have at least a basic understanding of what deploying an algorithm through QuantConnect looks like, let me outline how you can connect an actual interactive broker's account to UQC algorithm. For this, it is very important to understand some of the intricacies involved in connecting an IB account. So let's go over some prerequisites and important notes first. It is both possible to connect an interactive broker's paper trading account or a real account to QuantConnect. Regardless of whether you already paper traded your algorithm in QuantConnect, I would highly recommend first trying to connect your ALGO to an IB paper trading account to fully understand the process. That said, the first requirement is having an IB pro account. API trading is currently not supported through a LITE account. Secondly, it is important to understand that interactive brokers does not support more than one active session per login credentials. This is the case both for paper and live trading accounts. Here, an active connection to QuantConnect also counts as one active session. This means, while you are connected to QuantConnect, you cannot use the same login credentials to log into your IB trader workstation to make trades. If you attempt to do this, you will receive the following error. You are logged in without trading slash market data permissions, you can still manage your account but can't trade. If you would then click Continue, you would automatically sever the connection to QuantConnect and your algorithm would run into an error. So to still be able to trade from your account, besides through your algorithm, there are two possible solutions. The first would be to exclusively trade through QuantConnect's live trading page. This is possible since you can still place manual trade orders directly through QuantConnect. But an alternative solution would be to create a second user under the same name for your account. This resolves the issue since these are new login credentials for the same account which means you will be able to have another active session in that account. I'd recommend creating a second account and using this exclusively for algo trading. One more thing to be aware of is QuantConnect's compatibility with IB security measures. As of the making of this video, QuantConnect is not compatible with Interactive Brokers 2-Factor SMS authentication. So if you want to use 2-Factor authentication for better security, you will have to do so through the Interactive Brokers mobile app. Note that if you use 2-Factor authentication, you will have to re-authenticate the connected account once a week to maintain the connection to QuantConnect. With that out of the way, let me now present to you how you can connect an actual Interactive Brokers account to your QC algorithm. For this demonstration, I will simply use my IB paper trading account, but note that the process is equivalent for a real account. So once again, we click on the Go Live button to open the Live Deployment selection view. Instead of choosing QuantConnect paper trading, we select Interactive Brokers this time. Now you will have to enter your IB account username, account ID and password. Note that an Interactive Brokers paper trading account uses a different login and account ID than your actual main account. So if you want to set up the paper trading, make sure to enter your paper trading account credentials here and not your live trading account login. Also note that QuantConnect does not save your Broker login credentials. Before deploying the algorithm, you can now also select the data source that you would want your algorithm to use. If you want to use the Interactive Brokers feed as the data source, you need to have purchased a live data subscription from IB since you otherwise might get delayed data. Alternatively, you can also just use QuantConnect or both as a data source. If you choose both as a data source, the algorithm will first attempt to get the data from QuantConnect and then it requested it from IB if that was not possible. Note that you need to select IB only or both as a data source if you want to access options or index data. Lastly, you can turn on automatic restarts for your algorithm. If this is turned on, the algorithm will automatically attempt to relaunch once it runs into an error. If this happens, you will be notified by mail. However, if an algorithm fails to restart five times in a row, it will no longer attempt to redeploy. When deploying the algorithm, make sure that you are not currently logged into your IB Broker account with the same credentials anywhere else. Otherwise, the algorithm will run into an error and fail to deploy. Once the algorithm is connected with your IB account, you will see the same view as for QuantConnect paper trading. Then you can once again place manual trade orders through the QuantConnect interface. But here it is important to understand that the equity chart and performance metrics of the QuantConnect Live trading view will take many of your positions and already existing positions of your portfolio into account as well. So keep this in mind, for example, when wondering why your out-of-sample backtest performance might be deviating a lot from the show-on performance. In general, if you want to truly test the performance of your algorithm strategy, you should not interfere with it too much since you otherwise won't get representative results. If you ever want to change the code of a live running algorithm, you will have to stop and redeploy it. Since this algorithm uses an hourly resolution, let's let some time pass and then come back to see if it's actually doing what it's supposed to do. As you can see, the algorithm has now opened a position and the performance chart is no longer completely flat. We can scroll down and view our portfolio as well as any order sent out over the lifetime of this algorithm. Furthermore, we can check the logs in which we printed the moving average value as well as QQQ's most recent price. As you can see, the SMA is above the price, which means that the algorithm considers there to be a downtrend. That is why we invested 80% of the available capital into BND and only 20% into QQQ. Now, let me log into my IB account to check if we can actually see if it's connected correctly. Since I already connected my paper trading account to the algorithm, I won't be able to fully connect to the IB platform using the same credentials. However, instead, I can just use the real account credentials to log into paper trading mode. This will still allow me to view the portfolio holdings of the paper account and it won't lead to any double connection problem. Like I said earlier, if you connect your main account to QQQ, you can just create a second user for the same account to work around this problem. But since you can't create a second user for paper trading account, this is not possible here. After logging in, we can see that almost our entire portfolio is invested in equities, which is exactly what we expect. If we click on the detailed view of the invested stock portion of our portfolio, we can actually see how much is invested in which stocks. Here we can see that, just like we want it to be, about 80% is invested in BND while 20% is invested in QQQQ. Last but not least, let me go over some important aspects to be aware of when switching from backtesting to live trading and some general risks to know. First of all, when developing and testing your algorithms, make sure to put some thought into how your algorithm handles any potential existing positions. In many previous videos, we used if statements that test whether the portfolio is invested to make trading decisions. While this works great in backtesting since the portfolio will only be invested if the algorithm decides to open a position, this is not necessarily true in reality. You might already have other open positions when deploying the algo or want to place manual trade orders in certain scenarios. So always make sure to do asset specific checks instead of portfolio wide actions if that's not what you want. Another example of this would be using the liquidate method without passing a specific position. This will not only close positions that were opened by your algorithm, but it will close all open positions in your portfolio even if they existed long before you deployed your algo. To simplify things, it can be a good idea to have a completely separate account just for algo trading. Also note that your algorithm relies on data as well as a stable connection to your broker's API. If the data feed or API is down, your algorithm will not be able to function properly. This usually does not happen very often, but nonetheless you should be aware of these risks and try to add certain safeguards for situations like this. That's also why actively monitoring server uptimes and your algorithm is a very good idea. Furthermore, please remember that just because an algorithm performs well in backtesting is by no means a guarantee that it will perform great in reality. Unrealistic slippage, execution, commission settings, short share availability and biases are just a few examples that could lead to deviations in backtest and real-world results. That's why I highly recommend live paper trading your algorithm before deploying it with any real money. And even after that, start with small amounts of capital instead of rushing into it all at once. But like I mentioned earlier, try to go through the entire deployment process in paper trading mode to familiarize yourself with everything first. Also, don't make the mistake and just set and forget your algorithm. Especially in the beginning, it's a very good idea to actively monitor what it's doing to make sure that it's indeed doing what you want it to do. That being said, I hope you enjoyed this video and now understand how you can deploy your own algorithm to live trade for you. If you did, make sure to smash the like button, subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/19-trading-bot-50-lines.md b/course/19-trading-bot-50-lines.md new file mode 100644 index 0000000..a3d6228 --- /dev/null +++ b/course/19-trading-bot-50-lines.md @@ -0,0 +1,23 @@ +# 19 · 50-Line Trading Bot（50 行代码 · 标普成分股调整） + +- **系列**：Full Algorithmic Trading Using Python +- **频道**：TradeOptionsWithMe ｜ **本集**：50-Line Trading Bot（50 行代码 · 标普成分股调整） +- **时长**：17 分 29 秒 ｜ **原视频**：https://youtu.be/EbUEMW5135M +- **本地视频**：[videos/19-trading-bot-50-lines.mp4](videos/19-trading-bot-50-lines.mp4) + +## 🎯 本集要点（中文导读） + +**50 行代码的选股 bot**：围绕 **标普 500 指数成分股调整** 套利（新手友好）。 + +- **策略逻辑**：标普 500 定期**增删成分股**；新纳入的股票会迎来关注度与被动买盘，被剔除的会被抛售。于是——**买入新纳入的股票、卖空（或卖出）被剔除的股票**（通常同时发生，故常同时开一多一空）。入场后用**跟踪止损（trailing stop）** 做风控。 +- **数据准备**：从维基百科《List of S&P 500 companies》的增删表格抓数据，用 **Jupyter + pandas**（`read_html`）清洗成可用格式（作者提供了数据链接）。 +- **实现**：用 QuantConnect，**不到 50 行 Python** 完成。 +- ⚠️ 提醒： + - 维基数据的**质量/完整性无保证**，用自定义数据要警惕数据质量问题（含此前提到的 **ticker/symbol 陷阱**，可用 QuantConnect 的 SPY 成分股数据集改进，其中还含各成分权重）。 + - 实盘需**实时数据源**；可扩展方向：把"剔除原因"标准化后纳入决策、或扩展到其它大指数。 + +## 📝 完整文稿（英文 · 自动转写） + +> 由本地 ASR（faster-whisper）从视频音轨转录，未人工校对，供检索/精读使用。原始讲解以上方视频为准。 + +Welcome back to another video. In this video we will create yet another trading algorithm using the QuantConnect Algorithmic Trading Platform. The bot that we will implement will look at changes in the constituents of the biggest and most followed market index, namely the S&P 500 and trade based on these changes. As you probably are aware of, the S&P 500 index tracks the 500 biggest companies trading on US stock exchanges. Since markets are in constant motion, the list of the biggest companies occasionally changes. Sometimes certain stocks are removed from the index while others are included. Unsurprisingly, the addition or removal of a stock into one of the most followed market indexes will attract some attention. Furthermore, a wide plethora of funds and investors have holdings in the S&P and will thus allocate capital to the stock newly included in the index. The idea of our strategy is to capitalize on this spike of attention and buy pressure by buying stocks that are being newly added to the index and sell stocks that have been removed. Since most of the time a stock will be removed at the same time as another stock will be added to the index, we will usually open a long and a short position at the same time. Once a position has been opened, we want to use a trailing stop loss a few percentage points away from the entry price for risk management. In this video, we will code this entire strategy in Python in under 50 lines of code using QuantConnect. I highly recommend following along, so if you haven't already, make sure to use the link in the description box below to create your free account and code along. Note that this will be a relatively easy beginner-friendly bot to code along with. However, in case you want a complete walkthrough of how to develop your own trading bots using the QuantConnect API, check out my free algorithmic trading course in the description box below. That said, before we can actually start implementing the code for this trading bot, we first need to find the data of the constituent changes and preprocess it so that we can actually use it as an input for our trading algorithm. Luckily, I found some data on changes in the S&P 500 members on Wikipedia in the article list of S&P 500 companies. Here we can see a table with additions and removals of stocks and a reason for the change going back at least 20 years. Note that I don't have any information on the quality or completeness of this data, but nonetheless I will use this data for the testing of our strategy. However, before we want to import this data and actually use it in our trading algorithm, we need to preprocess this data and transform it into the right format. For this, I will open a new Jupyter notebook to scrape the data and perform these changes. Note that I will make this data available through a link in the description box below so that you can copy and download it for your own trading algorithm. The first thing we want to do in our Jupyter notebook is import pandas so that we can work on the data in a pandas data frame. Next, we use the pandas readHTML method to read the data from the Wikipedia page. What this method does is it takes all HTML and tables on the given page and turns them into data frames and then adds these to a list. Since the table we are interested in is the second one, we index this list to get the right data frame. As you can see, we now have the data here, however, instead of having the newer states at the top, we want the reverse order. Next up, we want to flatten the column names of the data frame since right now it consists of multiple layers. Furthermore, we are actually only interested in the columns, date, ticker, added and ticker removed. Before we move on to refactoring this data a bit more, let's create a list of all the tickers mentioned in the remove and add columns. We will use this list inside of QuantConnect to later add data for these stocks to our algorithm. To create this list, we turn the columns into sets and union them after dropping non-values. After this, we want to refactor the data a bit. More specifically, right now the problem with the data is that we have multiple rows with the same date. Instead of having many rows with the same date, we want to have a list of all added and removed stocks for each date. For this, we can simply loop through the rows of the data frame and create a dictionary with the lists for each date in the data frame. Then all we have to do is turn that dictionary into a new Pandas data frame, reset the indexing, sort it correctly and we basically have the format we want. One last thing we want to do however is turn the lists of stocks into strings where the tickers are separated by spaces instead of commas. We do this since we will later import the data into QuantConnect as a comma-separated file and thus not using commas here as separators will make this step easier later. With that, we now have the data in the format that we desire. So now we can export it to a CSV file. This CSV file I will now upload to Dropbox. We can then use the Dropbox URL to this data in our algorithm to access this data and use it in our algorithm inside of QuantConnect. I will leave the Dropbox URL in the description box below so that you can download this data too. With that being said, we can now head over to QuantConnect and start writing the code for the actual trading bot. To create the trading bot, we will go to the lab tab and click on Create New Algorithm. Once a blank template algorithm was being created, we want to start by implementing the initialize method in which we can set some algorithm variables and backtest settings. First off, we set the backtest time frame to 2015 until 2021. Note that to backtest a strategy properly, you need to specify a backtest time frame for which we actually have the custom data. If you backtest on a time frame without the data, you won't get any meaningful results. After that, I will leave the starting cache balance for the backtest at $100,000. In the next step, we will take advantage of QuantConnect's algorithm framework which allows us to specify a risk model that is pre-made for us. Here I will go with the Trading Stop Risk Model and pass a percentage of how far I want the trading stop loss to follow our positions. For this parameter, I will go with 5%. This will automatically create a trading stop loss for us so we won't have to worry about any risk management in the remainder of the implementation. Next up, I will paste the ticker list that we created from the custom data before and save it to a variable. Note that there is a potential problem with this. To understand this, we have to know that QuantConnect differentiates between symbols and tickers. Symbols are permanent identifiers that stay consistent even after big changes. While tickers can change and lead to some ambiguity problems. In other words, since some of these companies and tickers are old, there might be a chance that the algorithm will think this is a different security. One approach to fixing this problem would be to use QuantConnect's curated SPY Constituents dataset that I linked to in the description box below. However, here we have to know that the SPY Constituents might slightly vary from those of the S&P. Furthermore, I want to demonstrate how we can work with custom data since this opens up so many doors for developing interesting strategies. That's why I will still stick to this approach for now, but for more details on the differences between tickers and symbols, check out the third video in my algorithmic trading course. We will now subscribe to the data of all these tickers at an hourly resolution. Note that this could be optimized since we might not need all the data depending on the back test time frame, but since there aren't that many symbols, this is just the easiest approach. Using Minitly or an even higher resolution would allow us to access quote information. This would then lead to more accurate back tests since the trades are based on the bits and asks, but at the same time the back test would take way longer due to the abundance of data points. That's why we stick to hourly data for now. After adding the equities data, it is time to add the custom data file that we uploaded to Dropbox. To import custom data to QuantConnect, you need to create and implement a custom class. We will name this class SPX change and add the data at an hourly resolution and save it symbol to a class variable name change. Before we can actually use this data, however, we must implement this SPX change class. This class has to inherit from the Python data class and it needs to implement two methods, namely the get source and reader methods. Like the name implies, the get source method is there to get the source of the custom data. In this method, we will specify as the source of the data the dropbox you are ill from before. However, note that here we need to set the download flag to 1 instead of the default 0. Then next up, we specify that this is a remote file that we want to get the data from and return a subscription data source object. After that, we can move on to implementing the reader method, which is there to read from the custom data source and return a custom data object. This method basically receives the data from our data file line by line. So first, we need to check that the line that the method received is in fact one with the right data. If for instance, we already went through all the data and receive an empty line, we don't want to return anything. Otherwise, we want to split the line by commas since it's a comma separated file and create a custom data object. An important thing we need to do here is set the symbol and datetime of the just created custom data object. Since the time is specified in the first column of our data file, we can index it with zero. However, then it is important to tell the method what format the data has. Since the Wikipedia data did not have any time of day information, we'll just make it available at 10am. Next up, we need to set some other attributes of the custom data object so that we can access the added and removed tickers. By default, there is the value attribute for custom data that we can use for this. But this is only for decimal data, which we don't have here. So instead, we just set the attribute to zero, since it's not recommended to not set it to anything. Then we create two more attributes in which we save lists of the added and remove tickers respectively. Lastly, if while doing any of this, we run into an error, we just want to return nothing. In case everything works out fine, we return this custom data object. Now that we have implemented the custom data class, we can move on to implementing the actual trade logic, which will only be a few lines of code. For this, we will use the on data method that is called every time the algorithm receives new data. This new data is submitted in the data parameter. Since we only want to perform any opening trades in case the stock has been added or removed from the index, we first need to check if we have any custom data object in our data parameter. In case we do, we want to loop through the added tickers and set 10% of our portfolio to invest into this stock. However, before we do so, we first check whether we have data for the given stock since a few of the tickers might not be supported or have data issues. Furthermore, we basically want to do the same for the stocks that have been removed from the index. However, here we want to open a short position with only 5% of our portfolios capital. We open a smaller position here since short positions are considered more risky. To achieve the right allocation, we use portfolio targets and set holdings. With that, we are now actually already done with implementing our trading bot. As you can see, this didn't even take 50 lines of code. So let us now build and backtest this algorithm to see the results over the backtest time frame. After a relatively short time, the backtest should have finished and you will see a performance report like this one. As you can see, the performance of this bot seems to be relatively good. However, I will not go over this report in any detail. But note that, if you haven't coded along, you can clone the entire algorithm, test it and play around with it yourself. I left a link in the description box below for that. Note that this algorithm in general plays it quite safe considering that the S&P 500 constituents don't change very often and it only invests about 15% at once into a long and short position. Instead of analyzing these results in detail, let me provide some suggestions on how this strategy could be improved. Firstly, usually the news that a stock will be added or removed from the S&P 500 is known before it actually happens. So a possible improvement of the strategy would be to try to find out about an inclusion or removal earlier so that the bots can act earlier. Possible approaches to this would be to track the inclusion metrics of the S&P 500 and check which stocks fulfill the criteria. You can, for example, find the requirements on Wikipedia. Another approach for this would be to analyze news and social media from mentions of stocks being included or removed. You might be able to get a rough idea of what impact getting the information earlier might have by making the custom data available a bit earlier. However, note that getting in earlier will mean that you will be exposed to directional movements for some time before the actual inclusion or removal. So here it might potentially also make sense to adjust the trailing stop. Besides that, another variation would be to track the reason for the addition or removal to the index as well. The idea behind this is that some reasons might impact the price more than others, so accounting for this in the strategy might be interesting to try out. This would not be very hard to implement since the Wikipedia data actually includes the reason already. However, note that this would require some additional pre-processing since the reasons aren't standardized. So you would first have to standardize the reason column to the most common reasons such as market cap requirements or acquisitions to name two common examples. Besides that, it might be an interesting idea to expand the strategy to more big name indexes than just the S&P 500. With that being said, note that for live trading you would have to find a real-time data source, but since the S&P 500 is one of the most well-known indexes, this shouldn't be too hard to find. Furthermore, please keep in mind that I can't guarantee the accuracy of the Wikipedia data in this video. In general, especially when working with custom data, always make sure to consider any potential data quality issues. For example, here you should remember the ticker and symbol problem that I referenced earlier. Like I said then, this could be improved by using QuantConnect's SPY constituents dataset. The cool thing about this dataset is that it also includes the allocation weight of each member of the SPY which could also be incorporated into the decision making of your strategy. With that being said, I really hope you enjoyed this video. If you want to learn more about developing your own trading algorithms using Python and QuantConnect, make sure to check out my free algorithmic trading course. Otherwise, make sure to smash the like button, subscribe and turn on the notification bell. Thanks for watching. diff --git a/course/README.md b/course/README.md new file mode 100644 index 0000000..efd46e9 --- /dev/null +++ b/course/README.md @@ -0,0 +1,37 @@ +# Full Algorithmic Trading Using Python — 课程教程 + +> 本目录是对 YouTube 频道 **TradeOptionsWithMe** 的《Full Algorithmic Trading Using Python》系列的**学习笔记 + 视频归档**。 +> 播放列表：https://youtube.com/playlist?list=PLtqRgJ_TIq8Y6YG8-G-ETIFW_36mvxMLad + +- **集数**：19 集 ｜ **总时长**：约 6 小时 45 分 +- **每集内容**：📌 中文要点导读（提炼） + 📝 英文完整文稿（本地 ASR 转写） + 🎬 本地视频（360p） +- **提醒**：文稿由自动语音识别生成，**未人工校对**；仅供个人学习检索，版权归原作者所有。视频画质上限 360p（YouTube 未登录下载限制）。 + +## 目录 + +| 集 | 标题 | 时长 | 链接 | +|---|---|---|---| +| 01 | Introduction（课程总览） | 7:19 | [📄 笔记](01-introduction.md) · [🎬 视频](videos/01-introduction.mp4) | +| 02 | How Trading Algorithms Work（算法原理与开发流程） | 10:03 | [📄 笔记](02-episode-2.md) · [🎬 视频](videos/02-episode-2.mp4) | +| 03 | Key Concepts（关键概念：时间处理·骨架·Symbol） | 10:05 | [📄 笔记](03-key-concepts.md) · [🎬 视频](videos/03-key-concepts.mp4) | +| 04 | Handling Data（处理数据 · 第一个算法） | 18:01 | [📄 笔记](04-handling-data.md) · [🎬 视频](videos/04-handling-data.mp4) | +| 05 | Trading & Orders（下单、订单管理与调试） | 28:46 | [📄 笔记](05-trading-and-orders.md) · [🎬 视频](videos/05-trading-and-orders.mp4) | +| 06 | Indicators & Historical Data（指标与历史数据） | 28:32 | [📄 笔记](06-indicators-history.md) · [🎬 视频](videos/06-indicators-history.mp4) | +| 07 | Consolidators & Rolling Windows（合并器、滚动窗口、事件调度） | 22:40 | [📄 笔记](07-consolidators-rolling-windows.md) · [🎬 视频](videos/07-consolidators-rolling-windows.mp4) | +| 08 | Dynamic Universes（动态证券池与基本面选股） | 25:30 | [📄 笔记](08-dynamic-universes.md) · [🎬 视频](videos/08-dynamic-universes.mp4) | +| 09 | Twitter Trading Bot（自定义数据 · 推文情绪） | 26:52 | [📄 笔记](09-twitter-trading-bot.md) · [🎬 视频](videos/09-twitter-trading-bot.mp4) | +| 10 | Backtesting & Performance Analysis（回测与绩效评估） | 25:45 | [📄 笔记](10-backtesting-performance.md) · [🎬 视频](videos/10-backtesting-performance.mp4) | +| 11 | Forex Trading（外汇 · 均值回归） | 13:27 | [📄 笔记](11-forex-trading.md) · [🎬 视频](videos/11-forex-trading.mp4) | +| 12 | Options Trading（期权入门） | 15:28 | [📄 笔记](12-options-trading.md) · [🎬 视频](videos/12-options-trading.mp4) | +| 13 | Options Code-Along（保护性看跌期权实战） | 26:06 | [📄 笔记](13-options-code-along.md) · [🎬 视频](videos/13-options-code-along.mp4) | +| 14 | Crypto Trading Bots（加密货币 · RSI 动量） | 15:51 | [📄 笔记](14-crypto-trading-bots.md) · [🎬 视频](videos/14-crypto-trading-bots.mp4) | +| 15 | The Algorithm Framework（算法框架） | 34:14 | [📄 笔记](15-algorithm-framework.md) · [🎬 视频](videos/15-algorithm-framework.mp4) | +| 16 | Data-Driven Research（研究环境 · QuantBook） | 21:52 | [📄 笔记](16-data-driven-research.md) · [🎬 视频](videos/16-data-driven-research.mp4) | +| 17 | Bitcoin ML Bot（用神经网络做比特币） | 31:07 | [📄 笔记](17-bitcoin-ml-bot.md) · [🎬 视频](videos/17-bitcoin-ml-bot.mp4) | +| 18 | How to Live Trade（实盘部署与 Paper Trading） | 26:03 | [📄 笔记](18-live-trading.md) · [🎬 视频](videos/18-live-trading.mp4) | +| 19 | 50-Line Trading Bot（50 行代码 · 标普成分股调整） | 17:29 | [📄 笔记](19-trading-bot-50-lines.md) · [🎬 视频](videos/19-trading-bot-50-lines.mp4) | + +## 关于本归档 +- 视频：经 yt-dlp 下载归档（360p），讲稿：faster-whisper 本地转写，导读：小五整理。 +- 系列主线为 **Python + QuantConnect（Lean 引擎）** 开发算法交易：从概念 → 下单/指标/数据 → 各类资产（股/汇/期权/币）→ 框架/研究/ML → 实盘。 +- 生成日期：2026-09-12。 diff --git a/course/videos/01-introduction.mp4 b/course/videos/01-introduction.mp4 new file mode 100644 index 0000000..d6d7312 Binary files /dev/null and b/course/videos/01-introduction.mp4 differ diff --git a/course/videos/02-episode-2.mp4 b/course/videos/02-episode-2.mp4 new file mode 100644 index 0000000..4a2a371 Binary files /dev/null and b/course/videos/02-episode-2.mp4 differ diff --git a/course/videos/03-key-concepts.mp4 b/course/videos/03-key-concepts.mp4 new file mode 100644 index 0000000..db13a63 Binary files /dev/null and b/course/videos/03-key-concepts.mp4 differ diff --git a/course/videos/04-handling-data.mp4 b/course/videos/04-handling-data.mp4 new file mode 100644 index 0000000..1df9df8 Binary files /dev/null and b/course/videos/04-handling-data.mp4 differ diff --git a/course/videos/05-trading-and-orders.mp4 b/course/videos/05-trading-and-orders.mp4 new file mode 100644 index 0000000..424388d Binary files /dev/null and b/course/videos/05-trading-and-orders.mp4 differ diff --git a/course/videos/06-indicators-history.mp4 b/course/videos/06-indicators-history.mp4 new file mode 100644 index 0000000..a8d0563 Binary files /dev/null and b/course/videos/06-indicators-history.mp4 differ diff --git a/course/videos/07-consolidators-rolling-windows.mp4 b/course/videos/07-consolidators-rolling-windows.mp4 new file mode 100644 index 0000000..fa762e9 Binary files /dev/null and b/course/videos/07-consolidators-rolling-windows.mp4 differ diff --git a/course/videos/08-dynamic-universes.mp4 b/course/videos/08-dynamic-universes.mp4 new file mode 100644 index 0000000..66b9c07 Binary files /dev/null and b/course/videos/08-dynamic-universes.mp4 differ diff --git a/course/videos/09-twitter-trading-bot.mp4 b/course/videos/09-twitter-trading-bot.mp4 new file mode 100644 index 0000000..eece903 Binary files /dev/null and b/course/videos/09-twitter-trading-bot.mp4 differ diff --git a/course/videos/10-backtesting-performance.mp4 b/course/videos/10-backtesting-performance.mp4 new file mode 100644 index 0000000..f21bc2c Binary files /dev/null and b/course/videos/10-backtesting-performance.mp4 differ diff --git a/course/videos/11-forex-trading.mp4 b/course/videos/11-forex-trading.mp4 new file mode 100644 index 0000000..c623bee Binary files /dev/null and b/course/videos/11-forex-trading.mp4 differ diff --git a/course/videos/12-options-trading.mp4 b/course/videos/12-options-trading.mp4 new file mode 100644 index 0000000..dfa8838 Binary files /dev/null and b/course/videos/12-options-trading.mp4 differ diff --git a/course/videos/13-options-code-along.mp4 b/course/videos/13-options-code-along.mp4 new file mode 100644 index 0000000..01853e0 Binary files /dev/null and b/course/videos/13-options-code-along.mp4 differ diff --git a/course/videos/14-crypto-trading-bots.mp4 b/course/videos/14-crypto-trading-bots.mp4 new file mode 100644 index 0000000..33ccf8d Binary files /dev/null and b/course/videos/14-crypto-trading-bots.mp4 differ diff --git a/course/videos/15-algorithm-framework.mp4 b/course/videos/15-algorithm-framework.mp4 new file mode 100644 index 0000000..bcc8486 Binary files /dev/null and b/course/videos/15-algorithm-framework.mp4 differ diff --git a/course/videos/16-data-driven-research.mp4 b/course/videos/16-data-driven-research.mp4 new file mode 100644 index 0000000..c858305 Binary files /dev/null and b/course/videos/16-data-driven-research.mp4 differ diff --git a/course/videos/17-bitcoin-ml-bot.mp4 b/course/videos/17-bitcoin-ml-bot.mp4 new file mode 100644 index 0000000..ae2b639 Binary files /dev/null and b/course/videos/17-bitcoin-ml-bot.mp4 differ diff --git a/course/videos/18-live-trading.mp4 b/course/videos/18-live-trading.mp4 new file mode 100644 index 0000000..00e17ee Binary files /dev/null and b/course/videos/18-live-trading.mp4 differ diff --git a/course/videos/19-trading-bot-50-lines.mp4 b/course/videos/19-trading-bot-50-lines.mp4 new file mode 100644 index 0000000..d4c957a Binary files /dev/null and b/course/videos/19-trading-bot-50-lines.mp4 differ