SHIFT: High Frequency Trading Simulation System

Ionut Florescu -

SHIFT is a test-bed that can simulate realistically the behavior of modern high-frequency financial markets and will become a key tool for improving the functioning of these markets, by allowing investors, market makers, managers of exchanges, and regulators to test scenarios and strategies under much more realistic conditions -- prior to implementing those strategies in a live market.

This will reduce the risk of “rogue” behaviors propagating in live markets as new software is turned up (such as the Knight Capital scenario and many other “glitches” seen in a variety of markets in recent years). It will enable regulators and exchange managers to experiment safely with possible rule-changes. It will allow market participants to reduce the uncertainty associated with innovations in trading strategies.

SHIFT is designed to be very versatile. The system supports several application areas, including:

  • Teaching micro structure finance and algorithmic trading courses. See FE 570 and FE 670
  • Designing and back testing high frequency trading algorithms.
  • Answering research questions about the impact and interaction of HFT algorithms in the market.
  • Designing and studying the impact of new regulations.

Research Topics:

The system developed is in fact a complete state of the art Agent Based modeling simulator. Therefore, it can answer a variety of questions that are currently being addressed at a very small scale with simulated (read little intelligence) agents.

These questions include but are clearly not limited to:

  • Testing new regulatory/rule changes at the exchange level and their effect to the trading ecosystem. The contemplated rules include the minimum life order rule, maximum order flow limits, half second resting time, order to trade ratio constraints, batch auction mechanism and many other rules. We will statistically measure the impact that each rule will have on the market participants as well as on the order fill ratio at the exchange level.
  • Testing the market impact caused by the interaction of different HFT algorithms. This may include situations in which HFT traders artificially create new opportunities that other HFT traders take advantage of, but that wouldn’t necessarily exist if HFT traders were not part of the market. Testing market reaction to stress scenarios. To this end we will create rogue algorithms and test measures that may be put in place to detect and shut down such algorithms.
  • The connection to the news engine will allow to gather the reaction of algorithms that use Natural Language Processing in trading to shocking words and phrases. We will test the impact of such news to the well-being of the market.
  • Since 2023 the SHIFT system has been used to design and test Automated Market Makers as well as Reinforcement Learning Algorithm. Please note the links below.

Publications:

T. Winkler, Z. Ye, I. Florescu. "SHIFT Project: A Tool for Teaching and Research". 7th Annual Stevens Conference on High Frequency Finance and Analytics (2016).

I. Florescu, T. Winkler, Z. Ye. "Algorithmic trading / Machine learning in Finance: Assessing the Algorithms’ Interaction and Impact". The Eastern Conference on Mathematical Finance (2017).

Florescu, I., Alves, T., Bozdog, D., & Calhoun, G. Using SHIFT a laboratory environment to test changes to financial markets. In 2020 Fall Central Sectional Meeting. AMS.

Winkler Alves, T. (2020). A laboratory environment for financial markets (Order No. 28088758). Available from Dissertations & Theses @ Stevens Institute of Technology. (2456827355). Retrieved from https://stevens.idm.oclc.org/login?url=https://www.proquest.com/dissertations-theses/laboratory-environment-financial-markets/docview/2456827355/se-2

Kashyap, S. (2020). Dynamic high frequency trading algorithm (Order No. 28152480). Available from Dissertations & Theses @ Stevens Institute of Technology. (2503955846). Retrieved from https://stevens.idm.oclc.org/login?url=https://www.proquest.com/dissertations-theses/dynamic-high-frequency-trading-algorithm/docview/2503955846/se-2

Alves, T. W., Florescu, I., Calhoun, G., & Bozdog, D. (2020). SHIFT: A Highly Realistic Financial Market Simulation Platform. arXiv preprint arXiv:2002.11158.

Alves, T. W., Florescu, I., & Bozdog, D. (2023). Insights on the Statistics and Market Behavior of Frequent Batch Auctions. Mathematics, 11(5), 1223.

Awards and Grants:

  • USPTO patent number US 62/239,351
  • CME Group Foundation support 2015 - 2017
  • CAPCO support 2018-2019

Researchers:

Former Members of the Team:

Research Papers:

Alves, T. W., Florescu, I., Calhoun, G., & Bozdog, D. (2020). SHIFT: A Highly Realistic Financial Market Simulation Platform. arXiv preprint arXiv:2002.11158.

Alves, T. W. (2020). A Laboratory Environment for Financial Markets. Stevens Institute of Technology. Ph.D. Thesis

Kashyap, S. (2020). Dynamic High Frequency Trading Algorithm (Master's thesis, Stevens Institute of Technology). Master Thesis.

Ye, Z., & Florescu, I. (2019). Extracting information from the limit order book: New measures to evaluate equity data flow. High Frequency, 2(1), 37-47.

Ye, Z., & Florescu, I. (2020). Covid-19 and the equity market. A March 2020 Tale. A March.

Alves, T. W., Florescu, I., & Bozdog, D. (2023). Insights on the Statistics and Market Behavior of Frequent Batch Auctions. Mathematics, 11(5), 1223.

Yao, Z., Florescu, I., & Lee, C. (2024, May). Control in Stochastic Environment with Delays: A Model-based Reinforcement Learning Approach. In Proceedings of the International Conference on Automated Planning and Scheduling (Vol. 34, pp. 663-670).

Yao, Z., Li, Z., Thomas, M., & Florescu, I. (2024). Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior. arXiv preprint arXiv:2403.19781.

System Features:

Development Road map:

We believe we are now in a position where the system is capable of streamline and answer research questions as well as becoming an extremely important teaching tool for market inner workings and market operation. The final phase of the project is to concentrate on usability and marketability of the system.

To this end we propose to develop the following features for teaching purpose:

  1. Allow a super account (the teacher) to design and start a dedicated server and market environment just for that course.
  2. Include in the web interface the ability to populate the market with artificial traders (market makers, noise traders, institutional traders with their big orders, etc.)
  3. The teacher will have the ability to select the instruments traded. Equity is already implemented but we will add commodities futures, treasury futures, FX Futures, and options on all these underlying assets thanks to the data donated by the CME Foundation
  4. The students will have the ability to start automated trading strategies directly from the interface and study their effects as a result of parameter choice market dynamic etc.
  5. The professor will have the capability of introducing stress scenarios.
  6. A news engine will be linked to the system and provide news events that will drive the market. The students will be reacting to the news.
  7. Lecture notes and case studies for various scenarios.

Another important application is to design and create a competition around the system. The competition will be similar to the CME Challenge, but it will be centered on algorithmic intraday trading strategies. The students will compete in groups similar to the traditional model but would now be able to implement their own strategies using our API. In its current form, the API is entirely written in C++, as this is usually the language of choice for HFT strategies, but it could be expanded to other programming languages (R, Python, Matlab, etc.) if demanded, even though it would require significant effort.

Development History:

Phase 1:

Phase one concentrated on developing the back-end of the three subsystems:

  • Matching Engine subsystem
  • Database and Database Engine subsystem
  • Desktop Client interface

A tremendous amount of work has been put into creating the system in a modular and versatile way.

Phase 2:

Phase two of development has been devoted to creating new user front-ends and improving our back-end subsystems. To streamline our development process and augment the system scaling capabilities, most of the back-end subsystems had to be rewritten and a new back-end was added. The Brokerage Center now acts as a broker between the exchange and the customers. Essentially in this phase we have designed and refined:

  • Brokerage Center, our new subsystem controlling all the communication between users of the platform and the Matching Engine subsystem (the exchange). Its responsibilities include managing users’ portfolio and book keeping information, as well as controlling investment margins.
  • A complete new Web Client application for access to the broker, expanding the reach of the system to the platform of choice of its users (including mobile users). We believe this will be the main point of access of the system, being especially useful in the classroom environment.
  • The development of a complete API in C++ for Direct Market Access, allowing users to direct connect their algorithms direct to the market, without having to use one of the GUI we provide. This API is also shared by our other GUI clients, streamlining its development, and it has been successfully used during our initial Research endeavors listed below.
  • We have developed a complete new subsystem where the market is completely driven by automated traders without any interference from historical data. Much work has been put into creating a stable trading flow that can be disrupted by rogue orders and other such market disruptions.