How the World's Largest Options Clearinghouse Moved From Batch to Stream Processing to Manage Risk

In this session, we will review how the OCC performs complex analytics using Kafka and Flink on a data stream containing prices for over 2 million financial instruments per second and billions of open option positions. We will also review how Ness and the OCC satisfied the SEC's requirement to build isolated Kafka + Flink infrastructure in both AWS and an on-premises data-center for disaster recovery purposes. Using Kafka + Flink, the OCC radically improved performance for their core margin calculation system, going from an 8+ hour, overnight batch process, to a 5 minute intraday operation.

Due to exponential growth in trading volumes and heightened regulatory requirements, OCC embarked on a process to replace its 2 decade old batch and mainframe based valuation/margin process. After evaluating several technology options, OCC chose Kafka + Flink as the platform to build its new risk management system. A key challenge we will focus on is the use of windowing strategies to provide consistency across clients and portfolios on a variable stream of price data.

The Options Clearing Corporation (OCC) is the world's largest equity derivatives clearing organization. It is a systemically important financial institution, clearing over 10 Billion contracts each year. In its role as guarantor and central counterparty, OCC ensures that the obligations of the contracts it clears are fulfilled. Through a novation process, OCC becomes the buyer for every seller and the seller for every buyer, protecting its members from counterparty risk.

To guard against counterparty risks, OCC holds approx. USD 180 billion in margin against clearing member's contracts. Margin requirements are calculated by running a full portfolio Monte Carlo simulation.