Remember me

Register  |   Lost password?


arXiv logo for blog page

Managing the Complexity of Processing Financial Data at Scale -- an Experience Report. (arXiv:1908.03206v1 [cs.DB])

Sun, 11 Aug 2019 23:02:25 GMT

Financial markets are extremely data-driven and regulated. Participants rely
on notifications about significant events and background information that meet
their requirements regarding timeliness, accuracy, and completeness. As one of
Europe's leading providers of financial data and regulatory solutions vwd
processes a daily average of 18 billion notifications from 500+ data sources
for 30 million symbols. Our large-scale geo-distributed systems handle daily
peak rates of 1+ million notifications/sec. In this paper we give practical
insights about the different types of complexity we face regarding the data we
process, the systems we operate, and the regulatory constraints we must comply
with. We describe the volume, variety, velocity, and veracity of the data we
process, the infrastructure we operate, and the architecture we apply. We
illustrate the load patterns created by trading and how the markets' attention
to the Brexit vote and similar events stressed our systems.