Building a Scalable and Reliable AI System for Financial Markets: Insights and Challenges

Introduction

Artificial intelligence is transforming industries worldwide, and its application in financial markets has become increasingly critical. AI systems in this domain must handle complex, unstructured data with high precision and reliability while maintaining scalability and security. This piece addresses those subjects while exploring the architecture, hardware and processes we use in production.

Our approach reflects years of development and operational experience, continuously refining systems to deliver reliable and high-performing AI solutions for our clients.

Overview of the architecture

At the core of our system is a client-facing workflow designed to extract meaningful insights from unstructured financial market chat data. This pipeline processes conversations between bank representatives and their clients, identifying key entities such as requests for quotes, client interest, sentiment, and historical data on broker quotes.

• We store the data exactly as it comes — which helps if we need to re-run any post-processing or manual curation
• Curation involves significant manual analysis, e.g. annotations
• Inference and training is where the magic happens
• Knowing up-to-date production f-scores lets us keep a close eye on client experience and evaluate new models

Client-facing pipeline

The client-facing workflow begins by storing raw input data and the inference results; this helps with traceability, measuring latency and accuracy for each request. Over time it gives us the picture of the level of client experience, how fast we improve, and whether the content of the chats itself is shifting.

Internal monitoring and iterative improvement

Analysis of historical data over longer periods of time — months or years — lets us fine-tune to specific problems faced by a specific industry. We were able to gradually improve our pipeline performance (f-score) from 85% for a generic vanilla product, to north of 95% and in some cases 99% accuracy.

Tech stack

Our technology stack is intentionally streamlined to prioritise reliability and scalability. Python is used throughout, from data processing to production systems, ensuring consistency across teams and simplifying collaboration. ElasticSearch handles text optimisation while PostgreSQL manages structured relational data — both critical for managing large volumes of financial market data. For dashboards and UIs, tools like Grafana and Streamlit provide visual insights, reducing the need for extensive custom development.

This minimalist approach focuses resources on refining data flows and training models, reducing complexity and operational risk while ensuring a robust and maintainable system architecture

Hardware and cloud infrastructure

Most of the technology scales well nowadays — cross-cloud across Azure, AWS and GCP, our very own datacentre, a lot of Python, a bit of ElasticSearch, some PostgreSQL and some web tech for the UI. With two exceptions in our case: data curation and GPUs.

Cloud utilisation

We leverage all three major cloud providers. Our initial reliance on Azure was driven by the banking industry’s familiarity and trust in Microsoft. As AWS gained adoption in financial markets, we added it to our stack. Most recently GCP joined our ecosystem, ensuring a broad, cloud-agnostic approach that maximises flexibility.

Our primary use of cloud is the Infrastructure-as-a-Service model. Not a trendy approach nowadays among cloud architects by a long margin, but it gives us the flexibility and speed we need to deploy what and where we need. Our clients have different priorities when it comes to choice of cloud providers and geographic regions. With IaaS we own the entire software layer and can deploy on any server.

Scaling challenges: GPUs and infrastructure

GPU availability

Our major challenge in the recent past has been GPU availability on cloud platforms. For example, mid-2021 a six-month scarcity of T4 cards with one of our cloud providers caught us off guard. This experience underlined the importance of owning GPUs, particularly for time-sensitive operations — and we started work on our own datacentre the very same year.

GPUs and model scaling

We started with NVIDIA T4 GPUs, versatile but limited in capacity for larger models. As our requirements grew, we transitioned to higher-performing models such as RTX 6000 and A100 cards, enabling faster fine-tuning and inference for larger models. Looking ahead, we are exploring H100 and H200 clusters, which offer cutting-edge performance and scalability for next-generation AI.

Data curation: the heart of model training

The data curation step involves annotating chat data by people, to extract meaningful entities like financial instrument, trade details and client sentiment. This data is later used to train the model. It’s a critical step, and for now largely manual.

To streamline curation we initially used third-party annotation tools, but as we developed new processes to improve annotation effort, we realised the need for a tailored solution specific to the banking domain, to enhance productivity and quality. That tool is now a key component of our intellectual property, offering unmatched precision and efficiency for our domain-specific needs — optimising away every unnecessary click.

Reliability and deterministic results

Ensuring deterministic behaviour in AI systems is critical for production reliability. We achieved this by eliminating randomness during inference, enabling consistent results for identical inputs. This foundation supports robust testing.

• “The usual” — unit tests, integration tests and so on
• Sanity tests — automated checks on basic and edge cases, ensuring fundamental functionality remains intact
• Regression testing — comparing current and new model outputs to validate improvements and prevent regressions
• Sanity and regression are part of our automated CI/CD pipeline
• Ongoing evaluation — monitoring f-scores and analysing misclassified cases to refine models continually

By maintaining deterministic outputs we avoid regressions and ensure high f-score accuracy. This approach has allowed us to build trust with clients, who rely on consistent and accurate results for critical decision-making. Our production system is deterministic from the client’s point of view: if an identical chat is sent several times over, the result is the same every time.

Evaluating production

We track production f-score over time. We can see how it improves after releasing fixes, or drops sharper when encountering new edge cases, or gradually decreases as it falls out of alignment while the language in the chats evolves. Our in-house analysis tools let us analyse this data — click on an individual score and drill into individual chats, see which metrics failed, and explore a side-by-side comparison of expected versus predicted results.

We also run a regression test to compare how two different model versions perform against the same test set. We run this before every release. It gives us comprehensive insight into how the new model differs from the previous one and how it would impact the production system.

Security and data governance

Client expectations

Data security is paramount in financial markets. As a data-driven organisation, data is at our core. This is why each client operates in a segregated environment, with unique API URLs, databases and access controls. Internally, access to client data is limited to essential personnel, ensuring confidentiality and compliance with regulatory standards.

Backup strategies

We adopt an aggressive approach to data backups, in places exceeding the traditional 3-2-1 rule — three copies, on two different media, one off-site. Multiple copies are stored across different media, with rigorous testing to ensure integrity. Long-term backups enable recovery from issues that may only surface months later, safeguarding our most valuable asset: data.

Our backup strategy reflects a deep commitment to data resilience. By implementing redundant systems and regular validation, we minimise the risk of data loss and ensure continuity in the event of unexpected disruptions.

Building for the future

We train in Europe — all data ends there — and we can run inference in or outside of Europe. Clouds give us this means to be flexible with the production system location, with data copied to Europe for training.

Geographic expansion

Currently our data centres operate within the EU, adhering to regional regulations. As we scale globally, compliance with varying data laws and infrastructure demands will become a significant focus. Expanding into new regions requires careful planning, particularly in navigating cross-border data transfer regulations and local operational requirements.

Conclusion

The journey of building a scalable and reliable AI system for financial markets is a continuous process of learning and adaptation. By focusing on robust architecture, thoughtful hardware decisions and rigorous data curation, we have created a system capable of delivering high accuracy and reliability. As we look to the future, the challenges of scaling, securing data and refining models will guide our efforts toward innovation and excellence.

Our commitment to reliability, scalability and security has positioned us as a trusted partner in the financial markets domain. Through ongoing investment in technology, infrastructure and processes, we aim to stay at the forefront of AI innovation.