The efficiency of every day operational processes can be improved by removing older unused data into a separate system or archive. In regulated industries such as Financial Markets, there are rules governing how long this old data or records need to be kept and also ensuring that historical information cannot be changed.
Firms often use external services to archive voice, chat, and email. They may well have more than one enterprise content management system to handle office files and scanned images, with formal workflows to produce records. The most difficult area is system applications that typically use RDBMS. These may well be archived into separate relationally structured data warehouses or have application code to write records out, but of course each will have its own set of formats, tables and keys. Long-term storage for these platforms will generally include Write Once Read Many (WORM) storage, to meet Sec 4a immutability rules, or cheaper storage such as Hadoop.
Use of this historical data generally falls into two requirements. First, from a business intelligence viewpoint, older operational data can provide a rich data lake for statistical mining. Second, older information may be needed for investigations – either regulatory or legal. In the latter, investigations can lead to extending the safekeeping time for the dataset, sometimes known as a legal hold.
Generally, internal audit is tasked with making sure processes are being maintained properly.
One forensic firm addresses an “event” (criminal, civil, regulatory, or otherwise) by taking samples from a range of different systems — spreadsheets, records of transactions, documents and even communication to try and piece the transaction history together. These types of firedrills are disruptive and expensive. If the internal team can’t pull it together, an external firm will be brought in to help re-assemble — at a great cost.
So recreating histories is near impossible as long as data silos (in this case, data archive silos) persist.
Based on my 30+ years of experience in financial markets, I can identify four major challenges to solve:
To deal with yesterday’s, today’s and tomorrow’s requirements you need to architect a platform that can handle archive data as you would operational data.
Instead of creating yet another series of archive silos, consider creating a compliance archive on a common infrastructure as your Regulatory Reporting Platform. If the platform is built on a multi-model platform, you can create a single archive with mixed formats and multiple schemas.
This multi-model database platform that allows you to load data without upfront ETL — and instead lets you develop a data harmonization layer where data can either be transformed or, in the case of preserving data integrity, metadata is created. By creating metadata to harmonize, you can provide consistent data terminology while maintaining governance. Through the use of MarkLogic’s Universal Index, you can search on values and terms that were not predetermined. Any reporting system needs to have fine-grained security controls on who can see what, and of course redact fields if necessary. Finally, you need your platform to integrate with Hadoop and WORM for cost-effective and compliant storage.
For three, top-tier investment banks, one asset manager and one of the top three global brokers, MarkLogic’s database platform is the choice to satisfy MiFID II requirements like transaction reporting, transparency reporting, and record keeping. It is the ideal infrastructure for providing a datastore for compliance archives – that also can be extended to a full Regulatory Reporting Platform. In fact, the aforementioned customers have extended their regulatory frameworks to comply with other regulations, e.g. Dodd Frank, MAS 610 and GDPR and also get in control of any upcoming regulations.
And unlike solutions built on top of relational databases, MarkLogic’s data integration approach offers the flexibility, cost efficiency and faster time to market needed to address current, future and additional regulatory requirements. We consistently out-perform benchmarks:
Archives are going to remain a fact of life for financial services. But they don’t have to be the (costly) headache they used to be.
View all posts from Don Butler on the Progress blog. Connect with us about all things application development and deployment, data integration and digital business.
Let our experts teach you how to use Sitefinity's best-in-class features to deliver compelling digital experiences.Learn More
Subscribe to get all the news, info and tutorials you need to build better business apps and sites
You can also ask us not to share your Personal Information to third parties here: Do Not Sell or Share My Info
We see that you have already chosen to receive marketing materials from us. If you wish to change this at any time you may do so by clicking here.
Thank you for your continued interest in Progress. Based on either your previous activity on our websites or our ongoing relationship, we will keep you updated on our products, solutions, services, company news and events. If you decide that you want to be removed from our mailing lists at any time, you can change your contact preferences by clicking here.