Mainframe Copybook to NoSQL

Mainframe Copybook to NoSQL

Posted on March 31, 2014 0 Comments
Person using a tablet

My colleague Ken Krupa recently found himself time-traveling between his early grunt years as a COBOL programmer and his current role as chief architect of an Enterprise NoSQL company. A meeting with a (huge) organization, heavily reliant on (massive) amounts of data generated from legacy mainframes had him playing with COBOL “copybooks”to bring them into MarkLogic.

COBOL has been around a few more years than Ken and was at the heart of many mission-critical systems across virtually every sector, from banking to manufacturing and healthcare to retail. Much of the structure lives in those aforementioned copybooks — sometimes embedded inside of the COBOL programs themselves. Getting this data into a NoSQL database would take the tarnish off this legacy data and spin it into a valuable commodity.

Legacy data structure from COBOL copybook.
Legacy data structure from COBOL copybook.

How hard was it to leapfrog from one architecture to another? “With MarkLogic and a set of open source Java libraries it was easy to ingest such mainframe data and also maintain the structure from the copybooks in a self-describing way via XML,” Ken told me. “More important however, having the data fully indexed in MarkLogic, regardless of the shape or structure of the data allows companies to glean new insights into such data like never before.”

Conversion of mainframe data files as defined by copybooks into a document-store like MarkLogic is a natural progression that was never well-suited in relational technologies.

Elegant XML is ingested into MarkLogic for easy analysis of legacy data.
Elegant XML is ingested into MarkLogic for easy analysis of legacy data.

“Copybooks are hierarchical by nature, as is XML,” Ken explained. “Also repeating items that might appear in an OCCURS clause of a copybook are handled easily with XML but are troublesome with relational. Additionally the poly-schema capabilities of NoSQL (e.g. different shapes of things with the same name) map well to the REDEFINES clause of copybooks. And then all of the typical stuff like not having to create tables or pre-define a landing place for the data just highlights the advantages further.”

An avid blogger, Ken chronicled his journey down memory lane — and  finding the right librariesto work his magic (and impress the prospect). In part 2 he shows the steps every step he took — including obfuscating actual data  by creating  a sample data set using custom copybook creators, the configuration of MarkLogic that anyone can download, mapping from COBOL to XSD. Questions? feel free to reach out to Ken via Twitter @kenkrupa.

Diane Burley

Responsible for overall content strategy and developing integrated content delivery systems for MarkLogic. She is a former online executive with Gannett with astute business sense, a metaphorical communication style and no fear of technology. Diane has delivered speeches to global audiences on using technologies to transform business. She believes that regardless of industry or audience, "unless the content is highly relevant -- and perceived to be valuable by the individual or organization -- it is worthless." 


Comments are disabled in preview mode.

Sitefinity Training and Certification Now Available.

Let our experts teach you how to use Sitefinity's best-in-class features to deliver compelling digital experiences.

Learn More
Latest Stories
in Your Inbox

Subscribe to get all the news, info and tutorials you need to build better business apps and sites

Loading animation