Challenges in Data Access for New Age Data Sources

Challenges in Data Access for New Age Data Sources

January 23, 2013 0 Comments

The Big Data and Cloud “movements” have acted as catalysts for tremendous growth in fit-for-purpose databases. Along with this growth, we see a new set of challenges in how we access the data through our business-critical applications. Let’s take a brief look at the evolution of these data access methods (and why we are in the mess we are in today).

Back in the 80s the development of relational databases brought with it a standardized SQL protocol that could be easily implemented within mainframe applications to query and manipulate the data. These relational database systems supported transactions in a very reliable fashion through what was called “ACID” compliance (Atomicity, Consistency, Isolation, and Durability), which provided a very structured and reliable method of dealing with data. But ACID compliance also brings along a lot of overheard process. Hence a downfall – they are not optimized to handle large transaction requests, nor can they handle huge volumes of transactions. To counteract this, we’ve done some significant performance and throughput enhancements within data connectivity drivers that lights a fire under the SQL speeds and connectivity efficiencies.

Now move to the 90s and early 2000s, where vendors are experimenting with more efficient ways of storing the data and the advent of “NOSQL” (aka Not Only SQL). We now have multiple applications trying to access a database with new requirements for performance, scalability, and volume. These databases employ one of several different storage models:

  • Key-Value, designed to handle massive amounts of data
  • BigTable, column-oriented relational databases based on Google technology
  • Document, similar to key-value where the value is the document
  • Graph, scaling to the complexity of the data and modeling the structure of the data

These databases sacrifice transaction-oriented access for speed and scalability. However, there is no standardized, optimized access method like SQL. In general, the best way to query is through the REST API and Web services. Each NOSQL database usually has a proprietary method of accessing, but that causes frequent API changes to your applications when dealing with multiple databases.

And that brings us to the needs of today for multiple applications requiring access to multiple fit-for-purpose databases using alternate data storage models and different access methods. Here comes NewSQL, which is supposed to fill the gap left by NOSQL with better support for ACID transactions while retaining the performance and scalability characteristics. NewSQL fulfills the needs for today’s data markets with highly scalable, high-performance, transaction-intensive data store options. While the adoption of these NewSQL alternatives is slow, I expect to see a rise once more tools become available to support it. The challenge here is having to rewrite how we access this data. The access method is a hybrid SQL, so it will take some effort before more vendor tools and middleware drivers support it. Plus, the impact to application development will have to be considered, given the new ways required to access the data.

All of these – SQL, NOSQL, NewSQL (and more, like in-memory) – have a distinguished place in today’s world of data. Each is customized for fulfilling the needs of the megatrends like Big Data and Cloud Computing. They’ve opened up new markets for better access methods that have low impact on existing business-critical applications. And being able to connect to the world’s data in a fast and consistent fashion will continue to be keys to the data castle.

Jeff Reser

View all posts from Jeff Reser on the Progress blog. Connect with us about all things application development and deployment, data integration and digital business.

Read next OData FAQs: Why Should REST API Developers Use OData?
Comments are disabled in preview mode.