One of the most popular interfaces for exposing and consuming data is using a REST API, and one of the best ways to store data at a low cost and analyze the data later is HDFS.
Data Architects encounter scenarios where the business wants to store data from REST API’s as they get updated information, so that the BI/Analytics team can run analytics on top of this data. To do this currently we need a custom plugin/solution to bring that data from the REST API to HDFS. Wouldn’t it be great if you could use an existing tool like Sqoop to bring in the data from REST API with out having to write a single line of code?
That is exactly what we are going to do with the help of Sqoop and Progress DataDirect Autonomous REST Connector. Autonomous REST Connector allows you to query API data using standard SQL by normalizing the semi-structured JSON response data into structured data that can be used with SQL.
In this tutorial we will use Autonomous REST Connector with Sqoop, where Autonomous REST Connector will connect to Alpha Vantage REST API (this is an example, you can use any REST API) to get stock exchange data. Sqoop will ingest this data into HDFS for your future analytical needs. Let’s take a look at how you can do this.
Before you start the tutorial, we assume you have the following environment and tools setup:
Note: Please use your api key in the #default property under apikey json object.
It’s so easy to import any REST API to HDFS using Progress DataDirect Autonomous REST Connector and you won’t have to write a single line of code to do so. Feel free to try out Autonomous REST Connector with any API. If you have any questions, feel free to talk to us.