Services

Move MongoDB Data into Elasticsearch via JDBC Driver

Introduction

Elasticsearch is a search engine that can be used to search all kind of documents in a near real-time. MongoDB, on the other hand, is a very popular NoSQL database which uses JSON data model for storing your data.  A common architecture around these two is using MongoDB for storing your data and Elasticsearch for searching the data. In this tutorial, we will walk you through on how you can import or migrate data from MongoDB to Elasticsearch for your analysis using the Progress DataDirect MongoDB JDBC connector.

Prerequisites

Before you go ahead with the tutorial, we assume that you have access/ installed the following.

  1. Elasticsearch 2.3.4
  2. MongoDB
    1. If you don’t have data, you can use the Restaurant sample data provided by MongoDB here, which will be used in this tutorial. 

Install the Progress DataDirect MongoDB connector

  1. Download Progress DataDirect MongoDB connector from here.
  2. Extract the PROGRESS_DATADIRECT_JDBC_MONGODB_WIN.zip and run the PROGRESS_DATADIRECT_JDBC_INSTALL.exe to install MongoDB connector.

Configure JDBC importer for Elasticsearch

1. Download the JDBC importer for Elasticsearch from here and extract the contents.

2. Copy the Progress DataDirect MongoDB JDBC connector (mongodb.jar) from C:\Program Files\Progress\DataDirect\JDBC_60\lib to \\path\elasticsearch-jdbc-2.3.4.0\lib folder

3. Now go to the path \\path\elasticsearch-jdbc-2.3.4.0\bin and create a new bat file with following script and save it.

 

@echo off

set DIR=%~dp0
set LIB=%DIR%..\lib\*
set BIN=%DIR%..\bin

REM ???
echo {^
"type" : "jdbc",^
"jdbc" : {^ "url" : "jdbc:datadirect:mongodb://:27017;DatabaseName=;TransactionMode=Ignore",^
"user" : "",^
"password" : "",^
"sql" : "select BOROUGH, RESTAURANT_ID, CUISINE, NAME, _ID from restaurants",^
"elasticsearch" : {^
"cluster" : "elasticsearch",^
"host" : "localhost",^
"port" : 9300^
},^
"index" : "restaurants"^
}^ }^ | "%JAVA_HOME%\bin\java" -cp "%LIB%" -Dlog4j.configurationFile="%BIN%\log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"

 

 

Import Data

  1. After you have established the connection, go to Elasticsearch install folder and run it if it’s not already running by using the command

     

    bin\elasticsearch.bat

     

  2. Go to the \\path\elasticsearch-jdbc-2.3.4.0\bin and execute the batch file you just created to start importing data from MongoDB to Elasticsearch. If everything is successful, the jdbc.log file in log folder shouldn’t have any errors.

Search Your Data

Now that your MongoDB data is in your Elasticsearch node, you can begin searching it. Some examples for your reference.

(These links will work only after you follow the steps above to set up your local Elasticsearch environment.)

http://localhost:9200/restaurants/

http://localhost:9200/restaurants/_search

http://localhost:9200/restaurants/_search?q=CUISINE:Chinese

Progress DataDirect provides the only SQL connectivity solution that exposes MongoDB data in the way SQL applications expect–as normalized relational tables–and the only solution that supports normalizing all data in your MongoDB database regardless of the depth and complexity of the documents stored. It’s relational connectivity to MongoDB data with no compromises. Try it free for 15 days.

JDBC TUTORIAL

Move MongoDB Data into Elasticsearch via JDBC Driver

jdbc

DataDirect
ODBC connectors

Real-Time Access to XML Data

community

DataDirect
ODBC Tutorials

Discussions, tips and tricks for
DataDirect Connect drivers