Redshift challenge to load 1M rows via ODI/JDBC WITHOUT S3 buckets | LIVE at #OOW14

Redshift challenge to load 1M rows via ODI/JDBC WITHOUT S3 buckets | LIVE at #OOW14

September 15, 2014 0 Comments

Welcome to the Amazon Redshift insert challenge (no s3 or ice buckets).  So LIVE at Oracle OpenWorld 2014 #OOW14, I have been challenged by data professionals to load 1 million records into Amazon Redshift from Oracle Data Integrator WITHOUT using an S3 bucket during a live #OOW14 session: Oracle Data Integration: A Crucial Ingredient for Cloud Integration [CON7926]:

Beyond Redshift | Oracle and Cloud Data Integration:

Amazon Redshift is just one cloud data source, so we'll also demo and talk about connecting various cloud data sources to build an Oracle data warehouse with ODI listed below.  We know this is a hot topic since my last two sessions on Salesforce data integration at Dreamforce 12 and 13 both sold out.

  • Oracle Marketing Cloud (Eloqua)
  • Oracle Service Cloud (RightNow)
  • Marketo
  • Microsoft Dynamics
  • Google Analytics / Adwords

Challenge not looking good so far ...

  • ODI Job: Loading 1M records into the supplier sample table in Amazon Redshift
  • Work tables are created in Redshift target for ELT
  • Source data is in an Oracle 11g Database
  • Data Integration platform is Oracle Data Integrator 12c
  • Open source 8.4 PostgreSQL JDBC driver
  • Everything is running on a local OracleVM

Since 1 million rows takes awhile, I put a filter to only load 40,288 rows which is taking 790 seconds or ~13 minutes.  We're looking at close to 6 hours to load 1M at this rate. odi-redshift1



How do I get myself out of this?

  • Is it even possible to get this load within 10 minutes?
  • Moscone may have flaky wifi?
  • Laptop could be confiscated by TSA!
  • May fall off the GG bridge during the 5K OpenWorld Bridge Run Sunday.

But I'm not worried about it since I am backed by the greatest R&D team for data connectivity in the world at Progress DataDirect; and plan to use the new DataDirect Amazon Redshift JDBC Driver.

If we pull this off, I will ask Julien Testut, Principal Product Manager for ODI, who is hosting the session to buy everyone in the audience a local treat (Morning Bun or Cup of Philz?).

Learn more about how DataDirect makes the impossible possible in data connectivity

Get started with a trial download today.

Or tweet to @DataDirect_News, talk to us on live chat or call 1-800-876-3101.

Sumit Sakar

Sumit Sarkar

Sumit Sarkar is a Chief Data Evangelist at Progress, with over 10 years experience working in the data connectivity field. The world's leading consultant on open data standards connectivity with cloud data, Sumit's interests include performance tuning of the data access layer for which he has developed a patent pending technology for its analysis; business intelligence and data warehousing for SaaS platforms; and data connectivity for aPaaS environments, with a focus on standards such as ODBC, JDBC, ADO.NET and ODATA. He is an IBM Certified Consultant for IBM Cognos Business Intelligence and TDWI member. He has presented sessions on data connectivity at various conferences including Dreamforce, Oracle OpenWorld, Strata Hadoop, MongoDB World and SAP Analytics and Business Objects Conference, among many others. 

Comments are disabled in preview mode.
Latest Stories in
Your Inbox
More From Progress
The New Mobile Development Landscape
Download Whitepaper
IDC Spotlight Sitefinity Thumbnail
Choosing the Right Digital Experience Platform to Improve Business Outcomes
Download Whitepaper
The Fastest Way to Build Mobile Apps With Cloud Data
Watch Webinar