This challenge was set forth on Sept-12 to the public (read article) to perform a 1M record load LIVE at an Oracle OpenWorld session on Oct-2 using Oracle Data Integrator titled, Oracle Data Integration: A Crucial Ingredient for Cloud Integration [CON7926]. The operation takes ~6 hours with an open source Postgres JDBC driver v8.4 and there was no proof this was even possible to do.
Here's the play by play which came down to the wire, and we ended up doing a little SaaS Cloud integration with Salesforce as well with the unexpected extra time :)
Hacking away with @brodymessmer at Duke's cafe at #oow14 for the Amazon Redshift challenge in < 48 hours #oow14 pic.twitter.com/mFg2lSuIde — Sumit Sarkar (@SAsInSumit) September 30, 2014
Hacking away with @brodymessmer at Duke's cafe at #oow14 for the Amazon Redshift challenge in < 48 hours #oow14 pic.twitter.com/mFg2lSuIde
(Probably should have skipped the beers in hindsight)
All eyes on Moscone South as the @awscloud redshift challenge from ODI begins at 9:30am in rm 270 #oow14 pic.twitter.com/zzmBRxbXaK — Sumit Sarkar (@SAsInSumit) October 2, 2014
All eyes on Moscone South as the @awscloud redshift challenge from ODI begins at 9:30am in rm 270 #oow14 pic.twitter.com/zzmBRxbXaK
It's getting interesting. @SAsInSumit kicking off 1 mn record load LIVE. #oow14 #odi12c #cloud — Madhu Nair (@nmadhu2k3) October 2, 2014
It's getting interesting. @SAsInSumit kicking off 1 mn record load LIVE. #oow14 #odi12c #cloud
Yikes! Redshift challenge by @SAsInSumit at #OOW14 to load 1M rows via ODI/JDBC WITHOUT S3 buckets http://t.co/SsIZpBNQ6c — Russell Rothstein (@RussRothsteinIT) September 29, 2014
Yikes! Redshift challenge by @SAsInSumit at #OOW14 to load 1M rows via ODI/JDBC WITHOUT S3 buckets http://t.co/SsIZpBNQ6c
And it worked over the #OOW14 wifi! @SAsInSumit #ODI12c #DataDirect — Michael Rainey (@mRainey) October 2, 2014
And it worked over the #OOW14 wifi! @SAsInSumit #ODI12c #DataDirect
Live demo for extracting data from #Salesforce using #ODI12c and DataDirect @SAsInSumit @JulienTestut pic.twitter.com/ku8QTvptEv — Irem Radzik (@Irem_Radzik) October 2, 2014
Live demo for extracting data from #Salesforce using #ODI12c and DataDirect @SAsInSumit @JulienTestut pic.twitter.com/ku8QTvptEv
@markrittman cool, this leverages the Progress driver ;-) — David Allan (@i_m_dave) October 2, 2014
@markrittman cool, this leverages the Progress driver ;-)
Sent a tweet with the results to the Amazon Redshift product team following the action from Seattle just after the session.
@SAsInSumit @rahulpathak @awscloud @ProgressSW awesome — David Allan (@i_m_dave) October 2, 2014
@SAsInSumit @rahulpathak @awscloud @ProgressSW awesome
And all your chatter made it to Larry Carvalho from IDC:
@robustcloud Great meeting you and you meant DataDirect did it under 10 min :) Cant wait to try the chai cart on market st next week #PRGS14 — Sumit Sarkar (@SAsInSumit) October 8, 2014
@robustcloud Great meeting you and you meant DataDirect did it under 10 min :) Cant wait to try the chai cart on market st next week #PRGS14
ODI Studio 12c showing session results for 1,000,000 row insert (total of 2,000,000 processed with ELT)
ODI Studio 12c showing mapping with ELT expression staging the 1M rows
1 week to go and first bead of sweat just rolled off - @awscloud Redshift 1M row challenge at #OOW14 http://t.co/ccH59KuBM2 via @ProgressSW — Sumit Sarkar (@SAsInSumit) September 25, 2014
1 week to go and first bead of sweat just rolled off - @awscloud Redshift 1M row challenge at #OOW14 http://t.co/ccH59KuBM2 via @ProgressSW
Download the DataDirect Amazon Redshift ODBC/JDBC driver and tell us you want to take the challenge so we can deliver you the latest patch since it may still be in QA. If you're not running Oracle Data Integrator, then give it a try against your data integration platform of choice which can leverage efficient loading via DataDirect:
Technology researcher, thought leader and speaker working to enable enterprises to rapidly adopt new technologies that are adaptive, connected and cognitive. Sumit has been working in the data access infrastructure field for over 10 years servicing web/mobile developers, data engineers and data scientists. His primary areas of focus include cross platform app development, serverless architectures, and hybrid enterprise data management that supports open standards such as ODBC, JDBC, ADO.NET, GraphQL, OData/REST. He has presented dozens of technology sessions at conferences such as Dreamforce, Oracle OpenWorld, Strata Hadoop World, API World, Microstrategy World, MongoDB World, etc.
Subscribe to get all the news, info and tutorials you need to build better business apps and sites
Progress collects the Personal Information set out in our Privacy Policy and Privacy Policy for California Residents and uses it for the purposes stated in that policy.
You have the right to request deletion of your Personal Information at any time.
You can also ask us not to pass your Personal Information to third parties here: Do Not Sell My Info
Let our experts teach you how to use Sitefinity's best-in-class features to deliver compelling digital experiences.
Copyright © 2021 Progress Software Corporation and/or its subsidiaries or affiliates.All Rights Reserved.
Progress, Telerik, Ipswitch, and certain product names used herein are trademarks or registered trademarks of Progress Software Corporation and/or one of its subsidiaries or affiliates in the U.S. and/or other countries. See Trademarks for appropriate markings.