Move Big Data, Not Bottlenecks: Multipart Upload/Download for AWS S3 in the Progress DataDirect Autonomous REST Connector
In today’s data-driven world, cloud storage is no longer optional—it’s essential. Organizations rely on platforms like Amazon S3 for durability, scalability and cost efficiency. With support for objects up to 5 TB, S3 has become the backbone for data exchange, analytics and ETL workflows.
But moving large files between systems and S3 can be challenging. Network interruptions, timeouts and memory limits often stand in the way of smooth transfers. That’s where the Progress DataDirect Autonomous REST Connector (ARC) steps in.
Why AWS S3 Matters
AWS S3 is more than just storage; it is a strategic layer for:
- Data pipelines: Acting as an intermediary for bulk data movement
- Analytics workflows: Storing large datasets for BI and ETL processes
- OEM integrations: Enabling scalable, secure data exchange between systems
What’s New in the Progress DataDirect Autonomous REST JDBC Connector?
The Progress DataDirect Autonomous REST JDBC Connector had existing support for file upload and download to S3, but transfers were previously limited to 2 GB due to memory constraints. With our latest enhancements, DataDirect ARC supports multipart upload and download, making large file transfers seamless and more reliable.
Key Features
- Multipart Upload: Automatically splits files larger than 100 MB into parts and uploads them in parallel, following AWS best practices
- Multipart Download: Uses byte-range requests to download files in chunks, enabling efficient retrieval of objects up to 5 TB
- Configurable Chunk Size: Fine-tunes performance using the new connection property, FileTransferPartSize
- Robust Error Handling: Retries logic and detailed logging for operational visibility
How It Works
You can configure this feature using:
Connection Property: FileTransferPartSize
New Stored Procedures:
getObjectSize – Retrieves the size of an object in S3
uploadFile – Uploads a file from the local system to S3
downloadFile – Downloads a file from S3 to the local system
What This Means for You
The Connector affords organizations like yours:
- Scalability: Moves files up to 5 TB without memory bottlenecks
- Reliability: Reduces the risk of failure with the allowance of retries via multipart operations
- Integration: Leverages S3 for robust data exchange in OEM solutions that embed DataDirect ARC
By bridging REST APIs and cloud storage, Progress DataDirect ARC strengthens its position as a versatile connector for modern data architecture. Learn more.
Also, watch this webinar for more on how to connect any REST Data Source using the Progress DataDirect Autonomous REST Connector.
Kumar Vikesh
Senior Technical Product Manager
Kumar has been at Progress for the past 15 years, having worked as a developer, engineer and most recently as a Technical Product Manager.
Projects
MYSQL-UDT: The project involves design and implementation of new SQL statement for User Defined data type (UDT) for distinct types. The statements would be ANSI SQL standard statement. The aim of the project is to do the changes to MYSQL database engine to support the distinct User Defined Types. This project will add new code to the different component of the database engine, such as syntaxes, parser, and executor modules as appropriate.
Environment: Linux
Role: Involved in Preparing Requirement specification, Preparing High Level Design and Low level design for the project. I developed the module that will create UDT, Drop UDT and use UDT in Create table, alter table etc.
SQL Grammar Test Framework: This is a framework that generates SQL in an automated manner. The objective of this framework would be as follows: Complementing the feature and systems integration testing that each of the database solutions providers may already have, as part of their testing process.
Environment: Linux
Role: Involved in preparing the requirement specification, design documents.I worked on the development of UI part and Query generation module using BNF grammar file.
MATRIX: Developing an antivirus solution for the Vmware virtual infrastructure. The Software provides security against malware (virus, spyware, Trojans etc.) for OSes running within the virtualization environment. The technologies involved are C/C++. Target platforms are Windows Server 2003 and for RTA (Real-time agent) component Win2k, 2k3, XP, Vista for 32/64-bit platforms.
Environment: Windows, VMware
Role: Developed the Update Module of the software. This also involves Reading and Modifing the registry, mounting the Registry Hive of a remote virtual machine into the local registry. Implementing the Proxy setting page for getting pattern and engine updates. VmWare installations.
Specialties
C, C++, VC++, Unix, System Programming Windows, Win32 SDK Prograaming, MFC, PERL Scripts, GDB, NETBEANS, Vi editor, Database Internals, Antivirus Software, SDLC, Rational Rose
