Move Big Data, Not Bottlenecks: Multipart Upload/Download for AWS S3 in the Progress DataDirect Autonomous REST Connector

In today’s data-driven world, cloud storage is no longer optional—it’s essential. Organizations rely on platforms like Amazon S3 for durability, scalability and cost efficiency. With support for objects up to 5 TB, S3 has become the backbone for data exchange, analytics and ETL workflows.

But moving large files between systems and S3 can be challenging. Network interruptions, timeouts and memory limits often stand in the way of smooth transfers. That’s where the Progress DataDirect Autonomous REST Connector (ARC) steps in.

Why AWS S3 Matters

AWS S3 is more than just storage; it is a strategic layer for:

• Data pipelines: Acting as an intermediary for bulk data movement
• Analytics workflows: Storing large datasets for BI and ETL processes
• OEM integrations: Enabling scalable, secure data exchange between systems

What’s New in the Progress DataDirect Autonomous REST JDBC Connector?

The Progress DataDirect Autonomous REST JDBC Connector had existing support for file upload and download to S3, but transfers were previously limited to 2 GB due to memory constraints. With our latest enhancements, DataDirect ARC supports multipart upload and download, making large file transfers seamless and more reliable.

Key Features

• Multipart Upload: Automatically splits files larger than 100 MB into parts and uploads them in parallel, following AWS best practices
• Multipart Download: Uses byte-range requests to download files in chunks, enabling efficient retrieval of objects up to 5 TB
• Configurable Chunk Size: Fine-tunes performance using the new connection property, FileTransferPartSize
• Robust Error Handling: Retries logic and detailed logging for operational visibility

How It Works

You can configure this feature using:

• Connection Property: FileTransferPartSize

• New Stored Procedures:

  • getObjectSize – Retrieves the size of an object in S3

  • uploadFile – Uploads a file from the local system to S3

  • downloadFile – Downloads a file from S3 to the local system

  For more details, view the official documentation.
  

What This Means for You

The Connector affords organizations like yours:

• Scalability: Moves files up to 5 TB without memory bottlenecks
• Reliability: Reduces the risk of failure with the allowance of retries via multipart operations
• Integration: Leverages S3 for robust data exchange in OEM solutions that embed DataDirect ARC

By bridging REST APIs and cloud storage, Progress DataDirect ARC strengthens its position as a versatile connector for modern data architecture. Learn more. 

Also, watch this webinar for more on how to connect any REST Data Source using the Progress DataDirect Autonomous REST Connector.