Polling unreliable systems

Use this scenario when polling a source that may be temporarily unavailable or is expected to periodically return errors.

This basic retry strategy can be used to handle failures in sources, destinations, scripts, and other task steps.

How It Works

The Repeat step allows iteration errors to occur without immediately failing the task.

When an iteration fails:

• The error is logged.
• Remaining steps for the current iteration are skipped.
• The Repeat step continues with the next iteration.

The task fails only after the configured number of consecutive iteration errors is exceeded.

Configure the Task

1. Add a Repeat step around the steps that communicate with the unreliable source.

2. Enable Continue on Iteration Error.

   This setting allows the Repeat step to continue running when an iteration encounters an error, rather than immediately failing the task.

3. Set Maximum number of consecutive iterations with error to the desired value.

   For example, set the value to 3 to allow up to three consecutive iteration errors before the Repeat step fails and the task ends.

4. Within the Repeat step, add a Download step for the source being polled.

5. Set Collect only files newer than the last successfully processed file to true.

   This setting ensures that each iteration collects only files that are newer than the last successfully processed file, preventing the same files from being processed multiple times.

6. Within the Repeat step, after the Download step, add an Upload step and configure the destination where the collected files should be uploaded.