Using Looping in Automations to move 450 records from Staging Table to Production Table

[steve.midcalf]

Hi everyone,

I have a use case with Automations where I want to move 450 unit serial numbers from a Staging table (written to by a 3rd party vendor using the Table API when a work order is released/modified from ERP system).

The Automation is triggered by a record being added to a trigger table with the ID of the work order.

I am able to use a connector function to get the list of records filtered by work order number, and then start looping through them but I get an Error message “Limited by rate”.

My guess is that the 60 second timeout is what the issue is. I’ve gotten around this by limiting the first Automation to only process 100 records and then making a 2nd Automation for the next 100 and then cascading them by having the final step of the first Automation be to add a record to another table to trigger the second one.

Some of these work orders have 450 units on them so I’ll need to make a 3rd, 4th, and 5th Automation (and 3rd, 4th, and 5th trigger Table) to process 100 records each to move them all over.

Hopefully this image captures the complexity of cascading Automations from new trigger Table writes.

Just wondering if anyone else has encountered this issue and if this is the best approach to solve it or if there’s a better way.

Thanks!

[steve.midcalf]

During testing I found a better way to do this.

Instead of writing to a second trigger table and using a second Automation (and a 3rd, 4th, & 5th), the last step if there’s more records to process is to delete the original trigger record and then re-add it, which will trigger the Automation again as many times as needed to finish all the records.

So it basically flows like this:

This also gets around the issue of what if the next order has 600 records? Or 700? The above method of cascading 100 at a time only works if you have enough Automations and trigger tables to handle all the records in advance.

I’d be happy to share more details if anyone else has this same use case - comment below or send me a message!