Bulk create records - with template

Hi,

I’m making a product parameter database managing application.
I’d like to have the possibility to add new products (± 215 parameter records)

I’ve made a connector function with inputs for all the fields of the record.
I made a loop where I use an aggregation (Unique ID) of my table query (filtered by a template PRODUCT and by an INDEX column). I pop the ID’s off the aggregation array & load the record, which results in an iteration of the queried table. Every iteration I create a new record with the values of the template record. But this can only loop a certain amount because of the trigger limit (101 times).

Any ideas on how to approach this?

I found a way by going to a different step (with a button) if the counter reaches it max, the button resets the counter and goes back to the loop step. I do this until the array length is <=0.

Seems that the trigger limit takes in account all events that are sequential, not only the events in the one step. Or I’m missing something. So breaking the loop sequence is the way to go for now.

Hi @stijn.devriendt,

By default, Tulip has a limit of 100 triggers that fire sequently off one another (i.e. triggering a step open trigger off of a transition). This is to avoid infinite loops in your application. At the instance level, we can extend that limit beyond 100 triggers within reason (typically a few hundred triggers).

Do you know how many triggers you would need? If you send me a message with your instance name and approval from an Account Owner, I’d be happy to extend this - it doesn’t require any downtime. I do want to reiterate that limit is for all apps, so if you change it and had previously built apps relying on that limit, they could be impacted.

Thanks!

Hi Grant,

I completely understand the need for a limit. But I was wondering if it would be possible to have this limit as a developer parameter accessible for the account owner. Or even have the option to have unlimited triggers but with a timeout break (also a potential parameter in the developer options) in the trigger loop.

On a separate topic, is it possible to fire python scripts with connectors? Or relay data to containers running on a local server? I have a gateway set up on the network this server is running on.

Thanks for the quick response.

Hi Grant,

Sorry I forgot to mention my instance:

Iff-Leper.tulip.co

PS. The name of our site is in fact Ieper, not Leper. I guess there was a lowercase typo-misread. I tried changing it in the options but this didn’t change anything.

Hi @stijn.devriendt,

I see you’re an Account Owner - do you want me to extend the trigger limit for your environment? Just let me know what you’d like me to extend it to (would recommend <500 for now). We’re also working on formally supporting looping in the platform to provide an easier mechanism to run actions like this.

You certainly could fire Python code from a connector function! You might want to look into Flask in setting up an endpoint that you could hit and would need it running on a reachable server. You could also look into something like Lambda functions in AWS to run from an HTTP request.

Let me know if that makes sense/answers your question.

1 Like

Hi, @Grant Levy,

Yes I’d like to have a trigger limit of 300 if possible. A feature like this would be greatly appreciated by the whole community I’d suppose!

I will have to take a look into setting up a flask environment, lambda function is a bit to limited for my use-case I think.

I’m actually setting up functions that take data from a linewalk in Tulip (pictures & comments) & turns this into a PowerPoint presentation on our server.

While also registering this linewalk in our internal database.

Maybe on a broader note, Is there a replication function to copy the tulip tables to an SQL database on some sort of tickrate with a connector?

I would like to have a local copy of the database at all times, because I’d like to relay some data to our PCN/OMN network (which requires a firewall exception) at some point.

Hi @stijn.devriendt,

I’ve gone ahead and increased your limit to 300 triggers. We don’t have the ability to trigger a replication function, but you could look at running a number of GET requests from a scripted service every time period (minutes, hours, days, etc.) and write the return to a SQL database. This would help keep your data up to sync without requiring a write for each event.

Let me know if that makes sense!

Hi @stijn.devriendt,

May I ask what you are using to accomplish this looping? I am trying to to do something similar in my instance (where I pop from an array of a variable length and create new records) but have yet to find something that is suitable to do the loop. Any help is appreciated. Thanks!

Hi @marcusreed98,

The most widely used form of looping in Tulip apps today (that I’m aware of) typically follows along the general principle of using step open triggers to repeat actions while iterating through an array. It sounds like you have your array already in mind, so I’ll try to describe more of the first part.

To start the loop, you may use a button, timer, or some other source of a trigger to initiate the action. In this block, users will typically set an iterator (i.e. an index variable to 0) and a length they want to run the loop for (may be length of array or total number of loops). This then uses a transition event to move to the looping step.

On the looping step, an on step enter trigger can be built where you:

  1. Check condition if index variable < length, if yes:
  2. Get array element by index variable
  3. Do whatever steps you want to implement in the loop (i.e. write to a table)
  4. increment the index variable by one
  5. Transition to the same looping step, which will re-execute the step open action
  6. If condition from step #1 is false:
  7. break out of the loop by moving to next step or not running transition from step #5

As mentioned above, we have limits for how many of these step open triggers can be run consecutively before Tulip exits the loop. However, it’s very easy to manually continue as you’ll have the index and length variable to still re-iterate through.

Let me know if that helps or if you have any specific questions!

Hi @marcusreed98,

Grant pretty much perfectly answered your question,

I use a table aggregation (unique fields) on a table query to determine how many records (loops) I need to create. The table consists of records which are parameters per product [ID][PRODUCT][PARAMETER].

So I query the table on unique products to add a parameter to all products = loop per unique product.

Or I query the table on unique parameters to add a product (with all parameter records) to the table. = loop per unique parameter.

This method can be used for all sorts of actions (deleting, creating, linking, loading, …)