In short, I’m reading a CSV file
I do a loop with an iteration of 500
I read the 500 records for specific fields, and add up the number of page. Everything works, but if I have more than 500 records, the rest will be dismissed, I don’t want to put an absurd number like 50000 just to make sure I get all entries. So how can I get the number of record in the data, is there a variable for it? Or how can I define a new variable with the number of records so I can loop trough all records without missing anything?
This is easily achieved with metadata. Instead of a Loop task, add a Create Metadata task and set it to Passthrough mode. This will create, by default, one metadata DataPage element per record in your CSV file. Then, immediately after the Create Metadata task, add a Metadata Sequencer task and set the level to Data page, with the sequencing based on every 1 instance.
The sequencer will act as a loop, repeating the rest of the branch for every single DataPage element found in the metadata. The only difference in the rest of your process is that your data selections no longer need to be incremented using the {%loop} variable since the sequencer will always be using one record at a time. So you no longer the Set Job Infos task either. Regardless of the number of records in your CSV files, be it 10 or 20000, this will work.
In the end, your process will look something like this:
