I have a datamap which uses a preprocessor step. Every time I go to edit the datamap or its associated template, I first have to go to the preprocessor step, select the preprocessor script and click on Apply.
This doesn’t seem right. Should not the default action be to apply the preprocessor steps? Am I missing something?
That behaviour is by design. A pre-processor step could contain one or more pre-processing procedures, and each of them could require a certain amount of time to run. Running them automatically when you load the data mapping config or each time you make a change to the config could quickly become very annoying if it prevents you from navigating/modifying the config while they run.
Also, some pre-processors do not need to run every time you open a config. For instance, say your pre-processor converts an incoming file from JSON to XML. The best way to proceed would be to run it once and save the resulting XML. Then, use that XML file as an additional sample file for your data mapping config, so you can use it to design the rest of the DM config. Your list of sample files could conceivably contain several versions of your data in different states, as each pre-processor runs.
At runtime, obviously, all the pre-processors will run in sequence so that’s not an issue.
Now there are some cases where the pre-processor is used to augment the incoming data (for instance, by querying a database to obtain additional information). It’s true that for such cases, you have to run the pre-processor manually.
For those cases, we were thinking of adding an option that would allow you to specify which pre-processors you want to run automatically. That way, you’d be in control of whether or not you want to incur the potential delays caused by running those pre-processors automatically.
I’d be interested to learn if other members of this forum think such a user option would be valuable.
Time to speak up, folks!
1 Like
Yes please 
Also, to be clear, I would want this automated preprocessor to only run when the data mapper is opened. Therefore not needing to go to the mapper, select the unchecked preprocessor script and then click apply.
Regards,
S
Philippe
Thanks for the explanation. Saving half-processed data had not occurred to me.
Your proposed option seems like a fine idea.
Stuart