Retry for write actions in the Workflow

OL Connect Professional Workflow
1

Some time ago we moved our PlanetPress servers to Azure. Eventually more and more systems will move to there but at the moment we have a hybrid environment where we still have a server rack in a data center and a couple of servers in Azure, all connected via a VPN connection.
We have used OL Connect for a lot of years now and there were never issues with writing information to a storage location or sending/receiving information via an API but lately I noticed that we get small network issues from time to time (if this is due to the VPN or just the nature of having part in Azure and another part in a data center or something with latency I do not know).

So, now I am in the process of thinking out some kind of retry option but unfortunately OL Connect Workflow is not designed with this in mind, so the relevant plugins (e.g. Send To Folder, Load External File) have no ability to do a retry when it fails. Sure, I can call an Error process when a plugin fails and from there take some actions to maybe retry the process but it is still a quite some work. Of course it is also possible to have a script retry something a couple of times until it finally fails or succeeds (with a try/catch or something). But this has the problem that it will hold up the rest of the process until the script is finally completed. And that can take some time with a back-off period for example.

My idea (which I have implemented for the most part already) is for every file that is going to be stored/archived I place it in a queue for a sub process who handles that. This has the advantage that other processes do not have to deal with the archiving stuff and I can have retry functionality in 1 place only. Also, whenever there is a serious network issue everything can stay in that queue until it is resolved and once it is, the backlog can be processed again.

Unfortunately having to write scripts for everything file related does add to the complexity of all my workflows and dragging + dropping a simple “Send To Folder” plugin will no longer work anymore. Of course I have written some global Javascript functions that I can call from any process to make it simpler so I do not have to write the same code everywhere.

I am curious if there are other/better ways of handle things properly when a network error occurs during a write action for example?

Thanks,
Dennis

2

Hello @dvdmeer,

A possible solution is to:

  1. Write both the folder path (to the network location) and the file name (that needs to be send to the network location) to a Job Info variable.
  2. Use an error management process that will store the file forwarded in a specific folder and write both the folder path and file name to a text file and save this text file in a specific folder.
  3. Create a separate process that will pick up the text file, extract the folder path and file name from the text file and tries to send the file to the same network directory again. Make sure that step 1 is also implemented in this separate process, should an error occur again.

“[…] and tries to send the file to the same network directory again.”—This part can be achieved by executing the Folder Capture and Send To Folder tasks after each other.

Update: You may also need to built in some kind of count and to stop retrying when the count has reached a certail value. Otherwise, an infinite loop occurs.