I have a project for property tax notice creation. There are about 100,000 records. After the data file generation, the workflow is starting to plug the data into data mapper and template. However the Executing Data Mapper took about 1.5 hours, the Create Print Content took about 9.5 hours, the Job Creation took about 14 minutes and the Create Output took at least 2.5 hours. We are using version of 8.7.1.12148 workflow and version of 1.7.1.4448 Connect Designer. The data file generated is about 2.8 GB. What possible issues could cause the slowness?
To make Connect run as fast as possible, the main thing to address is Content Creation speed.
Step 1 optimizing the template. Make sure to not only optimize the scripting, but also make sure the resources are optimized (for instance, no transparent images if they are not needed). For the scripting:
make sure that scripts are only enabled for the sections and master pages where they are needed
profile the scripts (Click on Context tab > Profile Scripts)
review any scripts that stand out in the profiling
sometimes a bottle-neck script can be handled by making sure the DOM is only updated once and by doing the data/text replacement with JavaScript search/replace
getting the target DOM element and changing that to text
running a JavaScript replace on the html text
setting the resulting html text as the html of the result
Assign a CSS ID or Class selector to each field placeholder placed on the page. Targeting\searching for CSS ID or Class selectors (or in combination with Text selectors) in the DOM is faster than searching for Text selectors only. Text selectors will be searched line by line in the DOM until the end of the document is reached for each record, whereas scripts will immediately exit once an ID is found, (since they are unique) and carry on with the next record instead of continuing the search until the end of the DOM is reached.
Where many sections and master pages share the same content and group of fields\variables, create these contents as Shared Contents and assign them CSS Classes. This not only make the form easily maintainable, but it also increases performance. Changes are only made to the shared snippets and these will automatically replicate to any master page or section which uses them
Step 2: run as many merge engines as possible on the given hardware. For print, the number of merge engines or their speed, is not limited in any way by the license.
if the system is allowed to run jobs one by one (so no complications by running multiple jobs simultaneous), as a rule of thumb, run as many merge engines as the system has full cores (so don’t use hyper threading cores for merge engines)
set your scheduling so a medium or large job gets all merge engines available.
run some tests, if the CPU is not hitting 100% during content creation, perhaps you can even run some more merge engines.
Add as much RAM to the system as you can afford. Note that each engine uses 640MB of RAM by default. So if you are going to increase the number of merge and weaver engines used at any one time and especially if you are going to increase the amount of RAM used by each weaver engines as explained in the link below, then you need to make sure the system has sufficient RAM.
If you can get above 1,000 ppm per Merge engine (check the merge engine log files, it mentions ppm for each run), then you are in good shape. I am not sure if your hardware will reach that: it makes quite a difference if you are running a Xeon @2.3GHz or @3GHz. In addition, as you have PlanetPress Connect, your license entitles you to up to 6 Weaver engines. Therefore, your system must have a minimum of 6 physical CPU cores (no hyperthreading). I will go for more core to give the OS some room.
To get best single job speed: run 1 Weaver engine and give all speed units to medium and large jobs. If the Weaver engine is not hitting 3,000 ppm for batch output, check its logs to see if there is anything like flattening going on.
Assuming that on your system
a merge engine can achieve close to 1000 ppm with an optimized template
you can run 6 merge engines on your server
there are no impediments for the Weaver engines to perform normally
You should get decent ppm.
Suppose you are able to double the CPU cores (perhaps the server will fit an extra CPU) and add a performance pack, then with 12 merge engines and a faster output speed, you could probably get over 10k ppm. Disclaimer: this is speculation based on a number of assumptions. It is not a guarantee. You may need some professional services assistance to help setup, configure the system and optimize the templates and workflow processes.
We have very simple scripts I the template, so the scripting should be ruled out.
The server in Intel Xeon with 2.9GHz (4 processors - 4 cores) and the RAM is 16 GB. During the process of Content Creation, I watched the usage of CPU and RAM. The usage of CPU was about 50-60%,RAM was about 40%.
I am very interested in the number of Merge engine you mentioned (run as many merge engines as possible on the given hardware). I noticed there were 2 or 3 Mergeengine.exe, but not sure how to add more to speed up the process.
You can increase the number of merge engines from Start > Programs >Objectif Lune > Connect Server Configuration.
Then click on the Scheduling page and set what you can consider the maximum number of records in small jobs and minimum number of records in large jobs.
Under the Scheduling page, you will find the Scheduling - Merge Engine and Scheduling -Weaver Engine when you can enable or disable Merge and Weaver engines.
If you are running on Connect v1.7 and below, you will need to restart the OLConnect_Server and OLConnect_MySQL services if you make any changes to the Server scheduling preferences.