I have a data that can contain up-to and not excessive of 1000 records. But for some random reason an output record gets generated and it’s a record from this data file as text. This happens randomly, but has started to become more frequent. When I re-process this file, I get my output and is processed correctly as what I want.
The data file is being split into chunks then processing these chunks are processed one at a time. The setup in the workflow is really straight forward and easy. The process has a self-replication process ticked and set so 5%. I have had issues with this in previous jobs, so I tend to keep it at between 5%-10%. This is so when I try to do something like extracted data and writing into a text file
I have a data that can contain up-to and not excessive of 1000 records. But for some random reason an output record gets generated and it’s a record from this data file as text
I assume that you have a input file (CSV?) which contains 1,000 records and that you sometimes have a output file which contains data of a single record as text instead of the expected output, can you please confirm if this is correct?
Were you able to check the PlanetPress Watch logging of such a moment? If so, do you see anything strange in the logging – For example: Certain Workflow plugins has been skipped because a Condition is true where it should be false – or any error or warning messages?
Thank you for your reply. It made me think a bit more
The data file is text that is delimited with a FF (FormFeed) character.
Looking at the logs on the Datamapperengine, it looks like there are some warning exeptions being generated “The step is out of bound (DME000130)” step(s) being out of bounds. Though a generated PDF file is on disk. This is okay.
Where there is an Error message with “com.objectiflune.datamining.internal.exceptions.EStepFailed: The step is out of bound (DME000130)” and I think this is when the text file is just output as text.
I have noticed this only happen when I run the file through the workflow. If I run it through the designer via an output creation so that all records are created, all file are generated okay.
There is also the “ERROR [main] com.objectiflune.contentcreation.html.mergeengine.impl.a.raiseError(HtmlMergeCoreUtils.java) uncaught exception: out of memory.
com.objectiflune.layoutengine.html.embed.GeckoJSRuntimeError: uncaught exception: out of memory”
Followed by “ERROR [IPC Input] com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeToPDF.exportToPdf(HtmlMergeToPDF.java) Error during content creation (CC3000023)
org.eclipse.core.runtime.CoreException: uncaught exception: out of memory.”
The above logs are from the “Mergeengine log file” and no errors in the Datamapperengine or Weaverengine log files. I guese the Merger engines need more memory.
I will keep testing/changing somethings to correctly indentify the issue.
Connect (incorrectly, in my never humble opinion) interprets “line termination” characters as “new line” characters. This will cause a spurious extra, empty, record at the end of your job. This MIGHT be the source of your issue.
You can override this behavior by checking the “Ignore CR/LF/FF at end of file” checkbox on the Settings tab of the Data Map.
Hi @TDGreer, I have the check box ticked as I thought this initially. I will have a look and do some twicks on the Mergeengine as this is where the error originates. Thanks for your advice.
The only thing I can find is this below from the mergeengine logs file.
Any ides what the issue could be? There is enough memory to process the file but I don’t know what the issue is. Maybe somene here in the forum can give me some pointers.
[2022-08-03 16:29:52,887] ERROR [main] com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeMasterPageJSCallback.function(HtmlMergeMasterPageJSCallback.java) Error during on-demand merge of masterpage: org.eclipse.core.runtime.CoreException: C:\Users\xxxxxxxxxxx\AppData\Local\Temp\TZCBUWfW.HTM.part could not be saved, because the source file could not be read.¶¶Try again later, or contact the server administrator. (CC3000018)>>>line endings sanitized<<< (Core_6250)
org.eclipse.core.runtime.CoreException: C:\Users\xxxxxxxxxxx\AppData\Local\Temp\TZCBUWfW.HTM.part could not be saved, because the source file could not be read.
Try again later, or contact the server administrator.
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.callInUIThread(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.runInUIThread(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.b.loadResources(b.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.b.getUnit(b.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.q.merge(q.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeMasterPageJSCallback.function(HtmlMergeMasterPageJSCallback.java)
at com.objectiflune.layoutengine.html.embed.e.callJava(e.java)
at org.mozilla.xpcom.internal.XPCOMJavaProxy.callXPCOMMethod(Native Method)
at org.mozilla.xpcom.internal.XPCOMJavaProxy.invoke(XPCOMJavaProxy.java:201)
at com.sun.proxy.$Proxy97.oL_evaluateJS(Unknown Source)
at com.objectiflune.layoutengine.html.embed.u.a(u.java)
at com.objectiflune.layoutengine.html.embed.u.evaluate(u.java)
at com.objectiflune.layoutengine.html.embed.Browser.evaluate(Browser.java)
at com.objectiflune.layoutengine.html.MozillaBrowser.a(MozillaBrowser.java)
at com.objectiflune.layoutengine.html.MozillaBrowser.evaluate(MozillaBrowser.java)
at com.objectiflune.layoutengine.html.pagination.j.run(j.java)
at com.objectiflune.layoutengine.html.pagination.PaginationProvider.a(PaginationProvider.java)
at com.objectiflune.layoutengine.html.pagination.PaginationProvider.paginate(PaginationProvider.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.p.run(p.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.runAndWait(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.a.b(a.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.a.mergeAndPaginate(a.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.mergeAndPaginate(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.mergeAndPaginate(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeToPDF.lambda$1(HtmlMergeToPDF.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.lambda$0(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.a.run(a.java)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:185)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4001)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3629)
at com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineApplication.start(MergeEngineApplication.java)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
at org.eclipse.equinox.launcher.Main.run(Main.java:1461)
at org.eclipse.equinox.launcher.Main.main(Main.java:1434)
[2022-08-03 16:29:52,896] ERROR [main] com.objectiflune.contentcreation.html.mergeengine.impl.a.raiseError(HtmlMergeCoreUtils.java) Error: org.eclipse.core.runtime.CoreException: C:\Users\xxxxxxxxxxx\AppData\Local\Temp\TZCBUWfW.HTM.part could not be saved, because the source file could not be read.¶¶Try again later, or contact the server administrator.>>>line endings sanitized<<< (Core_6250)
com.objectiflune.layoutengine.html.embed.GeckoJSRuntimeError: Error: org.eclipse.core.runtime.CoreException: C:\Users\xxxxxxxxxxx\AppData\Local\Temp\TZCBUWfW.HTM.part could not be saved, because the source file could not be read.
Try again later, or contact the server administrator.
at com.objectiflune.layoutengine.html.js2java.JavascriptErrorHandler.onJavascriptError(JavascriptErrorHandler.java)
at com.objectiflune.layoutengine.html.js2java.JavascriptErrorCallback.function(JavascriptErrorCallback.java)
at com.objectiflune.layoutengine.html.embed.e.callJava(e.java)
at org.mozilla.xpcom.internal.XPCOMJavaProxy.callXPCOMMethod(Native Method)
at org.mozilla.xpcom.internal.XPCOMJavaProxy.invoke(XPCOMJavaProxy.java:201)
at com.sun.proxy.$Proxy97.oL_evaluateJS(Unknown Source)
at com.objectiflune.layoutengine.html.embed.u.a(u.java)
at com.objectiflune.layoutengine.html.embed.u.evaluate(u.java)
at com.objectiflune.layoutengine.html.embed.Browser.evaluate(Browser.java)
at com.objectiflune.layoutengine.html.MozillaBrowser.a(MozillaBrowser.java)
at com.objectiflune.layoutengine.html.MozillaBrowser.evaluate(MozillaBrowser.java)
at com.objectiflune.layoutengine.html.pagination.j.run(j.java)
at com.objectiflune.layoutengine.html.pagination.PaginationProvider.a(PaginationProvider.java)
at com.objectiflune.layoutengine.html.pagination.PaginationProvider.paginate(PaginationProvider.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.p.run(p.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.runAndWait(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.a.b(a.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.a.mergeAndPaginate(a.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.mergeAndPaginate(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.mergeAndPaginate(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeToPDF.lambda$1(HtmlMergeToPDF.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.lambda$0(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.a.run(a.java)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:185)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4001)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3629)
at com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineApplication.start(MergeEngineApplication.java)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
at org.eclipse.equinox.launcher.Main.run(Main.java:1461)
at org.eclipse.equinox.launcher.Main.main(Main.java:1434)
[2022-08-03 16:29:52,902] ERROR [IPC Input] com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeToPDF.exportToPdf(HtmlMergeToPDF.java) Error during content creation (CC3000023)
org.eclipse.core.runtime.CoreException: Error: org.eclipse.core.runtime.CoreException: C:\Users\xxxxxxxxxxx\AppData\Local\Temp\TZCBUWfW.HTM.part could not be saved, because the source file could not be read.
Try again later, or contact the server administrator.
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.callInUIThread(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.runInUIThread(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeToPDF.exportToPdf(HtmlMergeToPDF.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.a(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.createContent(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineCommandProcessor.createContent(MergeEngineCommandProcessor.java)
at com.objectiflune.contentcreation.mergeengine.product.internal.f.run(f.java)
at com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineCommandProcessor.a(MergeEngineCommandProcessor.java)
at com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineCommandProcessor.a(MergeEngineCommandProcessor.java)
at com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineCommandProcessor.processMessages(MergeEngineCommandProcessor.java)
at com.objectiflune.contentcreation.mergeengine.product.MergeEngineThread.instantiateCommandProcessor(MergeEngineThread.java)
at com.objectiflune.core.ipc.EngineThread.run(EngineThread.java)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.objectiflune.layoutengine.html.embed.GeckoJSRuntimeError: Error: org.eclipse.core.runtime.CoreException: C:\Users\xxxxxxxxxxx\AppData\Local\Temp\TZCBUWfW.HTM.part could not be saved, because the source file could not be read.
Try again later, or contact the server administrator.
at com.objectiflune.layoutengine.html.js2java.JavascriptErrorHandler.onJavascriptError(JavascriptErrorHandler.java)
at com.objectiflune.layoutengine.html.js2java.JavascriptErrorCallback.function(JavascriptErrorCallback.java)
at com.objectiflune.layoutengine.html.embed.e.callJava(e.java)
at org.mozilla.xpcom.internal.XPCOMJavaProxy.callXPCOMMethod(Native Method)
at org.mozilla.xpcom.internal.XPCOMJavaProxy.invoke(XPCOMJavaProxy.java:201)
at com.sun.proxy.$Proxy97.oL_evaluateJS(Unknown Source)
at com.objectiflune.layoutengine.html.embed.u.a(u.java)
at com.objectiflune.layoutengine.html.embed.u.evaluate(u.java)
at com.objectiflune.layoutengine.html.embed.Browser.evaluate(Browser.java)
at com.objectiflune.layoutengine.html.MozillaBrowser.a(MozillaBrowser.java)
at com.objectiflune.layoutengine.html.MozillaBrowser.evaluate(MozillaBrowser.java)
at com.objectiflune.layoutengine.html.pagination.j.run(j.java)
at com.objectiflune.layoutengine.html.pagination.PaginationProvider.a(PaginationProvider.java)
at com.objectiflune.layoutengine.html.pagination.PaginationProvider.paginate(PaginationProvider.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.p.run(p.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.runAndWait(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.a.b(a.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.a.mergeAndPaginate(a.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.mergeAndPaginate(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeEngine.mergeAndPaginate(HtmlMergeEngine.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeToPDF.lambda$1(HtmlMergeToPDF.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.HtmlMergeCoreUtils.lambda$0(HtmlMergeCoreUtils.java)
at com.objectiflune.contentcreation.html.mergeengine.impl.internal.a.run(a.java)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:185)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4001)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3629)
at com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineApplication.start(MergeEngineApplication.java)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
at org.eclipse.equinox.launcher.Main.run(Main.java:1461)
at org.eclipse.equinox.launcher.Main.main(Main.java:1434)
[2022-08-03 16:29:52,939] ERROR [IPC Input] com.objectiflune.contentcreation.html.mergeengine.impl.HtmlMergeToPDF.exportToPdf(HtmlMergeToPDF.java) org.eclipse.core.runtime.CoreException: uncaught exception: out of memory
[2022-08-03 16:29:52,939] ERROR [IPC Input] com.objectiflune.contentcreation.mergeengine.product.internal.MergeEngineCommandProcessor.createContent(MergeEngineCommandProcessor.java) Error: org.eclipse.core.runtime.CoreException: C:\Users\xxxxxxxxxxx\AppData\Local\Temp\TZCBUWfW.HTM.part could not be saved, because the source file could not be read.¶¶Try again later, or contact the server administrator.>>>line endings sanitized<<< (Core_6250)
Can you let me know please if you were able to find out what the memory usage of the Merge Engine was the moment the reported incident occurred and if it was equal to or higher than the value set via the Connect Server Configuration application? In version 2022.1 of PReS Connect this is set via: OL Connect Server Configuration application > (Preferences window) > Engines > Memory > Merge Engine (MB).
We currently had the memory lower than this as shown in the image and only 1 Data Converter. So ended up bumping it to 2. It’s way more than enough I think! Hope this helps.
It’s just driving me insane. When these files are output as text, when I drop them back in to process there is no issue. There is no condition in the workflow that is being missed or anything. I am now thinking of prrocessing the whole file in an output crreation instead. As it’s the only solution at this moment.
We were using the PlanetPress Connect Workflow version 2021.2.1.
We have just updraded to version 2022.1.5 with some issues along the way but now running the same file seems to be okay. I guese is a threading issue or something as I have not changed anytging apart from the upgrade we did.
