Hey guys! I have an emergency!!!
I`m trying to retrieve a recorddata set with 8k record and I receive this error
The output is JSON and after retrieve I have a script that creates a file.
Thanks!!!
8K records is really not a lot… unless each individual record is huge. If you have a lot of detail tables in each record, they are all included in the JSON structure (whereas if you output to metadata, then you only get the main records, without the detail tables). So you may want to test if you get the expected results when outputting to metadata.
Then you could try and stop the Connect Service and start it again. Your server will be down for a couple of minutes, mind you, until all engines spin back up.
Hi Phil,
I had been fixed. I had to restart my workflow and everything ran well afterwards. Luckily, for this client we do not have detailed tables, just records with 280 fields :)))).
I will test out the output metadata, but have not find yet the right syntax to access and output to a file some information related to metadata.
Usually, I have to create a report for each containing some fields from those 280, but I am not sure how it works with metadata.
Regards,
fsh22
If you still have the logs for when the issue occurred, you may wish to analyze what else was going on at the same time. Workflow is (sadly) still a 32-bit app, so there is a 1.8 GB memory limit for the entire service. If it so happened that other processes were concurrently handling large amount of data, it may just have been bad luck due to timing.
Note that the definition of “large amount of data” can be deceptive. A 50 MB XML file seems small, but due to the nature of XML processing, it takes about 500 MB of memory. Workflow is usually pretty good to use memory only when needed, but timing…
Hi Fortiny,
I checked my logs, there were no other process running in the same. I have a workflow cfg file with two process, it is designed to run one after another, so we cannot say it was due to bad timing…
Scrolling through my log I have discovered the following:
WPROC: PROCES B (thread id: 12788) - 12:52:21
DEBUG: 12:52:21.211 [0001] Starting plugin Folder Capture - 12:52:21
DEBUG: 12:52:21.211 [0001] Mask: *.txt
DEBUG: 12:52:21.211 [0001] Parsed mask: *.txt
DEBUG: 12:52:21.211 [0001] Input path: C:\Users\ol\Desktop\TEST\Folder capture export_extins_enriched
DEBUG: 12:52:21.211 [0001] Parsed path: C:\Users\ol\Desktop\TEST\Folder capture export_extins_enriched
DEBUG: 12:52:21.211 [0001] Number of masks: 1
DEBUG: 12:52:21.211 [0001] Using mask: *.txt
DEBUG: 12:52:21.212 [0001] Found 1 new files in “C:\Users\ol\Desktop\TEST\Folder capture export_extins_enriched”
DEBUG: 12:52:21.212 [0001] 1 file(s) found
DEBUG: 12:52:21.212 [0001] Processing file C:\Users\ol\Desktop\TEST\Folder capture export_extins_enriched\9fd6b35e-8741-47f4-9848-06dbd982b423_sorted_export_extins Somatie de plata RCS-RDS.txt
INFO : 12:52:21.218 [0001] File captured : C:\ProgramData\Objectif Lune\PlanetPress Workflow 8\PlanetPress Watch\Spool\1.tmp\job0109BIKRGH0I081B08EC0FD.dat, size: 1117475 bytes
INFO : 12:52:21.218 [0001] Plugin Folder Capture completed successfully - 12:52:21 (elapsed time: 00:00:00:007)
DEBUG: 12:52:21.218 [0002] Entering branch
DEBUG: 12:52:21.220 [0003] Starting plugin Set Job Infos and Variables - 12:52:21
DEBUG: 12:52:21.220 [0003] Value: %O
INFO : 12:52:21.220 [0003] JobInfo/Variable %{fileName} is set to “9fd6b35e-8741-47f4-9848-06dbd982b423_sorted_export_extins Somatie de plata RCS-RDS”
INFO : 12:52:21.220 [0003] Plugin Set Job Infos and Variables completed successfully - 12:52:21 (elapsed time: 00:00:00:000)
DEBUG: 12:52:21.220 [0004] Starting plugin Run Script - 12:52:21
INFO : 12:52:21.220 [0004] Run embedded script…
INFO : 12:52:21.249 [0004] Data file processed : job0109BIKRGIIIN82B08EC10D.dat, size: 1117475 bytes
INFO : 12:52:21.249 [0004] Plugin Run Script completed successfully - 12:52:21 (elapsed time: 00:00:00:029)
DEBUG: 12:52:21.249 [0005] Starting plugin Set Job Infos and Variables - 12:52:21
DEBUG: 12:52:21.249 [0005] Value: %{exportExtinsID}
INFO : 12:52:21.249 [0005] JobInfo/Variable %{runID} is set to “9fd6b35e-8741-47f4-9848-06dbd982b423”
DEBUG: 12:52:21.249 [0005] Value: %{templateID}
INFO : 12:52:21.249 [0005] JobInfo/Variable %{templateID} is set to “Somatie de plata RCS-RDS”
INFO : 12:52:21.249 [0005] Plugin Set Job Infos and Variables completed successfully - 12:52:21 (elapsed time: 00:00:00:000)
DEBUG: 12:52:21.249 [0006] Starting plugin Execute Data Mapping - 12:52:21
INFO : 12:52:21.256 [0006] Sending file data source…
DEBUG: 12:52:21.291 [0006] File Data Source ID: 1943118
INFO : 12:52:21.291 [0006] Sending data mapping command…
DEBUG: 12:52:21.292 [0006] Using cached configuration file (id = 1938299).
DEBUG: 12:52:21.292 [0006] Starting data mapping operation…
DEBUG: 12:52:21.329 [0006] [2021-02-03 12:52:21.329] Record creation> Retrieving results
DEBUG: 12:52:29.502 [0006] [2021-02-03 12:52:29.502] Record creation> Generating JSON data
DEBUG: 12:54:14.024 [0006] New JSON data file created: job0109BIKRGIIIN82B08EC10D.json
INFO : 12:54:14.024 [0006] Data successfully created from : Export_extins_enriched.OL-datamapper
INFO : 12:54:14.024 [0006] Plugin Execute Data Mapping completed successfully - 12:54:14 (elapsed time: 00:01:52:775)
DEBUG: 12:54:14.024 [0007] Starting plugin Run Script - 12:54:14
INFO : 12:54:14.024 [0007] Run embedded script…
INFO : 12:54:15.153 [0007] Data file processed : job0109BIKRGIIIN82B08EC10D.json, size: 6983621 bytes
INFO : 12:54:15.153 [0007] Plugin Run Script completed successfully - 12:54:15 (elapsed time: 00:00:01:129)
DEBUG: 12:54:15.153 [0008] Starting plugin Update Data Records - 12:54:15
DEBUG: 12:54:15.154 [0008] Create the rest client
DEBUG: 12:54:15.154 [0008] Original JSON field: %9
DEBUG: 12:54:15.390 [0008] Parsed JSON field: [{“schema”:{“columns”:{“cheie_primara”:“STRING”,“cheie_secundara”:“STRING”,“judet”:“STRING”,“localitate”:“STRING”,“tip_artera”:“STRING”,“strada”:“STRING”,“numar”:“STRING”,“cod_postal”:“STRING”,“infocodplus”:“STRING”,“oficiu_postal”:“STRING”,“directie_postala”:“STRING”,“recipient”:“STRING”,"confirmat…
DEBUG: 12:56:18.162 [0008] [2021-02-03 12:56:18.162] Print content creation> Sent JSON for field update
INFO : 12:56:18.169 [0008] Plugin Update Data Records completed successfully - 12:56:18 (elapsed time: 00:02:03:016)
DEBUG: 12:56:18.169 [0009] Starting plugin Delete - 12:56:18
INFO : 12:56:18.169 [0009] Deleting file : job0109BIKRGIIIN82B08EC10D.json, size: 6983621 bytes
INFO : 12:56:18.169 [0009] Plugin Delete completed successfully - 12:56:18 (elapsed time: 00:00:00:000)
DEBUG: 12:56:18.172 [0010] Entering branch
DEBUG: 12:56:18.174 [0011] Starting plugin Retrieve items - 12:56:18
INFO : 12:56:18.174 [0011] Retrieve using a condition rule.
INFO : 12:56:18.175 [0011] Launch find on the OL Connect server.
DEBUG: 13:13:01.059 [0011] Result received from OL Connect server.
DEBUG: 13:13:01.059 [0011] Saving result in current Metadata
INFO : 13:13:01.136 [0011] Plugin Retrieve items completed successfully - 13:13:01 (elapsed time: 00:16:42:962)
DEBUG: 13:13:01.136 [0012] Starting plugin Create Print Content - 13:13:01
DEBUG: 13:13:01.136 [0012] Template File: %{templateID}.OL-template
DEBUG: 13:13:01.136 [0012] Parsed Template File: Somatie de plata RCS-RDS.OL-template
INFO : 13:13:01.136 [0012] Source data is MetaData
DEBUG: 13:13:01.136 [0012] Creating the rest client
WARN : 13:13:09.944 [0012] Cached version not available, attempting to upload file.
DEBUG: 13:13:10.083 [0012] Template C:\ProgramData\Objectif Lune\PlanetPress Workflow 8\PlanetPress Watch\OLConnect\Template\Somatie de plata RCS-RDS.OL-template uploaded to server
DEBUG: 13:13:10.083 [0012] Starting content creation operation…
DEBUG: 13:13:10.312 [0012] [2021-02-03 13:13:10.312] Print content creation> Retrieving results
DEBUG: 13:24:44.919 [0012] Content created from template and metadata
DEBUG: 13:24:44.991 [0012] Metadata file saved
INFO : 13:24:44.991 [0012] Content successfully created from : Somatie de plata RCS-RDS.OL-template
INFO : 13:24:44.991 [0012] Plugin Create Print Content completed successfully - 13:24:44 (elapsed time: 00:11:43:855)
DEBUG: 13:24:44.991 [0013] Starting plugin Create Job - 13:24:44
DEBUG: 13:24:44.991 [0013] Runtime Parameters JSON:
DEBUG: 13:24:44.991 [0013] Parsed Runtime Parameters JSON:
DEBUG: 13:24:45.054 [0013] Create the rest client
INFO : 13:24:45.054 [0013] Creating Job…
DEBUG: 13:24:45.055 [0013] Using cached config file (id = 1938357).
DEBUG: 13:24:45.055 [0013] Starting job creation operation…
DEBUG: 13:24:45.088 [0013] [2021-02-03 13:24:45.088] Job creation> Retrieving results
DEBUG: 13:27:31.248 [0013] Metadata file saved
INFO : 13:27:31.248 [0013] Job successfully created from : SORTARE_NRPLIC.OL-jobpreset
INFO : 13:27:31.257 [0013] Plugin Create Job completed successfully - 13:27:31 (elapsed time: 00:02:46:266)
DEBUG: 13:27:31.257 [0014] Starting plugin Create Output - 13:27:31
DEBUG: 13:27:31.353 [0014] Using cached config file (id = 1942689).
DEBUG: 13:27:31.353 [0014] Starting output creation operation…
INFO : 13:27:31.395 [0014] Output successfully created from : DataMatrix_Separation_EOS.OL-outputpreset
INFO : 13:27:31.405 [0014] Plugin Create Output completed successfully - 13:27:31 (elapsed time: 00:00:00:148)
DEBUG: 13:27:31.414 [0015] Starting plugin Delete - 13:27:31
INFO : 13:27:31.415 [0015] Deleting file : job0109BIKRGH0I081B08EC0FD.dat, size: 1117475 bytes
INFO : 13:27:31.415 [0015] Plugin Delete completed successfully - 13:27:31 (elapsed time: 00:00:00:001)
WPROC: PROCES B (thread id: 12788) complete - 13:27:31 (elapsed time: 00:35:10:204)
-
Why retrieving items and creating plugin contents lasted for 16, respectively 11 minutes?
-
We talk about 8k records with no detail table lines…
I am a bit concerned because all our jobs will be (more or less) running on the same architecture as in this process posted aboved. We have various situations regarding the number of records, ranging from 10k to 1mil.
Regards,
fsh22
I have recreated a similar setup on my machine: about 6000 records, from which I am extracting 300 fields per record. When using the Retrieve Items task after that, I also encountered the Out Of Memory error from time to time. I am not quite sure why it happens.
The data mapping process itself takes only 15-20 seconds when I set it its output type to IDs in metadata. I therefore wrote a script to retrieve the records as a JSON file, which takes about 3 minutes to execute and which never resulted in an error message.
I will report the issue to our R&D team. In the meantime, you should open a support ticket with our team so they can help you with a workaround.
Hi @Phil,
I have received a workaround for the problem that I reported. But i still need some clarification if you don`t mind. Below is the script:
var rest = new ConnectREST();
var myRecordSet = Watch.ExpandString("GetMeta(_vger_recordset_id[0], 10, Job.Group[0])");
var timer = new Timer();
// Get List of all data records
//Watch.Log("Retrieving record IDs",2);
var response = JSON.parse(rest.get("/rest/serverengine/entity/datasets/"+myRecordSet));
if (response) {
var recordList = {
recordids: response.identifiers,
recursive: true,
explicitTypes: true
}
// Watch.Log("Retrieved "+response.identifiers.length+ " records in "+timer.diff(),2);
// Watch.Log("Retrieving record values",2);
var response = rest.post("/rest/serverengine/entity/datarecords/values",JSON.stringify(recordList));
if(response){
// Watch.Log("Retrieved "+response.length+ " bytes in "+timer.diff(),2);
// Watch.Log("Writing records to file",2);
var f = new BinaryFile(Watch.getJobFileName());
f.WriteAll(response);
// Watch.Log("New job file created in "+timer.diff(),2);
} else {
Watch.Log("No record found",2);
}
} else {
Watch.Log("Empty data set",2);
}
/*
HELPER FUNCTIONS/OBJECTS BELOW
*/
function ConnectREST(){
try {
var serverInfo = JSON.parse(Watch.GetConnectToken());
} catch (e) {
if(!serverInfo) {
serverInfo = {
protocol: "http",
host : "localhost",
port: 9340,
token: ""
};
}
}
this.baseURL = (serverInfo.protocol || "http")+ "://" + serverInfo.host + ":" + serverInfo.port;
this.authToken = serverInfo.token;
this.xhr = new ActiveXObject("MSXML2.XMLHTTP.6.0");
this.get = function(endPoint){
// Watch.log("GET: "+this.baseURL+endPoint,4);
this.xhr.open("GET",this.baseURL+endPoint, false);
if(this.authToken) this.xhr.setRequestHeader("auth_token", this.authToken);
this.xhr.send();
return (this.xhr.status==200) ? this.xhr.responseText : null;
}
this.post = function(endPoint,payload){
// Watch.log("POST: "+this.baseURL+endPoint,4);
this.xhr.open("POST", this.baseURL+endPoint, false);
if(this.authToken) this.xhr.setRequestHeader("auth_token", this.authToken);
this.xhr.send(payload);
return (this.xhr.status==200) ? this.xhr.responseText : null;
}
}
function jsonGetRequest(url, token){
var res=null;
var xhr = new ActiveXObject("MSXML2.XMLHTTP.6.0");
xhr.open("GET",url, false);
xhr.setRequestHeader("auth_token", token);
xhr.send();
if(xhr.status==200) {
res = xhr.responseText;
}
xhr = null;
return res;
}
function jsonPostRequest(url, token, payload){
var res=null;
var xhr = new ActiveXObject("MSXML2.XMLHTTP.6.0");
xhr.open("POST", url, false);
xhr.setRequestHeader("auth_token", token);
xhr.send(payload);
if(xhr.status==200) {
res = xhr.responseText;
}
xhr = null;
return res;
}
// Thanks to https://ofstack.com/javascript/21749/jscript-method-for-reading-and-writing-binary-files.html
// for this handy object
function BinaryFile(filepath){
var adTypeBinary=1,adTypeText=2;
var adSaveCreateNotExist=1,adSaveCreateOverWrite=2;
var adReadAll=-1,adReadLine=-2;
this.path=filepath;
this.WriteAll = function(content){
var Stream = new ActiveXObject("ADODB.Stream");
Stream.Type = adTypeText;
Stream.CharSet = "iso-8859-1";
Stream.Open();
Stream.WriteText(content);
Stream.SaveToFile(this.path, adSaveCreateOverWrite);
Stream.Close();
Stream = null;
}
this.ReadAll = function(){
var Stream = new ActiveXObject("ADODB.Stream");
Stream.Type = adTypeText;
Stream.CharSet = "iso-8859-1";
Stream.Open();
Stream.LoadFromFile(this.path);
var content = Stream.ReadText(adReadAll);
Stream.Close();
Stream = null;
return content;
}
}
function Timer() {
this.prev = new Date();
this.diff = function(){
var newDate = new Date();
var res = newDate-this.prev;
this.prev = newDate;
return timeString(res);
}
this.reset = function(){
this.prev = new Date();
}
}
function timeString(time){
var second = 1000;
var minute = second * 60;
var hour = minute * 60;
var hours = Math.floor(time / hour % 24);
var minutes = Math.floor(time / minute % 60);
var seconds = Math.floor(time / second % 60);
var millisecs = (time / second % 1);
return hours + 'h ' + minutes + "m " + (seconds + millisecs).toFixed(3) + "s";
}
I was suggested to set my DataMap output to metadata ID`s only, instead I need the fix after Retrieve Items.
Quick question:
- After the provided script can I continue with another script in order to create my file, or do I have to adjust this one in order to have the desired result?
Many thanks in advance,
fsh22
Is it still the case the app is still only 32bit? If so then retrieve items is going to cause problems when using PDF input data.
Surely it must have been converted to 64bit now hopefully
The format of the input data is irrelevant when using the Retrieve Items task because at that stage, the data has already been extracted and is stored in a unified data model inside Connect’s database.
One possible cause for memory issues, however, is the number of fields being extracted for each record, as outlined in the discussion above.
And yes, the app is still 32-bit. Porting an app to 64-bit is unfortunately not as straightforward as simply recompiling the app. The app has literally hundreds of dependencies, therefore all of those dependencies have to be available in 64-bit, which is not always the case.
It makes sense now why we are constantly loosing work when using workflow and getting out of memory issue now and we were hoping it was fixed the newer version, but guess we’ll have to keep remembering to save every time we make a change so we don’t loose work when it crashes with the out of memory issues