It is not possible to insert a text script for nested detail fields

[Xanathon]

Over a year ago we started developing a template for invoices. That project was postponed, since another project took precedence. Now I had to revisit the old datamapper and template.

In our training we asked the trainer about our invoice xml structure, he acknowledged a software problem and gave us a workaround. He told us to do a loop on the first xml node (Rechnung) despite there always is only one Rechnung-Node in a dataset and not multiple.

That is this construct and back then it worked:

I was able to use the fields from the extract “Extraction Basisdaten” in the template in Designer by just dragging and dropping from the datamodel.

In one of the last versions the software behavior was changed, however. Now the data from the extraction is seen as looped data and if I try to drag and drop the data fields into the template I get the following error:

I took half a day to try to rescue the datamapping and template. But even if Datamapper allows dragging and dropping of steps and copy & paste of steps, the datamodel is damaged and no longer working if I do that (drag and drop the extraction outside of the Rechnung-loop or copy & paste the extraction outside of the Rechnung-loop).

I could just delete the extraction and recreate it, but since I had to rename the fields for usability and to avoid redundant field names that would be a lot of effort and would be prone to error, since I would have to reconstruct all field names. Unfortunately I cannot copy & paste a list of field names from the datamodel, to paste the field names later.

What can be done to salvage and rescue the datamodel and the template? Is there an option to deactivate the “nested detail fields”-message and just use the fields as non-looped fields?

TIA

[Sander]

Hi @Xanathon,

Perhaps I don’t fully understand what you mean, sorry. Are you dragging a detail field from the Data Model view to a template editor?

If you are doing that the “not possible” message will always pop up, no matter what your data model looks like. It is not a serious error, we show it because it is not clear what the user intends to do.

Can you please explain what you expect to happen? Do you expect a script to be created? What would that script do?

[Xanathon]

If I want a data field from the data model in the template I drag & drop it into the template to a position where I want it. There a placeholder is created.

E.g. if I drag the field “name” into the template, at the position where I dragged it to a placeholder in the form @name@ is created (and also a script dynamically filling that placeholder from the data field).

In the past that simply worked, even with a field from an extraction in a loop. Now I get that error message. But only if the field comes from the first extraction in the Rechnung-loop. A field from outside any loops is placed as expected.

[Sander]

You’re saying “data field” but I said “detail field”. Dragging a regular field to the editor should work fine, but dragging a detail field to the editor has never been supported.

Whether a field is a detail field depends on the hierarchy. If your data model looks like this:

record
	field1
	detail_table
		field2

… only field2 is a detail field. You can drag field1 to the editor since it is a regular field. That will create a placeholder and a script. However, for field2 you will get that “not possible” message.

[Xanathon]

I am sorry that I do not know your exact nomenclature.

Since when? In the past I was able to drag & drop the fields from below “Rechnung” without any problem whatsoever. The Rechnung loop was only a workaround for a another problem anyway, provided by one of your trainers. In one of the past versions that handling must have been changed, since back then I could drag and drop all fields below “Rechnung” without any problem whatsoever and without getting this error message.

Because of that change now datamodel and template have become completely unusable and I would loose days if not weeks of work. The question is still: how I can salvage this?

If I try to drag the extraction step out of the loop, the data fields are empty. The same when I copy & paste. If this is not possible, why are there options to drag&drop or copy&paste anyway?

[Sander]

I’m not complaining about terminology, I’m sorry for giving you that impression. I’m only trying to make myself clear. There is a difference in behavior between a regular field and a detail field.

I’m out of my depth. Hopefully someone who is more familiar with the data mapper component can help you.

[Xanathon]

Does not look that way …

[Erik]

Was reading your initial question. I would like to understand the “software problem” encountered and the workaround provided. Would you be able to share your Data Mapping configuration? Let’s see if this is something we can fix (which version are you using?).

Detail tables are meant for data that can be iretated, therefore the default behavior is linked to the dynamic table wizard. For what I read it is now used to group information. From that standpoint I would like to understand why it is not possible to have this data in the root of your data model.

To group information we introduced the JSON field type (believe this was in v2022.1). An example of this is found the “Basic letter” sample which you could Download from the New section in the Welcome screen of OL Connect Designer 2023.1. The image below is taken from that template.

Placeholders for properties in a JSON field are created using drag and drop. This approach also works when not using Handlebars expressions but in that case it will generate scripts.

To come back to your question. We may opt for a change in the software where selecting No in the warning dialog (shown in your initial question) inserts a placeholder for the field in the current row of the detail table. Perhaps even including an option to remember the selected option.

Especially when using the new Handlebars expressions this should be relatively easy. The expression would look something like this {{Rechnung[0].Auftragsherkunft}} and doesn’t require a script. For the scenario where Handlebars expressions is not active we could fallback to creating a script to set the text of the placeholder (something like: results.text( record.tables.Rechnung[0].fields.Auftragsherkunft )).

FYI. I typically create the optimal data model for the design first, after which I map the data to the model. This approach may not work for everyone, but it helps me to simplify the logic in my templates and keep the data model manageable.

I hope this is of some help,

Erik

[Xanathon]

Hello Erik,

thank you for your answer. Unfortunately I cannot share the data mapping configuration, since there are real customer data in the example files and I cannot share those because of GDPR. I would have to remove all example data from the datamapper and provide an example data file that has no real customer data, but that is some work. i’ll try to do that, but a single example file is not enough, because the xml invoice files differ a lot depending on the order details.

We are using 2023.1. The datamapping and template were created around november 2021, I do not know what was the version then. We updated all versions relatively fast because we had reported heaps of bugs that needed fixing and some of those were fixed in the updates. That probably was a wrong approach.

We have to derive data from a very complex xml-stucture to generate invoices with lots of special cases and multiple daughter companies in one design document. When we had our training in november 2021 we presented the trainer with the xml file and to avoid some problem, he told us to do a loop on the first “Rechnung”-element (there is always only one of those, because they represent one invoice and we split the xml file on “Rechnung”), but no one here remembers what that problem possibly could have been. But we do remember it was a workaround for some problem in that older Datamapper version, we were told that explicitly.

Whatever that problem was, I would guess it was removed in some version afterwards and that leads to the problems we have now, since the behavior has changed. Then we could drag & drop fields from that unnecessary loop, now we cannot.

We are not using handlebars, since we would first have to look into how to use that and if it is reasonable for our use case.

With the complexity of our data it is not possible to first completely do the datamapping and then the template, we have to go back and forth adding data fields in datamapper as we need them in designer. This is not possible and way too much effort otherwise. The main problem here is that we had a working datamodel and now we can no longer use that because of the changes in the software that now prevents us from using that datamodel.

I took a whole day trying to repair the datamapping last week. I tried to move or copy&paste the extraction step as described in a post above, but in all cases the datamodel was broken. I tried to delete the extraction step to remap outside of the loop but then I had orphaned fields in the datamodel I was not able to delete via context menu, next refresh and they were back again.

I will try to get anonymized example data so I can provide the datamodel.

regards,
Xanathon

[Erik]

Thanks for the update. Looking at your data model I assume your template will also have an impressive list of scripts. Handlebar expressions may simplify your template and signficatly reduce the number of scripts. In addition expressions are simple text strings so there is no need to wrap placeholders with elements with a class or ID, I’m sure this will further simplify your template.

The examples in the Welcome Screen use expressions. Give them a look, it may inspire you and help move things forward.

Best,

Erik

[Xanathon]

I understand that Handlebars may be a great improvement for future projects, but at the moment my primary goal is to save the lots and lots of hours of work that were already done on the Datamapper and Template. And so far I see no way to do that, even after a lot of experiments. I cannot even remove an extraction step to recreate it outside the loop, because then the fields in the datamodel are orphaned and cannot even be deleted.

Actually there are not so much scripts in the template so far, since we are (fortunately) still in an early stage of development, as I mentioned earlier. If we were far more advanced in the development process the damage done by the drastically changed software behavior would be even greater.

If I delete the extraction step, the fields stay in the datamodel with no way to delete them.

If I try to first delete all extraction fields in the extraction step that does not work, since there has to be at least one data field in an extraction step. So I always have an orphaned field in the datamodel that I cannot remove because rightclick->contextmenu->delete does not actually delete the fields, they are back on next load or reload.

Why are fields in the datamodel not deleted if I delete the extraction step? How can I actually delete the orphaned data fields from the datamodel? How can I delete ALL fields from an extraction step before deleting the step itself?

If I could delete that extraction step and get rid of all the corresponding fields in the datamodel I could just do a new extraction outside of the unnecessary loop and then simply resuse the fields with the same field names in the template.

[Phil]

When the fields remain in the data model after deleting the extraction step, you can get rid of them by multi-selecting them in the data model panel, right-clicking and then selecting “Delete”. This will permanently remove them from the data model.

[Xanathon]

I know that it should theoretically work that way (as I already wrote above). I tried that multiple times, but it does not work. The fields stay in the data model. They first appear in red after the delete

and after a refresh or reload they are back and still there in grey:

There is no way I can see to get rid of the fields.

Edit: Same if I delete the group: After a refresh it is back with all fields.

Edit 2: The whole Datamodel pane seems to be buggy. If I add new fields from a new extraction and group them, the group is only shown after I do multiple refreshes. The display behavior of that pane seems inconsistent to me.

[Phil]

Make sure you close the template and only have the DataMapper config open when you do that. If the template is open and actively uses those fields, then the DataMapper takes for granted that the fields must be kept in the data model.

[Xanathon]

Oh my god … yes, that works. Thanks. That is not intuitive, since you usually always have datamodel and template open together. Datamapper should take precedence imho, after a requester asking “do you really want to delete?”.

[Phil]

I understand your position, but imagine that your data mapper configuration is used with several different templates: it would be a bit too easy to screw up all the other templates just because you no longer need certain fields in one specific template. Mind you, that can still happen with the method I outlined earlier… but it requires an extra step that may help users remember that there could be unforeseen consequences to deleting fields from the data model.

In the end, it’s a design choice we made. I still believe it’s the proper one, but if several users tell me I am mistaken, then I will be more than willing to reconsider.

[Xanathon]

That may or may not be so, but for usability and UX it is at least advisable to show the user a requester that states “to delete the fields you have to close the template first” (or something like that). Else as a user you are extremely frustrated and get nearly mad about not being able to delete the fields (as in my case, I tried literally for days). And if you close the template before deleting the fields you still can screw up other templates, so I do not understand the reasoning here …

Aside from that: From what I saw from the software so far I would NEVER reuse a datamapping configuration for multiple templates for exact the reasons you just gave. Especially if there are probably multiple devs working on said templates and datamappers. The maintainability of this really is not good. Workflows and use cases are different. What may look as a good “design choice” for you may be a very bad one for a customer’s use case.

[Phil]

I agree with you: a notification that you should close the template first before deleting fields would be useful.

As for re-using data mapping configurations, again, I understand your position but we have thousands of customers to consider when we make these design choices.

Having a single, standardized data mapping config for multiple documents is not uncommon. In some cases (and I think you fall into that category), users prefer to have a single template (with a single DM config) that conditionally generates different documents. That’s very useful when you have to make a single change that impacts multiple documents, but it also makes managing that template more cumbersome.

In other cases, each document has its own template, and they all share the same DM config. The documents are therefore much easier to maintain individually, but making a global change to all of them is more difficult.

It’s a design choice (on the part of the user, this time).

But in any event, this is a very useful conversation: we know we need to cater to all kinds of different needs with our applications, but sometimes we may over-privilege one method over another. This kind of feedback helps us try to balance things better.

Thanks for the comments… And if any one else wants to chime in, feel free to do so!

[Xanathon]

I wanted to provide a solution for the drag & drop-problem of Extract steps from inside a loop to outside of a loop, if the loop actually is not necessary, for users that may look for a solution for a similar problem in the future:

Inside the loop the extracted XPath for the field “AnzahlPakete” looks like this:

./@AnzahlPakete

if you drag & drop the Extract step outside the loop, you have to correct the Xpath for the attributes (also see my xml structure screenshot above), in my case:

./Rechnung/@AnzahlPakete

Using this on all fields I was just able to salvage that extract step so I can still use the datamapping configuration with my template (keeping my changed field names) and I am able to drag and drop the fields again in Designer.

[Xanathon]

@Phil in the past with PlanetPress Suite 7 we had eight different templates for different distribution channels. When management wanted design changes we had to touch all eight single templates to implement those changes. The maintainability was very bad and that lead to a lot of unnecessary work and was very prone to error. From those experiences we decided we will implement all those distribution channels in one template in Connect to avoid those problems.