Hello there
TL;DR
Datamapper createHTTPRequest does not encode POST request properly to UTF8.
I did stumble over the following problem (code below):
In my datamapper i send some data to an external server with json data.
When my json contains special characters (like ö,ä,ü – I’m Swiss, so thats not that special after all) the server receives the data, but the encoding is wrong.
Wrong result
php json_decode results in:
JSON_ERROR_UTF8 Malformed UTF-8 characters, possibly incorrectly encoded
My guess, whats happening here:
When javascript is used on proper webpage, the encoding is defined by the page and internally javascript uses UTF16 (I think). And its transparently handled by the engine itself.
But here i think that the request just sends the UTF16 string to the server as-is and does not properly convert it to UTF8 encoding. Xmlhttprequest has to use UTF8 encoding acc. to the docs.
A (bad?) fix
I did find a solution when i use utf8.js lib [2] to encode the extracted strings “again” the json seems to be valid. But i dont know if i can trust this library at all its very old.
Version of PPC
I do use the quite old Version 2019.2. Maybe there was a fix in the newer versions?
My code datamapper extraction script field (reduced to the core part):
var payload = {
"uuid": "75a567ef-7796-4a6f-a0d2-133df059ae7d",
"Field": record.fields['FieldWithSpecialChars'], // This doesn't work but ': "ä"' does not work either
};
var myJSON = JSON.stringify(payload);
var request = createHTTPRequest();
var my_url = "https://webhook.site/xxxxx"; // To check the request
request.open("POST",my_url,"","");
request.setRequestHeader("Content-type", "application/json");
request.send(myJSON);
uuid;
[1] Datamapper createHTTPRequest() - Bad Request: Received a GET request with a text/plain; charset=UTF-8 Content-Type - DataMapper - Upland OL User community
[2] GitHub - mathiasbynens/utf8.js: A robust JavaScript implementation of a UTF-8 encoder/decoder, as defined by the Encoding Standard.
