Ahhh… I see: your Job does not necessarily contain all the Content Items from any specific Content set, which is why you have to loop through the JSON Page Details List in order to determine if the content item is included or not in the job.
In that case, your approach is the correct one.
As for the duplicate content items in the Get Content Items for Job enpoint, I would think that if they are duplicated, then they would also appear twice in your output (because the Output Creation is internally using that same endpoint to fetch the list of items to include in the output). Could it be that your Job Preset is somehow set to duplicate some items?