Do you really need to cut on FF as you already cut after 67 lines? Otherwise you can simply disable the cut on FF option.
It might also help to delete the last lines through the Add/Remove lines plugin in Workflow.
1- I can’t simply delete the last line because that last line might be valid data that needs extracting. The last line might not always be empty. We sometimes have the totals values on that last line.
2- I had tried your first suggestion by turning off the Cut on FF before posting the question. If I do that, then the datamapper fails to set the record boundary correctly as it now doesn’t know when a page starts and when it ends. So it thinks the whole text file is one long page and I sometimes get 1 record instead of the correct number of X records.
There is an option to ignore CRLF at end of file, I guess OL did not anticipate this scanrio for FF. We need this option for FF at start AND end of file.
In addition, I have also found that the FF could even be at the begining of the file and in this case I get a out of bound execption in the first record. Likewise I can’t simple delete the first line as it could be valid data.
I really need to be able to remove FF at start and end of the file if they are there and keep the ones in between the pages of the file. Any other idea?
Then you only choice is to do this in a script. It will look at the 3 first lines and 3 last lines (why 3? no idea but we have to start at something ), and remove any FF found.
Or safer, you could remove any FF found prior any data at the beginning of the file and after any data at the end of file.