I tested this on Mac OS 10.11.6 with ID CC 2017 and on Windows 7 with ID CS4 (6.0.6), exporting using standard ‘High Quality Print’. I checked the PDFs using Acrobat Pro DC on Mac and and Acrobat Reader DC on Windows and couldn’t see any spurious spaces. But, when I opened the test files in Illustrator (CS4 on Windows; CC 2017 on Mac) it is clear that there are spurious characters in the PDF. In both cases, there appears to be a series of end-of-paragraph marks where the discretionary hyphen was. In Illustrator, I couldn’t get the cursor to run past the hidden characters in either direction nor could I select them.
I could select and copy the passage in using Acrobat Pro DC on Mac and and Acrobat Reader DC on Windows. Pasting into WordPad showed no anomalies but, when I pasted into BBEdit, there was an extra line return at the end of the word with the discretionary hyphen. Very odd.
Looks like a bug to me and I’m surprised it has persisted so long. For the nonce, the only solution for you I can think of is to eliminate discretionary hyphens in the source file, which may not be feasible.