Heimo: Yes. Select the “on” state and make any changes you want. Then click the “off” state.
You should think of the “on” and “off” states as though they were different objects on the page. So when you choose “on” then InDesign hides the “off” state objects and shows the “on” state objects. You can change the text, the color, the effects… anything!

Heimo: Yes, what you are describing is the “Normal On” selection in the Buttons and Forms panel. You select the checkbox button, then you can change the look of it for the “Off” state and the “On” state (or even the rollover and click states if you want). When you export to Interactive PDF, it should work.
However, the problem is that sometimes it doesn’t work… especially depending on the PDF reader. These often don’t work on mobile devices, for example. And even if it does work, sometimes the “look” gets covered up by Acrobat’s big blue box, so you cannot see that it is working!
There may be some help here:
https://indesignsecrets.com/pdf-button-tips-tricks-and-problem-solving.php

I wonder if this is a Windows issue… 
Also: Richard, do you have more than one version of InDesign installed on your computer? Perhaps you’re running one version but Bridge tries launching a different version?

What is the font? Perhaps you only have a bold version of the font installed? Or perhaps the “regular” is bold?