I wish there was a 1-step solution, but I fear not. This involved making a scalloped mask to place over the top of the image.
1. Make a rectangle the same width as your image.
2. Apply the Wavy stroke style.
3. Bump up the thickness of the stroke until the waves are the size to your liking.
4. Copy and paste the rectangle into Illustrator.
5. View in Outline mode (Cmd + Y). Use the white arrow to delete the outside rectangle clipping path.
6. Then use the black arrow to select the wavy rectangle and copy and paste it back into InDesign.
7. Use the white arrow to delete the sides and top of the rectangle (Be sure to get all the points).
8. Use the pen tool to click on one of the end points, then close the path by shift clicking down, then across, then on the last point.
9. Fill with white and remove the stroke.
Now that you know how to make a wavy line, you could alternatively, delete the top edge of your image frame, and then connect this wavy edge using the pen tool. Repeat with the bottom edge. Then save the scalloped frame as a snippet or in a library.
.
