Does removing the metadata make a photo anonymous?
No, and no tool can promise it: the way the file is compressed still points at the camera or the program. What goes is what is written down: the position, the serial number, the names and the dates.
Why is the orientation tag kept?
Without it a phone photo comes out on its side in most viewers, and it says which way is up and nothing about you. Untick it if every tag must go.
Does it change the picture?
No. The compressed image data is copied byte for byte, so the pixels of the result are the pixels of the original.
What are the hidden extras?
Bytes after the picture ends: the video of a motion photo, a depth map, an HDR gain map. A redraw on a canvas drops them without saying so; here they are named, and yours to keep or drop.