This is exactly how mine was when I got it used. The oval rubber type thing pops out with a tooth pick or as suggested above a small screwdriver. I hunted for ages for a replacement but ended up having to make a new port cover from some of the watches packaging that is a similar material. As you still have it just super glue the oval port seal back onto the cover and you're done!
XDA Developers was founded by developers, for developers. It is now a valuable resource for people who want to make the most of their mobile devices, from customizing the look and feel to adding new functionality. Are you a developer?