@mirolg is correct for a stock ROM. For fully-unlocked ROMs, it's possible for a XAP to put files into the Windows folder when installed (fully unlocked == everything, including the XAP unpacker, runs with full permissions).
In this case, though, I believe the app just copies those files over to the Windows folder. Any app with sufficient permissions can do that ("sufficient permissions" meaning via OEM driver using ID_CAP_INTEROPSERVICES the way older homebrew apps worked, through rolicy elevation using WP7 Root Tools, through token replacement using HtcRoot, or of course on a full-unlock ROM where everything has permissions for everything). Deleting those files from the Windows folder is probably a good start toward uninstalling, if you really want to do that, but you will probably need to revert some other changes (such as registry values, which are harder to find on cursory examination) as well.