Pull the battery out the device, wait about 15 seconds, replace the battery, press and hold the volume DOWN , home, and power button to boot into download mode, press the volume up button to continue into download mode, proceed to flash.
How can I flash modem on rooted device?
Flashing modem only with odin doesn't seem to work for me.
Should I flash the entire tar file and then root it again?
Not nearly enough information. You'd get a lot more tailored help if you described in detail exactly how you tried to change your modem. Then we could spot what went wrong. Be sure to include the versions of your OS, modem file and your flashing program.
XDA Developers was founded by developers, for developers. It is now a valuable resource for people who want to make the most of their mobile devices, from customizing the look and feel to adding new functionality. Are you a developer?