Verified with native player and Mobo(HW playback). Modified steps 3-3.1 to reflect HP settings, added HP preset and sample. Enjoy!
Since this question gets asked a lot lately and my post on one of the thread was buried, so I figured a separate thread would be helpful.
As I understand it, Tegra 2 should playback up to High Profile 720p and Main Profile 1080p. However, with limitation on Honeycomb and/or lack of driver release by Nvidia, current crop of HC tablets do not playback anything above Baseline Profile smoothly. Future updates by Google/Nvidia should fix this issue, but it looks like we may have to reencode High/Main profile media for the time being.
Note that there are few software codec players(Mobo/Vital) that are able to playback High/Main profile H264 media files. My experience with these players were less than satisfactory and I wanted to stick with hardware decoding to maximize smoothness.
HandBrake is an open-source, GPL-licensed, multiplatform, multithreaded video transcoder, available for MacOS X, Linux and Windows. It offers variety of H264 encoding options via GUI and easy to use. Best of all, it is FREE.
You can obtain HandBrake from this link.
Once you have HandBrake installed and have a source file opened, there are only few tabs you need to setup.
1. Select the mp4 container and set the picture width to 1280. Anamorphic should be loose and Modulus 16. Large File Size can be check for better compatibility.
2. Under Video Tab, Make sure the options are set to "H.264","Same as source", and "Constant Quality". You can keep it a RF:20. Optionally, you can change the RF slider for better quality or smaller size. Higher RF number means smaller size but less quality. Also, you can choose "Average Bitrate" or "Target Size" option if you like.
3. Update: For Honeycomb 3.1, Follow step 3.1 for high profile settings. Last Tab but most important, set your advanced options exactly same as picture below. You can play with settings but CABAC, 8x8, and P-Frames has to be checked off to keep it baseline profile.
3.1 This is High Profile settings for HC 3.1 TF101.
That's it. Start your encode and enjoy your 720p on your Transformer.
Included below is my presets and sample movies I encoded using the profiles.
Baseline Sample on Megaupload
High Profile Sample on Megaupload