[PRJ] SYS Builds: What's the Difference?

---

I'm putting up this thread because there is a good bit of questions that get asked about the difference between the SYS builds. While Microsoft and its partners don't give us a real breakdown of them, we can at least get results on which one is faster than the other. That is the purpose of this thread. If you have any relevant information for me to add to this, let me know. If I add your suggestion, I will, of course, credit you. This is relevant to pretty much all WM6 devices, but I'm making this specific to the Rhodium as that is the only device I plan to support with benchmark results. I'm following the same pattern from the benchmark thread that was started HERE by daeron.be.

To start off, Da_G, a1d2catz, and Cotulla originally posted about the different builds. OndraSter posted to elaborate on a couple of the builds.

Branches of WM Development: Here is what all these different version numbers relate to, and a summary of their features.

212xx = AKU1, all builds leading up to and including WM 6.5
213xx = MOT motorola
214xx = ???
215xx = SAM samsung
216xx = HTC htc
217xx = COM1, continuing dev of 6.5.0.1 - 6.5.0.40
218xx = COM2, continuing dev of 6.5.0.50
219xx = MD, feature test branch, pretty much dead now. (unstable features are added here, this tree is based on COM1, so older base OS code, but the UI/UX code is newer)
22xxx = SEMC sony ericsson
*230xx = COM3, continuing development
*234xx = COM4, appears to be abandoned.
*235xx = COM5, more GUI changes here. New Outlook Interface.
*236xx = LG Electronics Branch
*24xxx = Possible HTC branch
*25xxx = SEMC - Sony Ericsson
*280xx, 282xx = WMD. This is a continuation of com3 from 23090. Most of the changes appear to be with IE
235xx is the only branch that has threaded email natively
290xx = Unknown branch

There was 219xx about half year ago, with numbers ending about 21936 (or maybe 45 or about that). It was test branch, where new features (like... supernew, something like previous 230xx where appeared huge softkeybar etc) were added. This branch was dead, just as looks 234xx now.

But, there was also COM2, with numbers 218xx. Well it reached its maximum, 21899. But instead of making new number line, MS/HTC chose to use already used 219xx. Here is where the mess comes from

The WM version numbers are as follows:
216xx = 6.5.3
219xx = 6.5.0
220xx = 6.5.3
231xx = 6.5.3
235xx = 6.5.5
236xx = 6.5.3
246xx = 6.5.3
250xx = 6.5.3
282xx = 6.5.3
290xx = 6.5.3

Updates:

14 March 2011: Finished benchmarks on 21689, 28245, 29011, 29013.

09 March 2011: New builds have begun rolling out. Testing begins tonight on 21687, 29008, and 29009.

...trimmed every month...

Results

I based these blank uncompressed ROMs on the latest shipped T-Mobile 6.5 ROM (1.91.531.4), disabled all ext packages, and swapped builds. I've listed all of the builds that I currently have. If you have one for me to test that's not on the list, send it my way as long as it's WVGA.

SYS builds tested:

216xx: 21659, 21661, 21663, 21665, 21671, 21674, 21677, 21679, 21680, 21681, 21682, 21683, 21684, 21685, 21686, 21687

219xx: 21904, 21905, 21907, 21908, 21909, 21911, 21914, 21916

220xx: 22013, 22018, 22019, 22021, 22022, 22024, 22027, 22031, 22036, 22037, 22038, 22040, 22041, 22042, 22044, 22046, 22047

231xx: 23120, 23121, 23129, 23130, 23132, 23133, 23134, 23135, 23136, 23138, 23139, 23140, 23142, 23144, 23145, 23146, 23148, 23149, 23150, 23151, 23152

235xx: 23569

236xx: 23651, 23654, 23656, 23658, 23659, 23662, 23664, 23667, 23670, 23676, 23678, 23680, 23683, 23686, 23688, 23689, 23690, 23691, 23694, 23697, 23698, 23699

246xx: 24609, 24610, 24611, 24614, 24618, 24619, 24620, 24626, 24627, 24628, 24630, 24631, 24635

250xx: 25018, 25024, 25026, 25027, 25028

282xx: 28233, 28236, 28237, 28238, 28240, 28242, 28243, 28244

290xx: 29002, 29003, 29005, 29007, 29008, 29009


Custom ROMs tested so far:

NRGZ28 Energy: 21916, 21684, 28244, 23148, 23699 - Update coming soon!

Ondraster LBFAR: 21899, 23569

dotcompt DeepShining: Coming soon!



Problem builds:
These are builds I was either unable to import into my kitchen, or cook for various reasons:
22016, 21664, 23123, 23125, 23126, 23127, 23566, 23694

Method and relevant information

If you are planning to send me results or anything, be sure to include the version numbers of the programs you used. SPBBenchmark and Test OpenGL are both freeware. Coreplayer Mobile costs around $30.

Benchmark programs used:
1. SPBBenchmark 1.6
2. Test OpenGL 18.01.2010
3. CorePlayer Mobile 1.2.5


Test Method for blank builds:
1. task29 then flash the ROM without the SIM card.
2. The storage card is left in to install the benchmark cabs.
3. Go into airplane mode.
4. Disable switching screen off and turning the device off.
5. Install benchmarking apps
6. Soft Reset
7. Run SPBbenchmark and disable the following:

• Built-in applications (whole category)
• Arkaball frames per second
• Copy 1 MB using memcpy

8. Run TestOpenGL for about 3 cycles of benchmarks (each cycle is 4 benchmarks) and then turn it off
9. Run three Avatar trailers, note down the results (%). Use default Coreplayer settings and run benchmark immediately after files are loaded.

• Trailer found HERE. Note that trailers ARE copyrighted material and are NOT subject to fair use. The link provided is to show what I am using. I own the BluRay disk and ripped it from there. The link is also 720p. My original clip was 1080p. Each exported trailer was set to 800x480 resolution.
• One trailer was converted to 25fps, 768 kbps - XHQ
• One trailer was converted to 23.976fps, 512 kbps - HQ
• One trailer was converted to 20fps, 384 kbps -MED
• I wasn't able to determine the original qualities from daeron.be's thread, so I made my own. If you know more about it, please let me know for more accurate results.

10. Run Video Flash benchmark with PIE using a downloaded version of the animation found HERE. If you plan on doing this yourself, download it to your device and run it with IE. Just put it in a root dictionary (\) and then input \benchmark.swf address in IE. On some ROMs you might have to wait a while before the flash starts (there won't be any signs of loading). Run the benchmark (if you want your result to be compatible with mine you MUST run it with flash file in full screen).
11. Collect results

Test Method for Custom ROMs:
Same as above with three exceptions.
1. Run SPB Benchmark with ONLY "Built-in applications (whole category)" test disabled. Leave everything else on.
2. Connect the device to the PC after running the SPB Benchmark test. You may have to go to Start > Settings > Connections > USBtoPC and untick the "Enable faster data synchronization" box to get it to work.
3. Copy the "Spb Benchmark Results.xml" from \My Documents to the PC and grab the results from there with the decimals. I use every decimal in the file. I then trim the trailing zeros in the Excel results file.

I figure since some people will probably use this as a guide of which ROMs are "best" or "fastest," that the results should be as accurate as possible, which is why I'm including the decimals in the custom ROM results, but not in the blank builds.

FAQ

When posting to this thread, please use proper English only, and do not use profanity.
You should not type like a high school girl, either. Leave the "plz," "sum1," and "u" back on the playground.
Ask for help like an adult and be treated like one.


1. What are these benchmarks for?
Basically this is just a measure of performance on different SYS builds to see which one is the fastest. These tests do not indicate which build is really “the best.” It is a measurement of speed on the process of copying files, reading directories, and playing video. To truly “test” a ROM, you need to spend a few days with it.

2. Which radio version are you using?
To the best of my knowledge, this has no affect on the results. However, my current radio version can be found in my sig. I do not change it for these tests.

3. Does a task29 before each flashing affect the results?
There are problems with "ghosting" in ROMs where residual data is left over from a previous ROM. To avoid this, I perform a task29 first. If you want to know more about task29, then visit that thread.

4. Are those benchmarks accurate?
I test the ROMs with SPBBenchmark once as it averages the results of 13 tests each. I use three rounds of OpenGL and use those results as it takes the highest result from there, which is what we want. An average in that category is useless as we want to know what the best result is for this, not the average. I run each Avatar trailer three times and take that average. Flash is pretty straightforward and kind of just for fun, so I only do one test. All of this means that the results may vary by a very small margin. If you don't believe my findings, you are always welcome to test yourself. Anything within a 1% margin of error is acceptable.

5. Why do you disable "Built-in applications (whole category)," "Arkaball frames per second," and "Copy 1 MB using memcpy" for SPB Benchmark on the blank builds?
They don't make a noticeable difference in the results. I could grab the *.xml results file from \My Documents, but it's not that big of a deal in the blank builds as they are just a stepping stone for chefs to see what they like and don't like. If you want to see the results, test them. I use them for custom ROMs because it makes sense to do so. People will interpret these results loosely, and it is better to be more accurate with them.

6. Why aren't you using Linpack?
Great question! I would if this were Android as there are current programs for it. The only one available for use on the WinCE platform is almost three years old at this point. Also, SPBBenchmark measures MFlops, so I don't really need it, do I?

7. Why don't you test battery life?
Battery life testing takes too long, plain and simple. When you flash a new ROM, getting the battery set properly takes about two or three cycles (drained to 10% and then charged to 100%) before it "settles." Even if I did nothing but play movies with the backlight and sound at max while downloading files via wifi and using Bluetooth file transfer, it would still take about an hour or so to drain the battery enough to charge it back again. You would need to drain and then charge AT LEAST once, preferably twice, before running a battery meter to test. This means I would be stuck on one ROM for hours before moving to the next one. My current set up allows for about 30-45 minutes per ROM to test. This is much more reasonable than 4 hours each, don't you think? If you decide to do test it yourself and want to share, then be my guest.

8. I didn't see you use xxxxx version of a build. Why is that?
I either don't have access to that build, or it isn't a WVGA build. If it is a WVGA build and I don't have it listed, send me a link, and I'll test it out.

9. Can I test the ROMs too?
Sure, why not? I won't use your results, though. There was a massive flame war in the aforementioned original thread that was partly fueled by other users posting results. I don't plan to start any nonsense here.

10. Would you mind testing a custom ROM for me?
YOU MUST GET THE CHEF'S PERMISSION FOR THEIR ROM'S RESULTS TO BE POSTED. I need them to PM me with an ok. I basically want it in writing. NOTE: Once a chef says it's ok to test, I will not remove them from the tests; make sure they understand this.

11. Why am I getting different results?
There are several little nuances that can influence the results in minor ways. If you're getting significantly different results than I am, then I'll look into it again. Keep in mind that using a different device than I am may affect the results as well. However, I will not use any other users' results without verifying them myself.

12. Some of this material looks familiar. Did you steal this?
Yes, basically I did. I figured I'd restart something similar for reasons I've mentioned as well as making it more Rhodium specific. Besides, the original poster, daeron.be, hasn't even logged on in a long time. His thread hasn't been updated since September of 2009 as well. I'm going to use most of their formatting and previous material to build on. If someone has a legitimate problem with this, then PM me.

13. I don't believe that you got "X" results with "X" ROM! I think it's WAY better than all the rest!
I don't care what you think. That isn't a question anyhow. I'm doing this for myself and sharing with you all. Either take it for what it's worth or do it yourself.

14. You didn't respond to my post. What gives?!
I have a life. I'm on XDA way more than I should be as it is. I will generally respond to questions and the like. If you just post to say "Good job" or "Thanks, man," don't expect me to respond. I might, just don't expect it. Just consider this answer here as a general "Thanks" and "You're welcome."

15. You're from New Orleans? What's it like?
People are wonderful; food is amazing; Mardi Gras rocks. Let's keep the thread on topic, please.

Requests

• If someone wants to change the BG colors of the spreadsheet to something they like better, be my guest. If I like the colors, I'll keep it.
• Please take the poll at the top of the page. If enough people use this thread with other devices, then I will consider moving it to the Chef's Central forum.
• I am slowly adding custom ROMs to the results. I need a few people to notify me when the ROMs you're using are updated. This includes Energy, Simplicity, DeepShining, Core Cell, and other popular ROMs.

Custom ROM testing

Ask your chef if he/she would mind having their ROM(s) tested and compared in my results. See below for rules:

• All custom ROMs would be separate of the clean builds, but compared all together.
• Understand that once permission is given, it cannot be taken away. (I don't want anyone trying to revoke permission after results are posted to take advantage of my system.)
• Have the chef PM me with the green light if they want it included.
• All results will either be performed by me, or verified by me.
• I will not measure battery performance. See FAQ.
• These results are not conclusive to picking a "good" ROM. It's a jumping off point. Test yourself and see which one works for you. This tests certain factors of the ROMs. This doesn't test for stability or feel of the ROM. Do that yourself.
• Chefs are allowed to "pick and choose" which ROMs they want tested. If they cook multiple flavors or builds, they can decline to have certain ones excluded from results.
• If you have a problem with my methods, you are welcome to do them yourself. I am doing my best to ensure that this is as objective as possible.

Contact
If you feel like contacting me or need to request info or whatever, you may PM, e-mail, or message me on Twitter.

Donate
Coffee keeps me cranking out results. DONATE to keep the caffeine flowing. If I see that you've donated, I will probably take a bit more care in listening to your problem/suggestion/comment. Make sure that you include your XDA user name so I can give you credit.

beautiful... cant wait to see results

Subscribed.


Imho you should test blank WM builds. Or at last be sure that the EXT/XIP packages are same. However the idea of testing different builds fits me.

Shame that usually the fastest builds leak memory like hell.

Originally Posted by Jackos

Subscribed.


Imho you should test blank WM builds. Or at last be sure that the EXT/XIP packages are same. However the idea of testing different builds fits me.

Shame that usually the fastest builds leak memory like hell.

I used to believe that until we started seeing the 21680 and 21681 builds come out. They really are quite faster than most of the others. When NRG was using the 23xxx builds, that was the case where the faster they ran the more memory was "spirited away," so to speak. I'm finding a much more stable experience with the 21680 and 21681 builds than most that I've seen.

In any case, the results will speak for themselves. The first three should be posted in a few hours or so.

build question

---

Originally Posted by cajunflavoredbob

I used to believe that until we started seeing the 21680 and 21681 builds come out. They really are quite faster than most of the others. When NRG was using the 23xxx builds, that was the case where the faster they ran the more memory was "spirited away," so to speak. I'm finding a much more stable experience with the 21680 and 21681 builds than most that I've seen.

In any case, the results will speak for themselves. The first three should be posted in a few hours or so.

Hi cajunflavoredbob,
After doing a little study on the benchmark masterlist, I was just wondering why the chefs aren't using build 23667 or 23691 (23667 with better results). Reading the above quote gives a little insight but was it because of the memory loss? The reason I'm asking is that I cannot find any ROMs based on these builds by energy, deepshining, etc. (possibly i'm not looking correctly).
Its just that i'm a nut for performance and the rhodium ain't a powerhouse with winmo on it, lol (also, 23667 stacks up very closely with 28244, which i really like)

Regards,
-Techkit

Originally Posted by cajunflavoredbob

Requests

The first thing I could use help with to speed up my testing is for someone to point me to the reg key that disables Sense. This would make it easier for me to just drop it into XDA_UC to have the changes made automatically.

[..]

This isn't a reg key, but I think there are mortscripts that disable Sense. Maybe this might help?

http://forum.xda-developers.com/show....php?p=8684203

I think JVH3 talked about scripts that disable sense too, so perhaps he might be able to help you out there too?

Great work, this is really interesting!