[Updated 11-Dec-2018][oxygen] Battery Life / Performance Benchmarks - AOSP Roms

[svbarbosa]

[22-Nov] NEW: ArrowOS v8 with alternative cpu governors. Check it out in post #127.

Here is my first series of PC Mark Battery Life Benchmarks for the Xiaomi Mi Max 2.

Let me first say that is not a "best rom" thread. There is no absolute best rom. What is best for one user may not be best for another. To give a simple example, battery life is very important for some users while performance is priority for others.

Benchmark results are usually misleading, but PC Mark battery life benchmark shines by using day to day tasks to measure phone performance and battery drain. It repeats for many hours, until battery drops below 20%, a series of tasks including web browsing, photo and video editing, text editing and data manipulation. Then it extrapolates the SOT that would be obtained with 95% of the battery charge.

For each rom I made a clean install and a simple setup with basic settings and a added a few apps necessary to monitor/admin the runs.

The tests are run with the screen brightness calibrated at 200 lumens as required by PC Mark and all sound levels set to zero, so results are better suited for comparison.

Each benchmark run takes between 9 and 11 hours. So this test is also a stability test. Some rom/kernel combinations simply fail to complete the benchmark.

If you want to learn more about PC Mark take a look at the site: PC Mark for Android.

These are the roms included so far:

• AospExtended-v4.6-oxygen-20180425-0949-UNOFFICIAL by @eva0034 - [ROM][7.1.2_r29][UNOFFICIAL] AospExtended ROM V4.6 [OMS/DUI]
• lineage-14.1-20180318-UNOFFICIAL-oxygen by @ashwin007 - [ROM+TWRP][UNOFFICIAL][oxygen][7.1.2] LineageOS 14.1
• DOT-N-v1.2-20180503-oxygen-UNOFFICIAL by @Jay_B - [Unofficial][ROM][7.1.2] dotOS VoLTE (Airtel and Vodafone Working)Final
• Lineage 14.1 (30.10.2017) by @xuefer - LineageOS 14.1 for MI MAX 2 (oxygen)
• XenonHD-180329-Unofficial-oxygen by @Jay_B - [UNOFFICIAL] XenonHD_Oxygen 7.1.2 With Substratum Support Final
• RR-N-v5.8.5-20180412-oxygen-Unofficial by @iscle - [ROM][FINAL][7.1.2][2018-04-12] ResurrectionRemix 5.8.5 [UNOFFICIAL]
• RR-N-v5.8.5-20180321-oxygen-Unofficial by @kenzolo - [21March][ROM][UNOFFICIAL] Resurrection Remix v5.8.5 [7.1.2][Nougat]

Some might be wondering why it took so long for me to add first results.

I have already run literally hundreds of benchmarks, mainly for the Samsung Galaxy Note 3 and the Xiaomi Redmi Note 4. I had very few situations where a benchmark would not finish.

With the Mi Max 2 I had to spend almost a month testing a few roms and modifying my test environment so that a benchmark would finish. It was happening with every rom I tried. Very frustrating.

Eventually I reduced the number of apps I use during tests from 22 to 8 and was able to start completing benchmarks. Failures still happen, but now at least I have an around 60% success rate.

I added a line with the number of failures for each rom in my results table. Keep in mind that failing the benchmark does not indicate that the rom is unstable and not suited to be a daily driver. These roms may never fail in normal day to day use.

I also noticed that battery SOT varied a lot between my two Mi Max 2, so I had to give up using both. All tests were run on the same phone.

Anyway now, even with only one phone, progress will be faster.

Enough talk. Now Let's see the results:

Score is a performance index - the higher the better.

SOT (Screen On Time) is an estimate of the number of hours the device may be used until battery drops almost dead (from 100% to 5%). It's a very good estimate because the test starts with the battery at 80% and ends when the battery drops below 20%.

There is no clear advantage comparing the roms tested so far. They all offer similar performance and battery drain.

So any of these roms will serve you well and you should pick one based mainly on features you want the most.

11-Jun-2018 Update:

Added three more roms to the results table.

I was so wrong when I predicted next results would come faster.

Lost about 10 day trying every possible way to benchmark ashwin´s LA15.1, but benchmarks would simply not finish.

Also had big variances between runs for Jay_B´s XenonHD and iscle´s RR-N. So I ran extra benchmarks to have more meaningful averages.

With more Oreo roms showing up my plan is to focus on Oreo roms for the next benchmarks.

29-Jul-2018 Update:

For the last (revised) preview of Oreo Roms head to post #87.

16-Aug-2018 Update:

For the last preview of Oreo Kernels head to post #95.

22-Nov-2018 Update:

Time for Pie, thanks to @hardrock_82. Check it out in post #105.

.

 

[svbarbosa]

Instructions to vote in poll:

If you are using an app to access the thread, the poll will not be displayed.

To access the poll you must open the thread using a browser, login, and then you see the poll and be able to vote.

In XDA Labs, for example, once viewing the thread, you may open the three dots menu on the top right and just use the "Load in browser" option.

In other apps there is probably a similar option.

 

[svbarbosa]

Also mine!

 

[svbarbosa]

One more!

 

[svbarbosa]

Ok. Enough.

 

[jbmc83]

haha i especially like the part about @pranavaryans slowing down for you to catch up

prioritizing via poll is a good idea, curious to see what the users will choose

what kind of benchmarking tools will you be using? same as with the redmi note 4?

and what kind of rom settings? especially concerning performance mode, kernel settings and powered sensors (wifi, bt, gps, etc.)

Sent from my Xiaomi Mi Max 2 (Oxygen) using Tapatalk

 

[congminhmc]

Xuefer's LOS 14.1 still the best rom for me :laugh: with "Brevent" Module to increase the battery life and no bugs in my opinion

 

[jbmc83]

Xuefer's LOS 14.1 still the best rom for me :laugh: with "Brevent" Module to increase the battery life and no bugs in my opinion

@svbarbosa contact me via pm if ure interested to include xuefer´s LOS 14.1 build

 

[svbarbosa]

What kind of benchmarking tools will you be using? same as with the redmi note 4?

I use PC Mark for Android - Battery Life Benchmark.

Benchmark results are usually misleading, but PC Mark battery life benchmark shines by using day to day tasks to measure phone performance and battery drain. It repeats a series of tasks including web browsing, photo and video editing, text editing and data manipulation.

I am predicting each test run on the Mi Max 2 will take 10-12 hours. During all time the phone has to remain untouched. Receiving a call or SMS makes the test stop.

For more info take a look at their site: PC Mark for Android.

And what kind of rom settings? especially concerning performance mode, kernel settings and powered sensors (wifi, bt, gps, etc.)?

I am a little paranoid with the benchmarks. I try my best to have comparable results by doing the following:

- Clean install the rom
- Install always the same 20 apps, mainly support utilities, using Titanium Backup to restore them
- Use the same basic settings (no optimizations - no kernel tweaks)
- Sound always set to zero
- Brightness always set to 200 lx using the same meter (as required by PC Mark)
- All auto brightness settings disabled
- Wifi on and always connected to the same router
- Bluetooth always on but not connected to any device
- Location set to High Accuracy
- Benchmark always start with battery at 80%
- Reboot phone before each test
- Phone stays still always in the same place in my home office

 

[jbmc83]

I use PC Mark for Android - Battery Life Benchmark.

Benchmark results are usually misleading, but PC Mark battery life benchmark shines by using day to day tasks to measure phone performance and battery drain. It repeats a series of tasks including web browsing, photo and video editing, text editing and data manipulation.

I am predicting each test run on the Mi Max 2 will take 10-12 hours. During all time the phone has to remain untouched. Receiving a call or SMS makes the test stop. [emoji14]

For more info take a look at their site: PC Mark for Android.



I am a little paranoid with the benchmarks. I try my best to have comparable results by doing the following:

- Clean install the rom
- Install always the same 20 apps, mainly support utilities, using Titanium Backup to restore them
- Use the same basic settings (no optimizations - no kernel tweaks)
- Sound always set to zero
- Brightness always set to 200 lx using the same meter (as required by PC Mark)
- All auto brightness settings disabled
- Wifi on and always connected to the same router
- Bluetooth always on but not connected to any device
- Location set to High Accuracy
- Benchmark always start with battery at 80%
- Reboot phone before each test
- Phone stays still always in the same place in my home office

very thought out experimental setup my friend, good job there

Sent from my Xiaomi Mi Max 2 (Oxygen) using Tapatalk

 

[chetchuakaka]

Good job man

 

[pranavaryans]

lol

Nice to know that my name is mentioned there.
You are doing great work on that OP, keep it up.

And yes, i have slowed down for now.

 

[destroyer71]

And what is the meaning of these synthetic tests? They do not give information in full. There are so many other factors. I know 3 Roms, where there are problems with voice communication. On 2 other "departing" applications. Sense in such tests? For example, we see one of the highest scores in these tests. But there will be problems with communication and other things. What is the point then? Only because of the greatest number of points in the test? For daily use, they are not suitable. Everyone has their own personal preferences. I tested all the roms for this device. For me personally, the best of them - crDroid and AOSP Extended from pranavaryans. Now I use AOSP Extended from pranavaryans
Excellent rum. But why is it not in the survey. All this is my personal opinion. Personally for me, such a waste of time. Everyone is very individual. The test is just numbers. They do not give complete information.
But ... maybe someone will be interested. Good luck

 

[jbmc83]

And what is the meaning of these synthetic tests? They do not give information in full. There are so many other factors. I know 3 Roms, where there are problems with voice communication. On 2 other "departing" applications. Sense in such tests? For example, we see one of the highest scores in these tests. But there will be problems with communication and other things. What is the point then? Only because of the greatest number of points in the test? For daily use, they are not suitable. Everyone has their own personal preferences. I tested all the roms for this device. For me personally, the best of them - crDroid and AOSP Extended from pranavaryans
. Now I use AOSP Extended from pranavaryans
Excellent rum. But why is it not in the survey. All this is my personal opinion. Personally for me, such a waste of time. Everyone is very individual. The test is just numbers. They do not give complete information.
But ... maybe someone will be interested. Good luck

this thread isnt supposed to be an "end all discussion" solution concerning the "best" rom. u said it yourself, everyone has their own personal preferences! so what if someone doesnt care about those "departing apps" because hes using different apps, but really cares about cpu / gpu performance due to gaming?

this data doesnt give you an excuse not to do your due diligence, so ALWAYS read the respective rom threads to find out what kind of bugs / functionality they have.

BUT

it helps you make a more informed choice when, say, youre stuck between 2-3 roms which might be very similar in mods and functionality / design. having more information at your disposal is always good, so this is far from being a "waste of time"


I dont get this negative vibe from a lot of users here. trying to shoot down threads, discourage devs or criticize projects such like this which take a LOT of time and work for their initiators to complete. And all that because "they personally" are not interested in it and think its "a waste of time"? Well guess what: not everyone shares your opinion / your everyday usage cases / your preferences concerning ROMs, Apps, settings, design choices, etc. And thats totally fine! Everyone has different requirements, so lets just collect all the info we can so that everyone can make his/her own choice.

What would you get if everything was shut down just because we have different requirements? Well then there wouldnt be ANY choices left. Congratz, youve just made XDA superfluous

 

[destroyer71]

this thread isnt supposed to be an "end all discussion" solution concerning the "best" rom. u said it yourself, everyone has their own personal preferences! so what if someone doesnt care about those "departing apps" because hes using different apps, but really cares about cpu / gpu performance due to gaming?

this data doesnt give you an excuse not to do your due diligence, so ALWAYS read the respective rom threads to find out what kind of bugs / functionality they have.

BUT

it helps you make a more informed choice when, say, youre stuck between 2-3 roms which might be very similar in mods and functionality / design. having more information at your disposal is always good, so this is far from being a "waste of time"


I dont get this negative vibe from a lot of users here. trying to shoot down threads, discourage devs or criticize projects such like this which take a LOT of time and work for their initiators to complete. And all that because "they personally" are not interested in it and think its "a waste of time"? Well guess what: not everyone shares your opinion / your everyday usage cases / your preferences concerning ROMs, Apps, settings, design choices, etc. And thats totally fine! Everyone has different requirements, so lets just collect all the info we can so that everyone can make his/her own choice.

What would you get if everything was shut down just because we have different requirements? Well then there wouldnt be ANY choices left. Congratz, youve just made XDA superfluous

You misunderstood what I wanted to say. I'm not against flows, quite the opposite. I expressed only a personal opinion about such tests. If it helps someone, that's fine. The results of daily use will differ from the test results. That's what I wanted to say

 

[jbmc83]

You misunderstood what I wanted to say. I'm not against flows, quite the opposite. I expressed only a personal opinion about such tests. If it helps someone, that's fine. The results of daily use will differ from the test results. That's what I wanted to say

yes, totally agree on that. battery life is extremely dependant on personal usage cases. still, since your own usage pattern basically stays the same no matter which rom ure using, it makes sense to compare the roms with one and the same strategy.

lets say testing shows that ROM A provides 2h more SOT than ROM B, with a total of 14h SOT on A vs. 12h SOT on B.

With your usage pattern you might only get 10h SOT on ROM A due to heavy usage of GPS and LTE mobile data. Still, due to overall higher system energy consumption on ROM B, you know based on the given testing data above that your own usage pattern will most likely net less SOT on ROM B than on ROM A. So in your case, you already know that you might get 8h SOT on ROM B just by looking at the 14h vs. 12h data from testing

 

[mesarabi]

When you start to test roms?did you wait for specific vote number?I'm just doesn't unlock my device because of your test
I'm waiting to see the results then compare them to stock miui and then decide what to do because I can't see the big devlepors(with respect to all of them) or teams for this device(like leedroid or viper or eliminator purefusion for htc)
So I make a choice totally base on your tests
Thank you so much for your time spending for community

 

[jbmc83]

When you start to test roms?did you wait for specific vote number?I'm just doesn't unlock my device because of your test
I'm waiting to see the results then compare them to stock miui and then decide what to do because I can't see the big devlepors(with respect to all of them) or teams for this device(like leedroid or viper or eliminator purefusion for htc)
So I make a choice totally base on your tests
Thank you so much for your time spending for community

i can tell you right now that it will not make sense for you to wait for his tests to be complete this will take weeks if not months, i doubt ull wait that long

 

[mesarabi]

i can tell you right now that it will not make sense for you to wait for his tests to be complete this will take weeks if not months, i doubt ull wait that long

I thought he starts soon(for example this week or next) and test roms with some scripts and then test another one
So I thought test will take at most two weeks
Am I wrong?he doesn't use script for test?or benchmark app?like gsmarena or others

 

[svbarbosa]

And what is the meaning of these synthetic tests? They do not give information in full. There are so many other factors. I know 3 Roms, where there are problems with voice communication. On 2 other "departing" applications. Sense in such tests? For example, we see one of the highest scores in these tests. But there will be problems with communication and other things. What is the point then? Only because of the greatest number of points in the test? For daily use, they are not suitable. Everyone has their own personal preferences. I tested all the roms for this device. For me personally, the best of them - crDroid and AOSP Extended from pranavaryans. Now I use AOSP Extended from pranavaryans
Excellent rum. But why is it not in the survey. All this is my personal opinion. Personally for me, such a waste of time. Everyone is very individual. The test is just numbers. They do not give complete information.
But ... maybe someone will be interested. Good luck

You misunderstood what I wanted to say. I'm not against flows, quite the opposite. I expressed only a personal opinion about such tests. If it helps someone, that's fine. The results of daily use will differ from the test results. That's what I wanted to say

Yes, benchmarks are misleading most of the times. I never liked benchmarks myself. Real life usage is affected by so many factors.

But PC Mark Battery Life is not the usual benchmark. You should visit their site to better understand what it does.

PC Mark for Androd - A Better Benchmark for Android Devices

In a few words, this benchmark runs a series of simulations of day-to-day tasks , including web browsing, video watching, text writing and photo editing. The series is repeated until the battery drops below 20% and then you get an extrapolated SOT and a performance index.

I start the test with the battery precisely at 80%, so it uses about 60% of the battery charge, and the extrapolated SOT is very meaningful. The phone has to remain still during all time. If you run any app or receive a call/SMS while the benchmark is running, PC Mark simply aborts the test.

Since the benchmark runs for hours it is also a stability test. If the rom has, for example, memory leaks, it fails the test. I had it happening a few times with the two other devices I tested.

I truly believe my work produces useful info to help users make a more educated rom choice.

Of course there are many other important points that should be checked when choosing a custom rom, like customization features, active development/support, user base, etc... and the info I provide should never be the only basis for a decision.

When you start to test roms?did you wait for specific vote number?I'm just doesn't unlock my device because of your test
I'm waiting to see the results then compare them to stock miui and then decide what to do because I can't see the big devlepors(with respect to all of them) or teams for this device(like leedroid or viper or eliminator purefusion for htc)
So I make a choice totally base on your tests
Thank you so much for your time spending for community

Tests will start this weekend. I am having an unusualy tough work week, with long hours.

The votes will only help me defining a test sequence and have a feeling of how much people are really interested.

Running these benchmarks is a very time consuming task. The phones remain unusable for days/weeks.

The more people interested, the more motivation I have to keep benchmarking roms and kernels (yes, kernels eventually will appear for the Mi Max 2).

Right now I am benchmarking all important kernels for the mido (Redmi Note 4). There are 10+ kernels available.

The Mi Max 2 will be the third device I run benchmarks.

It all started for my main device, the venerable Samsung Galaxy Note 3, because its SOT is usually 3-4 hours, and battery life is a really important factor when choosing a rom. For the Note 3 I made a few scattered XDA posts. No dedicated thread.

Then I got my midos and decided to open a thread for my benchmarks. It was very welcome by the mido community, including developers, because the results helped them adjusting roms to better achieve their goals.

Now here I am with two Mi Max 2. Let´s see how it goes....