Xiaomi Mi Band data extraction, analytics and Google Fit sync tools
- Thread starter xmxm
- Start date
Hi!
I extracted files with Bonoboo tasker extraction tool, and got a Mi_Band_Table.csv
I tried to use fitnesssyncer to synd and i got the error "Unknown error has occurred. (Sun Feb 08 16:06:14 PST 2015)"
Any ideas? Maybe the csv from tasker is different? I wanted to use your extraction tool to try, however, my OSX isnt capable to adb right now (always show device offilne...)
I extracted files with Bonoboo tasker extraction tool, and got a Mi_Band_Table.csv
I tried to use fitnesssyncer to synd and i got the error "Unknown error has occurred. (Sun Feb 08 16:06:14 PST 2015)"
Any ideas? Maybe the csv from tasker is different? I wanted to use your extraction tool to try, however, my OSX isnt capable to adb right now (always show device offilne...)
Did you use latest version of Bonoboo tool?
How many columns do yo have in CSV file? There should be 23 columns for import to work properly.
If you already have 23 columns, could you please send me the CSV file or at least few lines from it for troubleshooting?
Last edited:
Thanks xmxm for your help.
Tried with the extraction tool apk and it's working... I will update the tasker to make it work...
Hope fitnesssyncer syncs with google fit someday... cause it would be wonderful to have all my tracks in one place (food on fatsecret, gym at google fit and runkeeper, and steps and sleep on miband...)
Tried with the extraction tool apk and it's working... I will update the tasker to make it work...
Hope fitnesssyncer syncs with google fit someday... cause it would be wonderful to have all my tracks in one place (food on fatsecret, gym at google fit and runkeeper, and steps and sleep on miband...)
Seems that fit2 works great but one thing though ;] What to do when I have fit installed on my phone and it gatheres steps and I add data from mi band. They sum up and I have twice as much steps ;p Will I have to uninstall google fit from my phone?
I hope Bonoboo updates his program, but in the meantime you can just take db\miband.sql file from v5.71 and replace the version of same file that comes with Tasker package.
If you have Google Fit on your phone and it already counts your steps, why would you want to Sync the data from Mi Band?
As far as I understand the way Google Fit works, my app could only delete data that was created by app itself, so you'll need to pick just one app to feed the data into Fit.
I guess it's because google fit app is less accurate than miband, and because we wear miband all day and not the mobile phone.
I think i understand his question, because i also have google fit app installed, so i can add sport activities, however i would like the steps to come from miband. Unfortunatly, google fit app doesnt let disable step count.
It might be technically possible but I do not know Unix/OSX shell scripting well enough to translate "for /F" behaviour and calling subroutines. You may be able to get native curl and jq apps for OSX and then try to manually reproduce all steps from the batch, but this might be a complex undertaking. I think that at this point it might be easier to install VM with Windows or use some Wine implementation to run it natively under Windows.
So my understanding is that you'd like to have steps coming from Mi Band, but to be able to enter other activities from Google Fit, right? I haven't used Fit app on my phone much, but I do not think it is counting steps for me. When I open settings in Google Fit on my smartphone, I see "Activity detection" switch which reads "allow Google Fit to collect & store data, including your location". I have this setting OFF. Could you try changing it to OFF and see if it stops step counting?
Last edited:
I don't know it it was the other user doubt... However for me it solved.
Thanks
I'd recommend to look into tools made by Bonoboo: http://xdaforums.com/showpost.php?p=56311839
I think that this might match what you are looking for: http://xdaforums.com/showpost.php?p=58714678
Similar threads
Top Liked Posts
-
There are no posts matching your filters.
-
56Current Version: Extraction Tool v5.92, Analytics Tool v5.60, iOS Extraction Tool v0.2, Fit Sync Tool v0.2 (Updated 6/20/2015)
Here is a tool I wrote for Xiaomi Mi band which provides following functionality:
1. Exports data from Xiaomi Mi band on both rooted and non-rooted Android phones and jailbroken (and possibly non-jailbroken) iPhones under Windows and Linux, generating file in CSV format which can be reviewed in Excel.
Bonoboo maintains a script to perform extraction directly on your smartphone (see link in post 3).
2. Extracts detailed (per minute) walking and running information from Mi Band database (thanks to the input provided by mritsch and Osid).
3. Produces files that can be uploaded to fitnesssyncer.com/ to syncronize data with various fitness services: FitBit/HealthVault/RunKeeper/JawBone and soon Google Fit! See 3rd post for details on data mapping for fitnesssyncer.
4. Provides multiple analytics reports to review extracted data. These reports can be shared on cloud drives to be accessed from anywhere (see post 3 for details)
5. Supports localized UI (currently supported languages are English, Russian, French, Spanish and Italian)
6. Support for synchronization data with Google Fit Service - see post #2 for details
How to use:
Preparation steps:
1. If you plan to use multiple packages, unpack them all to same directory.
1a. If you want to extract data from smartphone, download miband_extract package. To extract data from iOS devices, see to "To extract data from iOS devices" step.
1b. If you want to display local charts of your extracted data on your computer, download miband_analytics package.
1c. If you want to synchronize your extracted data with Google Fit, download miband_fit package.
2. Make sure you have USB drivers for your device properly installed and that your device is accessible by ADB when you connect it through USB
3. If you use it on windows, extract package has all binaries included, for Linux/OSX see comments below.
Checking configuration settings:
1. Review SDPath parameter value in run.bat/run.sh. The program will copy files from Mi app location to folder specified in SDPath before pulling them to desktop. In most cases default value (/sdcard) shoud work fine, however if your phone does not have this directory, find the path where your Internal/External SD is mounted and put that path string into SDPath value. Second most common value might be /storage/sdcard0
2. Review config.js and make any changes to your liking (set Goals for sleep hours and daily steps, force override UI language to specific value)
3. If you do not want main report being open every time you run extract, change OpenHTML=Y in run.bat/run.sh to OpenHTML=N
4. If your device is not rooted or have any issues with first (root) method that application uses and prefer to skip straight to the second (backup) method, set ForceBackupMode value to Y in run.bat.
5. If you want to generate Raw data extract in addition to summary data, set ExtractRaw parameter to Y, update Height parameter to your height in centimeters and update Weight parameter to your weight in kilograms. Please note that data extracted from raw blobs may be slightly different from total numbers in summary data.
If you are planning to use ADB over Wifi:
For Linux/OSX users:
Running application:
1. Connect phone through USB and make sure USB debugging setting is enabled on your phone. Unlock phone screen.
2. Execute run.bat - if your phone is rooted, the data would be pulled automatically. If your phone is not rooted you would see backup screen and you need to press "Back up my data" button in the bottom left corner.
3. Data from your mi band will be saved to extract.csv file and extract.js. After extraction is complete, if OpenHTML is set to Y, mi_data.html will be opened automatically to show charts for your Mi usage.
4. HTML reports are using Google Charts framework and Google TOS does not allow storing their scripts offline along with the application, therefore you will need to have working internet access for reports to work. Your data is not being sent to Google, the internet connection is only used to download latest version of Google Charts javascripts.
Troubleshooting connectivity issues:
To extract data from iOS devices:
Screenshots:
9Report descriptions:
mi_data.html:
This report shows overview of the data being extracted. It consists of 3 elements:
a. The chart on the left shows sleep data over time. Total bar height represents overall sleep time and colored segments represent light and deep sleep parts of it.
b. The chart on the right shows daily activity metrics over time. It shows total distance, number of steps and calories burned.
c. The chart on the bottom shows all data inside text table. You can sort the data by any column by clicking on the title of that column. If you select any specific row in the table, same day will be hightlighted in two other charts.
Both graphic charts also show target goals in green line, so you could see how well your results are matching target values. You can dynamically change target values by pressing + and - buttons. The table below has arrows next to Total Sleep and Total Steps columns which would be green for days when goal was met and red for days when goal was not achieved.
mi_data_byDoW.html:
This report allows you to see the scatter of your activities grouped by the day of the week.
This would be a good chart to see if you are performing better on any specific day of the week.
There are 4 charts on that report:
a. The left chart shows various metrics in scatter chart. You can select a specific metric from the dropdown located above the chart. You can also see a daily goal trendline in green and an approximation line in black. If the approximation line is not flat and you have statistics over many weeks, this might mean that your performance is consistently different depending on the day of the week.
b. The right chart shows line chart data for the metric selected in the left chart. It has time zoom controls at the top to only show the data over last day, last 5 days, 1 week, 1 month, 3 months, 6 months and 1 year. You can also select time interval manually by dragging time sliders at the bottom of the chart.
c. The chart at the bottom is a calendar view for your activity-related metrics. It shows a calendar and each days recorded activity is colored based on whether the value for that activity is above or below the goal. The days that have metric value below daily goal will be colored red, the days with metric value above the goal will be colored blue. The higher the deviation from the goal, the more intense color would be. In other words, the days where results are significantly above the goal would be colored in saturated blue and the days where results are significantly below the goal would be colored in saturated red.
This chart could help searching for more complicated patterns in your performance, patterns that are not just days of the week.
You can also change goal values dynamically by using + and - buttons at the top and see green trendline moving on the left chart and days changing colors on the bottom chart.
d. The last chart shows a timeline of your sleep on a daily level. This chart will only be visible if you select "Sleep Timeline" metric from the dropbox. This chart shows the time you went to sleep and the time you woke up. It shows this information for each day the history data is available and weekends are colored in a different color. This chart could help you to see how consistent your sleep/wake hours across different days and also if you have same sleeping patterns on weekends.
Google Fit Syncronization:
Limitations:
1. Daily summary statistics is synced, so charts by Week and Month will show correct data; chart by day will show correct totals, but not correct hourly breakdown.
Prerequisites:
1. You need to have at least version 5.71 of miband_extract package.
2. You will need to have Internet access and your browser should be configured to enable pop-ups at least from a specific domain below.
3. You need to open following URL: Get Fit Token and allow it to access your Google Fit data. Latest version of Chrome worked for me, while IE 11 had issues with popups.
Installation:
1. Download latest version of meband_fit package and unpack it to the directory where you have miband_extract v5.71 or above.
2. If you upgrade from previous version of fit sync package, you will need to clean up old data populated by old version. To do so run following command:
fit clean
Running Sync:
1. Synchronize your data from smartphone to desktop.
2. Obtain a new OATH key for syncing data with Fit - open Get Fit Token and copy resulting string to your clipboard.
3. Run fit.bat and when asked for new token paste the string from previous step and press enter. If fit.bat did not ask you for token and failed with Authentication error, just run it again.
Troubleshooting:
1. The process is logging into fit\*.log files. If you have errors, review these logs first.
2. The token from step 2 is valid for 1 hour - you will need to refresh the token after that. After 1 hour subsequent fit syncs will start failing until you obtain a new token.
3. Current token is cached in GKey environment variable. Once batch determines that old token is expired, it would clean the variable and will ask you for a new key on a subsequent run.6Related links:
Latest version of Bonoboo port for smartphone extraction.
Known Issues:
Version History:
Syncronizing with fitnesssyncer.com
Accessing your charts directly from Dropbox:
Accessing your charts directly from Google Drive:
Using Helium backup instead of Android backup:
Plans for future versions:
How to translate application to your language:
5
I'll share my findings.
Sorry for my bad English.
Mi band sends ActivityData to app. Each ActivityData is 1 minute long and are 3 bytes.
(category, intensity, steps)
steps is the most obvious.
You'll find those bytes in data blob of date_data table or, if you have miband_log.txt on /sdcard in JSON sended to server (data property, base64 encoded).
App "rewrites" them in SportData class.
SportData is
(timeIndex, activity, mode, runs, step)
timeIndex = minute of day (1-1440)
step = steps in a minute
the others are calculated as follows (pseudo Java code):
category, intensity, steps
if (category == 0x7f) {
mode = category;
} else {
mode = category & 0xf;
if (step > 0) {
runs = category >> 4;
}
activity = intensity
step = steps
Now, some formula which Mi fit uses to calculate distance and calories burned every minute
d = height_in_cm * 3 / 100
iDistance = (step <= 90) ? (step*d*0.9) : (step^2*d)/(step > 120 ? 125 : 100)
runDistance = (((runs * 2) + 3) * iDistance) / 15;
For calories it uses a lookup table of doubles:
calList = {40.233d, 0.95d}, {53.645d, 1.19d}, {67.056d, 1.41d}, {80.467d, 1.57d}, {93.878d, 1.78d}, {107.29d, 2.36d}, {120.7d, 2.97d}, {134.11d, 3.79d}, {160.94d, 4.67d}, {187.76d, 5.24d}, {214.58d, 5.62d}, {241.4d, 6.1d}, {268.23d, 6.91d}, {295.05d, 7.62d}, {321.87d, 9.05d}, {348.69d, 9.43d}, {375.52d, 10.95d}};
calList index is the max of calList 1st value less than iDistance.
cals = (weight_in_kg * 2.2046 * calList i-th tuple[1] * iDistance) / (60 * calList i-th tuple[0])
runCalories = (cals * ((runs * 2) + 3) / 15);
Some examples taken for logcat:
03-13 23:05:54.220 5092-5140/com.xiaomi.hm.health D/dataProcess? DataAnalysis i=2274:steps=67
03-13 23:05:54.220 5092-5140/com.xiaomi.hm.health D/dataProcess? DataAnalysis iDistance=42.547680,p=1,cals=119.911552
03-13 23:05:54.220 5092-5140/com.xiaomi.hm.health D/dataProcess? DataAnalysis i=2275:steps=100
03-13 23:05:54.220 5092-5140/com.xiaomi.hm.health D/dataProcess? DataAnalysis iDistance=70.560000,p=3,cals=123.047801
03-13 23:05:54.220 5092-5140/com.xiaomi.hm.health D/dataProcess? DataAnalysis i=2276:steps=97
03-13 23:05:54.220 5092-5140/com.xiaomi.hm.health D/dataProcess? DataAnalysis iDistance=66.389904,p=2,cals=126.227996
BTW: I'm 168cm and 62kg
After that algorithm finds intervals. How?
for (Iterating over all sportDatas, i used as index [minute]) }
startMode = sportData.mode;
if (spordData.mode != sportData[i-1].mode) {
if ((sportmode[i - 1] == 0 && sportmode - startmode >= 10) || !(sportmode[i - 1] == 1 || sportmode[i - 1] == 2 || sportmode[i - 1] == 0)) {
this is an interval startMode -> i-1
}
} else {
i++;
}
}
Intervals are discarded.
The remaining gaps are filtered (I have't still reverse engineered Java method filterData) and under some conditions it finds sections which are the real sections showed in graphs on Mi Fit app.
I haven't see SleepAnalysis.
I've developed, for internal use, an Xposed module which hooks into MiLiProfile class of Mi Fit and through intents is able to control vibration, LEDs, realtime steps, sensors, etc.
Maybe, in the future, I'll release it with Tasker integration and/or as a standalone app (without Xposed) if there's some interest.
Hope to help you
