WhatsApp backup smaller, strangely. Anyone else?

[xda-eh]

A recent WhatsApp backup was around 100MB smaller than the previous one. Whereas I would have thought that it should have been slightly bigger. The same thing happened like two or three months ago as well, where a backup was also around 100MB smaller than the previous one (which used to be almost 500MB).

Is anyone aware of any changes that WhatsApp implemented that caused the backups to be smaller due to a new compression method or something else?

Thank you.

 

[xda-eh]

So it's possible you could've deleted some messages, right? Since there's no way its going to decrease for no reason.

No. Deleted messages would take away a few KB or MB at most. Not the huge percentage that's been reduced.

 

[Schrubbi]

I'm facing the same issue for three times already: From 13th to 14th of April '22 by message database shrunk by ~20% Between Jan and Feb '23 by another 10% Between 4th and 7th of April by almost 60%!

In total the database shrunk from 140MB to now 46MB, in one year, without having any messages deleted. All events seem unrelated to .cryptX changes. It's hard to believe that the compression has been improved by 3x. Whatsapp support was unable to provide any technical information or background on the first occurrence last year.

Another report of this issue can be found here: https://android.stackexchange.com/q...backup-shrinks-in-size-for-no-apparent-reason

I would assume it's easily possible that also the unencrypted message database is reduced in size, unfortunately can't check the internal files due to lack of root access.
Maybe anyone could confirm?

 

[WoKoschekk]

I'm facing the same issue for three times already: From 13th to 14th of April '22 by message database shrunk by ~20% Between Jan and Feb '23 by another 10% Between 4th and 7th of April by almost 60%!

In total the database shrunk from 140MB to now 46MB, in one year, without having any messages deleted. All events seem unrelated to .cryptX changes. It's hard to believe that the compression has been improved by 3x. Whatsapp support was unable to provide any technical information or background on the first occurrence last year.

Another report of this issue can be found here: https://android.stackexchange.com/q...backup-shrinks-in-size-for-no-apparent-reason

I would assume it's easily possible that also the unencrypted message database is reduced in size, unfortunately can't check the internal files due to lack of root access.
Maybe anyone could confirm?

I compared an older msgstore.db of Whatsapp from 2020 (Titanium Backup) with my current msgstore.db from /data/data/com.whatsapp/databases and did an analysis with the Termux command file. The output was the same for the *.db meta data. No changes.

It's possible to compress databases but it creates a special file format. So, the output of file shows that the current database isn't compressed. To cross check it I also archived the current database as a zip file and that reduced the size more than 50%. I assume that wouldn't be possible if it was already compressed before.

In total the database shrunk from 140MB to now 46MB,

A *.db file contains only raw text and even with hundreds of chat entries you'll never get a database > 100MB. The most of your 140MB should be media data that is not part of msgstore.db.

 

[Schrubbi]

I compared an older msgstore.db of Whatsapp from 2020 (Titanium Backup) with my current msgstore.db from /data/data/com.whatsapp/databases and did an analysis with the Termux command file. The output was the same for the *.db meta data. No changes.

It's possible to compress databases but it creates a special file format. So, the output of file shows that the current database isn't compressed. To cross check it I also archived the current database as a zip file and that reduced the size more than 50%. I assume that wouldn't be possible if it was already compressed before.


A *.db file contains only raw text and even with hundreds of chat entries you'll never get a database > 100MB. The most of your 140MB should be media data that is not part of msgstore.db.

The 140MB refers to msgstore.db.crypt15 in it's former version.
I haven't looked into this db for some years, but it didn't use to contain any media files, except maybe for tiny previews not larger than a few kb. Do you know what's exactly contained in that file?
As my database contains thousands of messages of more than 10 years now, I wouldn't exclude that they can make up to several 10s of MBs.

Thanks for looking into the db file meta data.
Not sure what to make out of it though, as the only possible conclusion from your statement would be that the backup now contains less than a third of my original messages. I wouldn't expect that Whatsapp is so sloppy of making backups lose such an amount of data.
But still I don't have any better hypothesis what's actually causing the reduced file size.

Maybe I could use a formerly retrieved encryption key to open the .db and compare their contents.

 

[WoKoschekk]

To open the encrypted msgstore.db.crypt14 files you'll need the key stored as ~/com.whatsapp/files/key. Only accessible with root.

Another way would be to install a VM like BlueStacks, root it and install Whatsapp on it OR ask someone with a rooted device. With both methods you can extract the msgstore.db.
It contains everything related to your chats except the media files and any settings. See attachment for all the tables inside it. All databases of Whatsapp:

axolotl.db
chatsettings.db
commerce.db
companion_devices.db
daily_metrics.db
emojidictionary.db
hsmpacks.db
location.db
media.db
msgstore.db
payments.db
stickers.db
sync.db
wa.db

 

[world]

To open the encrypted msgstore.db.crypt14 files you'll need the key stored as ~/com.whatsapp/files/key. Only accessible with root.

Another way would be to install a VM like BlueStacks, root it and install Whatsapp on it OR ask someone with a rooted device. With both methods you can extract the msgstore.db.
It contains everything related to your chats except the media files and any settings. See attachment for all the tables inside it. All databases of Whatsapp:

axolotl.db
chatsettings.db
commerce.db
companion_devices.db
daily_metrics.db
emojidictionary.db
hsmpacks.db
location.db
media.db
msgstore.db
payments.db
stickers.db
sync.db
wa.db

This thread started a whole theory......

lol

 

[xda-eh]

This thread started a whole theory......

Makes sense, considering WhatsApp's strange behavior lately.

I wouldn't expect that Whatsapp is so sloppy of making backups lose such an amount of data.

I also wouldn't.

But recently I looked at a chat that started in the last few months and saw that my first message disappeared, and the conversation starts with the other person's reply to me. I subsequently looked at other chats and/or groups and noticed that a few of them also have one or more missing messages at the beginning. So it would seem that WhatsApp indeed started getting sloppy recently.

 

[Schrubbi]

I subsequently looked at other chats and/or groups and noticed that a few of them also have one or more missing messages at the beginning. So it would seem that WhatsApp indeed started getting sloppy recently.

That's really odd behavior! Have you tried to contact Whatsapp support?

After pushing them a bit in the quite obvious direction, they were at least able to confirm that this shrink is intentional. I'll let them double confirm there's no lost data due to it, but it seems like Whatsapp did indeed optimize drastically the way message backups are stored. Still surprising it's not signed with a change in .cryptX but if that really only refers to encryption methodology, that could explain it.
Overall very much appreciated I have to say.

 

[xda-eh]

Have you tried to contact Whatsapp support?

Yes. Completely worthless. A bot or dumb human kept on replying with long generic messages related to keywords in my message, but no actual answer to my specific question.

 

[Pralltiller]

I've just stumbled across some WhatsApp backups from mine (from several devices, dating back to 2013) and I also wondered about the drop in file size on my newest device. Google lead me to this thread and since nobody came up with a plausible reason, I registered an account here to share my findings with you:

I decrypted two backups of mine. One is from Feb '23 (65.4 MB) and the other one is from yesterday Jun '23 (24 MB). I have never deleted any messages so I was curious as of what data has changed between the backups.

I decrypted both backups (using wa-crypt-toolswa-crypt-tools). Both backups have a .db.crypt14 extension (although I had to recover them using different keys). After decryption the uncompressed file size changed to this:

• Feb '23 Backup:
  • Encrypted/compressed: 65 MB
  • Decrypted/uncompressed: 127 MB
• Jun '23 Backup:
  • Encrypted/compressed: 24 MB
  • Decrypted/uncompressed: 50 MB

Note that I haven't used WhatsApp in a long time, so the changeset of these two files should be minimal (just a few group messages).

I used sqldiff do compare what has changed between the two databases. And from the looks of it they simply dropped a few indexes (without replacement) and migrated some tables to a new structure. Just by dropping the indexes on the old database lead to a decrease in size from 127 MB to 90MB. I am still not certain how the remaining 40MB file size difference (90MB vs 50MB) can be explained, but I assume it comes from the the new table structures (which involves dropping a few tables).

This is an excerpt of the changeset, for those of you that are currious:

sqldiff msgstore.february.db msgstore.latest.db | grep "DROP":

Spoiler: Dropped indexes and tables

DROP INDEX chat_assignee_index;
DROP INDEX agent_device_index;
DROP INDEX call_link_token_index;
DROP INDEX call_log_key_index;
DROP INDEX call_log_participant_key_index;
DROP INDEX chat_group_type_index;
DROP INDEX deleted_chat_job_index;
DROP INDEX frequent_index;
DROP INDEX group_notification_version_index;
DROP INDEX group_participant_device_index;
DROP INDEX group_participant_user_index;
DROP INDEX group_participants_index;
DROP INDEX group_participants_history_index;
DROP INDEX group_past_participant_user_index;
DROP INDEX group_past_participant_user_timestamp_index;
DROP INDEX jid_raw_string_index;
DROP INDEX jid_key_new_index;
DROP INDEX jid_lid_map_jid_index;
DROP INDEX joinable_call_log_call_id_index;
DROP INDEX joinable_call_log_group_jid_row_id_index;
DROP INDEX labeled_jid_index;
DROP INDEX labeled_jids_index;
DROP INDEX labeled_messages_index;
DROP INDEX labels_index;
DROP INDEX message_key_index;
DROP INDEX message_starred_index;
DROP INDEX message_type_chat_index;
DROP INDEX message_chat_id_index;
DROP INDEX message_sort_id_index;
DROP INDEX message_type_index;
DROP INDEX message_chat_sort_id_index;
DROP INDEX message_starred_sort_id_index;
DROP INDEX message_add_on_key_index;
DROP INDEX message_add_on_parent_message_row_id_index;
DROP INDEX message_add_on_chat_status_index;
DROP INDEX message_add_on_status_index;
DROP INDEX message_add_on_orphan_key_index;
DROP INDEX message_add_on_orphan_parent_key_index;
DROP INDEX message_add_on_poll_vote_selected_option_message_add_on_row_id_index;
DROP INDEX message_add_on_receipt_device_index;
DROP INDEX message_add_on_receipt_device_jid_index;
DROP INDEX message_details_author_device_jid_index;
DROP INDEX message_ephemeral_expire_timestamp_index;
DROP TABLE message_ftsv2_content;
DROP TABLE message_ftsv2_docsize;
DROP TABLE message_ftsv2_segdir;
DROP TABLE message_ftsv2_segments;
DROP TABLE message_ftsv2_stat;
DROP INDEX message_link_index;
DROP INDEX message_media_chat_index;
DROP INDEX message_media_hash_index;
DROP INDEX message_media_original_file_hash_index;
DROP INDEX message_media_interactive_annotation_index;
DROP INDEX message_media_interactive_annotation_vertex_index;
DROP INDEX message_media_vcard_count_index;
DROP INDEX mentions_index;
DROP INDEX message_orphaned_edit_key_index;
DROP INDEX message_poll_option_message_row_id_index;
DROP INDEX quoted_mentions_index;
DROP INDEX message_quoted_vcard_index;
DROP INDEX message_system_chat_participant_index;
DROP INDEX message_template_index;
DROP INDEX message_template_button_index;
DROP INDEX messages_thumbnail_key_index;
DROP INDEX message_row_id_index;
DROP INDEX message_vcard_index;
DROP INDEX message_vcard_jid_index;
DROP INDEX message_view_once_media_state_index;
DROP INDEX missed_call_log_participants_key_index;
DROP INDEX missed_call_logs_key_index;
DROP INDEX mms_thumbnail_metadata_transferred_index;
DROP INDEX parent_group_participants_parent_jid_index;
DROP INDEX message_payment_transaction_id_index;
DROP INDEX attachments_quick_reply_id_index;
DROP INDEX receipt_device_index;
DROP INDEX receipt_device_table_device_index;
DROP INDEX receipt_orphaned_index;
DROP INDEX receipt_user_index;
DROP INDEX receipts_key_index;
DROP INDEX user_device_index;
DROP INDEX user_device_info_index;

 

[xda-eh]

I've just stumbled across some WhatsApp backups from mine (from several devices, dating back to 2013) and I also wondered about the drop in file size on my newest device. Google lead me to this thread and since nobody came up with a plausible reason, I registered an account here to share my findings with you...

Thank you for that information.

 

[diramidon]

Anyone?

I mentioned such strange behaviour of WhatsApp database at least a year ago, may be even a couple of years. It was easy to notice because my DB updates once a week, so it's easy to see the difference. Surprisingly, it's size never exceed 500 Mb: as soon it becomes bigger, the new one appears with smaller size under 500 Mb. Sometimes it cut about 100 Mb and it makes me negatively surprised: internal storage of my phone big enough, so no need to compress files. The only explanation I may image is that earlier messages have been deleted. But I can't prove that, because I can't scroll my main chat to the very beginning: unfortunately, WhatsApp have no such option.

 

[WoKoschekk]

I mentioned such strange behaviour of WhatsApp database at least a year ago, may be even a couple of years. It was easy to notice because my DB updates once a week, so it's easy to see the difference. Surprisingly, it's size never exceed 500 Mb: as soon it becomes bigger, the new one appears with smaller size under 500 Mb. Sometimes it cut about 100 Mb and it makes me negatively surprised: internal storage of my phone big enough, so no need to compress files. The only explanation I may image is that earlier messages have been deleted.

There's no limitation for internal backups and their uploads to Drive. It's not unusual that users have backups in size of 1GB+ due to all their media files.
Backups on Drive will be completely deleted after 5 months of inactivity and that's all.

But I can't prove that, because I can't scroll my main chat to the very beginning: unfortunately, WhatsApp have no such option.

Use WA desktop version.

 

[xda-eh]

Use WA desktop version.

Actually it's the mobile app that retains the full history, whereas the desktop version only keeps the more recent one.

 

[WoKoschekk]

Actually it's the mobile app that retains the full history, whereas the desktop version only keeps the more recent one.

Then open a chat > 3 dots (upper right corner) > More > Export chat. This creates a *.txt file and the first line of the chat = 1 post in chat with its date.

 

[xda-eh]

diramidon you mistakenly inserted your response into your quote of me (which should have just been one word), as if I wrote what you actually wrote. See attached image.