Small command-line utility to safely merge multiple WhatsApp backups (msgstore.db) into one.

Overview

whatsapp-database-merger

A small command-line utility that can be used to merge multiple WhatsApp databases (msgstore.db) into one. A few notes:

  • input databases must be already decrypted
  • output database will also be in decrypted form. To use it, it must either be moved to /data/data/com.whatsapp/databases on a rooted phone, or otherwise encrypted again and restored as usual.
  • Java 11 should be installed to build and run the program.

Usage

You can either build the tool yourself or download the binary from the releases page. The tool requires a single argument - a path to a directory on your local machine. For example:

./whatsapp-database-merger /Users/Me/Desktop/WhatsApp

In this example, the program will look for input databases in the /Users/Me/Desktop/WhatsApp/input folder, and will create the merged database in /Users/Me/Desktop/WhatsApp/output.

How it works

The program is inspired by whapa. WhatsApp msgstore.db is just a SQLite database and as such it can be read and written to with common tools.

The program will merge all relevant tables, not just messages and chats. See src/main/kotlin/dev/natario/Table for a list.

When merging them, extra care is taken:

  • primary keys, when possible, are given the appropriate offset in order to not collide with existing entries in the other database(s).
  • When this happens, we also apply the same offset in every other column of the database that were referencing them, ensuring consistency of data.
  • Special logic addresses the case where entries could not be inserted due to unique constraint defined in the database, in which case all references to the skipped entry will be updated to match the existing entry which prevailed.

For example, messages._id will receive an offset to avoid collision, then the program will also modify all other columns pointing to messages, like chat.last_read_message_row_id.

Known issues

The tool can have troubles with binary data, especially in the message_thumbnails table. If the data contains many null bytes (\u0000), the SQL string will include many concatenations (like 'foo'||char(0)||'bar') and there is a hard limit in the driver about how many concatenation operations it can handle for a single statement.

By default, the merger will simply drop thumbnails that it was not able to copy.

It doesn't work for me!

I'm sorry. This happened to work for me in March 2022, but I am unlikely to spend too much time on this utility in the future, especially since the database schema can change very quickly. However, I am happy to review and accept pull requests. The source code should be very easy to understand.

Comments
  • Too soon for an issue

    Too soon for an issue

    Hey, thanks for the work. I have very limited experience with Java, tried building the project but got some error. I know very little to conclude what went wrong. Here's the screenshot.

    image

    Would it be possible for you to release a jar for this?

    opened by YuvrajRaghuvanshiS 18
  • Convert old database to new schema

    Convert old database to new schema

    Sorry if this is a bit out of topic.

    Currently I want to merge two databases, an old one where messages are stored in messages table, and a new db where messages are stored in message table.

    I read the new changes made by MrBarbie that makes merging between databases with the new, mid-2022 schema possible. But mine is on different schemas.

    I tried restoring the old db into the latest whatsapp version, and creating a backup, then restoring again using the newly generated backup file. But the new backup file is still using old schema, and I'm at a loss here.

    So back to the question, is there a way to convert the old db schema into the new one? I know that it is possible to generate a SQL query to do that but the whatsapp db structure is a bit overwhelming for me.

    opened by Azfarrr 1
  • Database is not consistent

    Database is not consistent

    After my whatsapp databases are already using the latest/new schema, tried using the version updated by @MrBarbie.

    But got some database consistency error, some are just as warning and stopped at RuntimeException for column message_vcard_jid.vcard_row_id. Reading from issue #2, @natario1 suggested to add the column into suggestion.

    The thing is I don't know how to compile using gradle and kotlin. Want to try using sencodemaker's merger but it still not updated to account for the latest database schema.

    The log is as below.

    [*] Input databases:
            - input\msgstore.db
            - input\msgstore1.db
    [*] Checking consistency of input\msgstore.db
            Column chat.jid_row_id is consistent! All 698 unique entries (out of 698) exist in the referenced table jid.
            Column chat.display_message_row_id is consistent! All 286 unique entries (out of 698) exist in the referenced table message.
            Column chat.last_message_row_id is consistent! All 286 unique entries (out of 698) exist in the referenced table message.
            Warning: Database input\msgstore.db is not consistent. Column chat.last_read_message_row_id references table message, where we expected to find a list of 235 entries, but only 234 were found.
    Total count (source): 95991, total count (target): 698
    Missing=[100846]
            Warning: Database input\msgstore.db is not consistent. Column chat.last_read_receipt_sent_message_row_id references table message, where we expected to find a list of 286 entries, but only 285 were found.
    Total count (source): 95991, total count (target): 698
    Missing=[100846]
            Column chat.last_important_message_row_id is consistent! All 2 unique entries (out of 698) exist in the referenced table message.
            Column chat.change_number_notified_message_row_id is consistent! All 3 unique entries (out of 698) exist in the referenced table message.
            Column chat.last_read_ephemeral_message_row_id is consistent! All 0 unique entries (out of 698) exist in the referenced table message.
            Column chat.last_message_reaction_row_id is consistent! All 9 unique entries (out of 698) exist in the referenced table message.
            Column chat.last_seen_message_reaction_row_id is consistent! All 9 unique entries (out of 698) exist in the referenced table message.
            Warning: Database input\msgstore.db is not consistent. Column chat.last_read_message_sort_id references table message, where we expected to find a list of 196 entries, but only 195 were found.
    Total count (source): 95991, total count (target): 698
    Missing=[100846]
            Column chat.display_message_sort_id is consistent! All 217 unique entries (out of 698) exist in the referenced table message.
            Column chat.last_message_sort_id is consistent! All 218 unique entries (out of 698) exist in the referenced table message.
            Warning: Database input\msgstore.db is not consistent. Column chat.last_read_receipt_sent_message_sort_id references table message, where we expected to find a list of 187 entries, but only 186 were found.
    Total count (source): 95991, total count (target): 698
    Missing=[100846]
            Warning: Database input\msgstore.db is not consistent. Column message.chat_row_id references table chat, where we expected to find a list of 457 entries, but only 456 were found.
    Total count (source): 698, total count (target): 95991
    Missing=[-1]
            Column message.sender_jid_row_id is consistent! All 702 unique entries (out of 95991) exist in the referenced table jid.
            Column message.sort_id is consistent! All 95990 unique entries (out of 95991) exist in the referenced table message.
            Column message_quoted.message_row_id is consistent! All 10651 unique entries (out of 10651) exist in the referenced table message.
            Column message_quoted.chat_row_id is consistent! All 118 unique entries (out of 10651) exist in the referenced table chat.
            Column message_quoted.parent_message_chat_row_id is consistent! All 152 unique entries (out of 10651) exist in the referenced table chat.
            Column message_quoted.sender_jid_row_id is consistent! All 303 unique entries (out of 10651) exist in the referenced table jid.
            Column message_vcard.message_row_id is consistent! All 67 unique entries (out of 71) exist in the referenced table message.
            Column message_vcard_jid.message_row_id is consistent! All 57 unique entries (out of 86) exist in the referenced table message.
    Exception in thread "main" java.lang.IllegalStateException: Database input\msgstore.db is not consistent. Column message_vcard_jid.vcard_row_id references table message_vcard, where we expected to find a list of 61 entries, but only 60 were found.
    Total count (source): 71, total count (target): 86
    Missing=[-1]
            at dev.natario.ConsistencyKt.verifyConsistency(Consistency.kt:21)
            at dev.natario.ConsistencyKt.verifyConsistency$default(Consistency.kt:3)
            at dev.natario.MainKt.main(Main.kt:50)
    
    opened by Azfarrr 29
  • Tool not working since mid july due to changes in database schema

    Tool not working since mid july due to changes in database schema

    Seems like in version 2.22.7.74 the schema changed a lot, most likely to this feature of being able to switch between iOS and Android more seamlessly.

    This cool author blogpost has some info to get started with the changes, the TLDR is that the messages (plural) tables have been dropped and now they are called message and message_* and there are a few more.

    https://thebinaryhick.blog/2022/06/09/new-msgstore-who-dis-a-look-at-an-updated-whatsapp-on-android/

    Do you think you could find some time to update the tool? Ideally it would detect the version and would be able to merge both and migrate to the new schema for the output database.

    I'm very interested in this so if you say that you won't be able to look at it anytime soon I might try to give it a chance but I'm new to Kotlin, only being able to run the project in IntelliJ took me an hour LOL.

    Thanks in any case for this contribution!

    opened by AlbertCalvet 22
  • How to use?

    How to use?

    Can you tell me a detail instruction on how to use the tool? Especially for windows user with little knowledge.

    For example, how to arrange the msgstore.db; eg: I want to combine mgstore (6March) and mgstore (7March), how to do it? How about the naming scheme? naming it mgstore1.db for 7 March, and mgstore2.db for 6 March?

    What command to make in cmd (windows) to combine it? How to define output file?

    Hope you can answer this ASAP as I really on my end needs here :D Sorry for any bad English used.

    opened by DarkXonline 10
Owner
Mattia Iavarone
I am available for support, development, training and consulting for your Android project. Send an e-mail!
Mattia Iavarone
A command line utility to help you investigate the sensitive data associated with Macie findings.

Macie Finding Data Reveal This project contains a command line utility to help you investigate the sensitive data associated with Macie findings.

AWS Samples 8 Nov 16, 2022
A command line tool for NFT stuff

MartaKli is a command line tool to help you generate your NFT pfp project. Current features: Generate batch of images using different layers and corre

Martabak Cult 2 Oct 4, 2022
KDoctor - A command-line tool that helps to set up the environment for Kotlin Multiplatform Mobile app development

KDoctor is a command-line tool that helps to set up the environment for Kotlin Multiplatform Mobile app development.

Kotlin 331 Dec 29, 2022
A logger with a small, extensible API which provides utility on top of Android's normal Log class.

This is a logger with a small, extensible API which provides utility on top of Android's normal Log class. I copy this class into all the little apps

Jake Wharton 9.8k Dec 30, 2022
A small utility to record Android device screen to a GIF

RoboGif A small utility to record Android device screen to an optimized GIF so you can paste it to GitHub or a similar service. Requirements Python 2.

Jernej Virag 526 Dec 9, 2022
A collection of small utility functions to make it easier to deal with some otherwise nullable APIs on Android.

requireKTX is a collection of small utility functions to make it easier to deal with some otherwise nullable APIs on Android, using the same idea as requireContext, requireArguments, and other similar Android SDK methods.

Márton Braun 82 Oct 1, 2022
Small utility used to add Homewizard Energy Socket tiles to Android Quick Settings

TileWizard [Alpha! Unstable!] Settings Result Functionality: Add up to 5 Wi-Fi Energy Sockets to Android Quick Settings. What do you need: Android dev

Niels Masdorp 2 Oct 19, 2021
Recover deleted messages for whatsapp and Facebook lets you recover messages that sender deleted

Recover deleted messages for whatsapp and Facebook lets you recover messages that sender deleted. It also recover deleted media Images, Audio, Video etc

S M Khurram Khan 6 Jul 17, 2022
Handy library to send & receive command with payload between subscribers for Android.

Commander Handy library to send & receive command with payload between subscribers for Android. Features Subscription based No dependency on Framework

Romman Sabbir 3 Oct 19, 2021
Command framework built around Kord, built to be robust and scalable, following Kord's convention and design patterns.

Command framework built around Kord, built to be robust and scalable, following Kord's convention and design patterns.

ZeroTwo Bot 4 Jun 15, 2022
Utility for detecting and notifying when your Android app goes background / becomes foreground

Foredroid Utility for detecting and notifying when your Android app goes background / becomes foreground. API-level 14+. Usage: Initialise Foreground

Steve Liles 151 Nov 29, 2022
Utility tool to make you a computer ninja.

Cmd Window Never spend 6 minutes doing something by hand when you can spend 6 hours failing to automate it - Zhuowej Zhang What is this about? This to

Marcin Radoszewski 3 Feb 1, 2022
FractalUtils - A collection of utility functions and classes, with an emphasis on game related utilities

A collection of utility functions and classes written in Kotlin. There is some emphasis on utilities useful for games (Geometry, Random, Time, Updating, etc).

null 2 Nov 11, 2022
Utility library that utilizes KSP to generate Room type converter classes.

Roomie Roomie is an annotation processing library that utilizes KSP to geaRoomie is an annotation processing library that utilizes KSP to generate TypeConverter classes for Room. TypeConverter classes most often involve same boiler-plate code and Roomie makes it really easy to quickly create them with a single annotation.nerate TypeConverter classes for Room. TypeConverter classes most often invol

Chukwuka Eze 12 Aug 26, 2022
CHAOS - Like a utility knife for discord

CHAOS - Like a utility knife for discord. Currently under development. If you feel inclined, please support me by contributing to this project. Or alt

caffeine.moe 20 Nov 13, 2022
Obsi-bot: the next generation discord utility bot 🔥

obsi-bot obsi-bot is the next generation discord utility bot. It is developed in Kotlin using kordex and kord Help me translating Feel free to help me

mooziii 2 Nov 7, 2022
A library to quickly and easily enable multiple monitoring & support platforms for your mobile apps

You have a small team. Setting up crash reporting tools, event tracking tools, and log management services is not what you want to spend your hours do

Percolate 65 Aug 8, 2022
Convert OkHttp requests into curl logs.

Ok2Curl Convert OkHttp requests into curl logs. Usage Add library to project dependencies. Library is hosted on jcenter. repositories { jcenter()

Michal Moczulski 373 Jan 1, 2023
A simple Android utils library to write any type of data into cache files and read them later.

CacheUtilsLibrary This is a simple Android utils library to write any type of data into cache files and then read them later, using Gson to serialize

Wesley Lin 134 Nov 25, 2022