Page MenuHome GnuPG

Document how to switch akonadi database backend to sqlite
Closed, ResolvedPublic

Description

With a slight possibility that Akonadi might one day find its way into the Gpg4win / GnuPG VS-Desktop package it is important that we can use sqlite as a database backend. Since this might also attract more users to KMail in general and thus bring us more testers I would like to start using it.

But since I have ~10 identities configured and various send / receive accounts I do not want to loose that.

Problems I can imagine is that mapping things like sent mail folder / drafts / trash etc. might loose their connection. This could be similarly a problem for Mailfilteragent based filters. Which I personally do not use.

I would like to see the current state how to do this documented somewhere. Even if the documentation is just "no chance" to keep anything, but you have to delete / modify the following configuration files for a clean switch. Maybe under: https://userbase.kde.org/Akonadi (And give that page some love while you are at it.)

Event Timeline

aheinecke created this task.
dvratil moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Dec 1 2023, 9:49 AM

Requests to be able to migrate to SQLite without losing data/configuration has been requested multiple times since we blogged about the improvements in SQLite support.

There are probably three ways how to migrate, ultimately it's up to you to decide which way I should go.

Automated Conversion of the DB from X to SQLite

This basically requires implementing a migration tool that would create a new SQLite DB, copy all data from MySQL/PSQL, reconfigure Akonadi to use SQLite and delete the old DB. This way, database IDs are preserved so that the IDs stored in configuration files continue pointing to the same collections even after migration. This would make the migration hassle-free and accessible to anyone.

There's demand for such a feature in the community, so I was planning to implement it on my own time anyway (probably as akonadictl migrate). If you don't want to invest into me writing the migrator (a totally legitimate decision!), I can go and just update the Akonadi documentation in general and document the migration process later, once the tool is done.

Manual Conversion of the DB from X to SQLite

Write an extensive guide how to perform the migration manually. It's possible, but probably a rather lengthy and error-prone procedure. Will still probably invoke running some Python script to convert the SQL dump from MySQL/PSQL SQL dialect to SQLite, which is impossible to do manually, unless you really hate yourself.

Document How to Wipe the DB and Start From Scratch

Write a documentation how to wipe MySQL/PSQL, configure Akonadi to use SQLite and try to cover all the possible places where the folder mapping must be updated to point to real collections. This is a viable approach, but I'm afraid it will go out-of-date quickly.

It is okay for you to start with option so "akonadictl --migrate" but I would suggest deleting the old DB only as an option. Especially if error handling is not 100% perfect there should be also an easy way to restore your old config.

And with all the config files involved I am not sure if you can easily say "please backup directory and file XY before starting" ?

dvratil moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Feb 5 2024, 10:25 AM

Well, so the documentation is that there is no way to migrate and you have to delete everything and then set it up manually again? In that case it is still missing the deletion of the KMail settings etc.

I tried to use pimdataexporter to cerate a backup. In ~/.config called: rm akonadi -rf akonadi* ../.local/share/akonadi*
created the akonadiserverrc and then started kontact.
That made a bit of bäm. Since it started to create resources and so on right away based on other configuration files I guess.

Then I removed the files again and this time started with "pimdataexporter" to import the old backup. That froze at around 80% so hard that I had to send sigkill to close it. But the resource configuration was imported,.. But I then had to reset all the folder Ids, as you akso mentioned.
I left it syncing for a while and noticed afterwards that my SMTP servers were triplicated. But I could remove them except for one confiuguration value for each of the SMTP Servers I use.

But for Kaddressbook and KOrganizer the CalDav / CardDav resource used were also not restored.

All in all I have to admit that you did not document it this way, so that users should use pimdataexporter. But Yeah pimdataexporter also does not seem to work very robustly.

And yes it would have been cheaper for me to just set up from scratch :)

Daniel wrote a migration tool which was merged in January (https://invent.kde.org/pim/akonadi/-/merge_requests/154), i.e. a few weeks after he wrote the documentation. He foresaw that the documentation "will go out-of-date quickly". ;-)

aheinecke changed the task status from Open to Testing.Jul 29 2024, 9:15 PM
aheinecke claimed this task.
aheinecke added a subscriber: dvratil.

Ah, that was what daniel and me actually talked about. So I was a bit dissapointed when I only found the documentation. :)

I will restore my state from before my manual migration from backups and try again using the tool. So far everything works fine using sqlite by the way. But as mentioned I did not have my calendar resource anymore but that is more of an issue with pimdataexporter I think, since that froze up several times and did not appear to be very reliable.

I tested it and updated the community wiki documentation to refer to it.