Have something to say?

Ready to be published? LXer is read by around 350,000 individuals each month, and is an excellent place for you to publish your ideas, thoughts, reviews, complaints, etc. Do you have something to say to the Linux community?

A KMail Breakthrough.

Posted by Ridcully on May 1, 2016 4:57 AMLXer Linux News; By Dr Tony Young

LXer Feature Update: 27-Apr-2016

This tells the story of how I finally managed a successful transfer of email data from KMail version 1.13.6 to version 4.11.5. It is a non-technical essay exploring the obstacles I encountered, my options, and the methods I used to achieve my aim. It was written partly to give the information, but also with the hope that readers will both enjoy and be amused by the story of the "battle of KMail" that was ultimately won against "incredible odds". Links to the earlier articles discussing problems with KMail 4x are given at the end.

Introduction

This article has come about purely because I was downright stubborn - and that is putting it mildly. My current laptop is an old HP Compaq 6710b with a 2-core processor, running a 32 bit Linux system comprising openSUSE 11.4, KDE4.6 and KMail version 1.13.6. Yes, it's antique, but if it does the job that I need, then fine.....if it ain't broke, don't fix it. Unfortunately, while not broken, this dear old laptop is now beginning to really show its age....silly things like keys that may not always work at the moment you need them, or speed, or versions of software that are simply too old to do the job really well any more; just minor irritations, but they all add up.

Last year, I undertook a project with a software development group in Brisbane, and in order to do the work necessary, a new 64 bit laptop was required, so ultimately I found myself the proud possessor of an HP Elitebook 8540P (64bit, 4 core) which was running Windows 7 when I got it. Allow me to confess immediately, that the laptop was NOT brand new......it was a second hand ex-government laptop but in "as-new" condition. And the reason I insisted that it come with Windows 7 was that I knew, for my own ease of use, that the computer would be running the traditional BIOS software chip and therefore I would not have to fight with UEFI. I also made sure that I was able to replace the hard disk drive easily, and that meant that I could remove the Win7 drive and replace it with a blank drive ready to receive Linux. Before someone steps in and says: "Dual Boot", I have never operated that way and don't want to. My preference is to swap the hdd's, and because I know exactly what to do by swapping drives, I prefer to continue that way.

With a blank hdd in place, I loaded the new laptop up with openSUSE 13.1 and proceeded to "play". I chose 13.1 because it is the version that will be supported by the Evergreen project and so 13.1 will retain support for about another two years after SUSE itself stops support. I tried a number of things to get the KMail email files from my old laptop into KMail in this new 64bit machine and in the process of which, altered some settings for KDE. Eventually I found real problems (which I won't discuss here), decided to dump the lot and erased the drive. I then decided that perhaps version 13.2 would be better. Within an hour after installation of 13.2, I regretted that move so much that I once more erased the drive and re-installed 13.1. I also promised myself that under no circumstances would I tamper with any settings involving Akonadi, and I would alter only the minimal ones required to get KMail on line. From then on, I would just simply see what happens when various techniques are tried.

Finally in this introduction, let me say this. I strongly prefer KDE as my workplace and even more strongly prefer KMail as my email client software. I am at home with KDE and I use both the classic menu and the folder view. Once you have those in place, KDE4 closely resembles dear old KDE3.5, but KDE 4 is a far better piece of software apart from its enforced dependence on Akonadi for the "Personal Management Suite and KMail" set of programs - well, that's my opinion. Also, I am now a dedicated Dolphin user as regards file management. I very much like its settings abilities and its split screen mode is wonderful. I'd hate to be without it.

However, in my humble opinion, while KMail is an excellent email client with a superb display, it has been made overcomplex and it can be an absolute "stinker" to work with while you are setting it up if you have never used it before. If you know where to go, you can get the settings for classic style, flat date layout, folder view down the side, etc., but it is exceptionally easy to end up with an extremely messed up display and a "difficult to use" program if you put in the wrong workplace view settings. As for its Akonadi dependence (the Wallet problem appears to have been fixed), words almost fail me as regards this disgusting set of affairs. I retain precisely the same very adverse perceptions that I fully documented in a series on KMail published on LXer a year or so ago. Nevertheless my preference for KMail was such that I soldiered on stubbornly and tried to get this newer version to do what I wanted - and that was not necessarily what the KDE4 and the KMail teams had unilaterally decided all the users wanted.
You have to love that arrogance - remove the user's choice fellers, that's the way to go in Linux - NOT !.

Getting the Mail Files Across from KMail 1.13.6 to KMail 4.11.5

The problem seemed insurmountable. KMail 4x can do a simple import of a previously prepared KMail archive file which will contain all your emails and settings etc........BUT.......the much, much older and earlier KMail 1.13.6 does NOT have the ability to archive its files, and therefore you cannot export an archive to be taken up by KMail 4x. So, could you do what used to be so simple with all versions of KMail up to KMail 1.13.6 ?

All you did was make a complete copy of the kmail directory in ..../.kde4/share/apps/kmail/ and drop that kmail copy into the same location in the new installation........KMail immediately recognised all files, folders etc. and away it went. Simple huh ? But you cannot, cannot, cannot do that now.

I can tell you that thanks to the fact that KMail is now integrated with Akonadi, an SQL based server, the email files are apparently saved as part of the Akonadi resources in the location:
....../.local/share/akonadi_maildir_resource_0.
But it gets better. If you open that folder, you will find the folders you have just set up on KMail 4x and inside each are the usual "cur", "new" and "tmp" folders. However, my inspection showed that there seems to be a random logic in whether or not the files shown on the KMail display are in any of those folders, or none. Further, that mail directory location is set up as an Akonadi resource, and although it can be handled by the file manager, it does not correspond to the older KMail Unix file system resource. My experience so far strongly suggests that unless an email has passed through the KMail software itself, Akonadi will not know it is there and in turn KMail depends on Akonadi for its recall of email data. I decided that sleeping dogs were best left lying and that discretion was the better part of valour - there had to be an alternative.

(Note: Sadly, what I have written in the paragraph above suggests to me that unlike KMail 1.13.6, where you could so easily use your file manager to make a flawless copy of all your emails perfectly arranged in their respective folders, KMail 4x prohibits this situation. I hope I am wrong, but so far nothing has changed my mind. I suggest some options later.)

Evolution Gives the Answer

It shows how stubbornly frustrated I was getting with the situation in that I also tried two other email client programs, Thunderbird and Evolution. Neither of them appealed to me as much as KMail, although I suppose with time and knowledge and familiarity (especially the last) I am sure I could have grown to enjoy both. In my very early years with Linux, Evolution was my main email software, but that is going way back to around 2003. However, it was Evolution that gave me a key to the problem and it is sad that I cannot acknowledge the writer of this short article, because the name hasn't been included. I had searched on methods of getting KMail files into Evolution, and here is the result that finally "opened the door":

As you will see, if you open the article, it is a tutorial which explains the series of short steps that you need to take in order to export the file contents of email folders in KMail into mailbox files (.mbox) that can be imported directly into Evolution. I am not going to repeat the tutorial steps here; they are very clearly written and extremely easy to follow. So, could my old copy of KMail actually do this export ? The answer was very rapidly, "YES", and the process followed the exact steps described in the tutorial written by "eyemeansit". I created a folder called "tempmbox" and eventually I ended up with 23 ".mbox" files of varying sizes which corresponded perfectly to the contents of the 23 folders in my old KMail 1.13.6 software. Naturally, each mbox file was named so that I knew the name of the new folder that would need to be created in KMail 4.11.5, eg. "Crossover.mbox". And of course, there were mbox files corresponding to "inbox.mbox" and "sent-mail.mbox". Okay, that now had things to the point where every email currently held in my old version of KMail had been exported to a series of mbox files that would (or should) be instantly recognisable by both Evolution and hopefully, much later versions of KMail.

Importing into KMail 4.11.5

The next part of the process is tedious, but reasonably simple. You should create the additional folders you need by selecting "Local Folders" in the side panel of KMail 4.11.5, then right click on it and select "Add Folder". It helps if you also have the temporary folder with all the mbox files open in Dolphin (or your preferred file manager) and you can go through that file list one by one while you are adding the names of the KMail folders so that each relevant mbox file will eventually be sent to its same folder name in KMail. Once that is done, you will have your new version of KMail with a set of ready and empty mail folders. I mention here that when importing the .mbox files into KMail, you can send an mbox file into any KMail folder you wish to choose, but in this case, the aim is to duplicate the structure of the original KMail layout.

Now select "File" on the menu bar, and from the drop down menu, select "Import Messages". This will open a subwindow like the one below which was taken from my present installation of KMail 4.11.5. Next, you need to open the selection panel at the top and make sure that the choice of "Import mbox Files (UNIX, Evolution)" is selected from the drop down list. It's right near the top.

{Special Note: it was while editing and checking this article, that for the first time I looked at ALL the options available for importing into KMail 4.11.5, and about halfway down the long list, came across this little gem:
"Import KMaildirs and Folder Structure". It may seem odd, but I had never noticed it before because I was searching for an Evolution solution and that option is found virtually at the top of the drop down list - therefore I didn't look further. I have no idea if this option will work correctly on the older KMail structures but at least I do know for certain that the mbox option does. Also, I can only wonder why this particular option specifically for KMail is not at the very head of the list. However, I digress.....back to the battle.}

Next, from the bottom selection panel, open the selection dialogue and select the particular folder in KMail you wish to import email data into. It is at this point that ensuring the names of the mbox files correspond to the KMail folder names makes your task very, very easy. Once you have selected the particular KMail folder into which you want to import data, select "Next". A new subwindow will open which will allow you to select the relevant mbox file.

From here on, just follow the instructions and you will usually be astonished at how fast the process works. You will end up with all the relevant emails from the mbox file imported into their correct KMail folder - sort of; see the next paragraph for the last little "tweak". Doing this procedure again and again is slow and tedious compared to importing a single archive file, but it works. It is quite possible that all the inbox emails will be marked as "unread". If this is the case, then an even more tedious job is waiting for you, because as far as I know, you have to select the files one by one so they are marked as read.

There is also another minor and tedious task to be undertaken. Each time you import an mbox file into the KMail folder, it does NOT go immediately into the folder itself, but instead the emails are copied into a subfolder of the folder you are working with. What you then do is open that subfolder, select all the emails in it and "drag and drop move" them into the designated folder and delete the subfolder. Okay, messy, but once again, it does work.

Last But Not Least.

As I indicated above, there seems to be no way you can drop a Unix based copy of the entire set of KMail files into the present version of KMail using the standard file managers. I'd be enormously happy to be corrected if I am wrong in this matter. As I presently see it, there are two methods available to the KMail user if some sort of backup system is to be applied - there may be others. The first is to use the "mbox" export method and keep that set of files as an "unreadable but savable" copy. The second is to use KMail's archive process and make a standard archive.

I'd also stress that this article contains my perceptions of the situation as it unfolded. There are probably other ways of doing what I wanted, but I found a way to effectively solve a problem and tedious or not, I'm very content with the results. It is very possible that some of my assumptions are incorrect and readers should recall that I am just a user who has fought his way through a problem and seen it from a particular perspective. I'm very happy to be told of any errors I've made in this text, just as long as I also get the real answer.

At the moment, KMail 4.11.5 seems to be running perfectly. I have altered no settings which concern the Akonadi system. I cordially detest the fact that KMail is wholly dependent on Akonadi, but I don't dare touch any of its default settings for fear of rendering KMail inoperable. I have no objections to Akonadi itself, I simply believe that users should have the choice/option of operating KMail completely independently from Akonadi and preferably with Akonadi wholly disabled.

I have gone through some "convoluted process" which seems to have stored the server password in KMail rather than the wallet, but if you ask me how I did it, I simply don't quite know. Just luck that I pushed the right buttons, I think. I do recall that when the Wallet kept on asking me if I wanted to Keep the password in the system, I selected Keep each time, and I think the Wallet got sick of asking. What I do know, is that when I just looked at my account, the password is clearly shown as having been registered in the KMail software itself. If that is the case, then I congratulate the KMail team for restoring that user option - I repeat, my only problem is that I don't know precisely how I did it. I have always argued that a user should have the option of either keeping the password either in the system or in the Wallet, and the option should be very clearly displayed.

Assuming that the option to use or not use the Wallet has now been restored, KMail's absolute dependence on Akonadi to store its files remains the sole major problem as far as I personally am concerned. But for the rest, KMail remains a superb piece of software, settings complexity or not - I can live with that. And one last thing is that the 4-core 64bit laptop runs both KDE 4 and KMail very fast indeed and at least some of the problems noted in the earlier articles are now un-noticeable.

So there we are.......I think I have managed to "bash" KMail 4.11.5 into a form that I can use comfortably. It's been a very steep learning curve and I wasn't sure I was going to win. To be honest, I think the KMail team would be perfectly justified in saying that I should have expected some sort of difficulty in trying to migrate emails from such an old/antique version into one of the latest versions. And I cannot help but agree - but in mitigation KMail team, I believe I used an "antique method" to get my files across (sorry, but that strikes me as funny). I have noted that the Accounts section in the setting up window has a "Local Mail" line......I'm not sure what it does, but it's of no use to me, and so I have left it strictly alone. Like other parts of KMail now, I am reluctant to try to modify or delete areas about which I lack information and whose change might disturb the smooth running of the software in some way.......What an absolute pain. But, thankyou once again KMail team for the Password/Wallet option alteration and the difficulties that were laid in my path (and I strongly emphasise that I am not being sarcastic at this point). Without those problems, I would not have explored and learned about the mbox concept - and best of all, it worked.

Links to the Earlier Articles on openSUSE 13.1, KDE and KMail

Here is a link to my earlier article discussing problems with KMail in these latest versions. I'd add here, that later events have "killed" the Patience option described here. The latest versions of KDE4 have been modified to the extent that the old version of Patience will no longer run. Sad. The new swinging and dancing version of KDE 4 KPat does NOT have the crispness and quality of the card decks available in the old KDE3.5 KPat. Well, that's my opinion.