On Sun, 16 Oct, 2005 at 07:47:28 -0400, Ken Schneider wrote:

On Sun, 2005-10-16 at 12:01 +0200, Jon Clausen wrote:

...

It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?

Look into using rsync to transfer the files/folder structure from one machine to the other.

Right. I've already tried copying some folders "behind Cyrus' back". I must've not been doing the right 'reconstruction' afterwards, because Cyrus didn't recognize the added folders. So if I'm to go down the road of copying the contents "outside" of IMAP, what would be the procedure to get Cyrus to accept everything afterwards? TIA /Jon -- YMMV

Jon Nelson wrote:

...

So if I'm to go down the road of copying the contents "outside" of IMAP, what would be the procedure to get Cyrus to accept everything afterwards?

Well, I recommend John Goerzen's offlineimap which can copy from imap-to-imap as of 4.0. I've not used that capability myself but it shouldn't be a problem. Alternatively, have you considered asking the Cyrus folks what to do? I'd bet they've been asked before and have an answer. ;-)

The question is related to "How do I recover Cyrus from scratch with my backup only". Maybe it's a good idea to gather enough knowledge so you can use it in an emergency case also. (^-^) It's also a very good opportunity to test your backup strategy. What is always recommended on cyrus sites is to - either stop cyrus for backup or do a snapshot with lvm - copy all data folders - copy all databases - if necessary (for migration/upgrade definitely!) export the databases in text format to be able to reimport them in case the db version differs. Last time this happened was from bdb 4.1 -> bdb 4.2 (imho Suse 9.2 -> 9.3), they were not compatible... So, how do you export the databases and what databases do you need to export? The most essential is the Database with the folder structure, the mailbox-db: su - cyrus -c "ctl_mboxlist -d" > $CONFIGDIRECTORY/mailboxlist.txt If you reimport this structure on your new server you should at least see all folders and emails. What you are missing at this point is the read/unread status of the mails. su - cyrus -c backupseen.sh backupseen.sh: ------------------------ for file in `find /var/lib/imap/user -name \*.seen`; do /usr/lib/cyrus/bin/cvt_cyrusdb ${file} skiplist ${file}.txt flat done ------------------------ On your new Server you need to import these databases again: su - cyrus -c restoreseen.sh restoreseen.sh: ------------------------ for file in `find /var/lib/imap/user -name \*.seen.txt`; do /usr/lib/cyrus/bin/cvt_cyrusdb $file flat ${file/seen.txt/seen} skiplist done ------------------------ The user cyrus is needed to perform these tasks, otherwise the permissions of the directories/files will probably be messed up. I haven't yet done any restore from scratch, but it should work this way. So I recommend to check each step and test the result. See also: http://acs-wiki.andrew.cmu.edu/twiki/bin/view/Cyrus/Backup Sandy -- List replies only please! Please address PMs to: news-reply (@) japantest (.) homelinux (.) com

Jon Clausen wrote:

...

It's also a very good opportunity to test your backup strategy.

My bakkah-stratahwhatnow?

Just something to keep the users/powers-that-be from lynching you. (^-^)

Thanks a bunch. That looks very much like what I needed to know.

I still would like for cyradm to be able to pull contents from another server, but either I'm missing something or else it's just not been implemented yet.

Unfortunately this is something most server applications tend to neglect: How to backup/restore your configuration/data.

In any event Sandy's got a good point in 'learn how to recover'.

Only consideration left is this;

Circumstances dictated that I offlined the old server and onlined the new simultaneously. Also I needed immediate access to some of the folders, which I copied over using Mutt.

I am afraid that was a big mistake. I prefer to test and check the migration with a test system and a small database, then use the real data to check, if there is a hidden bug and see how much time is needed. I then double the time to have space for the inevitable bugs that occur in the real migration. Then I set an offline window and tell the users to store mails needed in the mean-time in off-line folders. That way they don't bug me when I need my concentration and nerves for the actual migration. First rule is always: Don't let yourself be talked into things that you simply can not guarantee. Rather find ways around to solve the immediate problem and then proceed according to plan.

As a consequence I now have a number of folders which are severely out of sync (between "old" and "new"). Question;

If I do a complete backup of everything on old-server and "restore" that on new-server, will it

1) only overwrite "old" content present on new-server leaving "new" content intact?

Tha depends on how you "restore" the data. If you overwrite the files on the new system you will definitely destroy the new databases, so all new mailboxes, folders and permissions on these are lost with the restore, even if the folders and mails are physically still present. One hare-brained way to get them back might be to export the mailbox.db on the old system and the new system, merge these files and then use unique to delete double entries. This might work, but I have never tried something that reckless. (^-^) PLEASE backup your mailbox.db incl. the text export to recover if that doesn't work!! All the databases are practically useless, at least I don't know any tool to merge the seen dbs, the deliver.db (not absolutely critical) If both servers are still functional you might be better off to tell all users to login to both servers at once and then copy the content they want to keep. Sandy PS: if you use sieve-scripts, please don't forget to copy those as well. (^-^) -- List replies only please! Please address PMs to: news-reply (@) japantest (.) homelinux (.) com

Jon Clausen wrote:

Right. I guess I should have pointed out earlier, that I am in fact the only user on either of the servers.

Ah, in that case there is at least no one breathing fire down your neck to get the problem solved. (^-^)

My original plan was to copy the contents using IMAP, and if all else fails that is still a viable solution. Even if it means I have to depart from the 'vanila' state of the new server.

Why don't you do that? That is the usual way to migrate a mailbox to another server, when there is no tool available. I know I once migrated a POP3 Mailbox to Imap that way. Just add another server account in your mail client and copy the contents of the old server to the new. It will take some hours but that shouldn't be a problem.

INBOX; Toplevel, 'synced' manually INBOX.1st-sub; Mailinglists, workrelated etc. some in sync some not INBOX.2nd-sub; Timeslice archives of above (quarterly, monthly etc)

What I'm looking at restoring, at this point, is mostly just the 2nd sublevel. So I think that the basic approach outlined previously would still apply, with modifications to only work on the 2nd level?

Webcyradm should have the option to reconstruct a mailbox. Have you tried that?

...

One hare-brained way to get them back might be to export the mailbox.db on the old system and the new system, merge these files and then use unique to delete double entries. This might work, but I have never tried something that reckless. (^-^)

I like that :)

By the way, I think Suse already performs a text export of the mailbox database every night. Have a look at /var/lib/imap/backup. Sandy -- List replies only please! Please address PMs to: news-reply (@) japantest (.) homelinux (.) com

On Wed, 19 Oct, 2005 at 07:07:57 +0200, Jon Clausen wrote:

On Tue, 18 Oct, 2005 at 12:57:58 +0200, Sandy Drobic wrote:

...

Just add another server account in your mail client and copy the contents of the old server to the new. It will take some hours but that shouldn't be a problem.

Quite so. It's just that I tend to think that one of the reasons we have computers in the first place, is to *avoid* time-consuming, simple and repetitive tasks... :P

<snip>

I think I'll go back a couple of steps, and look into imapsync suggested earlier in the thread.

Which is what I did: Imapsync needs the perl module Mail::IMAPClient which I installed from CPAN. The --include option makes it do what I want (copy 2nd sublevel folders). I just had to realize how to construct a regex that matches the folders I wanted. It's running now, and from what I've seen so far it's still going to take hours to finish. The difference (from 'manually' copying, using a mail-client) is that it runs without human intervention, leaving me free to attend to other activities... :) Conclusion; The backup/restore approach is probably magnitudes faster (since it avoids the IMAP overhead, and you're free to compress/decompress contents for the actual transfer). It does however require you to think ahead and, not least, *some* familiarity WRT the inner workings of your imapd of choice. Imapsync otoh, takes little effort to set up, seems reliable, is fairly straightforward to operate, but takes time to run. For me, imapsync provided the path of least resistance. Of course it's no substitute for a backup strategy, but it satisfies my immediate need. Thanks for everyone's input. /Jon -- YMMV

On Mon, 17 Oct, 2005 at 09:38:21 +1000, Mark Keir wrote:

On 10/16/05, Jon Clausen wrote:

...

It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?

Use the perl module imapsync. Have used this in the last couple of months on the same SuSE combination. Watch out for mailboxes with LOTS of files or very large data volumes (eg >1Gb) as the wordlist return is greater than the maxlength allowed. This tool is very good as it preserves the SEEN state for Cyrus mailboxes.

I already googled around and found imapsync. From what I found, it depends on Mail:IMAPClient, which is not included with the distro. Now, I guess I could just install that module from CPAN. In the end I probably will. But I'm just wondering if there really isn't a tool that's *included*? Up till now I've made *some* effort to stay 'clean' (nothing installed from anywhere but the 9.3 installation sources, and everything configured through the SUSE-provided means) on this machine. hmmm /Jon -- YMMV