Copying IMAP, server -> server
I'm about to retire my old mail server. I'd like to copy the complete IMAP-structure from old -> new. Old is; SuSE 8.2, Cyrus 2.1.12 - not 'live' New is; SUSE 9.3, Cyrus 2.2.12 - 'live' The whole thing is largeish (couple hundred megabytes) and complexish (several layers of folders/subfolders). Large enough and, particularly, complex enough that I'm going to spend days creating folders and copying mail, if I'm to do it 'manually' using an imap client running on my desktop machine. Both of the servers are physically here. If at all possible I'd like to avoid installing/changing anything on the old server. So basically what I'm after is; Something running on New-server, to 'pull' stuff off of Old-server. But mostly I'm just asking for pointers; It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro? TIA /Jon Clausen -- YMMV
On Sun, 2005-10-16 at 12:01 +0200, Jon Clausen wrote:
I'm about to retire my old mail server.
I'd like to copy the complete IMAP-structure from old -> new.
Old is; SuSE 8.2, Cyrus 2.1.12 - not 'live' New is; SUSE 9.3, Cyrus 2.2.12 - 'live'
The whole thing is largeish (couple hundred megabytes) and complexish (several layers of folders/subfolders).
Large enough and, particularly, complex enough that I'm going to spend days creating folders and copying mail, if I'm to do it 'manually' using an imap client running on my desktop machine.
Both of the servers are physically here.
If at all possible I'd like to avoid installing/changing anything on the old server.
So basically what I'm after is;
Something running on New-server, to 'pull' stuff off of Old-server.
But mostly I'm just asking for pointers;
It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?
Look into using rsync to transfer the files/folder structure from one machine to the other. -- Ken Schneider UNIX since 1989, linux since 1994, SuSE since 1998
On Sun, 16 Oct, 2005 at 07:47:28 -0400, Ken Schneider wrote:
On Sun, 2005-10-16 at 12:01 +0200, Jon Clausen wrote:
It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?
Look into using rsync to transfer the files/folder structure from one machine to the other.
Right. I've already tried copying some folders "behind Cyrus' back". I must've not been doing the right 'reconstruction' afterwards, because Cyrus didn't recognize the added folders. So if I'm to go down the road of copying the contents "outside" of IMAP, what would be the procedure to get Cyrus to accept everything afterwards? TIA /Jon -- YMMV
On Mon, 17 Oct 2005, Jon Clausen wrote:
On Sun, 16 Oct, 2005 at 07:47:28 -0400, Ken Schneider wrote:
On Sun, 2005-10-16 at 12:01 +0200, Jon Clausen wrote:
It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?
Look into using rsync to transfer the files/folder structure from one machine to the other.
Right.
I've already tried copying some folders "behind Cyrus' back". I must've not been doing the right 'reconstruction' afterwards, because Cyrus didn't recognize the added folders.
So if I'm to go down the road of copying the contents "outside" of IMAP, what would be the procedure to get Cyrus to accept everything afterwards?
Well, I recommend John Goerzen's offlineimap which can copy from
imap-to-imap as of 4.0. I've not used that capability myself but it
shouldn't be a problem. Alternatively, have you considered asking the
Cyrus folks what to do? I'd bet they've been asked before and have an
answer. ;-)
--
Carpe diem - Seize the day.
Carp in denim - There's a fish in my pants!
Jon Nelson
Jon Nelson wrote:
So if I'm to go down the road of copying the contents "outside" of IMAP, what would be the procedure to get Cyrus to accept everything afterwards?
Well, I recommend John Goerzen's offlineimap which can copy from imap-to-imap as of 4.0. I've not used that capability myself but it shouldn't be a problem. Alternatively, have you considered asking the Cyrus folks what to do? I'd bet they've been asked before and have an answer. ;-)
The question is related to "How do I recover Cyrus from scratch with my backup only". Maybe it's a good idea to gather enough knowledge so you can use it in an emergency case also. (^-^) It's also a very good opportunity to test your backup strategy. What is always recommended on cyrus sites is to - either stop cyrus for backup or do a snapshot with lvm - copy all data folders - copy all databases - if necessary (for migration/upgrade definitely!) export the databases in text format to be able to reimport them in case the db version differs. Last time this happened was from bdb 4.1 -> bdb 4.2 (imho Suse 9.2 -> 9.3), they were not compatible... So, how do you export the databases and what databases do you need to export? The most essential is the Database with the folder structure, the mailbox-db: su - cyrus -c "ctl_mboxlist -d" > $CONFIGDIRECTORY/mailboxlist.txt If you reimport this structure on your new server you should at least see all folders and emails. What you are missing at this point is the read/unread status of the mails. su - cyrus -c backupseen.sh backupseen.sh: ------------------------ for file in `find /var/lib/imap/user -name \*.seen`; do /usr/lib/cyrus/bin/cvt_cyrusdb ${file} skiplist ${file}.txt flat done ------------------------ On your new Server you need to import these databases again: su - cyrus -c restoreseen.sh restoreseen.sh: ------------------------ for file in `find /var/lib/imap/user -name \*.seen.txt`; do /usr/lib/cyrus/bin/cvt_cyrusdb $file flat ${file/seen.txt/seen} skiplist done ------------------------ The user cyrus is needed to perform these tasks, otherwise the permissions of the directories/files will probably be messed up. I haven't yet done any restore from scratch, but it should work this way. So I recommend to check each step and test the result. See also: http://acs-wiki.andrew.cmu.edu/twiki/bin/view/Cyrus/Backup Sandy -- List replies only please! Please address PMs to: news-reply (@) japantest (.) homelinux (.) com
On Mon, 17 Oct, 2005 at 17:20:56 +0200, Sandy Drobic wrote:
Jon Nelson wrote:
So if I'm to go down the road of copying the contents "outside" of IMAP, what would be the procedure to get Cyrus to accept everything afterwards?
Well, I recommend John Goerzen's offlineimap which can copy from imap-to-imap as of 4.0. I've not used that capability myself but it shouldn't be a problem. Alternatively, have you considered asking the Cyrus folks what to do? I'd bet they've been asked before and have an answer. ;-)
I would suppose so... I never actually occured to me to look up any FAQ... I'm probably just spoiled by Google. Usually I just type in some question and out comes the answer; "Two kilometers south, go left at the 7/11, add two deciliters of water and leave simmering for ½ hour" or somesuch... :P
The question is related to "How do I recover Cyrus from scratch with my backup only". Maybe it's a good idea to gather enough knowledge so you can use it in an emergency case also. (^-^)
Quite.
It's also a very good opportunity to test your backup strategy.
My bakkah-stratahwhatnow?
What is always recommended on cyrus sites is to
<snip very step-by-step description> Thanks a bunch. That looks very much like what I needed to know. I still would like for cyradm to be able to pull contents from another server, but either I'm missing something or else it's just not been implemented yet. In any event Sandy's got a good point in 'learn how to recover'. Only consideration left is this; Circumstances dictated that I offlined the old server and onlined the new simultaneously. Also I needed immediate access to some of the folders, which I copied over using Mutt. As a consequence I now have a number of folders which are severely out of sync (between "old" and "new"). Question; If I do a complete backup of everything on old-server and "restore" that on new-server, will it 1) only overwrite "old" content present on new-server leaving "new" content intact? or 2) wipe out "new" content in the process? TIA /Jon -- YMMV
Jon Clausen wrote:
It's also a very good opportunity to test your backup strategy.
My bakkah-stratahwhatnow?
Just something to keep the users/powers-that-be from lynching you. (^-^)
Thanks a bunch. That looks very much like what I needed to know.
I still would like for cyradm to be able to pull contents from another server, but either I'm missing something or else it's just not been implemented yet.
Unfortunately this is something most server applications tend to neglect: How to backup/restore your configuration/data.
In any event Sandy's got a good point in 'learn how to recover'.
Only consideration left is this;
Circumstances dictated that I offlined the old server and onlined the new simultaneously. Also I needed immediate access to some of the folders, which I copied over using Mutt.
I am afraid that was a big mistake. I prefer to test and check the migration with a test system and a small database, then use the real data to check, if there is a hidden bug and see how much time is needed. I then double the time to have space for the inevitable bugs that occur in the real migration. Then I set an offline window and tell the users to store mails needed in the mean-time in off-line folders. That way they don't bug me when I need my concentration and nerves for the actual migration. First rule is always: Don't let yourself be talked into things that you simply can not guarantee. Rather find ways around to solve the immediate problem and then proceed according to plan.
As a consequence I now have a number of folders which are severely out of sync (between "old" and "new"). Question;
If I do a complete backup of everything on old-server and "restore" that on new-server, will it
1) only overwrite "old" content present on new-server leaving "new" content intact?
Tha depends on how you "restore" the data. If you overwrite the files on the new system you will definitely destroy the new databases, so all new mailboxes, folders and permissions on these are lost with the restore, even if the folders and mails are physically still present. One hare-brained way to get them back might be to export the mailbox.db on the old system and the new system, merge these files and then use unique to delete double entries. This might work, but I have never tried something that reckless. (^-^) PLEASE backup your mailbox.db incl. the text export to recover if that doesn't work!! All the databases are practically useless, at least I don't know any tool to merge the seen dbs, the deliver.db (not absolutely critical) If both servers are still functional you might be better off to tell all users to login to both servers at once and then copy the content they want to keep. Sandy PS: if you use sieve-scripts, please don't forget to copy those as well. (^-^) -- List replies only please! Please address PMs to: news-reply (@) japantest (.) homelinux (.) com
On Mon, 17 Oct, 2005 at 19:04:21 +0200, Sandy Drobic wrote:
Jon Clausen wrote:
I still would like for cyradm to be able to pull contents from another server, but either I'm missing something or else it's just not been implemented yet.
Unfortunately this is something most server applications tend to neglect: How to backup/restore your configuration/data.
Right. But since (current) cyradm has a 'transfer mailbox' function, at least it seems that it is being worked on. In my situation it does little good since it only supports transferring mailboxes *to* a different server. And the older version that's on the old server doesn't...
Circumstances dictated that I offlined the old server and onlined the new simultaneously. Also I needed immediate access to some of the folders, which I copied over using Mutt.
I am afraid that was a big mistake. I prefer to test and check the migration with a test system and a small database, then use the real data to check, if there is a hidden bug and see how much time is needed. I then double the time to have space for the inevitable bugs that occur in the real migration. Then I set an offline window and tell the users to store mails needed in the mean-time in off-line folders. That way they don't bug me when I need my concentration and nerves for the actual migration.
Right. I guess I should have pointed out earlier, that I am in fact the only user on either of the servers.
First rule is always: Don't let yourself be talked into things that you simply can not guarantee. Rather find ways around to solve the immediate problem and then proceed according to plan.
I was the one who talked myself into doing it like that. The hows and whys are immaterial at this point. Let's just say; the pressure was entirely from myself. Originally I didn't intend to do any 'behind the scenes' backup/restore for the transition. Which is why I never spent much time looking into the mechanics of that. My original plan was to copy the contents using IMAP, and if all else fails that is still a viable solution. Even if it means I have to depart from the 'vanila' state of the new server.
As a consequence I now have a number of folders which are severely out of sync (between "old" and "new"). Question;
At this point I should clarify about the folder structure; INBOX; Toplevel, 'synced' manually INBOX.1st-sub; Mailinglists, workrelated etc. some in sync some not INBOX.2nd-sub; Timeslice archives of above (quarterly, monthly etc) What I'm looking at restoring, at this point, is mostly just the 2nd sublevel. So I think that the basic approach outlined previously would still apply, with modifications to only work on the 2nd level?
One hare-brained way to get them back might be to export the mailbox.db on the old system and the new system, merge these files and then use unique to delete double entries. This might work, but I have never tried something that reckless. (^-^)
I like that :) If I were somewhat more comfortable with how Cyrus works, I might even try it...
PS: if you use sieve-scripts, please don't forget to copy those as well.
At least on that front there is no problem. I use Procmail for filtering. Which, incidentally, was my motivation to set up the 1st sublevel structure; So that the procmailrc's could be 'just cut/pasted' and start work immediately... which they did :) /Jon -- YMMV
Jon Clausen wrote:
Right. I guess I should have pointed out earlier, that I am in fact the only user on either of the servers.
Ah, in that case there is at least no one breathing fire down your neck to get the problem solved. (^-^)
My original plan was to copy the contents using IMAP, and if all else fails that is still a viable solution. Even if it means I have to depart from the 'vanila' state of the new server.
Why don't you do that? That is the usual way to migrate a mailbox to another server, when there is no tool available. I know I once migrated a POP3 Mailbox to Imap that way. Just add another server account in your mail client and copy the contents of the old server to the new. It will take some hours but that shouldn't be a problem.
INBOX; Toplevel, 'synced' manually INBOX.1st-sub; Mailinglists, workrelated etc. some in sync some not INBOX.2nd-sub; Timeslice archives of above (quarterly, monthly etc)
What I'm looking at restoring, at this point, is mostly just the 2nd sublevel. So I think that the basic approach outlined previously would still apply, with modifications to only work on the 2nd level?
Webcyradm should have the option to reconstruct a mailbox. Have you tried that?
One hare-brained way to get them back might be to export the mailbox.db on the old system and the new system, merge these files and then use unique to delete double entries. This might work, but I have never tried something that reckless. (^-^)
I like that :)
By the way, I think Suse already performs a text export of the mailbox database every night. Have a look at /var/lib/imap/backup. Sandy -- List replies only please! Please address PMs to: news-reply (@) japantest (.) homelinux (.) com
On Tue, 18 Oct, 2005 at 12:57:58 +0200, Sandy Drobic wrote:
Jon Clausen wrote:
Right. I guess I should have pointed out earlier, that I am in fact the only user on either of the servers.
Ah, in that case there is at least no one breathing fire down your neck to get the problem solved. (^-^)
Indeed.
My original plan was to copy the contents using IMAP ...
Why don't you do that?
Well... I would have, if any of the clients I know of supported the idea of copying folders (maybe even recursively)?
That is the usual way to migrate a mailbox to another server, when there is no tool available.
Indeed. But I've only just in the course of this correspondance become aware of the lack of (SUSE-provided) tools.
Just add another server account in your mail client and copy the contents of the old server to the new. It will take some hours but that shouldn't be a problem.
Quite so. It's just that I tend to think that one of the reasons we have computers in the first place, is to *avoid* time-consuming, simple and repetitive tasks... :P
Webcyradm should have the option to reconstruct a mailbox. Have you tried that?
No. Don't think I will either. I don't feel like introducing another layer of abstraction into the mix.
By the way, I think Suse already performs a text export of the mailbox database every night. Have a look at /var/lib/imap/backup.
Yep: plenty of mailboxes.txt.2005-xx-yy.zz.gz's there. I think I'll go back a couple of steps, and look into imapsync suggested earlier in the thread. Thanks for the responses anyway :) /Jon -- YMMV
On Wed, 19 Oct, 2005 at 07:07:57 +0200, Jon Clausen wrote:
On Tue, 18 Oct, 2005 at 12:57:58 +0200, Sandy Drobic wrote:
Just add another server account in your mail client and copy the contents of the old server to the new. It will take some hours but that shouldn't be a problem.
Quite so. It's just that I tend to think that one of the reasons we have computers in the first place, is to *avoid* time-consuming, simple and repetitive tasks... :P
<snip>
I think I'll go back a couple of steps, and look into imapsync suggested earlier in the thread.
Which is what I did: Imapsync needs the perl module Mail::IMAPClient which I installed from CPAN. The --include option makes it do what I want (copy 2nd sublevel folders). I just had to realize how to construct a regex that matches the folders I wanted. It's running now, and from what I've seen so far it's still going to take hours to finish. The difference (from 'manually' copying, using a mail-client) is that it runs without human intervention, leaving me free to attend to other activities... :) Conclusion; The backup/restore approach is probably magnitudes faster (since it avoids the IMAP overhead, and you're free to compress/decompress contents for the actual transfer). It does however require you to think ahead and, not least, *some* familiarity WRT the inner workings of your imapd of choice. Imapsync otoh, takes little effort to set up, seems reliable, is fairly straightforward to operate, but takes time to run. For me, imapsync provided the path of least resistance. Of course it's no substitute for a backup strategy, but it satisfies my immediate need. Thanks for everyone's input. /Jon -- YMMV
On 10/16/05, Jon Clausen
I'm about to retire my old mail server.
I'd like to copy the complete IMAP-structure from old -> new.
Old is; SuSE 8.2, Cyrus 2.1.12 - not 'live' New is; SUSE 9.3, Cyrus 2.2.12 - 'live'
The whole thing is largeish (couple hundred megabytes) and complexish (several layers of folders/subfolders).
Large enough and, particularly, complex enough that I'm going to spend days creating folders and copying mail, if I'm to do it 'manually' using an imap client running on my desktop machine.
Both of the servers are physically here.
If at all possible I'd like to avoid installing/changing anything on the old server.
So basically what I'm after is;
Something running on New-server, to 'pull' stuff off of Old-server.
But mostly I'm just asking for pointers;
It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?
TIA /Jon Clausen
-- YMMV
-- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
Use the perl module imapsync. Have used this in the last couple of months
on the same SuSE combination. Watch out for mailboxes with LOTS of files or very large data volumes (eg >1Gb) as the wordlist return is greater than the maxlength allowed. This tool is very good as it preserves the SEEN state for Cyrus mailboxes. Mark
On Mon, 17 Oct, 2005 at 09:38:21 +1000, Mark Keir wrote:
On 10/16/05, Jon Clausen
wrote:
It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?
Use the perl module imapsync. Have used this in the last couple of months on the same SuSE combination. Watch out for mailboxes with LOTS of files or very large data volumes (eg >1Gb) as the wordlist return is greater than the maxlength allowed. This tool is very good as it preserves the SEEN state for Cyrus mailboxes.
I already googled around and found imapsync. From what I found, it depends on Mail:IMAPClient, which is not included with the distro. Now, I guess I could just install that module from CPAN. In the end I probably will. But I'm just wondering if there really isn't a tool that's *included*? Up till now I've made *some* effort to stay 'clean' (nothing installed from anywhere but the 9.3 installation sources, and everything configured through the SUSE-provided means) on this machine. hmmm /Jon -- YMMV
On 10/17/05, Jon Clausen
On Mon, 17 Oct, 2005 at 09:38:21 +1000, Mark Keir wrote:
On 10/16/05, Jon Clausen
wrote: It seems like a standard enough kind of task, that some sort of migration -tool or -path 'ought' to be included in the distro?
Use the perl module imapsync. Have used this in the last couple of months on the same SuSE combination. Watch out for mailboxes with LOTS of files or very large data volumes (eg >1Gb) as the wordlist return is greater than the maxlength allowed. This tool is very good as it preserves the SEEN state for Cyrus mailboxes.
I already googled around and found imapsync. From what I found, it depends on Mail:IMAPClient, which is not included with the distro.
Now, I guess I could just install that module from CPAN. In the end I probably will. But I'm just wondering if there really isn't a tool that's *included*?
Up till now I've made *some* effort to stay 'clean' (nothing installed from anywhere but the 9.3 installation sources, and everything configured through the SUSE-provided means) on this machine.
hmmm
/Jon -- YMMV
Jon,
This can be done inside Cyrus with "xfer" but this requires a newer version than on the 8.2 box. This was what led me to imapsync. You can just take the files across and "reconstruct -r" but then you lose the mail information state so it all looks like it is unread or unsubscribed to (eg Thunderbird or Outlook). Mark
participants (5)
-
Jon Clausen
-
Jon Nelson
-
Ken Schneider
-
Mark Keir
-
Sandy Drobic