fetchmail -keep problem

Constant Brouerius van Nidek

5 May 2003 5 May '03

15:52

Newly installed fetchmail and configured it with fetchmailconf on my Suse8.1. I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw. Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails. Can that be solved with a command that I have overlooked?

Show replies by thread

Constant Brouerius van Nidek

5 May 5 May

16:23

New subject: [SLE] fetchmail -keep problem

Okido,If you have not found it since 7.3, there is only a small case that I am more lucky ;-). On Monday 05 May 2003 23:32, Bruce Marshall wrote:

...

On Monday 05 May 2003 11:52, Constant Brouerius van Nidek wrote:

...
Newly installed fetchmail and configured it with fetchmailconf on my Suse8.1. I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw. Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails. Can that be solved with a command that I have overlooked?

It's been that way ever since I started using fetchmail (7.3?)

There may be a way to prevent it but I've never found it.

Bruce Marshall

16:32

New subject: [SLE] fetchmail -keep problem

On Monday 05 May 2003 11:52, Constant Brouerius van Nidek wrote:

...

Newly installed fetchmail and configured it with fetchmailconf on my Suse8.1. I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw. Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails. Can that be solved with a command that I have overlooked?

It's been that way ever since I started using fetchmail (7.3?) There may be a way to prevent it but I've never found it.

gary

16:43

New subject: [SLE] fetchmail -keep problem

On Mon, May 05, 2003 at 12:32:45PM -0400 or thereabouts, Bruce Marshall wrote:

...

On Monday 05 May 2003 11:52, Constant Brouerius van Nidek wrote:

...
Newly installed fetchmail and configured it with fetchmailconf on my

...

...
I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw.

...

...
Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails.

It's been that way ever since I started using fetchmail (7.3?)

There may be a way to prevent it but I've never found it.

Long ago, I switched to getmail, a fetchmail alternative, written in python. This will cure the double download problem. -- Gary

Constant Brouerius van Nidek

17:05

New subject: [SLE] fetchmail -keep problem

Hai Gary, Do you think getmail is a good alternative for fetchmail? Because I will try to have my mail setup reorganised in two or three months and will then have all my mail on Suse. As a mater of fact, I set up the fetchmail in order to get the taste of it and experiment with it in order to make it the program of choice in my new setup. Does it have a grafic configuration (have to admit that I liked the fetchmailconf). spamfilter and so on as in fetchmail? Is it maintained somewhere? On Monday 05 May 2003 23:43, gary wrote:

...

On Mon, May 05, 2003 at 12:32:45PM -0400 or thereabouts, Bruce Marshall wrote:

...
On Monday 05 May 2003 11:52, Constant Brouerius van Nidek wrote:

...
Newly installed fetchmail and configured it with fetchmailconf on my

I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw.

Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails.

It's been that way ever since I started using fetchmail (7.3?)

There may be a way to prevent it but I've never found it.

Long ago, I switched to getmail, a fetchmail alternative, written in python. This will cure the double download problem.

David Krider

17:40

New subject: [SLE] fetchmail -keep problem

On Mon, 2003-05-05 at 10:52, Constant Brouerius van Nidek wrote:

...

Can that be solved with a command that I have overlooked?

It might help if you were to post your .fetchmailrc (with password redacted, of course)... I refuse to think that this can't be configured correctly. If fetchmail really had a bug that glaring, it would have been fixed a long time ago. I'm not saying that I can get it going. It's just that I have used fetchmail before, and might want to again. If there's a problem with it on SuSE, I want to get to the bottom of it too. Regards, dk -- David "Dunkirk" Krider, http://www.davidkrider.com Acts 17:28, "For in Him we live, and move, and have our being." Linux: Will you use the power for good... or for AWESOME?

gary

17:45

New subject: [SLE] fetchmail -keep problem

On Tue, May 06, 2003 at 12:05:38AM +0700 or thereabouts, Constant Brouerius van Nidek wrote:

...

Do you think getmail is a good alternative for fetchmail?

Absolutely, I trust the author completely, and have used many of his GNU-ware.. Fetchmail recently had another security alert. Getmail has never had one. It is very easy to set up, and the author maintains a mailing list which is very helpful. It too has a completely configurable getmailrc file, and will deliver to mbox or Maildir format, and it will distribute to individual email addresses... in other words, a mini procmail, if you wish it to. There is also a sample getmailrc file on the URL and in the package. Taken from the main page. Summary of Features Retrieve mail from an unlimited number of POP3 mailboxes and servers. Support for multidrop or domain mailboxes. Safe and reliable delivery to qmail-style Maildirs, as well as program (pipe) delivery for use with arbitrary external MDAs. Includes an MDA for mbox files that supports mboxrd format and fcntl-type flock locking. Does not destroy information by rewriting mail headers. Does not cause mail loops by doing SMTP injection, and therefore does not require that you run an MTA (like qmail or sendmail) on your host. Can remember which mail it has already retrieved, and can be set to only download new messages. Written in Python, and therefore easy to extend or customize. Simple to install, configure, and use. Won DaveCentral's "Best of Linux" award on 13 January 2000.

...

Is it maintained somewhere?

Yes, a quick google search provides http://www.qcc.ca/~charlesc/software/getmail-3.0/ -- Gary "All that is necessary for the triumph of evil is that good men do nothing." (Edmund Burke)

David Krider

17:48

New subject: [SLE] fetchmail -keep problem

On Mon, 2003-05-05 at 12:40, David Krider wrote:

...

On Mon, 2003-05-05 at 10:52, Constant Brouerius van Nidek wrote:

...
Can that be solved with a command that I have overlooked?

Basically, everything below leads to: do a `fetchmail -v' and inspect the output. It may just be that your _server_ is the problem, and fetchmail might not be able to do anything about it...

...

From the fetchmail man page, under RETRIEVAL FAILURE MODES,

The normal mode of fetchmail is to try to download only `new' messages, leaving untouched (and undeleted) messages you have already read directly on the server (or fetched with a previous fetchmail --keep). But you may find that messages you've already read on the server are being fetched (and deleted) even when you don't specify --all. There are several reasons this can happen. One could be that you're using POP2. The POP2 protocol includes no representation of `new' or `old' state in mes sages, so fetchmail must treat all messages as new all the time. But POP2 is obsolete, so this is unlikely. Under POP3, blame RFC1725. That version of the POP3 pro tocol specification removed the LAST command, and some POP servers follow it (you can verify this by invoking fetch mail -v to the mailserver and watching the response to LAST early in the query). The fetchmail code tries to compensate by using POP3's UID feature, storing the iden tifiers of messages seen in each session until the next session, in the .fetchids file. But this doesn't track messages seen with other clients, or read directly with a mailer on the host but not deleted afterward. A better solution would be to switch to IMAP. Another potential POP3 problem might be servers that insert messages in the middle of mailboxes (some VMS implementations of mail are rumored to do this). The fetchmail code assumes that new messages are appended to the end of the mailbox; when this is not true it may treat some old messages as new and vice versa. The only real fix for this problem is to switch to IMAP. Yet another POP3 problem is that if they can't make temp files in the user's home directory, some POP3 servers will hand back an undocumented response that causes fetchmail to spuriously report "No mail". The IMAP code uses the presence or absence of the server flag \Seen to decide whether or not a message is new. Under Unix, it counts on your IMAP server to notice the BSD-style Status flags set by mail user agents and set the \Seen flag from them when appropriate. All Unix IMAP servers we know of do this, though it's not specified by the IMAP RFCs. If you ever trip over a server that doesn't, the symptom will be that messages you have already read on your host will look new to the server. In this (unlikely) case, only messages you fetched with fetchmail --keep will be both undeleted and marked old. In ETRN and ODMR modes, fetchmail does not actually retrieve messages; instead, it asks the server's SMTP lis tener to start a queue flush to the client via SMTP. Therefore it sends only undelivered messages. -- David "Dunkirk" Krider, http://www.davidkrider.com Acts 17:28, "For in Him we live, and move, and have our being." Linux: Will you use the power for good... or for AWESOME?

Patrick Shanahan

18:01

New subject: [SLE] fetchmail -keep problem

* Constant Brouerius van Nidek [05-05-03 11:44]:

...

Okido,If you have not found it since 7.3, there is only a small case that I am more lucky ;-).

On Monday 05 May 2003 23:32, Bruce Marshall wrote:

...
On Monday 05 May 2003 11:52, Constant Brouerius van Nidek wrote:

...
Newly installed fetchmail and configured it with fetchmailconf on my Suse8.1. I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw. Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails. Can that be solved with a command that I have overlooked?

It's been that way ever since I started using fetchmail (7.3?)

There may be a way to prevent it but I've never found it.

Procmail: # ------------------------------------------------------- # remove duplicates ## from man procmailex examples ## # 12-13-2002 LOCKFILE = msgid.cache.lock :0 Wh: msgid.lock | formail -D 16384 msgid.cache LOCKFILE # ------------------------------------------------------- You will still download them but they will not appear in your mailbox. You may have to experiment with the cache size, depending on your traffic and the length of time you leave the mail on the server. -- Patrick Shanahan Please avoid TOFU and trim >quotes< http://wahoo.no-ip.org Registered Linux User #207535 icq#173753138 @ http://counter.li.org Linux, a continuous *learning* experience

Bruce Marshall

18:48

New subject: [SLE] fetchmail -keep problem

On Monday 05 May 2003 14:01, Patrick Shanahan wrote:

...

* Constant Brouerius van Nidek [05-05-03 11:44]:

...
Okido,If you have not found it since 7.3, there is only a small case that I am more lucky ;-).

On Monday 05 May 2003 23:32, Bruce Marshall wrote:

...
On Monday 05 May 2003 11:52, Constant Brouerius van Nidek wrote:

...
Newly installed fetchmail and configured it with fetchmailconf on my Suse8.1. I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw. Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails. Can that be solved with a command that I have overlooked?

It's been that way ever since I started using fetchmail (7.3?)

There may be a way to prevent it but I've never found it.

Procmail: # ------------------------------------------------------- # remove duplicates ## from man procmailex examples ## # 12-13-2002

LOCKFILE = msgid.cache.lock

:0 Wh: msgid.lock : | formail -D 16384 msgid.cache

LOCKFILE # -------------------------------------------------------

You will still download them but they will not appear in your mailbox. You may have to experiment with the cache size, depending on your traffic and the length of time you leave the mail on the server.

You've got to be kidding!! Right now I'm staring at 337 emails on my isp pop... and you want me to download them all every 6 minutes? Not a very good solution. (the reason for the 337 is that I am not at home right now... and want to keep them until I get there)

...

-- Patrick Shanahan Please avoid TOFU and trim >quotes< http://wahoo.no-ip.org Registered Linux User #207535 icq#173753138 @ http://counter.li.org Linux, a continuous *learning* experience

Bruce Marshall

18:51

New subject: [SLE] fetchmail -keep problem

On Monday 05 May 2003 13:48, David Krider wrote:

...

On Mon, 2003-05-05 at 12:40, David Krider wrote:

...
On Mon, 2003-05-05 at 10:52, Constant Brouerius van Nidek wrote:

...
Can that be solved with a command that I have overlooked?

Basically, everything below leads to: do a `fetchmail -v' and inspect the output. It may just be that your _server_ is the problem, and fetchmail might not be able to do anything about it...

From the fetchmail man page, under RETRIEVAL FAILURE MODES,

The normal mode of fetchmail is to try to download only `new' messages, leaving untouched (and undeleted) messages you have already read directly on the server (or fetched with a previous fetchmail --keep). But you may find that messages you've already read on the server are being fetched (and deleted) even when you don't specify --all. There are several reasons this can happen.

One could be that you're using POP2. The POP2 protocol includes no representation of `new' or `old' state in mes sages, so fetchmail must treat all messages as new all the time. But POP2 is obsolete, so this is unlikely.

Under POP3, blame RFC1725. That version of the POP3 pro tocol specification removed the LAST command, and some POP servers follow it (you can verify this by invoking fetch mail -v to the mailserver and watching the response to LAST early in the query). The fetchmail code tries to compensate by using POP3's UID feature, storing the iden tifiers of messages seen in each session until the next session, in the .fetchids file. But this doesn't track messages seen with other clients, or read directly with a mailer on the host but not deleted afterward. A better solution would be to switch to IMAP.

Another potential POP3 problem might be servers that insert messages in the middle of mailboxes (some VMS implementations of mail are rumored to do this). The fetchmail code assumes that new messages are appended to the end of the mailbox; when this is not true it may treat some old messages as new and vice versa. The only real fix for this problem is to switch to IMAP.

Yet another POP3 problem is that if they can't make temp files in the user's home directory, some POP3 servers will hand back an undocumented response that causes fetchmail to spuriously report "No mail".

The IMAP code uses the presence or absence of the server flag \Seen to decide whether or not a message is new. Under Unix, it counts on your IMAP server to notice the BSD-style Status flags set by mail user agents and set the \Seen flag from them when appropriate. All Unix IMAP servers we know of do this, though it's not specified by the IMAP RFCs. If you ever trip over a server that doesn't, the symptom will be that messages you have already read on your host will look new to the server. In this (unlikely) case, only messages you fetched with fetchmail --keep will be both undeleted and marked old.

In ETRN and ODMR modes, fetchmail does not actually retrieve messages; instead, it asks the server's SMTP lis tener to start a queue flush to the client via SMTP. Therefore it sends only undelivered messages.

Whatever the possible reason may be from all of the above, it seems to me that if Kmail with 'do not delete' checked for a pop server works, why shouldn't fetchmail?

...

-- David "Dunkirk" Krider, http://www.davidkrider.com Acts 17:28, "For in Him we live, and move, and have our being." Linux: Will you use the power for good... or for AWESOME?

Patrick Shanahan

19:17

New subject: [SLE] fetchmail -keep problem

* Bruce Marshall [05-05-03 13:08]:

...

On Monday 05 May 2003 11:52, Constant Brouerius van Nidek wrote:

...
Newly installed fetchmail and configured it with fetchmailconf on my Suse8.1. I am downloading and sorting out my email still on a seperate system in DrDos and I not yet ready with my home network setup so I want to keep the emails on my isp when I have a look at them with Suse. With "fetchmail -keep" that works but with a flaw. Everytime fetchmail thinks it should contact my ISP all emails which it had already downloaded are downloaded again. That means with the suse list only a rapid filling up of my harddisc with a multitude of the same emails. Can that be solved with a command that I have overlooked?

It's been that way ever since I started using fetchmail (7.3?)

There may be a way to prevent it but I've never found it.

man fetchmail under "RETRIEVAL FAILURE MODES" para3 The normal mode of fetchmail is to try to download only ew' messages, leaving untouched (and undeleted) messages you have already read directly on the server (or fetched with a previous fetchmail --keep). But you may find that messages you've already read on the server are being fetched (and deleted) even when you don't specify --all. There are several reasons this can happen. One could be that you're using POP2. The POP2 protocol includes no repre sentation of ew' or d' state in messages, so fetchmail must treat all messages as new all the time. But POP2 is obsolete, so this is unlikely. -- Patrick Shanahan Please avoid TOFU and trim >quotes< http://wahoo.no-ip.org Registered Linux User #207535 icq#173753138 @ http://counter.li.org Linux, a continuous *learning* experience

Carlos E. R.

19:23

New subject: [SLE] fetchmail -keep problem

The 03.05.05 at 14:51, Bruce Marshall wrote: [Please, trim your quotes]

...

Whatever the possible reason may be from all of the above, it seems to me that if Kmail with 'do not delete' checked for a pop server works, why shouldn't fetchmail?

As David said, when we see the fetchmailrc file - with passwords and login names removed - we will think it over :-) One reason could be that kmail has access to the email to do a compare. -- Cheers, Carlos Robinson

David Krider

19:23

New subject: [SLE] fetchmail -keep problem

On Mon, 2003-05-05 at 13:51, Bruce Marshall wrote:

...

Whatever the possible reason may be from all of the above, it seems to me that if Kmail with 'do not delete' checked for a pop server works, why shouldn't fetchmail?

First of all, you didn't mention that KMail was working where fetchmail wasn't. Why are you concerned about fetchmail if KMail is doing everything you need done? Second of all, what's that got to do with it? Whatever problem that fetchmail (a tiny command-line-only program) isn't working around due to considerations of size, performance, and standards compliance... What's to keep KMail from doing so? If this other program, "getmail," works around it, great! Use that one. I'd be very interested in hearing if it does. If not, *and you want a command-line program to do these sorts of things*, then you could always gank the KMail source and put it into your own fork of fetchmail... dk -- David "Dunkirk" Krider, http://www.davidkrider.com Acts 17:28, "For in Him we live, and move, and have our being." Linux: Will you use the power for good... or for AWESOME?

Bruce Marshall

19:37

New subject: [SLE] fetchmail -keep problem

On Monday 05 May 2003 15:23, David Krider wrote:

...

On Mon, 2003-05-05 at 13:51, Bruce Marshall wrote:

...
Whatever the possible reason may be from all of the above, it seems to me that if Kmail with 'do not delete' checked for a pop server works, why shouldn't fetchmail?

First of all, you didn't mention that KMail was working where fetchmail wasn't. Why are you concerned about fetchmail if KMail is doing everything you need done?

I normally use fetchmail/procmail to draw email off of several pop servers. Then I use Kmail to pop the mail off my own machine and into Kmail. But when I am away from home, I would like to retain the email on the pop server until I get back home. Thus, I have to use only Kmail (with 'do not delete') while I am away from home... without procmail and spamassassin and other filtering goodies... Just Kmail.

...

Second of all, what's that got to do with it? Whatever problem that fetchmail (a tiny command-line-only program) isn't working around due to considerations of size, performance, and standards compliance... What's to keep KMail from doing so?

Your point is?? Kmail can handle it, fetchmail doesn't. That's all I'm saying.

...

If this other program, "getmail," works around it, great! Use that one. I'd be very interested in hearing if it does. If not, *and you want a command-line program to do these sorts of things*, then you could always gank the KMail source and put it into your own fork of fetchmail...

It seems to work but it doesn't work in the global fashion that fetchmail/procmail can be made to work in. It's more of a 'user only' gizmo which means it's eash user for himself... and admin'ing a couple of machines would be more problematic. But it does appear to retain email on a pop server.

...

dk

-- David "Dunkirk" Krider, http://www.davidkrider.com Acts 17:28, "For in Him we live, and move, and have our being." Linux: Will you use the power for good... or for AWESOME?

gary

20:16

New subject: [SLE] fetchmail -keep problem

On Mon, May 05, 2003 at 03:37:30PM -0400 or thereabouts, Bruce Marshall wrote:

...

I normally use fetchmail/procmail to draw email off of several pop servers. Then I use Kmail to pop the mail off my own machine and into Kmail.

But when I am away from home, I would like to retain the email on the pop server until I get back home. Thus, I have to use only Kmail (with 'do not delete') while I am away from home... without procmail and spamassassin and other filtering goodies... Just Kmail.

...

It seems to work but it doesn't work in the global fashion that fetchmail/procmail can be made to work in. It's more of a 'user only' gizmo which means it's eash user for himself... and admin'ing a couple of machines would be more problematic. <snip>

getmail will work in global mode... just taken from their website: http://www.qcc.ca/~charlesc/software/getmail-3.0/getmail.html#summary getmail must have write permission to the Maildirs you are delivering to. This can be done by making the various destinations group writable, by only delivering to destinations owned by the user getmail is running as, or by running getmail as root. When delivering to Maildirs, getmail will attempt to chown any files it creates to the owner of the Maildir.

...

But it does appear to retain email on a pop server.

Yes, it does. I have recommended this to people all over the world, and have never had a complaint. -- Gary Why don't sheep shrink when it rains?

Bruce Marshall

6 May 6 May

00:19

New subject: [SLE] fetchmail -keep problem

On Monday 05 May 2003 15:23, Carlos E. R. wrote:

...

As David said, when we see the fetchmailrc file - with passwords and login names removed - we will think it over :-)

Well, I'm not that bothered by it.... I just don't ever count on fetchmail to keep mail on the server.

...

One reason could be that kmail has access to the email to do a compare.

Eh? After I delete some email from Kmail at 10AM, it's still going to be comparing things at 2PM? I don't think so.

7682

Age (days ago)

7683

Last active (days ago)

List overview

Download

16 comments

8 participants

participants (8)

Bruce Marshall
Carlos E. R.
Constant Brouerius van Nidek
David Krider
gary
gary
gary
Patrick Shanahan