sendmail/fetchmail - where does this duplication come from? is this a bug or a configuration error?
SuSE V8.2 sendmail on the sending side SuSE V9.2 sendmail on the receiving side fetchmail release 6.2.5+POP2+RPA+NTLM+SDPS+SSL+OPIE+NLS I am sending out one email with 3 recipients on the from line, cheryl@domain.topleveldomain, julie@domain.topleveldomain, sue@domain.topleveldomain, The mail is received by the SMTP server of an ISP, which buffers it in mailbox@isp.com. There are 3 messages in the pop mailbox, one for cheryl, one for julie and 1 for sue. All messages have the same message header and the same message ID. Fetchmail retrieves the 3 emails from mailbox@isp.com. The first mail ist distributed to all 3 recipients, so is the second and the third. Only one message left the sending smtp server, it had a unique message ID and 3 recipients. Received at the destination were 9 messages. This is a bit annoying. Is there any cure? Should there be 3 messages in the mailbox of the ISP? or should there be only 1? Thanks in advance for your help Peter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Monday 2005-10-24 at 18:19 +0800, Peter Sutter wrote:
SuSE V8.2 sendmail on the sending side SuSE V9.2 sendmail on the receiving side fetchmail release 6.2.5+POP2+RPA+NTLM+SDPS+SSL+OPIE+NLS
I am sending out one email with 3 recipients on the from line, cheryl@domain.topleveldomain, julie@domain.topleveldomain, sue@domain.topleveldomain,
I assume you meant "To:" line.
The mail is received by the SMTP server of an ISP, which buffers it in mailbox@isp.com. There are 3 messages in the pop mailbox, one for cheryl, one for julie and 1 for sue. All messages have the same message header and the same message ID.
First problem: is it a) _one_ pop mailbox with three copies, or is it b) three mailboxes, with one email each? I assume it is "a".
Fetchmail retrieves the 3 emails from mailbox@isp.com. The first mail ist distributed to all 3 recipients, so is the second and the third.
Only one message left the sending smtp server, it had a unique message ID and 3 recipients. Received at the destination were 9 messages.
Ok, I'll try to explain a bit. The three initial copies are generated on the sending side. The ISP box thus receives three emails: cheryl, julie, and sue. Fetchmail fetchs them, and forwards them to sendmail. Being a multidrop box (a) fetchmail probably doesn't know who is the "main" destination of each; it hands them over to sendmail, who sees in the destination three addresses... thus sends each one to all three destinations. A peek at the log would serve to check if my hypothesis is correct (perhaps calling fetchmail in verbose mode (-v)). Please read the fetchmail manual, paying attention at "multidrop mail boxes". Look: | Note that | fetchmail's reconstruction of MAIL FROM and RCPT TO | lines is not guaranteed correct; the caveats discussed | under THE USE AND ABUSE OF MULTIDROP MAILBOXES below | apply. This is what is happening... (IMO). Check if your ISP puts a "Delivered-To" line, you can use it to clarify things in fetchmail. Mmmm...: | Also, note that in multidrop mode duplicate mails are sup- | pressed. A piece of mail is considered duplicate if it has | the same message-ID as the message immediately preceding and | more than one addressee. Such runs of messages may be gener- | ated when copies of a message addressed to multiple users are | delivered to a multidrop box. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDXMFLtTMYHG2NR9URAoJoAJ97YCWolNmHXRcEY3onzhmfdtxdgQCdGzRd lMcWZtkS1SdsCkOZDx5zTCg= =hqfA -----END PGP SIGNATURE-----
On Monday 24 October 2005 19:10, Carlos E. R. wrote:
The Monday 2005-10-24 at 18:19 +0800, Peter Sutter wrote:
SuSE V8.2 sendmail on the sending side SuSE V9.2 sendmail on the receiving side fetchmail release 6.2.5+POP2+RPA+NTLM+SDPS+SSL+OPIE+NLS
I am sending out one email with 3 recipients on the from line, cheryl@domain.topleveldomain, julie@domain.topleveldomain, sue@domain.topleveldomain,
I assume you meant "To:" line.
Yes, a mental typo
The mail is received by the SMTP server of an ISP, which buffers it in mailbox@isp.com. There are 3 messages in the pop mailbox, one for cheryl, one for julie and 1 for sue. All messages have the same message header and the same message ID.
First problem: is it a) _one_ pop mailbox with three copies, or is it b) three mailboxes, with one email each? I assume it is "a".
Fetchmail retrieves the 3 emails from mailbox@isp.com. The first mail ist distributed to all 3 recipients, so is the second and the third.
Only one message left the sending smtp server, it had a unique message ID and 3 recipients. Received at the destination were 9 messages.
Ok, I'll try to explain a bit. The three initial copies are generated on the sending side. The ISP box thus receives three emails: cheryl, julie, and sue. Fetchmail fetchs them, and forwards them to sendmail. Being a multidrop box (a) fetchmail probably doesn't know who is the "main" destination of each; it hands them over to sendmail, who sees in the destination three addresses... thus sends each one to all three destinations.
A peek at the log would serve to check if my hypothesis is correct (perhaps calling fetchmail in verbose mode (-v)).
It is correct
Please read the fetchmail manual, paying attention at "multidrop mail
boxes". Look: | Note that | fetchmail's reconstruction of MAIL FROM and RCPT | TO lines is not guaranteed correct; the caveats discussed | under THE USE AND ABUSE OF MULTIDROP MAILBOXES below apply.
This is what is happening... (IMO). Check if your ISP puts a "Delivered-To" line, you can use it to clarify things in fetchmail.
Mmmm...: | Also, note that in multidrop mode duplicate mails are | sup- pressed. A piece of mail is considered duplicate if it | has the same message-ID as the message immediately preceding | and more than one addressee. Such runs of messages may be | gener- ated when copies of a message addressed to multiple | users are delivered to a multidrop box.
-- Cheers, Carlos Robinson
Thanks, Carlos. Fetchmail seems to be the culprit, and yes, the ISP puts a Delivered-To: line correctly in the message headers. All emails that have an pseudo or empty "To:" header end up in postmaster (actually multidrop as defined in fecthmail). I periodically run a little script that does the actual distribution. This aspect is pretty much under control. Where the problem starts, is that in all other circumstances, fechmail goes only for the To: Headers. I tried different settings of qvirtual "Delivered-To:", "Delivered-To: ", "Delivered-To", and envelope setting; none of it makes a difference, fetchmail ignores the qvirtual keyword and seems to interpret the "To:" header only. This goes that far that it tries to deliver mail locally that are for a different domain, i.e. if the To: line contains sutterp@sopac.com.au and other@otherdomain, it tries to deliver mail to other@localhost. fetchmail does not run in daemon mode, but is controlled by crontab. Here is the fetchmailrc file I use. defaults proto pop3 set postmaster "multidrop" set no bouncemail set properties "" poll mailserver.domain.topleveldomain with proto pop3 aka mydomain.topleveldomain ISPdomain.topleveldomain qvirtual "Delivered-To: " envelope "X-Envelope-To" localdomains mydomain.topleveldomain user "mailuser" there with password "mailpassword" to * here warnings 3600 antispam 571 550 501 554 Thanks for your help Peter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Tuesday 2005-10-25 at 18:25 +0800, Peter Sutter wrote:
I tried different settings of qvirtual "Delivered-To:", "Delivered-To: ", "Delivered-To", and envelope setting; none of it makes a difference, fetchmail ignores the qvirtual keyword and seems to interpret the "To:" header only. This goes that far that it tries to deliver mail locally that are for a different domain, i.e. if the To: line contains sutterp@sopac.com.au and other@otherdomain, it tries to deliver mail to other@localhost.
Does the email contain an "X-Envelope-To" header? If that's so, I think you don't need anything else. I would remove the qvirtual and envelope entries in your config, at least for the moment (I guess you are running tests, you are not "in production mode" yet): -E <line> | --envelope <line> (Keyword: envelope) This option changes the header fetchmail assumes will carry a copy of the mail's envelope address. Normally this is `X-Envelope-To' but as this header is not standard, practice varies. See the discussion of multidrop address handling below. As a special case, `envelope "Received"' enables parsing of sendmail-style Received lines. This is the default, and it should not be necessary unless you have globally disabled Received parsing with `no envelope' in the .fetchmailrc file. So you do not need to specify it. If that does not work, I think this paragraph in the man applies: By using the option `envelope Delivered- To:' you can make fetchmail reliably identify the original envelope recipient, but you have to strip the `mbox-userstr-' prefix to deliver to the correct user. This is what this option is for. I can't be more specific, sorry: I'm getting outside of my direct experience by this moment ;-) -- I know fairly well how fetchmail works, but I don't have multidrop boxes, so no direct experience there. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDYB8ZtTMYHG2NR9URAkUmAJ0QiwPwD9kuxGj/cWQc3N56Xm/B4ACfSstR foN1JlZmJiaxJOLfjweJncE= =O4f2 -----END PGP SIGNATURE-----
participants (2)
-
Carlos E. R.
-
Peter Sutter