SA/Sieve Woes
Hi all,
We have MS 2005Q6 core patch -58 installed. Below is the story and problems that we have encountered when setting up Spam Assassin.
SA is setup to label spam with addtag [SPAM] so that every spam email is tagged in the tcp_local channel. The spamfilter1_string_action looks like this:
require ["addheader","fileinto"];
addtag"[SPAM]";
addheader"$U";
fileinto"Spam";
stop;
Problem 1. In a user level sieve script, the (header :matches "Subject"
"[SPAM]*") did match nothing even though the [SPAM] tag did appear in the
final subject line. It looks as if the addtag processing is only done
after the header match operations, or the header match uses the "original"
subject line for the match. Is there anything wrong with the sieve script?
After some time, we modified the script with the intention to give every scanned email detailed sa results even for non-spams, and at the same time add a new header for user leve sieve scripts.
#mode=2,field= in spamassassin.opt
require ["spamtest","relational","comparator-i;ascii-numeric","fileinto","addheader"];
spamadjust"$U";
addheader"Spam-Test: $U";
if spamtest :value"ge" :comparator"i;ascii-numeric""5"{
addheader"Spam: Y";
addtag"[SPAM]";
fileinto"Spam";
}else{
keep;
}
Problem 2. The above script runs well, except one thing that cost me a full day debugging. Note the 2nd addheader line, which reads addheader "Spam: Y". The problem is, it seems that we cannot have a 2nd addheader in one single script, which is longer than 7 characters in length including the space in between. If you say addheader "ABCDE: Y", you are busted. The filter will just refuse to run. Is this a bug or something?
OK, never mind, who care about a 7 byte header. We want to have a fine-grain control over the traditional email forwarding feature that uses mailForwardingAddress attribute. I suppose Sieve is more flexible when it comes to filtering and redirecting mail. So we disable the mailForwardingAddress, and use Sieve. So now, we try to setup a user level sieve to redirect important emails, even it is identified as spam like this:
require ["fileinto","copy"];
if header :match"Spam""Y"{
fileinto :copy"Spam";
}
if header :match"Priority""Urgent"{
redirect"another@email.address";
}
Problem 3. When SA identifies a spam, it is marked as such with addtag [SPAM] (see code above). The tag should appear once. It is the case for the email left in the Spam folder of the server. But the email received by "another@email.address" contains 2 tags [SPAM] [SPAM] in the subject line. Clearly, SA is run twice, one for the incoming message, and one for the redirected message, and the log file confirms this. It seems that the redirect action cause the mail to be queued in the tcp_local channel, which causes SA to run. But it is interesting that the traditional emailForwardingAddress attribute kind of forwarding doesn't have this kind of behavior. Is there a way we can prevent SA from running twice in sieve to save resources and not to duplicate the tag?
Anyone can shed some lights on these matters?
Thanks in advance.
PY
[4320 byte] By [
e-jing] at [2007-11-26 11:00:38]
# 1
Hi,
> SA is setup to label spam with addtag [SPAM] so
> that every spam email is tagged in the tcp_local
> channel. The spamfilter1_string_action looks like
> this:
>
> > require ["addheader","fileinto"];
> addtag "[SPAM]";
> addheader "$U";
> fileinto "Spam";
> stop;
>
>
> Problem 1. In a user level sieve script, the (header
> :matches "Subject"
> [SPAM]*") did match nothing even though the [SPAM]
> tag did appear in the
> final subject line. It looks as if the addtag
> processing is only done
> after the header match operations, or the header
> match uses the "original"
> subject line for the match. Is there anything wrong
> with the sieve script?
Please provide the actual spamfilter1_string_action you used - not just the 'cleaned' version.
Also I should point out that matching on a modified subject header can lead to unexpected behaviour e.g. forwarded spam emails getting filtered to spam folder because [SPAM] appears in the subject line. It is much better to use the spamadjust/spamtest if possible.
> After some time, we modified the script with the
> intention to give every scanned email detailed sa
> results even for non-spams, and at the same time add
> a new header for user leve sieve scripts.
>
> > #mode=2,field= in spamassassin.opt
> require
> ["spamtest","relational","comparator-i;ascii-numeric",
> "fileinto","addheader"];
> spamadjust "$U";
> addheader "Spam-Test: $U";
> if spamtest :value "ge" :comparator "i;ascii-numeric"
> "5"{
>addheader "Spam: Y";
> addtag "[SPAM]";
>fileinto "Spam";
> lse{
>keep;
>
>
>
> Problem 2. The above script runs well, except one
> thing that cost me a full day debugging. Note the
> 2nd addheader line, which reads addheader "Spam:
> Y". The problem is, it seems that we cannot have a
> 2nd addheader in one single script, which is longer
> than 7 characters in length including the space in
> between. If you say addheader "ABCDE: Y", you are
> busted. The filter will just refuse to run. Is this
> a bug or something?
There is one limit you may be hitting (I found this one yesterday) and that is that spamfilterX_string_action is currently limited to a maximum of 256 characters - that is AFTER the expansion of the $U btw. Please log a support call (if you haven't already) and ask to be added to bug #6485247.
There is a workaround to this limit and that is to use channel-level filtering, refer to my last reply in a forum question yesterday:
http://forum.sun.com/jive/thread.jspa?threadID=109607
> OK, never mind, who care about a 7 byte header. We
> want to have a fine-grain control over the
> traditional email forwarding feature that uses
> mailForwardingAddress attribute. I suppose Sieve is
> more flexible when it comes to filtering and
> redirecting mail. So we disable the
> mailForwardingAddress, and use Sieve. So now, we try
> to setup a user level sieve to redirect important
> emails, even it is identified as spam like this:
>
> > require ["fileinto","copy"];
> if header :match "Spam" "Y" {
>fileinto :copy "Spam";
>
> if header :match "Priority" "Urgent" {
>redirect "another@email.address";
>
>
>
> Problem 3. When SA identifies a spam, it is marked
> as such with addtag [SPAM] (see code above). The tag
> should appear once. It is the case for the email
> left in the Spam folder of the server. But the email
> received by "another@email.address" contains 2 tags
> [SPAM] [SPAM] in the subject line. Clearly, SA is
> run twice, one for the incoming message, and one for
> the redirected message, and the log file confirms
> this. It seems that the redirect action cause the
> mail to be queued in the tcp_local channel, which
> causes SA to run. But it is interesting that the
> traditional emailForwardingAddress attribute kind of
> forwarding doesn't have this kind of behavior. Is
> there a way we can prevent SA from running twice in
> sieve to save resources and not to duplicate the
> tag?
>
>Anyone can shed some lights on these matters?
>Thanks in advance.
Well I'm not going to troubleshoot each of your issues as that will take me a ages - sieve scripts can be tricky. What is your overall goal of the filtering? From what you have mentioned so far it is:
At the system level:
-> add the [SPAM] tag to all spam emails
-> add the X-Spam: Y to all spam emails emails (you should use X- for custom headers btw.)
-> add the Spam-test: <score> to all scanned emails
At the user level:
-> filter all Spam emails to a spam folder
-> forward non-spam high priority emails to another address
Are you using separate front-end MTA/back-end store configuration and not using LMTP between the two? This will change whether spamadjust/spamtest can be used for the user level sieve rules.
Regards,
Shane.
# 3
Hi
> For problem 1,
> spamfilter1_string_action=data:,require
> ["addheader","fileinto"];addtag "[SPAM]";addheader
> "$U";stop;
It would appear in this case that the modification to the Subject header is not visible to the user-level sieve script in this case. I'm not sure how you can get around this.
> For problem 3, let me simply the problem here. When
> we use the redirect action in user level sieve, we
> notice that the mail is actually scanned twice by
> SA. We can tell from the subject line of the
> redirected mail, that contains 2 addtags like
> Subject: [SPAM] [SPAM]. We are trying to avoid it,
> not for the sake of mailbox tidyness, but for
> reducing the workload of the servers - we don't want
> every mail scanned twice, once for incomming, and
> once for redirecting. Any idea?
First, turn on sieve filter logging. Add the following to your option.dat and ./imsimta cnbuild; ./imsimta restart
LOG_FILTER=1
Then send an email which provides logging for the above scenario. Cut/paste here.
Please also supply the imta.cnf channel line where you have defined the [source|destination]spamfilter1optin line (which applies the filtering).
From what you have described, I think you may have just put the filteroptin line in the wrong spot, so emails are being unnecessarily scanned (e.g. on the ims-ms channel rather then the tcp_local channel).
Regards,
Shane.
