reconstruct -r -f on folder not working
Hi,
We have an iplanet 5.2 system running on Solaris 9 and we are wanting to move the INBOX to a new folder daily. I'm developing a script to move the INBOX folders (00 thru 99) to a new directory that I create and then run "reconstruct -m" and then "reconstruct -f -r user/NAME/FOLDER" . But the reconstruct -f is not making a store.idx file -- the prompt comes back very quickly. I know it should take longer because there is about one Gig of messages that get moved. I see by the documentation I can't do "mboxutil -r user /NAME/INBOX user/NAME/FOLDER". So my script does a "mv ? FOLDER" command in to the manually made directory of the account. Why wouldn't the forced reconstruct not work? What do I need to do to make it do the right thing?
Hal Huntley
SRI International
[804 byte] By [
halsria] at [2007-11-26 19:44:19]
# 1
Hi,
Firstly this is a bad idea. You should not be manually moving messages on disk - what do you expect will happen if you move the folders when emails are being delivered for example?
Why do you not just write a perl script to do the same thing (using say Net::IMAPClient)? It should be possible to do this in a few dozen lines.
> We have an iplanet 5.2 system running on Solaris 9
> and we are wanting to move the INBOX to a new folder
> daily. I'm developing a script to move the INBOX
> folders (00 thru 99) to a new directory that I
> create and then run "reconstruct -m" and then
> "reconstruct -f -r user/NAME/FOLDER" .
A better approach is to create the folder first, then move the messages inside. Therefore you don't even need to run the reconstruct -m command e.g.
mboxutil -c user/NAME/folder-name
mv <inbox path>/[0-9][0-9] <new folder path>/
reconstruct -r user/NAME/folder-name
Regards,
Shane.
# 2
Thank you for the reply.
I have thought about the issue of delivering while moving. Some more details. This account is one that is delivered to only by way of a rule in the imta.filter file. We use it for fighting SPAM. To stop delivery of the email I move the imta.filter out of the way, do an "imsimta refresh" and then do the "mv". Once that is done, I put the imta.filter back in place and do another refresh.
I did not know about the perl IMAP option. I'll check in to that. While I'm in that learning curve, I'd like to make this method work.
I'll modify my script to do the "mboxutil -c" method of making the folder.
What could be causing the reconstruct to not happen? Is there some other way to force it to go so that we can look at the messages in the folder?
I tried moving the new folder to another via "mboxutil -r user/NAME/FOLDER user/NAME/ANOTHERFOLDER", but that just hangs and nothing happens for a long time.
Thanks,
Hal
# 3
Hi,
> I have thought about the issue of delivering while
> moving. Some more details. This account is one that
> is delivered to only by way of a rule in the
> imta.filter file. We use it for fighting SPAM. To
> stop delivery of the email I move the imta.filter out
> of the way, do an "imsimta refresh" and then do the
> "mv". Once that is done, I put the imta.filter back
> in place and do another refresh.
If you are already using a sieve filter to deliver email, may I suggest an alternate approach:
1. Modify the script to deliver the messages to a *subfolder* directly, rather then the default INBOX e.g.
28Feb07+<email address>
This will deliver the emails to the folder "28Feb07" under the account. If the folder doesn't exist it will be created.
2. If you need to rotate say once a day, then just increment the folder name in the imta.filter script, and run the commands:
./imsimta cnbuild; ./imsimta restart
(btw. you shouldn't use ./imsimta refresh, please use the above commands instead in future -- in fact in the next release ./imsimta refresh no longer works).
This approach means you only have to restart the MTA once for the change to become active AND as an extra bonus, no manual moving of emails.
> I did not know about the perl IMAP option. I'll
> check in to that. While I'm in that learning curve,
> I'd like to make this method work.
This is definitely something you should be investigating -- if nothing else then it will provide you with background in the future for if you ever need to manipulate email accounts in some kind of scripted fashion without needing to run reconstructs etc.
> What could be causing the reconstruct to not happen?
> Is there some other way to force it to go so that we
> can look at the messages in the folder?
You have a classic chicken & egg scenario. Lets break down what you said in a previous email:
"I'm developing a script to move the INBOX folders (00 thru 99) to a new directory that I create and then run "reconstruct -m""
Firstly, reconstruct -m goes through the folder structures on disk looking for the store.idx files which indicate that there is a folder structure it should know about - this store.idx information is then used to rebuild the master folder database.
If all you moved was the underlying message folders [0-9][0-9] then the reconstruct -m would have skipped your new folder.
"and then "reconstruct -f -r user/NAME/FOLDER""
This command works off the master folder database information. If a folder is not registered (which due to the missing store.idx folder would be the case), then it will be ignored.
So the reconstruct -r -f won't work and create a new store.idx file because the folder isn't registered with the master folder database, and the reconstruct -m won't work because the store.idx file is missing.
The mboxutil -c command both creates the new directory on disk, and the required store.idx file and registers the folder with the master folder database.
Regards,
Shane.
# 4
Thanks for that information. I learned a lot from it.
I did not know of the chicken and egg problem with the mailbox folders. It explains several things we've had problems with over the last few years. I think I can fix the reconstruct problem now: I'll just create a new folder with mboxutil -c and then move the [0-9][0-9] directories in to it and do another reconstruct -r -f on the new folder.
We are working on upgrading to version 6 and should get there some time soon.
But let me ask a imta.filter question. Right now our filters take the form of:
if header :matches "Subject" "*xanax*" {
redirect "rejects@f.q.d.n.com";
}
We match against quite a few headers and keywords, but that is a good example.Note that I redirect to an email address with a FQDN. As I was reading about filters, I thought that was the requirement, but I gather from what you said I could do something like:
redirect "20070227+Rejects"
and it will move the message to the 20070227 folder in the Rejects account.
?Or are you saying I should write it as:
redirect "20070227+rejects@f.q.d.n.com"
I'm not sure what the syntax of the imta.filter file should take in this case.
I could then write the script to do a sed or awk replacement of the dated folder name in the imta.filter file once around midnight. I like that idea.
Thanks,
Hal Huntley
# 5
Hi,
> Thanks for that information. I learned a lot from
> it.
No worries... the more you know the less you need to ask.. win-win.
> I did not know of the chicken and egg problem with
> the mailbox folders. It explains several things
> we've had problems with over the last few years. I
> think I can fix the reconstruct problem now: I'll
> just create a new folder with mboxutil -c and then
> move the [0-9][0-9] directories in to it and do
> another reconstruct -r -f on the new folder.
In running a large messaging server install for several years in my previous job (100K staff/students over several mail-stores) I never had to run the reconstruct -m command. Usually if a customer says they are using this command for anything but a total master database corruption recovery, this usually rings alarm bells.
> We are working on upgrading to version 6 and should
> get there some time soon.
I would recommend that if you are moving to 6, that you consider holding off a few weeks more for 6.3 to come out. The underlying changes (especially the mshttpd process now talking IMAP) make deployment that much easier.
> Or are you saying I should write it as:
>redirect "20070227+rejects@f.q.d.n.com"
> sure what the syntax of the imta.filter file should
> take in this case.
You should always use the FQDN form of an email address as you listed above.
> I could then write the script to do a sed or awk
> replacement of the dated folder name in the
> imta.filter file once around midnight. I like that
> idea.
Yep much cleaner and simpler.
Regards,
Shane.
# 6
Shane,
Now I'm in "nitty gritty" mode :-). To test the sending to a specific folder I make a rule like:
if header :matches "Subject" "*foobar*" {
redirect "20070221+rejects@f.q.d.n.com";
}
If I send a message with subject "foobar", I get a "5.5.1 bounce (unknown or illegal alias: 20070221+rejects@f.q.d.n.com" type of error.
If I change the redirect to:
redirect "20070221+<rejects@f.q.d.n.com>"
the messages goes in to the standard rejects INBOX.
The 20070221 folder exists and has a store.idx file in it.
So, the syntax of "20070221+rejects@f.q.d.n.com" bounces and adding the angle brackets around the address just lets the message to to the regular INBOX, not the specified folder. So, I'm missing something in the correctness of the syntax for the rule.
Secondly, I'm trying to create a new folder with the form of the command "mboxutil -c user/Rejects/20070224" and it is about two hours so far and the prompt is not coming back. I can see the mboxutil process still there. An "lsof" of the mboxutil process id shows about 55 lines of open files and libraries.
I can create a similar folder in my personal mailbox directory and I get the prompt back within a few seconds. Does the folder creation depend on how big the account directory structure is? This "rejects" account has over 34 Gig in it, and a single folder may have up to about 2 Gig in it, with one to one and a half gig being about normal. The store.idx file is about 370 Meg.
How patient should I be in waiting for the folder creation? Or should I restart some process?
Hal
# 7
Hi,
> If I send a message with subject "foobar", I get a
> "5.5.1 bounce (unknown or illegal alias:
> 20070221+rejects@f.q.d.n.com" type of error.
My apologies, the format is:
<username>+<foldername>@<domain>
e.g. rejects+20070221@f.q.d.n.com
In testing I just found a flaw with this approach.
To enable direct delivery to a folder in 5.X you need to add this permission manually (go into messenger express, folders tab, select folder, click share button, click the enable direct delivery). In 6.X you can modify the permissions on the command-line (e.g. "./readership -s user/rejects/20070221 anyone p" allows direct-posting).
My original statement of the auto-creation of folders was related to the fileinto sieve action, which is not being used in this case.
The original approach can still work, but it needs two steps, one to tag the emails being redirected to Rejects account as needing to be filtered (e.g. add [SPAM] to the front of the subject), and the other, adding a fileinto rule to the Rejects account (mailsieverulesource:) which triggers off this flag.
This approach has the advantage of not even needing to restart the MTA since you are now changing a directory attribute instead - the change will become effective once the cache has expired for the account (by default 15 minutes).
> Secondly, I'm trying to create a new folder with the
> form of the command "mboxutil -c
> user/Rejects/20070224" and it is about two hours so
> far and the prompt is not coming back. I can see the
> mboxutil process still there. An "lsof" of the
> mboxutil process id shows about 55 lines of open
> files and libraries.
Sounds to me like there is a database lock on the folder. Are you able to run a full reconstruct of the account (reconstruct -r user/Rejects) and not have it hang as well?
> I can create a similar folder in my personal mailbox
> directory and I get the prompt back within a few
> seconds. Does the folder creation depend on how big
> the account directory structure is?
No.
> This "rejects"
> account has over 34 Gig in it, and a single folder
> may have up to about 2 Gig in it, with one to one and
> a half gig being about normal. The store.idx file is
> about 370 Meg.
Hmm, 34 gig seems to be a LOT of email for a single account, you may want to consider adding some kind of expiration rule to the account (e.g. remove any emails older then 30 days)
> How patient should I be in waiting for the folder
> creation? Or should I restart some process?
You may need to restart the entire store (./stop-msg;./start-msg) to clear the lock.
Regards,
Shane.
# 8
Shane,
Gee, I've written this reply twice now and both times I've been timed out because of needing to go pay attention to something else. Ok, the third time is a charm :-).
With your advice, I think I have a working script that will do what I need. I'll be using the method of creating a new folder with mboxutil -c and then moving the imta.filter out of the way while I move the INBOX [0-9][0-9] folders in to the new directory, then do the reconstructs to INBOX and the new folder. More comments in-line below.
>
> To enable direct delivery to a folder in 5.X you need
> to add this permission manually (go into messenger
> express, folders tab, select folder, click share
> button, click the enable direct delivery). In 6.X you
> can modify the permissions on the command-line (e.g.
> "./readership -s user/rejects/20070221 anyone p"
> allows direct-posting).
Since we want this as a cronjob, we'll pass on this option. When we go to 6.x, then I'll work on the "readership" command line way of doing things.
> My original statement of the auto-creation of folders
> was related to the fileinto sieve action, which is
> not being used in this case.
>
> The original approach can still work, but it needs
> two steps, one to tag the emails being redirected to
> Rejects account as needing to be filtered (e.g. add
> [SPAM] to the front of the subject), and the other,
> adding a fileinto rule to the Rejects account
> (mailsieverulesource:) which triggers off this flag.
I'll have to read up on the "fileinto" sieve command. "redirect" seemed intuitive to me when I first started the filter a few years ago.
> Sounds to me like there is a database lock on the
> folder. Are you able to run a full reconstruct of the
> account (reconstruct -r user/Rejects) and not have it
> hang as well?
Since Rejects is so large, I chose not to reconstruct the whole account.
> I can create a similar folder in my personal
> mailbox
> directory and I get the prompt back within a few
> seconds. Does the folder creation depend on how
> big
> the account directory structure is?
>
> No.
This is good to know. I'll expect the directory creation to not take too long. Otherwise I'll go the start/stop msg routine.
> Hmm, 34 gig seems to be a LOT of email for a single
> account, you may want to consider adding some kind of
> expiration rule to the account (e.g. remove any
> emails older then 30 days)
Yes it is big. I'm working on a part of the script that will toss folders over 62 days old. We need to keep things a couple of months in case someone didn't receive a message because of false positives.
> > How patient should I be in waiting for the folder
> > creation? Or should I restart some process?
>
> You may need to restart the entire store
> (./stop-msg;./start-msg) to clear the lock.
Would "./stop-msg store" then "./start-msg store" clear the lock? There are some who will complain if the imap process is not there for even a few seconds (or so it seems :-).
Hal
# 9
Just a quick note -- the mboxutil hung again, so I killed the process (need to do a "-9") and tried the "./stop-msg store; ./start-msg store". Well the POP and IMAP processes were stopped before the STORE daemon. And they were NOT restarted with the "./start-msg store" command. Sigh. ]
So, the answer is "no", it won't work. Just do the "./stop-msg" and then the "./start-msg".
> Would "./stop-msg store" then "./start-msg store"
> clear the lock? There are some who will complain if
> the imap process is not there for even a few seconds
> (or so it seems :-).
>
Hal
# 10
Yeah, you can start the store without the other processes, but, since the processes like IMAP, MSHTTP, etc all depend on the store, "stop-msg store" will stop everything. That's the only way to clear an orphan database lock.
# 11
I just got asked the question -- "What is a database lock?" (and an "orphaned" one) and I don't know the answer. Which database and how and why does it lock?Hal
# 12
Ok, here goes. . .
If you examine the .../store/mboxlist directory, you'll see several files that end in ".db". That's the database. Messaging Server maintains a database of users' mailboxes, folders, subscriptions, shares, and such, for rapid access.
All of the store accessing processes are allowed to read from that database in order to function properly. That's IMAP, HTTP, POP, and ims-master. When they need to read from the database, the "row" where the data is located is "locked" so that no other process can read or change the data while it's being read. When the read is completed, then the lock is released.
If one of the processes crashes, or is killed, it can leave such locks on the database, preventing other processes from reading or writing to the database. The only way to release such an "orphan lock", is to stop and restart the entire Messaging Server. Stored on startup removes these locks.
Stored itself is the only process allowed to write to the database. All other processes that need to change data write that data to one of the "transaction logs", the files in that mboxlist directory where the name starts with "log". If stored cannot write, due to an orphan lock, then these transaction logs start to build up, and the access processes cache that data. You can run out of disk and memory, if you don't notice and take action.
Under normal operation, you should see one or at most two transaction logs. While running reconstructs, you may see more. Doing extensive user modification, migrations, and such, you may see more. That's normal.
better?
# 13
Jay,Yes, that was very informative. Thank you for the thorough and quick response.Hal
