a user complained to me that his sent mail before a certain date had gone missing.
this happens to coincide with the date i ran an imsbackup - which backs up to an nfs mount (that has been acting weird lately -not sure if this is relevant)
anyway - what should i do - mboxutil, recontruct? should i look on the filesystem for the files. can i run these commands only his mailbox only?
thanks for any detailed info.
s7
Hi,
The first thing is to never take the users word for granted. Try proxyauth'ing their account using WebMail and seeing if the messages are in their Sent Folder (eliminate corruption at the email client end). If you haven't got proxyauth set up.. contact the customer and get them to check.
If the messages don't appear, run:
reconstruct -r user/<uid>
If for some reason the messages have disappeared from the store.idx file, it will be fixed and an error message will be provided.
If there is no error(s) mentioned after the reconstruct, you may have to restore the messages from backup (assuming they are there, it may be the messages were never there in the first place e.g. they are stored on a 'local folder').
I should note that in 4 years of running a 90K+ install, there were no cases of 'magically' disappearing emails that didn't turn out to be either:
1. Customer had moved/deleted emails themselves (either on purpose or by accident)
2. Filesystem level corruption
Hope this helps,
Shane.
Shane - Thanks for the reply.
I tried a recontruct -r -f (for the whole store) and then for this particular user - the messages did not re-appear. i can login to his account w/ his passwd.
i didn't see the messages anywhere.
i will try to restore from a backup. which as i mentioned was made using imsbackup to an nfs mount - that has become troublesome. weird RPC read errors - e.g. similar to --> NFS readdir+ failed for server xyz: error 2 (RPC: Can't decode result)
hopefully i'll be able to get the imsbackup file off of there, hopefully it's not damaged. in any event - is there a way to imsrestore only a particular folder for a user, rahter than the user's whole folder set?
could runnung imsbackup to this troublesome nfs device have caused this?
e.g. his messages to dissapear - it does coincide with when i ran it. he says he has no more sent messages (from before the day i ran it).
thanks,
s7
Hi,
Firstly, you shouldn't run a reconstruct -r -f unless it is really necessary (takes a lot of time depending on the size of your mail-store, makes a lot of unnecessary changes as it rebuilds all the store.idx files, and is very rarely required).
If the messages didn't re-appear then chances are they were moved by the user.
It is extremely unlikely that the imsbackup had anything to do with the messages disappearing. On the NFS front, sorry can't help you there. It sounds like it could be a lower-level NFS issue.
You can restore just the Sent messages in question by running something like:
bash-2.05# ./imsrestore -f /tmp/backup.bak /primary/user/user001/Sent
--
Tape Version : 2
Backup Date: Thu, 31 Aug 2006 14:43:37
Message Store : myhost.com
Block factor : 20
--
/primary/groupB/user001/Sent
/primary/groupB/user001/Sent/7 restoring...
/primary/groupB/user001/Sent/8 restoring...
/primary/groupB/user001/Sent/9 restoring...
/primary/groupB/user001/Sent/10 restoring...
user001 == uid for your user
primary == partition for your user
Shane.
Guys, reconstruct -r globally shouldn't cause any real problems.Even -r -f shouldn't, as the data that's there is used. However, it was an un-needed use of cpu time.
I would look at the disk itself, and see if messages are indeed there, or not.
I actually have seen messasges "dissappear" from a mailstore. Always, however, there is a reason. Such reasons include:
1. Manual intervention. Removal of message files can cause store.idx to become out of sync with the store, which can cause the "pre-deleted" issue, where new messages arrive, and are immediately marked, "deleted".
2. Disk problems. I've seen whole directories become files (say, "=+sent" directory, becomes a file, instead of a directory. This is a disk/OS problem, and will cause the mail server to be unable to find any such messages.
3, and most common.User moves or deletes files and blames the system.
NFS for backup should be ok, imsbackup doesn't have any of the dependancies that the store has on "atomic writes". If you're getting errors, it's possible that you have some problem with either the backup file itself, or your command line.
Remember, if your backup is larger than 2 gig, you should be using the -f - (dash f space dash" syntax.
Thank you both for all the info.
Good to know I didn't hurt anything with the reconstruct -r -f
you said when applied globally - if applied to a single user is it then problematic?
to look at the disk - what and where should i look? how can i find the user's folder easily? they seem to be burried in lots of numeric subfolders ...
how might i know if my disks are having trouble? iostat? ?
it may have been the user, but he said no. and it did coincide w/ my imsbackup ...
i didn't manipulate the store files.
I was able to copy the backup off this old Snap appliance over to a much newer Snap appliance. I was then able to imsrestore the messages in the folder he said were gone.
For some reason on the older device, Solaris9 got RPC can't decode errors when trying to imsrestore - but not on the new - I tried forcing older versions of nfs, different protos, etc. but in the end it became easier to copy the file using windows to the new server, and then set up the new mount.
so far, it's only been this one user, and it did coincide w/ when i ran imsbackup (to the old/bad nfs mount -in any event, it apparently wrote the backup file fine).
i am using:
/opt/SUNWmsgsr/sbin/imsbackup -f- / > /nfs-back/
w/o the space following the f ...
suppose i better change that. my backup file is 1.5gb.
> Thank you both for all the info.
>
> Good to know I didn't hurt anything with the
> reconstruct -r -f
> you said when applied globally - if applied to a
> single user is it then problematic?
Nope, not a problem. That's why we offer the ability.
>
> to look at the disk - what and where should i look?
> how can i find the user's folder easily? they seem
> to be burried in lots of numeric subfolders ...
Yes, the user's mailbox is buried.
we provide a "hashdir" utility to tell you where to look.
hashdir uid
will give the number subdirs to the user's mailbox. It is case sensitive.
>
> how might i know if my disks are having trouble?
> iostat? ?
var/adm/messages
>
> it may have been the user, but he said no. and it did
> coincide w/ my imsbackup ...
I can't imagine any mechanism where an imsbackup would remove messages from the store.
>
> i didn't manipulate the store files.
>
> I was able to copy the backup off this old Snap
> appliance over to a much newer Snap appliance. I was
> then able to imsrestore the messages in the folder he
> said were gone.
good
>
> For some reason on the older device, Solaris9 got RPC
> can't decode errors when trying to imsrestore - but
> not on the new - I tried forcing older versions of
> nfs, different protos, etc. but in the end it became
> easier to copy the file using windows to the new
> server, and then set up the new mount.
>
> so far, it's only been this one user, and it did
> coincide w/ when i ran imsbackup (to the old/bad nfs
> mount -in any event, it apparently wrote the backup
> file fine).
That's a very good thing. . .
>
> i am using:
> /opt/SUNWmsgsr/sbin/imsbackup -f- / > /nfs-back/
>
> w/o the space following the f ...
>
> suppose i better change that. my backup file is 1.5gb.
Hi,
I should probably just clarify that reconstruct -r -f shouldn't hurt anything but you are forceably changing/refreshing store.idx files for all accounts on the system which given that this problem appears to be isolated to one user, is a bit of overkill.
Where this does become an issue is when you have a massive mailstore (millions of files, TB of data) and it can take many hours to do a full-forced recontruct and will cause tremendous I/O on the system. So its just good practice not to run a full reconstruct over the entire mail-store when a reconstruct against a user/partition will do.
Using the reconstruct -r will only update the store.idx file when there are inconsistencies between the store.idx and the files on disk (e.g. missing .msg files, additional .msg files, bad permission .msg files, incorrectly sized .msg files).
If the customer reports inconsistencies with the actual contents of a message (e.g. header information) then a reconstruct -r -f will cause the header information to be re-read into the store.idx files.
For the NFS issues, nfsstat has information on NFS connectivity, but ideally you may need to look at the logs on the appliance - contact the appliance vendor for help.
To get access to a customers mailbox on disk, I usually create a script (you will need to change paths to reflect your environment):
++getdir.sh++
#!/bin/bash
export PATH=$PATH:/opt/SUNWmsgsr/sbin
echo "/opt/SUNWmsgsr/data/store/partition/*/=user/`hashdir $1`=$1"
Then you just run:
cd `getdir <uid>`
Shane
a user wants back some deleted messages. i have some backups (i do roughly 2/month).
can i restore a user's mailbox over his quota?
will his existing email remain (e.g. not get overwritten?)
should i expand his quota before the restore?
can i restore it to different box?
thanks,
s7
> a user wants back some deleted messages. i have some
> backups (i do roughly 2/month).
>
> can i restore a user's mailbox over his quota?
yes
>
> will his existing email remain (e.g. not get
> overwritten?)
yes
>
> should i expand his quota before the restore?
no need
>
> can i restore it to different box?
yes
>
> thanks,
> s7
