Managing user INBOX

Hi,

> I'm managing a SUN messaging server farm. Lately,

> many users have been complaining about not being able

> to access their INBOX(es) through their full mail

> client. When attempted to do so, they receive a

> "System I/O error" message.

You should also be seeing similar errors in the imap logs. That will give you a guide as to which users are experiencing the errors.

> I am feeling very uneasy about it. Also,

> the problem is intermittent; sometimes clients are

> able to gain access and sometimes they cannot. It

> was suggested somewhere that when the number of

> messages in a user's INBOX get too large, the

> store.idx file gets corrupted.

There is currently a hard limit on the size of the store.idx file of 2GB. This will prevent new emails from being received rather then access being denied. To get a store.idx file up to 2GB would require a user to have several 100,000's of emails (no exact number available as it depends on the size of email headers etc.)

> And since we don't

> have a quota policy, the inboxes can get to be very

> large. So then my questions are:

Quota policies are good things to have :) But I will leave that as an exercise for the user.

> . Where can I find documentation relating to this

> "system I/O error"?

System I/O error can mean a number of things. It can be due being unable to read a file from disk e.g. disk corruption, SAN connectivity issues, or a problem memory-mapping a file etc. Basically when the messaging server application asked the OS for a file/file-operation it was denied.

> 2. Is the suggestion about the INBOX size correct? if

> not, then please advise.

Yes and no.. I will explain more later.

> 4. What is the exact number of messages the INBOX (or

> system folders) can support without running into any

> problems?

Explained earlier.. there is no EXACT number of messages, but if the user has < 100,000 emails in a folder as a rough guide then the 2GB limit on store.idx won't be hit. If a user has this many email in a single folder, get them to move them to another folder... problem solved.

> Moreover, many senior executives at my company are

> experiencing this problem with their mail client. I

> recommended that they move messages out of their

> INBOX(es) in order to reduce its size. However, not

> all are willing and neither do they have to time to

> perform such action given their busy schedule. I

> wanted to work on the backend and move messages

> around manually out of the inbox to another folder.

> For example,

Ahh... the old battling with senior executives bit. Quite possibly the worst aspect of maintaining email systems. I hit this EXACT same issue (senior execs and all) so I can sympathise.

> lease advise. A prompt response will be very

> appreciated.

Based on the symptoms you described the problem is quite easy to solve.

Explanation: What you are hitting is a limit in the size of the IMAP process. For a 32 bit process you can only use ~3GB (I don't remember the exact number) which limits the amount of space that can be used for memory-mapping files. Every time a user clicks on a folder, messaging server memory-maps the store.idx file, so over time the larger these files get (more email in folders/more simultaneous connections) the larger the size of the imap process gets.

Eventually the process hits to 3GB limit. The next user to click on a folder (usually the inbox as it traditionally has the most emails hence the largest store.idx, and usually the 'power-users' for the same reason), gets the system i/o error as the OS was unable to memory-map the index file.

This is why the problem happens intermittently.. most likely during the peak usage periods and most likely due to the power-users first/most often for reasons mentioned. This is how the number of messages/folder translates back to this problem.

Solution: Increase the number of imap processes and reduce the number of connections per/process to distribution imap connections across more processes.

e.g.

Say you have the following set:

service.imap.maxsessions = 3000

service.imap.numprocesses = 2

(so you have a maximum number of 2x3000=6000 simultaneous connections)

You should increase these to

service.imap.maxsessions = 1500

service.imap.numprocesses = 4

(so you have a maximum number of 4x1500=6000 simultaneous connections)

Then run:

./stop-msg imap;./start-msg imap

Hope this helps,

Shane.

p.s. before anybody asks, yes this particular problem will go away with 64bit version of messaging server, and yes Sun is working on this for the next release and no I can't comment on when it will be available but odds on it will be before the end of this year.

shane_hjortha at 2007-7-8 2:30:25 >

Hi,

> I'm sure the original spec was implemented for

> optimal performance, but I can't help but wonder,

> should those same configurations (maxsessions and

> maxprocesses) have been configured on the MMPs too?

The MMP's don't have these same options. Let's deal with this issue one system at a time.

> What exactly is the behavior of an imap session

> (stateless or persistent or else) in the Messaging

> Server?

I'm not sure exactly what you mean by this. IMAP sessions are connection-oriented. So an email client establishes a TCP connection, logs in using IMAP protocol and that connection _can_ remain open indefinitely, or it can close it once it is finished checking folders, retrieving emails etc. This is very much client dependant. Therefore messaging server needs to keep track of these connections for the life of the connection (till the TCP connection is dropped).

> Through my investigation and research, I know that

> each open folder window uses one imap connection, and

> one imap connection is always kept cache for the

> inbox.

This is consistent for say something like Mozilla Thunderbird. Other clients may act differently.

> So, for a company with approximately 7000 active

> users, does that mean there will be 7000 persistent

> imap connections at any single day?

There are tools to check this (refer to global.numcurrentconnections):

bash-2.05# cd /usr/iplanet/server5/bin/msg/admin/bin

bash-2.05# ./counterutil -o imapstat

Monitor counteroobject (imapstat)

registry /usr/iplanet/server5/msg-mumble/counter/counter opened

counterobject imapstat opened

count = 1 at 1168469650 rh = 0xc55d8 oh = 0xc55b0

global.currentstarttime [4 bytes]: 19/Dec/2006:13:48:21 +1100

global.lastconnectiontime [4 bytes]: 11/Jan/2007:09:51:04 +1100

global.maxconnections [4 bytes]: 6

global.numconnections [4 bytes]: 3293

global.numcurrentconnections [4 bytes]: 0

global.numfailedconnections [4 bytes]: 0

global.numfailedlogins [4 bytes]: 0

global.numgoodlogins [4 bytes]: 4

> Say MSG Server 1 can have a maximum number of 16,000

> simultaneous connections (=8x2000).

>

> Say MSG Server 2 can have a maximum number of 16,000

> simultaneous connections (=4x4000).

Any particular reason why these numbers are different (e.g. different hardware?). A rule of thumb used to be 1 process per processor. This would reduce the swapping of processes off processors.

> Adding the 7000 connections for the inbox and the

> estimated 35,000 gives us 42,000 connections at any

> single time.

Check your actual stats using the command listed earlier, this saves on having to guesstimate.

> If I'm not completely incorrect, is that the source

> of these "system io errors"?

> The MMP logs are reporting errors like the

> following:

These are network related errors, in this case no data was transferred, this can happen if a client connects (TCP) then disconnects.

> ImapProxyAService.cfg

> "session start, client IP 10.32.254.3:4331, server IP

> 10.15.24.249:143

> (sid 0xfc1a14) client socket IO error 131

> (sid 0xfc1a14) 0 C->S bytes, 0 S->C bytes in 0

> seconds

> (sid 0xfc1a14) session end"

Did you check your imap logs as I noted earlier and check to see if it listed the I/O error?

Also check the current size of your imap processes to see whether they are hitting the 3GB threshold:

ps -ef -o pid,args,vsz,rss | grep imapd | grep -v grep

Regards,

Shane.

shane_hjortha at 2007-7-8 2:30:25 >

The total size of the process, resident and what's swapped out, is what we're talking about, here.

The process doesn't care what is resident and what's swapped out. "virtual memory" is all the same as real memory to the process.

I suggest that you go from 4 imap processes to 8, and cut your maximum sessions in half.

jay_plesseta at 2007-7-8 2:30:25 >

Hi All.

No problem Shane, I understand. You guys have been very helpful. Thank you for that the continued advice and assistance.

To touch base on the imap processes size issue:

"As you can see from the sizes though, your processes are indeed hitting the 3GB limit (~3000MB) so the solution is very straight-forward as outlined in previous posts by Jay and myself."

The aforementioned imap sizes are the size of the imap processes on my mail stores. I understand that changes need to be made there, as soon as I can convince the implementation team to do so. But first I have to prove to them that it's a change worthy of their efforts. They're avoiding the leaping before looking process, one of which I too willingly follow myself. And hence my due diligence persists.

I further investigated the messaging server implementation and, recently, I noticed a large number of the problematic users resides on a mail store that doesn't have the imap processes size issue; in fact, that server machine has twice the number of imap processes -mentioned before in this thread- running with an average size of 1.5G per imap process. That fact raised a red flag for me.

So then that led me to look at the mta processes running on my MMP machines. There was no surprise there.

Below is the imap processes running on my MMPs:

MMP 1: 49M 6576K imapd

MMp 2: 49M 4968K imapd

Using round-robin, these MMPs relay messages to the mailstores onto which a particular user mailbox resides.

As mentioned above, how can I explain the uneasy fact that many of the power-users live on a mail store whose imap processes doesn't come near to hitting the 3G size limit?

Once again, many thanks in advance for your continued assistance.

Regards,

Ken

kdavida at 2007-7-8 2:30:25 >

Hi,

> The aforementioned imap sizes are the size of the

> imap processes on my mail stores. I understand that

> changes need to be made there, as soon as I can

> convince the implementation team to do so.

I should point out that the change is straight-forward to implement and will result in minimal service outage to implement and to backout. You only need to restart the imap service as well (so webmail/store/pop can remain active) using the ./stop-msg imap; ./start-msg imap command. If the users are connecting via MMP's then they may not even realise there has been an outage (since the TCP connection to the MMP host will remain active).

> But first

> I have to prove to them that it's a change worthy of

> their efforts. They're avoiding the leaping before

> looking process, one of which I too willingly follow

> myself. And hence my due diligence persists.

If you wanted to follow proper due diligence, I would recommend in future logging a Sun support case at the same time (although as Jay noted already, it is possible you could get Jay as the contact person anyway).

> I further investigated the messaging server

> implementation and, recently, I noticed a large

> number of the problematic users resides on a mail

> store that doesn't have the imap processes size

> issue; in fact, that server machine has twice the

> number of imap processes -mentioned before in this

> thread- running with an average size of 1.5G per imap

> process. That fact raised a red flag for me.

Either you are reading your information incorrectly OR you have two issues. You are definitely hitting the 3GB limit -- so if users reside on a *mailhost* which doesn't have processes that large AND are receiving the *exact* same errors, then you should investigate this separately.

> So then that led me to look at the mta processes

> running on my MMP machines. There was no surprise

> there.

>

> Below is the imap processes running on my MMPs:

>

> MMP 1: 49M 6576K imapd

> MMp 2: 49M 4968K imapd

>

> Using round-robin, these MMPs relay messages to the

> mailstores onto which a particular user mailbox

> resides.

I think you are mixing terms here.

-> MTA, mail transfer agent - these are responsible for the system level *delivery* of email and include things like the job_controller/smtp_client type processes.

-> MMP, messaging multiplexor - responsible for accepting and channelling client connections to the appropriate store - runs as the AService.rc process for both IMAP/POP/SMTP multiplexing.

-> Mailstore processes - responsible for providing IMAP/POP/Webmail access to delivered emails - includes processes such as imapd/mshttpd/popd

Now an MMP host does NOT use the imapd process - that is why the imapd process in this case is very small. The MMP software runs as the AService.rc process.

So you end up with something like this in the typical deployment:

(email access - POP/IMAP)

Client system (outlook/thunderbird/whatever) -> MMP system (Aservice.rc process) -> Mailstore system (imapd process)

(email delivery - SMTP)

Client system (outlook/thunderibrd/whatever) OR MTA (hotmail.com/gmail.com/local server) -> MTA system (dispatcher/job_controller/smtp_client/smtp_server) -> Mailstore system (dispatcher/job_controller/smtp_client/smtp_server)

> As mentioned above, how can I explain the uneasy fact

> that many of the power-users live on a mail store

> whose imap processes doesn't come near to hitting the

> 3G size limit?

Already discussed. I think you are mistaking the imapd processes on the MMP/MTA system as being active, in reality an MMP/MTA ONLY system doesn't require the imapd/popd/stored/mshttpd processes to be running at all (and they can be disabled to reduce RAM footprint).

Regards,

Shane.

shane_hjortha at 2007-7-8 2:30:25 >