migrating from sendmail inbox to ms6.2 results in double msg download

Hi and thanks in advance for any assistance or insight into our problem...

Mail clients for pop (thunderbird, eudora, outlook all variations...) are downloading messages we migrated even though the messages were already downloaded to the client pre migration.

They were 'left on the server' so were migrated to the ms6.2 infrastructure.

We have verified with comms express that the inbox has no message duplication.

Our goal is to preserve the existing message id's so that pop clients can reliably check to see if they have been downloaded and leave the messages on the server without getting them down again as 'new' mail.

Can this be done easily...or at all?

Here's the background and supporting details. Again, thank you in advance for any input!

Chris.

Details:

Migration is using imsimport:

imsimport -v 1 -u $userid -s /var/mail/$userid -d INBOX

Our hunch on this problem is that when we perform the imsimport, the X-UIDL values

are getting ignored and the ms6.2 system is re-assigning them it's own internal one ,

and hence the pop clients think they haven't seen the message.

Here's a sample header (cleaned a bit to protect the innocent ;) )

- Original Message --

From: - Thu Feb 8 19:10:16 2007

X-Account-Key: account4

X-UIDL: 13-1170696686

X-Mozilla-Status: 0001

X-Mozilla-Status2: 00000000

X-UIDL: -PJ!!^^E"!Jo

Return-Path: <x.y@a.com>

Received: from [x.x.x.x] (a.b.c [y.y.y.y]) by mta02 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTPA id <0JCZ009UEUBJXA20@mta02> for x.y@a.com (ORCPT x.y@a.com); Mon, 05 Feb 2007 09:16:32 -0500 (EST)

Date: Mon, 05 Feb 2007 09:16:32 -0500

From: A B <C.D@test.com>

Subject: email 2

To: test.account@test.com

Message-id: <45C73C40.7030101@test.com>

MIME-version: 1.0

Content-type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary=ms000800070002010904020902

User-Agent: Thunderbird 1.5.0.9 (X11/20070104)

X-Filtered-With: renattach 1.2.0

X-RenAttach-Info: mode=badlist action=rename count=0

Status: RO

Prior to migration, the header X-UIDL was the alphanumeric string.

Post migration another header value was added: X-UIDL: 13-1170696686

So for example, a thunderbird popstate.dat file looks like this (pre migration)

# POP3 State File

# This is a generated file! Do not edit.

*host accountname

k J,$"!m_3"!5fM"!K#e!! 1170685402

k AiJ"!KYL!!MZ*"!NmH"! 1170685402

k -PJ!!^^E"!Jo<!!a-T!! 1170685401

k ~[(!!l*i"!VN+"!@0C!! 1170685401

k %kB"!5;m!!Ma""!4W@!! 1170685402

k ><7]"!BCe!!GH<!![B("! 1170685402

k BQC"!'$-!!$bd"!bR##! 1170685402

Now the popstate.dat file looks like this

k 6-1170696686 1170696990

k 1-1170696686 1170696989

k 3-1170696686 1170696990

k 5-1170696686 1170696990

k 7-1170696686 1170696990

k 8-1170696686 1170696990

k 2-1170696686 1170696989

k 4-1170696686 1170696990

(the header shown above does not corellate with these popstate.dat files by the way).

What happened to all the previous X-UIDLs? ?

We think we should be able to preserve the X-UIDL value from the original imported emails, and hence, prevent mail from being downloaded yet again to clients.

Comments and recommendations on how make this happen are welcome!

Obligatory imsimta version statement:

./imsimta version

Sun Java(tm) System Messaging Server 6.2-3.04 (built Jul 15 2005)

libimta.so 6.2-3.04 (built 01:43:03, Jul 15 2005)

SunOS xxxx 5.10 Generic_118833-03 sun4v sparc SUNW,Sun-Fire-T200>

[3834 byte] By [CP_In_Kingstona] at [2007-11-26 17:51:32]
# 1

Hi,

> Our goal is to preserve the existing message id's so

> that pop clients can reliably check to see if they

> have been downloaded and leave the messages on the

> server without getting them down again as 'new'

> mail.

>

> Can this be done easily...or at all?

At this point, I would say no. The UIDL value is defined by the server, this is a free-form undefined unique value so the format will change depending on the server software implementation.

> Our hunch on this problem is that when we perform the

> imsimport, the X-UIDL values

> are getting ignored

Ignored, or they don't exist at all? Have a look at the message on disk, I think you will find there is no X-UIDL header, and this header is being added by the email client during download based on the response to the UIDL command by the server.

I did a quick check myself, messaging server generates the UIDL based on the message number and the timestamp of the message:

e.g. from thunderbird (view message source)

X-UIDL: 11-1162502237

file this corresponds to (note the 11.msg)

./msg-mumble/store/partition/primary/=user/c4/31/=shjorth/00/11.msg

No idea what the timestamp refers to, but if I had to guess it was the creation time of the INBOX/account (you are welcome to test locally).

bash-2.05# perl -e 'print scalar localtime(1162502237)'

Fri Nov 3 08:17:17 2006

> and the ms6.2 system is

> re-assigning them it's own internal one ,

> and hence the pop clients think they haven't seen the

> message.

A fair conclusion.

> - Original Message --

> From: - Thu Feb 8 19:10:16 2007

> X-Account-Key: account4

> X-UIDL: 13-1170696686

> X-Mozilla-Status: 0001

> X-Mozilla-Status2: 00000000

> X-UIDL: -PJ!!^^E"!Jo

The headers above are added by Mozilla. You will not see them if you view the email headers via IMAP/messenger express for example.

> Prior to migration, the header X-UIDL was the

> alphanumeric string.

Sendmail probably uses a hash of the message to provide a 'unique' component.

> Post migration another header value was added:

> X-UIDL: 13-1170696686

That is the messaging server format, the 13 would refer to the fact that the message on disk is 13.msg.

> We think we should be able to preserve the X-UIDL

> value from the original imported emails, and hence,

> prevent mail from being downloaded yet again to

> clients.

Nope, I think you will have to live with this issue. Something to communicate with your uses during the migration. Fortunately this is a once-off.

> Obligatory imsimta version statement:

> ./imsimta version

> un Java(tm) System Messaging Server 6.2-3.04 (built

> Jul 15 2005)

> libimta.so 6.2-3.04 (built 01:43:03, Jul 15 2005)

> SunOS xxxx 5.10 Generic_118833-03 sun4v sparc

> SUNW,Sun-Fire-T200

You may want to patch this to -58 before going into production.

Regards,

Shane.

shane_hjortha at 2007-7-9 5:04:03 > top of Java-index,E-Mail, Calendar, & Collaboration,Sun Java System Messaging Server...