Crash occurred in NSAPI SAF send-cgi

Hi,

We are getting Crash every couple of days. But could not find why this should happen. Here is the error log and the pstack info. If anyone has idea, please let me know.

We have recently upgraded to sun servers6.1sp6 and solaris 10.

thanks,

log;

[20/Mar/2007:12:01:56] catastrophe (11443): CORE3260: Server crash detected (signal SIGSEGV)

[20/Mar/2007:12:01:56] info (11443): CORE3261: Crash occurred in NSAPI SAF send-cgi

[20/Mar/2007:12:01:56] info (11443): CORE3262: Crash occurred in function INTnetbuf_next from module /usr/iplanet/servers6.1s

p6/bin/https/lib/libns-httpd40.so

[20/Mar/2007:12:02:08] failure (11442): CORE3107: Child process closed admin channel

[20/Mar/2007:12:02:08] info (11160): CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.4.2_05] from [Sun Microsystems In

c.]

pstack:

core 'core.icorefep1.webservd.3205.1010.1171723612.2319' of 2319:webservd -r /usr/iplanet/servers6.1sp6 -d /usr/iplanet/servers6.1sp6/h

pstack: warning: librtld_db failed to initialize; symbols from shared libraries will not be available

-- lwp# 1 --

fe0c0e70 ? (0, 0, ffbff6e8, 0, 0, 3e8)

-- lwp# 4 --

fe0c0e70 ? (fd9fadb8, 1, fd9fad38, 0, 0, 1388)

-- lwp# 5 --

fe0c0f84 ? (d, fcffaedd, feb, 0, 2c00, 0)

-- lwp# 6 --

fe0c0144 ? (27c90, 4a970, fcefbcc8, 1, 0, 0)

-- lwp# 7 --

fe0c0528 ? (fcdfbeb8, 0, 0, fd7e0800, 0, 1)

-- lwp# 8 --

fe0c0144 ? (28668, 66ebf0, 0, 0, 0, 0)

-- lwp# 9 --

fe0c0e70 ? (0, 0, fcbfbea8, 0, 0, 0)

-- lwp# 11 --

fe0c0144 ? (286b8, 66ee98, 0, 0, 0, 0)

-- lwp# 12 --

fe0c0144 ? (7c1390, 7af2e8, fc9fbc90, 1, 0, 0)

-- lwp# 13 --

fe0c1378 ? (88490, 88478, fa67fd30, 3b9ac800, 1d8dbc20, 16)

-- lwp# 14 --

fe0c1378 ? (8c150, 8c138, f1430, 0, 0, 0)

-- lwp# 15 --

fe0c1378 ? (8c2b8, 8c2a0, f1638, 0, 0, f9e7f8a4)

-- lwp# 16 --

fe0c1378 ? (87c20, 87c08, ff000000, ff000000, 0, 0)

-- lwp# 18 --

fe0c1378 ? (88a30, 88a18, f1a48, 0, 4b8110, f7a7fd50)

-- lwp# 19 --

fe0c1378 ? (88ac0, 88aa8, f1c50, 0, 4b8110, 0)

fd194078 ? (88a60, 0, 8cd50, 4000, 4194, f1c50)

fd2aac7c ? (fd5358a9, fd5c1238, f1c50, 12c1d0, 31a08c, fd267ed8)

fd267f00 ? (f1c50, 13, 40, 0, 40, 1)

fd2643e0 ? (f1c50, e3780000, 0, 0, fd2642ac, 1)

fe0c00a0 ? (0, 0, 0, 0, 0, 0)

-- lwp# 20 --

fe0c1378 ? (88ac0, 88aa8, edb50, 0, 4b8110, 0)

fd194078 ? (88a60, 0, 8d020, 4000, 4194, edb50)

fd2aac7c ? (fd5358a9, fd5c1238, edb50, 12cde8, 31a08c, fd267ed8)

fd267f00 ? (edb50, 14, 40, 0, 40, 1)

fd2643e0 ? (edb50, e3500000, 0, 0, fd2642ac, 1)

fe0c00a0 ? (0, 0, 0, 0, 0, 0)

-- lwp# 22 --

fe0c0e70 ? (e2d7f750, 0, e2d7f7b8, 0, 0, 0)

fe061f24 ? (e2d7f750, fe0eba68, fe0eba68, 0, e2d7f7b8, 0)

fe06229c ? (0, 0, 0, 0, e2d7f840, e2d7f840)

fd10651c ? (0, ea60, 1, f7c160d0, c, 0)

fd10633c ? (f2880, 0, ea60, 1, 4, 0)

fd293258 ? (f2914, e2d7f9a8, 0, ea60, e8dfbf78, 0)

f7c0b96c ? (e2d7fa2c, b8, 0, 3c, 0, 0)

f7c05804 ? (e2d7faac, b7, 0, f7c16284, 8, e2d7f9c0)

f7c05804 ? (e2d7fb24, f38212b8, 0, f7c160d0, 4, e2d7fa50)

f7c05a8c ? (e2d7fb9c, 0, 0, f7c16420, 4, e2d7fac8)

f7c0010c ? (e2d7fc28, e2d7fe90, a, f3821ca0, 4, e2d7fb40)

fd15bcf8 ? (e2d7fe88, e2d7fcf0, e2d7fda8, f2880, f2880, e2d7fd00)

fd24a3e4 ? (fd582000, ab7120, e2d7fd9c, e2d7fd98, e2d7fda8, f2880)

fd25d5a8 ? (e2d7fe88, e2d7fe84, e2d7fe7c, e2d7fe74, e2d7fe6c, f2880)

fd26e8f8 ? (f2880, f2880, 30298, ab7120, 31a08c, fd267ed8)

fd267f00 ? (f2880, 16, 40, 0, 40, 1)

fd2643e0 ? (f2880, e2d80000, 0, 0, fd2642ac, 1)

fe0c00a0 ? (0, 0, 0, 0, 0, 0)

-- lwp# 23 --

fe0c0e70 ? (e1e7f750, 0, e1e7f7b8, 0, 0, 0)

fe061f24 ? (e1e7f750, fe0eba68, fe0eba68, 0, e1e7f7b8, 0)

fe06229c ? (0, 0, 0, 0, e1e7f840, e1e7f840)

fd10651c ? (0, ea60, 1, f7c160d0, c, 0)

fd10633c ? (f2c90, 0, ea60, 1, 4, 0)

fd293258 ? (f2d24, e1e7f9a8, 0, ea60, e8e053b0, 0)

f7c0b96c ? (e1e7fa2c, b8, 0, 3c, 0, 0)

f7c05804 ? (e1e7faac, b7, 0, f7c16284, 8, e1e7f9c0)

f7c05804 ? (e1e7fb24, f38212b8, 0, f7c160d0, 4, e1e7fa50)

f7c05a8c ? (e1e7fb9c, 0, 0, f7c163d0, 4, e1e7fac8)

f7c0010c ? (e1e7fc28, e1e7fe90, a, f3821ca0, 4, e1e7fb40)

fd15bcf8 ? (e1e7fe88, e1e7fcf0, e1e7fda8, f2c90, f2c90, e1e7fd00)

fd24a3e4 ? (fd582000, 8e3120, e1e7fd9c, e1e7fd98, e1e7fda8, f2c90)

fd25d5a8 ? (e1e7fe88, e1e7fe84, e1e7fe7c, e1e7fe74, e1e7fe6c, f2c90)

fd26e8f8 ? (f2c90, f2c90, 30608, 8e3120, 31a08c, fd267ed8)

fd267f00 ? (f2c90, 17, 40, 0, 40, 1)

fd2643e0 ? (f2c90, e1e80000, 0, 0, fd2642ac, 1)

fe0c00a0 ? (0, 0, 0, 0, 0, 0)

-- lwp# 76 --

fe0c1294 ? (6, 0, fe0a4ba8, ffffffff, fe0e8284, 6)

fe040038 ? (0, 1, fe0b579c, a8350, fe0eb298, 0)

fd499d98 ? (1, fd53ad22, e1d7f668, fd56f8a2, fd56f83a, 14)

fd37d35c ? (e1d7f678, fd5d8b3c, fd53aaec, fd546bb3, fd53ab90, fd546b70)

fd2f6fa4 ? (8f, 5000, 0, ff000000, 0, 48a284)

fd2c0188 ? (8f, fd5c364c, 20361e8, e1d7f590, e1d7f568, 1)

fd83e174 ? (203627c, e1d7f918, 8f, ffffffff, f7c162e4, 0)

f7c0b96c ? (e1d7f994, b8, 0, f7c162a0, 0, e1d7f898)

f7c05804 ? (e1d7fa2c, b8, 0, f7c16298, 4, e1d7f930)

f7c05804 ? (e1d7faac, f38b3ba0, 0, f7c16298, 4, e1d7f9c8)

f7c05a8c ? (e1d7fb24, f38212b8, 0, f7c16420, 8, e1d7fa48)

f7c05a8c ? (e1d7fb9c, 0, 0, f7c163d0, 4, e1d7fac8)

f7c0010c ? (e1d7fc28, e1d7fe90, a, f3821ca0, 4, e1d7fb40)

fd15bcf8 ? (e1d7fe88, e1d7fcf0, e1d7fda8, 20361e8, 20361e8, e1d7fd00)

fd24a3e4 ? (fd582000, ab6910, e1d7fd9c, e1d7fd98, e1d7fda8, 20361e8)

fd25d5a8 ? (e1d7fe88, e1d7fe84, e1d7fe7c, e1d7fe74, e1d7fe6c, 20361e8)

fd26e8f8 ? (20361e8, 20361e8, 309c8, ab6910, 31a08c, fd267ed8)

fd267f00 ? (20361e8, 4c, 7f, 0, 7f, 1)

fd2643e0 ? (20361e8, e1d80000, 0, 0, fd2642ac, 1)

fe0c00a0 ? (0, 0, 0, 0, 0, 0)

-- lwp# 25 --

fe0c0e70 ? (e1cfbc88, 1, e1cfbc08, 0, 0, 1388)

fe05cf48 ? (e1cfbc88, 1, 1388, 10624c00, 0, 0)

fedd8534 ? (e1cfbcf4, 1, e1cfbd24, e1cfbc88, 20, ffffffff)

fedd9dd8 ? (30748, e1cfbe58, ffffffff, 12, 0, ffffffff)

ff22a3ac ? (f42e8, e1cfbe58, ffffffff, ff000000, 0, 0)

ff220fc0 ? (108de50, 7, 8, 0, 1000, ffffe800)

ff106d68 ? (108de50, 78f110, 0, 0, 0, 1)

fedddf2c ? (78f110, ff106d44, 400, fedf6a98, 0, fedf87dc)

fe0c00a0 ? (0, 0, 0, 0, 0, 0)

-- lwp# 26 --

fe0c0144 ? (262a0, 498f8, e1bfbd28, 1, 0, 0)

fe0ba1d4 ? (262a0, 498f8, e1bfbd28, 0, 0, 0)

fe0ba64c ? (262a0, 498f8, e1bfbd28, 0, 0, 0)

fe0ba7e4 ? (262a0, 498f8, e1bfbeb8, ff000000, 0, 0)

fe0ba8d8 ? (262a0, 498f8, e1bfbeb8, 0, 0, c350)

fe0ba918 ? (262a0, 498f8, e1bfbeb8, 45d7155d, 0, 3618adb8)

fedd79fc ? (26298, 2, 78faa0, 0, 3618adb8, 0)

ff23b82c ? (2e58, 0, 0, ff2ef26c, ff2ef274, ff2ef260)

fedddf2c ? (78faa0, ff23b498, 400, fedf6a98, 0, fedf87dc)

fe0c00a0 ? (0, 0, 0, 0, 0, 0)

[7295 byte] By [indian_511a] at [2007-11-26 22:20:26]
# 1

Unfortunately, we can't tell much from that pstack output because it doesn't include symbols:pstack: warning: librtld_db failed to initialize; symbols from shared libraries will not be availableI don't know pstack couldn't initialize librtld_db. How big is the core file? Is it possible that it was truncated due to a lack of disk space or ulimit? Did you run pstack on the same system that generated the core file?

While I don't remember seeing a bug like this before, have you tried applying the latest service pack? For Web Server 6.1, that's 6.1SP7. Service packs contain important bug fixes.

elvinga at 2007-7-10 11:17:23 > top of Java-index,Web & Directory Servers,Web Servers...
# 2
Yes, I ran the pstack on the same machine where crash occured.We have recently upgrated to 6.1 server, so I hope that all the patches are applied. I will check with the admin.
indian_511a at 2007-7-10 11:17:23 > top of Java-index,Web & Directory Servers,Web Servers...