Kernel panic with ipf and patch 125014-02?
Hi,
After bringing a bunch of Sparc servers running Solaris 10 up to current patch level I've been experiencing strange and unstable behavior. All servers are running with an ipfilter configuration.
One server has been freezing on the network interface at least once every day without any syslog notice. A reboot is the only way up again.
Now today one server (a Internet proxy server) did a kernel panic twice - never seen that before on this server.
It seems like it was caused by the kernel module ipf. And since it has never happend before I guess it could be caused by the IP filter patch 125014-02.
Anyone experiencing something similar and am I on the right track with suspecting this patch to be bad?
Thanks
Kasper
Message from syslog:
Feb 20 14:53:00 ceres unix: [ID 836849 kern.notice]
Feb 20 14:53:00 ceres ^Mpanic[cpu0]/thread=2a10053dcc0:
Feb 20 14:53:00 ceres unix: [ID 340138 kern.notice] BAD TRAP: type=31 rp=2a10053ca70 addr=18 mmu_fsr=0 occurred in module "ipf" due to a NULL pointer dereference
Feb 20 14:53:00 ceres unix: [ID 100000 kern.notice]
Feb 20 14:53:00 ceres unix: [ID 839527 kern.notice] sched:
Feb 20 14:53:00 ceres unix: [ID 520581 kern.notice] trap type = 0x31
Feb 20 14:53:00 ceres unix: [ID 381800 kern.notice] addr=0x18
Feb 20 14:53:00 ceres unix: [ID 101969 kern.notice] pid=0, pc=0x7bb3ad30, sp=0x2a10053c311, tstate=0x80001602, context=0x0
Feb 20 14:53:00 ceres unix: [ID 743441 kern.notice] g1-g7: 0, 0, 2621c, 1aa10, ea0a, 16, 2a10053dcc0
Feb 20 14:53:00 ceres unix: [ID 100000 kern.notice]
Feb 20 14:53:00 ceres genunix: [ID 723222 kern.notice] 000002a10053c790 unix:die+78 (31, 2a10053ca70, 18, 0, 2a10053c850, 1076000)
Feb 20 14:53:00 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000001fff 0000000000000031 0000000001000000 0000000000002000
Feb 20 14:53:00 ceres%l4-7: 000000000181a1d8 000000000181a000 0000000000000000 00000000e85e2018
Feb 20 14:53:00 ceres genunix: [ID 723222 kern.notice] 000002a10053c870 unix:trap+9d4 (2a10053ca70, 10000, 1fff, 5, 0, 1)
Feb 20 14:53:00 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000000000 00000000018364c0 0000000000000031 0000000000000000
Feb 20 14:53:00 ceres%l4-7: ffffffffffffe000 0000000000f250af 0000000000000001 0000000000000005
Feb 20 14:53:01 ceres genunix: [ID 723222 kern.notice] 000002a10053c9c0 unix:ktl0+48 (b80c, c006, b7f2, 3511, 1a, 82e1)
Feb 20 14:53:01 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000000003 0000000000001400 0000000080001602 000000000101aa04
Feb 20 14:53:01 ceres%l4-7: 0000000000000008 00000600009a6e04 0000000000000006 000002a10053ca70
Feb 20 14:53:01 ceres genunix: [ID 562518 kern.notice] 000002a10053cb10 6 (0, 600009a6df0, 4, 600009a6e04, f332, 0)
Feb 20 14:53:01 ceres genunix: [ID 179002 kern.notice]%l0-3: 00000600009a6e14 0000000000000014 0000000000000006 0000000000000014
Feb 20 14:53:01 ceres%l4-7: 0000000000000028 0000000000000005 0000000000000045 0000000000000000
Feb 20 14:53:01 ceres genunix: [ID 723222 kern.notice] 000002a10053cbc0 ipf:appr_check+32c (2a10053cff8, 60002a0ef00, 0, 2a10053d000, 600010ad7b0, 0)
Feb 20 14:53:01 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000000000 00000600009a6df0 00000600009a6e04 0000000000000000
Feb 20 14:53:01 ceres%l4-7: 000000000000ffff 00000000701734b8 000000000000ffff 000000000000fc00
Feb 20 14:53:01 ceres genunix: [ID 723222 kern.notice] 000002a10053cc70 ipf:fr_natout+248 (2a10053cff8, 60002a0ef00, 1, 600009a6e14, a85a, fffff4e3)
Feb 20 14:53:01 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000000000 000006000115b500 0000060002a0f008 00000600009a6df0
Feb 20 14:53:01 ceres%l4-7: 0000000082e13511 0000000000000001 00000000701734b8 0000000000000001
Feb 20 14:53:01 ceres genunix: [ID 723222 kern.notice] 000002a10053cd20 ipf:fr_checknatout+4a8 (2a10053cff8, 6000115b500, 2a10053d000, 6000106bdf0, 70173388, fc)
Feb 20 14:53:02 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000080000000 0000000000040007 ffffffff00000000 0000000100000000
Feb 20 14:53:02 ceres%l4-7: 0000000082e13511 0000060002a0ef00 0000000000000000 0000000000000001
Feb 20 14:53:02 ceres genunix: [ID 723222 kern.notice] 000002a10053ce20 ipf:fr_fastroute+278 (600036e3000, 2a10053d188, 2a10053cff8, 0, 4, 6000106bdf0)
Feb 20 14:53:02 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000000000 0000000000000000 000006000106bdf0 0000000000000001
Feb 20 14:53:02 ceres%l4-7: 00000600036e3040 000002a10053cfa8 00000600009a6df0 00000600009a6df0
Feb 20 14:53:02 ceres genunix: [ID 723222 kern.notice] 000002a10053cef0 ipf:fr_send_ip+168 (2a10053d258, 600036e3040, 2a10053d188, 4000, ff, 600009a6df0)
Feb 20 14:53:02 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000040000000 0000000000000000 0000000000000000 000000000000000e
Feb 20 14:53:02 ceres%l4-7: 0000060000b969c0 000002a10053d410 000006000106beac 0000060000aee2a0
Feb 20 14:53:02 ceres genunix: [ID 723222 kern.notice] 000002a10053d0d0 ipf:fr_send_reset+258 (2a10053d258, c006ea0a, 600009a6e04, 600009a6df0, 0, 82e13511)
Feb 20 14:53:02 ceres genunix: [ID 179002 kern.notice]%l0-3: 000002a10053d260 0000000000000000 00000600009a7de0 0000000000000028
Feb 20 14:53:02 ceres%l4-7: 0000000000000006 0000000000000045 0000000000000040 0000000000000040
Feb 20 14:53:03 ceres genunix: [ID 723222 kern.notice] 000002a10053d190 ipf:fr_check+59c (3000, 0, 0, 701720d0, 600022d5900, 2a10053d518)
Feb 20 14:53:03 ceres genunix: [ID 179002 kern.notice]%l0-3: 000002a10053d258 0000000000000001 0000000000000000 0000000000000000
Feb 20 14:53:03 ceres%l4-7: 000002a10053d254 0000000000000000 0000000040009101 0000000000080000
Feb 20 14:53:03 ceres genunix: [ID 723222 kern.notice] 000002a10053d330 pfil:pfil_precheck+6c8 (0, 1, 14, 6000106bdf0, 0, 0)
Feb 20 14:53:03 ceres genunix: [ID 179002 kern.notice]%l0-3: 000002a10053d410 00000600009a7e08 0000060001109900 000006000311b1c0
Feb 20 14:53:03 ceres%l4-7: 00000000700cfef0 0000000000000000 00000600009a7de0 0000000000000800
Feb 20 14:53:03 ceres genunix: [ID 723222 kern.notice] 000002a10053d460 pfil:pfilmodrput+2c0 (60000aee2a0, 6000311b1c0, 2a100538000, 41, 6000106bdf0, 0)
Feb 20 14:53:03 ceres genunix: [ID 179002 kern.notice]%l0-3: 00000000010076e4 0000000000000006 0000004480001600 00000000000007c8
Feb 20 14:53:03 ceres%l4-7: 00000300000b3c80 0000000023800000 0000000000000042 0000000000000043
Feb 20 14:53:03 ceres genunix: [ID 723222 kern.notice] 000002a10053d520 unix:putnext+218 (60000aee490, 60000aee2a0, 6000311b1c0, 100, 60000aee530, 0)
Feb 20 14:53:03 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000000000 0000000000000000 0000000000000000 00000000000055d0
Feb 20 14:53:03 ceres%l4-7: 000000000000010d 000000007016ba40 000000007bb24418 fffffd5effac8000
Feb 20 14:53:04 ceres genunix: [ID 723222 kern.notice] 000002a10053d5d0 dld:dld_str_rx_fastpath+24 (60001043e08, 0, 6000311b1c0, e, 0, 0)
Feb 20 14:53:04 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000060001069f10 0000000000000006 000006000311b1c0 0000000000000000
Feb 20 14:53:04 ceres%l4-7: 0000000000000003 0000000000000003 0000060001069f54 0000000000001e71
Feb 20 14:53:04 ceres genunix: [ID 723222 kern.notice] 000002a10053d680 dls:i_dls_link_ether_rx+1c8 (0, 0, 133a850, 2a10053d740, 2a10053d748, 2a10053d730)
Feb 20 14:53:04 ceres genunix: [ID 179002 kern.notice]%l0-3: 000002a10053d738 0000000000000000 0000000000000000 00000600008c4940
Feb 20 14:53:04 ceres%l4-7: 0000060001061f80 0000060001061fa8 0000000000000001 0000000000000000
Feb 20 14:53:04 ceres genunix: [ID 723222 kern.notice] 000002a10053d770 mac:mac_rx+58 (6000105fce8, 0, 6000311b1c0, 133c5fc, 0, 6000105fa78)
Feb 20 14:53:04 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000000000005 00000600009a7d40 0000000000000002 0000000000000002
Feb 20 14:53:04 ceres%l4-7: 0000000000000001 0000000000000000 0000060000ac9308 0000000000000000
Feb 20 14:53:04 ceres genunix: [ID 723222 kern.notice] 000002a10053d820 bge:bge_receive+350 (60000e2fe10, 6000098f000, 0, 6000098fb40, 6000311b1c0, 6000311b1c0)
Feb 20 14:53:04 ceres genunix: [ID 179002 kern.notice]%l0-3: 00000300016c6800 00000600009908e0 000006000098f840 000006000311b1c0
Feb 20 14:53:04 ceres%l4-7: 00000300016db012 000000000000012e 0000000000000200 0000000000000040
Feb 20 14:53:05 ceres genunix: [ID 723222 kern.notice] 000002a10053d970 bge:bge_intr+108 (6000098f000, 18e0, 50000002a000000, 1800, 600009908e0, 1a20)
Feb 20 14:53:05 ceres genunix: [ID 179002 kern.notice]%l0-3: 0000000100000000 0000000000000000 0000000000000400 0000000000000002
Feb 20 14:53:05 ceres%l4-7: 0000000000001a20 0000000000006808 0000000000006800 00000300016db000
Feb 20 14:53:05 ceres genunix: [ID 723222 kern.notice] 000002a10053da20 pcisch:pci_intr_wrapper+b4 (300000c21c8, 60000b0b600, 0, 0, 0, 600010172d0)
Feb 20 14:53:05 ceres genunix: [ID 179002 kern.notice]%l0-3: 00000000018d3bb0 00000600009cc580 00000000018d3bf8 0000000000f26e20
Feb 20 14:53:05 ceres%l4-7: 00000300003e0970 000006000098f000 0000000000000000 000000007bb19110
Feb 20 14:53:05 ceres unix: [ID 100000 kern.notice]
Feb 20 14:53:05 ceres genunix: [ID 672855 kern.notice] syncing file systems...
Feb 20 14:53:05 ceres genunix: [ID 433738 kern.notice] [1]
Feb 20 14:53:05 ceres genunix: [ID 733762 kern.notice] 35
Feb 20 14:53:07 ceres genunix: [ID 433738 kern.notice] [1]
Feb 20 14:53:07 ceres genunix: [ID 733762 kern.notice] 28
Feb 20 14:53:08 ceres genunix: [ID 433738 kern.notice] [1]
Feb 20 14:53:29 ceres last message repeated 20 times
Feb 20 14:53:30 ceres genunix: [ID 622722 kern.notice] done (not all i/o completed)
Feb 20 14:53:33 ceres genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c1t0d0s0, offset 430374912, content: kernel
Feb 20 14:53:40 ceres genunix: [ID 409368 kern.notice] ^M100% done: 20967 pages dumped, compression ratio 4.92,
Feb 20 14:53:40 ceres genunix: [ID 851671 kern.notice] dump succeeded
