We got two SF4800 running Oracle RAC on Solaris 9 and Sun Cluster 3.1. We found the udpInCksumErrs counter dumps up suddenly when the server loading increases. We tried to use an packet analyzer software to monitor a mirrored network ports of them, but failed to find any UDP packets with incorrect check sum.
What is the meaning of udpInCksumErrs? Will it affect the system?
Thanks for your information.
We can't find any UDP packets with checksum error coming from our network. So, we suspect those corrupted packets are coming from heartbeats, but it is very difficult to trace.
Do they have the relationship or triggered by the external packets?
Does Sun Cluster has a tool to trace such problem?
Shift happens.
Intersting point though, i wonder if it occours on both nodes or just one.
Its quite worrying if it occours on the heartbeat interface, it might mean that the cluster looses heartbeats.
After all the interconnect should just be a direct connection with a crossed cable..
7/M.
As far as I know, the Sun Cluster heartbeat does not use UDP. It just uses raw ethernet frames of type 0833. You can see these if you do a snoop on the interface. I pretty sure that the UDP packets are from the Oracle RAC distributed lock manager/cache fusion.
I would have thought, again, that RAC would create the packets it needed and ask Solaris to send them. (I'm checking this with a colleague). From scanning the bugs database, it would appear that there are a number of recommendations:
* Ensure the latest patches have been applied
* Make sure the system has been installed in accordance with Sun's EIS checklist (if Sun installed the system this would have happened).
Which says:
When using supported network adapters which use the *ce* driver for private
transport, insert into file /etc/system:
set ce:ce_taskq_disable=1
Also see InfoDoc:79189
Tim
Tim
Thanks for your suggestion.
Actually, we got 3 Oracle RAC incidents before. All of them are related to the block locking problem. We suspect the bad checksum UDP packet is the root cause.
Our system was installed by a Sun certified service provider and I've checked the "system" file which already set ce_taskq_disable to 1.
I assume that your phrase 'set ce_taskq_disable to 1' is a typo as it should be 'set ce:ce_taskq_disable=1' (with a ':' not an '_').
I this is what you have I would open a case with Sun support as they'll need to go through a details check of the patches and settings you have in /etc/system.
Bottom line, it is probably a tuning issue. The corrupt headers will cause Oracle to have to retransmit the packets, resulting in lower performance.
Tim
