Link aggregation and "[kern.warning] WARNING: IP: Hardware address 'XXXXXXX

Hi,

I am trying to play with link aggregation feature on Sun x4100 (Solaris 10 06/06 + all patches)

># cat /etc/hostname.aggr1

>srv1

># cat /etc/hosts | grep srv1 | head -1

>10.2.33.3srv1.domain.com srv1

># cat /etc/inet/ipnodes | grep srv1 | head -1

>10.2.33.3srv1

># ifconfig -a

>lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

>inet 127.0.0.1 netmask ff000000

>aggr1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

>inet 10.2.33.3 netmask ffffff00 broadcast 10.2.33.255

>ether 0:14:4f:2a:11:48

>e1000g3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

>inet 10.2.34.98 netmask ffffffe0 broadcast 10.2.34.127

>ether 0:14:4f:2a:11:4b

># dladm show-link

>e1000g0 type: non-vlan mtu: 1500device: e1000g0

>e1000g1 type: non-vlan mtu: 1500device: e1000g1

>e1000g2 type: non-vlan mtu: 1500device: e1000g2

>e1000g3 type: non-vlan mtu: 1500device: e1000g3

>aggr1type: non-vlan mtu: 1500aggregation: key 1

># dladm show-dev

>e1000g0 link: upspeed: 1000 Mbpsduplex: full

>e1000g1 link: upspeed: 1000 Mbpsduplex: full

>e1000g2 link: downspeed: 0Mbpsduplex: half

>e1000g3 link: upspeed: 1000 Mbpsduplex: full

># dladm show-aggr

>key: 1 (0x0001) policy: L4address: 0:14:4f:2a:11:48 (auto)

>deviceaddressspeedduplex linkstate

>e1000g00:14:4f:2a:11:48 1000 Mbpsfullupattached

>e1000g10:14:4f:2a:11:49 1000 Mbpsfullupattached

># dladm show-aggr -L

>key: 1 (0x0001) policy: L4address: 0:14:4f:2a:11:48 (auto)

>LACP mode: off LACP timer: short

>deviceactivity timeout aggregatable sync coll dist defaulted expired

>e1000g0passive shortyes nonononono

>e1000g1passive shortyes nonononono

Some statistics:

># dladm show-aggr -s

>key: 1 ipackets rbytesopacketsobytes %ipkts %opkts

>Total62404327447842111 3474266265910602

>e1000g023435543040289643 180629312945140337.652.0

>e1000g138970674407709894 166809213646893962.4

>

># dladm show-link -s

>ipackets rbytes ierrors opacketsobytesoerrors

>e1000g0 0 000 00

>e1000g1 0 000 00

>e1000g2 0 000 00

>e1000g3 10069008008 541 0

>aggr163277547523738247 035293362704270060

># dmesg | grep -i aggr

>Sep 6 15:40:34 srv1 mac: [ID 543131 kern.info] NOTICE: aggr0/1 registered

Both NET0 and NET1 of my x4100 is plugged into Cisco 3560g gigabit ethernet switch.

NET0 is plugged into:

>cisco-3560g-1#sh runn inter GigabitEthernet 0/3

>Building configuration...

>

>Current configuration : 161 bytes

>!

>interface GigabitEthernet0/3

> switchport access vlan 2

> switchport mode access

> spanning-tree portfast

>end

NET1 is plugged into:

>cisco-3560g-1#sh runn inter GigabitEthernet 0/4

>Building configuration...

>

>Current configuration : 161 bytes

>!

>interface GigabitEthernet0/4

> switchport access vlan 2

> switchport mode access

> spanning-tree portfast

>end

Now in /var/adm/messages I can see a lot of messages like:

>Sep 7 12:43:13 ride ip: [ID 903730 kern.warning] WARNING: IP: Hardware address '00:14:4f:2a:11:48' trying to be our address 010.200.144.003!

Do I miss something? Should I do some extra setup on my cisco switch?

[3751 byte] By [tbricks] at [2007-11-26 9:59:17]
# 1
I have the same problem. Did you, by any chance, solve this issue, or know of any workarounds?
SirAl at 2007-7-7 1:26:21 > top of Java-index,General,Network Configurations...
# 2
Have you created the port-channel in the cisco switch? I don't see the "channel-group X mode on" lines in your config (X= number of port channel)
Gersius at 2007-7-7 1:26:21 > top of Java-index,General,Network Configurations...
# 3
Should I create port-channel? What encapsulation and mode type should I setup?.. What about LACP mode setting of the Cisco side?
tbricks at 2007-7-7 1:26:21 > top of Java-index,General,Network Configurations...
# 4
any body resolved this one i too have the same problem.
Tiger1234 at 2007-7-7 1:26:21 > top of Java-index,General,Network Configurations...
# 5

please check the speed of each physical ports an the Cisco Switch which you are trying to aggregrate, but itried trunking both on the server side and the swtich side.

i have done the configuration using SunTrunking 1.3 on Solaris 5.10 with E2900 server plus you have make the switch side to be in the same speed and duplex with trunking not aggregation

Tiger1234 at 2007-7-7 1:26:21 > top of Java-index,General,Network Configurations...
# 6

I managed to set link aggregation for 3 NICs (Solaris x86 u2 + all patches). Below is my current config.

Solaris side

-

Don't forget to set LACP mode to "active" on Solaris side

(since we have LACP mode as "passive" on Cisco side):

# dladm modify-aggr -l active 1

# dladm show-aggr

key: 1 (0x0001) policy: L2address: 0:14:4f:2a:11:48 (auto)

deviceaddressspeedduplex linkstate

e1000g00:14:4f:2a:11:48 1000 Mbpsfullupattached

e1000g10:14:4f:2a:11:49 1000 Mbpsfullupattached

e1000g20:14:4f:2a:11:4a 1000 Mbpsfullupattached

# dladm show-aggr -L

key: 1 (0x0001) policy: L2address: 0:14:4f:2a:11:48 (auto)

LACP mode: activeLACP timer: short

deviceactivity timeout aggregatable sync coll dist defaulted expired

e1000g0activeshortyes yesyes yes nono

e1000g1activeshortyes yesyes yes nono

e1000g2activeshortyes yesyes yes nono

# dladm show-aggr -s

key: 1 ipackets rbytesopacketsobytes %ipkts %opkts

Total1036003820424933065573967342497 288842640366

e1000g0239827120 80862894412 260672380 6199100410723.126.9

e1000g1229746435 68747894317 616528856 19197453352522.263.7

e1000g2566430274 27532227785290141279 3487711342654.79.3

Cisco side

-

1. Port-channel:

spb-cisco-3560g-1#sh running-config interface Port-Channel 3

Building configuration...

Current configuration : 120 bytes

!

interface Port-channel3

description link aggregation for RIDE

switchport access vlan 2

switchport mode access

end

2. Ports:

spb-cisco-3560g-1#sh running-config interface GigabitEthernet 0/3

Building configuration...

Current configuration : 191 bytes

!

interface GigabitEthernet0/3

description NET0 on RIDE (link aggregation setup)

switchport access vlan 2

switchport mode access

channel-group 3 mode passive

spanning-tree portfast

end

spb-cisco-3560g-1#sh running-config interface GigabitEthernet 0/4

Building configuration...

Current configuration : 191 bytes

!

interface GigabitEthernet0/4

description NET1 on RIDE (link aggregation setup)

switchport access vlan 2

switchport mode access

channel-group 3 mode passive

spanning-tree portfast

end

spb-cisco-3560g-1#sh running-config interface GigabitEthernet 0/5

Building configuration...

Current configuration : 191 bytes

!

interface GigabitEthernet0/5

description NET2 on RIDE (link aggregation setup)

switchport access vlan 2

switchport mode access

channel-group 3 mode passive

spanning-tree portfast

end

Check that LACP on Cisco is running fine:

spb-cisco-3560g-1#show lacp 3 counters

LACPDUs MarkerMarker ResponseLACPDUs

PortSentRecvSentRecvSentRecvPkts Err

Channel group: 3

Gi0/33268194 1012380000 0

Gi0/43269053 1012910000 0

Gi0/53268501 1014320000 0

spb-cisco-3560g-1#show lacp 3 internal

Flags: S - Device is requesting Slow LACPDUs

F - Device is requesting Fast LACPDUs

A - Device is in Active modeP - Device is in Passive mode

Channel group 3

LACP portAdminOperPortPort

PortFlagsStatePriorityKeyKeyNumberState

Gi0/3SPbndl32768 0x30x30x30x3C

Gi0/4SPbndl32768 0x30x30x40x3C

Gi0/5SPbndl32768 0x30x30x50x3C

You can also run:

spb-cisco-3560g-1#term monit

spb-cisco-3560g-1#debug lacp

[debug output is skipped]

tbricks at 2007-7-7 1:26:21 > top of Java-index,General,Network Configurations...