For the ICM class there are 12 different lab environments with 2 virtual ESXi host each. To deploy these ESXi hosts I clone them from a master ESXi template that is preconfigured and has a local datastore with all the necessary class files and software. After I deploy each virtual machine I test connectivity and before last nights class everything seemed fine.
Once the class started accessing the lab environments we started seeing some network issues. Connections to the lab ESXi hosts were being dropped every few seconds, restarting the management networks would bring back connectivity but only for a short time. The issue turned out to be that each of the 24 hosts had the same MAC address assigned to vmk0 – oops!
The vmk0 is a virtual interface and a MAC address is generated for it when it is created. When the ESXi template is cloned the vmknic and its configuration (Name, MAC, MTU, and IP settings) are also cloned. Accessing ESXi Shell and running esxcfg-vmknic -l will list the configured vmknics and the configurations (including the MAC) of each:
There are 2 ways to correct this in the cloned ESXi hosts. One way is to delete and recreate the vmk interface. The other is to set the vmknics to use the hardware MAC address of their physical uplink NIC. The process for both of these options can be found in this kb article.
For the lab environment I set the vmknic MAC to follow the hardware MAC.
esxcfg-advcfg -s 1 /Net/FollowHardwareMac
After setting this and rebooting the vmknic (vmk0) now uses the MAC address of its physical uplink nic.
Not sure why this issue did not show up during testing but it should be fixed now.
Probably just misunderstood what you were asking in the chaos. All is good. Had this same issue in the first ICM 4 class. Thought I had it fixed in the ESXi template machine. Should be right now, tested a few deployments and we should not have to worry about this issue again.
Just hit this issue, although – not entirely un-triggered by myself.
My situation started with a vSphere HA of a v5.1.0 host and a v5.5 host. I removed the host from the vSphere not by feature but by function of the halt command 😉
After powering off the host, reinstalling the OS drive with v5.5, rebuilding the RAID array and restoring to the network – I noticed I had a MAC collision taking place between the two IPs associated with the two hosts above.
Host #1 – 172.16.0.21 – was the v5.1.0 host (now upgraded).
Host #2 – 172.16.0.23 – is the v5.5 host.
The network was seeing a broadcast response for both IPs with the same MAC address. This was the MAC address of host #1! Not only did this confuse the hell out of the network causing disruption for anything management, it also confused the hell out of me!
Wasn’t until I started looking into the vSphere setup that I realised it does ARP takeover/spoofing in order to service requests from the network.
Quick look at the ‘esxcfg-vmknic -l’ showed my offending MAC address. Running ‘esxcfg-advcfg -s 1 /Net/FollowHardwareMac’ and rebooting fixes the issue and I’m back to one MAC per host.
Perfect!
Not exactly the same issue as yours but related none the less. Maybe this comes in useful for anyone else having a similar amount of fun 😉
Is there a possibility of changing the mac addres of vmknic without rebooting the host?
Restarting management network doesn’t seem to work. I recently upgraded an old 4.1 esxi to 5.5 simply by installing 5.5 on an usb drive on an identical server, and sure enough, after booting with it, i had a mac conflict on the network. The problem is i cannot reboot the host again.
If it’s not possible, can you manually set the mac of a new host? Because i’ll soon have to power on the server i first installed 5.5 on, and i’ll have the conflict again. If i can change the mac of this server, at least it will not conflict with the old un-rebootable one.
Gonna answer myself to the question i posted earlier.
Basically the command above requires a reboot, but i found another way of doing it without reboot.
Assuming you have ssh enabled on the host you want to change:
esxcfg-nics -l lists your physical nics on the machine, with their correct macs.
esxcfg-vmknic -l lists the management nics, which are the ones you want to change.
copy the good macs, then edit /etc/vmware/esx.conf, and scroll down.
Just replace the macs there.
save the file, then type “dcui” to get to the console screen, login, then simply restart the management network.
Ctrl-C to go back to the shell, and check again esxcfg-nics -l and esxcfg-vmknic -l . They should be the same.
Oh, I thought you said these weren’t clones. I saw this issue first time I did an esxi cluster in workstation, bit annoying. Glad you fixed it!
John,
Probably just misunderstood what you were asking in the chaos. All is good. Had this same issue in the first ICM 4 class. Thought I had it fixed in the ESXi template machine. Should be right now, tested a few deployments and we should not have to worry about this issue again.
Hope to see you on Thursday.
Hersey
Just hit this issue, although – not entirely un-triggered by myself.
My situation started with a vSphere HA of a v5.1.0 host and a v5.5 host. I removed the host from the vSphere not by feature but by function of the halt command 😉
After powering off the host, reinstalling the OS drive with v5.5, rebuilding the RAID array and restoring to the network – I noticed I had a MAC collision taking place between the two IPs associated with the two hosts above.
Host #1 – 172.16.0.21 – was the v5.1.0 host (now upgraded).
Host #2 – 172.16.0.23 – is the v5.5 host.
The network was seeing a broadcast response for both IPs with the same MAC address. This was the MAC address of host #1! Not only did this confuse the hell out of the network causing disruption for anything management, it also confused the hell out of me!
Wasn’t until I started looking into the vSphere setup that I realised it does ARP takeover/spoofing in order to service requests from the network.
Quick look at the ‘esxcfg-vmknic -l’ showed my offending MAC address. Running ‘esxcfg-advcfg -s 1 /Net/FollowHardwareMac’ and rebooting fixes the issue and I’m back to one MAC per host.
Perfect!
Not exactly the same issue as yours but related none the less. Maybe this comes in useful for anyone else having a similar amount of fun 😉
Hey Hersey,
Is there a possibility of changing the mac addres of vmknic without rebooting the host?
Restarting management network doesn’t seem to work. I recently upgraded an old 4.1 esxi to 5.5 simply by installing 5.5 on an usb drive on an identical server, and sure enough, after booting with it, i had a mac conflict on the network. The problem is i cannot reboot the host again.
If it’s not possible, can you manually set the mac of a new host? Because i’ll soon have to power on the server i first installed 5.5 on, and i’ll have the conflict again. If i can change the mac of this server, at least it will not conflict with the old un-rebootable one.
Gonna answer myself to the question i posted earlier.
Basically the command above requires a reboot, but i found another way of doing it without reboot.
Assuming you have ssh enabled on the host you want to change:
esxcfg-nics -l lists your physical nics on the machine, with their correct macs.
esxcfg-vmknic -l lists the management nics, which are the ones you want to change.
copy the good macs, then edit /etc/vmware/esx.conf, and scroll down.
Just replace the macs there.
save the file, then type “dcui” to get to the console screen, login, then simply restart the management network.
Ctrl-C to go back to the shell, and check again esxcfg-nics -l and esxcfg-vmknic -l . They should be the same.
Nice! Glad you were able to get it sorted out and thanks for posting the solution.