Working with a customer the other day to migrate his direct connect SimpliVity deployment to a 10 GbE switch topology to facilitate adding a couple of compute nodes to expand the CPU and memory resources available to the cluster. He went through the migration but could not get things to work correctly. The issue ended up being the MTU configuration on the physical switch.
Mis-configuration of the physical switch is probably the most common issue I come across when configuring virtual networking to use Jumbo Frames. The configuration varies from switch to switch – vendor to vendor. On some switches the configuration is done globally, on other switches it is configured on individual ports, and on some switches it must be enabled globally and configured per port.
Jumbo Frames must be configured correctly end-to-end. This means on the vSwitch, vmkernel interfaces, physical switches, endpoint – everything must be configured to support the larger frames end-to-end. Using larger frames will provide more efficient processing of network traffic (in many cases) but it does require a bit more complexity. If any part of the path is mis-configured, network frames larger than 1500 may not pass. If they do pass they will be fragmented, which defeats the purpose of using Jumbo Frames.
To configure Jumbo Frames on vSphere the MTU is set to 9000 on the vSwitch.
If Jumbo Frames is being used for a vmkernel service (NFS storage, iSCSI storage, vMotion, etc) the MTU is set to 9000 on the vmkernel interface.
On the ESXi Host the vmkping command can be used to test a vmkernel interface. This is useful for testing proper configuration of Jumbo Frames.
The following options are used to test Jumbo Frames configuration using vmkping:
-I Specifies the vmkernel interface to use
-d Sets the do not fragment (DF) bit on IPv4 packets – this ensures the packet will not be fragmented
-s Sets the size of the packet – to test Jumbo Frames use 8192
The full command to test vmk2 for correct Jumbo Frames configuration: vmkping -I vmk2 -d -s 8192 IP.ADD.RES.SS
If Jumbo Frames is configured correctly end-to-end the vmkping will be successful.
Jumbo Frames can provide network efficiency and increase network performance but it must be configured correctly. Hope these tips help.
One thought on “Jumbo Frames – Make sure it is setup end-to-end.”
A very good article. Just wanted to share one point of view:
I believe based on the technology and the support for jumbo frames by switches in different manner the statement “Jumbo Frames must be configured correctly end-to-end. This means on the vSwitch, vmkernel interfaces, physical switches, endpoint – everything must be configured to support the larger frames end-to-end” is not accurate.
There are cases from a number of vendor deployments where physical switches can be segregated into switching domains and the term ‘end-to-end’ will confuse even the mature engineers and architects.
There are switch from so many vendors that was not exactly the case 5 years ago; that can even have different MTU for different port-channel / LACP groups.
Imagine a storage connected with a switch over a port channel and carry ISCSI Vlan that leads to a VMKernal on a vDS. If someone does not understand all component in this path having a larger MTU and confuses it with the need to have a large MTU for the whole switch; they might be in for a blocker when it comes to impement Jumbo frames.
A very good article. Just wanted to share one point of view:
I believe based on the technology and the support for jumbo frames by switches in different manner the statement “Jumbo Frames must be configured correctly end-to-end. This means on the vSwitch, vmkernel interfaces, physical switches, endpoint – everything must be configured to support the larger frames end-to-end” is not accurate.
There are cases from a number of vendor deployments where physical switches can be segregated into switching domains and the term ‘end-to-end’ will confuse even the mature engineers and architects.
There are switch from so many vendors that was not exactly the case 5 years ago; that can even have different MTU for different port-channel / LACP groups.
Imagine a storage connected with a switch over a port channel and carry ISCSI Vlan that leads to a VMKernal on a vDS. If someone does not understand all component in this path having a larger MTU and confuses it with the need to have a large MTU for the whole switch; they might be in for a blocker when it comes to impement Jumbo frames.
Keep safe.
Cheers,
Imran