mellanox在vmware中的切割

本文以ESXI6.7-CX3的组合做示例

mellanox官网下载对应的驱动安装包

ConnectX® Ethernet Driver for VMware® ESXi Server

下载后的文件

MLNX-NATIVE-ESX-ConnectX-3_3.17.70.1-1OEM.670.0.0-15491048.zip

参考手册： Installation - ESXi 6.7 v4.17.15.16 - NVIDIA Networking Docs

将文件解压成

MLNX-NATIVE-ESX-ConnectX-3_3.17.70.1-1OEM.670.0.0-offline_bundle-15491048.zip

将解压后的文件上传到ESXI上，使用SSH登录到ESXI里

vib install -d /<路径>/MLNX-NATIVE-ESX-ConnectX-3_3.17.70.1-1OEM.670.0.0-offline_bundle-15491048.zip 

重启机器

[root@localhost:/vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB] esxcli software 
vib install -d /vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB/MLNX-NATIVE-ESX-Co
nnectX-3_3.17.70.1-1OEM.670.0.0-offline_bundle-15491048.zip 
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: MEL_bootbank_nmlx4-core_3.17.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-en_3.17.70.1-1OEM.670.0.0.8169922, MEL_bootbank_nmlx4-rdma_3.17.70.1-1OEM.670.0.0.8169922
   VIBs Removed: VMW_bootbank_nmlx4-core_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-en_3.17.9.12-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-rdma_3.17.9.12-1vmw.670.0.0.8169922
   VIBs Skipped: 
[root@localhost:/vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB] reboot

下载mellanox控制程序

Mellanox Firmware Tools (MFT) (nvidia.com)

安装对应版本

参考手册： Virtualization - ESXi 6.7 v4.17.15.16 - NVIDIA Networking Docs

vib install -v /<路径>/nmst-4.18.0.106-1OEM.650.0.0.4598673.x86_64.vib 

vib install -v /<路径>/mft-4.18.0.106-10EM-600.0.0.4598673.x86_64.vib

[root@localhost:~] esxcli software 
vib install -v /vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB/nmst-4.18.0.106-1O
EM.650.0.0.4598673.x86_64.vib 
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: MEL_bootbank_nmst_4.18.0.106-1OEM.650.0.0.4598673
   VIBs Removed: 
   VIBs Skipped: 
[root@localhost:~] esxcli software 
vib install -v /vmfs/volumes/62e74b89-e6922764-6655-dc9914c8e87f/VIB/mft-4.18.0.106-10E
M-600.0.0.4598673.x86_64.vib 
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: MEL_bootbank_mft_4.18.0.106-0
   VIBs Removed: 
   VIBs Skipped: 
[root@localhost:~] reboot

启动服务

[root@localhost:~] /opt/mellanox/bin/mst start
[root@localhost:~] /opt/mellanox/bin/mst status
MST devices:
------------
mt4099_pciconf0
mt4099_pci_cr0

查看当前拆分状态

[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 q

Device #1:
----------

Device type:    ConnectX3
Device:         mt4099_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          8
         WOL_MAGIC_EN_P1                     True(1)
         LINK_TYPE_P1                        VPI(3)
         LINK_TYPE_P2                        VPI(3)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               True(1)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               True(1)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)

观察这2个参数   SRIOV_EN = True(1)                NUM_OF_VFS  =  8  如果没有打开可以使用命令

[root@localhost:~] /opt/mellanox/bin/mlxconfig -d mt4099_pciconf0  set SRIOV_EN=1 NUM_OF_VFS=8
[root@localhost:~] reboot

使用max_vfs模块参数设置需要为 PF 创建的虚函数数。

[root@localhost:~] esxcli system module parameters set -m mlx4_core -p "max_vfs=8"