服务器IB网卡安装教程

1、系统环境检查

1.1 检查是否插入IB网卡

lspci |grep Mell
# 5e:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
# 5e:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

1.2 查看系统版本

cat /etc/redhat-release
# CentOS Linux release 7.9.2009 (Core)

2、下载驱动

下载链接（根据实际情况选择版本）

3、安装包解压

#上传至服务器然后解压
tar -zxvf MLNX_OFED_LINUX-5.8-4.1.5.0-rhel7.9-x86_64.tgz

4、安装报错

cd MLNX_OFED_LINUX-23.10-2.1.3.1-rhel7.9-x86_64
./mlnxofedinstall
# 报错
Logs dir: /tmp/MLNX_OFED_LINUX.26646.logs
General log file: /tmp/MLNX_OFED_LINUX.26646.logs/general.log
Verifying KMP rpms compatibility with target kernel...
The kernel KMP rpms coming with MLNX_OFED_LINUX are not compatible with kernel: 3.10.0-1160.95.1.el7.x86_64
See log at /tmp/MLNX_OFED_LINUX.26646.logs/is_kmp_compat_check.log

The 3.10.0-1160.95.1.el7.x86_64 kernel is installed, MLNX_OFED_LINUX does not have drivers available for this kernel.
You can run mlnx_add_kernel_support.sh in order to to generate an MLNX_OFED_LINUX package with drivers for this kernel.
Or, you can provide '--add-kernel-support' flag to generate an MLNX_OFED_LINUX package and automatically start the installation.

5、报错解决方案

./mlnxofedinstall --add-kernel-support

# 根据报错提示安装相应的包
yum install createrepo

# 重新执行
./mlnxofedinstall --add-kernel-support

# 根据报错提示安装相应的包
yum install kernel-devel-3.10.0-1160.95.1.el7.x86_64 python-devel

# 重新执行
./mlnxofedinstall --add-kernel-support

# 显示驱动安装成功
Installation finished successfully.

# 重启驱动
/etc/init.d/openibd restart
# 启动成功
# Unloading HCA driver:                                      [  OK  ]
# Loading HCA driver and Access Layer:                       [  OK  ]

6、查看网卡状态

ibstatus
# 显示active
Infiniband device 'mlx5_0' port 1 status:
        default gid:     fe80:0000:0000:0000:0c42:a103:0016:0854
        base lid:        0xffff
        sm lid:          0x0
        state:           1: DOWN
        phys state:      3: Disabled
        rate:            10 Gb/sec (4X SDR)
        link_layer:      InfiniBand

Infiniband device 'mlx5_1' port 1 status:
        default gid:     fe80:0000:0000:0000:0c42:a103:0016:0855
        base lid:        0x2
        sm lid:          0x3
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            100 Gb/sec (4X EDR)
        link_layer:      InfiniBand

驱动安装成功，网络连接正常！（需要配置一下IP等）