用于 Mellanox ConnectX-4 Lx 网络卡的 DPDK 适配器

一直想测试MCX网卡运行DPDK程序的性能、稳定性及丢包率。奈何绑定MCX后程序一直报检测不到port的错误。忙于其他工作就没有对这个问题进行解决。最近业务上又需要dpdk，但对端(agent)打过来的数据是基于IP + port，dpdk UIO把网卡抽离到用户态后会失去eth信息。调研的解决方式有多种：

1. DPDK KNI模块允许使用Linux标准网络工具ethtool, ifconfig和tcpdump管理DPDK端口
2. MCX网卡驱动mlx5等自己实现了轮询模式驱动程序库librte_pmd_mlx5，但是必须得安装专门的驱动
3. ......

考虑到本来就想测MCX以及学习KNI的周期，还是老老实实的解决这个问题......

1. 前期准备

注意：支持轮询的驱动需要按装如下图所示的OFED，而不是EN。

点击此链接，下载系统对应的版本，本文使用的是debian9.9，可以看到并不是所有的系统小版本都是有对应驱动的，如果想升级内核小版本，参考debian内核小版本升级(9.0->9.9)

2. 驱动安装

Step-1

wget https://www.mellanox.com/downloads/ofed/MLNX_OFED-5.2-1.0.4.0/MLNX_OFED_LINUX-5.2-1.0.4.0-debian9.9-x86_64.tgz
tar zxvf MLNX_OFED_LINUX-5.2-1.0.4.0-debian9.9-x86_64.tgz
cd MLNX_OFED_LINUX-5.2-1.0.4.0-debian9.9-x86_64/
# 此过程比较长
./mlnxofedinstall --upstream-libs --dpdk --add-kernel-support

Step-2

上面操作可能遇到的error（没错误可略过，执行step-3）

Installing python3-pyverbs-52mlnx1...
Failed to install python3-pyverbs DEB
Collecting debug info...
See /tmp/MLNX_OFED_LINUX.9495.logs/python3-pyverbs.debinstall.log
Failed to install MLNX_OFED_LINUX-5.2-1.0.4.0-debian9.9-ext for 4.9.0-3-amd64

查看日志内容如下，大致是python3-pyverbs包没有安装

Unpacking python3-pyverbs:amd64 (52mlnx1-1.52104) ...
dpkg: dependency problems prevent configuration of python3-pyverbs:amd64:
 python3-pyverbs:amd64 depends on libpython3.5 (>= 3.5.0~b1); however:
  Package libpython3.5 is not installed.

dpkg: error processing package python3-pyverbs:amd64 (--install):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Errors were encountered while processing:
 python3-pyverbs:amd64

执行apt install python3-pyverbs又报出以下错误

You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 python3-pyverbs : Depends: libpython3.5 (>= 3.5.0~b1) but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

导致安装失败的真正的原因：python3-pyverbs的依赖libpython3.5没有安装，执行apt install libpython3.5安装后再次执行./mlnxofedinstall --upstream-libs --dpdk --add-kernel-support即可安装成功

Step-3

安装成功会出现以下信息

Restart needed for updates to take effect.
Log File: /tmp/DeKhKl8b8j
Real log file: /tmp/MLNX_OFED_LINUX.28764.logs/fw_update.log
Device (82:00.0):
        82:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Link Width: x8
        PCI Link Speed: 8GT/s

Device (82:00.1):
        82:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Link Width: x8
        PCI Link Speed: 8GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/openibd restart

根据他的提示，需要加载新安装的驱动，继续执行/etc/init.d/openibd restart，会出现以下信息

Unloading HCA driver:                                      [  OK  ]
Loading HCA driver and Access Layer:                       [  OK  ]

继续关机重启reboot即可

3. DPDK的相关配置

下载DPDK源码后，不要着急编译，先修改config/common_base文件再编译。下面过程可供参考，DPDK的详细安装及配置可见这篇DPDK安装配置教程及hello world示例详解

wget http://fast.dpdk.org/rel/dpdk-20.02.1.tar.gz
tar zxvf dpdk-20.02.1.tar.gz
cd dpdk-stable-20.02.1
apt-get install libnuma-dev
# 打开config/common_base文件，修改CONFIG_RTE_LIBRTE_MLX5_PMD=y，再进行编译
vi config/common_base 
# 修改config/common_base文件
CONFIG_RTE_LIBRTE_MLX5_PMD=n  ===>   CONFIG_RTE_LIBRTE_MLX5_PMD=y
# 编译
make install T=x86_64-native-linuxapp-gcc
cd x86_64-native-linuxapp-gcc
# 加载驱动
modprobe -a ib_uverbs mlx5_core mlx5_ib

设置环境变量，大页内存等省略，详见上面那篇DPDK详细安装文章

4. 运行testpmd

testpmd相关eal参数及操作可见测试程序testpmd的用法

./testpmd -l 1-7 -n 4 -w 82:00.0 -w 82:00.1  -- -i --max-pkt-len=1524  --rxq=2 --txq=2  --rxd=1024 --txd=1024

参考

• http://doc.dpdk.org/guides-20.02/nics/mlx5.html
• https://bbs.huaweicloud.com/forum/thread-26292-1-1.html#
• https://blog.****.net/Jmilk/article/details/105980054
• https://blog.****.net/botao_li/article/details/108075832
• https://blog.****.net/baidu_35141290/article/details/100138087?utm_medium=distribute.pc_relevant.none-task-blog-baidujs_title-2&spm=1001.2101.3001.4242