I have tried to install mtcp with the netmap module.I have install the netmap module successfully. and compile the mtcp source successfully too.but when I run the test program in apps/examples It occurs an error,I run like this: ./epserver -p /home/ -f ./epserver.conf -N 4 ./epclient 127.0.0.1/example.txt 10000 -N 4 -c 80

but there always an error,it says there is a Segmentation fault when I run into the mtcp_init() fuction.

when I run the dmesg command it says like these: [ 2147.768239] epserver[6166]: segfault at 0 ip (null) sp 00007fff3f834eb8 error 14 in epserver[400000+21000] [ 2189.381506] epwget[6179]: segfault at 0 ip (null) sp 00007ffddff13548 error 14 in epwget[400000+22000]

did I do some thing wrong.do someone have some document that make mtcp working with netmap?

Hi, mtcp team,

I can build DPDK and mTCP successfully, but after binding Ethernet device(82599ES ) to IGB UIO module dpdk0 does not appear. The NIC: 82599ES 10-Gigabit SFI/SFP+ Network Connection

Have any opinion about this issue? Am I missing some step?
Thanks in advance.

Hi,

I am running ubuntu 14.04 as virtual guest on VMware ESXi, the guest is using adapter vmxnet3. I had following diff to mtcp/src/dpdk_module.c to make mTCP compile and run, but at the web server side when I run tcpdump, I see no packet coming in to server. I don't have access to VMware ESXi hypervisor, so not sure if the packet has egressed out hypervisor.

`diff --git a/mtcp/src/dpdk_module.c b/mtcp/src/dpdk_module.c index 33d349e..666dfd3 100644 --- a/mtcp/src/dpdk_module.c +++ b/mtcp/src/dpdk_module.c @@ -57,8 +57,8 @@ /*

• Configurable number of RX/TX ring descriptors */ -#define RTE_TEST_RX_DESC_DEFAULT 128 -#define RTE_TEST_TX_DESC_DEFAULT 128 +#define RTE_TEST_RX_DESC_DEFAULT 128 +#define RTE_TEST_TX_DESC_DEFAULT 512

  static uint16_t nb_rxd = RTE_TEST_RX_DESC_DEFAULT; static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT; @@ -124,7 +124,7 @@ static const struct rte_eth_txconf tx_conf = { * As the example won't handle mult-segments and offload cases, * set the flag by default. */

  •   .txq_flags =                    0x0,
    

  •   .txq_flags =                    ETH_TXQ_FLAGS_NOOFFLOADS|ETH_TXQ_FLAGS_NOMULTSEGS,
    

    };

  struct mbuf_table { `

I noticed the dpdk0 interface has empty MAC address as below:

4: dpdk0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff <==========EMPTY MAC ADDRESS inet 10.1.72.28/24 brd 10.1.72.255 scope global dpdk0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe00:0/64 scope link valid_lft forever preferred_lft forever

I reviewed dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h, noticed only IXGBE and IGB adapter were supported for mTCP to retrieve and attach MAC addresses for dpdk0 in Linux world.

do you think the empty MAC address for dpdk0 is the reason I see no packet from the server side?

if so, adapter vmxnet3 can be added to igb_uio.h like IGB adapter to resolve the issue?

below is the output I think relevant:

EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd EAL: PCI memory mapped at 0x7ff74ea00000 EAL: PCI memory mapped at 0x7ff74ea01000 EAL: PCI memory mapped at 0x7ff74ea02000 PMD: eth_vmxnet3_dev_init(): >> PMD: eth_vmxnet3_dev_init(): Hardware version : 1 PMD: eth_vmxnet3_dev_init(): UPT hardware version : 1 PMD: eth_vmxnet3_dev_init(): MAC Address : 00:50:56:86:10:76 Total number of attached devices: 1 Interface name: dpdk0 Configurations: Number of CPU cores available: 4 Number of CPU cores to use: 4 Maximum number of concurrency per core: 10000 Maximum number of preallocated buffers per core: 10000 Receive buffer size: 8192 Send buffer size: 8192 TCP timeout seconds: 30 TCP timewait seconds: 0

NICs to print statistics: dpdk0

Interfaces: name: dpdk0, ifindex: 0, hwaddr: 00:00:00:00:00:00, ipaddr: 10.1.72.28, netmask: 255.255.255.0

Number of NIC queues: 4

Loading routing configurations from : /etc/mtcp/config/route.conf Routes: Destination: 10.1.72.0/24, Mask: 255.255.255.0, Masked: 10.1.72.0, Route: ifdx-0

Destination: 10.1.72.0/24, Mask: 255.255.255.0, Masked: 10.1.72.0, Route: ifdx-0

Loading ARP table from : /etc/mtcp/config/arp.conf ARP Table:

IP addr: 10.1.72.17, dst_hwaddr: 00:23:E9:64:C0:03

Initializing port 0... PMD: vmxnet3_dev_configure(): >> PMD: vmxnet3_dev_rx_queue_setup(): >> PMD: vmxnet3_dev_rx_queue_setup(): >> PMD: vmxnet3_dev_rx_queue_setup(): >> PMD: vmxnet3_dev_rx_queue_setup(): >> PMD: vmxnet3_dev_tx_queue_setup(): >> PMD: vmxnet3_dev_tx_queue_setup(): >> PMD: vmxnet3_dev_tx_queue_setup(): >> PMD: vmxnet3_dev_tx_queue_setup(): >> PMD: vmxnet3_dev_start(): >> PMD: vmxnet3_rss_configure(): >> PMD: vmxnet3_setup_driver_shared(): Writing MAC Address : 00:50:56:86:10:76 PMD: vmxnet3_disable_intr(): >> PMD: vmxnet3_dev_rxtx_init(): >> rte_eth_dev_config_restore: port 0: MAC address array not supported <=====here done:

Checking link statusdone Port 0 Link Up - speed 10000 Mbps - full-duplex Configuration updated by mtcp_setconf(). CPU 0: initialization finished. [mtcp_create_context:1173] CPU 0 is now the master thread. [CPU 0] dpdk0 flows: 0, RX: 10(pps) (err: 0), 0.00(Gbps), TX: 0(pps), 0.00(Gbps) [ ALL ] dpdk0 flows: 0, RX: 10(pps) (err: 0), 0.00(Gbps), TX: 0(pps), 0.00(Gbps) Thread 0 handles 1 flows. connecting to 10.1.72.17:80 [CPU 0] dpdk0 flows: 1, RX: 25(pps) (err: 0), 0.00(Gbps), TX: 2(pps), 0.00(Gbps) [ ALL ] dpdk0 flows: 1, RX: 25(pps) (err: 0), 0.00(Gbps), TX: 2(pps), 0.00(Gbps)

Hi,

I'm having some troubles running example apps with mellanox driver. I've compiled provided dpdk with CONFIG_RTE_LIBRTE_MLX5_PMD=y and testpmd works just fine:

# ./testpmd -c 0xff00 -n 4 -w 0000:19:00.1 -- --rxq=2 --txq=2
EAL: Detected 64 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:19:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1015 net_mlx5
PMD: net_mlx5: PCI information matches, using device "mlx5_1" (SR-IOV: false, MPS: true)
PMD: net_mlx5: 1 port(s) detected
PMD: net_mlx5: MPS is enabled
PMD: net_mlx5: port 1 MAC address is ec:0d:9a:3a:3b:5b
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=203456, size=2176, socket=0
USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=203456, size=2176, socket=1
Configuring Port 0 (socket 0)
PMD: net_mlx5: 0xd171c0: TX queues number update: 0 -> 2
PMD: net_mlx5: 0xd171c0: RX queues number update: 0 -> 2
Port 0: EC:0D:9A:3A:3B:5B
Checking link statuses...
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=1 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages disabled
Logical Core 9 (socket 1) forwards packets on 2 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
  RX P=0/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding - CRC stripping enabled - packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=1
  RX queues=2 - RX desc=128 - RX free threshold=0
  RX threshold registers: pthresh=0 hthresh=0 wthresh=0
  TX queues=2 - TX desc=512 - TX free threshold=0
  TX threshold registers: pthresh=0 hthresh=0 wthresh=0
  TX RS bit threshold=0 - TXQ flags=0x0

It binds to the interface and the server stops responding on it. The epserver app on the other hand initializes without any errors but doesn't bind to any of the interfaces:

# ./epserver -p /var/www -f epserver.conf -N 1
Configuration updated by mtcp_setconf().
---------------------------------------------------------------------------------
Loading mtcp configuration from : epserver.conf
Loading interface setting
EAL: Detected 64 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:19:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1015 net_mlx5
PMD: net_mlx5: PCI information matches, using device "mlx5_0" (SR-IOV: false, MPS: true)
PMD: net_mlx5: 1 port(s) detected
PMD: net_mlx5: MPS is enabled
PMD: net_mlx5: port 1 MAC address is ec:0d:9a:3a:3b:5a
EAL: PCI device 0000:19:00.1 on NUMA socket 0
EAL:   probe driver: 15b3:1015 net_mlx5
PMD: net_mlx5: PCI information matches, using device "mlx5_1" (SR-IOV: false, MPS: true)
PMD: net_mlx5: 1 port(s) detected
PMD: net_mlx5: MPS is enabled
PMD: net_mlx5: port 1 MAC address is ec:0d:9a:3a:3b:5b
Configurations:
Number of CPU cores available: 1
Number of CPU cores to use: 1
Maximum number of concurrency per core: 10000
Maximum number of preallocated buffers per core: 10000
Receive buffer size: 8192
Send buffer size: 8192
TCP timeout seconds: 30
TCP timewait seconds: 0
NICs to print statistics:
---------------------------------------------------------------------------------
Interfaces:
Number of NIC queues: 1
---------------------------------------------------------------------------------
Loading routing configurations from : config/route.conf
Routes:
Destination: 172.16.1.0/24, Mask: 255.255.255.0, Masked: 172.16.1.0, Route: ifdx--1
---------------------------------------------------------------------------------
Loading ARP table from : config/arp.conf
ARP Table:
IP addr: 172.16.1.10, dst_hwaddr: EC:0D:9A:3A:4B:B3
---------------------------------------------------------------------------------

Checking link statusdone
[dpdk_init_handle: 263] Can't open /dev/dpdk-iface for context->cpu: 0! Are you using mlx4/mlx5 driver?
CPU 0: initialization finished.
[mtcp_create_context:1230] CPU 0 is now the master thread.

Here's my epserver.conf

# cat epserver.conf 
io = dpdk
num_mem_ch = 4
port = mlx5_1
max_concurrency = 10000
max_num_buffers = 10000
rcvbuf = 8192
sndbuf = 8192
tcp_timeout = 30
tcp_timewait = 0

No matter what I put as the port value, I always get the same behavior. The server responds to pings and trying to connect to port 80 gives connection refused. I was unable to find any examples of mTCP+Mellanox combo so I don't really know what to put the port value to.

Did I miss something?

Thank you,

Hrvoje Zeba

Hi,

I'm trying to launch epserver but I get "Error opening dpdk-face".

Indeed dpdk-iface.ko is loaded but there is no /dev/dpdk-iface entry. I use your dpdk version from git submodule.

I have a mlx5 NIC. Kernel 4.15. Ubuntu 18.04.

epserver.conf: io = dpdk num_mem_ch = 6 port = enp115s0f0 rcvbuf = 8192 sndbuf = 8192 tcp_timeout = 30 tcp_timewait = 0 stat_print = dpdk0

Thanks, Tom

Hi, Can I use only one physical machine, and open two virtual machine on it. Then one virtual machine run epwget and another one run epserver. Can it work?

Hi, Thanks for putting this great module. The userspace TCP is the hot cake in the market. I have been looking for one open source for sometime. I have few questions -

1. Can you please let me know what PPS you are able to achieve with this.
2. Do you plan to make it work for multiple processes similar to the NGINX architecture.
3. Do you plan to integrate this with netmap.

Thanks..Santos

Hi,

I build mtcp with the latest version of netmap which uses the ixgbe-5.3.7 driver.

But the test results show nearly zero throughput. I think the problem here is the RSS.

So how I correctly setup the RSS in the ixgbe-5.3.7 driver?

I do not modify seeds as suggested by the readme.netmap, since the function ixgbe_setup_mrqc() in 5.3.7 driver may be not a proper place to do it. Instead, I write the seeds in function ixgbe_init_rss_key().

Hi

the ported apachebench does not support SSL load test, I am wondering what client SSL load tool mtcp project used to evaluate SSLShader performance mentioned in the paper. I am interested to know if there is existing one or get some idea on how to port one on mtcp.

Thanks

We met an unsolvable problem. At first we had to run mtcp on a Redhat 6.2 machine whose kernel version was 2.6.32-220.el6.x86_64. We compiled a new kernel version 3.2.78. Then we met the problem.

When we tried to use the former mtcp version with dpdk-2.1.0, we met these strange problems when compiling dpdk-2.1.0 in it:

CC [M] /home/gj/mtcp/dpdk-2.1.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o In file included from /home/gj/mtcp/dpdk-2.1.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:34: /home/gj/mtcp/dpdk-2.1.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:378: error: unknown field ‘ndo_fdb_add’ specified in initializer In file included from /home/gj/mtcp/dpdk-2.1.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:39: /home/gj/mtcp/dpdk-2.1.0/lib/librte_eal/linuxapp/igb_uio/compat.h:53: error: redefinition of ‘pci_intx_mask_supported’ /home/gj/mtcp/dpdk-2.1.0/lib/librte_eal/linuxapp/igb_uio/compat.h:53: note: previous definition of ‘pci_intx_mask_supported’ was here /home/gj/mtcp/dpdk-2.1.0/lib/librte_eal/linuxapp/igb_uio/compat.h:76: error: redefinition of ‘pci_check_and_mask_intx’ /home/gj/mtcp/dpdk-2.1.0/lib/librte_eal/linuxapp/igb_uio/compat.h:76: note: previous definition of ‘pci_check_and_mask_intx’ was here make[10]: *** [/home/gj/mtcp/dpdk-2.1.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o] Error 1 make[9]: *** [module/home/gj/mtcp/dpdk-2.1.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio] Error 2 make[8]: *** [sub-make] Error 2 make[7]: *** [igb_uio.ko] Error 2 make[6]: *** [igb_uio] Error 2 make[5]: *** [linuxapp] Error 2 make[4]: *** [librte_eal] Error 2 make[3]: *** [lib] Error 2 make[2]: *** [all] Error 2 make[1]: *** [x86_64-native-linuxapp-gcc_install] Error 2 make: *** [install] Error 2

When tried new version with dpdk-2.2.0, error gone with:

CC [M] /home/mtcp/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o In file included from /home/mtcp/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:35: /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:218: error: expected declaration specifiers or ‘...’ before ‘netdev_features_t’ /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h: In function ‘netdev_set_features’: /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:221: error: ‘features’ undeclared (first use in this function) /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:221: error: (Each undeclared identifier is reported only once /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:221: error: for each function it appears in.) /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h: At top level: /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:229: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘netdev_fix_features’ cc1: warnings being treated as errors /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:283: error: initialization from incompatible pointer type /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:284: error: ‘netdev_fix_features’ undeclared here (not in a function) /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/igb_uio.h:285: error: unknown field ‘ndo_fdb_add’ specified in initializer In file included from /home/mtcp/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:42: /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/compat.h:66: error: redefinition of ‘pci_intx_mask_supported’ /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/compat.h:66: note: previous definition of ‘pci_intx_mask_supported’ was here /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/compat.h:89: error: redefinition of ‘pci_check_and_mask_intx’ /home/mtcp/dpdk-2.2.0/lib/librte_eal/linuxapp/igb_uio/compat.h:89: note: previous definition of ‘pci_check_and_mask_intx’ was here make[10]: *** [/home/mtcp/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o] Error 1 make[9]: *** [module/home/mtcp/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio] Error 2 make[8]: *** [sub-make] Error 2 make[7]: *** [igb_uio.ko] Error 2 make[6]: *** [igb_uio] Error 2 make[5]: *** [linuxapp] Error 2 make[4]: *** [librte_eal] Error 2 make[3]: *** [lib] Error 2 make[2]: *** [all] Error 2 make[1]: *** [pre_install] Error 2 make: *** [install] Error 2

In /home/gj/mtcp/dpdk-2.1.0/lib/librte_eal/linuxapp/igb_uio/compat.h, I added

ifndef COMPAT_H

define COMPAT_H

endif

and then solved the redefinition problem. But others are still.

However, when we turned to the original dpdk version, dpdk-2.1.0-rc4, it succeeded without any error. What's wrong? I thought error goes from the dpdk in mtcp. Who can help me? Thank you!

Hi,

I am getting No route to epserver in epwget client. I am using the below route.conf and arp.conf. Please do let know of I have done anything wrong

route.conf

ROUTES 1 10.10.10.222/32 port0

arp.conf

ARP_ENTRY 1 10.10.10.222/32 00:0c:29:74:12:9a

This is the error in epwget client

[mtcp_create_context:1352] CPU 0 is in charge of printing stats. [GetOutputInterface: 28] [WARNING] No route to 10.10.10.222 CPU 1: initialization finished. [GetOutputInterface: 28] [WARNING] No route to 10.10.10.222 Thread 1 handles 5000 flows. connecting to 10.10.10.222:80 [GetOutputInterface: 28] [WARNING] No route to 10.10.10.222

Thanks, Mohan

Dear contributors,

RunMainLoop() in mtcp/src/core.c:761 is supposed to be executed per core, but has static variables, "static uint16_t len, static uint8_t *pktbuf" at line 787 and 788 thereby being not thread-safe and occurring unexpected problems. I think just removing the keyword "static" will be okay. Please consider any changes to fix the problem. Thank you

I know, problems with the packet driver are not primarily a problem of mtcp, but I need to set the MAC address manually, and I have no source for the packet driver (may be I can disassemble it with "Sourcer", but then I still have no clue of it). Is there a way to set the MAC address manually (e.g. in the mtcpcfg file) ? These MX98715 PCI cards (3 of them) are integrated in the (Watchguard FB1000) mainboard, and I have no chance to replace them. The original Linux kernel (2.0.3.3) which ran with the Firebox does support the MX98715 cards, but I replaced the internal flash memory content with DOS. I am able to transfer files with MSKERMIT and the help of the serial port to and from the Watchguard, but I have not enough free space to install a whole development environment, so if there a chance to modify the source code of mtcp for the above described purpose, I have to do this elsewhere. I've found also an issue entry #51 with a similar problem, but I am not sure if this can be solved in exactly the same manner.

Hi experts,

I got the issue at last step for mtcp build. When I run make for mtcp after configure successfully, I got the error " /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: error: ‘__builtin_strncpy’ output may be truncated copying 1023 bytes from a string of length 1023 [-Werror=stringop-truncation] 106 | return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest)); "

I tried to update my gcc to gcc 10.3.0, still the same error, please anyone saw the same issue before?

My environment is: os: Ubuntu 20.04.4 LTS kernel : 5.15.0-43-generic compiler: gcc 10.3.0 dpdk : by default 18.05

Expected behavior:

1. Run dhcp.exe
2. Get valid configuration from server
3. Use any program from mtcp package until lease time run out.
4. Optionally get warning that lease time is running out, but continue to run until then.

Current behavior:

1. Run dhcp.exe
2. Get valid configuration from server
3. Use any program until lease time is less than 1 hour, then mtcp refuses to run.

Default lease time of some dhcp servers is one hour (i.e. dnsmasq) what might make some confusion.

I am currently running mTCP with netmap. My epserver seems to start fine but I am a bit confused on how to query it from a client. I am planning to use wrk as a http request generator from another node on the same network, but I'm not sure how to what IP address or port I have to address it too. (Especially since my NIC loses its ip address once insert the custom network driver). Any help would be much appreciated.

Regards, Suhas