Traffic Generation
The Colosseum allows for IPv4 traffic to be transmitted and received through different SRNs in order to test the capability of radios to deliver information. IP traffic is utilized by most internet applications today. IP traffic can provide good metrics for quality of success because IP packets can be counted and statistics such as bit rate throughput, latency, packet jitter inter-arrival time, and loss can be calculated. These metrics can be used for user practice, event scoring, and event visualization.
All traffic will be limited to IPv4 and utilize both TCP and UDP transport protocols. Traffic will be instantiated as individual flows. A flow will define the characteristics of the traffic along with the desired ingress and egress SRN pair. Multiple flows can be instantiated simultaneously to both the same or different SRN pairs. There is no plan to enable Jumbo Frames on the traffic generators or switches. Fragmentation of packets is allowed. The traffic generators will not be doing any path MTU discovery. Note however that “messages” of any size can be transmitted through the system. (IE: a 1Mb message could be sent and will be fragmented.)
Each user container will have a traffic interface named tr0 and have the IP address assigned 192.168.X.1/24. The source for all IP traffic will be 192.168.X.2-254 and the destination for all IP traffic will be 192.168.Y.2-254, however it will always enter one SRN and egress a completely different SRN. Each SRN has a /24 traffic network attached to it. Traffic will never ingress/egress the same subnet and thus, the same SRN. It will always be expected to go across the radio network.
From a single SRN, the traffic system may provide multiple flows for potentially multiple destinations and user radio applications need to be able to handle this.
Users’ radios are only responsible for passing unicast IPv4 data (TCP and UDP) which is between IP addresses on the 192.168.101-228.0/24 subnets. Additionally, users can expect to see ARP messages between the SRN tr0 interface and the directly-connected traffic generator. These ARP exchanges are necessary to ensure that traffic data can be passed between the SRN and directly-connected traffic generator. All other traffic (e.g., broadcast traffic, IPv6 traffic, etc.) will be minimized and users are not responsible for this traffic.
If users are using TUN/TAP interfaces as their radio application interfaces, it should be noted that these interfaces may also create additional traffic from networking processes within their containers. Users may want to configure interface and network settings within their containers to minimize extraneous traffic passing over their RF networks. For example, TUN/TAP interfaces by default may attempt to use ARP to discover hardware addresses associated with other nodes in the radio network, which may be overly burdensome or inconsistent with some radio application designs.
The MAC address of the traffic source and destination are statically assigned based on their function and the IP address. The MAC address of the traffic interface inside the user container will be static and mapped to each SRN. The Mac address of user container collaboration and CAN interfaces are randomly assigned.
Multi-Generator (MGEN) Traffic Generation Application
The IP traffic in Colosseum is currently generated by a modified version of the NRL’s Multi-Generator (MGEN) Traffic Generation Application. MGEN provides the ability to perform IP network performance tests and measurements using TCP and UDP/IP traffic. More details about MGEN can be found on the link here: https://www.nrl.navy.mil/itd/ncs/products/mgen
Each request to the traffic generation via the Colosseum CLI or Batch mode system will create new MGEN applications that will send traffic to the tr0 interface in the SRN container. During a traffic request, MGEN logs (.drc files) will be collected by the traffic generation system. At the end of a request, the MGEN logs will be provided in the team folder on the NAS. These files can be processed to generate traffic statistics such as latency, packet loss, and packet counts. NRL provides a tool called TRace Plot Real-time (TRPR) to process MGEN log files.
In interactive mode, all traffic starts 120 seconds after “tg start”. So for example, if the mgn file specifies a start of 15.0, then traffic won’t begin until 135 seconds after the user does “tg start.”
There is a slight modification to the MGEN output files generated in Colosseum. A new field called frag is added to the MGEN output. An updated version of the TRPR tool that processes these files is provided below.
Update (1/31/18): A new version of TRPR is now available which processes MGEN output files (which include the date in timestamps) generated since December 2017. This file can be found in the page attachments at the end of this article.
Update (5/10/18): The source code for the 1/31/18 update to TRPR is now available on this page for download. If and when this source code appears on a public git repository, the source code attached to this page should be considered deprecated. In addition, a diff is available showing the changes that were made to version 2.1b3 which is available at the link above. See the README.TXT file for build instructions, but please note that there will be no help desk support for modification and compilation of the source code.
Update (6/18/18): A new version of TRPR is now available which processes MGEN output files which include the TOS field in the listen-side logs marked as being generated by Mgen Version 5.02b-Kudu-2018_05_21. The compiled binary for linux and the source code for the update can be found in the attachments at the bottom of this page as trpr_frag_20180618.tar.gz and trpr_source_20180618.tar.gz.
Below is a detailed description of the path of IP traffic through the Colosseum.
Path of IP Traffic through Colosseum
IP Traffic will be sourced and received from servers external to the user allocated SRNs. IP Traffic will flow from the sending traffic generator server, to the user ingress SRN, across the channel emulator, and then out the egress SRN towards the listening traffic generator server. It is the user’s responsibility to route this IP traffic through the radio channel emulator and deliver to the receiving traffic generator server.
The following example will show how traffic will flow from a traffic generator on SRN 001 to a traffic sink on SRN 122. Each SRN has a traffic network interface that owns the IP 192.168.X.1/24. All traffic generator send/receive processes will have IP’s in the connected 192.168.X.0/24 network. The tap0 interfaces have addresses assigned in the diagram, but these are generated by the user and have no addressing requirements.
Packet Routing Setup
Users must devise a method to handle routing of traffic from the traffic interface to the radios on their radio networks. One way to do this is through a TUN/TAP interface and the use of routing software such as Quagga. There are many configurations that users may use to accomplish this objective and are not limited to these approaches. The description here assumes that users have set up their radio network routing.
Packet Routing Steps
Traffic Generator creates a packet with SRC IP = 192.168.101.2 and DST IP = 192.168.222.2
- Traffic Generator at 192.168.101.2 sends this packet to its default gateway, which is the tr0 interface on the connected SRN (192.168.101.1)
The Traffic Generator uses ARP requests to discover the MAC address of the tr0 interface on the connected SRN (192.168.101.1)
The user software receives the packet on its interface to the Traffic Network
- The user software does an IP route table lookup on the DST IP to determine the next hop in the path, which will be over the radio network
The user software must handle discovery of all IP subnets within its network (e.g., running a routing protocol or through static routes)
If a TUN/TAP interface is used as a radio application interface, by default it will attempt to discover Layer 2 hardware addresses using ARP. It is up to the user to either implement this functionality or otherwise configure the interface to be consistent with their radio application design.
The user software sends the packet to the next hop via the radio application running within the container
The user radio application routes the packet as necessary until the packet arrives at the SRN that is directly attached to the destination traffic network (here, 192.168.222.0/24)
- The user software in the destination SRN forwards the packet from the radio application to the traffic destination (192.168.222.2).
The container on the SRN uses ARP requests via tr0 to discover the MAC address of the connected traffic generator (192.168.222.2).
The receiving Traffic Generator receives the packet and processes it.
Initial ARP Discovery / Missing Initial Traffic Fix
Prior to Colosseum 3.4.0 (2019 July 12), at the beginning of scenario execution, approximately the first one to two seconds of traffic packets will sometimes not be delivered into user containers on the SRN. This is due to a delay in ARP resolution at several places along the entire route of the packet through the traffic, SRN, and RF systems. Given the amount of traffic being generated, too many packets built up within the traffic system and were dropped. Eventually ARP stabilizes, and end-to-end traffic is achieved.
There are two changes to the traffic system to resolve hardware addresses prior to the start of any scored traffic. One change addresses address resolution within the traffic system and another change enables user to construct static ARP tables on each SRN.
For each MGEN send/listen pair, an additional “dummy” packet is added to the very beginning of each MGEN script. This is a single 150 Byte UDP packet from source port 1000 to destination port 1000. This packet is sent approximately 15 seconds before any scored traffic is sent. The purpose of this packet is to force the traffic system to resolve the hardware address of the SRN traffic interface. This packet does not need to be routed by user and is not scored.
Additionally, the traffic generation containers are now configured to assign MAC addresses which are determined solely by the IP address. This means that user can construct static ARP tables since all IP-MAC pairs are deterministic and constant. When starting a traffic scenario, the traffic system will sequentially assign IP addresses to MGEN instances within the traffic network subnet for a given SRN. For a given SRN, the IP address of any MGEN container attached to that SRN will fall within the subnet 192.168.(100+SRN_ID).0/24. The 6-byte MAC address is constructed by prepending the IP address in hexadecimal with “02:10”.
So for example, on SRN 35, the traffic subnet would be 192.168.135.0/24 and IP addresses and hardware addresses are assigned as follows:
192.168.135.1 --> 02:10:c0:a8:87:01
192.168.135.2 --> 02:10:c0:a8:87:02
192.168.135.3 --> 02:10:c0:a8:87:03
... and so on
In order to ensure that your scenario traffic is not delayed due to ARP resolution, the Colosseum Team recommends you implement static ARP tables within your LXC images following the scheme described above. Since the implementation of this may be user-specific, the Colosseum Team will not be directly supporting or providing any particular static ARP table implementation, but if you feel that there is an issue with the Colosseum software or traffic generation system, please feel free to reach out to us via a help desk ticket. You may also choose to pass the dummy packet described above which will also help to ensure that all addresses are dynamically resolved prior to the start of scored traffic. Finally, please be aware that depending on your implementation, the ARP cache may time out, so please be aware of this.
Summary Diagram
This diagram shows a high-level overview of the flow described above. The MGEN sender and listener are the traffic source and destination, respectively, and the SRNs in this diagram are effectively acting as routers in order to pass traffic over the RF network subnet.
Detailed Diagram
This diagram expands on the summary diagram with more detail on an example container configuration. Users are not bound to this architecture and may implement a different solution. This is intended to provide an example with detail on the interfaces within and external to the container. Note that the tap0 interface must be created by the user within the container and the IP address assigned is defined by the users.
Selecting an IP Addressing Scheme for your Radio Network
There are many methods users may wish to use in order to assign IP addresses to the nodes in their radio network. One method can be to use the SRN number which can be found by running ‘hostname’ from within the container. Container hostnames are in the format teamname-srnXXX. Users can run the following bash command to extract the SRN number from their hostname:
~$ hostname | grep -oE "[0-9]{1,3}$"
This can be appended to a static subnet, e.g., 172.20.22.0/24, to create a unique IP address. For example, a reservation with SRNs 3 and 6 would assign the IP addresses 172.20.22.3 and 172.20.22.6 to the radio interfaces. Similar approaches can be used to assign a unique MAC or hardware address if one is needed.
Configuring a TUN/TAP Interface In Your LXC Container
A TUN/TAP interface is a Layer2/Layer3 virtual interface with direct hooks into the operating system’s network stack. For Colosseum traffic generation, a TUN/TAP interface acts as a bridge between the operating systems standard network stack (Ethernet and IP) and the RF domain (e.g., GnuRadio). Effectively providing an IP-over-RF capability.
To configure a TUN/TAP in your Container, follow the steps below.
NOTE: Repeat these steps for every Container; however, adjust the MAC address and IP address accordingly so that they are unique. For example, in Container one, use a MAC address of 12:34:56:78:90:01 and IP address of 172.20.22.1/24. In Container two, use 12:34:56:78:90:02 and 172.20.22.2/24. In Container three, use 12:34:56:78:90:03 and 172.20.22.3/24. And so on.
Create the TAP interface:
sudo ip tuntap add dev tap0 mode tap
Setup and bring up the TAP interface:
ifconfig tap0 hw ether 12:34:56:78:90:01 ifconfig tap0 mtu 1500 ifconfig tap0 172.20.22.1/24 ifconfig tap0 up
TAP configuration is complete. To verify TAP interface, run ifconfig:
ifconfig tap0
You should see tap0 and all its parameters should match the earlier ifconfig commands.
Installing Quagga In Your LXC Container
There are a number of ways in which users can implement IP-layer routing in their applications. A simple solution is Quagga which provides a solution which is ready nearly out-of-the-box. For more information on Quagga, see: http://www.nongnu.org/quagga/.
Quagga is a software-based routing suite for Linux systems – effectively allowing your Linux computer to participate and advertise IP routing with other routing devices. In regards to Coliseum traffic generation, Quagga is used to advertise a Container’s traffic subnets with other Container’s across the RF network. Thus allowing a Container to dynamically discover the next-hop IP address of traffic.
To install Quagga in your Container, follow the steps below. NOTE: After installing and configuring quagga, users may save and upload the image so it can be used for future reservations.
Install the Quagga software using the Apt repositories:
sudo apt-get install quagga
Enable Quagga’s Zebra and RIPd daemons:
vim /etc/quagga/daemonsChange the contents of ‘daemons’ so it matches the following:
zebra=yes bgpd=no ospfd=no ospf6d=no ripd=yes ripngd=no
Configure the Zebra daemon:
vim /etc/quagga/zebra.confChange the contents of ‘zebra’ so it matches the following:
! -*- zebra -*- ! ! zebra sample configuration file ! ! $Id: zebra.conf.sample,v 1.1 2002/12/13 20:15:30 paul Exp $ ! hostname Router password zebra enable password zebra ! ! Interface's description. ! interface tap0 multicast interface tr0 multicast ! ! ! Static defaultroute sample. ! !ip route 0.0.0.0/0 203.181.89.241 ! ! log file /var/log/quagga/zebra.log
Configure the RIPd daemon:
vim /etc/quagga/ripd.confChange the contents of ‘ripd’ so it matches the following:
! -*- rip -*- ! ! RIPd sample configuration file ! ! $Id: ripd.conf.sample,v 1.1 2002/12/13 20:15:30 paul Exp $ ! hostname ripd password zebra ! ! debug rip events ! debug rip packet ! router rip network tap0 network tr0 redistribute kernel ! route 10.0.0.0/8 ! distribute-list private-only in eth0 ! !access-list private-only permit 10.0.0.0/8 !access-list private-only deny any ! log file /var/log/quagga/ripd.log ! !log stdout
Restart Quagga:
service quagga stop service quagga start
Quagga configuration is complete. To verify operation, run tcpdump on the tap0 interface:
tcpdump -v -i tap0
You should see periodic RIPv2 advertisement messages, advertising the Container’s tr0 IP subnet.
Bring Up the RF Domain
Now that the TUN/TAP interface and Quagga routing software is up and running, bring up your Container’s RF modems. Quagga’s RIP route advertisements will most likely be the first IP traffic flowing across the RF domain. To verify successful route advertisements, use the ‘route’ utility:
On a Container, run route:
route -nIf the RIP route advertisements are successful, then the output of route will show the tr0 IP subnets of the other Containers. These routes will have a non-zero metric.
The final verification is to ping the other Container’s tap0 IP address. If the pings are successful, then the Containers are successfully communicating with each other using IP over the RF domain.