Tungsten Fabric as the predecessor of OpenSDN was widely known for its support of dual IP stack. Due to the dual IP addresses situation in the networking world today it is vital for corresponding programs to support both formats: on the one hand, IPv4 address space was exhausted several years ago and IPv6 is the only solution for modern industrial applications, but on the other hand many companies still use IPv4 addresses in private and public networks and they will continue to require them for some time to come.

The dual IP stack feature highlights OpenSDN/Tungsten Fabric/OpenContrail in comparison with other SDN platforms because it provides support for modern network applications while preserving backward compatibility with older IPv4 services.

The transition to IPv6 protocol support in Tungsten Fabric had not been 100% complete and some valuable network functions were limited to 4-byte IP addresses. Some restrictions were alleviated in the latest R24.1 release of OpenSDN:

• IPv6 Metadata service support had been introduced, making it possible to utilize pure IPv6 virtual networks without technical/auxiliary IPv4 networks, commits feee59b, feee59b, f6c2579, 61c062d;
• NAT66 (IPv6 to IPv6 Network Address Translation) had been implemented to allow floating IP for IPv6, commits 34c6468, ef617f8;
• virtual DNS service had been extended to work with IPv6 floating IP, commit eeba013;
• the code base had been updated to enable server foundation classes instances (TcpServer, HttpServer, etc) binding to IPv6 addresses, commit c0527f4.

The first release of OpenSDN has seen the world!

With this realease OpenSDN starts its own story as a separate project. It is a successor of Tungsten Fabric, OpenContrail and Contrail, but having it's own highly awaited features, such as:

• an enhanced VxLAN control plane;
• python 3 support;
• CQL Cassandra driver;
• C++11 - compliant source code;
• and many others.

The official release page can be found here. Other links:

• Dockerhub
• Telegram group
• GitHub group
• LinkedIn group

Implementation of NAT66 (IPv6 NAT) in Tungsten Fabric

Introduction

Network Address Translation (NAT) technique is widely used in IPv4 networks. This approach allows to connect hosts in private networks to the Internet, saves IPv4 addresses (which are now steadily becoming more and more expensive resource) and adds additional layer of network protection and security for customers.

With the advent of IPv6, the main reason (exhaustion of network addresses) has become obsolete and this network feature is not supported by many forwarding and routing devices. However, NAT protocol for IPv6 (NAT66) might be used in networks due to some other reasons, related to clouds exploitation. Today (October, 2, 2023), Tungsten Fabric vRouter Forwarder is not an exception and while it allows to perform all types of NAT (source/destination network address translation, source/destination port address translation(PAT)) for IPv4 standard, there are no similar tools for IPv6 standard.

The described here vRouter code modifications allow to enable NAT66 functionality in Tungsten Fabric and also show how to introduce changes into vrouter.ko module.

Code changes

The implementation of NAT66 for Tungsten Fabric will require modification of it's control and data planes components: vRouter Agent and vRouter Forwarder. Before we proceed to description of needed code changes, it is necessary to give brief description of loading the vRouter Forwarder modified source code. Afterwards, ...

How to load custom vrouter code into memory

vRouter Forwarder kernel module is compiled and loaded into the memory using special Docker container, therefore we should understand how to inject the modified source code into this container and how to run it. This procedure involves next 5 steps.

1. Prepare original sources as tarball. During this step we should create a compressed tarball which has an extension .tgz and contains next directories (with corresponding files):

   • dp-core, system - independent source code for vrouter.ko;
   • include, header files for vrouter.ko;
   • linux, system - dependent (linux) source code for vrouter.ko;
   • sandesh, source code for kernel-to-user interaction functionality; It is necessary to note here that content of sandesh directory is generated automatically during procedure of building from sources (tf-dev-env repository);
   • utils, source code for utilities;
   • GPL-2.0.txt, a version of the corresponding GPL license;
   • LICENSE, a license text;
   • Makefile, a GNU Makefile system instuction to build vrouter.ko.

2. Prepare a Dockerfile with necessary changes. New Dockerfile should contain list of all files that are to be added into the new docker image. For instance, in our example we should load previous default Docker image instructions and then we should update files: vr_flow.h, vr_flow.c, vr_proto_ip6.c:

   FROM the_name_of_original_image ADD ./the_name_of_tarball_with_sources.tgz /vrouter_src/ ADD ./vr_flow.h /vrouter_src/include/vr_flow.h ADD ./vr_flow.c /vrouter_src/dp-core/vr_flow.c ADD ./vr_proto_ip6.c /vrouter_src/dp-core/vr_proto_ip6.c

3. Stop previous services. Before loading the modified image into the memory, it is necessary to remove the old one: devices should be removed, dockers and modules should be stopped and removed. A script to do this work might look as:

    #!/usr/bin/bash
    
    ifconfig vhost0 down
    I=0
    N=3 # if pkt devices are numbered from 0 to 3
    for I in `seq 0 $N`
    do
        ifconfig pkt$I down
    done
   
    docker stop vrouter_vrouter-agent_1  # stop vRouter Agent container
    docker rm -f vrouter_vrouter-agent_1 # rm  vRouter Agent container
    
    docker stop vrouter_vrouter-kernel-init_1 # stop vRouter Forwarder Init
    docker rm -f vrouter_vrouter-kernel-init_1 # rm vRouter Forwarder Init
    
    rmmod vrouter
    
    kernels=`ls -d /lib/modules/[0-9]*` # remove all vrouter.ko for all kernels
    for k in $kernels
    do
        rm -rf $k/updates/dkms/vrouter.ko
    done
   

4. Load/create new docker image using common Docker commands:

    # remove previous version of the container
    docker image rm --force kmod_init:R2011-latest
    #
    docker image build -t kmod_init:R2011-latest .
    docker run  --mount type=bind,src=/usr/src,dst=/usr/src  --mount type=bind,src=/lib/modules,dst=/lib/modules  kmod_init:R2011-latest > run_log 2>&1
   

5. Load to memory. For this purpose you can use for example docker-compose:

    docker-compose -f ./docker-compose.yaml up -d
   

vRouter Agent changes

To enable NAT for IPv6 we will firstly modify PktFlowInfo class (pkt_flow_info.cc) methods:

• PktFlowInfo::FloatingIpDNat;
• PktFlowInfo::FloatingIpSNat;
• PktFlowInfo::UpdateFipStatsInfo.

Functions PktFlowInfo::FloatingIpDNat and PktFlowInfo::FloatingIpSNat are responsible for setting up Source NAT (SNAT) and Destination NAT forwarding on vRouter Forwarder. These function are executed per each flow entry (a FlowEntry entry object) and program vRouter Forwarder settings for that flow entry.

In method PktFlowInfo::FloatingIpDNat we replace

if (pkt->ip_daddr.to_v4() == vm_port->primary_ip_addr()) {
    return;
}

with

if (pkt->ip_daddr.is_v4()) {
    if (pkt->ip_daddr.to_v4() == vm_port->primary_ip_addr()) {
        return;
    }
}
if (pkt->ip_daddr.is_v6()) {
    if (pkt->ip_daddr.to_v6() == vm_port->primary_ip6_addr()) {
        return;
    }
}

which means that we check for the coincidence of a virtual machine primary address with the Floating IP address both for IPv6 and IPv4 formats.

Then we replace

if (pkt->ip_daddr.to_v4() != it->floating_ip_) {
    continue;

}

with

if (pkt->ip_daddr != it->floating_ip_) {
    continue;
}

And right after if (underlay_flow) { we add

if (pkt->ip_daddr.is_v6()) {
    return;
}

because Tungsten Fabric doesn't support IPv6 for underlay for the time being.

In function PktFlowInfo::FloatingIpSNat we remove next code to enable SNAT:

if (pkt->family == Address::INET6) {
    return;
    //TODO: V6 FIP
}

Then we relplace

if (it->fixed_ip_ != Ip4Address(0) && (pkt->ip_saddr != it->fixed_ip_))  {

with

if (it->fixed_ip_ != IpAddress() && (pkt->ip_saddr != it->fixed_ip_))  {

to make comparison in the if condition to be agnostic of IP version number.

There is some issue regarding NAT (NAT66) + VxLAN. Unfortunately, NAT and NAT66 doesn't work well with VRF NextHop, therefore we might go two ways:

• the adjustment of ACL;
• the adjustement of the source code.

Both solutions are described in the corresponding note (NAT + VxLAN in Tungsten Fabric). Solution number 2 (with the adjustement of the source code) seems simpler, because adjustment of ACL for cases with UDP/TCP might be cumbersome.

Finally, we should update PktFlowInfo::UpdateFipStatsInfo function to enable statics collection for IPv6 floating IP. However, these changes don't affect forwarding functionality of TF and they are not considered here.

vRouter Forwarder changes

Next, when we have modified vRouter Agent to program vRouter Forwarder, we should teach the latter one to work with IPv6 packets destined for NAT processing.

NAT processing of IP packets in vRouter Forwarder is attached to flow forwarding. Each time when flow is identified for a given combination of IP addresses, ports and protocol type, vRouter Forwarder applies the address and port translation according to settings specified by vRouter Agent. A function that performs the translation for IPv4 is called from vr_flow_nat. The whole process of flow indentification and configuration is triggered by vr_do_flow_lookup which is invoked:

• on arrival of a packet from virtual interfaces (vr_virtual_input);
• before a nexthop execution (nh_output) if a flow was not indentified previously;
• in the nexthop that is responsible for processing of IP packets (vr_ip_rcv), if that was not done previously;
• on an IP packet input processing (vr_ip_input) if that was not done previously;
• on a node flush (vr_flow_flush_pnode).

A full call stack from it's start in vr_flow_forward till NAT processing is next:

 vr_flow_forward
       |            +--> vr_inet6_flow_lookup --+
       v            |                           |
vr_do_flow_lookup --+                           +->- vr_flow_lookup
                    |                           |           |
                    +--> vr_inet_flow_lookup ---+           |
                                                            |
                                                            v
          vr_flow_nat <-- vr_flow_action_default <-- vr_flow_action
              |
              v
       vr_inet_flow_nat

To conform to the coding practices of TF vRouter Forwarder, a new procedure of NAT IPv6 packets processing is placed into function vr_inet6_flow_nat, which is called from vr_flow_nat.

The new function has been created on the basis of the original IPv4 NAT processing function (vr_inet_flow_nat) by accounting next peculiarities of IPv6 protocol:

• IPv6 packet header format is kept in different structure (vr_ip6);
• an IP packet payload checksum is not stored anymore in the IPv6 header;
• IPv6 addresses are stored in 16-byte arrays instead of 4-byte variables for IPv4.

Finally, in case we have TCP or UDP packets as a payload of our IP6 packet, we should update it's checksum after address translation. For this operation a function vr_ip6_update_csum (by analogy to original vr_ip_update_csum) has been created. Listings of all code changes are shown below.

• Changes to vr_flow_nat function: if (pkt->vp_type == VP_TYPE_IP6) return vr_inet6_flow_nat(fe, pkt, fmd);

• vr_inet6_flow_nat function code: flow_result_t vr_inet6_flow_nat(struct vr_flow_entry *fe, struct vr_packet *pkt, struct vr_forwarding_md *fmd) { uint32_t ip_inc = 0, port_inc = 0; uint16_t *t_sport = NULL, *t_dport = NULL; const uint8_t dw_p_ip6 = VR_IP6_ADDRESS_LEN / sizeof(uint32_t); uint8_t i = 0;

        struct vrouter *router = pkt->vp_if->vif_router;
        struct vr_flow_entry *rfe = NULL;
        struct vr_ip6 *ip6 = NULL, *icmp_pl_ip6 = NULL;
        struct vr_icmp *icmph = NULL;
  
    if (fe->fe_rflow < 0)
            goto drop;
  
        rfe = vr_flow_get_entry(router, fe->fe_rflow);
        if (!rfe)
            goto drop;
  
        ip6 = (struct vr_ip6 *)pkt_network_header(pkt);
        if (!ip6)
            goto drop;
  
        if (ip6->ip6_nxt == VR_IP_PROTO_ICMP6) {
            icmph = (struct vr_icmp *)((char *)ip6 + sizeof(struct vr_ip6));
  
            if (vr_icmp_error(icmph)) {
                icmp_pl_ip6 = (struct vr_ip6 *)(icmph + 1);
                if (fe->fe_flags & VR_FLOW_FLAG_SNAT) {
                    memcpy(icmp_pl_ip6->ip6_dst, rfe->fe_key.flow6_dip,
                        VR_IP6_ADDRESS_LEN);
                }
  
                if (fe->fe_flags & VR_FLOW_FLAG_DNAT) {
                    memcpy(icmp_pl_ip6->ip6_src, rfe->fe_key.flow6_sip,
                        VR_IP6_ADDRESS_LEN);
                }
  
                t_sport = (uint16_t *)((uint8_t *)icmp_pl_ip6 +
                    sizeof(struct vr_ip6));
                t_dport = t_sport + 1;
  
                if (fe->fe_flags & VR_FLOW_FLAG_SPAT) {
                    *t_dport = rfe->fe_key.flow6_dport;
                }
                if (fe->fe_flags & VR_FLOW_FLAG_DPAT) {
                    *t_sport = rfe->fe_key.flow6_sport;
                }
            }
        }
  
        if ((fe->fe_flags & VR_FLOW_FLAG_SNAT) &&
                (memcmp(ip6->ip6_src, fe->fe_key.flow6_sip, VR_IP6_ADDRESS_LEN) == 0)) {
            for (i = 0; i < VR_IP6_ADDRESS_LEN; i += dw_p_ip6) {
                vr_incremental_diff( *((uint32_t*)(ip6->ip6_src + i)),
                    *((uint32_t*)(rfe->fe_key.flow6_dip + i)), &ip_inc);
            }
            memcpy(ip6->ip6_src, rfe->fe_key.flow6_dip,
                VR_IP6_ADDRESS_LEN);
        }
  
        if (fe->fe_flags & VR_FLOW_FLAG_DNAT) {
            for (i = 0; i < VR_IP6_ADDRESS_LEN; i += dw_p_ip6) {
                vr_incremental_diff( *((uint32_t*)(ip6->ip6_dst + i)),
                    *((uint32_t*)(rfe->fe_key.flow6_sip + i)), &ip_inc);
            }
            memcpy(ip6->ip6_dst, rfe->fe_key.flow6_sip,
                VR_IP6_ADDRESS_LEN);
        }
  
        if (vr_ip6_transport_header_valid(ip6)) {
            t_sport = (uint16_t *)((uint8_t *)ip6 +
                    sizeof(struct vr_ip6));
            t_dport = t_sport + 1;
  
            if (fe->fe_flags & VR_FLOW_FLAG_SPAT) {
                vr_incremental_diff(*t_sport,
                    rfe->fe_key.flow6_dport, &port_inc);
                *t_sport = rfe->fe_key.flow6_dport;
            }
  
            if (fe->fe_flags & VR_FLOW_FLAG_DPAT) {
                vr_incremental_diff(*t_dport,
                    rfe->fe_key.flow6_sport, &port_inc);
                *t_dport = rfe->fe_key.flow6_sport;
            }
        }
  
        if (!vr_pkt_is_diag(pkt))
            vr_ip6_update_csum(pkt, ip_inc, port_inc);
  
        if ((fe->fe_flags & VR_FLOW_FLAG_VRFT) && pkt->vp_nh &&
                ((pkt->vp_nh->nh_vrf != fmd->fmd_dvrf) ||
                (pkt->vp_nh->nh_flags & NH_FLAG_ROUTE_LOOKUP))) {
            /* only if pkt->vp_nh was set before... */
            pkt->vp_nh = vr_inet6_ip_lookup(fmd->fmd_dvrf, ip6->ip6_dst);
        }
  
        return FLOW_FORWARD;
  
    drop:
        PKT_LOG(VP_DROP_FLOW_NAT_NO_RFLOW, pkt, 0, VR_PROTO_IP6_C, __LINE__);
        vr_pfree(pkt, VP_DROP_FLOW_NAT_NO_RFLOW);
        return FLOW_CONSUMED;
    }
  

• vr_ip6_update_csum function code: static void vr_ip6_update_csum(struct vr_packet *pkt, uint32_t ip_inc, uint32_t port_inc) { struct vr_ip6 *ip6 = NULL; struct vr_tcp *tcp = NULL; struct vr_udp *udp = NULL; uint32_t csum_inc = ip_inc; uint32_t csum = 0; uint16_t *csump = NULL;

        ip6 = (struct vr_ip6 *)pkt_network_header(pkt);
  
        if (ip6->ip6_nxt == VR_IP_PROTO_TCP) {
            tcp = (struct vr_tcp *)((uint8_t *)ip6 + sizeof(struct vr_ip6));
            csump = &tcp->tcp_csum;
        } else if (ip6->ip6_nxt == VR_IP_PROTO_UDP) {
            udp = (struct vr_udp *)((uint8_t *)ip6 + sizeof(struct vr_ip6));
            csump = &udp->udp_csum;
            if (*csump == 0) {
                return;
            }
        } else {
            return;
        }
  
        if (vr_ip6_transport_header_valid(ip6)) {
            /*
            * for partial checksums, the actual value is stored rather
            * than the complement
            */
            if (pkt->vp_flags & VP_FLAG_CSUM_PARTIAL) {
                csum = (*csump) & 0xffff;
            } else {
                csum = ~(*csump) & 0xffff;
                csum_inc += port_inc;
            }
            csum += csum_inc;
            if (csum < csum_inc)
                csum += 1;
  
            csum = (csum & 0xffff) + (csum >> 16);
            if (csum >> 16)
                csum = (csum & 0xffff) + 1;
  
            if (pkt->vp_flags & VP_FLAG_CSUM_PARTIAL) {
                *csump = csum & 0xffff;
            } else {
                *csump = ~(csum) & 0xffff;
            }
        }
  
        return;
    }
  

Tests

The implementation of address translation for IPv6 is tested for NAT and PAT cases separately. Since TF is able to use both MPLS and VxLAN technologies, all tests are performed for each overlay type. When connecting virtual machines reside on one hypervisor, transferred data between them is not encapsulated. When virtual machines belong to different computes, a tunnel is established between them. Therefore, 2 configurations should be considered for each overlay type: without tunnel (Interface) and with tunnel. For each configuration 3 types of L4 packets are considered: ICMP, TCP, UDP.

An inter-VRF packet forwarding is tested additionally for VxLAN overlay.

Drop of packets for cases where there is no reverse flow is verified separately.

Finally, we have the next map of trials:

1	2	3	4	5
1	NAT	MPLS	Intf	ICMP
2	NAT	MPLS	Tunn	ICMP
3	NAT	MPLS	Intf	TCP
4	NAT	MPLS	Tunn	TCP
5	NAT	MPLS	Intf	UDP
6	NAT	MPLS	Tunn	UDP
7	PAT	MPLS	Intf	TCP
8	PAT	MPLS	Tunn	TCP
9	PAT	MPLS	Intf	UDP
10	PAT	MPLS	Tunn	UDP
11	NAT	VxLAN	Intf	ICMP
12	NAT	VxLAN	Tunn	ICMP
13	NAT	VxLAN	Intf	TCP
14	NAT	VxLAN	Tunn	TCP
15	NAT	VxLAN	Intf	UDP
16	NAT	VxLAN	Tunn	UDP
17	PAT	VxLAN	Intf	TCP
18	PAT	VxLAN	Tunn	TCP
19	PAT	VxLAN	Intf	UDP
20	PAT	VxLAN	Tunn	UDP

Columns of the table have next meaning:

1. a trial's number;
2. an address translation type (NAT/PAT);
3. an overlay type (MPLS/VxLAN);
4. an inter-VM connection type (Intf corresponds to interface routes, Tunn corresponds to tunnel routes);
5. an L4 packet type (ICMP/TCP/UDP).

MPLS configuration

We have two VM's created, for example, using OpenStack. These machines reside on compute2. Each machine has: one public IPv4 interface for interaction with an external world and one private IPv6 interface for inter-VM connection.

It is expected that both interfaces are configured properly (i.e., ifconfig shows they are present and ip -6 r gives correct routes).

IPv6 interfaces are connected to a TF IPv6 virtual network (let say U6-net) with subnet aaaa::/32. One interface has address aaaa::11, another has address aaaa::21. The interface with an address aaaa::11 is linked to Floating IP with address cccc::11.

For ICMP tests we run ping command from aaaa::21 interface to cccc::11:

ping -6 cccc::11

For TCP tests we run on the machine with aaaa::11 (and cccc::11) interfaces:

nc -6 -nlv aaaa::11  8888

And on the machine with aaaa::21 interface:

nc -6 -nv cccc::11  8888

For UDP tests we run nc with the -u key.

When PAT capabilities are tested, next settings are to apply:

• Floating IP port mappings should be enabled in TF for cccc::11 (Floating IP --> Floating IP port mappings --> (src, protocol, dst). Where src is a port number on the client side (who requests a connection), dst is a port number on the server side) and protocol is a name of protocol (TCP or UDP);
• the assigned src port (e.g., 7777) should be used with the nc command in the virtual machine with the interface aaaa::21 and the assigned dst port (e.g., 8888) value should be used with nc command in the virtual machine with the interface aaaa::11.

For cases with tunnels betweens VM's, another VM with interface address aaaa::41 that is located on different compute node is used for outgoing ping and nc connections.

Security groups for all ports should allow egress and ingress connections for ICMP, TCP and UDP protocols.

For TCP and UDP inter-VRF tests, the permissive policy (e.g., pass for all addresses, ports and protocols) should be enabled for both networks (VRF instances) and for routing network (routing VRF instance) if we use ACL's.

Due to issues with checksum calculations in TF vrouter.ko module, it is recommended to turn it of on all computes:

ethtool -K ethX tx off rx off

Where ethX is a name of the ethernet interface that is connected to the vhost0 interface.

firewalld, iptables, ip6tables are disabled.

Packet mode is turned off for all ports, reverse flow forwarding is on for all networks.

VxLAN configuration

For the VxLAN configuration settings are analogous to MPLS configuration, however:

• interfaces of virtual machines are distributed among virtual networks: U6-net and V6-net;
• U6-net has subnet aaaa::/32 and V6-net has subnet bbbb::/32;
• interfaces of V6-net are used to create outgoing ICMP/TCP/UDP messages, interfaces of U6-net accept connections/messages from V6-net.

Networks U6-net and V6-net with respective subnets aaaa::/32 and bbbb::/32 are connected to a single VxLAN logical router with some predefined VNI - let's say, 112233.

To test ICMP and TCP/UDP packets, commands ping and nc are used.

An example of a packet path from when travelling via VxLAN from one VRF instance (V6-net) to another (U6-net) between virtual machines on different computes (VxLAN + VRF-T + NAT66):

compute 0

tap
|
v
bbbb::41       aaaa::/32      aaaa::31
-------- --->  --------- ---> --------
V6-net VRF     V6-net VRF     Rt VRF
                                |
                                v
                            ethX compute0
- - - - - - - - - - - - - - - - - - - - -
                            ethX compute2
                                |
                                v
                aaaa::11  NAT aaaa::31
       tap <--- -------- <--- --------
                 Rt VRF        Rt VRF
    
       compute 2

NAT66 Useful links

1. https://blogs.infoblox.com/ipv6-coe/you-thought-there-was-no-nat-for-ipv6-but-nat-still-exists/
2. https://raghavendrahiremathg.medium.com/ipv4-ipv6-tcp-udp-checksum-fd7056286562
3. https://stackoverflow.com/questions/30858973/udp-checksum-calculation-for-ipv6-packet
4. https://networkengineering.stackexchange.com/questions/68161/ipv6-extension-headers-and-udp-tcp-icmpv6-checksums
5. https://www.packetmania.net/en/2021/12/26/IPv4-IPv6-checksum/

The Internal Structure of Routge Tables of Tungsten Fabric vRouter Agent

Tungsten Fabric vRouter Agent component partially belongs to data plane and control planes, which fits well in the architecture proposed by Open Networking Foundation (ONF) [ONF_SDN_ARCH]. Therefore, it has dual role:

• it programmes vRouter Forwarder to forward packets between virtual machines of a hypervisor and external routers;
• it provides information for an SDN controller component about local routes to virtual machines of a hypervisor.

The process of packets forwarding in vRouter is performed in accordance with:

• Bridge table for L2 addressing (BridgeAgentRouteTable class);
• Inet unicaste table for L3 addressing (InetUnicastAgentRouteTable class).

There are also other tables that carry auxiliary functions: EVPN tables, multicast tables, nexthop table, interface table, etc. Nonetheless, all tables hold the same structure and derive from the same parent class. For example, on figure 1 an inheritance diagram is shown for the special branch of table classes (AgentRouteTable) that are used to represent forwarding information in vRouter Agent.

Figure 1.

Each table (class DBTable) in vRouter Agent is represented as a partitioned tree. Partitioning is organized as array (std::vector) of partitions (DBTablePartition class). And each partition contains a red-black tree of records that constitutes the table. For the implementation of a red-black tree, boost::intrusive::set template class of boost library is used.

Therefore, route tables in vRouter Agent (just like all other tables) are represented as partiotioned sets of records, where each record represents a route.

Each route in vRouter Agent is represented with class AgentRoute that holds information about a destination address (an IP prefix for L3 routes or a MAC address for L2 routes) and the list of paths that can be used to reach the destination.

A path to the destination is represented by:

• AgentPath class that holds information about a nexthop (NextHop class);
• a special marker that is called peer (Peer class).

When a packet travels from one virtual machine to another one, it passes through the sequence of nexthops, each of them directs the packet to another destination (router, network, etc) or processes it (encapsulates or decapsulates, for example). Examples of nexthops are:

• the interface (class InterfaceNH) nexthop directs a packet to the specified TAP (Terminal Access Point), therefore, usually it is the last nexthop on the route of a packet;
• the tunnel (class TunnelNH) nexthop encapsulates a packet (for example, in UDP packet) and directs it towards an external interface of a hypervisor;
• the vrf (class VrfNH) nexthop changes VRF instance of packet.

A peer marker is used to distinguish an origin of a path or it's purpose. Examples of peers are:

• local vm port peer shows that a path was created internally and points to a local interface (TAP usually);
• bgp peer depicts that a path arrived to vRouter Agent from a controller.

Let's traverse an EVPN Type2 table to illustrate this structure of a route table. If we want to traverse a route table in vRouter Agent to print its content, we should organize 2 cycles (see listing 1):

• one cycle walks through routes (records of table);
• one cycle loops through each path in a record.

RTTI is used to cast results of base DBTable class functions to needed final classes (e.g., EvpnRouteEntry as type of a route table record). In the current implementation of vRouter Agent, partitioning for AgentRouteTable is turned off and, hence during walk we use only partition number 0. However, in other table several partitions might be used and in this situation there are two choices:

1. organization of additional loop over table partitions;
2. employment of special methods of DBEntryBase class that returns partitions.

As it is seen of figure 2, we have:

• interface nexthops for IP addresses 10.1.1.0, 10.1.1.3, 10.1.1.31, 10.1.1.41 10.1.1.201, 10.100.100.0;
• tunnel nexthops for routes to 10.0.0.250, 10.1.1.0, 10.1.1.10, 10.10.10.0, 10.100.100.0;
• composite nexthops for routes to 10.1.3, 10.1.1.201.

Some routes were created by vRouter Agent of a hypervisor (LocalVmExportPeer peer type), and some routes were imported from SDN controller.

Figure 2.

Listing 1:

VrfEntry* routing_vrf = const_cast<VrfEntry*>(const_vrf);
EvpnAgentRouteTable *evpn_table = dynamic_cast<EvpnAgentRouteTable *>
    (routing_vrf->GetEvpnRouteTable());
if (evpn_table == NULL) {
    std::cout<< "VxlanRoutingManager::PrintEvpnTable"
             << ", NULL EVPN tbl ptr"
             << std::endl;
    return;
}
EvpnRouteEntry *c_entry = dynamic_cast<EvpnRouteEntry *>
    (evpn_table->GetTablePartition(0)->GetFirst());
if (c_entry) {
    if (c_entry->IsType5())
        std::cout << "Evpn Type 5 table:" << std::endl;
    else
        std::cout << "Evpn Type 2 table:" << std::endl;
}
while (c_entry) {
    const Route::PathList & path_list = c_entry->GetPathList();
    std::cout<< "  IP:" << c_entry->ip_addr()
             << ", path count = " << path_list.size()
             << ", ethernet_tag = " << c_entry->ethernet_tag()
             << std::endl;
    for (Route::PathList::const_iterator it = path_list.begin();
        it != path_list.end(); ++it) {
        const AgentPath* path =
            dynamic_cast<const AgentPath*>(it.operator->());
        if (!path)
            continue;
        std::cout<< "    NH: "
                    << (path->nexthop() ? path->nexthop()->ToString() :
                    "NULL")
                    << ", " << "Peer:"
                    << (path->peer() ? path->peer()->GetName() : "NULL")
                    << std::endl;
        if (path->nexthop()->GetType() == NextHop::COMPOSITE) {
            CompositeNH *comp_nh = dynamic_cast<CompositeNH *>
                (path->nexthop());
            std::cout<< "    n components="
                        << comp_nh->ComponentNHCount()
                        << std::endl;
        }
    }
    if (evpn_table && evpn_table->GetTablePartition(0))
        c_entry = dynamic_cast<EvpnRouteEntry *>
            (evpn_table->GetTablePartition(0)->GetNext(c_entry));
    else
        break;
}

Literature

[ONF_SDN_ARCH] SDN architecture, Issue 1. Open Networking Foundation, 2014.

---

NAT + VxLAN overlay in Tungsten Fabric

'ppose, you want to implement in Tungsten Fabric a network configuration that allows interconnection between internal nodes (virtual machines) from different TF networks. This can be accomplished with employment of a VxLAN feature of Tungsten Fabric. All networks connected to a Logical Router (LR) expose their interface routes to the exchange (or routing) Virtual Routing and Forwarding (VRF) instance making them public for all hosts, which connect to this LR. Private IP addresses of virtual machines can be hidden using special TF tools like Allowed Address Pairs (AAP) or Floating IP (FIP) which is actually a synonym to the NAT technology.

However, there are two issues that hinder usage of all TF powerfull features in this scenario:

1. if we want to hide the private IP of a connected Virtual Machine, we can use only L3-L2 Allowed Address Pairs, because original TF VxLAN manager handles only interface (or composite of interfaces) L3-L2 routes;
2. TF VxLAN manager uses VRF Translation technique to direct a packet from the one VRF instance into another one and this technique is not compatible with NAT tools of the SDN.

The first issue can be solved with the new experimental VxLAN joint, which is available in the master branch of Tungsten Fabric controller: https://github.com/tungstenfabric/tf-controller. The new VxLAN implementation allows to copy from bridge VRF instances into routing VRF instance all kinds of interface (and composite of interfaces) routes, including:

• basic IP instances;
• Floating IP;
• L3 and L3-L2 AAP;
• Interface Routing Tables (IRT);
• BGP-as-a-Service (BGPaaS);
• and other.

The second problem can be resolved by means of another Tungsten Fabric tool: Access Control List (ACL). Implementation of NAT in Tungsten Fabric uses information from the endpoint interface of a path (interface nexthop) to set up modification of a forwarded IP packet header. Specifically, NAT works in TF only when the first nexthop of a route is of interface type. When the first nexthop in a path from a bridge VRF instance to the routing VRF instance is not an interface but the vrf, then this prohibits usage of NAT.

However, the solution with ACL might be cumbursome when it is neccessary to implement forwarding of packets for several different protocols and many FIPs. Therefore, an alternative solution, which changes code of vRouter Agent is proposed at the end of the material.

In this case we can use ACL to transfer the packet from one VRF instance (bridge) into another VRF instance (routing) where the needed interface nexthop is present and condition for NAT is met.

Now as general considerations about configuration have been set out, let's consider a practical example. We're going to use master branch of TF controller with OpenStack for virtualization orchestration. Networks connectivity sketch is presented here:

+--------------------------------------------------------+
|                      hypervisor                        |
|   +---+                                        +---+   |
|   |VM1|                                        |VM2|   |
|   +---+                                        +---+   |
+-----|--------------------------------------------|-----+
      |                                            |
    +---+   FIP pool  +---+ VxLAN  +----+  VxLAN +---+
    | U |  ---------- | W | ------ | LR | ------ | V |
    +---+             +---+        +----+        +---+
   bridge             bridge       routing      bridge
   network            network      network      network

Let's say a virtual machine VM1 is connected to virtual network U ( subnet mask 10.1.1.0/24) via the IP address 10.1.1.21 and a virtual machine VM2 is connected to virtual network V (subnet mask 10.2.2.0/24) via the IP address 10.2.2.21. 'ppose, these virtual machines belong to a hypervisor compute2 and we want to connect them using VxLAN. Moreover, let's envsion also that we want to hide 10.1.1.0/24 addresses from V network. In this case we create network W, which will keep Floating IP addresses and connect it to our LR. We also connect network V to the LR.

NB. To enable Floating IP pool for W network, we should mark it as external in it's advanced properties. On the flip side, TF GUI doesn't allow to connect external networks to a LR. To overcome this, configuration steps should be executed in the next sequence:

• first, a network must be connected to the LR;
• then this network must be marked as external.

After that, TF GUI will hide this external network from the list of connected to the LR networks, but actually it will stay there.

To implement our configuration:

• networks U, V, W with their subnets (10.1.1.0/24, 10.2.2.0/24 and 10.9.9.0/24) are to be created;
• a new LR should be created;
• networks V and W should be connected to the created LR;
• network W should be marked as external;
• next we create virtual machine interfaces (VMI) that will connect virtual machines to virtual networks: VM1 will be connected U network as 10.1.1.11 and VM2 will be connected with V network 10.1.1.21 respectively;
• finally, we create FIP 10.9.9.91 (Static Floating IP) from the FIP pool of W network and we associate this FIP with the virtual machine interface from network U (those one that has 10.1.1.11 IP address).

When we connect the newly created VMI's to their virtual machines (virtual machines are assumed to be running) we will see next configuration of virtual networks:

• bridge network V has an interface route to 10.1.1.11/32 and a vrf route to 10.9.9.0/24 subnet;
• bridge network W has an interface route to 10.9.9.91/32 and a vrf route to 10.1.1.0/24 subnet;
• the routing network of the LR (the one with the name starting with __contrail_lr) has interface routes to 10.1.1.11/32 and to 10.9.9.91/32.

VRF Inet tables for V, W and routing networks are presented on figures 1a, 1b and 1c respectively.

Now the VMI from W network (VRF instance) can talk to the VMI from V network (VRF instance) via the LR: when a packet from W VRF VMI is send to the prefix 10.2.2.0/24, this packet hops into the routing VRF instance (according to vrf nexthop of 10.2.2.0/24 route). There the packet finds the interface nexthop of 10.2.2.21/32 route and proceeds to the TAP of the destination guest OS.

Guest OS's should also be configured to know about foreign networks subnets:

• on VM with 10.1.1.11 (U network proxied by W) we need to run: sudo ip r a 10.2.2.0/24 via 10.1.1.1 dev eth1
• on VM with 10.2.2.21 (V network) we need to run sudo ip r a 10.9.9.0/24 via 10.2.2.1 dev eth1

We use network 10.9.9.0/24 prefix because we don't want to expose IP addresses from virtual network U to network V.

It is assumed also that all virtual port security groups are open in ingress and egress directions for all needed addresses, protocols and ports ranges.

Solution of inter-VRF forwarding problem with ACL

If we didn't use Floating IP (which is actually the NAT) to hide addresses of U network, this configuration would be enough to connect virtual machines from different networks. To overcome the problem of the conflict between TF VxLAN and NAT implementations (see intro for more details), we are going to employ ACL facility of Tungsten Fabric:

• using Config Editor (Configure --> Config Editor of GUI) we find access-control-lists section and create a new ACL;
• in the appearing dialogue window we select:
  • default-domain:admin:V network for Virtual Network drop down (we want to change behaviour for packets from V network);
  • we set some name for our new ACL (e.g., to10-9-9-91);
  • in Access Control List Entries section we create two Acl Rules (No. 0 is already present by default);
  • for the Acl Rule No. 0 we set:
    • Direction to <>;
    • Ethertype to IPv4;
    • Protocol to any;
    • Src Port --> End Port to 65535;
    • Src Port --> Start Port to -1;
    • Src Address --> Ip Prefix to 0.0.0.0;
    • Src Address --> Ip Prefix Len to 0;
    • Src Address --> Virtual Network to any;
    • Dst Port --> End Port to 65535;
    • Dst Port --> Start Port to -1;
    • Dst Address --> Ip Prefix to 10.9.9.91;
    • Dst Address --> Ip Prefix Len to 32;
    • Dst Address --> Virtual Network to any;
    • Action List --> Assign Routing Instance we set to the name of the routing VRF instances, e.g. default-domain:admin:__contrail_lr...;
  • for the Acl Rule No. 1 we set:
    • Direction to <>;
    • Ethertype to IPv4;
    • Protocol to any;
    • Src Port --> End Port to 65535;
    • Src Port --> Start Port to -1;
    • Src Address --> Ip Prefix to 0.0.0.0;
    • Src Address --> Ip Prefix Len to 0;
    • Src Address --> Virtual Network to any;
    • Dst Port --> End Port to 65535;
    • Dst Port --> Start Port to -1;
    • Dst Address --> Ip Prefix to 0.0.0.0;
    • Dst Address --> Ip Prefix Len to 0;
    • Dst Address --> Virtual Network to any;
    • Action List --> simple_action we set pass.

Rule No. 0 is needed to transfer packets from the VRF instance of network V into the routing VRF instance where the Floating IP (10.9.9.91) is present and where NAT will be performed.

Rule No. 1 is needed to unblock passage of all other packets coming from or to network V.

ACL configurations for rules No. 0 and No. 1 are presented in JSON format on figures 2a and 2b respectively.

We can now try to ping VM1 from VM2 using FIP address 10.9.9.91 (figure 3).

And a final question: is it possible to connect internal networks with an external one using this approach? Use, it is possible, but in this case ACL is not needed, because a packet from an external network hits directly the routing VRF instance where FIP routes reside.

Solution of inter-VRF forwarding problem by modification of vRouter Agent code

Setting up ACL rules for many VRF instances, several protocols (IPv4 / IPv6, ICMP/TCP/ UDP) might be very time-consuming and error prone process. In this case, it is better to change vRouter Agent code. Inter-VRF packets forwarding is available through setting up a flow and this job is performed in class PktFlowInfo.

We will add a new method NatVxlanVrfTranslate to class PktFlowInfo that performs all neccesary operations inter-VRF packets forwarding. The declaration of the class is contained in file pkt_flow_info.h, and hence we put our first modification (inside PktFlowInfo scope) there:

void NatVxlanVrfTranslate(const PktInfo *pkt, PktControlInfo *in,
    PktControlInfo *out);

Afterwards, we should put body of the function in file pkt_flow_info.cc. The function contains checks that test whether all necessary is present and is allocated. We also check that an output VRF instance has an interface for the given destination IP address and this interface corresponds to some Floating IP entity. Finally, parameters of the flow are updated via functions ChangeVrf and UpdateRoute.

void PktFlowInfo::NatVxlanVrfTranslate(const PktInfo *pkt, PktControlInfo *in,
    PktControlInfo *out) {
    if (out == NULL ||
        in == NULL ||
        out->rt_ == NULL ||
        in->vrf_ == NULL ||
        in->vrf_->routing_vrf()) {
        return;
    }

    const NextHop *nh = out->rt_->GetActiveNextHop();
    if (nh == NULL || nh->GetType() != NextHop::VRF) {
        return;
    }

    const VrfNH *vrf_nh = static_cast<const VrfNH *>(nh);
    const VrfEntry *vrf = vrf_nh->GetVrf();
    if (vrf == NULL || vrf->routing_vrf() == false) {
        return;
    }

    InetUnicastRouteEntry *inet_rt = vrf->GetUcRoute(pkt->ip_daddr);
    const NextHop *rt_nh = inet_rt ?
        inet_rt->GetActiveNextHop() : NULL;
    if (rt_nh == NULL || rt_nh->GetType() != NextHop::INTERFACE) {
        return;
    }

    const Interface *intf = static_cast<const InterfaceNH*>
        (rt_nh)->GetInterface();
    if (intf == NULL || intf->type() != Interface::VM_INTERFACE ||
        static_cast<const VmInterface*>(intf)->FloatingIpCount() == 0) {
        return;
    }

    ChangeVrf(pkt, out, vrf);
    UpdateRoute(&out->rt_, vrf, pkt->ip_daddr, pkt->dmac,
        flow_dest_plen_map);
    UpdateRoute(&in->rt_, vrf, pkt->ip_saddr, pkt->smac,
        flow_source_plen_map);
}

Finally, we have to put call to this methods in a proper place. The correct place is PktFlowInfo::IngressProcess method just before the first call to PktFlowInfo::VrfTranslate method:

NatVxlanVrfTranslate(pkt, in, out);

This example was tested for Tungsten Fabric R2011, but it should also work for later versions.

Finally, vRouter Agent should be recompiled and the new binary must be copied into the proper docker container. After restart of the container, ping and nc commands can be used to test inter-VRF forwarding of packages.

---

Abbreviations

A non-exhaustive but still relevant list of terms and abbreviations used in networking.

A

• AAA - Authentication, Authorization, Accounting
• AAP - Authoritative Access Point
• AFI - Address Family Identifier
• ARP - Adress Resolution Protocol
• AS – Autonomous System
• ASIC - Application Specific Integrated Circuit

B

• BDR - Backup Designated Router
• BGP - Border Gateway Protocol
• BGPaaS - BGP as a Service
• BMP - BGP Monitoring Protocol
• BMS - Bare Metal Server
• BPDU - Bridge Protocol Data Unit
• BSS - Business Support System
• BUM - Broadcast Unknown Unicast Multicast

C

• CAR - Committed Access Rate
• CDN - Content Distribution Network
• CIDR - Classless Inter-Domain Routing
• CLI - Command Line Interface
• CNI - Container Network Interface
• CoS - Class of Service
• CP - Control Plane
• CPE - Customer Premises Equipment
• CPLD - Complex Programmable Logic Device
• CPS - Control Plane Services

D

• DC - Data Center
• DCI - Data Center Interconnect
• DHCP - Dynamic Host Configuration Protocol
• DMZ - DeMilitarized Zone
• DOCSIS - Data Over Cable Service Specification
• DP - Data Plane
• DPI - Deep Packet Inspection
• DR - Designated Router
• DSCP - Differenciated Services Code Point
• DV - Distance Vector
• DWRED - Distributed Weighted Random Early Detection

E

• ECMP - Equal Costs Multiple Paths
• EGP - External Gateway Protocol
• EPON - Ethernet Passive Optical Networks

F --

• FD - Forwarding Device
• FDB - Forwarding DataBase
• FIB - Forwarding Information Base
• FPGA - Field Programmable Gate Array

G

• GPON - Gigabit Passive Optical Networks
• GRE - Generic Routing Encapsulation
• GRO - Generic Receive Offload
• GSO - Generic Segmentation Offload

H

• HBS - Host Base Services
• HCL - Hardware Compatibility List
• HTML - HyperText Markup Language
• HTTP - HyperText Transfer Protocol

I

• ICMP - Internet Control Message Protocol
• IDS - Intrusion Detection System
• IF-MAP - The InterFace to Metadata Access Point
• IGP - Internal Gateway Protocol
• IP - Internet Protocol
• IPS - Intrusion Prevention System
• IPAM - IP Adress Management
• IRB - Integrated Routing and Bridging
• IS-IS - Intermediate System to Intermediate System

J

• JSON - JavaScript Open Standard

K

• KVM - Kernel-based Virtual Machine

L

• LAN - Local Area Network
• LB - Load Balancer
• LDP - Label Distribution Protocol
• LIB - Label Information Base
• LIR - Local Internet Registry
• LPM - Longest Prefix Match
• LS - Link State

M

• MAC - Media Access Control
• MANO - MAnagement aNd Orchestration
• MIB - Management Information Base
• MP - Management Plane
• MPLS - MultiProtocol Label Switching

N

• NAT - Network Address Translation
• NIC - Network Interface Card / Controller
• NFV - Network Functions Virtualization
• NFVI - NFV Infrastructure
• NI - Northbound Interface
• NLRI - Network Layer Reachability Information
• NOS - Network Operating System
• NVE - Network Virtualization Edge

O

• OCI - Open Container Initiative
• ONL - Open Network Linux
• ONIE - Open Network Install Environment
• OSPF - Open Shortest Path First
• OSS - Operation Support System
• OVF - Open Virtualization Format

P

• P4 - Programming Protocol-independent Packet Processors
• PCB - Process Control Block
• PCI - Peripheral Component Interconnect
• PDU - Protocol Data Unit
• POF - Protocol Oblivous Forwarding
• PWE3 - Pseudo-Wire Emulation Edge to Edge

Q

• QoS - Quality of Service

R

• RAN - Radio Access Networks
• RARP - Reverse Adress Resolution Protocol
• REST - REpresentational State Transfer
• RFC - Request For Comments
• RIB - Routing Information Base
• RIP - Routing Information Protocol
• RIR - Regional Internet Registry

S

• SAFI - Subsequent Address Family Identifier
• SAI - Switch Abstraction Interface
• SDN - Software Defined Network
• SI - Southbound Interface
• SNMP - Simple Network Management Protocol
• SR-IOV - Single Root I/O Virtualization
• STP - Spanning Tree Protocol

T

• TAP - Terminal Access Point
• TBB - Intel Threading Building Blocks
• TCG - Trusted Computer Group
• TCP - Transmission Control Protocol
• TE - Traffic Engineering
• TNC - Trusted Network Connect architecture
• TOR - Top Of the Rack
• ToS - Type of Service

U

• UDP - User Datagram Protocol

V

• VIM - Virtualized Infrastructure Manager
• VLAN - Virtual LAN
• VNC - Virtual Network Controller
• VNF - Virtual Network Functions
• VM - Virtual Machine
• VMI - Virtual Machine Interface
• VOD - Video On Demand
• VPN - Virtual Private Network
• VPLS - Virtual Private LAN Service
• VRF - Virtual Routing and Forwarding, in Juniper it is called as Routing Instance
• VTEP - VXLAN Tunnel End Point
• VXLAN - Virtual eXtensible LAN

W

• WAN - Wireless Access Networks
• WFQ - weighted Fair Queueing

X

• XMPP - Extensible Messaging and the Presence Protocol

Y

• YANG - Yet Another Next Generation data modelling language

Z

---

Markdown Rendering Using HTML

Once I needed a translator from Markdown into HTML formats to display contents of .md files on web pages. I intended to call that translator from C++ code, therefore I put several constraints:

• it should have C/C++ API;
• it should be open source with corresponding license (GPL/LGPL/MIT or similar);
• it should be portable and small;
• it should be fast enough to process tens of files on the fly imperceptibly.

I tried several solutions from the web:

1. Pandoc with libpandoc. Pandoc is a very popular and mature library that supports tens input and output formats and allows translation between them. However, it is written in Haskell and requires additional efforts to call it's functions from C++. This problem was solved in libpandoc project. Second drawback of Pandoc is size of distribution - it's installation on fresh Ubuntu 22.04 requires approximately 154Mb and that requirement seems excessive.
2. Markdownpp is another interesting solution. It is written using relatively modern C++, well-documented, and it supports mathematical equations via integration with LaTeX, syntax highlighting and different Markdown themes. However, this library has external dependencies, such as boost and hoedown for Markdown parsing (which was forked from sundown ).
3. maddy is a small single-header Markdown-to-HTML library written in C++. It doesn't have external dependencies except STL and therefore it is highly portable and easy to call in any C++ program. I've used it a lot, however Markdown-to-HTML conversion time might be large especially if you have several files of moderate size.
4. md4c is a pure C library to parse Markdown files and it also allows translation of .md files into HTML format. The library is extremely fast, it doesn't utilize any external dependencies (excepct libc). Therefore, I've chosen this solution.

Finally, there are next libraries that were not considered by me (but they could have their advantages):

• peg-markdown;
• Discount;
• Cpp-Markdown;
• libsoldout;
• cmark;
• cxxmarkdown as a wrapper over Discount;
• sundown or it's successor hoedown.

---

Developer & Testing Forum June 2023

The upcoming Linux Foundation Networkin (LFN) Developer & Testing Forum (D&TF) will be held virtually from June 6th to June 8th, 2023.

This event provides an opportunity for various specialists involved in LFN software development (both testers and developers) to collaborate and innovate on open-source networking projects. Next projects are usually presented at the event:

• OpenDaylight;
• ONAP;
• Anuket;
• Tungsten Fabric;
• XGVela;
• ODIM;
• EMCO;
• L3AF;
• 5G Super Blueprint;
• FD.io;
• Nephio.

The deadline for topic submissions is May 19th, 2023. Topics should be submitted via this link: https://wiki.lfnetworking.org/x/xgjxB.

Participants registration is available at: https://cvent.me/NKWY5M.

Questions regarding topics submission or participation should be directed to events@lfnetworking.org.

---

A list of Tungsten Fabric ports

Tungsten Fabric uses a lot of servers to provide tools for information, analysis and configuration. Sometimes they are called as Northbound and Soundbound bridges. It is possible to find tools addresses at the Juniper's site and in the source code. And also it might be valuable to keep them in one place.

Let's envision that the TF control node is located at the IP 172.16.0.26.

TF analytics API documentation

http://172.16.0.26:8081/documentation/index.html

TF configuration API documentation

http://172.16.0.26:8082/documentation/index.html

OpeStack UI

http://172.16.0.26

TF UI

http://172.16.0.26:8143

---

How To temporary disable loading of vrouter kernel module at the boot stage

In some circumstances it is neccessary to disable the TF vrouter kernel module right at the boot stage. For example, when a version of vrouter module with serious bug was compiled and installed in the kernel. In this case, reboot will not help and system continues to crash right after the boot stage. Or, for example, it is neccessary to switch off flow through the data plane provided by the TF at the boot.

In the aforementioned and other cases the editor of the GNU GRUB loader allows to disable vrouter kernel module just for one start of an OS. See, for example here.

The procedure contains 6 steps.

1. When system shows it's BIOS screen it is neccesary to press and hold the shift key.
2. When system shows the GRUB menu it is necessary to select item "Advanced options ...", Fig. 1.
3. Demanded version of the kernel can be selected there using up/down arrows buttons.
4. When the needed kernel version is selected, press on button 'e' should open an editor for the kernel boot options, Fig. 2.
5. In the editor, find the line that starts with the word linux and browse to it's end. Put space and write comma-separated list of disable modules after the keyword module_blacklist, e.g.: module_blacklist=vrouter, Fig. 3
6. Press Ctrl+x or F10 to boot the system using the modified kernel boot options.

Fig. 1

Fig. 2

Fig. 3

---

A list of Tungsten Fabric resources

1. https://tungsten.io/ is a main site of the Tungsten Fabric project.
2. https://wiki.tungsten.io/ is an LFN wiki dedicated to the Tungsten Fabric project.
3. https://github.com/tungstenfabric/ is a GitHub project for the Tungsten Fabric.
4. https://github.com/tungstenfabric/docs is a RTD (Read The Docs) documentation for the Tungsten Fabric.
5. https://www.juniper.net/documentation/product/us/en/contrail-networking/ is a documentation for the OpenContrail (a predecessor of the Tungsten Fabric).

---

Compilation of Tungsten Fabric R2011 from sources on CentOS 8

The compilation process of Tungsten Fabric (or the TF) on CentOS 8 is slightly different from CentOS 7 and requires some modifications of tf-dev-env scripts.

Next steps are needed to compile it on the fresh CentOS 8.

1. Obtain the latest version of the TF development environment, e.g.: git clone https://github.com/tungstenfabric/tf-dev-env.git .
2. Change files tf-dev-env/run.sh, tf-dev-env/build.sh, tf-dev-env/package-tpp.sh, tf-dev-env/build-operator-containers.sh by inserting commit_opts+='' instead of commit_opts='--format docker' (docker option --format is not supported anymore).
3. Install python-minimal and pip2.
4. Reinstall docker using instructions from here: https://docs.docker.com/engine/install/centos/ .
5. Link /usr/bin/python2 to /usr/bin/python .
6. Select the TF version: export GERRIT_BRANCH=R2011 in the shell before running the compilation script.
7. Finally, compile the TF from sources using standard procedure:
   1. tf-dev-env/run.sh fetch;
   2. tf-dev-env/run.sh configure;
   3. tf-dev-env/run.sh compile.

Please, write me if I miss anything in these steps.

---

Compilation of Tungsten Fabric R2011 from sources on Ubuntu 20.04

The Tungsten Fabric (or the TF) might work with many Linux distributions, such as: Ubuntu, CentOS, and others. The process of the TF compilation from sources is distribution-agnostic and is covered in tf-dev-env repository. However, this process might slightly vary depending on a type of the selected Linux distribution. While deviations from the common instructions are usually small, they might confuse a new TF user.

Here I summarize nuances of the TF R2011 compilation on the fresh Ubuntu 20.04.

Next steps are needed to compile it on the fresh Ubuntu 20.04.

1. Obtain the latest version of the TF development environment, e.g.: git clone https://github.com/tungstenfabric/tf-dev-env.git .
2. Change the packet name in tf-dev-env/common/functions.sh from python-minimal to python2-minimal .
3. Install python2-minimal and pip2 (for example, using this instruction: https://linuxize.com/post/how-to-install-pip-on-ubuntu-20.04/ ) .
4. Reinstall docker using instructions from here: https://docs.docker.com/engine/install/ubuntu/ .
5. Link /usr/bin/python2 to /usr/bin/python .
6. Select the TF version: export GERRIT_BRANCH=R2011 in the shell before running the compilation script.
7. Finally, compile the TF from sources using standard procedure:
   1. tf-dev-env/run.sh fetch;
   2. tf-dev-env/run.sh configure;
   3. tf-dev-env/run.sh compile.

Please, write me if I miss anything in these steps.

---