Versions Compared

Old Version 16

changes.mady.by.user Abhishek Kumar

Saved on

compared with

New Version 17

changes.mady.by.user Abhishek Kumar

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

Table of Contents

Introduction

CloudStack has support for IPv6 in basic zone/networking and in shared networks. Following a discussion started by Alex on dev/users ML, after figuring out a baseline a high-level proposal and proof-of-concept details for IPv6 support in Isolated and VPC networks with static routing were discussed and phase1 goals were identified:

https://markmail.org/message/ppkfzpuagxc4i6zz

This design document tries to explain the scope of this feature in its first phase that is generally accepted by all contributors who've discussed this on the ML thread above. 

Some design documents have been written in the past for reference:

IPv6 support

IPv6 in VPC Router

IPv6 in Basic Networking

IPv6 Prefix Delegation in Basic Networking

IPv6 in VPC Router (original version)

Purpose

To add support for IPv6 for isolated network and VPC tiers with static routing.

Future Work

Once this feature reaches production users, based on the feedback of community and contributors future work may involve adding support for dynamic routing using FRR, OSFP/BGP/ExaBGP.

Document History

VersionAuthor/ReviewerDate
1.0Rohit Yadav

 

1.1Rohit Yadav

 

1.2Abhishek Kumar

 

Proof-Of-Concept and Dev-Test Setup

A proof-of-concept was explored which would make it possible for any developer to  setup a IPv6 dev-test env irrespective of whether they've datacenter access or if their home/office ISP supports IPv6 using IPv6-in-IPv4 service from https://tunnelbroker.net

  1. Sign up for tunnelbroker and create a tunnel, and request/acquire /48 ipv6 prefix.
  2. Create a custom OpenWrt router (VM or physical host or Iot/RaspberryPi4) for your home/office lab to serve as upstream router, or setup IPv6-in-IPv4 tunnel on your laptop/desktop (refer to tunnerbroker docs).
    For testing and example purposes, my 6to4 tunnel and prefix looked  like:
    /64 default tunnel - 2001:470:36:3e2::/64 (allocated by default)
    /48 prefix - 2001:470:ed36::/48 (allocated by tunnel broker)
  3. My LAN configuration looked like:
    LAN IPv6 address: 2001:470:ed36:0::1/64
    Address mode: SLAAC+stateless (no dhcpv6)
    Check if you're able to ping say google.com:

    $ ping6 -c 1 google.com
    PING google.com(del11s05-in-x0e.1e100.net (2404:6800:4002:810::200e)) 56 data bytes
    64 bytes from del11s05-in-x0e.1e100.net (2404:6800:4002:810::200e): icmp_seq=1 ttl=117 time=329 ms

    — google.com ping statistics —
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 329.213/329.213/329.213/0.000 ms

  4. Now deploy a dev-test CloudStack env, and deploy a guest VM on an isolated network and enable firewall/pf rules for say ports 22/ssh and 80/http.

  5. In the isolated network VR:
    1. Enable IPv6:

      net.ipv6.conf.all.disable_ipv6 = 0
      net.ipv6.conf.all.forwarding = 1
      net.ipv6.conf.all.accept_ra = 1
      net.ipv6.conf.all.accept_redirects = 1
      net.ipv6.conf.all.autoconf = 1

    2. Next, we should simulate how nics/networks are created and setup semi-manually:

      echo iface eth0 inet6 auto >> /etc/network/interfaces
      echo iface eth2 inet6 auto >> /etc/network/interfaces
      /etc/init.d/networking restart


      Next, restart ACS isolated network without cleanup so all the nics, nat, firewall/pf rules are reconfigured.
      If you do "ip addr" you should see public IPv6 address autoconfigured (SLAAC) on the public interface (eth2) of the VR.

    3. Next for the guest network, we need to simulate assignment of a /64 IPv6 for which RA is enabled, and radvd can be used:

      Wiki Markup
      # cat /etc/radvd.conf
interface eth0
{
    AdvSendAdvert on;
    MinRtrAdvInterval 5;
    MaxRtrAdvInterval 15;
    prefix 2001:470:ed36:1::/64
    {
        AdvOnLink on;
        AdvAutonomous on;
    };
};
systemctl restart radvd


    4. The VR's guest network nic (eth0) and guest VM's nic should now get IPv6 address (SLAAC) in this ...:1::/64 network. In the VR check if you're able to ping6 google.com (or any other server), if not you may need to check and fix the DNS nameservers in /etc/resolve.conf
    5. Finally add a static route to your OpenWRT router or your laptop/desktop nics so it knows how to route back traffic from public Internet (the ipv6-in-ipv4 tunnel) to the VR, where the target is the /64 range (from our example, 2001:470:ed36:1::/64) and gateway is the IPv6 address of the VR public (eth2) nic.

    6. To test reachability you can run a webserver (say nginx) on the guest VM, enable any firewall rules on the router (note: there won't be any firewall rules in the VR yet) and use any external service, computer/phone or proxy (http://www.ipv6proxy.net) to check accessibility of your guest VM's IPv6 address.

Feature Specifications

In this section, we define the scope and high-level spec for the feature which is as follows:

  • IPv6 addresses management:
    • Root admin (or operator) has access to a /64 or bigger IPv6 prefix/range such as /48, /56 etc. which can add to a zone where a single /64 is specified and reserved for use in public/pod traffic (systemvms, VR's nics), rest of the /64 or larger IPv6 address/range can be used for guest traffic (guest VMs nics).
    • Each isolated network or a VPC tier is assigned a /64 from available and configured IPv6 prefix/ranges. The subnet/prefix allocation per network is tracked. (Note: not larger block allocation to VPC by default, such as /60, /62...)
    • SLAAC is supported and used, there is no DHCPv6 support. For public nics of VRs, it is assumed the address allocation uses IPv4.
    • The IPv4 address ranges are added in the DB which we can't for all IPv6 and therefore similar to the mac address calculation logic CloudStack calculate/find IPv6 address for VM's nic (that will get SLAAC-based assignment) based on VM's nic's mac address.
    • All systemvms (ssvm, cpvm, VRs etc) with a public nic should get automatic/SLAAC IPv6 address if zone has ipv6 /64 prefix dedicated/allocated. As discussed during implementation, IPv6 support for system VMs with a public nic will be done in the next phase.
  • Network offering:
    • A new parameter called supported stacks is added to an offering such as: ipv4 only, ipv4+ipv6 (future phases can introduce a new ipv6-only)
    • On upgrade or new installation, all existing network offerings migrate to ipv4-only enum/stack option.
    • A new offering property/service is added: "IPv6 routing" which is "static" (future phases can introduce dynamic routing for IPv6).
    • A new offering property/service is added: "IPv6 firewall" with VR as an option.
  • Firewall:
    • The IPv6 feature will only support IPv6 address management, static routing and firewall .
    • An upstream router with firewall may exist whose configuration is outside the scope of this feature (similar to static routing)
    • If the network offering supports IPv6 firewall, the isolated or VPC network VR should support IPv6 firewall or ACLs using `ip6tables`.
    • For isolated networks, a new UI tab called IPv6 addresses in which the Firewall is accessed in UI by going to network → IPv6 address (by default IPv6 address of the VR is shown).
    • For VPCs, the ACL feature is extended to support rules for IPv6.
    • Refer RFC 9099 - Operational Security Considerations for IPv6 Networks https://www.rfc-editor.org/rfc/rfc9099.html
  • IPv6 routes management:
    • Root admin (or operator) has access and ability to configure IPv6 static routes to their upstream router, either manually or using some automation/scripting.
    • Routes are added with the /64 address assigned to an isolated network or VPC tier with the VR's public nic's IPv6 address as the IPv6 gateway.
    • A section in the details tab or a new tab itself should list what IPv6 static routes lists should exist on the upstream router - this can be a root-admin only feature in UI.
    • A new health check to check if IPv6 is working properly up to the VR - for example, pinging (ping6) google.com or admin-configurable IPv6 AAA domain or address (global setting or network setting).
  • Support external automation:
    • CloudStack should announce events on its message bus such as to add or remove static route and firewall rules for IPv6
    • Export information on static routes that CloudStack expects as well as firewall rules by list APIs
    • The list API and message-bus/event bus (rabbitmq/amqp, kafka etc) can be used by root admin to write automation scripts to add call an API/CLI to add static routes or firewall rules on their upstream router
  • Misc:
    • Explore other related features/integration such as static routes, vpc gateway etc can be explored/supported.
  • Supported: Static Routes for IPv6 (both isolated and VPC networks)
  • IPv6 address usage: do first-class usage records similar to IPs, but for /64 ipv4 prefer blocks per network (isolated or vpc tiers)
  • Add VR diagnostic feature to do: nslookup, ping6 and traceroute6
  • IPv6 address management: allow admin to expand/shrink the prefix (update existing prefix) if there isn't any loss of ipv6 in use

Out of Scope for this Phase

  • Allocate prefix to an individual VM in shared network
  • VPN and Load-Balancing, port-forwarding
  • IPv6-only network/offering

Design & Implementation

API and Schema Changes

This section describes on a high-level the API and schema changes needed for root admins and users:

  • Address management:
    • Root admin-only: For public traffic/range, expand createVlanIpRange and related schema which supports adding ip6 range, or introduce a new API `addIPv6Range` which accepts IPv6 in /prefix format (allow accepting /64 or larger to validate input).
    • For isolated/VPC guest networks, expand the existing APIs and user_ipv6_address table for guest VM nics. The list VM response should include IPv6 address in the nics (similarly listNics).
  • Network offering:
    • Implement the spec, expand existing list APIs for network offering and VPC offering and related schema.
  • Routes and firewall:
    • For firewall rules (for ipv6 address in case of isolated networks or ACL rules) in case of such rules being added, removed or updated - expand/refactor existing APIs. A new addresstype (ipv4 or ipv6) may be introduced for firewall rules to differentiate whether such rules are configured via iptables or ip6tables. Alternatively, this can be determined in the UI or backend based on the rule (whether rule is ipv4/ipv6 address or CIDR).

Service Layer and VR Changes

For an IPv6 enabled network (isolated or VPC tier) in either single stack (ipv4 only) or dual-stack (ipv4+v6) configuration based on the network offering, CloudStack should check and allocate a /64 IPv6 prefix for the network by calculating and finding the next available block which is free (not in used) and with VM and network orchestration services/managers pass suitable configuration to the isolated or VPC VR. The firewall is implemented using ip6tables in the VR using a new chain/zone, if enabled by the network offering. The IPv6 address for VR's nic and guest VM's nics are calculated based on their mac address, also known as EUI-64 address (see the RFC or online guide such as https://kwallaceccie.mykajabi.com/blog/how-to-calculate-an-eui-64-address). VR should configure radvd (or frr in future?) to support the assigned prefix for the guest network or VPC tier and enable certain IPv6/icmpv6 packets (https://datatracker.ietf.org/doc/html/rfc4890).

New default VPC and isolated network offerings with IPv6 enabled (dual-stack)?

We need to explore if CloudStack should alter existing default network offerings, or add new ones? Once the network is started or restarted or implemented (when the first VMs is deployed), when the VPC/isolated VR is up and configured, the static routes and any firewall rules are announced on CloudStack's message bus, as well as when firewall rules or ACLs are created/updated/removed.

UI Changes

  • Address management:
    • For root admin enable a form to add IPv6 prefix such as /64 to /48 as well as dedicate the autoconfig/default IPv6 lan network (usually a /64) for use in public traffic for systemvms/VR. This may be an explicit option or some way to dedicate the range to systemvms.
    • All VPC tiers get automatic /64 assignments, no input from user required
    • Nics - in the list VM and nics response should export the nic's EUI-64 address, display that in infocard and other places
  • Network offering: (both network offering and VPC offering)
    • Based on API changes, ability for root admins to specify options: stack (single stack ipv4, dual stack ipv4+ipv6), IPv6 routing (static) and IPv6 Firewall (this is optional; maybe we can simply reuse the available firewall service with VR/VPC-VR as option?)
  • Firewall:
    • Isolated networks: Egress can be expanded to select/toggle between ipv4/ipv6 option to add rules; Introduce a new tab called IPv6 addresses in networks where firwalls are added via → ipv6 address → firewall tab (similar to current UI)
    • VPC: expand the existing ACLs feature where the ipv6 option may be specified to explicitly tell whether the rule is ipv4 or ipv6.
  • Routes:
    • Root admin only: Add a tab or add a detail in the Details tab for a network or VPC to list what routes should be added for the VPC or isolated network in the upstream router.

Marvin Tests

For this feature, the following test cases or QA considerations must be made:

  • Use of ULA (similar to IPv4 private network/range) for smoketests? Or, some static configuration in test_data.py
  • VM, VR and nic lifecycle tests around IPv6, as well as offering support
  • CRUD tests as user or root admin for all features listed above in API section