Kube-OVN

Enterprise

With contributions from the community, the Kube-OVN team is excited to announce the release of Kube-OVN 1.8.

Within this release cycle, we focus on the underlay network enhancement, datapath latency optimization ,and further improvement on VPC functions. With 10+ function improvements and 50+ security and stability improvements, the overall stability of Kube-OVN has been testified. Thanks for the support and contribution from the coummunity!

Underlay Network Enhancement

In Kube-OVN 1.8, the underlay network implementation has been refactored to achieve more flexible network setups. A new CRD ProviderNetwork is provided to map the underlay network into container network and provide options for complex real networks requirements, including:

Host with a single network interface to achieve both manage network and underlay container network

Host with multiple network interfaces which mapping into different VLANs

Hosts in a cluster with different network interface names

Some VLANs only exist for part of hosts in the cluster

Multicast in the network

Overlay and underlay networks co-exist

For more information, please read Vlan Support

Datapath Latency Optimization

Performance has long been a challenage topic. In v1.8 we dive into the OVN flow tables and kernel internals to profile the datapath and find the bottlenecks. With the help of profile results, some aggressive optimization methods have been found. With these methods, 45% latency can be reduced under the 1byte packet latency benchmark. These methods include:

OVN logical flow re-organized to avoid some CPU consuming operation

FastPath kernel module to reduce netfilter cost

Replace veth device with ovs internal port to reduce IRQ

Re-compile ovs kernel module with ISA related optimization

For more information, please read Performance Tunning

The 1-byte packet benchmark result for origin and optimized version:



The 1-byte packet benchmark comparison for Kube-OVN overlay/underlay mode, Calico IPIP and noEncap mode:



VPC Enhancement

As more and more users are trying to manage data center infrastructures with Kubernetes, VPC network functions in Kube-OVN are under rapid iteration. In this release, a new CRD SecurityGroup is provided to support VPC security management in VPC view.



Service support in custom VPC is also supported in this release, more L4 and L7 load balncer function is in the way.

For more information, please read VPC Usage

What's more

Kubernetes and Openstack share the same OVN deployment

Pod level mirror control

Manual route for multiple network interface

Dynamically modify tunnel IP

New OVN 21.03

Getting Started

New to Kube-OVN? Follow the Installation Guide.

Release

Release Note

Kube-OVN 1.8

In the joint efforts of the community members, Kube-OVN 1.11 has been officially released.



In this version, we have further enhanced the functionality, performance, stability, and usability of Kube-OVN. These enhancements include the intercommunication between Overlay and Underlay, Underlay nesting in k8s on k8s environments, support for LoadBalancer type Services, SR-IOV Network Operator support, internal load balancing in custom VPCs, and DNS in custom VPCs. Additionally, we have restructured the testing system, added extensive testing, and addressed stability and performance issues. A total of 32 community members have contributed to this release, and we would like to express our gratitude for their contributions and support.



Let's take a closer look at the new capabilities of Kube-OVN v1.11!



Overlay and Underlay Subnet Intercommunication



When both Overlay and Underlay subnets coexist in a cluster, by default, Pods in the Overlay subnet can access Pods in the Underlay subnet via NAT through the gateway. From the perspective of Pods in the Underlay subnet, the addresses in the Overlay subnet appear as external addresses that need to be forwarded through the underlying physical devices. However, the underlying physical devices are unaware of the addresses in the Overlay subnet and cannot perform the forwarding. As a result, Pods in the Underlay subnet cannot directly access Pods in the Overlay subnet using Pod IPs.



To enable intercommunication between Underlay and Overlay, the u2oInterconnection setting of the subnet needs to be set to true. In this case, Kube-OVN will use an additional Underlay IP to connect the Underlay subnet with the ovn-cluster logical router and set corresponding routing rules to achieve intercommunication. Unlike the logical gateway, this solution only connects the Underlay and Overlay subnets within Kube-OVN, and other traffic destined for the external network will still be forwarded through the physical gateway.



SR-IOV Network Operator



Previously, configuring Mellanox offload required manual operations. With the SR-IOV Network Operator, the process of configuring network cards for Mellanox can be automated, greatly simplifying the complexity of using Mellanox for Kube-OVN acceleration.



Internal Load Balancing in Custom VPCs

While Kubernetes itself provides the ability for internal load balancing through Services, it is limited by the Kubernetes implementation. The IP addresses of Services are globally allocated and cannot be duplicated. For users in VPC scenarios, they may need to define custom address ranges for internal load balancing, and the load balancing addresses in different VPCs may overlap. The built-in Service feature of Kubernetes cannot fully meet these requirements.



To address this scenario, Kube-OVN provides the SwitchLBRule resource, allowing users to define custom address ranges for internal load balancing.



Here is an example of a SwitchLBRule:





Custom VPC DNS



Due to the isolation between custom VPC and default VPC networks, it is not possible for custom VPCs to access coredns deployed in the default VPC. If users want to use the cluster-internal domain name resolution capability provided by Kubernetes within their custom VPC, they can utilize the newly introduced vpc-dns Custom Resource Definition (CRD) to achieve this.



The CRD ultimately deploys a coredns Pod with two network interfaces: one in the user's custom VPC and the other in the default VPC, enabling network communication. Additionally, the custom VPC internal load balancing is provided through the custom VPC.



Load Balancer Type Service



In previous versions, Kube-OVN supported Load Balancers in custom VPCs through vpc-nat-gateway. In this version, we have extended the functionality of vpc-nat-gateway to the default VPC and combined it with Load Balancer type Services. Users can now use similar mechanisms as in custom VPCs to implement Load Balancer type Services in the default VPC.



Other Features

● Controller startup time optimization

● Network interruption time optimization during upgrade

● Memory usage optimization for OVS and controller

● Controller election mechanism optimization

● ARP conflict detection in Underlay subnets

● Multicast optimization in Underlay subnets



Kube-OVN 1.11

As the creators of Kube-OVN, Alauda has the experience and expertise to help make your journey successful with Kube-OVN in production.

 Features

The most advanced Kubernetes feature set

Kube-OVN integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric that is feature rich and enterprise friendly.

 Looking to contribute?

Here is a quick development guide to help you get started with Kube-OVN.

 Meet us at [KubeCon + CloudNativeCon Europe 2021]

The CNCF’s flagship conference gathers adopters and technologists from leading open source and cloud native communities virtually from May 4 – 7. Join us at the Kube-OVN Project Office Hours 

Thu, May 6, 9:00 AM (CEST) / Fri, May 7, 9:00 AM (CEST)

 

The Most Advanced Kubernetes Network Fabric for Enterprises

Thank you to all the community partners who have accompanied Kube-OVN through another year of rapid development. With the widespread adoption of Kubernetes technology, the usage of CNI network plugins has been gradually increasing, and the Kube-OVN community has been growing continuously. Let's follow this article to explore Kube-OVN in 2022.



Continuous Enhancement of Product Features



In 2022, Kube-OVN focused on solidifying its foundation and consolidating its advantages, completing the upgrade from version 1.10 to 1.11.



In the two major versions released this year, Kube-OVN has significantly strengthened its features and performance. This includes the intercommunication between Overlay and Underlay, Underlay nesting in k8s on k8s environments, support for LoadBalancer type Services, SR-IOV Network Operator support, internal load balancing in custom VPCs, DNS in custom VPCs, and more.



Furthermore, Kube-OVN has continuously improved stability and usability. The testing system has been restructured, extensive testing has been added, and existing issues have been continuously addressed. The goal is to become the highest-performing and most reliable open-source container network plugin.



"If you miss the rich networking capabilities in the SDN field but have been struggling to find them in the cloud-native domain, then Kube-OVN is your best choice." In addition to continuously improving product features, the Kube-OVN community has introduced a special topic on advanced features to help everyone understand the working principles and implementation paths of advanced Kube-OVN features, enabling you to quickly advance in cloud-native networking.



Real-world Case Studies



With the further popularization of CNI network plugins, the high-performance CNI network plugin Kube-OVN has been widely adopted in various industries and scenarios.



As the first CNCF sandbox container network project, with the joint efforts of community partners from companies such as China Telecom, China Unicom, Intel, NVIDIA, F5, ByteDance, China Mobile, Inspur, and Yealink, Kube-OVN has continuously improved its features, performance, stability, and usability. It has become a recommended cloud-native networking solution by many large companies' technical teams, helping users solve challenges in cloud-native networking.



Diverse Application Scenarios



With the collective efforts of community members, the application scenarios for Kube-OVN have been continuously enriched and improved over the past year. As cloud-native technologies have extended from applications to data centers and infrastructure, more and more enterprises have started using Kubernetes and KubeVirt to run virtualized workloads, allowing unified control of virtual machines and containers. Kube-OVN, utilizing OVN/OVS widely used in traditional virtualization networks, has gained attention from many KubeVirt users since its open-source release. Some cutting-edge KubeVirt users have further enhanced Kube-OVN's networking capabilities based on their specific use cases, including fixed addresses, multi-tenant networks, SR-IOV, and OVS-DPDK, greatly enhancing the cloud-native virtualization experience of KubeVirt.



At the same time, the F5 CES solution has integrated Kube-OVN, solving the challenge of outbound traffic policy control in high-dynamic IP scenarios in a Kubernetes-native way. Through a hierarchical design, it addresses collaboration issues among enterprise security, network, platform, and application operation and maintenance departments, helping users better manage outbound policies for Kubernetes containers.



Continuous Growth of the Community



In 2022, the open-source community operation of Kube-OVN continued to improve and attracted more attention from IT professionals. The user base expanded beyond network engineers to include architects, developers, operations engineers, security engineers, and more technical experts.



According to incomplete statistics, the number of stars for Kube-OVN on GitHub has increased from 1.1k to 1.6k+, forks have risen from 300 to nearly 400, commits have exceeded 3000+ from 2000, and Docker image downloads have surpassed 2 million, with an increase of approximately 400k compared to last year. There has been a 25% increase in community contributors, a doubling of the number of PR contributions, a 92% increase in community users, and the community membership has exceeded 1.5k, representing a 50% growth...



Finally, we hope that more people interested in container networking can join the Kube-OVN community, strengthen the open-source power of K8s, and collaborate to create a more user-friendly CNI plugin, leading the development of container networking.



About Kube-OVN



Kube-OVN is an enterprise-grade cloud-native Kubernetes container network orchestration system independently developed by Alauda By bringing the mature networking capabilities from the OpenStack domain to Kubernetes, it greatly enhances the security, operability, manageability, and performance of Kubernetes container networking, bringing unique value to the deployment of the Kubernetes ecosystem.



In early 2021, Kube-OVN became the first open-source CNI network project hosted by the CNCF globally. It marked the first successful contribution of an independently designed and developed project by a Chinese container company to the CNCF Foundation. Moreover, it is one of the star projects under the Mulan community.

2022 Kube-OVN Open Source Community Annual Report

News

Recently, the Cloud Native Computing Foundation (CNCF) announced that Kube-OVN, an open-source container network project of Alauda, has officially entered the CNCF Sandbox Hosting. This is the first global open source CNI network project incorporated by CNCF, and it is also the first time that China Container Company successfully donated an independently designed and developed projects to the CNCF.



Kube-OVN is an enterprise-class cloud native Kubernetes container network orchestration system independently developed by Alauda Company. It is based on SDN network virtualization technology and provides flexible container networks for enterprises. As the first accepted open source CNI network project by CNCF Sandbox, Kube-OVN has greatly enhanced the security, operability, management and performance of Kubernetes container network by translating mature network functions in the OpenStack field to Kubernetes, integrating various characteristics such as security enhancement, intelligent operation and maintenance, hardware acceleration, etc. It brings unique value to the success of Kubernetes ecology. Kube-OVN joins CNCF sandbox will promote the development and maturity of Kubernetes operating environment solutions.

Since Kube-OVN opens source, it has harvested 700 + star on Github. Twenty-four contributors from Alauda, China Telecom, Intel, Huawei and many other companies participating in the code writing. At the same time, it has accumulated 50 + enterprise users, including many Giant companies such as Unicom, Telecom, Huawei, China Electronics, ByteDance and Rakuten (Japan). In less than two years, we achieve thousands of cluster-level large-scale enterprise projects, and overseas projects landing, as well as commercialization of the initial attempt.

Next, Kube-OVN will strenghthen community operations and launch a series of community activities. We hope to join hands with more developers and users; then combined innovative technologies with application requirements, and use codes to resolve users’actual demands so that we are able to make Kubernetes network system available and efficient, and truly release the value of open source communities.

Kube-OVN Website: http://www.Kube-ovn.io
GitHub: https://github.com/alauda/kube-ovn

Kube-OVN now joins CNCF as a Sandbox project

Kube-OVN 1.7: Hybrid overlay and underlay network, multiple ovn native network, VPC enhancement, active-active subnet gateway and more



With contributions from China Telecom, Inspur, Intel, and ByteDance The Kube-OVN team is excited to announce the release of Kube-OVN 1.7.

Within this release cycle, lots of requirements about complex network infrastructure orchestration in the real ente

rprise scene overwhelming us. We have built these abilities to resolve these complexities into Kube-OVN 1.7 including:

Hybrid overlay and underlay network

Pod with multiple ovn native network interface

VPC gateway with FloatingIP and NAT abilities

Active-Active subnet gateway

Hybrid overlay and underlay network

Previously, Overlay or Underlay is a cluster-level installation option. All subnets in a cluster must be the same type, overlay, or underlay. However, in enterprises more and more users need different types of networks for different use cases in one Kubernetes cluster. For example, run web applications in overlay network to achieve high flexibility, at the same time run middleware like db in underlay network to achieve high performance and connectivity with the external network.

In Kube-OVN 1.7, the installation network type only takes effect for the default subnet users can select network type dynamically at the subnet level. We also provide a new CRD ProviderNetwork to map the underlay network into container network and provide options for complex real network setups. e.g different nodes may belong to different physic networks, the network interface names are different across the cluster, certain underlay network only exists in a group of nodes, and so on.

For more information, please read Vlan Support

Pod with multiple ovn native network interface

With the increasing number of users who run VM in Kubernetes, the need for multiple network interface VM also increased. In the previous version, Kube-OVN only provides OVN type network to the primary interface, attachment networks have to be provisioned by other CNIs. From this version on, users can use Multus CRDs and Kube-OVN annotations to define a Pod with multiple network interfaces all provisioned by Kube-OVN. This also gives VMs on Kuberentes to run on multiple virtual networks all provided by OVN.

For more information, please read Multi network interace

VPC gateway with FloatingIP and NAT abilities

The VPC in 1.6 only provides the independent address space for multi-tenants. However, Pod in VPC can not connect to external networks and can not be visited outside the VPC. In 1.7 release, we provide a new CRD VpcNatGateway to provide the ability to connect VPC and external networks.

In the VpcNatGateway users can define the eips allocated to the VPC and select how to use these eips for the VPC. It provides abilities like:

SNAT: map a group of pods egress traffic to a selected eip

DNAT: map an eip IP:Porrt to a Pod IP:Port

FloatingIP: map a pod ingress and egress ip to a selected eip

For more information, please read VPC Usage

Active-Active subnet gateway

Kube-OVN subnet support to egress mode distributed and centralized. For centralized mode, users can select a group of nodes to act as the egress gateway. In the previous version, these node work in active-backup mode, only one node is in active status that processes the egress traffic. This node will be the bottleneck when dealing with a large volume of egress traffic. In 1.7 release, all nodes can work together at the same time, egress traffic will be distributed to all selected nodes and it will automatically check failed node and rebalance the traffic.

12

Getting Started

New to Kube-OVN? Follow the Installation Guide.

Release

Release Note

Kube-OVN 1.7





As cloud native technologies converge to data centers and the infrastructure, more and more enterprises are using Kubernetes and KubeVirt to run virtualized workloads and manage both virtual machines and containers from a unified control plane. However, the usage scenarios of VMs are significantly different from those of containers. At the same time, the emerging container networking solutions are not designed for virtualization scenarios - the feature set and the performance characteristics are quite different from those of the traditional data center networks. Networking has become the bottleneck of adopting cloud native virtualization.

Kube-OVN, as its name implies, is an integration of the OVN/OVS based networking with Kubernetes. Since OVN and OVS have been widely used in traditional virtualization scenarios, Kube-OVN has attracted a lot of attention from the KubeVirt community since it was open sourced. Some early adopters of KubeVirt have further improved the networking capabilities of Kube-OVN for their use cases. This resulted in a large number of new and optimized Kube-OVN features specifically targeted towards solving some of the networking challenges faced by KubeVirt.



VM Static IP Addresses

Container networks usually do not require that IP addresses be fixed. In VM usage scenarios, however, the IP addresses are often used as the primary access method. Users typically require that the VM addresses remain stable as the VM is restarted, upgraded, or migrated. Virtualization administrators also have a need to control address assignments and want the ability to specify VM addresses.

After several iterations, Kube-OVN now supports:

* IP address specification at VM creation, with addresses remaining stable throughout the lifecycle of the VM.

* Random IP address assignment at VM creation, with addresses remaining stable throughout the lifecycle of the VM.

* NICs remaining fixed during VM live migration.



Specifying static IP addresses at VM creation

To assign a specific IP address to a VM at its creation time, simply add an annotation ovn.kubernetes.io/ip_addressto the template of KubeVirt’s VirtualMachine resource:

apiVersion: kubevirt.io/v1alp

For VMs created this way, Kube-OVN will automatically assign the specified IP address according to the annotation field, and this address will remain stable for the remainder of the VM’s lifecycle.

https://github.com/kubeovn/kube-ovn/blob/master/docs/static-ip.md



Assigning random static IP addresses at VM creation

In some cases, the administrator does not want to manually assign an IP address each time a VM is created, but still want the address to remain fixed for the VM throughtout its lifecycle.

The ability to associate the assigned addresses with VirtualMachine instances has been introduced in Kube-OVN v1.8.3 and v1.9.1. The lifecycle of the assigned IP addresses in Kube-OVN will be aligned with that of the VirtualMachine instance, enabling a one-to-one mapping. Administrators do not need to manually assign VM addresses in order to keep them stable.

To enable this feature, simply add the following parameter to the kube-ovn-controller.

--keep-vm-ip=true

https://github.com/kubeovn/kube-ovn/issues/1297https://github.com/kubeovn/kube-ovn/pull/1307



Stable IP addresses across VM live migration

KubeVirt's live migration faces a lot of networking challenges, especially in the fixed IP scenario, which makes it more complicated to avoid IP conflicts. KubeVirt uses the default network for state synchronization during live migration. By using the multus-cni to separate the service network from the migration network, we can manage IP address migration at the functional level.

The default network is only used for live migration and requires multus-cni to attach Kube-OVN as an attachment NIC to the VM.

Add the corresponding annotation to the VM to enable fixed IP and live migration support:

<attach>.<ns>.ovn.kubernetes.io/allow_live_migration: 'true’

KubeVirt's DHCP will set the wrong default route. You need to add an annotation:

<attach >.<ns>.ovn.kubernetes.io/default_route: 'true'

to select the correct default routing NIC.







With the above configuration, Kube-OVN will take over the transfer process of the network during the live migration and choose the right point in time for the network switchover.

https://github.com/kubeovn/kube-ovn/pull/1001



Multi-tenant Networks

In traditional virtualization usage scenarios, multi-tenancy with VPCs is generally considered a best practice in many enterprise IT environments. However, the lack of built-in support for tenant level isolation in Kubernetes has led to shortcomings in the ability to support multi-tenancy in container networks and KubeVirt.

Kube-OVN enables multi-tenant networking in Kubernetes by introducing a new set of networking CRDs: VPC, Subnet, NAT-Gateway. The virtualization control plane in KubeVirt can manage VPCs and Subnets so that different VMs reside in different tenant networks. This enables complete multi-tenancy capabilities in virtualization scenarios.

In addition, Kube-OVN provides control knobs around LB/EIP/NAT/Route Table, etc. This enables administrators to manage a cloud native virtualization network as if it were a traditional data center network.





kube-ovn/vpc.md at master · kubeovn/kube-ovn



SR-IOV and OVS-DPDK support

KubeVirt's default Pod mode network suffers from long network data paths and poor performance due to the need to adapt to CNI specifications. On the other hand, SR-IOV mode suffers from inflexible features and complex configurations.

Kube-OVN achieves high performance by directing SR-IOV devices to KubeVirt VMs through the OVS offload capability of the smart NIC. Kube-OVN also retains the address allocation and OVN logical flow table capabilities, enabling complete SDN networking and striking a good balance between performance and functionality.



Although upstream KubeVirt has no support for OVS-DPDK type networks, many users in the Kube-OVN community have independently developed KubeVirt support for OVS-DPDK . Therefore, even a normal NIC can enhance the network throughput within a VM when accelerated by OVS-DPDK.



kube-ovn/hw-offload-mellanox.md at master · kubeovn/kube-ovn

kube-ovn/hw-offload-corigine.md at master · kubeovn/kube-ovn

kube-ovn/dpdk-hybrid.md at master · kubeovn/kube-ovn



To sum up

In this blog post, we described several interesting use cases and special considerations when designing the network fabric for cloud native virtualization. The KubeVirt community has contributed to a number of virtualization related features in Kube-OVN, including VM static IP addresses, multi-tenant networks and SR-IOV and OVS-DPDK. We see Kube-OVN as a great companion for KubeVirt, and when used together, they really light up the cloud native virtualization experiences. We hope more users from the KubeVirt community will try out these features and give us feedback.



Enterprise

Contact Us

Send us a message

Enterprise

Contact us

Contact Us

Send us a message