Detailed Semantics

A ‘Calico’ network is a Neutron network (either provider or tenant) whose
connectivity is implemented, on every compute host with instances attached to
that network, by the calico plugin or ML2 mechanism driver. There can be
just one Calico network, or any number of them. This page describes the
connectivity that Calico provides between instances attached to the same
network, and between instances attached to different Calico networks, and
between instances and the Internet; and explains how and why this connectivity
is in some details different from traditional Neutron API semantics.

Connectivity between instances on the same network

Calico provides IP connectivity, but not layer 2 (L2) adjacency, between
instances attached to the same Calico network. This means that:

An instance can ping the IP of another instance, and make other IP-based
connections to other instances. (Unless restricted by security group
configuration.)

If an instances probe the IP path to another instance, it will find that
there are intermediate IP hops in the path; or in other words, that the
instances are not directly connected.

Applications or protocols that actually require L2 adjacency - such as
routing protocols like OSPF - will not run successfully on instances on a
Calico network. But the vast majority of applications that are IP-based will
be just fine.

Traditionally, a Neutron network has always provided L2 adjacency between its
instances, so this is the first way that Calico differs from traditional
Neutron semantics. Up to and including the Mitaka release, L2 adjacency was an
assumed property of a Neutron network; so deployments using Calico simply had
to understand that Calico networks were different in this detail.

As of the Newton release, Calico’s IP-only connectivity is expressible in the
Neutron API, as a Network whose l2_adjacency property is False.
However work is still needed to make Calico networks report l2_adjacency
False, so at the moment - unfortunately - it still has to be understood that
Calico networks do not provide L2 adjacency, even though they report
l2_adjacency True when queried on the API.

Note: Calico’s connectivity design, based on IP routing, allows unicast IP
and anycast IP. Anycast IP also requires support for allowed-address-pairs,
or some other way of assigning the same IP address to more than one instance;
work for allowed-address-pairs support is in progress at
https://review.openstack.org/#/c/344008/. Multicast IP support is on our
roadmap but not yet implemented. Broadcast IP is not possible because it
depends on L2 adjacency.

Connectivity between different Calico networks

Calico provides exactly the same connectivity between instances on different
Calico networks, as it does between instances on the same Calico network.

It is important to note that this is equally true for ‘provider’ and ‘tenant’
Calico networks (i.e. for networks that are provisioned by the cloud operator,
or by a particular tenant or project), and for connectivity between any mix of
those networks. There is no way, with Calico, to get a tenant network that is
isolated by default at the connectivity level, per standard Neutron API
semantics for a tenant network that is not connected to a router, even if the
Calico tenant network is not connected to a Neutron router, or if there are
no Neutron routers in the deployment at all.

Calico works this way because it targets use cases where instances are attached
either to provider networks directly, or (in Neutron data model terms) to
tenant networks that are attached through a router to a provider network.
One reason for the latter case is to use floating IPs with Calico, because in
current Neutron the target of a floating IP has to be an instance attached to a
tenant network. For more on this, see Floating
IPs.

Flat IP addressing

An implication of that connectivity between networks is that Calico assumes
that the IP addresses it handles are all in a single, flat address space. For
example, if one network has a subnet with CIDR 10.65.0.0/24, and another
network has a subnet with CIDR 172.18.0.0/16, an instance with IP 10.65.0.2 can
directly address an instance on the other network with IP 172.18.3.23, and the
IP packet will travel all the way between them with source IP 10.65.0.2 and
destination IP 172.18.3.23. There is no NAT anywhere along this datapath.

Evaluation against Neutron semantics

Calico targets use cases that correspond to two Neutron data model patterns.

Firstly, where instances are attached directly to provider networks:

Secondly, where instances are attached to an externally-connected tenant
network:

In the general case those patterns may be combined - so in general there may be
any number of Calico provider networks, and any number of Calico tenant
networks, so long as each of the tenant networks is connected through some
router to a provider network. The purpose of using tenant networks - instead
of always using provider networks - is only so as to enable floating IPs. The
purpose of provisioning multiple networks of either kind - instead of just
one - is typically to allow the user to control what kind of fixed IP an
instance gets.

However many Calico networks there are, all their IP addresses (in associated
Neutron subnet or subnet pool objects) must be defined or understood as
belonging to a single, flat address space.

Finally it must be understood that there is no L2 adjacency between any
instances, even those that are attached to the same network. In a future
OpenStack release, we hope to make this explicit, by arranging for Calico
networks to report l2_adjacency False.

Subject to those restrictions and understandings, we believe that
networking-calico fully implements Neutron semantics, i.e. that it provides the
connectivity that an operator would expect for a given sequence of Neutron API
setup calls.