vSphere with Tanzu - Zero Trust with NSX-T DFW

To enable zero trust for the Supervisor Cluster and the guest cluster, you need to first define a default deny-all rule in the NSX-T distributed firewall. You can then allow the required ports as per https://ports.esp.vmware.com/home/vSphere-7 (Filter with the keyword “Tanzu “)

The problem?

vSphere with Tanzu expects to have a default allow-all rule. Specifically for egress (Source is Master VM Subnet and Destination is the whole cluster CIDR block) After enabling zero trust, the default deny-all rule blocks both ingress and egress traffic.

The NCP (NSX Contianer Plugin. More information here) auto creates policy rules in NSX-T DFW. The default baseline policy rules that are created are based on the assumption that the default rule is allow-all for both ingress and egress traffic.

Up until vSphere 7.0.2 and NSX-T 3.1.2, the default policy rules created by NCP are:

Allow intra-namespace, masterVM, LB sourceIP to namespace ingress traffic
Allow any to Master VM ingress traffic
Deny all other ingress traffic

Example default rules from my lab:

Workaround

Patch the default policy rules to allow egress traffic. The scope will be similar to the deny-all rule that exists already. This will allow vSphere with Tanzu to work in a Zero trust environment.

The default policy rules can not be edited from the NSX-T manager as they are created by the NCP principal identity. We will have to use a REST API call using Postman or some other client.

For more information on the PATCH REST API:

Help->API Documentation->All Methods->Search for “Patch a rule” : Policy > Security > East West Security > Distributed Firewall > Rules > Patch a rule

Example REST API Call to patch and add the allow-all-egress rule:
PATCH https://nsx-t.gs.labs/policy/api/v1/infra/domains/domain-c8:dad7e875-3357-449b-809a-bf1783e3430d/security-policies/ds_domain-c8:dad7e875-3357-449b-809a-bf1783e3430d/rules/allow-all-egress
{
  "display_name": "allow-all-egress",
  "id": "allow-all-egress",
  "sequence_number": "98",
  "source_groups": [
      "ANY"
  ],
  "logged": false,
  "destination_groups": [
      "ANY"
  ],
  "scope": [
      "/infra/domains/domain-c8:dad7e875-3357-449b-809a-bf1783e3430d/groups/dg_domain-c8:dad7e875-3357-449b-809a-bf1783e3430d"
  ],
  "action": "ALLOW",
  "services": [
      "ANY"
  ],
  "direction": "OUT"
}

Screenshots from Postman:

The required ports from the official documentation are:

Port	Protocol	Source	                                            Destination
22      TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
53      TCP	        Tanzu Cluster IP address range                      Tanzu Cluster IP address range
53      UDP	        Tanzu Cluster IP address range                      Tanzu Cluster IP address range
80      TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
443     TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
2379	TCP	        Tanzu Management IP address range	                Tanzu Management IP address range
2380	TCP	        Tanzu Management IP address range	                Tanzu Management IP address range
2381	TCP	        Tanzu Management IP address range	                Tanzu Management IP address range
5000	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
6443	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
8000	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
8073	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
8080	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
8081	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
8383	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
8443	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9402	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9440	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9441	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9808	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9844	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9845	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9846	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9847	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9848	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9850	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9851	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9853	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9874	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9875	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9876	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9877	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9878	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9880	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9887	TCP	        Tanzu Management IP address range	                Tanzu Management IP address range
9944	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9945	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9946	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
9950	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
10250	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
10256	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
10450	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
29000	TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
*       TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range
*       TCP	        Tanzu Cluster IP range, Tanzu Management IP range	Tanzu Cluster IP range, Tanzu Management IP range

FIX

With vSphere 7.0.3 and NSX-T 3.2, NCP will auto create an additional rule to allow all egress traffic like below

Allow intra-namespace, masterVM, LB sourceIP to namespace ingress traffic
Allow any to Master VM ingress traffic
Allow all egress traffic
Deny all other ingress traffic

The problem?#

Workaround#

FIX#

The problem?

Workaround

FIX