AKS workshop: Azure CNI Overlay + Cilium + Azure Kubernetes Application Network¶

Educational preview lab, not a production reference architecture

Azure Kubernetes Application Network is in public preview (no SLA, best-effort support). Running it on Azure CNI Overlay with the Cilium dataplane is not documented by Microsoft. The lab runs that combination only after you explicitly accept it as an experiment (approval gate G4). A working lab is runtime evidence, not proof of Microsoft support. Read Support & sources before you start.

What you will build¶

Two Linux AKS Standard clusters in two peered VNets (same region by default). You'll see:

Azure CNI Overlay powered by Cilium, with Kubernetes NetworkPolicy (L3/L4) enforced by Cilium.
Application Network (managed Istio ambient, no sidecars) with both clusters in one Application Network. It provides:
- cross-cluster service discovery
- mTLS with a shared managed root CA
- L4 identity authorization (ztunnel) and L7 authorization (waypoints)
A Gateway API ingress gateway restricted to your IP, with HTTP and HTTPS and routing to local and remote backends.
JWT validation and claim routing, weighted cross-cluster routing, fault injection, metrics/logs, lifecycle operations, a safe failure drill, and a full, audited cleanup.

flowchart LR
  you["You (ALLOWED_CLIENT_CIDR)"] -->|HTTP/HTTPS| ig
  subgraph v1["VNet 1 10.10.0.0/16"]
    subgraph c1["AKS cluster 1"]
      ig["ingress gateway"] --> pp["productpage"]
      pp --> wp1["waypoint"]
      ew1["east-west gw (internal)"]
    end
  end
  subgraph v2["VNet 2 10.20.0.0/16"]
    subgraph c2["AKS cluster 2"]
      ew2["east-west gw (internal)"] --> rv["reviews v2/v3, ratings"]
    end
  end
  v1 <-->|VNet peering| v2
  wp1 -->|"mTLS (HBONE)"| ew2
  an["Application Network<br/>(Azure-managed control plane + CA)"] -.-> c1
  an -.-> c2

Peering provides underlay reachability. Application Network provides service discovery, workload identity and encrypted mesh traffic. Overlay pod IPs are not routable across peering.

Get the lab¶

git clone https://github.com/ohadse-work/aks-appnet-cilium-workshop.git
cd aks-appnet-cilium-workshop
cp config/lab.env.example config/lab.env     # edit SUBSCRIPTION_ID, TENANT_ID, LAB_OWNER
scripts/00-check-local-tools.sh              # local tools only
scripts/01-preflight-azure.sh                # READ-ONLY Azure preflight

Then follow the Lab guide module by module. Nothing billable, preview-related or cluster-changing runs without an approval gate, where you type approve <GATE>.

Workshop at a glance¶


Audience	Engineers new to AKS networking and service mesh. Comfortable with a terminal.
Duration	About 8 hours of hands-on work. See the agenda for one-day and two-day formats.
Cost	Roughly USD 0.80–1.00/hour while both clusters run, plus optional monitoring. Application Network preview pricing isn't published. Not free.
You need	An Azure subscription where you can create resource groups and role assignments; macOS or Linux (WSL2 is fine); Azure CLI ≥ 2.86.0, kubectl, kubelogin, jq, curl, openssl
Not installed	AKS Istio add-on, self-managed Istio, a second Cilium, ACNS, Cilium mTLS/L7/Cluster Mesh, the AKS app-routing add-on

Modules¶

#	Module	Key evidence you'll collect
1	Planning, prerequisites, cost	Read-only preflight report
2	Networks and AKS clusters	Overlay + Cilium proven from Azure and in-cluster; no kube-proxy
3	Baseline L3/L4 NetworkPolicy	Timeouts proven to be policy, not DNS or a missing backend
4	Onboard Application Network	Members, ztunnel, east-west reachability, no sidecars
5	Cross-cluster discovery	Remote-only service answered by named cluster-2 pods
6	mTLS and identity	Shared root CA; plaintext rejected under STRICT
7	Ingress gateway	IP-restricted NSG rule; HTTPS with verified cert
8	L4 identity authorization	`CONN_RESET` for the wrong identity with the network open
9	L7 authorization via waypoints	`403 RBAC`, bypass blocked, ingress gap closed
10	Layered NetworkPolicy + mesh	Impostor pod: label passes, identity fails
11	Advanced traffic management	401 vs 403 vs 404; 20/80 split; injected delay/abort
12	Observability and lifecycle	Waypoint metrics, access logs, PromQL/KQL
13	Safe failure exercise	Endpoint-set behaviour, no failover claims
14	Cleanup	Leftover scan: nothing billable left

How to read results¶

Every exercise classifies each request, so you can tell the enforcing layer apart:

Result	Meaning
`TIMEOUT` / `UPSTREAM_CONNECT_TIMEOUT`	Dropped by NetworkPolicy (Cilium)
`CONN_RESET` / `UPSTREAM_RESET`	Rejected by ztunnel (mTLS STRICT or L4 identity policy)
`RBAC_DENIED` (403)	L7 AuthorizationPolicy in a waypoint or gateway
`JWT_REJECTED` (401)	Token failed validation
`NOT_FOUND` (404)	Route miss or app 404, never proof of authentication