Skip to content

Step-by-step lab: two AKS clusters, Azure CNI Overlay + Cilium, Azure Kubernetes Application Network

Educational preview lab. Not a production reference architecture. Application Network is in preview. Running it on Azure CNI Overlay with the Cilium dataplane is not documented by Microsoft (see support-and-sources.md). Every "expected output" below is ILLUSTRATIVE (from the docs or from how the tools format output) until you observe it. Record real results in acceptance.md.

Contents

How to use this guide

  • Working directory: always the lab root (the folder that contains README.md).
  • Scripts are small and step-based. scripts/exNN-*.sh help lists the steps in order. Every command a script runs is printed first (lines starting with +), so you can see and re-type it yourself.
  • Explicit contexts. Every kubectl/istioctl call uses the lab-only kubeconfig (.kube/appnet-lab.kubeconfig) and --context appnet-c1 or --context appnet-c2. Nothing touches ~/.kube/config.
  • Approval gates. Steps that create billable resources, register previews or change cluster lifecycle stop and ask you to type approve <GATE>. For unattended runs, export LAB_APPROVE_<GATE>=yes, but only after you've read the gate text.
  • Copy-paste setup (for running the commands shown under "Under the hood"):
cd <lab-root>
export LAB_ROOT="$PWD"; set -a; source config/lab.env; set +a
export KUBECONFIG="$LAB_KUBECONFIG"
export PATH="$LAB_ROOT/.tools/.istioctl/bin:$PATH"     # after script 08
You see Most likely cause Layer
TIMEOUT (curl exit 28 after DNS resolved) Packet dropped by NetworkPolicy (Cilium) L3/L4 network
UPSTREAM_CONNECT_TIMEOUT (Envoy 503 "…connection timeout") A proxy's upstream was dropped (NetworkPolicy) L3/L4 network
CONN_RESET (curl exit 56/52) ztunnel accepted then rejected: mTLS STRICT or L4 AuthorizationPolicy L4 mesh
UPSTREAM_RESET (Envoy 503 "…connection termination") A proxy's upstream was rejected by ztunnel L4 mesh
RBAC_DENIED (HTTP 403 "RBAC: access denied") Envoy (waypoint/gateway) L7 AuthorizationPolicy L7 mesh
JWT_REJECTED (HTTP 401) Token failed validation L7 authn
NOT_FOUND (HTTP 404) Route miss or app 404, not an auth result Routing/app
DNS_FAILURE / DNS_TIMEOUT Name didn't resolve / DNS traffic blocked DNS
FAULT_ABORT / NO_HEALTHY_UPSTREAM Injected fault / no endpoints Mesh routing

Architecture

flowchart LR
  subgraph Azure["One subscription / one tenant / one region (default eastus2)"]
    direction LR
    AN["Application Network resource<br/>(managed control plane per member,<br/>managed root CA)"]
    subgraph V1["VNet 1 10.10.0.0/16"]
      subgraph C1["AKS cluster 1 (appnet-c1)"]
        N1["CNI Overlay + Cilium<br/>pods 100.64.0.0/16<br/>services 10.110.0.0/16"]
        Z1["ztunnel + istio-cni<br/>(DaemonSets, applink-system)"]
        EW1["east-west gateway<br/>(internal LB)"]
        WP1["waypoint (bookinfo)"]
        IG["ingress gateway<br/>(public LB, your IP only)"]
        APP1["productpage · details · reviews-v1"]
      end
    end
    subgraph V2["VNet 2 10.20.0.0/16"]
      subgraph C2["AKS cluster 2 (appnet-c2)"]
        N2["CNI Overlay + Cilium<br/>pods 100.65.0.0/16<br/>services 10.120.0.0/16"]
        Z2["ztunnel + istio-cni"]
        EW2["east-west gateway<br/>(internal LB)"]
        WP2["waypoint (bookinfo)"]
        APP2["reviews-v2 · reviews-v3 · ratings"]
      end
    end
    V1 <-->|"VNet peering (both directions)<br/>routes VNet ranges only"| V2
    AN -. "config, certs, discovery" .-> C1
    AN -. "config, certs, discovery" .-> C2
  end
  User["Your machine<br/>(ALLOWED_CLIENT_CIDR)"] -->|"HTTP/HTTPS"| IG
  EW1 <-->|"mTLS (HBONE) over peering"| EW2

Key idea: peering provides underlay reachability between the VNets, which includes the internal east-west gateway IPs. Application Network provides service discovery, identity and encrypted mesh traffic on top of it. Overlay pod IPs are never routable across peering. A pod's traffic leaving its cluster is SNATed to its node IP.

Cross-cluster request path (Module 5):

sequenceDiagram
  participant P as productpage (c1)
  participant Z1 as ztunnel (c1)
  participant W1 as waypoint (c1)
  participant E2 as east-west gw (c2)
  participant Z2 as ztunnel/pod (c2)
  participant R as reviews-v2 (c2)
  P->>Z1: plain HTTP to reviews:9080 (captured in-pod)
  Z1->>W1: HBONE (mTLS, port 15008)
  W1->>E2: HBONE over VNet peering (endpoint in remote network)
  E2->>Z2: HBONE (mTLS)
  Z2->>R: plain HTTP inside the pod

The exact hops between the east-west gateway and the pod (for example, whether a cluster-2 waypoint is involved) are observed in Modules 5 and 10, not assumed.

Module 1: Planning, prerequisites and cost

Objective. Confirm tools, identity, permissions, region, versions, SKUs and quota, and understand the cost before anything is created.

Concepts. - Resource group: a folder for Azure resources. This lab creates three, all tagged lab=<LAB_TAG_VALUE>, so cleanup can prove what it owns. - Preview feature registration: a subscription-wide opt-in (Microsoft.AppLink/PublicPreview).

Resources and traffic path. None created. Read-only Azure API calls.

Working directory and variables. Lab root. Copy and edit the config:

cp config/lab.env.example config/lab.env
# set SUBSCRIPTION_ID, TENANT_ID, LAB_OWNER (and later ALLOWED_CLIENT_CIDR)
az login --tenant <TENANT_ID>
az account set --subscription <SUBSCRIPTION_ID>

Platform notes.

macOS Linux
Azure CLI brew install azure-cli Microsoft package repo
kubectl + kubelogin brew install kubectl Azure/kubelogin/kubelogin or az aks install-cli sudo az aks install-cli
jq brew install jq apt-get install jq (or distro equivalent)
bash Scripts work with the built-in bash 3.2 bash 4/5
openssl LibreSSL (built in) works OpenSSL

Steps. 1. scripts/00-check-local-tools.sh: checks tools and versions (az ≥ 2.86.0) and the appnet-preview extension. 2. az extension add --name appnet-preview if reported missing (local change only). 3. scripts/01-preflight-azure.sh (read-only). Checks: - the region is in both the documented region list and the provider location list - the CIDR plan has no overlaps and avoids AKS-reserved ranges - feature/provider state - Kubernetes 1.35 is offered as GA - live az appnet list-versions (after registration) - SKU restrictions and vCPU quota - your role assignments - pre-existing resource groups and Azure Policy assignments 4. Read support-and-sources.md. Then gate G1: scripts/02-register-preview.sh registers the feature and provider and prints az appnet list-versions. 5. Re-run scripts/01-preflight-azure.sh. Choose APPNET_VERSION (or leave empty for the documented N-1 default) and set ISTIO_VERSION/ISTIO_SAMPLES_REF to the matching Istio minor (supported-versions table: AppNet 1.4 → Istio 1.29, 1.5 → 1.30).

Expected output (ILLUSTRATIVE).

==> 1. Region support (eastus2)
[ OK ] eastus2 is in the documented Application Network region list
...
==> 5. Application Network versions compatible with Kubernetes 1.35
[WARN] SKIPPED: 'az appnet list-versions' needs the preview registration (gate G1_PREVIEW).
...
[ OK ] Preflight passed with 2 warning(s).

Tests. Positive: preflight passes. Negative: set POD_CIDR_C2="172.30.0.0/16" and preflight must fail with OVERLAP POD2 ... RESERVED_172_30 (revert afterwards).

Interpreting results. Any [FAIL] blocks resource creation. [WARN] lines are informational or will be resolved by a gate.

Node sizing. Each cluster has 2 × 4 vCPU / 16 GiB nodes. Ambient adds one ztunnel and one istio-cni pod per node, plus per-namespace waypoints, one east-west gateway and the ingress gateways. Bookinfo's reviews pods are Java and the heaviest part. Two nodes per cluster leave headroom for a surge node during upgrades (quota check counts it).

Cost estimate (assumptions: eastus2, pay-as-you-go retail, Linux, prices read on 2026-09-30).

Item Quantity Unit price Per hour
Nodes Standard_D4as_v7 4 $0.182/h $0.728
OS disks (64 GiB Premium SSD, P6) 4 $9.28/month (~$0.0127/h) ~$0.051
Public IPs (2 outbound + 1 ingress) 3 $0.005/h $0.015
AKS control plane (tier free) 2 $0 (Standard tier would be $0.10/h each) $0
Standard Load Balancer rules + data processed 2 LBs see LB pricing (not auto-retrieved) small, check
VNet peering per GB $0.01/GB each direction ~$0 for lab traffic
Internet egress (image pulls are ingress) small bandwidth pricing ~$0
Application Network (preview) 1 + 2 members no published price found unknown
Monitoring (Module 12, optional, gated) per GB / per sample Log Analytics $2.76/GB, managed Prometheus pricing varies

Rough total: about USD 0.80–1.00 per hour (~USD 20–25 per day) while both clusters run, before monitoring and before any Application Network charge. This lab is not free. Stop or clean up when you pause (Module 14). Check Cost Management over the following days.

Common errors. SubscriptionNotRegistered → gate G1 not done. NotAvailableForSubscription → change NODE_VM_SIZE. Quota failure → request quota or reduce NODE_COUNT.

Rollback. Nothing to roll back (read-only). Registration (G1) is intentionally left in place.

What you demonstrated. A reproducible, evidence-based go/no-go before spending money.

Module 2: Networks and AKS clusters

Objective. Build two VNets with bidirectional peering and two AKS clusters, then prove their actual configuration.

Concepts. - Pod: the smallest runnable unit in Kubernetes (one or more containers sharing a network namespace/IP). - Deployment: keeps N identical pods running and replaces them when they fail. - Service: a stable virtual IP + DNS name in front of a set of pods (selected by labels). - Namespace: a scope for names and policies inside a cluster. - Service account: the identity a pod runs as (used later for mesh identity). - CNI (Container Network Interface): the plugin that gives pods their network. Here it's Azure CNI with the Cilium eBPF dataplane, which also enforces NetworkPolicy and replaces kube-proxy. - Overlay: pods get IPs from a private range (100.64.0.0/16) that isn't part of the VNet. Nodes route pod traffic among themselves, and traffic leaving the cluster is SNATed to the node IP.

Resources and traffic path. 3 RGs, 2 VNets + node subnets, 2 peerings, 2 user-assigned identities (Network Contributor on their own subnet only), 2 AKS clusters with public API servers. Traffic: none yet.

Steps. 1. Gate G2: scripts/03-create-network.sh. Creates the RGs, VNets and subnets, and peering in both directions. Waits until both peerings are Connected/FullyInSync. Under the hood: az network vnet peering create ... --allow-vnet-access true. 2. Gate G2: scripts/04-create-identities.sh. Creates the UAMIs and subnet-scoped role assignments (with bounded retries). 3. Gate G3: scripts/05-create-clusters.sh. Under the hood (per cluster):

az aks create -g "$RG_C1" -n "$CLUSTER1_NAME" -l "$LOCATION" \
  --kubernetes-version "$KUBERNETES_VERSION" --tier free \
  --nodepool-name system --node-count 2 --node-vm-size "$NODE_VM_SIZE" --node-osdisk-size 64 \
  --network-plugin azure --network-plugin-mode overlay --network-dataplane cilium \
  --pod-cidr 100.64.0.0/16 --service-cidr 10.110.0.0/16 --dns-service-ip 10.110.0.10 \
  --vnet-subnet-id <subnet-1-id> --assign-identity <uami-1-id> \
  --enable-aad --enable-azure-rbac --enable-oidc-issuer --enable-gateway-api \
  --auto-upgrade-channel none --no-ssh-key --tags ... --no-wait

Then it assigns you AKS RBAC Cluster Admin + Cluster User Role scoped to each cluster. --auto-upgrade-channel none stops the cluster being upgraded to a Kubernetes version outside the Application Network compatibility table in the middle of the lab. 4. scripts/06-get-credentials.sh. Writes a dedicated kubeconfig with contexts appnet-c1/appnet-c2, converts it for kubelogin (Entra ID), and checks each context's API server FQDN. 5. scripts/07-verify-clusters.sh (read-only). Checks these from Azure and from inside each cluster: - networkPlugin=azure, networkPluginMode=overlay, networkDataplane=cilium, networkPolicy=cilium - the pod and Service CIDRs - OIDC issuer, managed Entra ID and Azure RBAC are on - not private, no authorized IP ranges, no Istio add-on - a cilium DaemonSet exists and there's no kube-proxy - Gateway API CRDs are managed-by: aks, channel standard

It also records the Cilium settings that matter for ambient (bpf-lb-sock-hostns-only, cni-exclusive, enable-bpf-masquerade) in out/cilium-ambient-compat.txt. 6. scripts/08-install-istioctl.sh: istioctl ISTIO_VERSION into .tools/ (local).

Expected output (ILLUSTRATIVE).

==> Azure view of appnetlab-c1
[ OK ] networkProfile.networkPluginMode = overlay
[ OK ] networkProfile.networkDataplane = cilium
[ OK ] No kube-proxy DaemonSet (Cilium replaces kube-proxy, as documented)
==> Cilium settings relevant to Istio ambient on cluster-1 (evidence only)
kube-proxy-replacement       true
bpf-lb-sock-hostns-only      <observed value>
cni-exclusive                <observed value>
enable-bpf-masquerade        <observed value>

Tests. Positive: all checks [ OK ]. Negative: kubectl --context appnet-c1 get nodes without KUBECONFIG set to the lab file must not reach the lab cluster (proves isolation of your default kubeconfig).

Interpreting results. A mismatch in plugin/mode/dataplane means the cluster isn't what the lab requires. Delete and recreate it rather than patching. RISK lines about Cilium settings are evidence for gate G4, not failures.

Common errors. - LinkedAuthorizationFailed / subnet permission errors: role assignment not propagated yet. Wait and re-run 05. - kubectl asks for device login or returns Forbidden: Azure RBAC propagation. 06 waits up to 10 minutes. - Policy denies (e.g. "subnets must have NSG"): see troubleshooting.

Rollback. scripts/cleanup-2-azure.sh removes everything (Module 14). Clusters are never modified in place.

What you demonstrated. Two independent VNets and clusters. The configuration is proven from two angles, and your kubeconfig is isolated.

Module 3: Baseline connectivity and L3/L4 NetworkPolicy

Objective. Learn Kubernetes NetworkPolicy (enforced by Cilium) before any mesh exists: baseline, default deny, selective allow, and egress with DNS preserved. Prove a denial is caused by policy, not by broken DNS or a missing backend.

Concepts. - NetworkPolicy: an allow-list for pod traffic by labels, namespaces, IP blocks and ports. Once a pod is selected for a direction (ingress or egress), only allowed traffic passes. Cilium drops the rest, so clients see timeouts. - Policies select labels, not identities, and they're per cluster.

Resources and traffic path. Cluster 1, namespace netpol-lab (not in the mesh): backend (agnhost /hostname returns its pod name, with a kubelet readiness probe), plus client-allowed and client-denied. Path: client pod → Service backend:80 → pod :8080.

Steps (scripts/ex03-netpol-baseline.sh <step>). 1. deploy 2. baseline: both clients get HTTP 200 and the body is the backend pod name. 3. deny applies 30-default-deny-ingress.yaml. Both clients TIMEOUT. The script then proves the backend has ready EndpointSlices, is Ready (kubelet probes still pass), and that curl's error is a connect timeout. So DNS resolved. 4. allow applies 40-allow-backend-from-client-allowed.yaml (TCP 8080, the pod port, not the Service port 80). 5. egress-deny applies an egress default-deny without DNS: client-allowed now fails with DNS_TIMEOUT/DNS_FAILURE. 6. egress-allow allows kube-dns UDP/TCP 53 and backend:8080. client-allowed works again, and client-denied still times out because of the backend's ingress policy. 7. evidence, then reset.

Expected output (ILLUSTRATIVE).

+ [cluster-1] netpol-lab/client-denied: curl  http://backend.netpol-lab.svc.cluster.local/hostname
     result: class=TIMEOUT http=000 curl_exit=28
     | curl: (28) Connection timed out after 5002 milliseconds
[ OK ] EXPECTED TIMEOUT: client-denied is still dropped (negative test)

Tests. Positive: baseline, allow (client-allowed), egress-allow. Negative: deny, allow (client-denied), egress-deny.

Interpreting results. TIMEOUT with DNS resolved means a policy drop. DNS_* means DNS is blocked. CONNECT_REFUSED would mean nothing is listening (a backend problem, not policy).

Common errors. AKS LocalDNS enabled: pods query 169.254.10.0/24, so allow it (Cilium doc FAQ). ipBlock can't select pod/node IPs with Cilium: use pod/namespace selectors.

Rollback. reset (deletes all NetworkPolicies in netpol-lab). teardown deletes the namespace. Do this before Module 4.

What you demonstrated. Default deny, selective allow, egress with DNS preserved, and a disciplined way to prove that a failure is a policy decision.

Module 4: Onboard Application Network

Objective. Create one Application Network, join both clusters, verify the managed components and east-west reachability, and enroll only the intended namespace into ambient without sidecars.

Concepts. - Service mesh (ambient mode): a layer that secures and controls service-to-service traffic. "Ambient" means no sidecar containers. - ztunnel: a per-node L4 proxy (DaemonSet). It captures an enrolled pod's traffic inside that pod's network namespace and wraps it in HBONE (HTTP CONNECT over mTLS, port 15008). - istio-cni: a per-node agent that sets up that traffic capture. - East-west gateway: the entry point for cross-cluster mesh traffic (internal load balancer by default). - mTLS: mutual TLS. Both sides present certificates, so each proves its identity and the traffic is encrypted. - Trust domain: the naming scope of identities (cluster.local here). Every workload certificate chains to the Application Network root CA.

Resources and traffic path. - Application Network resource in RG_APPNET, plus 2 members. - In each cluster, applink-system gets ztunnel, istio-cni, the Istio CRDs and an east-west gateway. - The control plane is Azure-managed, outside your clusters.

Steps (scripts/ex04-appnet-onboard.sh <step>). 1. acknowledge (gate G4_EXPERIMENTAL): shows what's documented and what isn't, plus your Cilium evidence. You accept running this as an experiment. 2. create (gate G5_APPNET): az appnet create --resource-group ... --name ... --location ... --identity-type SystemAssigned. 3. join (gate G5_APPNET): az appnet member join ... --member-resource-id <aks id> --upgrade-mode SelfManaged [--version X]. The mode is fixed after join. 4. verify: az appnet member list/show (provisioningState=Succeeded, eastWestGateway.visibility=Internal). Then, in-cluster: - ztunnel/istio-cni-node are fully Ready - the Istio CRDs are present (the doc sample lists 14) - the ztunnel image tag matches your istioctl minor - the LoadBalancer Service for the east-west gateway exists 5. underlay: from a non-mesh test pod in each cluster, TCP-connect to the other cluster's east-west gateway IP:port over peering. This proves the documented requirement that you provide gateway-to-gateway reachability. The port is read from the live Service (15008 preferred), not assumed. 6. enroll: creates bookinfo in both clusters and labels only it istio.io/dataplane-mode=ambient. It lists all enrolled namespaces. 7. no-sidecars: deploys test clients, confirms there's no istio-proxy container, shows the ambient.istio.io/redirection annotation (UPSTREAM), and runs istioctl ztunnel-config workload.

Expected output (ILLUSTRATIVE, from the doc sample).

NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
istio-cni-node   2         2         2       2            2           kubernetes.io/os=linux   10m
ztunnel          2         2         2       2            2           kubernetes.io/os=linux   10m
...
[ OK ] cluster-1 -> cluster-2 east-west TCP 15008 reachable
curl-6d8f...   containers=curl   init=   ambient-redirection=enabled

Tests. Positive: members Succeeded, DaemonSets ready, underlay TCP connect both ways. Negative: kube-system, applink-system and netpol-lab must not appear in the enrolled list, and no pod may have istio-proxy.

Interpreting results. Underlay failure means peering, NSG or UDR problems (troubleshooting). A ztunnel image minor that differs from istioctl means aligning ISTIO_VERSION.

Common errors. A join error mentioning the Istio add-on, OIDC or Entra ID: the cluster prerequisites. Expected exactly 1 LoadBalancer Service: inspect applink-system Services and adapt.

Rollback. Un-enroll: kubectl --context <ctx> label ns bookinfo istio.io/dataplane-mode-. Member removal happens in Module 14 (it requires un-enrollment first).

What you demonstrated. Managed ambient onboarding, a sidecar-free data plane, explicit enrollment scope, and verified underlay reachability between east-west gateways.

Module 5: In-cluster and cross-cluster service discovery

Objective. Deploy the official distributed Bookinfo, then prove that requests are served by the remote cluster using pod identity. An HTTP 200 alone proves nothing.

Concepts. - Waypoint: an optional per-namespace Envoy proxy for L7 (HTTP) features: routing, L7 policy, fault injection. It's a Gateway API Gateway with class istio-waypoint. - Global service (istio.io/global=true): a Service whose endpoints from all member clusters are merged.

Resources and traffic path. - Cluster 1 runs productpage, details, reviews-v1 and the curl clients. - Cluster 2 runs reviews-v2, reviews-v3, ratings and the curl clients. - There's one waypoint per cluster in bookinfo. - reviews-local (c1 pods only) and reviews-remote (zero c1 endpoints; c2 v2+v3) come from the doc's traffic-shifting section. - Each reviews deployment gets CLUSTER_NAME=cluster-1|2, which Bookinfo echoes in every response.

Steps (scripts/ex05-cross-cluster-discovery.sh <step>). 1. deploy: the doc's kubectl apply -f <bookinfo.yaml> -l <selector> commands (namespace bookinfo), plus set env CLUSTER_NAME. 2. waypoint: istioctl waypoint apply -n bookinfo --enroll-namespace --wait in both clusters. 3. global: label reviews and waypoint Services global in both clusters (doc). 4. discovery: istioctl ztunnel-config service. Global services list two VIPs, one from 10.110.0.0/16 and one from 10.120.0.0/16. That proves the definitions from both clusters were merged. 5. remote-only: shows 0 local EndpointSlice endpoints for reviews-remote in cluster 1, then 5 requests from cluster 1. Every response must say "clustername": "cluster-2" with a reviews-v2/v3 pod name that exists only in cluster 2. 6. global-mix: 20 requests to global reviews, counted per serving cluster (no weights configured). 7. productpage: end to end, reading "Reviews served by: on cluster ". 8. not-global: observation. ratings isn't global and has no Service in cluster 1. 9. evidence: ztunnel-config workload (PROTOCOL HBONE) and cluster-2 ztunnel log lines for the serving pod.

Expected output (ILLUSTRATIVE).

NAMESPACE  SERVICE NAME    SERVICE VIP               WAYPOINT  ENDPOINTS
bookinfo   reviews         10.110.x.y,10.120.x.y     waypoint  3/3
bookinfo   reviews-remote  10.110.x.y,10.120.x.y     waypoint  2/2
...
[ OK ] served by pod reviews-v3-7d9c... on cluster-2
  Summary: cluster-1=7 cluster-2=13 unknown=0

Tests. Positive: remote-only requests served by cluster 2. Negative: reviews-remote has 0 local endpoints (removing any doubt that a local backend answered). not-global shows discovery is opt-in.

Interpreting results. If remote-only fails but underlay passed, check istio.io/global labels on both sides and the waypoint Programmed status.

Scope note (contradiction C1). The overview page says "single-cluster". The architecture page and this walkthrough document multi-cluster. Treat multi-cluster as documented preview behaviour with an inconsistent scope statement.

Common errors. See troubleshooting.md: Module 5: In-cluster and cross-cluster service discovery for symptom → diagnosis → fix with exact commands.

Rollback. Keep Bookinfo for later modules. Full removal is in Module 14.

What you demonstrated. Cross-cluster discovery and routing, with the serving pod and cluster identified in every response.

Module 6: mTLS and identity

Objective. Tell automatic encryption apart from mandatory mTLS. Show authenticated traffic in-cluster and cross-cluster, then show a plaintext client rejected while its network reachability stays intact.

Concepts. - PERMISSIVE (documented default): encrypted and unencrypted connections are accepted. - STRICT: only mTLS peers are accepted (PeerAuthentication, namespace-scoped here). - Identity: spiffe://cluster.local/ns/<namespace>/sa/<service-account>. There is no cluster name in it. The same namespace + service account in either cluster is the same principal. - CA hierarchy (documented): - one root CA per Application Network (12-month validity, rotated every 6 months, backed by Key Vault) - one intermediate CA per member (90 days / 45 days) - workload certificates (24 hours / 12 hours)

Rotation is automatic, and the root certificate is published in ConfigMap applink-system/istio-root-cert. It's public data. The lab never prints private keys.

Resources and traffic path. Namespace lab-outside (cluster 1, not in the mesh) with curl-outside and a non-mesh plain-echo. The path is curl-outside → Cilium → productpage pod → ztunnel inbound (plaintext).

Steps (scripts/ex06-mtls-identity.sh <step>). 1. deploy-outside 2. identity: - ztunnel-config workload (HBONE for bookinfo) - the SPIFFE IDs derived from service accounts - ztunnel-config certificate <ztunnel> (summary, no keys) - the root CA subject, issuer, expiry and SHA-256 fingerprint in both clusters, which must be identical 3. permissive: mesh and non-mesh clients both succeed, and ztunnel logs show which connections carried a peer identity. 4. strict: applies 10-peer-authentication-strict.yaml in both clusters. 5. test-strict: - Positive first: mesh client → productpage OK, and mesh client → reviews-remote OK served by cluster-2 (authenticated cross-cluster hop). - Negative: curl-outside → productpage gives CONN_RESET. - Control: curl-outside → plain-echo OK, and no NetworkPolicy exists. So the rejection is mTLS policy, not networking. 6. reset: removes STRICT (returns to the documented default so later modules behave as the docs describe).

Expected output (ILLUSTRATIVE).

cluster-1:
     subject=... issuer=... notAfter=...
     sha256 Fingerprint=AB:CD:...
cluster-2:
     sha256 Fingerprint=AB:CD:...
[ OK ] Same root CA fingerprint in both clusters
...
     result: class=CONN_RESET http=000 curl_exit=56
[ OK ] EXPECTED CONN_RESET: plaintext request to a STRICT workload is rejected

Interpreting results. Evidence of mTLS comes from HBONE in the ztunnel config, identities in the logs and certificates, the rejection under STRICT, and a working control path. HTTP 200 alone proves none of this. A TIMEOUT under STRICT points to a leftover NetworkPolicy (Module 3/10).

Common errors. See troubleshooting.md: Module 6: mTLS and identity for symptom → diagnosis → fix with exact commands.

Rollback. reset.

What you demonstrated. The difference between opportunistic and mandatory mTLS, a shared trust root across clusters, and namespace/service-account identity scope.

Module 7: Ingress gateway

Objective. Expose Bookinfo north-south with the Gateway API, route to local and remote backends, and add HTTPS for the client-to-gateway hop.

Concepts. - Gateway (Gateway API): declares a listener (port, protocol, TLS). With class istio, Istio deploys an Envoy gateway plus a LoadBalancer Service and ServiceAccount named <gateway>-istio (UPSTREAM naming). - HTTPRoute: path and host routing rules attached to a Gateway. This is the "application routing" in this lab. - North-south vs east-west: north-south traffic enters from outside the mesh through the ingress gateway. East-west traffic flows between services, and between clusters through the east-west gateways. - Client-to-gateway TLS vs mesh mTLS: HTTPS ends at the ingress gateway. The gateway → productpage hop is a separate hop protected by mesh mTLS (HBONE).

Resources and traffic path. One public Standard LB frontend + public IP (cluster 1). service.beta.kubernetes.io/azure-allowed-ip-ranges=<ALLOWED_CLIENT_CIDR> is set through spec.infrastructure.annotations, so AKS creates NSG rules for your IP only. The path is: you → public IP → ingress gateway → productpage (c1) → reviews (c1 or c2).

Steps (scripts/ex07-ingress.sh <step>). Set ALLOWED_CLIENT_CIDR first (curl -s https://api.ipify.org → x.x.x.x/32). 1. deploy (gate G6_INGRESS): 10-bookinfo-gateway.yaml + 20-httproute-local-remote.yaml. 2. nsg-evidence: lists the AKS-managed NSG rules whose destination is the gateway IP. The source must be your CIDR. 3. test-http: - /productpage → 200, with the serving reviews pod/cluster shown - /lab/local/reviews/1 → cluster-1 - /lab/remote/reviews/1 → cluster-2 (experiment G-9; the result is recorded either way) - /not-a-route → NOT_FOUND (a route miss) 4. https: creates a 7-day lab CA and a server certificate with SAN TLS_HOST in .secrets/tls/ (git-ignored, keys chmod 600), creates the Secret bookinfo-tls (never printed), and adds the HTTPS listener (30-bookinfo-gateway-https.yaml). 5. test-https: - with --cacert: OK_2XX - without trusting the CA: TLS_ERROR (proves the certificate is actually verified) - prints the served certificate's subject and SAN

Expected output (ILLUSTRATIVE).

bookinfo-gateway   istio   20.x.x.x   True   40s
...
AllowAnyCustomAnyInbound-...   Allow   Inbound   src=203.0.113.10/32   ports=80
...
     result: class=TLS_ERROR http=000 curl_exit=60

Out of validated scope. External-client mTLS (client certificates) isn't documented for Application Network, so it isn't included. Never send tokens or passwords to the HTTP URL.

Common errors. - Gateway Programmed=False: kubectl describe gateway. - No response from your machine: ALLOWED_CLIENT_CIDR doesn't match your current egress IP, or a corporate proxy is in the way. - 404 on /productpage: the HTTPRoute isn't attached (kubectl get httproute -o yaml, check status.parents).

Rollback. Keep the gateway: Modules 8–11 use it. teardown removes it.

What you demonstrated. A restricted public entry point, path routing to local/remote backends, and TLS termination kept separate from mesh mTLS.

Module 8: L4 identity authorization

Objective. Allow one named service-account identity and reject another at L4, while the network stays fully open. Then any rejection is purely mesh authorization.

Concepts. - Authentication: who are you? (the peer's SPIFFE ID, proven by its mTLS certificate). - Authorization: are you allowed? (an AuthorizationPolicy decision about that identity). - A policy with selector is enforced by ztunnel at L4: it can match identities, namespaces and ports, but not HTTP.

Resources and traffic path. - The documented policy productpage-ztunnel: only sa/bookinfo-gateway-istio may connect to productpage pods. - A clean namespace l4-lab (ambient, no waypoint) with echo (cluster 1) and client-allowed/client-denied (both clusters). echo is global.

Steps (scripts/ex08-l4-authz.sh <step>). 1. precheck: no NetworkPolicy in bookinfo/l4-lab. Ingress and mesh paths return 200. 2. doc-policy: - ingress → 200 (allowed identity) - curl → productpage service → UPSTREAM_RESET (503). Because of the namespace waypoint, the rejected identity is actually sa/waypoint. - curl → productpage pod IP (skips the waypoint) → CONN_RESET, with sa/curl rejected directly. 3. l4lab-deploy, l4lab-baseline: both identities work before any policy. 4. l4lab-policy: allows cluster.local/ns/l4-lab/sa/client-allowed only. client-allowed gets 200, client-denied gets CONN_RESET, and the ztunnel log shows the decision. 5. cross-cluster (observation): client-allowed in cluster 2 is expected to be allowed by a policy written in cluster 1, because the principal has no cluster component. 6. reset.

Expected output (ILLUSTRATIVE).

     result: class=CONN_RESET http=000 curl_exit=56
     | curl: (56) Recv failure: Connection reset by peer
[ OK ] EXPECTED CONN_RESET: other identity rejected by ztunnel at L4 (negative)

Interpreting results. Compare with Module 3: TIMEOUT meant the packet was dropped (network). CONN_RESET means the connection was accepted, the identity checked, and access refused (mesh L4 authorization).

Common errors. See troubleshooting.md: Module 8: L4 identity authorization for symptom → diagnosis → fix with exact commands.

Rollback. reset (the workloads stay for Module 10).

What you demonstrated. Identity-based L4 allow/deny, independent of network reachability, and the cross-cluster meaning of an identity.

Module 9: L7 authorization through waypoints

Objective. Allow GET and deny DELETE (and deny other identities) at L7. Test every path to catch policy bypass, including ingress.

Concepts. - A policy with targetRefs: Service is enforced by that service's waypoint, at L7. - Mesh traffic now arrives at productpage from the waypoint, so the L4 policy must allow sa/waypoint (doc). - UPSTREAM: the waypoint only handles service-addressed traffic by default. Pod-IP traffic skips it, and ingress gateway traffic skips it unless istio.io/ingress-use-waypoint is set and istiod has ENABLE_INGRESS_WAYPOINT_ROUTING=true (default false; AppNet manages istiod, gap G-5).

Steps (scripts/ex09-l7-authz.sh <step>). 1. attach: the WAYPOINT column in ztunnel-config service. 2. apply: 10-productpage-waypoint-l7.yaml + 20-productpage-ztunnel-l4-with-waypoint.yaml. 3. test-direct: - curl GET → 200 - curl DELETE → RBAC_DENIED - curl-other GET → RBAC_DENIED - pod-IP DELETE → CONN_RESET (the L4 policy closes the bypass) 4. ingress-gap: ingress GET → 200. Ingress DELETE gets the app's status (e.g. 405), not RBAC: access denied. That demonstrates the gap. 5. close-gap: 30-ingress-gateway-deny-writes.yaml (AuthorizationPolicy targetRefs: Gateway, UPSTREAM feature). Ingress DELETE → RBAC_DENIED, GET → 200. 6. reset.

Expected output (ILLUSTRATIVE, doc).

     | RBAC: access denied
[ OK ] EXPECTED RBAC_DENIED: sa/curl DELETE denied at L7 by the waypoint
[ OK ] GAP DEMONSTRATED: ingress DELETE got HTTP_4XX (HTTP 405) from the APP, not 'RBAC: access denied'

Common errors. See troubleshooting.md: Module 9: L7 authorization through waypoints for symptom → diagnosis → fix with exact commands.

Rollback. reset.

What you demonstrated. L7 method/identity rules, how waypoint attachment works, and why every entry path needs its own enforcement point, with L4 identity policy as the backstop.

Module 10: Layered Cilium NetworkPolicy plus Application Network

Objective. Re-introduce scoped L3/L4 NetworkPolicies while ambient is on, without broadly allowing "all mesh traffic". Show that network isolation and identity authorization are independent controls, and that NetworkPolicy is cluster-local.

Concepts (documented upstream behaviour you must account for). - Mesh traffic reaches pods on HBONE port 15008. Policies that allow only the app port break the mesh. - Ambient SNATs kubelet probes to 169.254.7.127. On Cilium, a default-deny then blocks probes unless that /32 is allowed. (A plain ipBlock is used here. Cilium can't use ipBlock for pod/node IPs, but this link-local address is neither.) - Remote-cluster traffic enters through the local east-west gateway pod, so the network-level source is that pod, not the remote client (whose overlay IP isn't visible). - Source selectors are read from the live Services (waypoint, bookinfo-gateway-istio, and the east-west gateway Service), never guessed.

Steps (scripts/ex10-layered-netpol.sh <step>). 1. precheck: no leftover NetworkPolicy, AuthorizationPolicy or PeerAuthentication. All positive paths work. 2. discover: writes out/ex10-selectors.env with each Service's .spec.selector. 3. deny: 10-default-deny-ingress.yaml selects only app pods. Ingress becomes UPSTREAM_CONNECT_TIMEOUT and echo becomes TIMEOUT. Runtime check: does the echo pod turn NotReady (probe SNAT)? 4. probes: allows 169.254.7.127/32 only. Readiness recovers, and app traffic is still blocked. 5. allow: gateway → productpage (15008, plus 9080 for the gateway only), waypoint → details/reviews/productpage (15008), role=echo-client → echo (15008). - Positive: ingress, mesh via waypoint, and client-allowed all work. - Negative: client-denied gets TIMEOUT, and the pod-IP bypass to productpage is dropped. 6. impostor: a pod with the allowed label (role=echo-client) but sa/client-denied. The NetworkPolicy lets it through, and the L4 AuthorizationPolicy returns CONN_RESET. 7. remote: - cluster-2 client → echo is blocked, because the east-west source isn't allowed yet - allow the local east-west gateway pods on 15008 - cluster-2 client-allowed → 200, while cluster-2 client-denied passes the network rule but is rejected on identity 8. cluster2: the same pattern for reviews/ratings. Cluster 1 → reviews-v2/v3 still works, and the pod-IP bypass to ratings is dropped. 9. reset: deletes everything labeled lab.appnet/exercise=10 plus the impostor and its policy.

Expected output (ILLUSTRATIVE).

     result: class=UPSTREAM_CONNECT_TIMEOUT http=503 curl_exit=0
     | upstream connect error or disconnect/reset before headers. reset reason: connection timeout
[ OK ] Confirmed at runtime: default-deny also blocked the SNATed kubelet probe
...
[ OK ] EXPECTED CONN_RESET: right label (NetworkPolicy passes) + WRONG identity: ztunnel rejects

Traffic decision order for a request to echo:

flowchart LR
  C["client pod"] --> N{"Cilium NetworkPolicy<br/>(labels, ports; per cluster)"}
  N -- "drop" --> T["TIMEOUT"]
  N -- "allow :15008" --> Z{"ztunnel L4 AuthorizationPolicy<br/>(SPIFFE identity)"}
  Z -- "deny" --> R["CONN_RESET"]
  Z -- "allow" --> W{"waypoint L7 policy?<br/>(only if attached)"}
  W -- "deny" --> D["403 RBAC"]
  W -- "allow / none" --> A["application"]

Common errors. See troubleshooting.md: Module 10: Layered Cilium NetworkPolicy plus Application Network for symptom → diagnosis → fix with exact commands.

Rollback. reset (mandatory before Module 11).

What you demonstrated. Defense in depth without over-broad rules, the ambient-specific NetworkPolicy requirements, and why labels are not identities.

Module 11: Advanced traffic management

Objective. Run the documented examples: JWT authentication and claim-based routing, cross-cluster weighted routing, and fault injection. Interpret each result precisely.

A. JWT (scripts/ex11-traffic-management.sh jwt-*). - The gateway uses an internal LB. Tests call applink-ingressgateway-istio.applink-ingress.svc.cluster.local from cluster 1, so tokens never cross a public or plaintext Internet path. - Tokens: kubectl create token <sa> --audience <OIDC issuer> --duration 10m, piped over stdin straight into the pod. Never stored, echoed or logged.

Case jwt-test (doc resources) jwt-require (+ require-token policy) What it proves
No token NOT_FOUND 404 RBAC_DENIED 403 404 = route miss (auth not required); 403 = authorization requires a principal
Malformed token JWT_REJECTED 401 JWT_REJECTED 401 Validation runs
Valid token, sub=...:curl-other NOT_FOUND 404 NOT_FOUND 404 Authenticated, but the claim route doesn't match
Valid token, sub=...:curl OK_2XX OK_2XX Validation + claim-based routing

B. Traffic shifting (shift, shift-reset). 20% reviews-local / 80% reviews-remote, 40 requests counted by the CLUSTER_NAME of the serving pod. The expected count is about 8 vs 32, but a spread of roughly ±5 is normal at n=40. HBONE connection reuse between the waypoint and the east-west gateway can pin streams (the doc says the same), so treat ratios as approximate.

C. Fault injection (fault-setup → fault-delay-7s → fault-delay-1s → fault-abort → fault-reset). - Scoped to the sample ratings service in cluster 2, and to the end-user: jason header for the delays. - Measured directly (curl -H 'end-user: jason' ratings from cluster 2, timing printed) and end to end (a login as jason inside the cluster, then productpage). - The doc's 1s sample is corrected (see support-and-sources §5). - fault-reset deletes every VirtualService and restores reviews-remote. Always run it.

Expected output (ILLUSTRATIVE).

     with end-user: jason -> 200 7.01s
     without header      -> 200 0.01s
     | HTTP 200 in 6.1s
     |       1 Error fetching product reviews
...
[ OK ] EXPECTED FAULT_ABORT: ratings returns an injected 503 ('fault filter abort'), not a real outage

Common errors. See troubleshooting.md: Module 11: Advanced traffic management for symptom → diagnosis → fix with exact commands.

Rollback. jwt-reset, shift-reset, fault-reset.

What you demonstrated. The difference between authentication, authorization and routing, plus statistically honest weighted routing and scoped, reversible fault injection.

Module 12: Observability and lifecycle

Objective. Use supported metrics and logs to correlate traffic, denials, latency and faults. Understand Application Network versus AKS lifecycle.

Documented limits: no tracing and no control-plane metrics (control-plane logs are supported).

Steps (scripts/ex12-observability.sh <step>). - Free: - traffic - waypoint-stats: port-forward 15020, istio_requests_total by service/code (doc method) - access-logs: Telemetry API, read from the waypoint container logs - Gate G7_MONITORING (billable): - enable-metrics: a new Azure Monitor workspace in RG_APPNET, az aks update --enable-azure-monitor-metrics, and the doc ConfigMap (any existing one is backed up) - enable-logs: Log Analytics in RG_APPNET, az aks enable-addons --addons monitoring, and a diagnostic setting (category istiod) on each member - monitoring-disable - queries: PromQL and KQL, for example p95 latency (delay injection), 403/503 counts (L7 deny / abort), ztunnel connections by connection_security_policy, MeshControlPlane errors. - Lifecycle: - lifecycle (read-only): list-versions, member upgrade profiles, az aks get-upgrades, maintenance configs - Gate G8: maintenance-window (aksManagedNodeOSUpgradeSchedule) and upgrade-member (SelfManaged N→N+1, one member, disruptive)

Lifecycle rules (documented). - Application Network manages only mesh components. It never upgrades AKS, Kubernetes or node images. - SelfManaged: you choose the minor (default N-1). Upgrades go N→N+1 only, with one rollback to N-1. The maintenance window gates patches only. - FullyManaged: the Rapid/Stable channel chooses versions. The window gates minor and patch upgrades. A channel change applies immediately. - The upgrade mode can't be changed without removing and rejoining the member. Keep AKS within the compatibility table. - ztunnel/istio-cni upgrades are per node and may drop long-lived connections. Windows are best effort.

Common errors. See troubleshooting.md: Module 12: Observability and lifecycle for symptom → diagnosis → fix with exact commands.

What you demonstrated. Evidence-based troubleshooting with supported telemetry, and safe version management across two independently upgraded products.

Module 13: Safe failure exercise

Objective. Remove only sample backends, observe, and restore. Make no claim of automatic failover.

Steps (scripts/ex13-safe-failure.sh <step>). 1. precheck: no VirtualServices or policies. Baseline counts. 2. remove-remote: scale reviews-v2/v3 (cluster 2) to 0. - reviews-remote returns 5xx (expected NO_HEALTHY_UPSTREAM). - Global reviews is served by cluster 1 only. - productpage still renders. 3. restore-remote 4. remove-local / restore-local: the mirror image. 5. restore: safe at any time.

Interpreting results. A global service load-balances over its current endpoint set. When backends disappear, the set shrinks. That isn't a configured failover policy: no outlier detection or locality failover is configured, and short error bursts are possible while endpoints update.

Common errors. See troubleshooting.md: Module 13: Safe failure exercise for symptom → diagnosis → fix with exact commands.

What you demonstrated. Blast-radius-limited failure testing and precise language about resilience.

Module 14: Cleanup

Objective. Undo mesh enrollment, remove members and resources in the documented order, delete only lab-owned resources, and find anything billable that's left.

Steps. 1. Run the reset steps of any exercise still applied (or go straight to step 2). 2. scripts/cleanup-1-kubernetes.sh (gate G9_CLEANUP_K8S): - removes istio.io/use-waypoint, istio.io/use-waypoint-namespace, istio.io/dataplane-mode and istio.io/global from lab namespaces, services and pods (doc prerequisite for member removal) - deletes the lab mesh/Gateway objects - deletes only namespaces labeled app.kubernetes.io/part-of=appnet-lab - restores the Azure Monitor ConfigMap backup 3. scripts/cleanup-2-azure.sh (gate G9_DELETE_AZURE): - previews exact resource IDs in the three lab RGs and the AKS node RGs - refuses RGs without the lab tag - refuses while namespaces are still enrolled - removes both members, deletes the Application Network, then deletes the three resource groups (bounded waits) 4. scripts/cleanup-3-leftovers.sh (read-only) checks: - resource groups (including MC_*) - tagged resources - name matches - disks, public IPs, LBs, DCRs/DCEs, Prometheus rule groups, workspaces, Key Vaults - orphaned role assignments - shared default monitoring RGs (reported, never deleted)

Add --local to delete .kube/, .secrets/, .tools/ and out/ after confirmation. 5. Not undone on purpose: the Microsoft.AppLink/PublicPreview registration (other deployments may use it) and the appnet-preview CLI extension (az extension remove --name appnet-preview if you want).

What you demonstrated. Reversible, auditable teardown that can't delete shared resources.