Troubleshooting¶

Format: Symptom → Diagnosis (commands) → Fix. All commands assume the copy-paste setup from step-by-step-lab.md:

export LAB_ROOT="$PWD"; set -a; source config/lab.env; set +a
export KUBECONFIG="$LAB_KUBECONFIG"; export PATH="$LAB_ROOT/.tools/.istioctl/bin:$PATH"

Reading probe results¶

The classifier in scripts/lib/common.sh is a heuristic. Confirm each class with the evidence in the right-hand column before you conclude anything.

Class	Typical cause	Confirm with
`TIMEOUT`	NetworkPolicy drop (Cilium)	`kubectl --context C -n NS get networkpolicy`; the endpoints exist; curl text says "Connection timed out" (so DNS resolved)
`UPSTREAM_CONNECT_TIMEOUT`	Proxy → pod dropped by NetworkPolicy	Same as above, on the destination pod's namespace
`CONN_RESET`	ztunnel rejected: STRICT mTLS or L4 AuthorizationPolicy	`kubectl -n NS get peerauthentication,authorizationpolicy`; ztunnel log on the destination node
`UPSTREAM_RESET`	Waypoint/gateway → pod rejected by ztunnel	ztunnel log on the destination node shows the proxy's identity
`RBAC_DENIED`	L7 AuthorizationPolicy in a waypoint or gateway	`kubectl -n NS get authorizationpolicy -o yaml` (look for `targetRefs`)
`JWT_REJECTED`	Token failed validation (issuer, audience, signature, expiry, format)	`RequestAuthentication` issuer/audiences vs token claims
`NOT_FOUND`	Route miss (no matching HTTPRoute/VirtualService) or app 404	`kubectl get httproute,virtualservice -A`; `server: istio-envoy` + empty body usually means a route miss
`DNS_FAILURE` / `DNS_TIMEOUT`	Name doesn't exist / DNS egress blocked	`kubectl -n kube-system get svc kube-dns`; egress NetworkPolicy
`NO_HEALTHY_UPSTREAM`	No endpoints (local or remote)	`kubectl get endpointslices -l kubernetes.io/service-name=SVC`; `istioctl zc service` ENDPOINTS column
`FAULT_ABORT`	Injected abort	`kubectl -n NS get virtualservice -o yaml` (look for `fault:`)
`KUBECTL_ERROR`	kubectl exec failed (pod missing, RBAC, wrong context)	`kubectl --context C -n NS get deploy`; `kubectl auth can-i create pods/exec -n NS`

General diagnostics¶

# Which cluster am I talking to?
kubectl --context "$CTX1" config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo
# Recent events in a namespace
kubectl --context "$CTX1" -n "$APP_NS" get events --sort-by=.lastTimestamp | tail -20
# Mesh view
istioctl --context "$CTX1" -i applink-system ztunnel-config workload -n applink-system --workload-namespace "$APP_NS"
istioctl --context "$CTX1" -i applink-system ztunnel-config service -n applink-system | awk -v ns="$APP_NS" 'NR==1||$1==ns'
istioctl --context "$CTX1" -i applink-system ztunnel-config policy -n applink-system
# ztunnel on the node of a pod
NODE=$(kubectl --context "$CTX1" -n "$APP_NS" get pod <pod> -o jsonpath='{.spec.nodeName}')
kubectl --context "$CTX1" -n applink-system get pods --field-selector spec.nodeName="$NODE" -o name | grep ztunnel
kubectl --context "$CTX1" -n applink-system logs <ztunnel-pod> --since=5m | grep <pod>
# Whole-lab snapshot (read-only)
scripts/verify-lab-state.sh

Module 1: Planning / preflight¶

Symptom	Diagnosis	Fix
`SubscriptionNotRegistered` from `az appnet`	`az feature show --namespace Microsoft.AppLink --name PublicPreview`	Run `scripts/02-register-preview.sh` (gate G1). Registration can take several minutes.
`az: 'appnet' is misspelled or not recognized`	`az extension show --name appnet-preview`	`az extension add --name appnet-preview`
SKU `NotAvailableForSubscription`	`az vm list-skus -l $LOCATION --size Standard_D4 -o table`	Pick an unrestricted size and set `NODE_VM_SIZE`
Region fails one of the two lists	Contradiction C2 in support-and-sources	Choose a region that's in both lists
`aks-preview` warning	`az extension list -o table`	Not required. If `az aks` behaves unexpectedly, run the lab with a clean extension dir: `export AZURE_EXTENSION_DIR=$HOME/.azure/cliextensions-appnetlab` and `az extension add --name appnet-preview`

Module 2: Networks and clusters¶

Symptom	Diagnosis	Fix
`az aks create` fails with subnet `LinkedAuthorizationFailed`	`az role assignment list --assignee <uami-principal-id> --all -o table`	Wait 2–5 min for role propagation, then re-run `05` (existing clusters are skipped)
Policy error ("subnets must have an NSG", "public IPs denied")	The `az aks create` error message; `az policy assignment list`	Ask the subscription owner for an exemption on the lab RGs, or add an NSG that allows VNet traffic + Azure LB + your IP on 80/443 (AKS doesn't manage BYO subnet NSGs)
kubectl prompts for login repeatedly	`kubelogin --version`; kubeconfig `users[].user.exec`	Re-run `06-get-credentials.sh` (it runs `kubelogin convert-kubeconfig -l azurecli`)
`Forbidden` from kubectl	`az role assignment list --assignee <you> --scope <aks-id> -o table`	Wait for Azure RBAC propagation (up to ~5 min); `05` assigns the roles
`networkPolicy` isn't `cilium` in 07	`az aks show ... --query networkProfile`	Recreate the cluster. Don't patch the network mode in a lab
Cilium `RISK` lines in 07	`kubectl -n kube-system get cm cilium-config -o yaml`	You can't change them on AKS. Record them, decide at gate G4, and expect waypoint or probe problems

Module 3: NetworkPolicy baseline¶

Symptom	Diagnosis	Fix
`baseline` fails	`kubectl --context $CTX1 -n netpol-lab get pods,endpointslices`; `get networkpolicy`	Leftover policy: `ex03 reset`
`deny` returns `CONNECT_REFUSED`, not `TIMEOUT`	Backend not listening: `kubectl -n netpol-lab logs deploy/backend`	Wait for readiness, then re-run
`egress-allow` still gives a DNS failure	`kubectl --context $CTX1 -n kube-system get pods -l k8s-app=kube-dns --show-labels`	If the label differs, or LocalDNS (169.254.10.0/24) is enabled, adapt 60-clients-allow-dns-and-backend.yaml per the Cilium LocalDNS FAQ
Want packet-level evidence	`kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- cilium-dbg monitor --type drop`	Optional: the binary/container name may differ on AKS-managed Cilium

Module 4: Application Network onboarding¶

Symptom	Diagnosis	Fix
Join fails on prerequisites	`az aks show --query "{oidc:oidcIssuerProfile.enabled, aad:aadProfile.managed, mesh:serviceMeshProfile.mode}"`; Gateway API CRDs	Enable the missing prerequisite. The Istio add-on must be off
Member stuck or `Failed`	`az appnet member show ... -o jsonc`	Check `properties` for error details. Remove and rejoin (the mode is fixed at join)
`ztunnel`/`istio-cni-node` not Ready	`kubectl -n applink-system describe ds ztunnel`; pod logs	Node pressure (size), or CNI chaining. Compare with the Cilium evidence (`cni-exclusive`)
`underlay` TCP test fails	`az network vnet peering list ... -o table`; effective routes/NSG on a node NIC; the east-west Service IP	Peering must be Connected/FullyInSync both ways. No NSG/UDR may block VirtualNetwork → east-west ILB on the listed port
More than one LoadBalancer Service in `applink-system`	`kubectl -n applink-system get svc -o wide`	Identify the east-west Service by its ports (15008) and adjust `ew_endpoint`/`ew_selector`
ztunnel image minor ≠ istioctl	`kubectl -n applink-system get ds ztunnel -o jsonpath='{..image}'`	Set `ISTIO_VERSION`/`ISTIO_SAMPLES_REF` to match and re-run `08`

Module 5: Cross-cluster discovery¶

Symptom	Diagnosis	Fix
`remote-only` gets `NO_HEALTHY_UPSTREAM`	`istioctl zc service` on cluster 1: does `reviews-remote` list a second VIP and 2 endpoints?	Label `istio.io/global=true` on the Service in both clusters, and on the `waypoint` Service
`remote-only` gets `DNS_FAILURE`	`kubectl --context $CTX1 -n $APP_NS get svc reviews-remote`	The local (selector-less-match) Service must exist in cluster 1
Waypoint not Programmed	`kubectl -n $APP_NS describe gateway waypoint`	Wait, check events. `istioctl waypoint apply --wait` has a 90s default timeout
`clustername` is `null`	`kubectl -n $APP_NS get deploy reviews-v1 -o jsonpath='{..env}'`	Re-run the `deploy` step (it sets `CLUSTER_NAME`)
Responses only from one cluster in `global-mix`	Connection reuse / small sample	Re-run and increase the sample. Not a failure by itself

Module 6: mTLS¶

Symptom	Diagnosis	Fix
Outside client still succeeds under STRICT	`kubectl --context $CTX1 -n $APP_NS get peerauthentication -o yaml`	Apply in the correct namespace. Wait ~10s for propagation
Mesh client fails under STRICT	ztunnel logs on both nodes; `istioctl zc workload` PROTOCOL	The client namespace must be enrolled (HBONE). Gateways must be mesh-aware
Root fingerprints differ between clusters	`kubectl -n applink-system get cm istio-root-cert -o yaml` (public cert)	Both members must belong to the same Application Network. Record it and ask support
`TIMEOUT` instead of `CONN_RESET`	`kubectl -n $APP_NS get networkpolicy`	Remove leftover NetworkPolicies (`ex10 reset`)

Module 7: Ingress¶

Symptom	Diagnosis	Fix
Gateway has no address	`kubectl -n $APP_NS describe gateway bookinfo-gateway`; `get svc bookinfo-gateway-istio`	Check Service events for LB provisioning errors (quota, policy denying public IPs)
Local curl times out	`ex07 nsg-evidence`; your current egress IP	Update `ALLOWED_CLIENT_CIDR` and re-run `deploy`. Corporate proxies and VPNs change your source IP
NSG rule shows `Internet` as source	Annotation missing on the generated Service	Check `kubectl get svc bookinfo-gateway-istio -o yaml` annotations. `spec.infrastructure.annotations` must be propagated (UPSTREAM)
HTTPS listener not `Programmed`/`ResolvedRefs=False`	`kubectl -n $APP_NS get gateway bookinfo-gateway -o yaml` (listener conditions)	The Secret `bookinfo-tls` must be in the Gateway's namespace, with type `kubernetes.io/tls`
`TLS_ERROR` even with `--cacert`	`openssl x509 -in .secrets/tls/tls.crt -noout -ext subjectAltName`	The SAN must equal `TLS_HOST`. Use `--resolve TLS_HOST:443:IP`
`/lab/remote/...` fails	Gap G-9 (undocumented)	Record the result. Use the productpage path for documented cross-cluster routing

Module 8: L4 authorization¶

Symptom	Diagnosis	Fix
`client-denied` still allowed	`istioctl zc policy -n applink-system`; `kubectl -n l4-lab get authorizationpolicy`	The policy must use `selector` (ztunnel). Wait for propagation
Ingress broken after `doc-policy`	ztunnel log shows the gateway principal	The principal must be `cluster.local/ns/$APP_NS/sa/bookinfo-gateway-istio` (matching your namespace)
Cross-cluster client rejected	ztunnel log in cluster 1; `echo` global in both clusters	Record the observation. Check the east-west path (Module 4 `underlay`)

Module 9: L7 authorization¶

Symptom	Diagnosis	Fix
DELETE not denied on the mesh path	`istioctl zc service` WAYPOINT column for productpage	The namespace must have `istio.io/use-waypoint=waypoint`. The policy must use `targetRefs` Service
Everything via the waypoint gets 503	ztunnel log on the productpage node rejecting `sa/waypoint`	Apply 20-productpage-ztunnel-l4-with-waypoint.yaml
`close-gap` has no effect	`kubectl -n $APP_NS get authorizationpolicy bookinfo-gateway-deny-writes -o yaml`	`targetRefs` must name the Gateway `bookinfo-gateway` (group `gateway.networking.k8s.io`)

Module 10: Layered policies¶

Symptom	Diagnosis	Fix
`allow` step: mesh path still `UPSTREAM_CONNECT_TIMEOUT`	`cat out/ex10-selectors.env`; `kubectl get pods -l <selector>` in the right namespace	Re-run `discover`. Confirm the source pods carry those labels
echo never becomes Ready after `probes`	`kubectl -n l4-lab describe pod -l app=echo` (probe failures)	Istio doc alternative: a `CiliumClusterwideNetworkPolicy` allowing `fromCIDR 169.254.7.127/32`. Record it as runtime evidence
Cross-cluster fails even after `remote`	cluster-1 ztunnel logs for echo; Cilium drops	The east-west gateway may forward on another port, or via another pod. Observe the source and port, then add that scoped rule (never allow-all)
`impostor` allowed	`kubectl -n l4-lab get authorizationpolicy`	The L4 policy from Exercise 8 must be applied (the `impostor` step applies it)

Module 11: Traffic management¶

Symptom	Diagnosis	Fix
Valid token returns 401	Issuer/audience: `az aks show --query oidcIssuerProfile.issuerUrl`	Token audience must equal the issuer URL (the lab passes `--audience`). The issuer URL must end with `/`
`kubectl create token` forbidden	`kubectl auth can-i create serviceaccounts/token -n $APP_NS`	You need `serviceaccounts/token` create (included in AKS RBAC Cluster Admin)
No-token request returns 200	Claim route too broad	Check the VirtualService `match` in 20-jwt-authn-and-claim-routing.yaml
Shift shows 0 for one side	`istioctl zc service` endpoints for reviews-local/remote	Endpoints missing, or connection pinning. Re-run
Delay not observed	`kubectl --context $CTX2 -n $APP_NS get vs ratings -o yaml`	The header must be `end-user: jason`. Traffic must go through the cluster-2 waypoint
Faults left behind	`kubectl get vs -A` on both clusters	`ex11 fault-reset`

Module 12: Observability¶

Symptom	Diagnosis	Fix
No Prometheus series	`kubectl -n kube-system get pods \\| grep ama-metrics`; ConfigMap present	Wait 5–15 min. The doc ConfigMap enables ztunnel/istio-cni. Pod-annotation scraping needs `prometheus.io/*` annotations
No `MeshControlPlane` rows	`az monitor diagnostic-settings list --resource <member-id>`	The setting must target the member resource ID, category `istiod`
`waypoint-stats` port-forward fails	Is port 15020 in use locally?	Stop the other process, or change `port=` in the script

Module 13: Safe failure¶

Symptom	Diagnosis	Fix
`precheck` fails	`scripts/verify-lab-state.sh`	Run the earlier `reset` steps
Backends don't come back	`kubectl -n $APP_NS get deploy -l app=reviews`	`ex13 restore` (it scales each to 1)

Module 14: Cleanup¶

Symptom	Diagnosis	Fix
cleanup-2 refuses: namespaces still enrolled	`kubectl get ns -l istio.io/dataplane-mode`	Run `cleanup-1-kubernetes.sh`. Only if clusters are unreachable: `LAB_FORCE_MEMBER_REMOVE=yes`
RG refused: "not tagged"	`az group show -n RG --query tags`	The lab never deletes untagged groups. Delete manually only if you're certain
Leftovers after deletion	`scripts/cleanup-3-leftovers.sh`	Review each item. Shared `DefaultResourceGroup-` groups are not* lab-owned