How I Consolidated Milvus Across AZs

We had a standalone Milvus deployment whose PersistentVolumeClaims (PVCs) had been scheduled across three different AWS availability zones. Because AWS EBS volumes are strictly zonal, that single Milvus instance was forcing our cluster autoscaler to keep nodes alive across all three AZs. We were paying for extra compute and cross-AZ traffic on a workload that did not need multi-AZ high availability.

Here is the exact runbook I used to consolidate Milvus, its 4-node MinIO object store, and its 3-node etcd cluster onto a single tainted Kubernetes node in one availability zone while keeping every collection and vector intact.

---

Context

• Cluster: Production kOps cluster on AWS
• Namespace: example-ml-namespace
• Workload: Milvus standalone (deployed via the official Zilliz Helm chart, backed by StatefulSet etcd and MinIO)
• The pain: PVCs were scattered across three AZs (us-east-1a, 1b, 1c), pinning nodes in all three zones because of EBS volume node affinity.

Goal: Consolidate the entire stack onto a single tainted instance group in one target AZ, migrate every PVC into that AZ, and lose zero vectors along the way.

---

Strategy in One Page

1. Provision a dedicated, tainted kOps instance group pinned to a single target AZ.
2. Add nodeSelector and tolerations to the Helm values for Milvus, etcd, and MinIO.
3. Snapshot the existing EBS volumes via VolumeSnapshot (EBS volumes can’t cross AZs, but EBS snapshots live in S3 and can be restored into any AZ in the region).
4. Recreate the PVCs from those snapshots so the EBS CSI driver binds new volumes in the target AZ.
5. Bootstrap etcd from the restored etcd-0 snapshot using ETCD_FORCE_NEW_CLUSTER=true, then scale back to 3 replicas.
6. Bring up MinIO (which holds the actual vector segments and index files).
7. Bring up Milvus standalone.
8. Verify that every collection and segment loads cleanly, then drain and delete the old nodes.

Key architectural calls:

• VolumeSnapshots over application-level backup: Snapshotting EBS volumes at the storage layer let us move both MinIO segments and etcd metadata across AZ boundaries without exporting gigabytes of vectors over the network.
• Preserve MinIO data, force-rebuild etcd quorum: When etcd pods wake up on new IPs after a PVC restore, they deadlock trying to reach their old peer IPs. Restoring etcd-0 with ETCD_FORCE_NEW_CLUSTER=true cleanly resets peer membership while keeping all Milvus collection metadata.
• Single-AZ trade-off: Trading away automatic AZ failover on this internal workload saved two extra worker nodes and cross-AZ data transfer costs.

---

What Went Wrong (and How I Fixed It)

1) AZ mismatch blocks scheduling

• Symptom: Pods stuck in Pending with 0/N nodes are available: volume node affinity conflict.
• Cause: The instance group still had subnets spanning multiple zones, so the replacement node came up in a different AZ than the newly restored PVCs.
• Fix: Lock the kOps instance group spec to a single subnet in the target AZ, roll the instance group, and terminate the out-of-zone node.

2) StatefulSet PVCs stuck in old AZs

• Reality: Once a PersistentVolume is provisioned in us-east-1a, its nodeAffinity is immutable. You cannot kubectl patch a volume into us-east-1b.
• Fix: Take a VolumeSnapshot, delete the original PVC, and recreate a PVC with the exact same name referencing the snapshot as its dataSource. With WaitForFirstConsumer on the StorageClass, the CSI driver provisions the new EBS volume in whatever AZ the pod lands in.

3) Missing IAM permission for snapshot restores

• Symptom: PVC stuck in ProvisioningFailed with UnauthorizedOperation on ec2:CreateVolume from a snapshot ARN.
• Fix: The kOps default EBS CSI controller policy allowed ec2:CreateVolume on volumes, but not on arn:aws:ec2:*:*:snapshot/* resources. Adding this statement to the control-plane/CSI IAM role and restarting the ebs-csi-controller deployment unblocked provisioning immediately:

{ "Effect": "Allow", "Action": "ec2:CreateVolume", "Resource": "arn:aws:ec2:*:*:snapshot/*" }

4) etcd membership deadlock after restore

• Symptom: etcd pods crash-looping with rafthttp: failed to find member / No active endpoints in cluster.
• Cause: The restored etcd data directory still remembered the old pod IPs and peer URLs from before the migration, so quorum could never form.
• Fix:
  1. Restore only etcd-0’s PVC from its snapshot (delete the stale etcd-1 and etcd-2 PVCs so they start fresh).
  2. Force etcd-0 to form a new single-node cluster from its existing WAL/snapshot data:

kubectl set env statefulset/milvus-release-etcd \
  ETCD_FORCE_NEW_CLUSTER=true ETCD_INITIAL_CLUSTER_STATE=new -n example-ml-namespace

kubectl scale statefulset milvus-release-etcd -n example-ml-namespace --replicas=1

3. Once etcd-0 reports 1/1 Ready, remove ETCD_FORCE_NEW_CLUSTER and scale the StatefulSet back to 3 so etcd-1 and etcd-2 join as clean learners.

5) Empty collection list against a fresh etcd cluster

• Symptom: When we first tested starting Milvus against a fresh etcd cluster (even with all MinIO volumes intact), list_collections() returned [].
• Reality: Milvus stores vector segments and index files in MinIO, but the catalog mapping collection names to segment IDs lives entirely in etcd. Without the original etcd keyspace, the objects in MinIO are orphaned blobs.
• Fix: Once we restored etcd-0 from its VolumeSnapshot via the step above, Milvus reconciled the etcd catalog against MinIO and loaded every collection and segment on startup.

6) PVC selector immutability

• Lesson: Don’t waste time trying to edit PV node affinities or PVC storage selectors in place. Snapshot → delete PVC → recreate PVC with WaitForFirstConsumer is the only clean path.

---

Step-By-Step Execution

Phase 1 - Prep

Pin the dedicated node group to the target zone with a taint:

kops edit ig example-node-group --state s3://example-state-store
# Set one target-zone subnet and an example-workload=true:NoSchedule taint.

kops update cluster example.k8s.local --state s3://example-state-store --yes

Prepare Helm values with nodeSelector and tolerations for all three components:

# /tmp/milvus-migration-values.yaml
standalone:
  {
    nodeSelector: { kops.k8s.io/instancegroup: example-node-group },
    tolerations: [{ key: example-workload, operator: Equal, value: 'true', effect: NoSchedule }],
  }
etcd:
  {
    nodeSelector: { kops.k8s.io/instancegroup: example-node-group },
    tolerations: [{ key: example-workload, operator: Equal, value: 'true', effect: NoSchedule }],
    replicaCount: 3,
  }
minio:
  {
    nodeSelector: { kops.k8s.io/instancegroup: example-node-group },
    tolerations: [{ key: example-workload, operator: Equal, value: 'true', effect: NoSchedule }],
    replicaCount: 4,
    persistence: { size: <size-for-your-data> },
  }

Take VolumeSnapshots of etcd and MinIO PVCs:

kubectl apply -f VolumeSnapshotClass(ebs.csi.aws.com)

kubectl apply -f snapshots for etcd-0, etcd-2, minio-0, minio-1

kubectl wait volumesnapshot/<name> -n example-ml-namespace --for=jsonpath='{.status.readyToUse}'=true --timeout=300s

Phase 2 - IAM

1. Grant ec2:CreateVolume on arn:aws:ec2:*:*:snapshot/* to the EBS CSI controller role.
2. Roll the ebs-csi-controller pods in kube-system.

Phase 3 - Scale down & delete old PVCs

kubectl scale sts milvus-release-etcd -n example-ml-namespace --replicas=0
kubectl scale sts milvus-release-minio -n example-ml-namespace --replicas=0
kubectl delete deploy milvus-release-standalone -n example-ml-namespace
kubectl delete pvc <pvc-names-to-restore> -n example-ml-namespace

Phase 4 - Restore PVCs into the target zone

# Recreate PVCs from snapshots without a zone selector; CSI follows node placement.
kubectl apply -f restored-pvcs.yaml

Phase 5 - Lock the instance group to the target zone and replace nodes

kops edit ig example-node-group  # ensure only the target-zone subnet remains
kops update cluster example.k8s.local --state s3://example-state-store --yes
kubectl delete node <nodes-in-source-zones>

Phase 6 - Bring up MinIO

kubectl scale sts milvus-release-minio -n example-ml-namespace --replicas=<replica-count>
kubectl wait -n example-ml-namespace -l app.kubernetes.io/name=minio --for=condition=Ready pod --timeout=300s

Phase 7 - etcd recovery

kubectl delete pvc <stale-etcd-pvc> -n example-ml-namespace
kubectl set env sts/milvus-release-etcd ETCD_FORCE_NEW_CLUSTER=true ETCD_INITIAL_CLUSTER_STATE=new -n example-ml-namespace
kubectl scale sts milvus-release-etcd -n example-ml-namespace --replicas=1
kubectl wait pod/milvus-release-etcd-0 -n example-ml-namespace --for=condition=Ready --timeout=120s
kubectl delete pvc <second-stale-etcd-pvc> -n example-ml-namespace
kubectl scale sts milvus-release-etcd -n example-ml-namespace --replicas=<replica-count>

Phase 8 - Start Milvus

helm upgrade milvus-release zilliztech/milvus -n example-ml-namespace --reuse-values -f /tmp/milvus-migration-values.yaml
kubectl wait -n example-ml-namespace -l app.kubernetes.io/name=milvus --for=condition=Ready pod --timeout=300s

Phase 9 - Clean up extra nodes

1. Verify every pod in example-ml-namespace is scheduled onto the single target node.
2. Cordon, drain, and delete any remaining old nodes.

---

Validation Checklist

Infrastructure & PVC zones:

kubectl get nodes -l kops.k8s.io/instancegroup=example-node-group -o custom-columns='NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,STATUS:.status.conditions[-1].type'
kubectl get pods -n example-ml-namespace -o wide
for pvc in $(kubectl get pvc -n example-ml-namespace -o jsonpath='{.items[*].metadata.name}'); do
  vol=$(kubectl get pvc $pvc -n example-ml-namespace -o jsonpath='{.spec.volumeName}')
  zone=$(kubectl get pv $vol -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}')
  echo "$pvc | $zone"
done
# Expect every zone to match your chosen target zone.

Milvus health & collection verification:

kubectl get pods -n example-ml-namespace
kubectl logs -l app.kubernetes.io/name=milvus -n example-ml-namespace --tail=200 | grep -i "loaded segment metadata"
kubectl port-forward -n example-ml-namespace svc/milvus-release 19530:19530 &
python - <<'EOF'
from pymilvus import connections, utility
connections.connect(host="localhost", port="19530")
print(utility.list_collections())
EOF
pkill -f "port-forward.*milvus"

MinIO segment sanity check:

kubectl exec -it milvus-release-minio-0 -n example-ml-namespace -- ls /export/<bucket>/file/index_files/

---

Lessons You Can Reuse

Kubernetes

• Use VolumeSnapshot to cross availability zones. EBS volumes cannot move between AZs, so snapshot and restore is the fastest way to relocate stateful storage.
• With WaitForFirstConsumer enabled on the StorageClass, bind the pod to the target node via nodeSelector and the CSI driver will create the volume in that node’s AZ automatically.
• Pass explicit YAML files (-f values.yaml) on helm upgrade instead of --set flags when dealing with Bitnami/Zilliz subcharts (like etcd and MinIO), which otherwise regenerate passwords on upgrade.

AWS

• EBS volumes are strictly zonal; EBS snapshots are regional.
• Check your EBS CSI driver IAM policy before the maintenance window, since ec2:CreateVolume needs arn:aws:ec2:*:*:snapshot/* in its Resource list when restoring from snapshots.

Milvus

• MinIO stores the raw segment and index files; etcd stores the collection and segment catalog. Without the original etcd keyspace, MinIO buckets are just unindexed UUID prefixes.
• Whenever you restore a StatefulSet etcd cluster onto new pod IPs, bootstrap a 1-node cluster from etcd-0 with ETCD_FORCE_NEW_CLUSTER=true first, then scale out fresh replicas.

---

Optional Cleanup & Monitoring

Keep an eye on memory pressure and error logs for the first 24 hours:

kubectl top node
kubectl top pods -n example-ml-namespace
kubectl logs -l app=milvus-release -n example-ml-namespace --since=24h | grep -i error

Once you’re confident in the migration, clean up the temporary snapshots:

kubectl delete volumesnapshot -n example-ml-namespace <snapshots-approved-for-deletion>

---

Appendix - Minimal IAM Addition for Snapshot Restore

{
	"Effect": "Allow",
	"Action": "ec2:CreateVolume",
	"Resource": "arn:aws:ec2:*:*:snapshot/*"
}

Attach this statement to your EBS CSI controller IAM role and restart the controller deployment.