Consolidating Milvus Across AZs

I migrated a standalone Milvus deployment from a three-zone Kubernetes setup to a single dedicated node in one target availability zone. The migration preserved the vector data, consolidated the persistent volumes, and restored the collections with full integrity while untangling AWS, Kubernetes, Helm, and etcd constraints.

This post documents the end-to-end path: decisions, traps, exact commands, and final checks. No fluff.

---

Context

• Cluster: a production-like kOps cluster; identifying names are replaced with examples
• Namespace: example-ml-namespace
• Milvus: standalone, deployed with the official Helm chart
• Initial pain: PVCs spanned three AZs, forcing nodes in all three to satisfy volume affinity. Standalone Milvus didn’t need multi-AZ.

Goal: One dedicated node group with taints in a target availability zone, with every PVC in that zone and zero data loss.

---

Strategy in One Page

1. Create a dedicated instance group with taints, pinned to the target zone.
2. Add nodeSelectors/tolerations for Milvus, etcd, MinIO via Helm values.
3. Snapshot the PVCs; restore them into the target zone (EBS volumes can’t cross AZs; snapshots can).
4. Repair etcd membership from snapshot using ETCD_FORCE_NEW_CLUSTER=true.
5. Bring up MinIO (object storage with vector data), then Milvus.
6. Validate collections and segments; clean up.

Design calls:

• Snapshots over cloning to cross AZ boundaries.
• Preserve MinIO (data), rebuild etcd (metadata) from snapshot with FORCE_NEW_CLUSTER.
• Single AZ for simplicity and cost (dev/test trade-off accepted).

---

What Went Wrong (and How I Fixed It)

1) AZ mismatch blocking scheduling

• Symptom: volume node affinity conflict.
• Cause: The node remained in a source zone while PVCs were bound to the target zone.
• Fix: Move the instance group to the target zone, apply the change, and replace the old nodes.

2) StatefulSet PVCs stuck in old AZs

• Reality: PVC zone affinity is immutable.
• Fix: VolumeSnapshot → delete PVC → recreate PVC from snapshot; let CSI bind in the target zone.

3) Missing IAM for snapshot restores

• Symptom: UnauthorizedOperation on ec2:CreateVolume from snapshot.

Fix: Add:

{ "Effect": "Allow", "Action": "ec2:CreateVolume", "Resource": "arn:aws:ec2:*:*:snapshot/*" }

Restart EBS CSI controller.

4) etcd membership deadlock after restore

• Symptom: CrashLoop, “No active endpoints in cluster”.
• Cause: Restored data contained old member IPs.
• Fix (disaster recovery):
  • Restore only etcd-0 PVC from snapshot.
  • Scale to 3; etcd-1/2 join fresh.

Start one replica with:

kubectl set env statefulset/milvus-release-etcd \
  ETCD_FORCE_NEW_CLUSTER=true ETCD_INITIAL_CLUSTER_STATE=new -n example-ml-namespace

kubectl scale statefulset milvus-release-etcd -n example-ml-namespace --replicas=1

5) “Missing collections” scare

• Reality: Milvus stores metadata in etcd and vectors in MinIO.
• Fix: Once etcd metadata was restored from snapshot, Milvus mapped names→IDs and loaded segments. Data intact.

6) PVC selector immutability

• Lesson: Don’t try to patch PVC zone/selector. Use snapshot→recreate. With WaitForFirstConsumer, node placement determines AZ.

---

Step-By-Step Execution

Phase 1 - Prep

Dedicated node group (target zone, tainted):

kops edit ig example-node-group --state s3://example-state-store
# Set one target-zone subnet and an example-workload=true:NoSchedule taint.

kops update cluster example.k8s.local --state s3://example-state-store --yes

Helm values with selectors/tolerations (Milvus/etcd/MinIO):

# /tmp/milvus-migration-values.yaml
standalone:
  {
    nodeSelector: { kops.k8s.io/instancegroup: example-node-group },
    tolerations: [{ key: example-workload, operator: Equal, value: 'true', effect: NoSchedule }],
  }
etcd:
  {
    nodeSelector: { kops.k8s.io/instancegroup: example-node-group },
    tolerations: [{ key: example-workload, operator: Equal, value: 'true', effect: NoSchedule }],
    replicaCount: 3,
  }
minio:
  {
    nodeSelector: { kops.k8s.io/instancegroup: example-node-group },
    tolerations: [{ key: example-workload, operator: Equal, value: 'true', effect: NoSchedule }],
    replicaCount: 4,
    persistence: { size: <size-for-your-data> },
  }

Create VolumeSnapshots:

kubectl apply -f VolumeSnapshotClass(ebs.csi.aws.com)

kubectl apply -f snapshots for etcd-0, etcd-2, minio-0, minio-1

kubectl wait volumesnapshot/<name> -n example-ml-namespace --for=jsonpath='{.status.readyToUse}'=true --timeout=300s

Phase 2 - IAM

• Add ec2:CreateVolume on arn:aws:ec2:*:*:snapshot/*; restart EBS CSI controller.

Phase 3 - Scale down & delete old PVCs

kubectl scale sts milvus-release-etcd -n example-ml-namespace --replicas=0
kubectl scale sts milvus-release-minio -n example-ml-namespace --replicas=0
kubectl delete deploy milvus-release-standalone -n example-ml-namespace
kubectl delete pvc <pvc-names-to-restore> -n example-ml-namespace

Phase 4 - Restore PVCs into the target zone

# Recreate PVCs from snapshots without a zone selector; CSI follows node placement.
kubectl apply -f restored-pvcs.yaml

Phase 5 - Lock the instance group to the target zone and replace nodes

kops edit ig example-node-group  # ensure only the target-zone subnet remains
kops update cluster example.k8s.local --state s3://example-state-store --yes
kubectl delete node <nodes-in-source-zones>

Phase 6 - Bring up MinIO

kubectl scale sts milvus-release-minio -n example-ml-namespace --replicas=<replica-count>
kubectl wait -n example-ml-namespace -l app.kubernetes.io/name=minio --for=condition=Ready pod --timeout=300s

Phase 7 - etcd recovery

kubectl delete pvc <stale-etcd-pvc> -n example-ml-namespace
kubectl set env sts/milvus-release-etcd ETCD_FORCE_NEW_CLUSTER=true ETCD_INITIAL_CLUSTER_STATE=new -n example-ml-namespace
kubectl scale sts milvus-release-etcd -n example-ml-namespace --replicas=1
kubectl wait pod/milvus-release-etcd-0 -n example-ml-namespace --for=condition=Ready --timeout=120s
kubectl delete pvc <second-stale-etcd-pvc> -n example-ml-namespace
kubectl scale sts milvus-release-etcd -n example-ml-namespace --replicas=<replica-count>

Phase 8 - Start Milvus

helm upgrade milvus-release zilliztech/milvus -n example-ml-namespace --reuse-values -f /tmp/milvus-migration-values.yaml
kubectl wait -n example-ml-namespace -l app.kubernetes.io/name=milvus --for=condition=Ready pod --timeout=300s

Phase 9 - Cleanup extra nodes

• Verify all pods on the single node; cordon/drain/delete any stragglers.

---

Validation Checklist

Infra:

kubectl get nodes -l kops.k8s.io/instancegroup=example-node-group -o custom-columns='NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,STATUS:.status.conditions[-1].type'
kubectl get pods -n example-ml-namespace -o wide
for pvc in $(kubectl get pvc -n example-ml-namespace -o jsonpath='{.items[*].metadata.name}'); do
  vol=$(kubectl get pvc $pvc -n example-ml-namespace -o jsonpath='{.spec.volumeName}')
  zone=$(kubectl get pv $vol -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}')
  echo "$pvc | $zone"
done
# Expect every zone to match your chosen target zone.

Milvus health & data:

kubectl get pods -n example-ml-namespace
kubectl logs -l app.kubernetes.io/name=milvus -n example-ml-namespace --tail=200 | grep -i "loaded segment metadata"
kubectl port-forward -n example-ml-namespace svc/milvus-release 19530:19530 &
python - <<'EOF'
from pymilvus import connections, utility
connections.connect(host="localhost", port="19530")
print(utility.list_collections())
EOF
pkill -f "port-forward.*milvus"

MinIO contents (sanity):

kubectl exec -it milvus-release-minio-0 -n example-ml-namespace -- ls /export/<bucket>/file/index_files/

---

Lessons You Can Reuse

Kubernetes

• Snapshot first. It’s the only sane way to cross AZs with EBS.
• With CSI WaitForFirstConsumer, node placement → AZ. Don’t fight PVC immutability.
• Use values files for Helm upgrades; avoid subchart auth traps.

AWS

• EBS volumes don’t cross AZs; snapshots do (within region).
• IAM for CSI is granular: creating a volume from a snapshot needs explicit rights.

Distributed Milvus

• In Milvus: MinIO = data, etcd = metadata. Protect MinIO PVCs; snapshot etcd.
• etcd DR: Start one restored member with ETCD_FORCE_NEW_CLUSTER=true, then scale.

---

Final State

• Single dedicated node in the target zone, tainted and isolated.
• All PVCs consolidated in that zone.
• Services: Milvus standalone with its etcd and MinIO dependencies.
• Data: The expected collections and segments passed integrity checks.

---

Optional Cleanup & Monitoring

Watch stability and resources:

kubectl top node
kubectl top pods -n example-ml-namespace
kubectl logs -l app=milvus-release -n example-ml-namespace --since=24h | grep -i error

Remove snapshots if policy allows:

kubectl delete volumesnapshot -n example-ml-namespace <snapshots-approved-for-deletion>

---

Appendix - Minimal IAM Addition for Snapshot Restore

{
	"Effect": "Allow",
	"Action": "ec2:CreateVolume",
	"Resource": "arn:aws:ec2:*:*:snapshot/*"
}

Add to your EBS CSI controller role, then restart the controller.

---

Outcome: single-AZ Milvus, clean scheduling, lower cost, no data loss, reproducible steps.