A Pitfall Log of Deploying Milvus Standalone on a Single-Node Kubernetes Cluster

目录

I was tinkering with a RAG project at home and needed a vector database. So I deployed Milvus onto my single-node K8s cluster at home.

Environment Info

My lab environment is a physical machine with Rocky Linux 10.1 installed, running a single-node Kubernetes cluster:

• OS: Rocky Linux 10.1
• Container runtime: cri-o
• CNI: Flannel
• Helm: v3.x

Pre-Checks: Storage and Network

Before deploying Milvus, two things must be confirmed first: storage and network.

Milvus depends on etcd and minio, both of which need PVCs. First check whether the cluster has a default StorageClass:

kubectl get sc

If the output is empty, there’s no storage class configured. For a single-machine test environment I used hostPath to set up a few local PVs, which is enough for lab scenarios:

mkdir -p /home/wenjun/local-storage/pv-etcd
mkdir -p /home/wenjun/local-storage/pv-milvus
mkdir -p /home/wenjun/local-storage/pv-minio

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: hostpath-pv-etcd
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  hostPath:
    path: /home/wenjun/local-storage/pv-etcd
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: hostpath-pv-milvus
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  hostPath:
    path: /home/wenjun/local-storage/pv-milvus
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: hostpath-pv-minio
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard
  hostPath:
    path: /home/wenjun/local-storage/pv-minio
EOF

Note: the hostPath capacity is only a declaration; the actual usage depends on how much data is written, but you must make sure the host machine’s disk space is sufficient.

Adding the Helm Repo and Installing

Milvus officially provides a Helm Chart; first add the repo:

helm repo add zilliztech https://zilliztech.github.io/milvus-helm/
helm repo update

I chose the Standalone mode, with all components running in a single Pod and Woodpecker as the message queue. For a single-machine test, this combination saves resources while being sufficient:

helm install milvus zilliztech/milvus \
  --set image.all.tag=v2.6.14 \
  --set cluster.enabled=false \
  --set pulsarv3.enabled=false \
  --set standalone.messageQueue=woodpecker \
  --set woodpecker.enabled=true \
  --set streaming.enabled=true \
  --set etcd.replicaCount=1 \
  --set minio.mode=standalone \
  --set minio.replicas=1

After installing, check the Pod status:

kubectl get pods -o wide
kubectl get pvc

Normally you should see all three Pods in Running state:

Pod	Expected status
milvus-etcd-0	1/1 Running
milvus-minio-xxx	1/1 Running
milvus-standalone-xxx	1/1 Running

But my situation wasn’t so smooth—the following three pitfalls popped up one after another.

Pitfall One: Image Pull Failure (docker.io Unreachable)

Symptom: The Pod status is stuck in ImagePullBackOff, and in the Events you can see records of connection to docker.io timing out.

Cause: My network environment can’t reach the official docker.io registry.

Solution: Configure a registry mirror for cri-o. I used DaoCloud’s image accelerator:

sudo tee /etc/containers/registries.conf.d/mirror.conf << 'EOF'
[[registry]]
prefix = "docker.io"
location = "docker.io"

[[registry.mirror]]
location = "docker.m.daocloud.io"
EOF

sudo systemctl restart crio

After configuring and restarting cri-o, delete the Pods that failed to pull and let the kubelet re-pull the images.

Pitfall Two: Pod-to-Pod Network Unreachable (firewalld Blocking CNI Traffic)

This was the most troublesome pitfall.

Symptom: etcd and minio both start normally, but milvus-standalone stays in CrashLoopBackOff. The logs repeatedly show this kind of error:

dial tcp 10.244.0.xxx:2379: connect: no route to host

Inside the Pod, not only can it not reach etcd, but DNS resolution also fails and the ClusterIP can’t be pinged either.

Cause: firewalld‘s nftables rules were blocking traffic on Flannel’s cni0 bridge. Combined with the kernel parameter bridge-nf-call-iptables=1, bridge traffic enters nftables and is then dropped, cutting off all Pod-to-Pod communication.

Solution: Run the following on the host machine with root privileges:

# 1. Stop and disable firewalld
sudo systemctl stop firewalld
sudo systemctl disable firewalld

# 2. Flush the leftover nftables rules
sudo nft flush ruleset

# 3. Restart kube-proxy so K8s rebuilds the Service DNAT rules
kubectl delete pod -n kube-system $(kubectl get pod -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[0].metadata.name}')

After restarting kube-proxy, Pod-to-Pod communication, DNS resolution, and ClusterIP access all returned to normal. milvus-standalone could finally reach etcd too.

It’s advised to run systemctl disable firewalld in advance, to avoid firewalld starting again after a node reboot and interfering with the network.

Pitfall Three: etcd Permission Error

Symptom: The etcd Pod won’t start, and the logs report:

cannot access data directory: mkdir /bitnami/etcd/data: permission denied

Cause: bitnami’s etcd image runs as a non-root user by default, but the directory mounted by hostPath doesn’t have enough permission.

Solution: Crude but effective:

chmod -R 777 /home/wenjun/local-storage/pv-etcd

Then delete the etcd Pod and let it be recreated.

Exposing the Service

The Service deployed by default is of type ClusterIP, accessible only within the cluster. I chose the NodePort approach, which suits single-machine testing:

kubectl patch svc milvus -p '{"spec":{"type":"NodePort","ports":[{"name":"milvus","port":19530,"targetPort":19530,"nodePort":30001},{"name":"metrics","port":9091,"targetPort":9091,"nodePort":30002}]}}'

The access addresses after exposure:

Purpose	Address
Milvus gRPC/REST API	http://<node IP>:30001
Milvus WebUI	http://<node IP>:30002

Connection Test

After deployment, verify with the Python client:

from pymilvus import MilvusClient

client = MilvusClient(
    uri="http://192.168.1.58:30001",   # replace with your node IP
    token="root:Milvus"
)

print(client.get_server_version())  # v2.6.14

# Create a test collection
client.create_collection("test_collection", dimension=128)
print("Collection created")

Or use the REST API to view the list of databases and collections:

# View the database list
curl -u root:Milvus http://192.168.1.58:30001/v2/vectordb/databases/list

# View the collection list
curl -u root:Milvus http://192.168.1.58:30001/v2/vectordb/collections/list

Note: directly visiting http://<ip>:19530/ in a browser returns 404 page not found, because this port serves the gRPC/REST API, not a web root page.

Uninstall and Cleanup

If you no longer need it, you can uninstall with one command:

helm uninstall milvus
kubectl delete pvc data-milvus-etcd-0 milvus milvus-minio

To thoroughly clean up the PVs and local data:

kubectl delete pv hostpath-pv-etcd hostpath-pv-milvus hostpath-pv-minio
rm -rf /home/wenjun/local-storage

Deployment Parameter Notes

Finally, a summary of the key Helm parameters used this time:

Helm parameter	Value	Description
cluster.enabled	false	Standalone mode, all components running in a single Pod
image.all.tag	v2.6.14	Milvus image version
pulsarv3.enabled	false	Disable Pulsar
woodpecker.enabled	true	Enable Woodpecker as the message queue
streaming.enabled	true	Enable the Streaming node
etcd.replicaCount	1	Single etcd replica
minio.mode	standalone	MinIO standalone mode
minio.replicas	1	Single MinIO replica

Summary

This Milvus deployment experience once again confirmed a truth: with a local test environment, never underestimate the two basic problems of network and storage.

Of the three pitfalls, firewalld blocking CNI traffic was the most hidden and the most time-consuming. On the surface it looked like Milvus couldn’t connect to etcd, but in fact the Pod-to-Pod network of the entire cluster was broken. To troubleshoot this kind of problem, the most direct method is to manually curl or telnet another service’s ClusterIP from inside the Pod, to quickly locate whether it’s a DNS problem, a routing problem, or a firewall problem.

If you’re also deploying Milvus on a single-node K8s, I hope this article helps you step into a few fewer pitfalls.

---

References

• Milvus official docs - Helm installation
• Milvus Helm Chart repo

中英对照

文章作者：阿文

文章链接： https://www.awen.me/post/c708a4e4.html

版权声明：本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 阿文的博客！