Currently, my LAN is 172.16.254.0/24 and my Ceph network is 10.10.10.0/24 (separated over a different NIC and VLAN)

How do I give my Kubernetes access to Ceph? I want to create a separate pool and use object storage.

I guess I have to start with adding a virtual NIC for each Kube node into the 10.10.10.0/24 network so ther can communicate with the Proxmox Ceph Mons - am I on the right track here or is there a better way to do this?

Turns out that is not the right way to do it. The right way is to update the config so that the OSDs are running on 10.10.10.0/24 network (dedicated 10gbps) and put the monitors/managers on your 172.16.254.0/24 (or whatever) network.

In the below scenario, I am setting up a "Public" network for my ceph in the 172.16.254.0/24 subnet (with the monitor IPs changing from 10.10.10.1-3 to 172.16.254.50-52).

Below is a quick-n-nasty way to do this. Note that this requires cluster downtime.

First, export the ceph monitor map to a file ("MONMAP_FILE" in this case).

At this point, you're supposed to be able to restart the monitors and then the OSDs one by one and everything should just work (TM) but I couldn't get this working. I rebooted all 3 of my hosts - everything now functioning as normal.

I'd appreciate if anyone can chime in on why it doesn't work without a reboot.

Quick Navigation

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick support. 13,000+ satisfied customers have Proxmox subscriptions. Get your own in 60 seconds.