What are the storage options for on-premises and hybrid Kubernetes?
Storage for on-premises Kubernetes comes in four types: local disks attached to one node, replicated block storage that copies volumes across nodes (such as Rook-Ceph or Longhorn), an existing SAN or NAS connected through a CSI driver, and S3-compatible object storage that applications use through an API. Most clusters combine two or three of them. In hybrid setups, object storage and backups are what keep data portable between your data center and a cloud.
In the cloud, a storage class backed by the provider’s block volumes is usually ready when the cluster starts. On-premises, nothing provisions volumes until you install a storage system and its Container Storage Interface (CSI) driver.
Access modes decide what you can run
A PersistentVolume declares how it can be mounted. The Kubernetes access modes are:
ReadWriteOnce(RWO): read-write by a single node. Databases and most StatefulSets use it.ReadOnlyMany(ROX): read-only by many nodes.ReadWriteMany(RWX): read-write by many nodes. Shared uploads, some CI caches and VM live migration need it.ReadWriteOncePod(RWOP): read-write by a single Pod across the cluster.
Block storage gives RWO. RWX needs a shared file system such as NFS or CephFS.
The four types of on-premises storage
| Type | Examples | Access modes | Survives node loss | Good for |
|---|---|---|---|---|
| Local disks | Kubernetes local volumes, LVM-based provisioners, OpenEBS local engines |
RWO | No, data stays on one node | Databases that replicate themselves, caches, scratch space |
| Replicated block storage | Rook-Ceph, Longhorn, OpenEBS replicated engines | RWO; RWX through CephFS or Longhorn’s NFS-based shares | Yes, replicas on other nodes | General stateful workloads without an external array |
| External SAN or NAS | Storage vendors’ CSI drivers, the NFS CSI driver | RWO for block, RWX for NFS | Yes, if the array is redundant | Organizations with storage arrays and teams to run them |
| Object storage | Ceph Object Gateway through Rook, other S3-compatible stores | Not a volume; accessed over the S3 API | Yes, with replication or erasure coding | Backups, data lakes, ML datasets, artifacts |
Rook runs Ceph on Kubernetes and provides block, file and object storage from the same cluster. It is a CNCF graduated project since October 2020. Ceph is the most capable of the options and needs the most operating knowledge. Longhorn, a CNCF incubating project since November 2021, is a lighter replicated block store with a web UI and built-in snapshots and backups to S3 or NFS. OpenEBS, a CNCF sandbox project, offers several engines from local volumes to replicated storage.
Local disks are the fastest option and the easiest to set up, but a volume is tied to its node: if the node fails, the Pod cannot start elsewhere until the node returns. That is acceptable for systems such as PostgreSQL operators, Kafka or Elasticsearch, which keep their own replicas on other nodes.
Replication is not backup
Replicated storage protects against a failed disk or node. It does not protect against a deleted namespace, a bad migration or ransomware, because the damage is replicated too. Keep backups outside the storage system:
- Volume snapshots through the CSI snapshot API for fast, local rollback.
- Cluster backups with a tool such as Velero, which saves Kubernetes objects and volume data to object storage and restores them into the same or another cluster.
- Application-level backups, such as database dumps or write-ahead log archiving, for consistent restores of databases.
Performance
Replicated storage writes every block to several nodes before confirming it, so network bandwidth and latency between nodes set its write performance. Put replication traffic on a fast, dedicated network where possible, and use NVMe drives for latency-sensitive volumes. Benchmark with your real workload before choosing replica counts.
Storage in hybrid clusters
When one platform spans on-premises servers and public clouds, each environment has its own storage classes: local or Ceph volumes on-premises, the provider’s block volumes in each cloud. Block volumes do not move between environments, so stateful Pods stay where their data is.
To keep data portable:
- Use object storage as the exchange format. S3-compatible APIs exist on-premises and in every major cloud, so applications and pipelines run unchanged in both places.
- Replicate or back up across environments. Back up on-premises volumes to object storage in a cloud region, or the reverse, and test restores there.
- Tier hot and cold data. Keep hot data on local NVMe or replicated block storage near the compute, and move cold data to object storage with lifecycle rules.
- Account for transfer costs. Moving data out of a cloud costs money per gigabyte; see cloud egress fees.
Portable data is also the core of avoiding cloud vendor lock-in. For the rest of the on-premises setup beyond storage, see running Kubernetes on-premises.
How Cloudfleet fits
Cloudfleet Kubernetes Engine (CFKE) clusters can use local disks and replicated storage such as Longhorn or Rook-Ceph on self-managed nodes, and the CSI drivers for Amazon EBS, Google Cloud Persistent Disk and Hetzner Cloud Volumes on cloud nodes, all in one cluster. See the storage overview.