Everything You Never Needed to Know About Volumes, File Systems and Storage

When you build a computer, it needs storage. When you build a computer on the cloud, you try to get by with the smallest disk drives (or volumes) you can to fit the software and data that you have, so that you can avoid paying for disk space you don't need.

Everything You Never Needed to Know About Volumes, File Systems and Storage

When you build a computer, it needs storage. When you build a computer on the cloud, you try to get by with the smallest disk drives (or volumes) you can to fit the software and data that you have, so that you can avoid paying for disk space you don't need. However, the positive side of this optimization game is that you can change the size of your volume in minutes when you run out of space. No tiny screwdrivers required! But you do need to know some things.

Here, we cover some basics about volumes, partitions, file systems and mount points that are important to managing storage in RONIN, but that you probably never needed to know unless you also moonlighted as your lab's system administrator. We focus here on Linux, but the concepts also apply to Windows.

Disk Drives

You are probably familiar with a disk drive, called a volume in RONIN. Data that you store on a volume stays there, like on a physical hard drive. You can use a volume to hold a database, your operating system (root) data, or file data, because you can access individual blocks of data on a volume. This is called "block storage". You will typically copy data from object storage to block storage so that you can work with it more flexibly on your machine.

Because these volumes exist in the cloud, you can attach them to machines and detach them to machines virtually, as opposed to opening up the computer case to swap hardware. However, this process is a bit more complicated than plugging in flash drives. Read on if you dare!

Partitions and File Systems

Volumes can be split up into one or more logical partitions. The simplest configuration is to just have one partition per volume. However, each partition forms the basis of a file system, which is a directory structure on Linux. The advantage to creating multiple partitions per volume is that if one file system/partition fills up, it will not affect the others.  For example, if you have a file system for keeping the operating system and all of its supporting log files, putting that file system on a separate partition would prevent it from filling up the space which a researcher needs for data. In practice though, because we can choose volumes to be the size that we need, in RONIN we will generally create one partition per volume.

Therefore, you will have volumes that hold snapshots of everything related to the computer (the operating system, your home directory, and all the temporary and log files) and volumes that you set aside for applications and data files that you wish to move around between computers.

Mount Points

File systems are attached to the Linux directory tree at special directories called mount points. Figure 1 shows a machine with two volumes. Each volume has a single file system. One volume is mounted on /, and all the directories and files beneath it are located on that volume —  except that the second volume is mounted on /mnt/volume1. Once a user changes directory to /mnt/volume1, they are now writing to the second volume. By default, when you create a new storage volume when creating a machine, it will be mounted in /mnt. Storage for application files and data on a cluster will be mounted on /shared and /apps.

Figure 1. Mounting drives

You can permanently configure where a drive attached to a machine should be mounted by editing the file /etc/fstab.

Increasing the size of a volume

Inevitably, you create a volume, begin doing your work, and realize that you have run out of space. This may be on your root volume, or it may be on one of your other attached and mounted volumes. Some useful commands for learning about your drives, how full they are, the type of file system and where they are mounted are as follows:

#Find your disks (volumes) and their partitions
#see where (if) they are mounted, and their size
lsblk

#See how full they are (human readable)
df -H

#See what type they are (e.g. ext4 or xfs- default for RedHat)
df -T

On RONIN you can make a snapshot of your drive, restore it as a larger size volume, and then attach it to your machine. However, after you do that you may need to carefully let the operating system know what you have done (keep your snapshot until you are sure everything worked as expected). We don't do this for you because we know you would hate us if we accidentally deleted any of your data.

Suppose you make a snapshot of your 10GB drive attached on /mnt/volume1, restore it to a 25GB larger volume, and reattach that new volume to your machine. The command lsblk (Figure 2) shows that it is indeed 25GB, but it has a 10GB partition mounted on /mnt/volume1. Your data are all there, but you cannot yet use the space on the larger drive.

Figure 2. lsblk output after resizing a data volume from 10GB to 25GB.

We know from the command df -T above that this is an ext4 file system, and the commands we use below assume this to illustrate the basic concepts. For the black belt rundown, see the AWS documentation.

First we need to resize the partition to be the size of the volume. We can do this with the command growpart, which must be run as superuser. The first argument is the drive (volume) name, which is type disk in Figure 2. The second argument is the partition number, which we know from the p1 at the end of the partition name beneath the disk name is 1.

 sudo growpart /dev/nvme0n1 1

Now lsblk looks more accurate, but if we do a df, we still see the 10GB file system haunting us! (Figure 3)

Figure 3. After changing the partition size, we still need to make the file system fit it. 

The problem is that we need to extend the file system to fill up the resized partition. We can do this with the command resize2fs:

sudo resize2fs /dev/nvme0n1p1

Now you can use the entire, larger volume. Verify with the command df -H.

To Summarize...

A volume (a virtual disk drive) normally has a partition on it that corresponds to a file system. You mount a file system to access it by extending the directory tree to the files that are located there. To resize a volume you need to tell the operating system this is what you want to do, which requires some Linux system administration skills but is not impossible. And with these skills, you can go out and make a volume big enough to store just about anything!