When running out of disk space
What is it
All data gathered on your ProjectServer sits in the Docker volumes on your servers /var filesystem. Each stack you deploy eats up a bit of this space and almost inevitably the time will come when you run short of disk space. As we live in the cloud era adding more space does no longer incur fiddling with physical drives, it is just a matter of accessing a web UI and increasing the volume size. But this just means that your server has more space available, the filesystem where the data reside does not grow dynamically. Luckily this is something you usually can accomplish even without having to reboot your system.
CAUTION: Nevertheless this is a heavyweight process! If you make a mistake when creating the new partition you definitely risk losing all your data! So better do a backup before (most cloud and VM platforms offer a snapshot feature which is perfect for this task) and think twice before you act.
To find out whether you have a disk space shortage issue df -h /. This displays the percentage of use and the still available space.
Prerequisites
a) Make sure that your system contains the "partprobe" and "resize2fs" utilities: Enter which partprobe resize2fs and check whether the paths to both utilities are displayed. They usually reside in /sbin/. If you miss one or both install it from the proper Debian package: apt install parted e2fsprogs.
b) Have a current backup of the volume you plan to resize. Usually it is best to make such a backup while the server is not running. The easiest way (if offered by your platform) is a snapshot.
c) As user root execute fdisk -l to determine your disk configuration and identify the device name of the volume you want to resize. The standard PS4 setup uses one disk (volume) /dev/sda containing just one partition named /dev/sda1. This is what we are going to cover here. fdisk -l should display the following (numbers and possibly pathes may differ in your case):
How to achieve it
You must be logged in as superuser (root) to be able to execute the following commands.
Increase the volume size
You may skip this step if you find from the fdisk -l output that the size of volume /dev/sda (41943040 in the above example) is considerably larger than the partition size /dev/sda1 (here: 41940992). This means you have still space available which is not yet consumed by your filesystem.
Otherwise increase the volume/storage/disk size of the volume you have provided for your server. How to accomplish this depends on the concrete cloud or VM platform. In many cases this does not require to shut down your Linux server but it may nevertheless be necessary to make a useful backup before increasing the volume.
msg.Cloud
- Open the msg.Cloud dashboard, locate your server and click on its name to get into the detail view.
- In the Storages section identify the storage (=volume) you want to increase (in most cases there is just one volume listed). Enter its detail view.
- The storage detail pane now allows you to create a snapshot (see the Take Snapshot button).
- To increase the volume size simply use the respective UI element (a slider or an entry field).
- Press Update to apply the change. The increased volume is instantly available.
VirtualBox
- Shutdown the VM.
- In the Virtual Box Manager GUI open "File->Manager for virtual media" (or press CTRL-D).
- Select the virtual disk you want to grow.
- At the bottom of the windows is a slider and an entry box where you can choose the new disk size.
- Then press button Save.
You can achieve the same on the DOS command prompt through the VBoxManage modifymedium command with option --resize <megabytes>. To identify the disk you need its UUID which can be found in the "Manager for virtual media" under Information or by using VirtualBox command VBoxManage list hdds.
Grow the partition and resize the filesystem
Start your VM and check the output of fdisk -l. In the first line you should see the increased volume size.
What we now basically do is
- extend the partition by changing the number of its last disk sector, then
- extend the filesystem which resides on this partition.
So we do not move any byte of data, we simply tell the system that the partition now stretches out more on the volume. Changing the end sector is only possible by deleting and recreating the whole partition with the fdisk utility. Important: Nothing changes on the disk until you issue the "w" command to fdisk.
Note again that the following descriptions assumes that your system has just one volume. In the commands we use /dev/sda which is the most common case.
| Step | Command | What to do | |
|---|---|---|---|
| 1 | df -T | grep ^/dev/ | Check the name of your system partition. The command should display exactly one line of output. The first word is the partition path, usually /dev/sda1 and we will use this value as placeholder in our command examples. Also make sure that the second word (the filesystem type) is ext4. If any of these checks fail or you receive more than one output line you have a non-standard case. If unsure which partition to use please contact your system administrator, Linux expert or XT. | |
| 2 | fdisk /dev/sda | We enter the disk utility for our volume (not partition, so do not write /dev/sda1 here, just /dev/sda!). You should now see the prompt Command (m for help): | |
| 3 | p | This lists the disk partitions. Note down the total number of sectors, the start sector and the partition id (see highlighted data in the above screenshot). As you have already increased the disk size the total number of sectors should be considerably greater then the End sector of the current partition. | |
| 4 | Find the highest multiple of 2048 which is less or equal to the total number of sectors and subtract 1 from this value. This becomes the new last sector of your partition. Example: Your increased volume has 83886080 sectors. Usually this is already a multiple of 2048, as it is in this example. So we simply subtract one, giving 83886079. That's the number we will use as new last sector in step 6. (Note that fdisk usually offers this number as default. Nevertheless it is better to recalculate it to make sure that this default is what you in fact expect. If yes then you can simply press Enter below to accept the default. If your number and the one displayed by fdisk differ better recheck your calculation.) | ||
| 5 | d | Delete the partition. (This does not change anything on the disk yet as changes are only executed when you use the "w" command.) | |
| 6 | n | Recreate the partition with a new end sector and answer the prompts so: - Partition type: p ("primary") - Partition number: 1 - First sector: Should be exactly the same what you had on your old partition (usually it is 2048). - Last sector: Should be the number you calculated above. Usually the displayed default is already correct - When asked whether you want to remove an existing ext4 signature then answer N for "no". If unsure you can always abort the disk utility by pressing CTRL-C and restart at step 2. No changes have been made up to this point as we have not yet written the changes back to disk. | |
| 7 | p | List the new status. The Size column should show the new size of the partition. Make sure path, start and end sector are as expected. Also confirm that the Id is the same as on your old partition. If not then change it with command t. | |
| 8 | Last chance of making sure that all is well. If unsure type CTRL-C to abort. | ||
| 9 | w | Now we change the system as our partition layout changes are now written to disk. A message might appear informing you that the kernel still uses the old partition table which is ok. | |
| 10 | q | Usually this terminates fdisk and returns to the shell prompt. If not then exit the fdisk utility by entering "q". | |
| 11 | partprobe | Inform the kernel about a change of the disk partition layout. If this command is not available or does not work you can instead try kpartx or simply reboot your server (which then of course would possibly affect other users). | |
| 12 | resize2fs /dev/sda1 | This will now extend your filesystem and increases its size by the newly available space. | |
| 13 | df -h /dev/sda1 | Check your current disk usage, the percentage should now be considerably lower and the available space higher. | |
| 14 | If you created a backup snapshot consider when to remove it as cloud providers usually charge money as long as the snapshot exists. |