VCSA filesystem is out of disk space vcsa 6.5

This week while doing the normal check of the environment I came to run into a warning on the VCSA 6.5 stating: PostgresSQL Service Health Alarm

This was the first time I saw this kind of warning on the environment.

After logging in to the vSphere Appliance Managment I could see more detailed information about the issue.

Time for a deeper investigation

Nothing else is visible relate to this incident so I logged into VCSA using SSH to check the respective filesystem disk space using the command:

df -h

Here it is the problem, filesystem /dev/mapper/log_vg-log mounted on /storage/log has ZERO Available space

After identifying the low space filesystem there are two options that we can follow, increase the size of the disk on VCSA and for this one that is several blog post about it (Settlersoman) or try to find out the file and clean it.

I followed the second one and for that, I used the following command to search the filesystem for huge files/folders:

du -a /storage/log/ | sort -n -r | head -n 20

messages.1 with 20GB in size?????? This should not happen!!!

Now we know what is consuming the free space. For some reason, the file is not being compressed, I deleted the file using the command:

rm -Rf messages.1

Hoping I fixed it I checked again the filesystem usage and there was no change in the free space, /storage/log was still without free space.

Now what?


It looks like VCSA was not reclaiming the free space… After searching some blogs and VMware community posts the solution was to reboot VCSA but that was out of the question for now as it would impact production, and we don’t want that.

Why was the filesystem still full? I just deleted a 20GB file…

In Linux when a file is removed/deleted/unlinked if it’s still held open by any process only the entry of the file is removed, not the file’s data. To be able to free the data we have to close the process that is using the file.

Let’s see how many open deleted files we have on the system by running the following command:

lsof +L1

After identifying the file we have two options, or we kill the process or we restart the process. Because it wasrsyslogI decided to restart it using the command:

service rsyslog restart

After I checked the filesystems and there it is, 19GB free on /storage/log

df -h output

Leave a Reply

Your email address will not be published. Required fields are marked *

RELATED POST

Using WinSCP with the VMware vCenter Server Appliance VCSA 6 7 8

Connecting to VCSA with WinSCP If you try to connect to the VCSA with WinSCP in a default configuration, you…

VMWARE VSPHERE – DEPLOY .OVF ERROR:VIM.FAULT.FILENOTFOUND

I recently upgraded my vSphere host from vSphere 5.5 to 7.0. When I tried to deploy one of my .ovf…

How to Reset Forgotten VMware ESXi Root Password

step 1: Boot your ESXi server from USB Live CD linux. website download image boot live cd boot: https://gparted.org/download.php link…

Recreating a missing VMware virtual machine disk descriptor file (.vmdk) (1002511)

Details This article provides steps to recreate a lost virtual disk descriptor file (.vmdk). You may need to recreate missing…