Understanding and remedies for Datastore FULL issues

This article explains about various issues encounter for datastores. We have a more insight on possible remedies and prevention measures by taking proactive approach.
Storage LUN is discovered and added to esxi hosts is called as datastore. Which is formatted with filesystem known as VMFS. Virtual machines reside on datastores.
When we create virtual machine, we select datastore to place VM in it. So, for datastores there are possibilities of issues may encounter such as datastore over provisioned and datastore usage full.
Now we see when these issues occur. For example, your datastore size is 4 TB, you create a VM hard disk of size 6TB with Thin format which you can place in 4TB however when user start to dump the data in hard disk which increases hard disk size in datastore. If the hard disk size goes beyond 4TB your VM will not be accessible. It causes outage for that VM which this below error message.
Solution: Always create VM hard disk of Thick Eager Zero format for better performance and to avoid datastore over provision issues. Make ensure datastore have at least 10-15% free reserved space for snapshots.
Some VMware infra have backup solution which takes snapshot of VMs as a backup. So daily backup snapshots will consume additional space in the datastores. This may also result in datastore over provision.
As a proactive approach you can configure 2 alarms which save you from VM outages.
1.   Datastore Over provision alarm: alerts will trigger after VM hard disks consumption size is greater than total capacity.
  • Which alerts you about datastore status while you create virtual machine.
  • When snapshots of VMs is taken and it results in usage of disks more than datastore capacity.
2.   Datastore Full Usage alarm: alerts trigger before VM hard disks consumption reaches maximum datastore capacity. In this alarm you can set threshold values for warning as 85% and red alert for 90%
Alarms configurations
  1. Give appropriate name for alarms
  2. Give alarm type as datastores – monitor for specific conditions or state
  3. Click on Enable this alarm
  4. Define threshold values to trigger alarms
  5. You can set actions to send email notifications, SNMP traps, run some script.
  6. There is an option to select – actions will repeat until the alarm type changes.
    Note: Once alarm is triggered take an action and acknowledge the alarm. If there is no action required for that particular datastore you can go to that specific datastore – right click and disable alarms.
    Datastore over provision alarm configuration
Datastore Usage on disk alarm configuration