The reason for this post is that I often come across CommVault environments that have hosts which have not been decommissioned correctly. This results in the unnecessary consumption of both licensing and storage space/media. It seems that the correct way to decommission servers from Simpana isn’t something that gets highlighted enough during CommVault training. This also seems to be one of the first things people get wrong when they inherit a system. Backup administrators often complain that jobs aren’t being aged correctly, or they have unexpectedly used up all their licensing. Incorrectly decommissioned servers are the #1 reason for this.
Depending on how the server you are decommissioning is protected, will change how it needs to be removed in Simpana. Licenses may need to be released, and backup jobs may need to be cleaned up.
The following flow chart describes the considerations and procedures required to successfully decommission servers from CommVault:
Here are the procedures outlined in the flow chart:
A. Go through each installed agent and note down with which Storage Policies it is associated. It is important to do this first because after releasing the license this will become more difficult.
B. Right click on the host in the CommCell browser and select All Tasks -> Release License. More info is available on BOL. In completing this step you have now freed up the licenses utilised on this server. For people with capacity based licensing this will only be reflected when Data Aging is run (normally scheduled to run daily at 12pm).
C. Here is where most people get caught out. If you remember back to your CommVault training you will recall that retentions for copies are specified in days and cycles. Days are pretty straight forward and cycles refer to the number of complete backup cycles (new ones are started every full backup). Both need to be met before the job can be aged. Wait a minute, how can that be? The cycles dependency will never be met for the last full backup cycle on a deconfigured agent since it will never produce any new jobs. That’s right. Simpana will keep the last backup cycle (full + any incrementals) forever until you manually delete the jobs.
There are two solutions to this challenge:
- (D.) Manually going through each copy deleting the backup jobs, and/or creating calender items to remind yourself to clean those jobs up later, or…
- Enabling the ‘Ignore Cycles Requirement on Deconfigured clients‘ option.
I strongly suggest doing the latter, since it makes management much easier. It’s really how most people would expect the system to operate.
E. Simply right click on the subclient and select delete. Historic jobs will be aged as per normal using the configured days retention. Since this is subclient level stuff, cycles don’t apply, so you don’t need to worry about manually deleting any jobs. To restore VMs from these old backup jobs, do your Browse/Restore operation on the backup set, or alternatively browse history at the backup set level.
F. Browse the properties of the subclient associated with the VM, and delete it from the contents. For capacity based customers, licensing will be freed up when the subclient next runs a full backup.
I hope you found this useful, don’t hesitate to contact me with any comments or questions.
1. Pre-EmptThey say prevention is better than cure, here is a list of things that you should be asking yourself:
- Have you reminded your users to scrutinise every email. Most CryptoLocker infections are from emails convincing users that they have received a speeding fine or a package tracking notice that required them to download and run an executable. Being asked to run executables from the internet should always be a cause to be wary. If you smell deceit, hit delete!
- Are you using a robust backup product? When was the last time you tested restoring from backups? Do you keep backups off site? Don’t get caught out. Good, reliable backups are CRITICAL. How long is your backup retention? What if you only notice files encrypted weeks later…
- Do you have a transparent proxy that filters web traffic? Palo-Alto, Sopho UTM, McAfee, Checkpoint etc.
- How about on your workstations? Is the antivirus up to date?
- Have you applied an auditing policy on your file server? This won’t stop CryptoLocker from running, but will making finding the infection much easier. Simply enable file system auditing, and apply an auditing rule on your file shares to capture, file creation/deletion. This way you can find out from the logs which user logged into which machine deleted documents and created encrypted versions.
- Do you trust your antivirus? There are hash based file blocks you can configure using Group Policy to add further protection. This will from a Windows policy level stop CryptoLocker from running if it is a variant that matches the policy.
2. Identify SourceOnce you are infected your first step should be to identify which workstation is infected. Otherwise you can restore your files all you like, they will just get re-encrypted.
3. Clean the InfectionCryptoLocker is not a worm, the encrypted files won’t infect anyone else, and unless you run the executable on another machine the infection will not spread. Still it’s best to isolate the workstation as soon as possible by unplugging the network cable. Use your usual malware removal tools such as Malwarebytes to clean the system of CryptoLocker.
4. Identify Damage DoneNow its time to determine the extent of the damage. Scan the user’s mapped drives for .encrypted files. Here is a simple powershell two-liner to do it. Change the value of $path to the location you want to scan. The script will output a text file called encryptedList.txt which will contain all the files that have been encrypted.
6. CleanOnce you are satisfied that you have restored everything, you will want to delete the encrypted files, and the DECRYPT_INSTRUCTIONS.html files that got created. Here is another simple powershell script.
7. Post-MortemNow that you have survived CryptoLocker, its time to ask the important questions.
- How did the infection occur? Did it originate from an email? How did it get past our defences? Was the virus vendor aware of this variant?
- Were you appropriately prepared? Did you know how to use your backup product or were you second guessing every option?
- How do we stop future infections? Better web filters? Better antivirus? Do users need further education?