Blog

Fixing Backup Exec Advanced Open File Option VSS Snapshot errors

 

For several weeks, I had been receiving the error message below on most of my backup jobs for servers on my network.  The errors seemed to have started all of a sudden, and I want to share my research and troubleshooting steps that finally allowed me to resolve this frustrating problem.  There is a wealth of information out there about this error – some of it good, some of it not so much.  Hopefully this blog will help bring together the points that I found most helpful in my quest to find a resolution.

Backup- servername.DOMAIN.COM

– AOFO: Initialization failure on: “\servername.DOMAIN.COMShadow?Copy?Components”. Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11231 – VSS Snapshot error. Activity on a volume is preventing all volumes from being snapped. Try selecting the option ‘Process logical volumes for backup, one at a time’ , and then run the job again.

 

First things first, I started with the recommendations given to me in the error message.  To reconfigure our backup job to “Process logical volumes for backup, one at a time”, see this Support Document from Symantec: http://www.symantec.com/business/support/index?page=content&id=TECH69327&actp=search&viewlocale=en_US&searchid=1285083683731

Also, since the error specifically states that activity on the volume being backed up is preventing the snapshot from occurring, I did several things to quiet disk activity during the backup.  Here are a few things that I found that may have contributed to excessive disk activity on my servers:

  • Scheduled tasks that run during the backup
  • SQL Server Maintenance plans that run during the backup
  • Microsoft Indexing Service and Microsoft Search (If your server is a file server or Sharepoint server, this can be useful, otherwise, I would think that these services should be disabled)
  • Anti-Virus scheduled scans that may run during the backup
  • Defrag jobs scheduled during the backup (more on this later)

 Here are some other general tips that I found useful in ensuring my servers were optimally configured so that VSS snapshots can run successfully:

  • Free up disk space.  Having a low amount of free space can cause issues with VSS, backups, and system performance.  Here are some tips for freeing up space on the C drive:
  • Delete temp files in temp, WindowsTemp and your profile temp, which you can find by typing %temp% in the Run dialog and clicking OK.  
  • Check your log files in Windowssystem32LogFiles, as IIS can be reconfigured to start logging to a drive with more space.
  • NTUninstall folders in your Windows folder can be moved to another drive, or deleted.  However, you cannot uninstall updates once you do this. 
  • Many times, the WindowsSoftwareDistribution folder will get very large from cached Windows Updates.  You can safely delete this folder by stopping the “Background Intelligent Transfer Service” and the “Automatic Updates” services, deleting the folder, and restarting those services. 
  • Another place on your C drive that will sometimes grow very large is the WindowsInstaller folder.  If you download the Windows 2003 Support Tools, you can run msizap.exe –G to remove any orphaned patch files that live in the WindowsInstaller folder.
  • Once you free up disk space, you should defrag your volumes.  If you have off-hours where your server is not heavily used and you can schedule defrag jobs, I would highly recommend doing so.  I have found several references out there on the Internet that indicate a highly fragmented file system can cause VSS errors.  At the beginning of my quest to resolve this problem, I discovered that many of my volumes were highly fragmented.  I now have scheduled defrag jobs that run (outside of my backup window, of course!)
  • Make sure that you are up to date on all Microsoft Windows Updates and Service Packs.
  • Run Live Update to install all updates for Backup Exec.  Ensure that you are running all of the latest Service Packs (currently SP4 for Backup Exec 12.5) and Hotfixes.

Please note that after installation of Backup Exec Service Packs and some Hotfixes, it is necessary to update the Backup Exec Remote Agents on your remote servers.  You can redeploy those agents from within Backup Exec the same way you installed them originally – by going to Tools, and then selecting “Install Agents and Media Servers on Other Servers”.  Should that fail (as several of mine did), you can navigate to Program FilesSymantecBackup ExecAgents on your media server and copy the folder of the Windows agent for your server architecture – RAWS32 for 32-bit servers, and RAWSX64 for 64-bit servers to the remote server.  Then on the remote server, execute the setupaa.cmd to update your agent.  A reboot will likely be required.

Microsoft has released a VSS Update Rollup that addresses multiple VSS issues.   If you have the netdiag.exe tool installed from the Windows Server 2003 Support Tools, you can quickly find out if you have this update applied by typing netdiag | find “940349”.   If there is no output from the command, or it does not show v3 of this update, then you should download and apply this update.  This should be done on all servers being backed up, and this will also require a reboot on each server.  Information and a link to download the update are found here: http://support.microsoft.com/kb/940349

It can also be helpful to reregister the DLL files required for Microsoft Volume Shadow Copy.  I have included a sample batch file that can help reregister those files.

Cut and paste the following into notepad, and save it as FIXVSS03.BAT.  Run FIXVSS03.BAT to reset VSS configuration.

——begin cut here——

rem FILENAME: FIXVSS03.BAT
rem
net stop “System Event Notification”
net stop “COM+ Event System”
net stop “Microsoft Software Shadow Copy Provider ”
net stop “Volume Shadow Copy”
cd /d %windir%system32
net stop vss
net stop swprv
regsvr32 /s ole32.dll
regsvr32 /s oleaut32.dll
regsvr32 /s vss_ps.dll
vssvc /register
regsvr32 /s /i swprv.dll
regsvr32 /s /i eventcls.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s vssui.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml4.dll
Cd /d %systemroot%syswow64
regsvr32 /s ole32.dll
regsvr32 /s vss_ps.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml4.dll
net start “COM+ Event System”

—–End Cut Here—–

Note: The syswow64 folder will not exist on 32-bit Windows 2003 servers, so the batch file may give errors during that part of the execution.  These errors can be ignored.

Finally, I found this Microsoft Knowledge Base article on changing the initial storage area that VSS uses when it creates snapshots.   See the More Information section at bottom of this document: http://support.microsoft.com/kb/826936

I found recommendations that this be set to the maximum 3GB size (3000 decimal).  You need to be sure that you have 3GB of disk space available if you increase this number to the max.  I set the size of the MinDiffAreaFileSize key to 3000 decimal and most of my servers then stopped having the VSS errors during the backup.  However, on my main file server, I received another VSS error that shows up in the Application log on my main file server that says:

Event ID 7001 in the Application log

VssAdmin: Unable to create a shadow copy: Insufficient storage available to create either the shadow copy storage file or other shadow copy data.

Command-line: ‘C:WINDOWSsystem32vssadmin.exe Create Shadow /AutoRetry=5 /For=\?Volume{0e23512e-ab34-11db-902d-0002b3c80089}’.

I then went back to that registry key and lowered theMinDiffAreaFileSize to 2000 (after making the registry change, you can cut the information in the event information after “Command-line:” and can paste it in a command prompt window to run a snapshot immediately.  Then go check the application log again to see if the snapshot was successful or not).  The snapshot failed at 2000 also, and finally has been running successfully set at 1000 for a couple weeks now.

I have now had successful backups now for a couple of weeks.  Hopefully this information is helpful is solving your issues with Backup Exec and VSS snapshot errors.