Overview:
Backup server: running server 2008 r2 standard with backup exec 2010 r3 installed on domain A
Sharepoint client: Virtual server running server 2008 r2, sharepoint 2005 on domain B
SQL client: Virtual server running server 2003 x64 with SQL 2005 on domain B
Issue:
All backups have been running successfully for several years. The domains have a 2-way trust between them. The credentials set in the backup job have not changed and have always worked successfully.
This issue started a few weeks ago when I noticed the remote agent on the sharepoint client would hang at "starting". ultimately I had to reinstall the remote agent and re-establish the security connection between the server and the client.
Once the remote agent was reinstalled the job would fail with invalid credentials. I verified the service account was not locked out. It is a domain admin for domain B and is statically set in the backup job. The domain admin group has full access to both sharepoint and the sql databases.
I statically set the service account as local admin on both servers and gave it site administration for sharepoint and sysadmin for SQL. This did not resolve the issue. I then granted full access to all of the sharepoint sites and added the account to dbreader dbwriter, dbowner and backupoperator for the SQL databases.
I recreated the selection list and the backup job. It ran successfully and I thought the issue was resolved. When the scheduled job ran it failed with invalid credentials. When I tried to view the server farm from the selection list it would error stating the server farm was not responding.
After further troubleshooting I noticed if I remove the server farm from the selection list,save it and then add it back the job will run successfully once without issue. After it is run again (either manually or scheduled) it will fail with invalid credentials.
I have a support case opened with symantec and it has been escalated to another level of support as first level could not resolve it. It has been more than a day since it has been escalated and I have not heard back from them.
Does anyone have any suggestions? Both clients have the same version of the remote agent (13.0 Rev. 5204.127). The backup server is running version 13.0 Rev. 5204 (64 bit).