Currently, Failover Service and Disaster Recovery are supported only for servers running on Windows OS. 

Benefits of HA Configuration   

1. Availability of AssetExplorer at all times.
   
2. Smooth and automatic take-over of the failed server.
   
3. Provides quick recovery from hardware or software failures.
   
4. Prevents data loss during server failure.

Workflow of HA   

1. Primary Server - The server where the application is currently hosted.
   
2. Secondary Server - The standby backup server that monitors the primary server's health and constantly replicates files from it.
   
3. Database Server - The primary and secondary servers are linked to a shared database server. The database server monitors the primary server's health and replicates its data.
   

The primary server, secondary server, and database server are connected to a common network.    

1. DB heartbeat Mechanism: The primary server communicates its status to the database server every minute. If there is no update, the secondary server will take over. This is the default method to monitor a peer machine's health.
   
2. Polling Mechanism: In this method, when the primary server goes down, the backup server is pinged about the status, and the secondary server takes over.
   

Polling Mechanism requires an SSL certificate and can be configured only in the HTTPS protocol. In case of database failure, the application automatically sets up this method. 

Failover Service vs Disaster Recovery   

Prerequisites 

1. Two 64-bit server machines with high network connectivity.
   
2. AssetExplorer 64-bit .exe installation is preferred.
   
3. Two virtual or physical servers with different NIC card addresses.
   
4. A common IP Address for the primary and secondary servers.
   
5. Both servers must have two-way read-write access for the ManageEngine folder (where AssetExplorer is installed).
   

AssetExplorer Configuration Requirements  

1. The application must be installed in the same location on both servers.
   
2. Purchase an FOS license as an add-on for your application.
   
3. The database must be externalized from primary and secondary servers but remain accessible to both. You can either use an MSSQL database or migrate to version 7520 and use an external PGSQL database.
   
4. Set File attachment network path to the same domain and accessible to both servers.
   
5. Obtain HTTPS certificate for Alias URL of common IP.
   
6. Bind the common URL (for which the SSL certificate is obtained) to the common IP address configured in the DNS.
   
7. Start the application as a service using Java Service Wrapper.
   
8. Applications on primary and secondary servers must run on the same port.
9. Use the same polling and retry parameters in both servers.
   

 Prerequisites for Endpoint Central installed within AssetExplorer   

For customers who have purchased Endpoint Central separately, please refer here for steps to follow if you do not have a Failover Server setup. 

1. Endpoint Central > Admin > Tools > Database Backup > Backup Directory.
   
2. Endpoint Central > Admin > Software Repository > HTTP Repository > New Location.
   
3. Endpoint Central > Patch Mgmt > Downloaded Patches > Settings > Patch Repository Location.
   

Setup Process      

1. Run mirrorSetup.bat from the \bin directory after installing the application on the primary server.
   
2. However, the application will not start as a service. To start it as a service, run sd_service.bat -i from the {AssetExplorer_Home}/bin directory on the secondary server.
   

Folder Permissions

1. Share the ManageEngine folder between the primary server and the secondary server.
   
2. Set the folder permission to Everyone to ensure the servers have full read/write permissions.
   
3. To access the shared folder on your server, go to the start menu > Run > \\<machineIP>\ManageEngine. 
   
1. Provide the username and password (if needed) to make sure an IPC connection is established between the machines.
   
4. If you want to restrict the folder access to one particular user account,
   
5. Run services.msc.
   
6. Search for AssetExplorer and go to Properties.
   
7. Select the Log on tab and choose This account option.
   
8. Enter the login credentials of the domain user and save it.
   

Configure Failover Service   

1. Go to Admin > General Settings > HA Configuration.
   
2. Select the HA Mode as Failover Service.
   
1. Select the Enable FOS Startup Mode.
   
2. Use the pointers below to configure the primary and secondary server details.
   
3. Primary Server IP: Enter the IP address of the server where the application is running.
   
4. Secondary Server IP: Enter the IP address of the standby server.
   
5. NIC of Primary Server/NIC of Secondary server: Enter the NIC address of the primary and secondary servers in the respective fields within curly braces { }. For example, {117C3D5B-4395-4369-8812-741EEA26D76D}. Click here to learn how to identify the NIC addresses.
   
6. Subnet Mask of Primary Server/Subnet Mask of Secondary Server: Enter the subnet mask of your primary and secondary servers in the respective fields. Refer here for the valid subnet values.
   

How to identify NIC address? 

AssetExplorer contains an inbuilt provision to help you identify your NIC address.

1. Go to \bin directory in your secondary server.
   
2. Execute the iflist.exe tool.
   

1. Locate the NIC Card whose Adapter Status is UP. The adapter name is the NIC address for the IP address to which the application is bound.
   
2. Copy the value in the Adapter Name field to the application.
   

Valid Subnet Mask Values 

/30	255.255.255.252
/29	255.255.255.248
/28	255.255.255.240
/27	255.255.255.224
/26	255.255.255.192
/25	255.255.255.128
/24	255.255.255.0
/23	255.255.254.0
/22	255.255.252.0
/21	255.255.248.0
/20	255.255.240.0
/19	255.255.224.0
/18	255.255.192.0
/17	255.255.128.0
/16	255.255.0.0

 

How to test a virtual IP address? 

The virtual IP address is a common IP address in the local network, that is not bound to any specific machine. A simple way to check if an IP address can be used as a common one is to ping the IP address. If the IP is not reachable, then it can be used as a common IP address. 

 To test the virtual IP address, 

1.  Open your command prompt and execute the following command: ping {insert IP address}. 
   
2.  If you receive a request timed-out message, it indicates that the virtual IP address is available for use. 
   

1. Virtual IP: Specify the IP address to which the primary and secondary servers must be bound. Click here to learn how to test a virtual IP address.
   

Ensure that the virtual IP address you configure belongs to the subnet mask and is not bound to any existing servers in the network. 

1. Common Alias Name: Specify an alias name to access the application.
   
2. In case of failover notify to: Enter the email address for server failure notifications. Separate multiple addresses with commas.  
   

For the notification emails to be sent, the outgoing mail server should be configured for the application.

1. Click Save and restart the application for the configurations to take effect.
   

After configuring FOS, set up file replication to schedule.

When the primary server is switched to the secondary server, the server configurations on the FOS page must be manually updated. 

Configure Disaster Recovery   

1. Navigate to Admin > General Settings > HA Configuration.
   
2. Select the HA Mode as Disaster Recovery.
   
3. Select the Enable DR startup mode check box.
   
4. Use the following pointers to configure the application in DR mode.
   
1. Primary Server IP: Enter the IP address of the server where the application is running.
   
2. Secondary Server IP: Enter the IP address of the standby server.
   
3. Common Alias Name: Specify an alias name to access the application.
   
4. In case of disaster notify to: Enter the email address for server failure notifications. Separate multiple addresses with commas.  
   

For the notification emails to be sent, the outgoing mail server should be configured for the application.  

1. Click Save and restart the application for the configurations to take effect.

After configuring Disaster Recovery, set up HA file replication configuration.   

When the secondary server takes over as the primary server, the configurations in the DR page must be updated manually.  

Modify Primary Server's Listening Time 

1. Navigate to {AssetExplorer_home} / conf.
   
2. Open the ha.conf file and find the entry #peer.status.check.time.period=
   
3. Uncomment the entry by removing the hashtag
   
4. Specify the required time limit in minutes. For example: peer.status.check.time.period=10.  
   

 Upgrade HA Configuration

Announcement for Users using AssetExplorer version 6957 and below 

The current FOS configurations will not work after the upgrade and you will be automatically migrated to the new FOS. Enable new FOS for continued support.

 Announcement for Users using AssetExplorer version 6971 and above 

Users upgrading to version 6971 and above are not required to mirror the updates in both servers as the changes are automatically pushed during the upgrade. 

1. Invoke \bin\shutdown.bat in the primary and secondary servers. This will stop AssetExplorer.
   
2. Upgrade the build in the primary server. Click here to learn how.
   
3. Mirror the settings in the secondary server via Robocopy by invoking \bin\mirrorSetup.bat. This step applies only to users upgrading to version ≥ 6900.
   
4. Invoke \bin\run.bat in both servers to start AssetExplorer.
   

Ensure that you backup your files before upgrading your application. Click here to know more. 

 

Restore HA Configuration

1. Invoke \bin\shutdown.bat in the primary and secondary servers. This will stop AssetExplorer.
   
2. Restore the data in the primary server. Click here to learn how.
   
3. Mirror the settings in the secondary server via Robocopy by invoking \bin\mirrorSetup.bat
   
4. Invoke \bin\run.bat in both servers to start AssetExplorer.
   

Why is FOS/DR disabled on restoring AssetExplorer and an application reboot is required? 

Backups with FOS/DR may fail to restore on a different machine, causing the application to not start due to an incorrect NIC address in ha.conf. To prevent this, FOS/DR will be disabled by default. SDAdmins can enable FOS or DR and configure the NIC address under Admin > General Settings > HA Configuration.  

Disable Failover Service/Disaster Recovery 

1. Go to Admin > General Settings > HA Configuration.
   
2. Uncheck the Enable FOS Service Startup Mode/Enable DR Startup Mode in the Failover Service and Disaster Recovery configurations respectively.
   
3. Invoke \bin\shutdown.bat in both the primary and secondary servers. This will stop AssetExplorer.
   
4. You can then restart the application in the servers individually by invoking \bin\run.bat command. On restarting, the servers will behave individually as two separate machines.   

Troubleshooting

Unable to Save HA Configuration or HA File Replication Configuration 

1. If yes, proceed with Step 2 directly.
   
2. Otherwise, start the application as a service in Windows service manager and then perform Step 2.
   

1. If configured, proceed with Step 3 directly.
   
2. Otherwise, configure a dedicated user account for the service and then perform Step 3.
   

Local system account is not recommended because of permission issues during HA file replication.

The configured user account should belong to the Admin group.

Step 3: Check whether the ManageEngine\<application_directory> folder is available on both primary and secondary application servers. For example, ManageEngine\AssetExplorer.

1. If available, proceed with Step 4 directly.
   
2. Otherwise, modify the folder structure as mentioned above and then perform Step 4.
   

1. Right-click ManageEngine folder, go to the Sharing tab, and check if Network Path contains <machine name>\ManageEngine.
   
2. If yes, proceed with Step 5 directly.
   
3. Otherwise, share the folder by clicking Share... and then perform Step 5.
   

Step 5: Check if Share name is ManageEngine:

1. Right-click the ManageEngine folder, go to the Sharing tab, and select Advanced Sharing... 
   
2. Check if Share this folder is enabled and Share name is ManageEngine.
   

          

1. If yes, proceed with Step 6 directly.
   
2. Otherwise,
   
1. Remove any other ManageEngine folder shared with ManageEngine as share name.  
   
2. Re-share the ManageEngine folder where the application is available as subdirectory and then perform Step 6.
   

1. If available, proceed with Step 7 directly.
   
2. Otherwise, add the user account to the Sharing and Advanced Sharing lists, provide read/write permissions and full control, and then perform Step 7.
   

1. Run the command \\<secondary machine IP>\ManageEngine and \\<primary machine IP>\ManageEngine on corresponding application servers.
   
2. If accessible, proceed with Step 8 directly.
   
3. Otherwise, fix the network issue and then perform Step 8.
   

HA Service Startup Failure on Primary/Secondary Application Server in Serving State 

1. For the FOS mode, proceed from Step 2.
   
2. For the DR mode, proceed from Step 5.
   

1. Check the log file logs/serverout0.txt for the following trace:
   

ERROR: AddIPAddress failed with error code [5].Access is denied.|

1. If found, add the user account configured for the service in the Admin group and then perform Step 3.
   
2. Otherwise, proceed with Step 3 directly.
   

1. Check the log file logs/serverout0.txt for the following trace:
   

ERROR CODE: [ 1,004 ] DESC [ ERROR_IF_DOWN ]

[1004] [ERROR_IF_DOWN]. Network Interface Card is NOT working

1. If found, disable FOS and configure the proper NIC address in HA Configuration and then perform Step 4.
   
2. Otherwise, proceed with Step 4 directly.
   

1. Check the log file logs/serverout0.txt for the following trace:
   

ERROR: AddIPAddress failed with error code [5010].The object already exists

1. If found, run the ping command to check whether the virtual IP is bound to any application server, and then perform Step 5.
   
2.  A Request timed out response indicates that the virtual IP is not bound to any application server.  
   
3. Otherwise, perform Step 5 directly.
   

1. Check the log file logs/serverout0.txt for the following trace:
   

Error: HA service can be started only with FOS Add-on license!!!

1. If found, disable FOS/DR, start the application as a standalone service, apply license with FOS add-on, enable high availability and then perform Step 6.
2. Otherwise, proceed with Step 6 directly.
   

 SERVER SHUTDOWN - FOS build configured and file attachment path is not set as a network path!

1. If found,
   
1. Disable FOS/DR.
   
2. Externalize the attachment path.
   
3. Enable high availability (under HA Configuration) and then perform Step 7.
   
2. Otherwise, proceed with the next step directly.
   

1. logs folder from the primary application server
   
2. conf/ha.conf file from the primary application server
   
3. conf/wrapper.conf file from the primary application server
   
4. Output of the query: select * from haconfig;
   

Steps to Disable FOS/DR     

1. Step 1: Run the following query:
   

update HAConfig set paramvalue='Disabled' where category='HAMODE';

1. Step 2: On both primary and secondary application servers, replace the file module-startstop-processors.xml in the /conf/persistence/directory with the one from a fresh installation.
   

HA Service Startup Failure on Primary/Secondary Application Server in Listening State   

1. If yes, proceed with Step 2 directly.
   
2. Otherwise, start the application as a service in Windows service manager and then perform Step 2.
   

1. If configured, proceed with Step 3 directly.
   
2. Otherwise, configure a dedicated user account for the service and then perform Step 3.
   

 Local System account is not recommended because of permission issues that occur during HA file replication.

1. If available, proceed with Step 4 directly.
   
2. Otherwise, modify the folder structure as mentioned above and then proceed with Step 4.
   

1. Right-click the ManageEngine folder, go to the Sharing tab, and check whether the value of Network Path contains <machine name>\ManageEngine.
   
2. If yes, proceed with Step 5.
   
3. Otherwise, share the folder by using the Share... button and then perform Step 5.
   

                                   

1. Right-click the ManageEngine folder, go to the Sharing tab, select Advanced Sharing... 
   
2. Check if Share this folder is enabled and Share name is ManageEngine.
   

                            

1. If yes, proceed with Step 6 directly.
   
2. Otherwise,
   
1. Remove any other ManageEngine folder shared with share name as ManageEngine.
   
2. Re-share the ManageEngine folder where the application is available as a subdirectory, and then perform Step 6.
   

1. If available, proceed with Step 7 directly.
   
2. Otherwise, add the user account to the Sharing and Advanced Sharing list, provide read/write permissions and full control and then perform Step 7.
   

1. Run the command \\<secondary machine IP>\ManageEngine and \\<primary machine IP>\ManageEngine on corresponding application servers.
   
2. If accessible, proceed with Step 8 directly.
   
3. Otherwise, fix the network issue and then perform Step 8.
   

1. If enabled, proceed with Step 9.
   
2. Else, enable HA under HA Configuration and then perform Step 9.
   

1. If configured, proceed with Step 10 directly.
   
2. Otherwise, configure the HA File Replication and then perform Step 10.
   

[UNSUPPORTED_CHARACTER_IN_PASSWORD]. exclamation is not supported in password. Please reset the password and try again

1. If found, change the password (with appropriate characters) under Admin > General Settings > HA File Replication Configuration, and then perform Step 11.
   
2. Otherwise, proceed with Step 11 directly.
   

Connection command execution failed. Message: The user name or password is incorrect

1. If found, check whether the username under HA File Replication Configuration is a domain account and contains '\\' as a separator and go to step 12.
   
2. Otherwise, proceed with Step 12 directly.
   

1. If yes, proceed with Step 13 directly.
   
2. Otherwise, copy the conf/customer-config.xml file from primary application server to secondary application server and then perform Step 13.
   

Replication schedule termination did not complete in [120]. Try starting the build

1. If found, check whether any CPU / memory spikes appear in task manager, start the secondary application server as service in Windows service manager, and then perform Step 14.
   
2. Otherwise, proceed with Step 14 directly.
   

Error message: Error in connecting to remote server. The specified network password is not correct.

1. If found, it indicates that the password entered in HA file replication configuration is incorrect. Therefore, update the correct password in HA file replication configuration and then perform Step 15.
   
2. Otherwise, proceed with Step 15 directly.
   

1. MapNetworkDrive.ps1 cannot be loaded because running scripts is disabled on this system
   
2. If found, set powershell execution policy as RemoteSigned for the current user.
   
3. Otherwise, proceed with Step 16 directly.
   

Error type: SYNC_FATAL_ERROR, Error message: Error in connecting to remote server. & : AuthorizationManager check failed.

1. Right-click the application_home/bin/MapToNetworkDrive.ps1 script file and select Properties.
   
2. In Properties, go to the Digital Signatures tab.
   
3. Select the Zoho certificate from the list, then click Details.
   
4. In Certificate, go to the Details tab.
   
5. Click Copy to File.
   
6. In Certificate Export Wizard, click Next.
   
7. Select DER encoded binary X.509 (.CER).
   
8. Choose a destination path and file name, click Next, and click Finish to export the certificate.
   
9. To open Certificate Manager, press Windows + R, type certmgr.msc, and press Enter.
   
10. In the left pane, expand Trusted Publishers under Certificates - Current User.
    
11. Right-click Certificates under Trusted Publishers, and select All Tasks > Import.
    
12. In Certificate Import Wizard, click Next, then browse to select the .cer file you exported earlier.
    
13. Click Next. Make sure that the certificate store is set to Trusted Publishers and click Finish.
    

1. logs from the primary application server
   
2. conf/ha.conf file from the primary application server
   
3. conf/wrapper.conf from the primary application server
   
4. Output of the following queries:
   
1. select * from haconfig;
   
2. select * from fosnodedetails;
   
3. select * from serverstatus;