Manual Rolling Upgrade

Introduction

The manual rolling upgrade provides a way to upgrade Verify Privilege Vault with little to no downtime. That is, users will continue to have secret access during the upgrade.

This procedure only applies to clustered (multiple Web node) Verify Privilege Vault environments environment.

Prerequisites

The administrator role needs the following permissions:

  • Administer Configuration
  • Administer Nodes
  • Administer Backup

In addition, the role:

  • Needs a database login with permission to change the database
  • Requires access with permission to update files on web servers
  • Must go through the current upgrade process
  • Must not turn on maintenance mode until needed

Procedure

Task One: Uploading the Upgrade

  1. Download the latest version of Verify Privilege Vault.

  2. Navigate to Admin > See All > Upgrade Verify Privilege Vault:

    image-20191125140741060

  3. Important: Click to select the Do not put Verify Privilege Vault in Maintenance Mode during the upgrade process check box.

  4. Backup the Verify Privilege Vault application folder.

    Important: Ensure the encryption.config file is backed up. It is located at c:\inetpub\wwwroot\SecretServer\encryption.config.

  5. Click the Backup button to back up the Verify Privilege Vault database.

  6. Click the Continue button. The Upgrade Verify Privilege Vault page appears:

    1567525370967

  7. Click the Advanced (not required) link. The Advanced section appears:

    1567525047708

  8. Click the Choose File button, and select the zip file you downloaded earlier to upgrade to.

  9. Click the Upload Upgrade File button. The new version appears as available for installation:

    image-20191125141009308

  10. Click the Manual Rolling Upgrade link. The Manual Rolling Upgrade wizard appears.

Task Two: Verifying SQL Changes (Wizard Step One)

  1. Click the Next Button. The Verify SQL Deltas tab appears:

    Clicking the "Cancel Manual Rolling Upgrade" link, at any time, will take you to the Install Verify Privilege Vault Upgrade page.

  2. Click the Verify SQL Deltas button. This tests the prospective changes to see if errors result. If errors result, please contact IBM Security Technical Support. If the verification succeeds:

Task Three: Generating the Upgrade File (Wizard Step Two)

  1. Click the Next button. The Generate Upgrade File tab appears:

    image-20191125141217466

  2. Click the Download Application Zip File button. This generates a zip file with only the changed files needed to upgrade the application files on the Web server nodes.

    This may take a few minutes to generate and download.

    image-20191125141601535

Task Four: Generating the SQL Script (Wizard Step Three)

  1. Click the Next button. The Generate SQL Script tab appears:

    1567530833082

  2. Click the Generate SQL Script button. This generates script file with all the database changes needed to upgrade the database. When finished:

    1567530997742

    The wizard proceeds to step four:

Task Five: Backing up and Staging (Wizard Step Four)

image-20191125141910025

  1. Click the Enable Maintenance Mode button.

  2. Back up Verify Privilege Vault: Type "backup" in the Admin search text box, and click the item that appears in the dropdown list to access the Backup Configuration page. Click the Backup Now button.

  3. Click the Disable Maintenance Mode button.

  4. Restore Verify Privilege Vault files to the staging location:

    1. Copy the backup zip file to the staging location.
    2. Unzip the backup file.
    3. Copy the files to the web application folder.

  5. Restore the Verify Privilege Vault database to a staging database:

    1. In SQL Server Management Studio, right click on Databases.

    2. Click Restore Database.

    3. In Source, select Device.

    4. Select and add the backup database file location.

    5. Click Ok.

  6. Go to Admin > Secret Nodes to confirm the staging system is in maintenance mode.

  7. Copy the contents of the generated application Zip file to the staging location's web application folder. Typically, this is C:\inetpub\wwwroot\SecretServer.

  8. Run the generated SQL script on the staging database.

  9. Log on the upgraded staging Verify Privilege Vault to verify the upgrade was successful.

  10. (Optional) Delete the restored staging location and database.

    Important: Keep the backup files till you verify the upgrade was successful. You may need them if an issue develops.

  11. Click to select the Staging Test Successful check box to confirm your staging upgrade was successful. This is your confirmation that there were no errors before performing the actual upgrade in your production environment. The confirmation is recorded.

Task Six: Starting Upgrade Mode (Wizard Step Five)

  1. Click the Next button. The Enter Upgrade Mode tab appears:

    image-20191125150444083

  2. Click the Enable Maintenance Mode button. This mode limits the activities of users on secrets, secret templates, password requirements, and others and can take several minutes to start. A confirmation popup appears.

  3. Click the Enable button to confirm the mode change. The popup disappears.

  4. Click the Enable Ignore Version Mismatch button. This prevents users from being redirected to the Version Mismatch page. A confirmation popup appears.

  5. Click the Enable button to confirm the setting change. The popup disappears.

  6. Click the Next button. The Manual Steps tab appears:

    image-20191125150639306

Task Seven: Upgrading Web Nodes (Wizard Step Six)

To upgrade Web nodes:

  1. Split your nodes into two approximately even groups (A and B) so that one group can service traffic while the other is upgrading.

  2. Ensure "maintenance mode" and "ignore version mismatch" are enabled on each node. You can change them from the Enter Upgrade Mode tab.

  3. On the load balancer, disable traffic to group B. To prevent traffic interruptions, ensure those nodes are all completely disabled before proceeding to the next step. Group A, alone, now handles the traffic. For example, on a F5 Big-IP load balancer you:

    1. Select the Members tab on the pool page.
    2. Select the node to disable.
    3. Click Force Offline.

  4. For each node in group B:

    1. Navigate to the Downloads folder.
    2. Extract all the files from the application zip file downloaded earlier.
    3. Copy the extracted files to the Web application folder.
    4. Log onto the node to ensure the site correctly loads and logs on.

  5. On the load balancer, enable the group B nodes to return them to the pool.

  6. Disable traffic to group A. To prevent traffic interruptions, ensure those nodes are all completely disabled before proceeding to the next step. Group B, alone, now handles the traffic.

  7. Execute the script you created on the database, confirming there are no errors. If there are errors, follow the rollback instructions.

  8. Log onto each group B node again to ensure the site correctly loads and logs on.

  9. For each node in group A:

    1. Navigate to the Downloads folder.
    2. Extract all the files from the application zip file downloaded earlier.
    3. Copy the extracted files to the Web application folder.
    4. Log onto the node to ensure the site correctly loads and logs on.

  10. On the load balancer, enable the group A nodes to return them to the pool, restoring the original configuration and returning traffic to all nodes.

  11. Click to select the Upgrade Successful check box.

  12. Click the Next button. The Finish tab appears:

    image-20191125150852514

Task Eight: Finishing up (Wizard Step Seven)

  1. Click the Disable Upgrade Mode button. A Finish Manual Rolling Upgrade popup appears. This popup both disables maintenance mode and disables the ignore version mismatch setting.

  2. Click the Disable button. The popup disappears, and a completion message appears:

    image-20191125151058172

Troubleshooting and Notes

Rolling Back to the Previous Version

If you encounter errors at any step of the upgrade, rollback to the previous Verify Privilege Vault version:

  1. Restore the database from the backup.

  2. Restore the application files from the backup files to the Web application folder on all nodes.

  3. On the load balancer, restore the original configuration, sending traffic to all nodes.

  4. For assistance, contact us.

Version Guard

If an uploaded upgrade file cannot be used to upgrade the current version of Verify Privilege Vault, then "Version Guard" will block the upgrade and provide instructions on how to continue:

1567605592248

This usually occurs when not completing the perquisite steps in order. Click the Return To Upgrade button to return you to the first upgrade page to remedy the situation.

image-20191125151442551

This page also list the blocking versions that you must upgrade to prior to running the manual rolling upgrade.

New Advanced Configuration Setting

There is a new setting called "Manual Upgrade: Allow version mismatch while in Maintenance Mode." This setting, which only applies in maintenance mode, prevents Verify Privilege Vault from redirecting users to the version mismatch message page.

New Audit Type

To support the manual rolling upgrade, there is a new audit type—ManualUpgrade. Its audits are stored in the tbAudit table and record the following actions:

  • CANCEL
  • COMPLETED
  • GENERATE DB SCRIPT
  • GENERATE UPGRADE ZIP
  • STAGING TEST
  • STARTED
  • VERIFY DELTAS