Distributed Engine Overview

Distributed engine configurations for Verify Privilege Vault On-Premises and Verify Privilege Vault Cloud are not 100% equivalent. To view the differences, please refer to our reference architectures for Verify Privilege Vault. They are, at minimum, refreshed every year and are created by our Professional Services Solutions Architect team.
Distributed engine upgrades are no longer mandatory for every release. The Distributed Engine Configuration page can set the minimum required engine version. Modifying this triggers an automatic update for any engine below this version.

In the action menu for an engine on the Sites page, you can start a manual upgrade for individual engines below the latest version, which prompts the engine to update when it next calls in.

When changes are made needing an upgrade, the minimum required version is updated during the update process, and all engines update immediately.

Overview

Out of the box, Verify Privilege Vault performs all functions from the Web server it is installed on; however, specific features can be routed through a distributed engine for enhanced performance. For example, synchronize and authenticate AD users can be done in Verify Privilege Vault via your local site or from a distributed engine (DE).

You can install a DE in a remote site and allow it to operate many functions. Communication with Verify Privilege Vault Cloud also requires the distributed engine to be installed.

Architecture and Workflow

Main Components

DEs support heartbeat, Remote Password Changing (RPC), and discovery. A DE is composed of site connectors, sites, and engines:

  • A distributedengine is a Windows service that does the actual work, such as password changing, heartbeat, discovery, and more. Each engine belongs to a site.
  • A site can be thought of as a bucket of work items for a particular network area. Each engine is assigned to a single site, but each site can include multiple engines, significantly increasing throughput.
  • A site connector is a Windows service that holds the work items for a number of sites. The site connector can be either RabbitMQ or MemoryMQ (a built-in service developed by IBM Security). Each site can only be assigned to a single site connector, but you can have multiple site connectors running on separate machines, each storing work items for multiple sites. Those sites, in turn, distribute the work items among multiple engines. The ability to add new Site Connectors, Sites, and Engines as needed makes Distributed Engine a highly-scalable solution.
For the highest scalability and reliability, IBM Security recommends using RabbitMQ. MemoryMQ is an easier but less capable alternative and can be used for trials and proof of concepts but should not be used for production environments. Two exceptions are very small deployments and customers that do not use open-source software for compliance reasons.

Figure: Distributed Engine Components

image-20200601134202909

The above diagram is a simplified, conceptual one, not a network diagram. It does not show a callback port from the DE to Verify Privilege Vault. DEs require either an HTTPS or TCP port to communicate with Verify Privilege Vault for initial activation, updates, and continuous periodic check of site and site connector settings.

Ports

DEs have two configurable ports: one for connecting to the site connector, and one for the engine to retrieve configuration information from Verify Privilege Vault at regular intervals. The callback port from an engine to Verify Privilege Vault can be configured to contact the website directly over HTTP, HTTPS, or TCP. HTTP and HTTPS connections use the existing IIS port bindings. All connections are outbound—no inbound connections are made from Verify Privilege Vault or the site connectors to the remote networks.

Default ports:

  • RabbitMQ: 5672 (non-SSL), 5671 (SSL)

  • MemoryMQ: 8672 (non-SSL), 8671 (SSL)

  • Verify Privilege Vault: existing IP address bindings or custom port over TCP. We reserve one port for legacy upgrades, usually port 9999.

  • Verify Privilege Vault Cloud:

    • 443 (Web sockets—the default)
    • 5671 and 5672 (AMQP)
    These ports are used for outbound traffic for engines to communicate with SSC instances. They are set by the "Azure ServiceBus Transport Type" global engine setting.

Security

Distributed engines have multiple security layers:

  • Engines must be approved within Verify Privilege Vault before they will be given access to a site.
  • Work items are encrypted with a site-specific symmetric key prior to sending them to the site connector.
  • Communication to the site connector supports SSL and TLS.
  • Direct communication from engine to Verify Privilege Vault uses a public-private key exchange.
  • The engine configuration file is DPAPI encrypted.

For more information about DE security, see the Distributed Engine Hardening.

Engine Workflow

When an engine Windows service starts, the following steps occur:

  1. The service contacts Verify Privilege Vault directly using the engine callback port.

  2. The service receives configuration information for the site connector to connect to and what site to process work items for.

  3. The service connects to the site connector and registers with the site for work item processing.

  4. The service fetches a work item from the site.

  5. The service processes the work item.

  6. The service gives the site the result of the processing, such as heartbeat success or discovery results.

  7. The service fetches another work item, and the process continues.

Configuring Distributed Engines

When DEs auto update, they remove the MySQL, Oracle, and other DLLs that were manually placed there. To forestall this, we recommend creating an ignore file for DE upgrades. Please see How to create an ignore file for Distributed Engine upgrades for details.

Configuration

Below is a summary of the steps required to configure DEs:

  1. Enable the DE and specify the engine callback settings.

  2. Configure and Install the site connector.

    • If you plan to use RabbitMQ (recommended), please see Installing RabbitMQ. You can find general information on using Verify Privilege RabbitMQ Helper to install RabbitMQ can be found in Installing Rabbit MQ.

    • If you plan to use MemoryMQ, create the site connector record within Verify Privilege Vault then click the Download Site Connector Installer button to get the MSI. Run the MSI on the desired host.

  3. Setup sites.

  4. Configure default DE settings. See Engine Settings.

  5. Install engines.

  6. Assign secrets to sites. Secrets can be assigned to a site through their Remote Password Changing tab or via a bulk operation on the Verify Privilege Vault dashboard. Once assigned to a site, all heartbeat or password changing operations take place through that site.

  7. Assign discovery sources to sites. To run discovery through a site, edit the discovery source and assign the site. Once assigned, all discovery operations for that discovery source take place through that site.

Discovery should (a) have a dedicated DE and/or (b) a dedicated Site.

Engine Settings

  1. If you click a site name, there is a Default Engine Settings button that allows you to pick which features of apply to new DEs connected to this Site:

  2. Click the Default Engine Settings button.

  3. When a new DE is installed, it inherits these settings from the site it is attached to. You can further customize engines by going to Show Settings in the Kebab menu for the engine.

  4. FIPS Compliance is the only setting only available at the site level, and all DEs on the site use the same value. You cannot have a mix FIPS and non-FIPS engines on a single site. If you toggle the FIPS Compliance setting, Windows OS FIPS setting needs to be set to match it, and the engine server should be rebooted. The next time an engine checks in, if the engine’s settings have been changed from those set on the site, the engine reloads itself and starts only performing the site's selected duties.

FAQ

What happens if Verify Privilege Vault sends work items to the site connector, but no engines are running to consume them?

Work items continue to build up in the site connector until a limit is reached. Heartbeat work items have a Time To Live (TTL) of 5 minutes, Password Changing work items have a TTL of 20 minutes. Expired work items are thrown away and will not be processed. Once a heartbeat or password changing work item is sent to the site connector, Verify Privilege Vault will not send the same work item to the queue until 5 minutes after the TTL is up (10 and 25 minutes for heartbeat and password changing, respectively). This prevents multiple pending heartbeat or password changing work items for the same secret at the same time.

How many Sites can a Site Connector hold?

MemoryMQ supports up to 100. RabbitMQ supports up to 200.

Can I cluster Site Connectors?

RabbitMQ supports clustering, MemoryMQ does not.

Can I use both RabbitMQ and MemoryMQ?

Yes. You can have as many site connectors, of either type, installed as needed. Note that while you can have both RabbitMQ and MemoryMQ installed on a single machine, you cannot have two RabbitMQ instances or two MemoryMQ instances on the same machine.

Can I convert a site connector from MemoryMQ to RabbitMQ or vice versa?

Yes. You can install the new site connector, swap the sites over to the new service, and then decommission the old site connector.

I've reinstalled my Distributed engine, but why is it not showing in the console?

If you have distributed engines installed and encounter an error or decide to reinstall distributed engine on a machine that had it previously, please be aware that the Add/Remove Programs option does not clear out the installation folder of the Distributed Engine (C:\Program Files\Delinea Software Ltd\Distributed Engine\)

This leaves behind some configuration files. If these are left behind during an uninstall, then during a reinstall it may cause issues where the distributed engine will not appear in the console for activation or assignment to a site.