Troubleshooting and Common Questions

IBM Security software includes diagnostic tools and log files to help you trace the source of problems if they occur. Diagnostic reports and log files allow you to periodically check for issues and view information about operations on the computers you manage. The information is useful for troubleshooting and in resolving cases with the help of IBM Security Support.

This chapter describes how to find log files, set the level of detail recorded in log files, and use diagnostic tools to retrieve information about the operation of the Agent and Verify Privilege Server Suite components. This chapter also covers common questions to help you identify and correct problems on the computers you manage.

Solving Problems with Logging On

After you have installed the Agent for Windows and joined the computer to a domain, users cannot log on without a role assignment. The role, however, can be assigned to a local account or a domain account, or the role can be assigned the right to access a remote computer. Consequently, users might encounter problems logging on after the agent is deployed. For example, you might find that users can log on to the computer using a local account but cannot log on using their domain account or have trouble connecting to a remote server.

If users report problems logging on, there are some things you can try to troubleshoot the issue:

Check the logon rights for the affected users.

To do this, log on as an administrator and execute dzinfo user-name (where user-name is the name of the user experiencing problems logging on). You can also check user logon rights using the Authorization Center.
Try to log on using a local user account or using a different domain account if you have more than one account available.
Determine whether the computer you are using is connected or disconnected from the network. In rare cases, authorization information might not be available when a computer disconnected from the network.
If users cannot log on to a remote computer, confirm that they have a role that has the remote logon system right and that the computer itself is configured to allow users to log on remotely. Open the Authorization Center to review the list of roles and their associated rights for any user.
Check the computer’s local security policy or applied group policies to verify whether the user is allowed to log on interactively or through a remote desktop connection. For example, most domain users are not allowed to log on locally on domain controllers.

Depending on how your organization has configured native Windows security policies, users might need to be members of a specific Windows security group—such as Server Operators or Remote Desktop Users—to log on to specific computers locally or remotely even if they have been granted access rights using the Windows Login role or a custom role definition.
Check to see whether the computer is in Rescue mode.

In Rescue mode, access to a computer is granted only to users who have Rescue rights. For information about adding Rescue rights to a role, seeSystem rights allow users to log on. In general, a computer enters Rescue mode because the Windows agent authorization service has stopped. Possible causes include the following:
- The computer is not connected and the local authorization cache has not been initialized or is corrupt.
- The local authorization cache cannot be updated because the file system is full.
```
See Working with the authorization cache on managed computers for more
information about the authorization cache and the conditions under which
a computer is considered to be not connected.
```

Accessing Network Computers with Privileges

Depending on how you have defined the roles users are assigned, it is possible for users to see potentially misleading information in certain applications or be unable to perform the administrative tasks as they expect. For example, if users select a role with administrative privileges to access an application such as SQL Server Configuration Manager or Microsoft SQL Server Management Studio and connect to a remote SQL Server instances, it might appear as if they have permission to start and stop services or perform other tasks. However, if the role does not include network access rights for the remote SQL Server instance, users will not have the appropriate permission to perform those tasks.

You can check whether the selected role includes network access rights using the Authorization Center. If the role being used does not include network access rights, check whether the user has additional network roles available to use in conjunction with the local role. If the role being used includes network access rights, you should check whether those rights are applicable on the network computer the user is attempting to manage. Users must be assigned to the role that has network access rights on the remote server.

Refreshing Cached Information on Managed Computers

Authorization information is cached on the local computer to improve performance and to allow the use of elevated privileges even if users are disconnected from the network. If you make changes to rights, role definitions, or role assignments, you can refresh the information stored in the cache on managed computers to ensure the agent has the most up-to-date information about current rights and roles. If users are experiencing authorization problems or issues with their access rights (for example, if the management console shows that a user has logon rights, but dzinfo or the authorization center does not show that the user has logon rights), you should try refreshing the cache to make sure any changes you have made take effect.

You can refresh the cache using agent configuration panel or the dzrefresh command line program in a Command Prompt window if you have the appropriate permissions.

Analyzing Information in Active Directory

One important way you can troubleshoot your environment is by running the Analyze command. The Analyze command enables you to selectively check the integrity of information stored in Active Directory. With the Analyze wizard, you can check for a variety of potential problems, such as empty zones, invalid role assignments, or orphaned role assignments.

Note: When you run the Analyze command, only the zones that are open are checked.

To check for problems in the Active Directory forest:

Open Access Manager.

If you are prompted to connect to a forest, specify the forest domain or domain controller to which you want to connect.
Select the root node, right-click, then click Analyze.
Select the types of checks you want to perform, then click Next to generate the report.

You can select All to perform a complete check of the Active Directory forest. However, some of the analysis options are only applicable for Linux and UNIX computers or UNIX user and group profiles. For more information about any analysis option, see the Access Manager help or theAdministrator’s Guide for Linux and UNIX.
Review the result summary, then click Finish.
If the result summary indicates any issues, you can view the details by selecting Analysis Results in the console tree and viewing the information listed in the right pane.
Select individual warnings or errors, right-click, then select Properties for additional information.

Common Scenarios that Generate Errors and Warnings

For most organizations, it is appropriate to check the data integrity of the Active Directory forest on a regular basis. Although running the Analyze command frequently may not be necessary for small networks with few domain controllers, there are several common scenarios that you should consider to determine how often you should check the forest for potential problems.

The most likely reasons for data integrity issues stem from:

Multiple administrators performing concurrent operations.
Administrators using different domain controllers to perform a single operation.
Replication delays that allow duplicate or conflicting information to be saved in Active Directory.
Insufficient permissions that prevent an operation from being successfully completed.
Network problems that prevent an operation from being successfully completed.
Partial or incomplete upgrades that result in inconsistency of the information stored in Active Directory.
Using scripts or ADSI Edit rather than the console to create, modify, or delete objects in Active Directory, which may lead to corrupted or invalid information.

Running Analyze periodically helps to ensure that the scenarios that can cause problems are reported in the Analysis Results, enabling you to take corrective action.

Responding to Errors and Warnings

Depending on the type of warning or error generated in the Analysis Results, you might be able to take corrective action or access additional information. For example, if a computer account lacks the necessary permission to update Active Directory with the agent version it has currently installed, the Analysis Result will enable you to update the computer’s account permissions to allow changes to that attribute.

To review additional information or take corrective action, select the error or warning in the list of Analysis Results after running the Analyze wizard, right-click, then select Properties. For more information about responding to analysis results, see the Access Manager help or the Administrator’s Guide for Linux and UNIX.

Running Diagnostics and Viewing Logs for the Agent

The Agent for Windows provides logging and diagnostic services. If you have administrative access on a local computer, you can generate diagnostic information about the operation of the Agent for Windows and view and save the current content of the log file from the agent configuration panel. For example, you can generate diagnostic information about user sessions, user roles, desktops, and elevated account access, as well as detailed information about auditing from the agent configuration panel.

There are three different types of diagnostics information available:

Audit & Monitoring Service provides the diagnostic information

related to the auditing and monitoring service.
Identity Platform provides the diagnostic information related

to Privileged Access Service, such as for MFA. This diagnostics tool runs

the following tests:
- Agent Service Connectivity Check: Checks to see if the agent is in
  service, and if the agent is running in a normal state. Also determines
  whether the agent is in a zone, or is configured to use zoneless mode.
- Connector Connectivity Check: Determines whether all
  connectors in the network can be connected properly.
  whether the certificates (IWA and cloud) have been installed properly.
  Also determines whether the agent can be connected without a trusted
  certificate problem.
- Identity Platform Connectivity Check: Determines whether a
  connection to the cloud tenant is functional. Checks for problems with
  DNS, the firewall, and proxy server settings.
- MFA Configuration Check: Determines whether the local computer has
  been configured properly. If the computer is in a zone, the test also
  checks whether MFA complies with the configuration defined in the zone.
- MFA Role and Permission Check: Verifies whether role permissions are set properly in the Privileged Access Service Admin Portal.
- Offline MFA Provisioning Check: Determines if the computer has been configured with an offline MFA profile or not.
Privilege Elevation Service provides the diagnostic information

related to privilege management.

You can view these diagnostics tools either from the Windows system tray or from the agent configuration panel.

To view diagnostics from the Windows system tray:

Log on to a computer where the Agent for Windows is installed.
In the Windows system tray, right-click the IBM Security icon and click

Troubleshooting, then select the service for which you want to view

diagnostic information (your options may vary depending on what services are

enabled on the computer):
- Audit & Monitoring Service opens a dialog box with a text-based summary of diagnostic auditing and monitoring information.
- Identity Platform runs a series of connectivity tests and lists out the results of each test.
- Privilege Elevation Service opens a dialog box with a text-based summary of diagnostic privilege elevation information.

To generate diagnostics or view the log file from the agent configuration panel:

Log on to a computer where the Agent for Windows is installed.
In the list of applications on the Windows Start menu, click Agent Configuration to open the agent configuration panel.
Select the service for which you want to view information:
- Audit & Monitoring Service opens a dialog box with a text-based summary of diagnostic auditing and monitoring information.
- Identity Platform runs a series of connectivity tests and lists out the results of each test.
- Privilege Elevation Service opens a dialog box with a text-based summary of diagnostic privilege elevation information.
Click Settings.
Click the Troubleshooting tab.
Click Diagnostics to generate diagnostic information.
Select the Diagnostic Information displayed, right-click, then select

Copy to copy and paste the output to a file for further analysis.
Click View Log to display the current log file for the local agent.
Click Options to see or change the location of the log file or the level

of detail recorded in the log file.

Sample Diagnostic Report

For example, if you are viewing information about the privilege elevation service, the diagnostic report might be similar to this:

Product: Infrastructure Services (Name and Version information)
Computer: DC2008R2-LG
Joined Domain: finsterwald.org
Zone: finsterwald.org/Acme Pubs/Zones/HeadquartersAgent State: Connected
Time: 2017-10-16 12:38:03.620 -08:00
Session information:
Session 1
SAM Name: FINSTERWALD\anton.splieth
Logon Type: Console
Always Audit: Yes
Desktops:
Default
GUID: de1dd94a-b671-4b37-baa4-9b2c1b70e776
DZ Logon Id: (0x0)
Local Role: Self
Network Roles: Self
Always Audit: Yes
Audit Flag: On
UAC Restrictions: No
SQL-DBA
GUID: fccb2382-3800-4f3c-9569-922048f91375
DZ Logon Id: (0x9ba99)
Local Role: SQL-DBA/Headquarters
Network Roles: Self
Always Audit: Yes
Audit Flag: On
UAC Restrictions: No
Network Drives: No

Logon information:
Logon ID (0x9ba99)
Logon GUID: 38407dd1-0165-458e-b45d-686a07e87805
Base Logon ID: (0x77163)
Base SAM Name: FINSTERWALD\anton.splieth
ElevatedAccount: (ElevatedSelfAccount, AdditionalGroups=(count=1, items=(S-1-5-32-544)))
Local Role: SQL-DBA/Headquarters
Network Roles: None
Should Audit: Yes
Logon ID (0x22bfee)
Logon GUID: 1b50b739-461c-410e-803c-ed52d4ba1e80
Base Logon ID: (0x77163)
Base SAM Name: FINSTERWALD\anton.splieth
ElevatedAccount: (ElevatedSelfAccount, AdditionalGroups=(count=1, items=(S-1-5-32-544)))
Local Role: SQL-DBA/Headquarters
Network Roles: None
Should Audit: Yes

Domain last access information:
Forest finsterwald.org: Connected
Domains: finsterwald.org: Connected
Multi-factor Authentication information: None

Done.

Enabling Detailed Logging for Audit and Monitoring Service Components

In addition to the log files for the Agent for Windows, there are log files for other audit and monitoring service components to record information about operations performed by those components on a local computer. If you have audit and monitoring service components installed, you can view the log files or change log file options for those components to assist IBM Security Support when troubleshooting issues.

Enabling Detailed Logging for an Audited Computer

If you are troubleshooting an audit and monitoring service-related issue, you should enable detailed logging for the audit and monitoring service on the computers being audited. For Windows computers, you can enable detailed logging using the agent configuration panel.

To enable detailed logging on an audited computer:

Log on to an audited computer.
In the list of applications on the Windows Start menu, click Agent Configuration to open the agent configuration panel.
Click Audit & Monitoring Service.
Click Settings.
Click the Troubleshooting tab.
Click Options, change the logging level to Trace messages, then click OK.
Note the log folder location or click Browse to specify a different location for the log file, then click OK.
Click View Log to view the current log file.

From the log file window, you can also click File > Save As to save the log file.
Send an email to IBM Security Support with the log file from the location specified in Step 7 as an attachment.
Click Options, change the logging level back to its default setting of Informational messages, then click OK.
Click Close to return to the agent configuration panel.

Enabling Detailed Logging for the Collector Service

If you are troubleshooting an audit and monitoring service-related issue, you should enable detailed logging for the collector service on the computers where the collector service runs.

To enable detailed logging on a collector:

In the list of applications on the Windows Start menu, click Audit Collector Control Panel to open the audit collector control panel.
Click the Troubleshooting tab.
Click Options, change the logging level to Trace messages, then click Apply.
Note the log folder location or click Browse to specify a different location for the log file, then click OK.
Click View Log to view the current log file.

From the log file window, you can also click File > Save As to save the log file.
Send an email to IBM Security Support with the log file from the location specified in Step 4 as an attachment.
Click Options, change the logging level back to its default setting of Informational messages, then click OK.
Click Close to return to the Collector Control Panel.

Enabling Detailed Logging for Audit and Monitoring Service Consoles

In most cases, troubleshooting audit and monitoring service-related issues requires information about the operation of the agent and the collector or database activity. However, in some cases, it might be necessary to capture detailed information about the operation of Audit Manager or Audit Analyzer.

To capture detailed information for Audit Manager or Audit Analyzer:

Log on to a computer where the Audit Manager or Audit Analyzer console is installed.
In the list of applications on the Windows Start menu, click Agent Configuration to open the agent configuration panel.
Click Audit & Monitoring Service.
Click Settings.
Click the Troubleshooting tab.
Click Options.
In the Log Settings tab, change the logging level to Trace messages, then click OK.
Note the log folder location or click Browse to specify a different location for the log file, then click OK.
Send an email to IBM Security Support with the log file from the location specified in Step 8 as an attachment.
Click Options, change the logging level back to its default setting of

Warning messages, then click OK.
Click Close to return to the agent configuration panel.

Enabling Audit and Monitoring Service Performance Counters for the Collector

If you have enabled audit and monitoring service and installed the collector service on a local Windows computer, you can add audit-specific performance counters to Performance Monitor to help you analyze and resolve audit-related issues. When you install the collector, the performance counters are added automatically. When you uninstall the collector, the counters are automatically removed from Performance Monitor.

For more information about troubleshooting in an audit installation, see the Auditing Administrator’s Guide.

Tracking Database Activity

Database traces are used to help diagnose problems in the management database or audit store databases. For example, database traces can help to identify inconsistencies caused by hardware errors or network interruptions. After you enable database tracing, DirectManage Audit tracks all of the SQL statements and debug messages from the audit management database or audit store, and records the information in the database server.

Note: Tracing database operations affects database performance. You should only activate a database trace if you require this information for troubleshooting. Before you start a database trace, try to reduce the load on the database instance as much as possible, then only perform the actions needed to reproduce the issue you are troubleshooting. Turn off database tracing as soon as you have logged the activity you need for the analysis of database operations. The trace for each database can take up to 800MB of server disk space. After you turn off database tracing, restart the SQL Server instance to reset the disk space.

Starting a Database Trace

You can start a database trace for a management database or an audit store database.

To start database tracing:

Open Audit Manager.
Select an installation name, right-click, then click Properties.
Click the Database Trace tab.

This tab displays basic information about the management databases and audit store databases for the selected installation. In the Trace Status column, you can see whether tracing is enabled or disabled for each database.
Select a management or audit store database in the list, then click Enable to start tracing on the database selected.
Click OK, then perform the database actions for which you want to capture information.

Stopping the Database Trace

You should turn off database tracing immediately after you have logged the activity you need for the analysis of database operations.

To stop database tracing:

Open Audit Manager.
Select the installation name, right-click, then click Properties.
Click the Database Trace tab.
Select the management or audit store database that has tracing enabled, then

click Disable to stop tracing on the database selected.
Click Export to save the database trace from the selected databases to a

file with comma-separated values (.csv).
Follow the prompts displayed in the Export Database Trace wizard to save the

information to a file.

Exporting the Database Trace for a Management Database

The Export Database Trace wizard prompts you for different information depending on whether the database trace is for a management database or an audit store database. For example, if you generate a database trace for a management database then click Export, the Export Database Trace wizard prompts you for user accounts.

To export the database trace:

Select a start date and time for the From filter and an end date and time for the To filter, then click Next.
Click Add to search for and select users, then click Next.

By default, you can search for users in the entire directory, you can click Object Types or Locations to change the scope of the search scope, or click Advanced specify other criteria.
Accept the default folder location or click Browse to select a different location, then click Next.
Review your selections, then click Next.

By default, the wizard save the file as installation_name.csv and opens the file location.
Click Finish, then click OK to close the installation properties.

Exporting the Database Trace for Audit Store Databases

When you select an audit store from the lower area of the Database Trace tab on the Properties page and click the lower Export button, the wizard opens with a date/time Export Criteria page. On the second page, the wizard asks you to pick the domain and computer.

To export the database trace:

Select a start date and time for the From filter and an end date and time for the To filter, then click Next.
Click Add to search for and select collectors, then click Next.

By default, you can search for computers in the entire directory, you can click Object Types or Locations to change the scope of the search scope, or click Advanced specify other criteria.
Click Add to search for and select management database computers, then click Next.
Accept the default folder location or click Browse to select a different location, then click Next.
Review your selections, then click Next.

By default, the wizard save the file as audit_store_name.csv and opens the file location.
Click Finish, then click OK to close the installation properties.

Delegating Database Trace Management

You can delegate the authority to manage database tracing by granting the Manage Database Trace permission to other users for a management database or an audit store database.

Controlling Audit Trail Events

By default, audit trail events are recorded when users log on, open applications, select roles that elevate their privileges, and perform other tasks. You can use domain group policies to control the global location of the audit trail events. For example, you might want to store audit trail events in the audit store database instead of the Windows event Application log if you want to make them available for querying and reports.

You can also override domain group policy and configure local or category-specific audit trail targets using a local administrative template or group policy.

To configure global or per-category audit trail targets using an ADM administrative template:

Note: These settings override the settings defined in the Set global audit trail targets group policy.

Open the Group Policy Object Editor to display Local Computer Policy, and

select Computer Configuration > Administrative Templates.
Right-click, select Add/Remove Templates, then click Add.
Navigate to the AuditManager folder, select auditrail.adm, click OK,

then click Close.
Open the Classic Administrative Templates folder and select AuditTrail.
Specify global or separate targets for audit trail events:
- Enable Set global audit trail target settings to configure a single location for audit trail events for Access Manager and the Agents.
- If you want to have separate targets for audit trail events, you can
  enable the other audit trail group policies to override the global
  policy setting with a different target.
Specify the location for saving audit trail events, and then click OK:
- 0 to disable audit trail events
- 1 to store audit trail events in the audit store
- 2 to send audit trail events to the Windows event Application log
- 3 to send audit trail events to both the audit store and the Application log.

To configure per-category audit trail targets using a local group policy from an XML template:

Note: These settings override the settings defined in the Set global audit trail targets group policy.

Ensure that the Audit Trail Settings were updated with the most

recent XML template.
Open the Group Policy Object Editor to display Local Computer Policy,

and select Computer Configuration > Audit Trail Settings.
In Audit Trail Settings, separate folders for each audit trail
category contain Send audit trail to Audit database and Send audit trail to log file group policies. Enable these group policies in each
category that you want to configure to use a specific audit trail target.
The target that you specify for each category is used instead of the target
specified in the Set global audit trail targets group policy.

Summary of Audit Trail Events

Different components log different audit trail events. For example, the auditing and authorization services on a managed Windows computer track successful logon attempts and the use of Window access rights. Access Manager audit trail events record changes to the configuration of zones, such as the delegation of administrative tasks, the assignment of roles, and changes to the user and group profiles in a zone. For your reference, the following sections summarize the audit trail events recorded by Agents on managed Windows computers.

Additional audit trail events for Access Manager, Audit Analyzer, Audit Manager, and UNIX commands can be recorded in the target you specify for the audit trail. The event message provides detailed information about the operation performed or unsuccessfully attempted, including in most cases the reason the operation was unsuccessfully.

For a complete list of audit trail event identifiers and their corresponding descriptions, see the AuditTrailEvent.xml file provided in the Documentation folder. This file is generated directly from the underlying source code and provides the most up-to-date information about the events on which you can query and report.

Offline MFA Profile Authentication

In some environments, using an offline MFA profile for multi-factor authentication is not compatible with FIPS mode. See the Multi-factor Authentication Quick Start Guide for details about this restriction.

Authentication Service Known Issues

When troubleshooting, be aware of the following issues and constraints:

Import users and groups before importing the sudoers file (Ref: IN-90001).

Sudoers Import creates user roles but not the users. It is recommended that you import users and groups prior to importing the sudoers file. Otherwise, no sysRights are created for the users.
Pre-create computers before importing computer role from sudoers file (Ref: IN-90001).

The computers contained in the sudoers file must either be joined to a zone or pre-created.
Delegating zone administration permissions for SFU zones (Ref: IN-90001)

Delegate permissions to add, remove or modify users for SFU zone are not supported.
Users with rights to import user and groups into a zone also gain rights to modify profiles (Ref: IN-90001)

Any users who are given the right to "Import users and groups to zone" are automatically also given the right to "Modify user/group profiles".
Using domain local groups to manage resources (Ref: IN-90001)

Domain local groups can only be used to manage resources in the same domain as the group. So, for instance, a domain local group in domain A may be used to manage a computer in domain A but not one in domain B, despite a trust relationship between the two domains.
Domain local groups from other domains shown in search dialog (Ref: IN-90001)

When using the search dialog in the Access Manager to delegate zone control to a group, domain local groups from child domains will be shown incorrectly in the results and should be ignored. The search results when using the ADUC extension do not show these domain local groups.
Analyze forest and SFU zones (Ref: IN-90001)

The analyze forest feature in the Access Manager does not report empty zones or duplicated users or groups in a SFU zone.
Working with users that have more than one UNIX mapping (Ref: IN-90001)

Authentication Service supports Active Directory users that have more than one UNIX profile in a zone. However, if you are upgrading fromDirectControl 4.x or earlier and have existing users with more than one UNIX mapping, you should use DirectControl 5.0.0 or later to remove all but one of the UNIX profiles for each of these Active Directory users and then re-add them.

In addition, you should always use DirectControl console 5.0.0 or later when modifying these users.
In the Profile tab of the Properties page of a computer joined to a hierarchical zone, you cannot move this computer to a classic zone. Nor can you move it to a zone in another domain. There are no such limitations with a computer joined to a classic zone. (Ref: IN-90001)
Extra results when analyzing duplicate service principal names (Ref: IN-90001)

When running the Analyze / Duplicate Service Principal Names report, kadmin/changepw is incorrectly returned as a duplicate. The SPN is actually found multiple times, but this is by Microsoft design as it is the default account for the Key Distribution Center service in all domains.
Secondary groups not imported from XML files (Ref: IN-90009)

Using the Import Wizard to import user information from XML files does not import secondary group membership.