Troubleshooting Tips and Tools

This chapter describes how to use diagnostic tools and log files to retrieve information about the operation of IBM Security Agents and provides tips to help you identify and correct problems on managed computers.

Addressing Log On Failures

In most cases, valid Active Directory users should be able to log on to computers where you have deployed the agent without any configuration. If an attempt to log on fails, the problem is typically caused by one of the following:

  • Users attempting to log on to a computer they are not authorized to use.
  • Users do not have a valid Active Directory user account in the appropriate forest.
  • Users have typed their non-Active Directory password or typed the wrong password more times than allowed.

If users report that they cannot access computer resources they think they should have access to, take the following steps to troubleshoot the problem:

  1. Verify that the user has an Active Directory user account in the forest or in a forest with a two-way trust relationship.

  2. Check that the account is not disabled or locked out because of repeated log-on failures.

  3. Verify that there is an Active Directory domain controller available and that the computer a user is unable to log on to can connect to it and open a communication channel.

    For example, log on to the UNIX computer using a locally authenticated user, and run the ping command with the name of a domain controller in the forest. If the command receives a reply from the domain controller, the DNS service is functioning and the local computer is able to locate the domain controller on the network.

    If the ping command does not generate a reply, check your DNS configuration and check whether the local computer or the domain controller is disconnected from the network.

  4. Use adinfo or Active Directory Users and Computers to check that the computer is joined to the domain.

  5. Use adinfo to check whether the agent is currently running or disconnected.

    If the adinfo command reports the mode is disconnected, try restarting adclient and testing network response time. On a slow network, adclient may drop the connection to Active Directory if there is a long delay in response time.

    If the adinfo displays an "unavailable" error, try running adleave to leave Active Directory, re-run the adjoin command to re-join the domain. If a problem still exists, check the DNS host name of the local computer and the domain controller, the user name joining the domain, and the domain name you are using.

  6. Check the clock synchronization between the local computer and the Active Directory domain controller.

    If the clocks are not synchronized, reset the system clock on the managed computer using the date command.

  7. Check the contents of the system log files or the centrifydc.log file after the user attempts to log on. You can use information in this file to help determine whether the issue is with the configuration of the software or with the user’s account.

  8. Check for conflicts between local user accounts and the user profile generated by the agent.

If these steps do not reveal the problem, you can enable detailed logging of adclient activity using the addebug command. You can use the information in the /var/log/centrifydc.log file to further diagnose the problem or to provide information to IBM Security Support.

Understanding Diagnostic Tools and Log Files

The agent includes some basic diagnostic tools and a comprehensive logging mechanism to help you trace the source of problems if they occur. These diagnostic tools and log files allow you to periodically check your environment and view information about agent operation, Active Directory connections, and the configuration settings for individual computers you manage.

Logging is not enabled by default for performance reasons. Once enabled, however, log files provide a detailed record of agent activity. This information can be used to analyze the behavior of adclient and communication with Active Directory to locate points of failure. However, log files and other diagnostic tools provide an internal view of operation and can be difficult to interpret. The log files are primarily intended for IBM Security Support and technical staff.

In most cases, you should only enable logging when you need to troubleshoot unexpected behavior, authentication failures, or problems with connecting to Active Directory or when requested to do so by IBM Security Support. Other troubleshooting tools, such as command line programs, can be used at any time to collect or display information about your environment.

Configuring Logging

By default, the agent logs errors, warnings and informational messages in the syslog and /var/log/messages files along with other kernel and program messages. Although these files contain valuable information for tracking system operations and troubleshooting issues, occasionally you may find it useful to activate IBM Security-specific logging and record that information in a log file.

Enabling Logging for the Agent

To enable logging on the agent:

  1. Log in as or switch to the root user.

  2. Run the addebug command:

    /usr/share/centrifydc/bin/addebug on

    You must type the full path to the command because addebug is not included in the path by default.

    After you run this command, all of the agent activity is written to the /var/log/centrifydc.logfile. If the adclient process stops running while you have logging on, the addebug program records messages from PAM and NSS requests in the /var/centrifydc/centrify_client.log file. Therefore, you should also check that file location if you enable logging.

For performance and security reasons, you should only enable logging when necessary. For example, if you open a case with IBM Security Support, the Support representative may request that you enable logging and submit log files to investigate your case. You should also limit logging to short periods of time while you or IBM Security Support attempt to diagnose a problem. You should keep in mind that sensitive information may be written to this file and you should evaluate the contents of the file before giving others access to it.

When you are ready to stop logging activity, run the addebug off command.

Setting the Logging Level

You can define the level of detail written to the log by setting the log configuration parameter in the centrifydc.conf configuration file:

log: level

With this parameter, the log level works as a filter to define the type of information you are interested in and ensure that only the messages that meet the criteria are written to the log. For example, if you want to see warning and error messages but not informational messages, you can change the log level from INFO to WARN. By changing the log level, you can reduce the number of messages included in the log and record only messages that indicate a problem. Conversely, if you want to see more detail about system activity, you can change the log level to INFO or DEBUG to log information about operations that do not generate any warnings or errors.

You can use the following keywords to specify the type of information you want to record in the log file:

Specify this level To log this type of information
FATAL Fatal error messages that indicate a system failure or other severe, critical event. In addition to being recorded in the system log, this type of message is typically written to the user’s console. With this setting, only the most severe problems generate log file messages.
ERROR System error messages for problems that may require operator intervention or from which system recovery is not likely. With this setting, both fatal and less-severe error events generate log file messages.
WARN Warning messages that indicate an undesirable condition or describe a problem from which system recovery is likely. With this setting, warnings, errors, and fatal events generate log file messages.
INFO Informational messages that describe operational status or provide event notification.

Logging Details for a Specific Component

By default, when you specify a logging level, it applies to all of the agent components that log activity. The logging system, however, provides a hierarchical organization of logical log names for the components within the agent and each of these logical logs can be configured to provide more targeted analysis of it specific operations. For example, if you set your base logging level to only report serious errors but you want to see informational, warning, and error messages for adclient, you can add a separate logging level parameter for the log messages generated by adclient:

# Use the following setting to set the base level of detail  
# for logging to record Error messages:  
 log: ERROR  
  
# Add the name of the adclient logical log and specify the
# logging level to use for it and its children:  
 log.com.centrify.adclient: INFO

Logging to the Circular In-memory Buffer

If the adclient process is interrupted or stops unexpectedly, a separate watchdog process (cdcwatch) automatically enables an in-memory circular buffer that writes log messages passed to the logging subsystem to help identify what operation the adclient process was performing when the problem occurred. The in-memory buffer is also mapped to an actual file, so that if there is a system crash or a core dump, the last messages leading up to the event are saved. Messages from the in-memory circular buffer have the prefix _cbuf, so they can be extracted from a core file using the strings command.

The in-memory circular buffer allows debug-level information to be automatically written to a log file even if debugging is turned off. It can be manually enabled by restarting the adclient process with the -M command line option. The default size of the buffer is 128K, which should be sufficient to log approximately 500 messages. Because enabling the buffer can impact performance, you should not manually enable the circular buffer or modify its size or logging level unless you are instructed to make the changes by IBM Security Support.

Collecting Diagnostic Information

You can use the adinfo command to display or collect detailed diagnostic and configuration information for a local computer. Options control the type of information and level of detail displayed or collected. The options you are most likely to use to collect diagnostic information are the --config, --diag, or --support options, which require you to be logged in as root. You can redirect the output from any adinfo command to a file for further analysis or to forward information to IBM Security Support.

For more information about the options available and the information returned with each option, see the adinfo man page.

To display the basic configuration information for the local computer, you can type:

adinfo

If the computer has joined a domain, this command displays information similar to the following:

Local host name: magnolia  
 Joined to domain: ajax.org  
 Joined as: magnolia.ajax.org  
 Current DC: ginger.ajax.org  
 Preferred site: Default-First-Site-Name  
 Zone: Auto Zone  
 Last password set: 2014-04-01 14:47:57 PST  
 CentrifyDC mode: connected  
 Licensed Features Disabled

Resolving Domain Name Service (DNS) issues

In some cases, you may encounter problems with authentication, authorization, or lookup requests because of your DNS configuration. The most common scenarios are:

  • The Windows DNS server role is not configured to dynamically update service locator (SRV) records. These records enable Active Directory to find the nearest domain controller, Key Distribution Center (KDC), and Global Catalog (GC) for the site.
  • The DNS servers do not publish the SRV records for the domain controllers that provide Active Directory service to the enterprise. These records must be available for computers to connect to Active Directory and locate required services.
  • The DNS servers for the enterprise run on UNIX servers that are not configured to locate Active Directory domain controllers. In many cases, DNS servers for an enterprise are configured with a different domain namespace than Active Directory or Active Directory domain controllers are considered internal servers and not registered in the enterprise DNS.

If you encounter problems, you should contact your Active Directory administrator to determine whether the DNS server role is being used and if it is configured to allow dynamic updates. If the Active Directory DNS server role is not being used to provide DNS to the enterprise, you should contact the DNS administrator to resolve the issue.

There are several possible scenarios:

  • If the enterprise uses UNIX-based DNS servers instead of Active Directory-based DNS servers and DHCP, computers should have a snameserver entry in /etc/resolv.conf file that points to a valid DNS server.
  • Forward and reverse lookup zones should be configured to allow enterprise DNS servers to locate Active Directory domain controllers.
  • If the Active Directory domain namespace is different from the namespace registered in enterprise DNS servers, you should use the --name and --alias join option to resolve the namespace differences.
  • If the enterprise DNS servers do not include records for Active Directory domain controllers, you can manually set the location of the Active Directory domain controller using parameters in the centrifydc.conf configuration file.