Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Tivoli Performance Tuning

Because your Tivoli environment becomes more complex as you add more Tivoli applications and systems managed by Tivoli software, you should revisit your initial hardware configuration to improve performance.

the Tivoli Enterprise Console server is CPU-bound and increasing the number or quality of CPUs will give the largest improvement in throughput. Total events per second at the server is the main consideration when calculating required hardware. The best way to save on processing power is to limit the number of incoming events before they reach the server.

The following tips are taken from a Tivoli Enterprise Console 3.9 performance report:

• The Tivoli Enterprise Console server is CPU-bound, which means the speed of the CPU directly impacts the event throughput of the server.
• Tivoli Enterprise Console can take advantage of multiprocessors. When deciding between deploying on a single 4-processor machine, or two 2-processor machines of equal speed (with the Tivoli Enterprise Console server on one, and RIM_host/RDBMS on another), note that there is a drop in throughput for the two machine environment due to network latency. For a simple or moderately complicated rulebase, choose a slower 2-processor machine over a faster 1-processor. When using a complex rulebase, consider a faster 1-processor machine because the rule engine will be the bottleneck.
• Use a 100 Mbit or better network connection between the server and RDBMS.
• Set Maximum number of event messages buffered in memory higher than the peak expected burst size. This prevents the server from doing added CPU and disk work if the buffer overflows.
• Tune the database bufferpools and check that the log spaces are of sufficient size. Database overflow can introduce latency into the event processing. Details on how to do this are discussed in Sample DB2 configuration updates for Tivoli Enterprise Console.
• Plan on additional CPU capacity for programs executed by the rulebase.
• Plan on an additional CPU capacity for excessive number of Java™ consoles or Web consoles. Both access the RDBMS directly and run extra RIM processes.
• The Tivoli Enterprise Console server is not memory-intensive. Two Gbytes is sufficient, even with RIM and the RDBMS on the same machine. More might be needed for complex rulebases and external programs executed by the rulebase.
• Event throughput is bound by the server speed. The number of gateways and endpoints has little effect. Therefore, total events per second is the major design criteria.

The following sections discuss options to consider.

The first thing is to run database on the same box as Tivoli. In this case recommendations for improvement of database performance and kernel tuning parameters are available from the corresponding database literature (Oracle or DB2).

In case database is on another box you need at least increase number of file descriptors  See  AIX file descriptors

Process tables limits (UNIX only)

One of the ways you can improve how the Tivoli environment runs on UNIX systems is to monitor the process tables for the Tivoli user and the entire system. Check to make sure that your system-wide process table is large enough, and then check that the per-user process limit is also high enough.

The following guidelines provide information about how to analyze the process tables (see your system documentation on how to run the sar, ulimit, and ps commands or other process management commands particular to your system):

1. While the Tivoli environment is running, check the overall number of processes.
2. Establish how many processes are running on the system.
3. Determine how many processes are running as the Tivoli user (the tmersrvd account on HPUX and the nobody account on AIX and Solaris).
4. Compare how many processes are currently running to how many processes are allowed to run. Make this comparison for both the system-wide process table and per-user process limit. When the limits are exceeded, you might need to change the number of processes allowed to run.

File table limits (UNIX only)

Another area where you can optimize performance for UNIX systems is in the configuration for the file table. Not only should you check the number of files open on the file system, but you should also check every socket connection that is open. When performing this check, run the netstat command to see how many socket connections are open. Also run the sar -v command (or your system's equivalent) to see file-handle-table statistics and process table statistics, among others. If your file table is approaching the limit, raise it and regenerate your kernel. Take into consideration your soft-file handle limit, if your operating system has both a hard and a soft limit. Set limits so that neither is approached.

Tuning AIX systems

• AIX file descriptors (ulimit)
   
  • Description: Specifies the various restrictions on resource usage on the user account. The ulimit -a command displays all the ulimit limits. The ulimit -a command specifies only the number of open files that are permitted. The default number of open files setting (2000) is typically sufficient for most applications. If the value set for this parameter is too low, errors might occur when opening files or establishing connections.
     
  • Because this value limits the number of file descriptors that a server process might open, a value that is too low prevents optimum performance.
     
  • How to view or set: Perform the following steps to change the open file limit to 10,000 files:
    1. Open the command window.
    1. Edit the /etc/security/limits file. Add the following lines to the user account that the WebSphere Application Server process runs on:

       nofiles =  10000 
       nofiles_hard = 10000

    2. Save the changes.
    3. Restart your AIX system.
    4. To verify the result, type the ulimit -a command on the command line. For example, type # ulimit -a.
        
  • Default value: For the AIX operating system, the default setting is 2000.
     
  • Recommended value: The value is application dependent. Increasing the ulimit file descriptor limits might improve performance. Increasing some of the other limits might be needed depending on your application.
    
    Any changes to the data or stack ulimits should ensure that data+stack < 256MB (for 32-bit WebSphere Application Server only).

Swap space used

Another performance problem can be found in the swap-space configuration. Check the amount of configured swap space, considering whether it is the primary or secondary device or file system. If you are using 90% to 100%, you need more swap space.

New processes, when spawned, need three types of memory allocated:

• Memory in physical RAM for code and data space
• Memory in swap space for code and data
• Memory in swap space reserved for code and data

Virtual memory and paging size (Windows only)

When you find that a Windows managed node or Tivoli server is running low on virtual memory or is hanging, you might need to increase the virtual memory on the machine. You also should consider increasing the total paging file size for the system.

Processing load for applications

When you have multiple Tivoli applications installed, you might have problems with the applications successfully completing their tasks, such as software distributions or inventory scans. It is important to properly schedule application tasks so that they do not process at the same time. This can greatly reduce resource problems in the Tivoli environment.

Physical RAM constraints

Tivoli Management Framework requires space to expand as your Tivoli environment becomes more complex. It needs space to spawn more processes (process and file tables, RAM, and swap space), create new threads (RAM, swap space, and file handles), and so on.

It is important to check for RAM usage. To improve performance in the Tivoli environment, try any of the following:

• Run a Tivoli desktop from a managed node instead of from the Tivoli server. Servicing desktops requires many threads and database activity.
• Eliminate unnecessary drivers from your kernel and eliminate unnecessary operating system processes. For example, if you do not intend to use a system as an NFS server, disable the NFS server subsystem.
• If you are booting or running Xterminal (xterm) from your Tivoli server, move it to a different system--xterm windows consume a large amount of RAM.

Performance considerations with endpoints

When you encounter problems with endpoint logins, there could be a problem with the Tivoli server being loaded down with too many requests. You can use additional throttling options to better distribute the load of endpoint logins being handled by the endpoint manager at one time. Use the wepmgr command with the set option to define attribute values, login_interval, max_install, max_sgp, max_after, and max_jobs. Setting these attributes assist you in throttling endpoint login requests.

Another consideration is how to implement endpoint policy. It is recommended that you use Perl scripts if pattern match searches are required. Perl is a compiled language, as contrasted with shell scripts, which are interpreted. Perl is therefore faster. In addition, while each command used in a shell script requires spawning of an entire child process (a resource-intensive activity), a compiled Perl script runs as one process, which makes it system-friendly. Perl is also much more portable across gateways, regardless of the interpreter type.

TCP Parameters for Tivoli

sg245392

• Important: By default, the tcp_keepinit value is 150 (75 seconds). We recommend you set the tcp_keepinit to around 40 (20 seconds) if the Endpoint Gateway machine plays only the role of Endpoint Gateway. This setting will improve the throughput of the distribution process in a large-scale environment. But, again, modify this value with caution.

Old News ;-)

Performance analysis report on Remote Management Agent and Tivoli Enterprise Console

Tivoli Enterprise Console server

The purpose of the testing was to verify the Tivoli Enterprise Console server's event processing capabilities and limitations. Like the Remote Management Agent server, the Tivoli Enterprise Console server is CPU-bound and increasing the number or quality of CPUs will give the largest improvement in throughput. Total events per second at the server is the main consideration when calculating required hardware. The best way to save on processing power is to limit the number of incoming events before they reach the server.

The following tips are taken from a Tivoli Enterprise Console 3.9 performance report:

• The Tivoli Enterprise Console server is CPU-bound, which means the speed of the CPU directly impacts the event throughput of the server.
• Tivoli Enterprise Console can take advantage of multiprocessors. When deciding between deploying on a single 4-processor machine, or two 2-processor machines of equal speed (with the Tivoli Enterprise Console server on one, and RIM_host/RDBMS on another), note that there is a drop in throughput for the two machine environment due to network latency. For a simple or moderately complicated rulebase, choose a slower 2-processor machine over a faster 1-processor. When using a complex rulebase, consider a faster 1-processor machine because the rule engine will be the bottleneck.
• Use a 100 Mbit or better network connection between the server and RDBMS.
• Set Maximum number of event messages buffered in memory higher than the peak expected burst size. This prevents the server from doing added CPU and disk work if the buffer overflows.
• Tune the database bufferpools and check that the log spaces are of sufficient size. Database overflow can introduce latency into the event processing. Details on how to do this are discussed in Sample DB2 configuration updates for Tivoli Enterprise Console.
• Plan on additional CPU capacity for programs executed by the rulebase.
• Plan on an additional CPU capacity for excessive number of Java™ consoles or Web consoles. Both access the RDBMS directly and run extra RIM processes.
• The Tivoli Enterprise Console server is not memory-intensive. Two Gbytes is sufficient, even with RIM and the RDBMS on the same machine. More might be needed for complex rulebases and external programs executed by the rulebase.
• Event throughput is bound by the server speed. The number of gateways and endpoints has little effect. Therefore, total events per second is the major design criteria.

Recommended  Links

Improving performance in a Tivoli environment

IBM Redbooks | AIX 5L Practical Performance Tools and Tuning Guide

• IBM developer kits for AIX, Java technology edition at http://www.ibm.com/developerworks/java/jdk/aix/service.html
   
• IBM developer kits - diagnosis documentation at http://www.ibm.com/developerworks/java/jdk/diagnosis/
   
• AIX Performance PMR Data Collection Tools at ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr/
   
• AIX 5L Performance Tools Handbook at http://publib-b.boulder.ibm.com/Redbooks.nsf/RedbookAbstracts/SG246039.html
   
• Understanding IBM eServer pSeries Performance and Sizing at http://publib-b.boulder.ibm.com/Redbooks.nsf/RedbookAbstracts/SG244810.html
   
• Fine-tuning Java garbage collection performance at http://www.ibm.com/developerworks/library/i-gctroub/
   
• Getting more memory in AIX for your Java applications at http://www.ibm.com/developerworks/eserver/articles/aix4java1.html
   
• AIX 5.2 performance tools update, Part 1 at http://www.ibm.com/developerworks/eserver/articles/Keung_AIXPerf.html
   
• AIX 5.2 Performance Tools update, Part 2 at http://www.ibm.com/developerworks/eserver/articles/AIX5.2PerfTools.html
   
• AIX 5.2 Performance Tools update: Part 3 at http://www.ibm.com/developerworks/eserver/articles/AIX5.2_performancetoolsupdatepart3.html