Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Tivoli Endpoint and Gateway Proxy

News Tivoli Management Framework (TMF) Recommended Links Documentation Tivoli Enterprise Console  
Troubleshooting Tivoli Framework Troubleshooting endpoints Gateway Troubleshooting Using Log Files to Troubleshoot Tivoli Environment Humor Etc

Tivoli Framework Security Toolbox (TFST) introduces the Endpoint and Gateway Proxy functions. Version is important for troubleshooting: old versions have memory leaks are unstable so upgrading is the first step you should do. Appendix A of Firewall Security Toolbox User's Guide covers troubleshooting

Here what they state:

Testing proxy configuration

When the components are started, they try to exchange signals, called a handshake. The parent component sends its children a who request. The children reply. Similarly, the children send their parents a tell message and the parents reply. These exchanges enable the components in a chain of communication to establish the labels of all the components in the chain.

When one of the components is not running, the handshake fails. A message in the log file of each component lists the component with which the handshake failed.

Because components do not take the same amount of time to start, the log file might record a failure. The order of startup does not matter, but the order in which components are started affects the messages in the log file.

To test the components, set the log level to 3. The messages in the following example assume that you start the gateway proxy first. In the gateway proxy log file, look for lines of the type:

01/12/18 17:38:06 3 161 routingManager: WHO command received [l=null]

If the gateway proxy has problems connecting to its parent (relay or endpoint proxy), the log file records a message. For example, on Windows 2000 operating systems, the entry is similar to the following:

01/12/18 17:43:07 1 130 ERROR multiplex.newsessopen: cannot open connection (-1)

To verify that the gateway proxy handshake reached its parent, check the log file on the parent machine for an entry similar to the following:

01/12/18 17:53:34 3 248 routingManager: WHO reply command received [l=ascotti_gwp1]

Do a similar check with the parent (endpoint proxy or relay). Start the parent component first. When the components communicate, the log file shows entries for each child similar to the following (for endpoint proxies, this log file is epp.log):

01/12/18 17:51:27 3 47 routingManager: TELL command received [l=ascotti_gwp1] 

If you do not see an entry similar to this, the components are not communicating, and the log file shows a message similar to the following:

01/12/18 17:43:07 1 130 ERROR multiplex.newsessopen: cannot open connection (-1)

To verify that the endpoint proxy or relay handshake reached its child, check the log file on the child machine for an entry similar to the following (for gateway proxies, this file is gwp.log):

01/12/18 17:40:16 3 179 routingManager: TELL reply command received

Debug each component individually to ensure that each is operating correctly. Do the following for each component:

  1. Stop the component.
  2. Restart the component you are testing and check the log file.

You can get version of with ./gwproxy -v

# ./gwproxy -v

Tivoli Firewall Security Toolbox ver. 1.3.1 - level 20061016A.

Interim fix ID = 1.3.1-TFS-0009.

Also you need to collect the following: (from the TMR server)

$DBDIR/epmgrlog

wlsinst -ah

(from the TMF gateway/TFST Endpoint Proxy)

$DBDIR/gatelog

tfst_install_dir/epproxy.cfg

tfst_install_dir/epp.log

(from the TFST Gateway Proxy)

tfst_install_dir/gwproxy.cfg

tfst_install_dir/gwp.log

Often the problem may recover by stopping and restarting the TFST components:

(at gateway proxy host)

cd tfst_install_dir

./gwproxy.sh stop

./gwproxy.sh start

(at endpoint proxy host)

cd tfst_install_dir

./epproxy.sh stop

./epproxy.sh start

Endpoint proxy runs as a process

[1] [email protected]:/var/adm/cron # ps -elf | grep epp
200001 A root 663688 598092 0 60 20 e9dac400 228 f100010039769308 10:37:13 pts/2 0:00 grep epp
240001 A root 667710 1 0 60 20 15bc34400 4484 * Oct 12 - 5:11 /opt/tivoli/EPProxy/epproxy /opt/tivoli/EPProxy

It writes log to

Typical problem may look like

09/10/13 10:39:21  1  7456 gatewayProxySender [gw=8] failure creating a routed s
ession for nti281-gwp
09/10/13 10:41:19  1  8503 routedSessionOpen: [rs=8130] peer replied with failur
e
09/10/13 10:41:19  1  8503 upcallSender [id=92,rs=8130] routedSessionOpen failed
09/10/13 10:41:33  1  7731 routedSessionCreate: [rs=8128] open session failed (e
=-2)
09/10/13 10:41:33  2  7731 gatewayProxySender [gw=9] failure creating a routed s
ession for nti281-gwp.
09/10/13 10:41:33  1  7731 gatewayProxySender [gw=9] failure creating a routed s
ession for nti281-gwp
09/10/13 10:42:42  1  7998 routedSessionCreate: [rs=8131] open session failed (e
=-2)
09/10/13 10:42:42  2  7998 gatewayProxySender [gw=11] failure creating a routed
session for nti281-gwp.
09/10/13 10:42:42  1  7998 gatewayProxySender [gw=11] failure creating a routed
session for nti281-gwp

On the secure side of the firewall, there is an endpoint proxy that connects to the gateway as if it were the endpoint itself. On the less secure side of the firewall, the endpoints are connected to a gateway proxy, which acts as if it were the real gateway. The gateway proxy and the endpoint proxy then communicate with each other through the firewall.

Just as multiple endpoints can connect to a single gateway and multiple gateways to a single server, multiple endpoints can connect to a single gateway proxy and multiple gateway proxies can connect to a single endpoint proxy. The endpoint proxy emulates all the endpoints to the gateway that manages them.

The endpoint and gateway proxies consolidate all communications between multiple endpoints and gateways over a single port. All such communications are encapsulated in a connection-based TCP protocol.

Another benefit to the endpoint and gateway proxies is that they allow customers to keep all of their Tivoli Management infrastructure (servers and gateways) inside their most secure zones, and only the endpoints and their gateway proxies outside.

The endpoint and gateway proxy solution effectively addresses the issues of communicating over a firewall between the gateway and the endpoint. But what if there is more than one firewall between these systems? In most installations, there will be multiple security zones or DMZs, each separated by a firewall layer, between a gateway and its endpoints. For these environments, the next component, relays, provides a solution.

NEWS CONTENTS

Old News ;-)

[Oct 14, 2009] developerWorks test Test tivoli forum [tme10] [FW] Firewall Toolkit - ...

RE: [tme10] [FW] Firewall Toolkit - endpoints won't log in.
Posted: Oct 10, 2003 04:10:04 PM in response to: Tim Meanor's post

Hi Tim,

You didn't say if you had a successful handshake between your gwp and epp. The endpoints are moot if you can't get the gwp & epp to talk to each other. I suggest that you edit your epproxy.cfg and gwproxy.cfg to set logging at a high level (like 9 or 10), then restart those processes and monitor the logfiles to see what is going on. Don't forget to read & use the troubleshooting tips in the TFST admin guide.

You should also watch the gwp.log and the lcfd.log (at debug level 3) while restarting an endpoint to see what happens ... does it try to contact the correct gateway? Does the gwp ever see the ep trying to login? Check ports on the endpoint hosts (I recommend "lsof" if you have that tool, netstat otherwise) to make sure that no other process has bound your ports.

Do you have a true picture of the network topology in your DMZ / firewalled environment? In my shop, even though I was responsible for setting up Tivoli in the DMZ (using TFST) I was not allowed to see the network topology and discovered after the fact that the production network did not look anything like the development network in which I had originally tested everything :-P. I've been fighting^H^H^H^H^H^H^H^Hworking with the network security folks here for nearly a year to get Tivoli working properly in our DMZ. Haha.

Good Luck!

John Guadagnino
Gap Inc.

===

Tim,

Have you enabled tracing on both components? Do you get the WHO and TELL commands being executed successfully?

Regards,

--------------------------------------------------------------------------------

Marco Castillo, MCSE (NT & W2K)

Tivoli Services ADP Employer Services [email protected]

Tel (905) 795-5400 x 7315

== Please show some details of your configuration files (gwproxy.cfg and epproxy.cfg). How did you configure the connection, uni-directional or bi-directional? What kind of error did you see for endpoint login failing?

Chunhui Zhong

Systems Management Engineering, VISA/Invoant

Tel: 650-432-3955

Mobile: 408-921-8930

Email: [email protected]

IBM - gwproxy Daemon Fails to Start

The daemon fails to start when stopping and then attempting to restart the gateway proxy daemon.

When the gateway proxy is shutdown, its connections to the endpoint proxy are put in a TIME_WAIT state. The gateway proxy will not start until its connections to its parent component are closed.
The gwp.log will show an error similar to the following:

02/11/12 15:05:30 1 1 ERROR tcpunidir.createServerSocket: cannot bind socket
t (125)
02/11/12 15:05:30 0 1 Port 1026 not available.
02/11/12 15:05:30 0 1 FATAL tcpunidir.constructor: cannot create server soc
ket
02/11/12 15:05:30 1 1 initRoutedSessionsManager: failure creating connectio
n manager for the parent
02/11/12 15:05:30 0 1 routed sessions manager initialization failed [cfg=gw
proxy.cfg;label=venus]
02/11/12 15:05:30 0 1 INIT ERROR: failure starting communication layer

Wait a few minutes until the connections have been terminated and restart the proxy.
One can check the state of the connections by running 'netstat -a |grep <gwproxy_port>' on the gateway proxy.

The gateway proxy is a component of the Tivoli Firewall Security Toolbox.

Starting and stopping the components on UNIX operating systems

On UNIX operating systems, to start the components, do the following:

  1. Go to the directory in which the component is installed.
  2. Enter the command:
    ./component.sh start

    where component stands for:

    epproxy
    The endpoint proxy
    eventsink
    The event sink
    gwproxy
    The gateway proxy
    relay
    The relay. If you have more than one instance of the relay installed on the machine, specify the directory numbered for the instance of the relay to be started or stopped.

To stop the components on UNIX operating systems, enter the command from the directory in which the component is installed:

./component.sh stop

Providing More Detail in the Log Files

If you need to troubleshoot or to contact customer support, set the log levels of the components to a higher level of detail as follows:

IBM - IY55176 Epproxy-Gwproxy could have many connections in CLOSE_WAIT and than the process may run out of threads

Tivoli Firewall Security Toolbox Interim Fix 1.3.1-TFS-0009
Tivoli Firewall Security Toolbox Interim Fix 1.3.1-TFS-0011

APAR status
Closed as program error.

Error description

Env: Tivoli Firewall Solution Toolbox 1.3-TFS-0003A

Problem: In this customers environment we saw Epproxy leaves tons of connections in CLOSE_WAIT state. These connections are between gateway and endpoint proxy emulating the endpoint.

This ultimately causes the Endpoint proxy to run out of threads. Epp.log shows

   03/09/09 16:41:01  1 1779102 upcallSender [line=3505]:
   accept() failed (e=24 - Too many open files)
   03/09/09 16:41:02  1 1779106 LEAVE open_db (abdb_open
   failed)
   03/09/09 16:41:02  1 1779106 find_record: wrong parameter
We ran pfiles on customers Solaris Endpoint proxy and saw that the files utilization increased over a period of time

Recreate: Havent been able to recreate in house.

Summary: After talking to L3 here is what we think is happening. There can be instances where one TFST peer (e.g. the gateway proxy) is for some reason unable to communicate to its peer (e,g, the endpoint proxy) that a routed session needs to be closed. When this occurs, TFST fails to close its end of a TFST<->TMF connection. We see this at this particular customer site epproxy<->gateway connections stuck in CLOSE_WAIT state.

This same problem can theoretically happen on the gateway proxy, leaving gwproxy<->endpoint connections stuck in CLOSE_WAIT state.

Problem summary

The endpoint proxy closes its end of a connection with the gateway after receiving a command from the gateway proxy that the proxied endpoint using this connection has no further data to send to the gateway. Likewise, the gateway proxy closes its end of a connection with an endpoint after receiving a command from the endpoint proxy that the gateway has no further data to send to the proxied endpoint.

If for some reason the command to close the connection is not received, the connection between the endpoint proxy and gateway (or between the gateway proxy and an endpoint) persists in CLOSE_WAIT state.

Over time, the number of connections in CLOSE_WAIT state may increase to the point where all file descriptors available to the process are consumed. This results in failed upcalls and downcalls.

Problem conclusion

This is fixed in interim fix 1.3.1-TFS-0003.

Temporary fix

Comments

APAR information
APAR number IY55176
Reported component name MGMT. FRAMEWORK
Reported component ID 5698FRA00
Reported release 371
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Special Attention NoSpecatt
Submitted date 2004-03-25
Closed date 2004-05-06
Last modified date 2004-05-06

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Modules/Macros

Publications Referenced

Fix information
Fixed component name MGMT. FRAMEWORK
Fixed component ID 5698FRA00

Applicable component levels
R371 PSN UP

IBM - IY55176 Epproxy-Gwproxy could have many connections in CLOSE_WAIT and than the process may run out of threads

Summary: After talking to L3 here is what we think is
happening. There can be instances where one TFST peer
(e.g. the gateway proxy) is for some reason unable to
communicate to its peer (e,g, the endpoint proxy) that a
routed session needs to be closed. When this occurs, TFST
fails to close its end of a TFST<->TMF connection. We see
this at this particular customer site epproxy<->gateway
connections stuck in CLOSE_WAIT state. This same problem
can theoretically happen on the gateway proxy, leaving
gwproxy<->endpoint connections stuck in CLOSE_WAIT state

Documentation

Firewall Security Toolbox User's Guide

Recommended Links

Firewall Security Toolbox User's Guide

[PDF] Firewall Magic: How Tivoli Delivers End-to-End System Management ...

File Format: PDF/Adobe Acrobat - View as HTML

[PDF] Tivoli Enterprise Management Across Firewalls

IBM - Tivoli Security Firewall Toolkit gateway proxy failover

Tivoli Firewall Security Toolbox changes

Installing the Tivoli Remote Control Proxy Using a Response File

IBM Redbooks Data Flow for Tivoli Remote Control Proxy on Top of Tivoli Firewall Security Toolbox in a Single-TMR Session



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 29, 2020