Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)

Classic Network Utilities

News See Also Recommended Links Recommended Articles Reference Network Troubleshooting Tools TCPreplay
ifconfig ethtool ndd solaris route Linux route command netstat Wireshark
ping traceroute mtr Snoop Tcpdump ngrep netcat
nmap ntop rsync snort Packet Generation Tools Humor Etc

There are at lease a dozen of classic network tools. Among them are:

Several network utilities are can pipable (netcat can create pipelines across network):

In Jan 2007 I added the page about TCP/IP troubleshooting tools to complement this page

A very good list of free network tools can be found at Top 100 Network Security Tools

Dr. Nikolai Bezroukov


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Jul 16, 2018] netstat to find ports which are in use on linux server

Another example of more or less complex pipeline using cat
Oct 02, 2008 | midnight-cafe.co.uk

Below is command to find out number of connections to each ports which are in use using netstat & cut.

netstat -nap | grep 'tcp\|udp' | awk '{print $4}' | cut -d: -f2 | sort | uniq -c | sort -n

Below is description of each commands ::

Netstat command is used to check all incoming and outgoing connections on linux server. Using Grep command you can sort lines which are matching pattern you defined.

AWk is very important command generally used for scanning pattern and process it. It is powerful tool for shell scripting. Sort is used to sort output and sort -n is for sorting output in numeric order.

Uniq -c this help to get uniq output by deleting duplicate lines from it.

[Jul 16, 2018] Listing TCP apps listening on ports

Jun 13, 2018 | www.fredshack.com

netstat -nltp

[Jan 14, 2018] Using telnet to debug connection problems

Jan 14, 2018 | bash-prompt.net

Telnet, the protocol and the command line tool, were how system administrators used to log into remote servers. However, due to the fact that there is no encryption all communication, including passwords, are sent in plaintext meant that Telnet was abandoned in favour of SSH almost as soon as SSH was created.

For the purposes of logging into a remote server, you should never, and probably have never considered it. This does not mean that the telnet command is not a very useful tool when used for debugging remote connection problems.

In this guide, we will explore using telnet to answer the all too common question, "Why can't I ###### connect‽".

This frustrated question is usually encountered after installing a application server like a web server, an email server, an ssh server, a Samba server etc, and for some reason, the client won't connect to the server.

telnet isn't going to solve your problem but it will, very quickly, narrow down where you need to start looking to fix your problem.

telnet is a very simple command to use for debugging network related issues and has the syntax:

telnet <hostname or IP> <port>

Because telnet will initially simply establish a connection to the port without sending any data it can be used with almost any protocol including encrypted protocols.

There are four main errors that you will encounter when trying to connect to a problem server. We will look at all four, explore what they mean and look at how you should fix them.

For this guide we will assume that we have just installed a Samba server at samba.example.com and we can't get a local client to connect to the server.

Error 1 - The connection that hangs forever

First, we need to attempt to connect to the Samba server with telnet . This is done with the following command (Samba listens on port 445):

telnet samba.example.com 445

Sometimes, the connection will get to this point stop and hang indefinitely:

telnet samba.example.com 445
Trying 172.31.25.31...

This means that telnet has not received any response to its request to establish a connection. This can happen for two reasons:

  1. There is a router down between you and the server.
  2. There is a firewall dropping your request.

In order to rule out 1. run a quick mtr samba.example.com to the server. If the server is accessible then it's a firewall (note: it's almost always a firewall).

Firstly, check if there are any firewall rules on the server itself with the following command iptables -L -v -n , if there are none then you will get the following output:

iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

If you see anything else then this is likely the problem. In order to check, stop iptables for a moment and run telnet samba.example.com 445 again and see if you can connect. If you still can't connect see if your provider and/or office has a firewall in place that is blocking you.

Error 2 - DNS problems

A DNS issue will occur if the hostname you are using does not resolve to an IP address. The error that you will see is as follows:

telnet samba.example.com 445
Server lookup failure:  samba.example.com:445, Name or service not known

The first step here is to substitute the IP address of the server for the hostname. If you can connect to the IP but not the hostname then the problem is the hostname.

This can happen for many reasons (I have seen all of the following):

  1. Is the domain registered? Use whois to find out if it is.
  2. Is the domain expired? Use whois to find out if it is.
  3. Are you using the correct hostname? Use dig or host to ensure that the hostname you are using resolves to the correct IP.
  4. Is your A record correct? Check that you didn't accidentally create an A record for something like smaba.example.com .

Always double check the spelling and the correct hostname (is it samba.example.com or samba1.example.com ) as this will often trip you up especially with long, complicated or foreign hostnames.

Error 3 - The server isn't listening on that port

This error occurs when telnet is able to reach to the server but there is nothing listening on the port you specified. The error looks like this:

telnet samba.example.com 445
Trying 172.31.25.31...
telnet: Unable to connect to remote host: Connection refused

This can happen for a couple of reasons:

  1. Are you sure you're connecting to the right server?
  2. Your application server is not listening on the port you think it is. Check exactly what it's doing by running netstat -plunt on the server and see what port it is, in fact, listening on.
  3. The application server isn't running. This can happen when the application server exits immediately and silently after you start it. Start the server and run ps auxf or systemctl status application.service to check it's running.
Error 4 - The connection was closed by the server

This error happens when the connection was successful but the application server has a built in security measure that killed the connection as soon as it was made. This error looks like:

telnet samba.example.com 445
Trying 172.31.25.31...
Connected to samba.example.com.
Escape character is '^]'.
Connection closed by foreign host.

The last line Connection closed by foreign host. indicates that the connection was actively terminated by the server. In order to fix this, you need to look at the security configuration of the application server to ensure your IP or user is allowed to connect to it.

A successful connection

This is what a successful telnet connection attempt looks like:

telnet samba.example.com 445
Trying 172.31.25.31...
Connected to samba.example.com.
Escape character is '^]'.

The connection will stay open for a while depending on the timeout of the application server you are connected to.

A telnet connection is closed by typing CTRL+] and then when you see the telnet> prompt, type "quit" and hit ENTER i.e.:

telnet samba.example.com 445
Trying 172.31.25.31...
Connected to samba.example.com.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
Conclusion

There are a lot of reasons that a client application can't connect to a server. The exact reason can be difficult to establish especially when the client is a GUI that offers little or no error information. Using telnet and observing the output will allow you to very rapidly narrow down where the problem lies and save you a whole lot of time.

[Jun 10, 2010] Deep-protocol analysis of UNIX networks

[Jun 08, 2010 | developerWorks
Parsing the raw data to understand the content

Another way to process the content from tcpdump is to save the raw network packet data to a file and then process the file to find and decode the information that you want.

There are a number of modules in different languages that provide functionality for reading and decoding the data captured by tcpdump and snoop. For example, within Perl, there are two modules: Net::SnoopLog (for snoop) and Net::TcpDumpLog (for tcpdump). These will read the raw data content. The basic interfaces for both of these modules is the same.

To start, first you need to create a binary record of the packets going past on the network by writing out the data to a file using either snoop or tcpdump. For this example, we'll use tcpdump and the Net::TcpDumpLog module: $ tcpdump -w packets.raw.

Once you have amassed the network data, you can start to process the network data contents to find the information you want. The Net::TcpDumpLog parses the raw network data saved by tcpdump. Because the data is in it's raw binary format, parsing the information requires processing this binary data. For convenience, another suite of modules, NetPacket::*, provides decoding of the raw data.

For example, Listing 8 shows a simple script that prints out the IP address information for all of the packets.

Listing 8. Simple script that prints out the IP address info for all packets
use Net::TcpDumpLog;
    
use NetPacket::Ethernet;
    
use NetPacket::IP;

    
my $log = Net::TcpDumpLog->new();
 
$log->read("packets.raw");
 
 
foreach my $index ($log->indexes)
       
{
    
    my $packet = $log->data($index);
           

    my $ethernet = NetPacket::Ethernet->decode($packet);

  
    if ($ethernet->{type} == 0x0800)
       
    {
    
        my $ip = NetPacket::IP->decode($ethernet->{data});
          

    
        printf("  %s to %s protocol %s \n",
               $ip->{src_ip},$ip->{dest_ip},$ip->{proto});
   }

} 
The first part is to extract each packet. The Net::TcpDumpLog module serializes each packet, so that we can read each packet by using the packet ID. The data() method then returns the raw data for the entire packet.

As with the output from snoop, we have to extract each of the blocks of data from the raw network packet information. So in this example, we first need to extract the ethernet packet, including the data payload, from the raw network packet. The NetPacket::Ethernet module does this for us.

Since we are looking for IP packets, we can check for IP packets by looking at the Ethernet packet type. IP packets have an ID of 0x0800.

The NetPacket::IP module can then be used to extract the IP information from the data payload of the Ethernet packet. The module provides the source IP, destination IP and protocol information, among others, which we can then print.

Using this basic framework you can perform more complex lookups and decoding that do not rely on the automated solutions provided by tcpdump or snoop. For example, if you suspect that there is HTTP traffic going past on a non-standard port (i.e., not port 80), you could look for the string HTTP on ports other than 80 from the suspected host IP using the script in Listing 9.


Listing 9. Looking for strong HHTP on ports other than 80
use Net::TcpDumpLog;
    
use NetPacket::Ethernet;
    
use NetPacket::IP;
    
use NetPacket::TCP;
    

    
my $log = Net::TcpDumpLog->new();
       
$log->read("packets.raw");
       

    
foreach my $index ($log->indexes)
       
{
    
    my $packet = $log->data($index);
       

    
    my $ethernet = NetPacket::Ethernet->decode($packet);
       

    
    if ($ethernet->{type} == 0x0800)
       
    {
    
        my $ip = NetPacket::IP->decode($ethernet->{data});
          

    
        if ($ip->{src_ip} eq '192.168.0.2')
       
        {
    
            if ($ip->{proto} == 6)
       
            {
    
                my $tcp = NetPacket::TCP->decode($ip->{data});
       
                if (($tcp->{src_port} != 80) &&
               
                    ($tcp->{data} =~ m/HTTP/))
       
                {
    
                    print("Found HTTP traffic on non-port 80\n");
    
                    printf("%s (port: %d) to %s (port: %d)\n%s\n",
    
                           $ip->{src_ip},
       
                           $tcp->{src_port},
       
                           $ip->{dest_ip},
       
                           $tcp->{dest_port},
       
                           $tcp->{data});
 
                }
    
            }
    
        }
    
   }
    
}

Running the above script on a sample packet set returned the following shown in Listing 10.


Listing 10. Running the script on a sample packet set
$ perl http-non80.pl
Found HTTP traffic on non-port 80
192.168.0.2 (port: 39280) to 168.143.162.100 (port: 80)
GET /statuses/user_timeline.json HTTP/1.1
Found HTTP traffic on non-port 80
192.168.0.2 (port: 39282) to 168.143.162.100 (port: 80)
GET /statuses/friends_timeline.json HTTP/1

In this particular case we're seeing traffic from the host to an external website (Twitter).

Obviously, in this example, we are dumping out the raw data, but you could use the same basic structure to decode and the data in any format using any public or proprietary protocol structure. If you are using or developing a protocol using this method, and know the protocol format, you could extract and monitor the data being transferred.

Using a protocol analyzer

Although, as already mentioned, tools like tcpdump, iptrace and snoop provide basic network analysis and decoding, there are GUI-based tools that make the process even easier. Wireshark is one such tool that supports a vast array of network protocol decoding and analysis.

One of the main benefits of Wireshark is that you can capture packets over a period of time (just as with tcpdump) and then interactively analyze and filter the content based on the different protocols, ports and other data. Wireshark also supports a huge array of protocol decoders, enabling you to examine in minute detail the contents of the packets and conversations.

You can see the basic screenshot of Wireshark showing all of the packets of all types being listed in Figure 1. The window is divided into three main sections: the list of filtered packets, the decoded protocol details, and the raw packet data in hex/ASCII format.

[Aug 6, 2009] Xplico 0.5.2

The goal of Xplico is to extract the applications data from an Internet traffic capture. For example, from a pcap file Xplico extracts each email (POP, IMAP, and SMTP protocols), all HTTP contents, each... VoIP call (SIP), and so on. Xplico isn't a packet sniffer or a network protocol analyzer; it's an IP/Internet traffic decoder or network forensic analysis tool (NFAT).

[Jan 2, 2008] vnStat

About:
vnStat is a console-based network traffic monitor that keeps a log of hourly, daily, and monthly network traffic for the selected interface(s). However, it isn't a packet sniffer. The traffic information is analyzed from the /proc filesystem. That way, vnStat can be used even without root permissions.

Release focus: Minor bugfixes

Changes:
This release fixes a bug that caused a segmentation fault if the environment variable "HOME" wasn't defined, which in turn caused most PHP/CGI scripts using vnStat to malfunction. Some minor feature enhancements are also included.

[Apr 15, 2007] freshmeat.net Project details for Tcpreplay by Aaron Turner

Tcpreplay 3.0.RC1 (stable)

This release improves OpenBSD, HP-UX, Cygwin/Win32, x86_64, and little endian support. Enhancements were made to allow editing packets with tcpreplay. libpcap detection was improved.

[Mar 24, 2007] freshmeat.net Project details for Tcpreplay

Tcpreplay 3.0.beta13 released

Tcpreplay is a set of Unix tools which allows the editing and replaying of captured network traffic in pcap (tcpdump) format. It can be used to test a variety of passive and inline network devices, including IPS's, UTM's, routers, firewalls, and NIDS.

Release focus: Major bugfixes

Changes:
This release fixes some serious regression bugs that prevented tcprewrite from editing most packets on Intel and other little-endian systems. Some smaller bugfixes and tweaks to improve replay performance were made.

Author:
Aaron Turner [contact developer]

[Mar 3rd 2006 ] freshmeat.net Project details for netrw by Jiri Denemark

netrw 1.3.1

About: netrw is a simple (but powerful) tool for transporting data over the Internet. Its main purpose is to simplify and speed up file transfers to hosts without an FTP server. It can also be used for uploading data to some other user. It is something like one-way netcat (nc) with some nice features concerning data transfers. It can compute and check message digest (MD5, SHA-1, and some others) of all the data being transferred. It can also print information on progress and average speed. At the end, it sums up the transfer.

Changes: A bug causing netread to sometimes end up in an endless loop after receiving all data was fixed.



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) in the author free time and without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: January, 15, 2018