Linux Policy Routing

Policy routing is similar to firewall: it allows to make routing decision not only based on target address (like traditional routing) but on any other component of the packet. Generally any combination of the packet header fields can be used to define a route.  There is a new Linux command called simply ip which allows to configure it. 

The most valuable is the ability to select route base of combination of source and target addresses elevates the source IP address to the same level as the destination. This allows to use multiple default routers (for example to two different ISPs see http://lartc.org/howto/lartc.rpdb.multiple-links.html ) and helps to fight spoofing: when a router is paying attention to the source address and it is looking for a specific set of source addresses, using a faked source address (spoofing) is limited

Some Linux distributions like Suse 11 block packets as "martian packets" packets from special networks like 10.x.x.x network directed to the second interface which serves different subnet. Policy routing implemented in kernel but you need to check if it is enabled in your particular distribution.  Here is how to check kernel compatibility with policy routing feature (from Novell support document 7009083: Configure multiple default routes on linux):

1. Ensure the server is ready for the process:

A. Make sure the appropriate packages are installed (a quick zypper se will confirm this):

i. kernel-source

ii. gcc

iii. ncurses-devel iv.

iproute2

B. Check for kernel compatiblity:

i. cd /usr/src/linux

ii. make menuconfig

iii. Follow: Networking -> Networking Options -> And make sure the following are selected:

TCP/IP Networking

IP: advanced router

IP: Policy Routing  

IP: use netfilter MARK value as routing key (SLES 10 only)

IP: Choose IP: FIB lookup algorithm (FIB_HASH)  (SLES 10 only)

2. Create a new policy routing table for each interface:

echo "1 corporate">> /etc/iproute2/rt_tables

3. Provide IP info / gateway to the new corporate table.

ip route add 192.168.0.0/24 dev eth0 src 192.168.0.99 table corporate
ip route add default via 192.168.0.1 dev eth0 table corporate

4. Create IP rules to handle inbound / outbound traffic on this network.

ip rule add from 192.168.0.99/32 table corporate
ip rule add to 192.168.0.99/32 table corporate 

To check if policy routing is enabled you can view /etc/iproute2/rt_tables. For example

vi /etc/iproute2/rt_tables

A simple introduction (with a nice easy example) to source based routing

On standard Internet systems, when you receive a packet and decide where to route it to, that decision is made only based on the destination of the packet.

For example:

 crb@firewall:~$ /sbin/route -n
 Kernel IP routing table
 Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
 203.167.218.65  0.0.0.0         255.255.255.255 UH    0      0        0 ppp0
 192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
 10.7.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
 0.0.0.0         203.167.218.65  0.0.0.0         UG    0      0        0 ppp0

In this, a simple routing table for a firewall, all traffic for 192.168.0.0/24 is routed out eth1; traffic for 10.7.0.0/24 is routed out eth0; and everything else is routed out ppp0 to the Internet.

However, let's deal with the situation where we have two interfaces ppp0 and ppp1 (a dual-homed situation, with two internet providers.) We will call the IP address on ppp0 $P0 and on ppp1, $P1.

You end up with a routing table that looks like this:

 crb@firewall:~$ /sbin/route -n
 Kernel IP routing table
 Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
 219.88.160.1    0.0.0.0         255.255.255.255 UH    0      0        0 ppp1
 203.167.218.65  0.0.0.0         255.255.255.255 UH    0      0        0 ppp0
 192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
 10.7.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
 0.0.0.0         203.167.218.65  0.0.0.0         UG    0      0        0 ppp0

If you get traffic for your machine come into ppp1 from $OUTSIDE, the machine will receive the packets, generate a reply, and the system will now have a packet from $P1, destined to $OUTSIDE. Because the system only looks at destination IP addresses, the packet will get routed out the default gateway, ppp0. Even if you disable ReversePathFiltering to allow this kind of traffic on all your interfaces, chances are high your ISP will be using it. (For example, TelstraClear does not allow any traffic on it's network that originates from another network's IP.)

So, this is where source based routing comes in. We need to take any traffic that originates from $P1 (replies to traffic that came in ppp1), and route it back out through ppp1.

To do this we need to have the iproute2 package, which provides the command /sbin/ip; giving you much finer grained control over routing. If you don't have the /sbin/ip command, install an iproute package (debian: apt-get install iproute). The command route cannot handle multiple routing tables.

You also need to have a couple of kernel options enabled: they are CONFIG_IP_ADVANCED_ROUTER (Networking/IP: Advanced Router) and CONFIG_IP_MULTIPLE_TABLES (Networking/IP: policy routing).

Then, what you do is you create another routing table by editing /etc/iproute2/rt_tables; in my example I wish to create routes for a jetstream connection, so I have called the table 'jetstream' <footnote 1> by adding the line

 100     jetstream

Now, you can create a rule that dictates what routing table to look at.

 ip rule add __from $P1__ table jetstream

Look at the rules with ip rule list to get an idea of what happens when a packet is to be routed. The important bit is the from $P1. If you forget it, depending at the priority of your table, you could send all traffic to that table by default. Now, when routing, a packet that comes from the IP address $P1 will be passed to the routing table 'jetstream' instead of the main routing table.

Populate this table with a new default route, and simple routes for the rest of your local interfaces:

 ip route add 10.7.0.0/24 dev eth0 table jetstream
 ip route add 192.168.1.0/24 dev eth1 table jetstream
 ip route add 127.0.0.0/8 dev lo table jetstream
 ip route add default via 219.88.160.1 table jetstream

And you're done. In my case, I'm doing this on a ppp interface, so I only need the routes to exist when the interface is up; I've therefore added scripts for this to /etc/ppp/ip-up.d/ (ip-down.d contains ip rule del; I leave the table there - it's no harm if it's not called, but you could remove it with ip route del).

Thanks to PerryLorier for explaining this all to me, and to the

Linux Advanced Routing and Traffic Control HOWTO for filling in the detail; specifically Routing for multiple uplinks/providers.

From http://www.policyrouting.org/PolicyRoutingBook/ONLINE/CH03.web.html

Where are the route by source address, route by packet header data, and the other selection mechanisms?

This is where the rule element comes into play. Think of the rule as a method for implementing ACLs (Access Control Lists) for routes. The rule allows you to specify the filters that match packets, and which route structure to select when the filter does match. Because the filter is part of the rule selection mechanism, you can also use rules to specify other advanced options such as destination targets and NAT functions.

Using a rule you can perform the most common Policy Routing function, route by source address. The rule can specify to select a packet based on whether or not the source address of the packet falls into a designated address scope. If it does match, the rule states which route structure to use or other destination to choose. But if you stop to think about this for a moment, you realize that on a system where you only have one routing table a rule set is usable only under limited conditions.

3.1.3.1 Multiple Routing Tables

In a single routing table system, such as current network router devices, or most operating systems, all of the routes specified are in a single group called a table. This table is then read through (in network speak the route is "looked up") sequentially and the longest match of the packet destination is made. This longest match then returns the gateway to which to forward the packet.

Suppose you have three routers to the same network. Each router has a different speed connection to your network core. Which one should you use in your routing table? Even under OSPF this type of routing structure still results in a single "best" route for the condition of use.

Consider this conundrum in a different light. Most of your network clients only need limited access to a particular network. They would be fine on a slow link. A select group of your network clients needs a higher rate access to this network. If you have only one routing table, you can only put in one route to this destination network. Which one do you use? In this case even if you use rules to select the traffic, where are you going to end up sending them? To the routing table.

Thus the implementation of the rule in Policy Routing implies that for true global structure you must also implement multiple routing tables. A complete Policy Routing structure is found in the Linux kernel, version 2.1 and higher. It provides full use of Address, Multiple Independent Route Tables, and a Rule selection mechanism that can interact bidirectionally with the route tables.

3.2 RPDB - The Linux Policy Routing Implementation

Under Linux, the implementation of Policy Routing structure is carried out through the mechanism of the Routing Policy DataBase (RPDB). The RPDB is the cohesive set of routes, route tables, and rules. Since addressing is a direct function of these elements, it also is part of the system. What the RPDB primarily does is provide the internal structure and mechanism for implementing the rule element of Policy Routing. It also provides the multiple routing tables available under Linux.

Linux's RPDB and the complete rewrite of the IP addressing and routing structures in kernel 2.1 and higher sustains 255 routing tables and 2^32 rules. That is one rule per IP address under IPv4. In other words, you can specify a rule to govern every single address available in the entire IPv4 a ddress space. That works out to over 4 billion rules.

The RPDB itself operates upon the rule and route elements of the triad. In the operation of RPDB, the first element considered is the operation of the rule. The rule, as you saw, may be considered as the filter or selection agent for applying Policy Routing.

The following text about the RPDB and the definition of Policy Routing is adapted from Alexey Kuznetsov's documentation for the IPROUTE2 utility suite, with Alexey's permission. I have rewritten parts of the text to clarify some points. Any errors or omissions should be directed to me.

Classic routing algorithms used on the Internet make routing decisions based only on the destination address of packets and, in theory but not in practice, on the TOS field. In some circumstances you may want to route packets differently, depending not only on the destination addresses but also on other packet fields such as source address, IP protocol, transport protocol ports, or even packet payload. This task is called Policy Routing.

To solve this task, the conventional destination-based routing table, ordered according to the longest match rule, is replaced with the RPDB, which selects the appropriate route through execution of rules. These rules may have many keys of different natures, and therefore they have no natural order except that which is imposed by the network administrator. In Linux the RPDB is a linear list of rules ordered by a numeric priority value. The RPDB explicitly allows matching packet source address, packet destination address, TOS, incoming interface (which is packet meta data, rather than a packet field), and using fwmark values for matching IP protocols and transport ports. Fwmark is the packet filtering tag that you will use in Chapter 6 and is explained later on in this Chapter in the section "System Packet Paths -IPChains/NetFilter."

Each routing policy rule consists of a selector and an action predicate. The RPDB is scanned in the order of increasing priority, with the selector of each rule applied to the source address, destination address, incoming interface, TOS, and fwmark. If the selector matches the packet, the action is performed. The action predicate may return success, in which case the rule output provides either a route or a failure indication, and RPDB lookup is then terminated. Otherwise, the RPDB program continues on to the next rule.

What is the action semantically? The natural action is to select the nexthop and the output device. This is the way a packet path route is selected by Cisco IOS; let us call it "match & set." In Linux the approach is more flexible because the action includes lookups in destination-based routing tables and selecting a route from these tables according to the classic longest match algorithm. The "match & set" approach then becomes the simplest case of Linux route selection, realized when the second level routing table contains a single default route. Remember that Linux supports multiple routing tables managed with the ip route command.

At startup, the kernel configures a default RPDB consisting of three rules:

Priority 0: Selector = match anything

Action = lookup routing local table (ID 255)

The local table is the special routing table containing high priority control routes for local and broadcast addresses. Rule 0 is special; it cannot be deleted or overridden.

 

Priority 32766: Selector = match anything

Action = lookup routing main table (ID 254)

The main table is the normal routing table containing all non-policy routes. This rule may be deleted or overridden with other rules.

 

Priority 32767: Selector = match anything

Action = lookup routing table default (ID 253)

The table default is empty and reserved for post-processing if previous default rules did not select the packet. This rule also may be deleted.

Do not mix routing tables and rules. Rules point to routing tables, several rules may refer to one routing table, and some routing tables may have no rules pointing to them. If you delete all the rules referring to a table, then the table is not used but still exists. A routing table will disappear only after all the routes contained within it are deleted. Remember that a route is the location of the address while the rule is the location of the route.

Each RPDB entry has additional attributes attached. Each rule has a pointer to some routing table. NAT and masquerading rules have the attribute to select a new IP address to translate/masquerade. Additionally, rules have some of the optional attributes that routes have, such as realms. These values do not override those contained in routing tables; they are used only if the route did not select any of those attributes.

The RPDB may contain rules of the following types:

unicast	The rule prescribes returning the route found in the routing table referenced by the rule.
blackhole	The rule prescribes dropping a packet silently.
unreachable	The rule prescribes generating the error Network is unreachable.
prohibit	The rule prescribes generating the error Communication is administratively prohibited.
nat	The rule prescribes translating the source address of the IP packet to some other value.

You will see how these rule actions operate primarily in Chapter 5 and Chapter 6. There you will make hands-on use of the command set and implement several Policy Routing structures.

The RPDB was the first implementation of and first mention within the Linux community of the concept of Policy Routing. When you consider that the ip utility was first released in late spring of 1997, and that Alexey's documentation was released in April of 1999 coinciding with the official Linux 2.2 kernel release in May of 1999, then you realize that the Linux Policy Routing structure is already over five years old. In Internet time that is considered almost ancient. But as with most new network subjects, such as IPv6 and Policy Routing, Linux leads the way.

The RPDB itself was an integral part of the rewrite of the networking stack in Linux kernel 2.2. The Policy Routing extensions are accessed through a defined set of additional control structures within the Linux kernel. These extensions are the NETLINK and RT_NETLINK objects and related constructs. If you are curious about the programmatic details you can look through the source to the ip utility itself. The call structure and reference to the kernel internals is laid out quite well.

One of the important features that makes the RPDB implementation so special is that it is completely backward-compatible with the standard network utilities. You do not need to use the ip utility to perform standard networking tasks on your system. You can use ifconfig and route and get along quite fine. In fact, you can even compile the kernel without the NETLINK family objects and still use standard networking tools. It is only when you need to use the full features of the RPDB that you need to use the appropriate utility.

This backward compatibility is due to the RPDB being a complete replacement of the Linux networking structure, especially as it relates to routing. The addressing modalities for Policy Routing, as discussed in the "Address" section earlier in this chapter (and illustrated in depth in Chapter 5), were also implemented as part of this change. But the main changes, besides the addition of the rule element, were the changes to the route element. Drawing upon Alexey's documentation again I provide the following information on the route element construct.

In the RPDB, each route entry has a key consisting of the protocol prefix, which is the pairing of the network address and network mask length, and optionally the TOS value. An IP packet matches the route if the highest bits of the packet's destination address are equal to the route prefix, at least up to the prefix length, and if the TOS of the route is zero or equal to the TOS of the packet.

If several routes match the packet, the following pruning rules are used to select the best one:

1. The longest matching prefix is selected; all shorter ones are dropped.
2. If the TOS of some route with the longest prefix is equal to the TOS of the packet, routes with different TOS are dropped.
3. If no exact TOS match is found and routes with TOS=0 exist, the rest of the routes are pruned. Otherwise the route lookup fails.
4. If several routes remain after steps 1-3 have been tried, then routes with the best preference value are selected.
5. If several routes still exist, then the first of them is selected.

Note the ambiguity of action 5. Unfortunately, Linux historically allowed such a bizarre situation. The sense of the word "the first" depends on the literal order in which the routes were added to the routing table, and it is practically impossible to maintain a bundle of such routes in any such order.

For simplicity we will limit ourselves to the case wherein such a situation is impossible, and routes are uniquely identified by the triplet of prefix, TOS, and preference. Using the ip command for route creation and manipulation makes it impossible to create non-unique routes.

One useful exception to this rule is the default route on non-forwarding hosts. It is "officially" allowed to have several fallback routes in cases when several routers are present on directly connected networks. In this case, Linux performs "dead gateway detection" as controlled by Neighbor Unreachability Detection (nud) and references from the transport protocols to select the working router. Thus the ordering of the routes is not essential. However, in this specific case it is not recommended that you manually fiddle with default routes but instead use the Router Discovery protocol. Actually, Linux IPv6 does not even allow user-level applications access to default routes.

Of course, the preceding route selection steps are not performed in exactly this sequence. The routing table in the kernel is kept in a data structure that allows the final result to be achieved with minimal cost. Without depending on any particular routing algorithm implemented in the kernel, we can summarize the sequence as this: Route is identified by the triplet {prefix,tos,preference} key, which uniquely locates the route in the routing table.

Each route key refers to a routing information record. The routing information record contains the data required to deliver IP packets, such as output device and next hop router, and additional optional attributes, such as path MTU (Maximum Transmission Unit) or the preferred source address for communicating to that destination.

It is important that the set of required and optional attributes depends on the route type. The most important route type is a unicast route, which describes real paths to other hosts. As a general rule, common routing tables contain only unicast routes. However, other route types with different semantics do exist. The full list of types understood by the Linux kernel is as follows:

unicast	The route entry describes real paths to the destinations covered by the route prefix.
unreachable	These destinations are unreachable; packets are discarded and the ICMP message host unreachable (ICMP Type 3 Code 1) is generated. The local senders get error EHOSTUNREACH.
blackhole	These destinations are unreachable; packets are silently discarded. The local senders get error EINVAL.
prohibit	These destinations are unreachable; packets are discarded and the ICMP message communication administratively prohibited (ICMP Type 3 Code 13) is generated. The local senders get error EACCES.
local	The destinations are assigned to this host, the packets are looped back and delivered locally.
broadcast	The destinations are broadcast addresses, the packets are sent as link broadcasts.
throw	Special control route used together with policy rules. If a throw route is selected, then lookup in this particular table is terminated, pretending that no route was found. Without any Policy Routing, it is equivalent to the absence of the route in the routing table, the packets are dropped, and ICMP message net unreachable (ICMP Type 3 Code 0) is generated. The local senders get error ENETUNREACH.
nat	Special NAT route. Destinations covered by the prefix are considered as dummy (or external) addresses, which require translation to real (or internal) ones before forwarding. The addresses to translate to are selected with the attribute via.
anycast	not implemented The destinations are anycast addresses assigned to this host. They are mainly equivalent to local addresses, with the difference that such addresses are invalid to be used as the source address of any packet.
multicast	Special type, used for multicast routing. It is not present in normal routing tables.

Linux can place routes within multiple routing tables identified by a number in the range from 1 to 255 or by a name taken from the file /etc/iproute2/rt_tables. By default all normal routes are inserted to the table main (ID 254), and the kernel uses only this table when calculating routes.

Actually, another routing table always exists that is invisible but even more important. It is the local table (ID 255). This table consists of routes for local and broadcast addresses. The kernel maintains this table automatically, and administrators should not ever modify it and do not even need to look at it in normal operation.

In Policy Routing, the routing table identifier becomes effectively one more parameter added to the key triplet {prefix,tos,preference}. Thus, under Policy Routing the route is obtained by {tableid,key triplet}, identifying the route uniquely. So you can have several identical routes in different tables that will not conflict, as was mentioned earlier in the description of action 5 and "the first" mechanism associated with action 5.

These changes to the route element provide one of the core strengths of the RPDB, multiple independent route tables. As you will see in Chapter 5, the rule element alone can only perform a selection or filter operation. It is still up to the route to indicate where the packet needs to go next. Adding on top of these elements the QoS mechanisms to determine and set the TOS field and the ability to route by the TOS field provides you with the most powerful and flexible routing structure available under IPv4 and IPv6.

In summary, the RPDB is the core facility for implementing Policy Routing under Linux. The RPDB streamlines the mechanism of dealing with rules and multiple route tables. All operations of the rule and route structure are centralized into a single point of access and control. The addition of various alternate actions and destinations for routes and rules through the RPDB allows you to fine tune the mechanism of Policy Routing without needing to hack sections of the networking code.

3.3 System Packet Paths - IPChains/NetFilter

Understanding the RPDB brings up the question of at what point within the system the RPDB operates. To understand this within the context of the system you need to first see the logic of packet traversal within the system. The best way to approach this traversal is to consider how the packet filtering mechanisms treat this flow.

The various packet filtering mechanisms within the Linux kernel structures deal directly with the conceptualization of the packet flow as a means to identify the control points. They do this so that they may apply their security mechanisms at the control points. These control points are also of interest to the Policy Routing structure because these are the same control points that you would think to operate upon with Policy Routing structures.

... ... ...

3.3.2 NetFilter - Kernel 2.3/2.4

This contention of packet path location brings up the latest iteration of packet filtering in Linux. NetFilter is the extension of the traditional IPChains to cover state tracking functions. The new concept is to consider pure packet selection mechanisms as defining packet filtering in contrast to defining any packet selection mechanisms that change the packet information as packet mangling. This makes sense from many standpoints. It even casts a good light on the traditional split of consideration between routing and TOS/QoS structures as you will see in Chapter 6.

What NetFilter does is make this division of function obvious. Consider the packet paths in Figure 3.3.2.

Note that the dotted line tying together the two routing diamonds indicates that these are the same function, the RPDB. The reason for the split is that the routing function is entered in different places in the packet path.

This is due to another change in the packet path policy within NetFilter. The Input(2) and Output(4) chains now only refer to the Local Machine. When you consider that the primary function of a firewall is to protect machines behind it, and that implies transverse packets, then the packet path for NetFilter is much cleaner. Additionally, by placing the INPUT and OUTPUT chains as operating only upon the Local Machine you can create secured server machines.

Consider the path for a transverse packet. It enters the system and is processed by the entrance packet mangling and tagging stage, Pre-Route(1). This stage is where you would apply packet mangling operations such as fwmark and TOS/QoS tagging. The packet then enters the RPDB to obtain routing. From the RPDB it enters the primary firewall chain, Forward(3). The Forward chain is where the firewalling decisions are made. After the Forward chain it enters the exit packet mangling and tagging stage, Post-Route(5). The Pre-Route and Post-Route locations are where you would also apply NAT and IP MASQUERADING functions. Note that these NAT functions are not the same as the RPDB NAT. Indeed you will see the differences between RPDB NAT and NetFilter NAT in Chapter 8.

This transverse packet path, assuming you do not do any packet mangling, only then needs to be inspected by two entities, the RPDB and the Forward firewall chain. This is a great improvement in speed and logic when you start considering the interactions of Policy Routing and firewalling. For the purposes of a secured service machine, things are also more logically handled.

Consider the path for an externally sourced packet destined for an internal service. It enters the system and is processed by the entrance packet mangling and tagging stage, Pre-Route(1). This stage is where you would apply packet mangling operations such as fwmark and TOS/QoS tagging or perhaps the NetFilter NAT. The packet then enters the RPDB to obtain routing and is routed to the Input(2) chain. The Input chain provides the firewalling functions for packets destined to the Local Machine services.

The reverse scenario is the packet path for an internal service sourced packet destined for an external system, such as the reply packet to the one described in the previous paragraph. It exits the Local Machine and enters the Output(4) chains, which provides the firewalling functions. It then enters the RPDB for route processing and exits the system via the exit packet mangling and tagging stage, Post-Route(5).

Note that in all of these packet paths, ignoring the Mangle functions of the Pre/Post ROUTING chains, the packet never crosses through more than one packet filter chain and in all cases the packet gets processed by the RPDB. So all of the functions associated with the Policy Routing structures under the RPDB may be applied to the packet.

3.4 Summary

You have seen the fundamental triad of Policy Routing and how these elements are implemented within the Linux kernel through the RPDB. You then traced through the logic of the packet paths for the packet filters and the RPDB action locations. Now you understand the logic of the traffic flow for Policy Routing in Linux.

Chapter 4 will cover the usage and operation of the ip utility from Alexey Kuznetsov. This is the main utility implementing all of Policy Routing with the exception of TOS/QoS. Within the utility suite, however, is the tc utility, which performs all of the TOS/QoS functions of Policy Routing. After you have learned about the ip utility you will start to delve into hands-on experiences with Policy Routing by implementing a series of increasingly complex examples.

As you go through the examples and usages of the utilities try to see how the Triad operates in those situations. Remember:

Address	defines the location of a service.
Route	defines the location of the address.
Rule	defines the location of the route.

Return to Table of Contents

NEWS CONTENTS

• 20120405 : Configuring Multiple Default Routes in Linux ( Nov 19, 2007 , Darien Kindlund's Blog )
• 200102 : Multihomed Linux server just like my old NetWare... ( )

Old News ;-)

[Apr 05, 2012] Configuring Multiple Default Routes in Linux

Nov 19, 2007 | Darien Kindlund's Blog

Assume you have a Linux system with more than one network interface card (NIC) - say eth0 and eth1. By default, administrators can define a single, default route (on eth0). However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.

This can be a bit of a problem - especially when the two NICs share the same parent network and you're trying to preserve sane traffic flows. In a nutshell, this post will explain how you can ensure traffic going into eth0 goes out only on eth0, as well as enforce all traffic going into eth1 goes out only on eth1.

You've found the one post that actually explains this issue; your googling has paid off. You wouldn't believe how many advanced Linux routing websites out there explain how to route everything including your kitchen sink - yet fail to clearly explain something as simple as this.

As always, we'll explain by example. Assume the following:

• eth0 - 10.10.70.38 netmask 255.255.255.0
• eth0's gateway is: 10.10.70.254
• eth1 - 192.168.7.126 netmask 255.255.255.0
• eth1's gateway is: 192.168.7.1

First, you'll need to make sure your Linux kernel has support for "policy routing" enabled. (As a reference, I'm using a v2.6.13-gentoo-r5 kernel.)

During the kernel compilation process, you'll want to:

cd /usr/src/linux
make menuconfig
Select "Networking --->"
Select "Networking options --->"
[*] TCP/IP networking
[*] IP: advanced router
Choose IP: FIB lookup algorithm (FIB_HASH)
[*] IP: policy routing
[*] IP: use netfilter MARK value as routing key

Next, you'll want to download, compile, and install the iproute2 [1] utilities. (Most Linux distributions have binary packages for this utility.) Once installed, typing ip route show should bring up your system's routing table. Type man ip for more information about this utility, in general.

Speaking of which, assume the system's initial route configuration looks like this:

# netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.10.70.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.7.1 0.0.0.0 UG 0 0 0 eth1

So, basically, the system is using eth1 as the default route. If anyone pings 192.168.7.126, then the response packets will properly go out eth1 to the upstream gateway of 192.168.7.1. But what about pinging 10.10.70.38? Sure, the incoming ICMP packets will properly arrive on eth0, but the outgoing response packets will be sent out via eth1! That's bad.

Here's how to fix this issue. Borrowing the method from a really sketchy website [2], you'll first need to create a new policy routing table entry within the /etc/iproute2/rt_tables. Let's call it table #1, named "admin" (for routing administrative traffic onto eth0).

# echo "1 admin" >> /etc/iproute2/rt_tables

Next, we're going to set a couple of new entries within this "admin" table. Specifically, we'll provide information about eth0's local /24 subnet, along with eth0's default gateway.

ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin

At this point, you've created a new, isolated routing table named "admin" that really isn't used by the OS just yet. Why? Because we still need to create a rule referencing how the OS should use this table. For starters, type ip rule show to see your current policy routing ruleset. Here's what an empty ruleset looks like:

0: from all lookup local
32766: from all lookup main
32767: from all lookup default

Without going into all the boring details, each rule entry is evaluated in ascending order. The main gist is that your normal main routing table appears as entry 32766 in this list. (This would be the normal route table you'd see when you type netstat -anr.)

We're now going to create two new rule entries, that will be evaluated before the main rule entry.

ip rule add from 10.10.70.38/32 table admin
ip rule add to 10.10.70.38/32 table admin

Typing ip rule show now shows the following policy routing rulesets:

0: from all lookup local
32764: from all to 10.10.70.38 lookup admin
32765: from 10.10.70.38 lookup admin
32766: from all lookup main
32767: from all lookup default

Rule 32764 specifies that for all traffic going to eth0's IP, make sure to use the "admin" routing table, instead of the "main" one. Likewise, rule 32765 indicates that for all traffic originating from eth0's IP, make sure to use the "admin" routing table as well. For all other packets, use the "main" routing table. In order to commit these changes, it's a good idea to type ip route flush cache.

Congratulations! You're system should now properly route traffic to these two different default gateways. For more than 2 NICs, repeat the table/rule creation process as necessary.

Please provide comments, if you find any errors or have corrections to this post. I don't claim that this method will work for everyone; this information is designed primarily to preserve my sanity, when configuring routing on future multi-NIC Linux systems.

References:

[1] http://www.policyrouting.org
[2] http://www.linuxhorizon.ro/iproute2.html

Update: Here are some additional resources, that I have found useful.
http://lartc.org/howto/lartc.rpdb.multiple-links.html
http://linux-ip.net/html/routing-tables.html

Update: Apparently, OpenBSD also now supports multiple default routes through a new feature called the Virtual Routing Table:
http://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/

1. • Darien

   • Posted February 7, 2008 at 12:16 pm
   • Permalink
   • Reply
   RE: Failover Default Routes

   Here are some articles that describe what you're looking for:

   http://www.uwsg.iu.edu/hypermail/linux/net/0009.1/0027.html
   http://www.linux.com/articles/113988
   http://mailman.ds9a.nl/pipermail/lartc/2002q1/002667.html

   OpenBSD also has a daemon called "hoststated" that provides this type of capability:
   http://www.openbsd.org/cgi-bin/man.cgi?query=hoststated&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html

2. • Anonymous

   • Posted February 7, 2008 at 8:17 pm
   • Permalink
   • Reply
   Thank you very much for the links…

   gc_timeout did the trick for me as the default is 5 min which is too long for a route to fail over.

3. • Darien

   • Posted February 9, 2008 at 2:17 pm
   • Permalink
   • Reply
   RE: ARP Cache Timeout

   Try this link, for information about how to control how long entries stay in the ARP cache:

   http://www.ussg.indiana.edu/hypermail/linux/net/0012.2/0035.html

4. • Paul

   • Posted September 26, 2008 at 6:45 pm
   • Permalink
   • Reply
   i'm actually kind of curious – the most common, default situation is where eth0 is the interface that packets are routed through - heck, that's noted in the first sentence - so i'm wondering why you constructed your example with the reverse configuration – with eth1 as the default? i found it kind of confusing, in an otherwise most excellent article….

5. • Darien

   • Posted September 27, 2008 at 5:20 am
   • Permalink
   • Reply
   Sorry about the confusion; in my example, the Linux distribution I used in my example (Gentoo) seemed to always specify eth1 as the default route - when both eth0 and eth1 were present and active (and when automatic configurations were used).

   Let me ask you this, in your Linux distribution, when you have both an eth0 and an eth1 on the system with active links for both - does your distribution automatically select eth0 as the default route? I'm guessing it's pretty much random, as the standard defaults are to use DHCP for both eth0 and eth1, in which case, whichever interface gets the DHCP lease (first? or last?) will ultimately decide which default route actually gets used by the system.

   I hope this explains where I was coming from.

6. • Paul

   • Posted September 28, 2008 at 5:40 am
   • Permalink
   • Reply
   oh, okay, that makes sense. i'm always running servers with fixed IP's, so DHCP never enters into the picture.

   for what i'm attempting, i actually found that going with bonded interfaces worked best – using a bond0 device and making both interfaces capable of "believing" that they're the server's IP, and the only path out. thus if one interface goes offline, traffic can still pass.

7. • Amit Agarwal

   • Posted October 2, 2008 at 6:42 am
   • Permalink
   • Reply
   I have a slightly different requirement.
   I have internet access both over WLAN and over DSL. I use the DSL for VoIP setup and WLAN for internet. But sometimes the WLAN or the DSL goes out of service. In this caseI want to use both the services – Internet and my VoIP setup over the either WLAN or DSL whichever is active.
   How could I do this?

   Thanks,
   Amit
   [email protected]

8. • Darien

   • Posted October 2, 2008 at 12:50 pm
   • Permalink
   • Reply
   Hi Amit,

   See my earlier comment about "Failover Default Routes":

   http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-10

   – Darien

9. • Darien

   • Posted November 24, 2008 at 3:55 pm
   • Permalink
   • Reply
   Each Linux distribution is slightly different. Some distributions offer a way to save these configurations within a configuration file under /etc to be used upon reboot.

   Other distributions require you to add the "echo" and "ip rule" commands into a script file that gets called as one of the last commands when the system boots up.

   For example, on Ubuntu Linux, there is a script file called "/etc/rc.local" that you can add all the "echo" and "ip rule" commands into, that will apply these configurations upon complete server bootup.

   However, please be aware that if you have other network services that require the multiple default routes to be set up BEFORE those services can successfully load (e.g., an automatic VPN/IPSec tunnel), then you'll have to create an rc.d init script, similar to those found in the "/etc/init.d" directory.

   The short answer:
   - Try adding the commands to "/etc/rc.local".
   - If other services fail, then make a copy of a simple script in "/etc/init.d"; then edit that script to reflect the commands needed to be run at boot time; then register that script to be started upon bootup ("man update-rc.d", for Ubuntu/Debian Linux).

   Hope this helps,
   – Darien

10. • Lasy

    • Posted November 25, 2008 at 9:28 pm
    • Permalink
    • Reply
    Hello,
    This is a good article, but when you say : "However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.", car you please explain or give examples where we will or can receive traffic on eth1 if we have eth0 as a gateway.

    Thanks
    Lasy

11. • Darien

    • Posted November 26, 2008 at 4:25 pm
    • Permalink
    • Reply
    Sure,

    Lasy, when you set a single default route, then that route is relevant for *all traffic outside your local subnet*.

    Let's say you have a public class-C subnet of 128.253.1.0/255.255.255.0 and eth0 is 128.253.1.1 while eth1 is 128.253.1.2 and your default route is through eth0.

    Then, let's say someone from a completely different subnet (from 128.5.1.1, for example) pings your ETH1 IP address (128.253.1.2). The ICMP request packets will arrive on eth1; however, when your system sends out the ICMP response packets, *because the destination is outside your local subnet*, these packets will go out on eth0, since that is your default route.

    The point of the article is that if you have two or more network interfaces and have only one default *route* set on the OS, then *for all non-local-subnet traffic*, all outgoing packets will go out on the interface marked as the default route - unless you follow the steps listed in this article.

    – Darien

12. GB

    • • Darien

      • Posted March 27, 2009 at 9:29 pm
      • Permalink
      • Reply
      Hi Ghostbuster,

      Yes, IP_ROUTE_FWMARK was removed from linux v2.6.20 and later releases. See this link for details:
      http://cateee.net/lkddb/web-lkddb/IP_ROUTE_FWMARK.html

      Presumably, the feature still exists, but it's probably named something else (or perhaps merged with some other feature).

      Hope this helps,
      – Darien

13. • Ghostbuster

    • Posted March 28, 2009 at 6:46 am
    • Permalink
    • Reply
    Hi Darien,

    Thanks for that pointer.

    I'm thinking it's the MARK command in Netfilter these days.

    http://www.linuxtopia.org/Linux_Firewall_iptables/x4368.html

    I've done some qdisc stuff so hopefully it won't be too alien.

    GB

14. • Ghostbuster

    • Posted March 28, 2009 at 6:58 am
    • Permalink
    • Reply
    Okay, scrap that last comment

    I'll try this instead hoping the support is all handled by iproute2 these days:

    http://www.physics.umd.edu/pnce/pcs-docs/Glue/linux-route-hack.html

    GB

15. • Darien

    • Posted May 29, 2009 at 4:48 pm
    • Permalink
    • Reply
    Hi Anderson,

    There is no specific file, as all the commands I've listed are performed from the command line. If you're looking to run these commands every time your OS boots, then you can put them all in /etc/rc.local (for Ubuntu Linux).

    Second, 'netstat -rn' can't show you "policy routes" properly. So even after you run the policy routing commands, it's likely 'netstat -rn' will still show you the same thing before and after the policy routes are in place.

    If you're looking to verify that policy routes are working properly, run the "ip rule show" before and after setting up policy routes. If you want more detailed information, see the man pages for the "ip" command, by typing "man ip".

    – Darien

16. • Darien

    • Posted May 30, 2009 at 11:20 pm
    • Permalink
    • Reply
    Hi Anderson,
    You can define your eth1 gateway in the set of commands you enter into /etc/rc.local

    The "ip rule" commands Listed in the article will properly configure your eth1 gateway, so you won't need to define it within /etc/sysconfig/network

    – Darien

17. • nobodyfamous

    • Posted June 17, 2009 at 6:46 am
    • Permalink
    • Reply
    What do you change if both NICs are on the same network segment, using the same gateway IP? I tried following the steps above but I was left with all traffic going out eth0 while some went in eth1.

    • • Darien

      • Posted June 17, 2009 at 2:11 pm
      • Permalink
      • Reply
      Hi nobodyfamous,
      If both NICs are on the same network segment, the policy routing isn't going to help you. The problem has to do with Linux kernel's TCP/IP stack employing short-circuit routing.

      Specifically, this problem is solved at a different layer than routing. What you're looking for is "Link Aggregation" (aka. making multiple NICs in your server act as one large NIC). There is an entire page dedicated to the concept, available here:
      http://en.wikipedia.org/wiki/802.3ad

      In order to get it to work, though, you'll need to make sure your upstream switch/router supports the 802.3ad protocol (most recent managed switches do support it; however, if you have an unmanaged switch, probably not).

      – Darien

      • • Anonymous

        • Posted June 18, 2009 at 4:27 pm
        • Permalink
        • Reply
        Yeah, bonding was the initial goal, but we don't have access to the switches in this case, so it had to be solved at the host level.

        We were able to make it work following a modified form of your instructions, FWIW:

        In this example, the gateway is 192.168.0.1 and the server's IPs are 192.168.0.211 and 192.168.0.212 on eth0 and eth1 respectively:

        printf "1\tuplink0\n" >> /etc/iproute2/rt_tables
        printf "2\tuplink1\n" >> /etc/iproute2/rt_tables

        ip route add 192.168.0.211/32 dev eth0 src 192.168.0.211 table uplink0
        ip route add default via 192.168.0.1 dev eth0 table uplink0
        ip rule add from 192.168.0.211/32 table uplink0
        ip rule add to 192.168.0.211/32 table uplink0

        ip route add 192.168.0.212/32 dev eth1 src 192.168.0.212 table uplink1
        ip route add default via 192.168.0.1 dev eth1 table uplink1
        ip rule add from 192.168.0.212/32 table uplink1
        ip rule add to 192.168.0.212/32 table uplink1

        By the way, in the second paragraph you say that the solution is applicable to cases where the two NICs are on the same network, that's why I followed your lead on this.

18. • Tekknogenius

    • Posted July 2, 2009 at 12:01 am
    • Permalink
    • Reply
    This is extremely helpful. Thank you.

    One question about routing in general, is there a way to have subnets share a common default gateway?

    • • Darien

      • Posted July 2, 2009 at 12:34 am
      • Permalink
      • Reply
      Hi Tekknogenius,
      If we're talking about two subnets sharing a common, physical gateway (such as a router), then yes, but that common gateway must have a valid IP address on each subnet. For example, assume you have two subnets:

      192.168.0.0/24
      192.168.1.0/24

      You can then configure your router to have the IPs of:
      192.168.0.254 and 192.168.1.254 for use by members of their respective subnets.

      Bottom line is that the gateway must have an address in each subnet; otherwise, you would have to figure out some other way to reach the gateway's IP address (e.g. static ARP entry if on the local LAN)

      – Darien

19. • JohnnyKnoxville

    • Posted July 7, 2009 at 1:46 pm
    • Permalink
    • Reply
    Hi – i have another slightly different goal. My Linux box has an eth0 (connected via a default gw to the internet) and a on-demand ppp0 (gprs) connection. My goal is to programmatically create sockets via both interface at the same time. I think i am stuck setting up the correct routes for this – do you know how to handle this issue?

    • • Darien

      • Posted July 7, 2009 at 1:57 pm
      • Permalink
      • Reply
      Hi Johnny,
      See my earlier comment as I've already addressed this issue:
      http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-10

      – Darien

      • • JohnnyKnoxville

        • Posted July 15, 2009 at 12:56 pm
        • Permalink
        • Reply
        Thanks for your quick reply. I'm not sure failover routing is the correct issue here, as both routes should be up all the time. My app decides on various criteria which connection should be used. I had read som article, whiches in short says: What goes out on one device, should comme in on that again. Is this the right idea? And how can I realize that?

        • • Darien

          • Posted July 15, 2009 at 3:20 pm
          • Permalink
          • Reply
          Hi Johnny,
          Okay, thanks for the clarification. I had assumed when you mentioned 'on-demand', that you somehow wanted to switch from eth0 to ppp0 when eth0 failed. Instead, you want both eth0 and ppp0 alive all the time.

          In that case, just replaced 'eth1′ with 'ppp0′ in the example and you can still use policy routing to accomplish what you're looking for.

          The real challenge, is if ppp0′s network configuration changes constantly. If the IP address, subnet, and route of ppp0 constantly changes, then you need to somehow have logic to detect when these changes occur and update your policy routes accordingly.

          In that case, you'll probably want to research DHCP notification code. Depending on your Linux distribution, there are scripts which execute when your network changes, which you could use to also update your corresponding policy routes.

          – Darien

20. • DevizioN

    • Posted September 8, 2009 at 10:29 am
    • Permalink
    • Reply
    Hey,

    Great article, helped me loads compared to the other sites were they just skimm abit on the top.

    Thou, having problem adapting it to my setup.

    I currently have 2 nic's on my server that is acting as router/gateway.

    1 of the physical nic's has 5 virtuell interfaces (g1-5), g1 goes to the router itself as it also goes as a webserver. The rest are nated straight to nic2(which has no virtuell interfaces). As i have 4 computer, g2-5 are nated 1:1 towards the inside local ip:s 10.10.1.X.

    I have a simple iptables rule to fix the NAT part, and i had it working "sortoff" since everything was working except that i could not reach the servers/pc's from the outside, they could only talk from the inside->out (torrents etc works flawlessly thou).

    Any hints? If you got a mail or something it would be great to get some advice..

    • • Darien

      • Posted September 8, 2009 at 3:23 pm
      • Permalink
      • Reply
      Hi DevizioN,

      It sounds like you have one-way NAT working and you want bi-directional NAT. From your post, it sounds like g1 goes to the router, as well as nic2 goes directly to the router … is that correct?

      Regardless, it sounds like setting up multiple default routes correctly will NOT fix your NAT problem… I would recommend you try and fix your NAT problem first, and then setup the multiple default routes.

      To be very specific, with your current NAT configuration, try configuring your server with only ONE upstream gateway and fix your NAT so that it's working in both directions (insideout).

      Once you get a single route working, then add your second upstream gateway. If your bi-directional NAT breaks and no longer works correctly, then let me know and we can troubleshoot further (as configuring multiple default routes may help you).

      The general approach is: reduce the complexity of your configuration, verify each piece works first (e.g., NATing, virtual interfaces), and THEN verify the total configuration works. Otherwise, it can be a nightmare to troubleshoot multiple unknown pieces at the same time.

      – Darien

21. example

    10.10.70.0/24
    10.10.71.0/24

    ip rules add and from are with /32 or /24?

    example

    ip rule add from 10.10.70.38/32 table admin
    ip rule add to 10.10.70.38/32 table admin
    ip rule add from 10.10.71.38/32 table admin2
    ip rule add to 10.10.71.38/32 table admin2

    or

    ip rule add from 10.10.70.38/24 table admin
    ip rule add to 10.10.70.38/24 table admin
    ip rule add from 10.10.71.38/24 table admin2
    ip rule add to 10.10.71.38/24 table admin2

    I don't hunderstand why you use all 32 bits.

    Thanks

22. • Mazzer

    • Posted October 16, 2009 at 12:28 pm
    • Permalink
    • Reply
    Thanks for a great article. This looks like kinda what I need. However, my situation is a bit different (aren't they all ). Anyway, I am basically, trying to setup openVPN on my Ubuntu 8.10 server. I got 2 NICs in my server. eth0 and eth1. eth0 is the external NIC while eth1 is the internal one (the LAN. eth0 is the default route. VPN connects through eth0 and I can ping my server and I can ping eth1, however I can't ping anything behind eth1 (which is my local LAN). From the server console itself I can ping my internal LAN. I think the problem lies with traffice coming via eth0 and since the default route is eth0 it only goes through eth0. So if I ping anything which is behind eth1 it is routed via eth0 and not eth1. Is it possible to route all VPN traffic meant for eth1 through eth1 even though it's not the default route?

    Does this make any sense or am I making you guys as confused as myself?

    Thanks for your help in advance.

    • • Darien

      • Posted October 16, 2009 at 4:40 pm
      • Permalink
      • Reply
      Hi Mazzer,

      1) As a sanity check, are you able to ping your eth1 network if you TURN OFF your VPN? I just want to make sure that the problem isn't more fundamental.

      2) On your server, once you have your VPN setup, you actually have THREE different interfaces: eth0, eth1, and (probably something like vpn0 - check your 'ifconfig -a' for details). Essentially, you have a third 'virtual' interface, representing the effective IP address of your server, once connected on the *inside* of your VPN network.

      Here's what I need to know:
      - The IP address and netmask of your eth1 network.
      - The IP address and netmask of your *vpn0* network (see above for what I mean).

      Specifically, I believe that perhaps your eth1 network and your vpn0 network may have conflicting IP addressing, which is causing you to lose connectivity to your eth1 network once a VPN is established. However, I need more information in order to accurately say that this is the problem.

      – Darien

23. • prasad

    • Posted October 16, 2009 at 6:22 pm
    • Permalink
    • Reply
    I used the same kinda configuring (but created two tables instead of one) one per interface. This makes sure i get the response to through the right interface.

    # Assign IP address on the interfaces eth0 and eth1
    ip route add 10.1.0.0 dev eth0 src 10.1.0.250
    ip route add 10.0.0.0 dev eth1 src 10.0.0.250

    # IP route and default gateway for eth0 in the table 100
    ip route add 10.1.0.0 dev eth0 src 10.1.0.250 table 100
    ip route add default via 10.1.0.1 table 100
    # IP route and default gateway for eth1 in the table 101
    ip route add 10.0.0.0 dev eth1 src 10.0.0.250 table 101
    ip route add default via 10.0.0.1 table 101

    # Assign the rule to corresponding particular table
    ip rule add from 10.1.0.250 table 100
    ip rule add from 10.0.0.250 table 101
    ip route flush cache

    Now, i can ping the box to the corresponding box from behind the
    gateway. Everything works fine according to my requirement.

    Now if i use the ping utility (from the box) with a destination address not in 10.1.X.X or in 10.0.X.X network using both the interfaces (or the
    ip-address) it works
    ping -I eth0 some-address (works)
    ping -I eth1 some-address (works)
    ping some-address (not working) with "network is unreachable" as the error

    Now this force me to use the default route on the main table which i
    thought was not neccessary using below command
    ip route add default via 10.1.0.1.

    Since with out any interface or ip-address explicity given, ping
    should have chosen the source ip-address as either 10.1.0.250 or
    10.0.0.250. If so , rule should have hit in routing table 100 or table
    101. But just wondering why it is hiting the main routing table. This
    forces me to have a two gateway programming option for each interface
    and another default gateway to select among two. Not sure why..

    • • prasad

      • Posted October 16, 2009 at 6:24 pm
      • Permalink
      • Reply
      Also how do we use the same concept in the DHCP enabled network. DHCP client running on both the interfaces. DHCP clients always deals with the main routing table (instead of the custom created routing table).

      • • Darien

        • Posted November 10, 2009 at 3:30 pm
        • Permalink
        • Reply
        Handling DHCP leases is tough. You'll need to use a DHCP client that supports custom extensions, so that you can call a script (which you create) that will setup the appropriate policy routes.

    • • Darien

      • Posted November 10, 2009 at 3:29 pm
      • Permalink
      • Reply
      What does your 'netstat -anr' and 'ip rule show' look like?

24. • Mazzer

    • Posted October 20, 2009 at 1:35 pm
    • Permalink
    • Reply
    Hi Darien,

    Thank you for your quick reply. I was away from my computer for a while so couldn't reply.

    As per your questions, I'll answer the ones I can right now.

    1) I can ping eth1.

    2) I know what you mean about the 3rd virtual interface. Sorry for not providing more information. I do have one and it's a tun interface (tun0). I'm using routing and not bridging.

    The IP address of eth1 is 10.10.2.102, netmask is 255.255.255.0 so thats the 10.10.2.0 subnet.

    The details about tun0 I'll get for you when I go home tonight.

    I think the problem lies between the virtual Interface and eth1. From the VPN server I can ping both eth1 and eth0 and everything behind those interface. I can ping everything behind the eth1 subnet (internal network) and I can ping everyhting behind the eth0 subnet (the internet). When I'm connect via VPN to the server. I can ping both eth0 and eth1 but can't ping anything behind eth1. Once connected via VPN, I can SSH into the VPN server (through the VPN connection) and one connected to the server console. I can ping everything behind eth1. I guess thats no surprise as I'm am then doing the pinging from the console. So as I said, I think the problem lies with the virtual interface having the default route of eth0 and so the pings arte going through eth0 even though they meant for eth1.

    Phew, I can't believe I wrote all that down . Thanks for your help though.

25. • Mazzer

    • Posted October 20, 2009 at 2:33 pm
    • Permalink
    • Reply
    Also Just so you know, I'm using OpenVPN as my VPN server and I've setup the redirect gateway option as well.

26. • Mazzer

    • Posted October 21, 2009 at 10:55 am
    • Permalink
    • Reply
    Got the details for the tun0 interface.

    IP Address: 10.30.2.6
    Subnet Mask: 255.255.255.252
    Default Gateway: 10.30.2.5
    DHCP Server: 10.30.2.5
    DNS Servers: 10.10.2.11

    The 10.10.2.0 subnet is the internal eth1 network so looks like the VPN server is pushing the DNS server to the client correctly.

27. • madinc

    • Posted November 10, 2009 at 1:10 pm
    • Permalink
    • Reply
    Good article.
    Forgive my ignorance, but what's the purpose of the "to" rule, ie

    ip rule add to 10.10.70.38/32 table admin

    that would be used only from incoming packets, and even in that case it doesn't seem very useful. But I'm sure I'm missing something obvious.

    Thank you.

    • • Darien

      • Posted November 10, 2009 at 3:34 pm
      • Permalink
      • Reply
      madinc,

      The "to" rule is to tell the tcp/ip stack that for all packets where the destination is 10.10.70.38/32, then route those packets to the "admin" interface. This is important for STATEFUL flow processing. When Linux constructs the appropriate reply (e.g., a TCP ACK), it will know which policy route to use, when sending the reply. Otherwise, without this rule, you'll find that your ACKs will get sent using the default route in the main table.

      • • madinc

        • Posted November 10, 2009 at 5:08 pm
        • Permalink
        • Reply
        Sorry for being slow to understand, but for ACK reply packets the "to" (destination IP) will NOT be 10.10.70.38, but rahter the IP of the host the ACK is directed to.
        So if you want to use a specific rule to route those packets out, you probably want a rule that matches on *from* 10.10.70.38, rather than "to".

        I still fail to see how that "to" rule can match anything other than incoming packets.

        Thank you!

        • • Darien

          • Posted November 10, 2009 at 6:13 pm
          • Permalink
          • Reply
          Hi madinc,

          The "to" rule is primarily important when BOTH of your ethernet interfaces are on the SAME SUBNET. If both interfaces are on separate subnets with NO possibility of getting overlapping traffic, then the "to" rule is less of an issue.

          However, when both interfaces are on the same subnet, then the "to" rule tells Linux which packets to ignore or listen to if each interface is getting duplicate traffic.

          • • Darien

            • Posted November 10, 2009 at 6:14 pm
            • Permalink
            Just to clarify, when I mean SAME SUBNET, I mean that both ethernet interfaces are connected to the same upstream hub/switch/router, where both ethernet interfaces receive copies of all network traffic destined to any of the interfaces.

28. • madinc

    • Posted November 11, 2009 at 9:25 am
    • Permalink
    • Reply
    When both ethernet interfaces are on the same subnet, only the one whose IP address is the destination IP address of the incoming packet will get it.
    It's like having many hosts in a LAN: only the destination host will reply to ARP and thus get the packet, despite all other hosts being in the same subnet.

    Forgive me, but I still don't see where the "to" rule comes into play

    Thank you.

    • • madinc

      • Posted November 11, 2009 at 11:17 am
      • Permalink
      • Reply
      Hm, on second thought…with Linux's weak host model, both interfaces will reply. So yeah, now I see where the "to" can be useful.

      Thank you and sorry for the noise.

      • • Darien

        • Posted November 11, 2009 at 3:39 pm
        • Permalink
        • Reply
        No problem; glad to help. In general, when getting real answers, I'd recommend actually testing against an actual Linux system, since a lot of these nuances could change, depending on which kernel patches (from which Linux distributions) are loaded on your test machine.

        On numerous occasions, I've found that "what should happen" is largely different than "what really happens".

29. • wcole

    • Posted November 23, 2009 at 5:15 pm
    • Permalink
    • Reply
    I tried using the steps that you used to configure Multiple Routes and I ran into an error when I ran the command "ip route add default vi 10.141.50.50 dev eth2 table ccsadmin". The error that I received was "error: either "to" is duplicate, or "10.141.50.50″ is a garbage. I'm not sure what steps do I need to take to solve this problem that I'm having. I'm running CentOS 4.7 kernel version 2.6.9-78

    • • Darien

      • Posted November 23, 2009 at 10:17 pm
      • Permalink
      • Reply
      Umm, I think you mis-typed the command:

      "ip route add default vi 10.141.50.50 dev eth2 table ccsadmin"

      Should be:

      "ip route add default via 10.141.50.50 dev eth2 table ccsadmin"

      Specifically, the "vi" should be "via".

30. • Richard Ford

    • Posted December 14, 2009 at 9:58 am
    • Permalink
    • Reply
    Hmmm…

    I keep getting:

    [root@bjwww001 ~]# ip route add 192.168.20.0/24 dev eth1 src 192.168.20.2 table hkvpn
    RTNETLINK answers: Invalid argument

    All my ifcfgXX files are correct and have broadcast and network IP's defined too….

    • • Darien

      • Posted December 14, 2009 at 2:26 pm
      • Permalink
      • Reply
      Did you enable "policy routing" in the Linux kernel and install the iproute2 utilities? (Both were mentioned above in the article.)

      • • Richard Ford

        • Posted December 14, 2009 at 3:11 pm
        • Permalink
        • Reply
        iproute yes… policy routing – will have to check. This is a stock CentOS 5.4 64 bit. WIll I need to recompile the kernel or is there a kernel flag I can set in sysctl?

        • • Darien

          • Posted December 14, 2009 at 3:29 pm
          • Permalink
          • Reply
          Policy routing requires a kernel recompilation, if it's not enabled by default. There's no way to dynamically enable/disable it via sysctl.

          Not sure about CentOS, but many Linux distributions will store a copy of the actual Linux kernel config in /proc/config.gz, so you could try inspecting that kernel config (if it exists) to see if Policy Routing has been defined. Otherwise, you'll have to do a recompile.

          • • Richard Ford

            • Posted December 15, 2009 at 4:04 am
            • Permalink
            OK. Yes the .config is included. Have't recompiled a kernel for probably 10 years now. Oh well – like riding a bike….

          • • Richard Ford

            • Posted December 15, 2009 at 11:06 am
            • Permalink
            This works too for CentOS with no kernel recompile!

            http://www.sysadminsjourney.com/content/2009/04/15/doing-simple-source-policy-routing-centos

31. • edscott

    • Posted February 4, 2010 at 11:16 pm
    • Permalink
    • Reply
    Worked great for ssh, but borked my apache server. I don't know why but requests arriving on one device to the apache server will be sent out on the other one. And with the ip rule will try to go the wrong way. Maybe its an apache bug (or feature?)

    • • Darien

      • Posted February 5, 2010 at 6:16 am
      • Permalink
      • Reply
      Do you have Apache listening on all interfaces or just one interface? You may need to adjust this setting to get your desired effect.

      http://httpd.apache.org/docs/2.2/bind.html

32. • Marcus

    • Posted February 24, 2010 at 4:09 pm
    • Permalink
    • Reply
    Since you were using Gentoo:

    How do you setup the IP address in the "admin" routing table during boot? How did your modify your "/etc/conf.d/net.eth0″-file?

    The default gateway can be added with "routes_eth0=( "default via 10.10.70.254 table admin" )

    The rules can be added in the "postup" section of the /etc/conf.d/net; but I simply cannot bring Gentoo to add the IP address in the other table. It either adds it in the default table; or throws a syntax error.

    • • Darien

      • Posted February 25, 2010 at 2:50 am
      • Permalink
      • Reply
      Instead of trying to figure out the proper way to add the routing table commands to the network startup scripts, just add them to your /etc/rc.local script (or equivalent). Essentially, you want to use the same absolute commands as listed in the examples (in the article) and write them to a script that will run after *all* automatic networking scripts are ran. On most systems, commands written to /etc/rc.local are literally the last commands executed before the system finished bootup and presents a login prompt to the user.

      • • Anonymous

        • Posted February 25, 2010 at 10:57 am
        • Permalink
        • Reply
        That would be "/etc/conf.d/local.start" on a Gentoo system.

        While it does work, of course, there is a problem if you restart the interface with an "/etc/init.d/net.eth0 restart", since the routing table will be empty afterwards.

        Alas, at least it works now. Thanks for pointing me in the right direction.

        • • Marcus

          • Posted July 6, 2011 at 8:38 am
          • Permalink
          • Reply
          Replying to my own comment:

          Using OpenRC with baselayout2 the /etc/conf.d/net.eth0 script would look like this:

          config_eth0="192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255″
          routes_eth0="default via 192.168.1.1
          192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.150 table dmz
          default via 192.168.1.1 dev eth0 table dmz"
          dns_domain_eth0="localdomain.local"
          dns_servers_eth0="dns.ip.1 dns.ip.2″
          metric_eth0="0″

          Modify to your own pleasure.

33. • Al Evans

    • Posted February 28, 2010 at 8:33 pm
    • Permalink
    • Reply
    Thanks so much for this! I had no idea, until I watched it with tcpdump, that a packet coming in on one interface would get an (attempted) response on another.

    The fix you give worked like a charm.

    –Al Evans

34. • Andy

    • Posted March 25, 2010 at 11:27 pm
    • Permalink
    • Reply
    Wow, this worked great! Really cool. Does anybody know if something like this can be done under Windows?

35. • mdewet

    • Posted April 12, 2010 at 10:30 am
    • Permalink
    • Reply
    Hi,
    Thanks for the tutorial, it helped a lot! One thing, is there a way to make the routing tables persistent? Or will the newly configured tables still be there after a system reboot?

    Regards,
    mdewet

    • • Darien

      • Posted April 12, 2010 at 1:41 pm
      • Permalink
      • Reply
      Hi mdewet,

      The policy routing settings should persist across reboots. If, for whatever reason, they do not, you can usually add these settings to /etc/rc.local (or equivalent), so that these settings are applied at the very end of system boot.

      – Darien

36. • Joel

    • Posted May 1, 2010 at 3:25 pm
    • Permalink
    • Reply
    Great article. Much appreciated!!!

37. • rodmar

    • Posted May 7, 2010 at 5:45 pm
    • Permalink
    • Reply
    Hi Darien,

    First of all great post.
    I just have one question. Is it possible to do something similar but with two interfaces connected to the same network? For instance two wireless cards connected to the same access point.

    • • Darien

      • Posted May 7, 2010 at 9:54 pm
      • Permalink
      • Reply
      Hi rodmar,

      Yes, the same technique can be used on 2 wireless NICs connected to the same access point - Linux treats those interfaces the same when performing policy routing.

      – Darien

38. • dir

    • Posted May 13, 2010 at 9:28 am
    • Permalink
    • Reply
    Thanks for the tutorial. Does this also work with configuring multiple default routes on vlan interfaces?

    • • Darien

      • Posted May 13, 2010 at 12:39 pm
      • Permalink
      • Reply
      Hi,
      Yes, this technique can be applied to vlan interfaces as well, because Linux treats those interfaces exactly the same as physical interfaces - just substitute eth0/eth1 with the corresponding vlan interfaces you want to perform policy routing on.
      – Darien

39. • Fred

    • Posted May 16, 2010 at 11:59 pm
    • Permalink
    • Reply
    Bless you, O Darien, for this gem on a subject which has stumped not only me but all who I have hit up for the solution, til now!

40. • Fred

    • Posted May 17, 2010 at 12:00 am
    • Permalink
    • Reply
    Bless you, O Darien, for this gem on a subject which has stumped not only me but many who I have hit up for the solution!

41. • Anonymous

    • Posted May 27, 2010 at 2:31 pm
    • Permalink
    • Reply
    Classful routing hasn't been used since 1994. The fact that the mask is included means it is a classless network. So stop saying 'class C', just because the Class C default mask was 255.255.255.0. It is simply a /24 subnet.

    • • Darien

      • Posted May 27, 2010 at 3:13 pm
      • Permalink
      • Reply
      Done.

42. • Will

    • Posted June 3, 2010 at 5:21 pm
    • Permalink
    • Reply
    Darien, Thnak you for this info..

    Do i have to remove the gateway in the interface file for this to work???

    • • Darien

      • Posted June 4, 2010 at 4:00 am
      • Permalink
      • Reply
      Hi Will,
      No, you don't need to remove the gateway in the interface file for this to work. Essentially, you want to perform the policy route configuration *after* basic networking (including setting the default gateway) has been done. This is why I recommend setting the policy route in /etc/rc.local, so that it is the *last* thing initialized, which guarantees the settings will not conflict with another daemon upon boot up.
      – Darien

      • • Will

        • Posted June 4, 2010 at 12:47 pm
        • Permalink
        • Reply
        Can you look at this and see what I'm doing wrong??? I would gladly pay you for your time.
        We have 2 NIC cards and when I try to set the eth1 gatway we lose SHH to the server.
        ETH0 is for (SSH<WEB<ECT.)
        ETH1 is for SIP traffic only
        ifconfig –a
        eth0 Link encap:Ethernet HWaddr 00:1c:23:d0:0b:2c
        inet addr:192.168.1.69 Bcast:192.168.1.255 Mask:255.255.255.0
        inet6 addr: fe80::21c:23ff:fed0:b2c/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:6215 errors:0 dropped:0 overruns:0 frame:0
        TX packets:59877 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:908397 (887.1 KiB) TX bytes:13233015 (12.6 MiB)
        Interrupt:16 Memory:f8000000-f8012100

        eth1 Link encap:Ethernet HWaddr 00:1c:23:d0:0b:2e
        inet addr:192.168.3.70 Bcast:192.168.3.255 Mask:255.255.255.0
        inet6 addr: fe80::21c:23ff:fed0:b2e/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:271 errors:0 dropped:0 overruns:0 frame:0
        TX packets:123 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:181558 (177.3 KiB) TX bytes:13875 (13.5 KiB)
        Interrupt:16 Memory:f4000000-f4012100

        lo Link encap:Local Loopback
        inet addr:127.0.0.1 Mask:255.0.0.0
        inet6 addr: ::1/128 Scope:Host
        UP LOOPBACK RUNNING MTU:16436 Metric:1
        RX packets:197 errors:0 dropped:0 overruns:0 frame:0
        TX packets:197 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:68135 (66.5 KiB) TX bytes:68135 (66.5 KiB)

        vi interfaces
        auto lo
        iface lo inet loopback
        # The primary network interface[m
        auto eth0
        iface eth0 inet static
        address 192.168.1.69
        netmask 255.255.255.0
        gateway 192.168.1.10

        auto eth1
        iface eth1 inet static
        address 192.168.3.70
        netmask 255.255.255.0
        #gateway 192.168.3.11 ß--When I try to use this line it locks me out oh SSH

        ip rule show
        0: from all lookup local
        32759: from 192.168.3.70 lookup sip
        32760: from all to 192.168.3.70 lookup sip
        32763: from all to 192.168.1.69 lookup admin
        32764: from 192.168.1.69 lookup admin
        32766: from all lookup main
        32767: from all lookup default

        ip route show
        192.168.3.0/24 dev eth1 proto kernel scope link src 192.168.3.70
        192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.69
        gla

        • • Darien

          • Posted June 4, 2010 at 9:01 pm
          • Permalink
          • Reply
          Hi Will,
          Your last post looks like some information was cut off, but I think I have enough information to help you.

          1) In /etc/interfaces, do NOT define the gateway for eth1. You are correct that this declaration will cause problems. So, remove the line:
          "gateway 192.168.3.11″ completely from /etc/interfaces.

          2) It looks like you're trying to define TWO (2) policy tables: "sip" and "admin". You do NOT need to do this. Please remove *both* "sip" and "admin" tables, and clear out all the custom "ip rule" and "ip route" commands you have defined.

          3) Once you've down that, reboot your system and verify that there are NO custom "ip rule show" and "ip route show" commands listed, and that only the eth0 default gateway is showing up in "netstat -anr".

          4) THEN, type the following commands:

          echo "1 admin" >> /etc/iproute2/rt_tables

          ip route add 192.168.3.0/24 dev eth1 src 192.168.3.70 table admin
          ip route add default via 192.168.3.11 dev eth0 table admin

          ip rule add from 192.168.3.70/32 table admin
          ip rule add to 192.168.3.70/32 table admin

          ip route flush cache

          Hope this helps,
          – Darien

43. • Ed

    • Posted July 5, 2010 at 11:34 am
    • Permalink
    • Reply
    Great article – worked like a charm.
    Thanks very much!

44. • route66

    • Posted August 2, 2010 at 3:41 am
    • Permalink
    • Reply
    Thanks very much for the excellent info!

    However there was an error for me for the route setting for the default gateway, which I tried to set to same as in the /etc/network/interfaces file, but from an online IP-calculator I found that the gateway should be the first usable host address from the range of machines on the sub network. After changing to that IP, it worked like a charm.

45. • Scheidel21

    • Posted September 2, 2010 at 2:08 pm
    • Permalink
    • Reply
    Many thanks for this I have been trying to consolidate some servers and I was combining our Proxy box with our openvpn box for a different subnet. I decided to use Untangle for the proxy as it was a quick easy set up. However, I knew that I would have to run my own instance of openVPN as Untangle only does routed VPNs and I needed a bridged VPN. All setup fine, but then when I pinged to the openVPN bridged interface through a second router from a PC on my LAN which is the one subnet being proxied I was getting no response, because it was killing them, I looked for a way to route the traffic back out the same interface it came in one for a couple of weeks and even posted questions about this at some forums, but your web page here finally provided me the information I needed to do it. I hope you don't mind I have resolved the questions I asked on the forums and linked back to your article. Many thanks.

46. • Dave

    • Posted November 23, 2010 at 11:42 pm
    • Permalink
    • Reply
    Darien – thanks this is very helpful!
    nobodyfamous and howard refer to same subnet and same gatway with 2 (or multiple) NICs. But would anything change in your precription if I want 2 gateways (10.0.0.1 and 10.0.0.2) with 2 NICS (10.0.0.11 and 10.0.0.22) on the same subnet. (This is different from link aggregation).

    To stay with your example – what if eth0 has ip 192.168.7.127
    and gateway 192.168.7.1

    • • Dave

      • Posted November 23, 2010 at 11:46 pm
      • Permalink
      • Reply
      oops I shouls have said:

      To stay with your example – what if eth0 has ip 192.168.7.127
      and gateway 192.168.7.2

      • • Darien

        • Posted November 24, 2010 at 1:07 am
        • Permalink
        • Reply
        Hi Dave,
        Take a look at the comments posted right after nobodyfamous:
        http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-102

        So, let's assume:
        eth0 – 192.168.7.127 netmask 255.255.255.0
        eth0′s gateway is: 192.168.7.2
        eth1 – 192.168.7.126 netmask 255.255.255.0
        eth1′s gateway is: 192.168.7.1

        Here are the corresponding commands:
        printf "1\tuplink0\n" >> /etc/iproute2/rt_tables
        printf "2\tuplink1\n" >> /etc/iproute2/rt_tables

        ip route add 192.168.7.127/32 dev eth0 src 192.168.7.127 table uplink0
        ip route add default via 192.168.7.2 dev eth0 table uplink0
        ip rule add from 192.168.7.127/32 table uplink0
        ip rule add to 192.168.7.127/32 table uplink0

        ip route add 192.168.7.126/32 dev eth1 src 192.168.7.126 table uplink1
        ip route add default via 192.168.7.1 dev eth1 table uplink1
        ip rule add from 192.168.7.126/32 table uplink1
        ip rule add to 192.168.7.126/32 table uplink1

        • • Dave

          • Posted November 24, 2010 at 6:19 pm
          • Permalink
          • Reply
          I have set it up as stated. It boots fine but I am unable to ping 10.0.0.211 (eth1) from another machine on the subnet – but I can ping 10.0.0.201 (eth0). From the machine itself I can ping 10.0.0.211. Here is some output:
          oft201:~# ip rule
          0: from all lookup local
          32766: from all lookup main
          32767: from all lookup default
          oft201:~# vi /etc/rc.local
          oft201:~# /etc/rc.local
          oft201:~# ip rule
          0: from all lookup local
          32762: from all to 10.0.0.211 lookup uplink1
          32763: from 10.0.0.211 lookup uplink1
          32764: from all to 10.0.0.201 lookup uplink0
          32765: from 10.0.0.201 lookup uplink0
          32766: from all lookup main
          32767: from all lookup default
          oft201:~# netstat -nra
          Kernel IP routing table
          Destination Gateway Genmask Flags MSS Window irtt Iface
          10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
          10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
          0.0.0.0 10.0.0.1 0.0.0.0 UG 0 0 0 eth0

          What gives? [I am assuming that policy routing must be enabled in the latest stable debian kernel:
          Linux oft201 2.6.26-2-686-bigmem #1 SMP Thu Sep 16 20:29:07 UTC 2010 i686 GNU/Linux]

          • • Darien

            • Posted November 24, 2010 at 6:58 pm
            • Permalink
            Okay, can you paste the contents of your /etc/rc.local file?

          • • Darien

            • Posted November 24, 2010 at 7:00 pm
            • Permalink
            Also, I assume you entered "ip route flush cache" and still ran into problems?

          • • Dave

            • Posted November 24, 2010 at 7:03 pm
            • Permalink
            I am able to get it to work if the line:
            ip route add 192.168.7.127/32 dev eth0 src 192.168.7.127 table uplink0
            is changed to
            ip route add 192.168.7.0/24 dev eth0 src 192.168.7.127 table uplink0
            and the line
            ip route add 192.168.7.126/32 dev eth1 src 192.168.7.126 table uplink1
            is changed to
            ip route add 192.168.7.0/24 dev eth1 src 192.168.7.126 table uplink1

          • • Darien

            • Posted November 24, 2010 at 7:05 pm
            • Permalink
            Okay, glad it's working for you.

47. • Dave

    • Posted November 24, 2010 at 7:05 pm
    • Permalink
    • Reply
    My rc.local with what I beleive to be the incorrect lines commented out:
    #!/bin/sh -e
    #
    # rc.local
    #
    # This script is executed at the end of each multiuser runlevel.
    # Make sure that the script will "exit 0″ on success or any other
    # value on error.
    #
    # In order to enable or disable this script just change the execution
    # bits.
    #
    # By default this script does nothing.

    #ip route add 10.0.0.201/32 dev eth0 src 10.0.0.201 table uplink0
    ip route add 10.0.0.0/24 dev eth0 src 10.0.0.201 table uplink0
    ip route add default via 10.0.0.1 dev eth0 table uplink0
    ip rule add from 10.0.0.201/32 table uplink0
    ip rule add to 10.0.0.201/32 table uplink0

    #ip route add 10.0.0.211/32 dev eth1 src 10.0.0.211 table uplink1
    ip route add 10.0.0.0/24 dev eth1 src 10.0.0.211 table uplink1
    ip route add default via 10.0.0.2 dev eth1 table uplink1
    ip rule add from 10.0.0.211/32 table uplink1
    ip rule add to 10.0.0.211/32 table uplink1

    exit 0

    • • Dave

      • Posted December 1, 2010 at 6:58 pm
      • Permalink
      • Reply
      Turns out that the above settings work but not as expected.
      If I run 'ifdown eth1′ and then try
      ping or ssh 10.0.1.211 (the interface associated with eth1) – I can still get through (and vice versa). This may be seen as a feature rather than a bug – but I am spooked by it because I dont understand it. I have abandoned the quest of dual gateway on same subnet. There is no compelling reason for having this anyway. I can live with the original prescription of Darien – which works as expected. (I used the latest debian lenny kernel and did not need to make any special kernel compile to get Dariens prescription working). Thanks again.

48. • why

    • Posted December 21, 2010 at 1:01 pm
    • Permalink
    • Reply
    Would have been a great article if it had explained where one wrote the admin routing information:

    ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
    ip route add default via 10.10.70.254 dev eth0 table admin

    Which file is this placed into? I don't know. The article does not say.

    Therefore, I am back to googgeling once more for another site. But at least I know I should Google for policy routing. Thus a partial thank-you.

    • • why

      • Posted December 21, 2010 at 1:02 pm
      • Permalink
      • Reply
      PS. I am using SLES 11. Maybe its obvious on gentoo.

      • • Darien

        • Posted December 21, 2010 at 2:12 pm
        • Permalink
        • Reply
        why: Read the corresponding comments for your answer. Specifically, this one:
        http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-43

        Better yet, just search the page for "rc.local". What you're asking for is very generic… you just want a way to have this commands auto-executed upon boot up.

        If you do a google search for "suse linux rc.local", you get this as a top hit:
        http://linux.derkeiler.com/Mailing-Lists/SuSE/2003-12/1428.html

        Which states: /etc/init.d/boot.local is the equivalent file of rc.local in SLES.

      • • why

        • Posted December 21, 2010 at 2:40 pm
        • Permalink
        • Reply
        Sorry, I have realised that these lines are actual commands.
        ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
        ip route add default via 10.10.70.254 dev eth0 table admin
        Please disregard my initial posts.

49. • whynot

    • Posted December 21, 2010 at 2:47 pm
    • Permalink
    • Reply
    Hi,
    I have had a hard time following the post, and am unsure its what I have to do. Would you be able to have a look at this scenario and add some helpers for it?

    Both interfaces are on different VLANs (Should not make an once of difference). The default route is the one that goes to the commodity Internet, while the 10/8 destinations are for the internal network.

    eth0 IP 10.254.60.90/22
    default 10.254.60.74 255.255.252.0 eth0
    10.0.0.0 10.254.60.51 255.0.0.0 eth0

    eth1 IP 10.254.41.13/22
    default 10.254.40.74 255.255.252.0 eth1
    10.0.0.0 10.254.40.51 255.0.0.0 eth1

    • • Darien

      • Posted December 21, 2010 at 3:44 pm
      • Permalink
      • Reply
      I'll take a look, but why are you defining two different netmasks for each interface? That makes no sense.

      If you're going to define a full Class-A /8, then you don't need to explicitly define a /22 for the upstream Internet gateway. The /22 serves no purpose in this context.

      I'll generate some rules that show you what I mean…

      • • Darien

        • Posted December 21, 2010 at 3:47 pm
        • Permalink
        • Reply
        I think I see what you're trying to do… each interface has two _virtual_ IPs assigned to it:

        eth0: 10.254.60.90, 10.254.60.51
        eth1: 10.254.41.13, 10.254.40.51

        I'll adjust accordingly…

        • • Darien

          • Posted December 21, 2010 at 3:49 pm
          • Permalink
          • Reply
          Actually, can you confirm each interface has two virtual IPs? The way you wrote the initial IP information is confusing…

          • • whynot

            • Posted December 21, 2010 at 4:08 pm
            • Permalink
            These are both IPs plumbs on real NICs, not virtual NIC (not eth0:1 etc), and neither are these bonded in anyway.

      • • whynot

        • Posted December 21, 2010 at 4:00 pm
        • Permalink
        • Reply
        Hi Darie,

        Apologies for the confusion.

        The 10/8 reference was only mean to refer to the second route below for all traffic to the 10.0.0.0 network, and this understandingly caused confusion.

        All networks use /22. The default route is for all traffic other than the 10.0.0.0/22 network, (which is really the public Internet and other private nets 192.168.n.n. and 172.16.n.n networks).

        The current system has one NIC eth0. Its routing table as defined is :
        default 10.254.64.74 255.255.252.0 -
        10.0.0.0 10.254.64.51 255.0.0.0 -

        I have to plumb a new eth1 onto it on another VLAN and keep the traffic separate. (I had never heard of priority routing until now.) The eth1 has to send the default Internet traffic via the router 10.254.40.74 on eth1, and the rest back to the internal LAN via 10.254.40.51 on eth1.

        • • whynot

          • Posted December 21, 2010 at 4:01 pm
          • Permalink
          • Reply
          ^Hi Darie,^Hi Darien,

        • • Darien

          • Posted December 21, 2010 at 4:03 pm
          • Permalink
          • Reply
          Okay, ignore my previous post… will provide documentation based on this new information, shortly.

          • • whynot

            • Posted December 21, 2010 at 4:13 pm
            • Permalink
            FYI
            These are all routers somewhere on the network:
            10.254.64.74
            10.254.64.51
            10.254.40.74
            10.254.40.51

        • • Darien

          • Posted December 21, 2010 at 4:13 pm
          • Permalink
          • Reply
          Okay, the real issue is that you have a single IP address defined for each physical interface (eth0 and eth1), and yet you have multiple _overlapping_ netmasks defined for the routes in each interface.

          For example, eth0:
          Class-A /8 – respond directly
          /22 – out to gateway

          This makes the issue not as easy to solve, because normally you would assign the same netmask for each interface - or you could define multiple virtual interfaces with different IP addresses and netmask combinations.

          In any event, I believe the following commands will work without having to define any virtual interfaces, but I have never tested this scenario, so you'll have to try it out and see for yourself:

          Here are the corresponding commands:
          printf "1\tuplink0\n" >> /etc/iproute2/rt_tables
          printf "2\tuplink1\n" >> /etc/iproute2/rt_tables
          printf "3\tuplink2\n" >> /etc/iproute2/rt_tables
          printf "4\tuplink3\n" >> /etc/iproute2/rt_tables

          ip route add 10.254.60.51/22 dev eth0 src 10.254.60.51 table uplink0
          ip route add default via 10.254.60.74 dev eth0 table uplink0
          ip rule add from 10.254.60.51/22 table uplink0
          ip rule add to 10.254.60.51/22 table uplink0

          ip route add 10.254.60.51/8 dev eth0 src 10.254.60.51 table uplink1
          ip rule add from 10.254.60.51/8 table uplink1
          ip rule add to 10.254.60.51/8 table uplink1

          ip route add 10.254.40.51/22 dev eth1 src 10.254.40.51 table uplink2
          ip route add default via 10.254.40.74 dev eth1 table uplink2
          ip rule add from 10.254.40.51/22 table uplink2
          ip rule add to 10.254.40.51/22 table uplink2

          ip route add 10.254.40.51/8 dev eth1 src 10.254.40.51 table uplink3
          ip rule add from 10.254.40.51/8 table uplink3
          ip rule add to 10.254.40.51/8 table uplink3

          • • whynot

            • Posted December 21, 2010 at 11:28 am
            • Permalink
            For example, eth0:
            Class-A /8 – respond directly
            /22 – out to gateway

            Yes, this is exactly this. There is a multitude of overlapping netmasks here.

            I shall try all this on a test server, and then report back here. This'll be later tomorrow, as its nearly home time here.

    • • Darien

      • Posted December 21, 2010 at 4:01 pm
      • Permalink
      • Reply
      Okay, here's the set of rules for this scenario. I assume that each interface actually has _two_ virtual IP addresses. In this case, you would be defining "eth0:0 and eth0:1″ as the virtual interfaces for eth0 and "eth1:0 and eth1:1″ for the virtual interfaces for eth1. If you're not sure what this means or how to set it up, the SLES documentation about virtual interfaces should help.

      Regardless, instead of thinking about two interfaces, think about this problem in terms of _four_ virtual interfaces. If you split the problem based upon each virtual interface, it's much easier to solve. See below for details.

      Assumptions:
      eth0:0 – 10.254.60.90 netmask 255.255.252.0
      eth0:0′s gateway is: 10.254.60.74

      eth0:1 – 10.254.60.51 netmask 255.0.0.0

      eth1:0 – 10.254.41.13 netmask 255.255.252.0
      eth1:0′s gateway is: 10.254.40.74

      eth1:1 – 10.254.40.51 netmask 255.0.0.0

      Here are the corresponding commands:
      printf "1\tuplink0\n" >> /etc/iproute2/rt_tables
      printf "2\tuplink1\n" >> /etc/iproute2/rt_tables
      printf "3\tuplink2\n" >> /etc/iproute2/rt_tables
      printf "4\tuplink3\n" >> /etc/iproute2/rt_tables

      ip route add 10.254.60.90/22 dev eth0:0 src 10.254.60.90 table uplink0
      ip route add default via 10.254.60.74 dev eth0:0 table uplink0
      ip rule add from 10.254.60.90/22 table uplink0
      ip rule add to 10.254.60.90/22 table uplink0

      ip route add 10.254.60.51/8 dev eth0:1 src 10.254.60.51 table uplink1
      ip rule add from 10.254.60.51/8 table uplink1
      ip rule add to 10.254.60.51/8 table uplink1

      ip route add 10.254.41.13/22 dev eth1:0 src 10.254.41.13 table uplink2
      ip route add default via 10.254.40.74 dev eth1:0 table uplink2
      ip rule add from 10.254.41.13/22 table uplink2
      ip rule add to 10.254.41.13/22 table uplink2

      ip route add 10.254.40.51/8 dev eth1:1 src 10.254.40.51 table uplink3
      ip rule add from 10.254.40.51/8 table uplink3
      ip rule add to 10.254.40.51/8 table uplink3

50. • whynot

    • Posted December 22, 2010 at 11:01 am
    • Permalink
    • Reply
    Apologies. I;m confused.

    Actually, your original thread was so close what I want to do, yet the ip rules you defined only ever mentioned eth0:
    # eth0 – 10.10.70.38 netmask 255.255.255.0
    # eth0′s gateway is: 10.10.70.254
    But never mentioned eth1:
    # eth1 – 192.168.7.126 netmask 255.255.255.0
    # eth1′s gateway is: 192.168.7.1

    Since the eth1 is not mentioned, then I presume that it has to be indirectly decided upon by iproute, but this is where I lost the plot. There are only these entries:
    ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
    ip route add default via 10.10.70.254 dev eth0 table admin
    ip rule add from 10.10.70.38/32 table admin
    ip rule add to 10.10.70.38/32 table admin
    Nor does it say what to add or remove for the /etc/sysconfig/network/routes table, which currently has some routing information in it

    Anyway, cheers for your time.

    • • Anonymous

      • Posted June 8, 2011 at 12:41 am
      • Permalink
      • Reply
      The default route for anything other than that subnet specified in the table is that which is set for the network. In other words: if those rules don't apply to the packets, they head on out eth1 towards 192.168.7.1 (unless the packets came from the 7.x network, in which case they would head on over to the machine that made the request.)

51. • Larry

    • Posted February 16, 2011 at 7:43 am
    • Permalink
    • Reply
    This may be an old post, but wonderfully explains how to properly set up the routes. I'm running Ubuntu 10.04. One note, if it's not clear, I defined a gateway (192.168.1.1) for eth0 as my default in the /etc/network/interfaces file. For eth1, I did not define a gateway, but rather set up the route as noted above. Again, many thanks.

52. • Vijay Upreti

    • Posted April 13, 2011 at 10:02 am
    • Permalink
    • Reply
    Excellent article. It served most of my problem, specially where multiple vlan interfaces setup for different network.

53. • KS

    • Posted May 3, 2011 at 4:18 am
    • Permalink
    • Reply
    Splendid post!! Thanks =))

54. • Otheus

    • Posted May 5, 2011 at 7:02 am
    • Permalink
    • Reply
    Thanks.

55. • Khalil

    • Posted May 10, 2011 at 3:18 pm
    • Permalink
    • Reply
    This is really great. Finally got my bridged virtual VBOX server environment connected to the internet through a separate nic, leaving my primary nic for the physical ubuntu server access only.

    This makes my virtual business environment finally completely safe and separate from my physical private environment.

    Thanks a lot !

    (If someone wants the setup details, leave a reply and I will post them.)

56. • rishi

    • Posted May 26, 2011 at 2:30 pm
    • Permalink
    • Reply
    http://reddragon-linux.blogspot.com/2011/05/linux-internet-gateway-server-setup.html

57. • Anonymous

    • Posted August 1, 2011 at 5:27 pm
    • Permalink
    • Reply
    Two years later, and this is the best post i found to accomplish the job without need to learn all about iptables, routes, rules, etc of a linux box (but still is necesary ).
    Thank you very much. You make my day a happy day

58. • RG

    • Posted August 11, 2011 at 10:26 am
    • Permalink
    • Reply
    Thank you Thank you Thank you,
    I've been trying to shovel several vlans across as many openvpn tunnels on a virtual server without success for what felt like eternity, now in 2 minutes flat it actually does stuff.

    Thanks again

59. • Sheck

    • Posted August 17, 2011 at 5:54 pm
    • Permalink
    • Reply
    Man, Thank You !!!!!!

    I spent days trying to solve this. This worked right away.

60. • Andy Baker

    • Posted August 23, 2011 at 5:22 pm
    • Permalink
    • Reply
    Darien,
    Thanks to you for great article and thanks to everyone else for some great reading.

    Set this up on BackTrack 5 – 32 bit and it worked fine. Now trying the same thing on BT5 R1 – 64 bit and it's not playing

    I have two distinct networks and never the twain shall meet.

    My home network (eth0) is a 192.168.2.0/24 with the gateway as 192.168.2.1 and this connects to the internet
    My 'special' network (eth1) is a 10.0.0.16/28 with the gateway as 10.0.0.17 and this connects nowhere due to some of the software sitting on the VM's that live on this network (vm's are fixed to eth1 only using VMWare Virtual Network Editor). Both nics have static ip's though it did work with DHCP when I had it running pre 64 bit install.

    Here is what i'm getting
    have added following to rc.local
    ip route add 192.168.2.0/24 dev oth0 src 192.168.2.30 table admin
    ip route add default via 192.168.2.1 dev eth0 table admin
    ip rule add from 192.168.2.30/32 table admin
    ip rule add to 192.168.2.30/32 table admin
    ip route flush cache

    ip rule show
    0: from all lookup local
    32764: from all to 192.168.2.30 lookup admin
    32765: from 192.168.2.30 lookup admin
    32766: from all lookup main
    32767: from all lookup default

    ip route show
    default via 192.168.2.1 dev eth0 metric 100
    default via 10.0.0.17 dev eth1 metric 100
    10.0.0.16/28 dev eth1 proto kernel scope link src 10.0.0.20
    192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.30

    netstat -anr
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
    0.0.0.0 10.0.0.17 0.0.0.0 UG 0 0 0 eth1
    10.0.0.16 0.0.0.0 255.255.255.240 U 0 0 0 eth1
    192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

    With eth1 enabled, I can't access the internet or if I use 'route del default gw 10.0.0.17′ it will let me connect to the internet until next time I reboot. I had it working before and am now baffled as to why it's playing silly buggers. Have I missed something obvious above or can you think of something else I can try?

    This system is a training rig and I need to be able to access the internet from the main machine whilst using the VM's on their special network.

    • • Darien

      • Posted August 23, 2011 at 9:44 pm
      • Permalink
      • Reply
      Here's your problem:

      "netstat -anr
      …
      0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
      …"

      You've _already_ specified a default route in the _admin_ table for eth0; as such, you should NOT specify a default route for eth0 in the _main_ table.

      To correct this, remove the following command [FLAGGED] from your list:
      "ip route show
      default via 192.168.2.1 dev eth0 metric 100 <- [REMOVE THIS ONE]
      default via 10.0.0.17 dev eth1 metric 100
      10.0.0.16/28 dev eth1 proto kernel scope link src 10.0.0.20
      192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.30"

61. • Anonymous

    • Posted September 6, 2011 at 12:31 pm
    • Permalink
    • Reply
    Thanks! This article really helped me.

62. • JuanV

    • Posted September 17, 2011 at 4:04 pm
    • Permalink
    • Reply
    Thanks man ! you saved my life i was looking this information from a long time ago.

    cheers!

63. • sun

    • Posted October 5, 2011 at 12:13 pm
    • Permalink
    • Reply
    hi, and thank you for your post! i have a question….

    eth0 – ip 10,60,90,45 gw 10,60,90,1
    eth1 – ip 10,60,91,45 gw 10,60,91,1
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    10.60.90.0 * 255.255.255.0 U 0 0 0 eth0
    10.60.91.0 * 255.255.255.0 U 0 0 0 eth1
    link-local * 255.255.0.0 U 0 0 0 eth0
    loopback * 255.0.0.0 U 0 0 0 lo
    default 10.60.90.1 0.0.0.0 UG 0 0 0 eth0

    if i need that a machine on 69,50,20,xx subnet can connect to 10,60,91,45 i add a new route

    69,51,20,0 10,60,91,1 255,255,255,0 UG 0 0 0 eth1

    but if i don't know all the subnets or more easy, how can i do to all traffic that goes through an interface is returned by the same interface.

    • • Darien

      • Posted November 15, 2011 at 10:34 am
      • Permalink
      • Reply
      Hi sun,
      Take a look at the comments posted right after nobodyfamous:
      http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-102

      So, let's assume:
      eth0 – 10.60.90.45 netmask 255.255.255.0
      eth0′s gateway is: 10.60.90.1

      eth1 – 10.60.91.45 netmask 255.255.255.0
      eth1′s gateway is: 10.60.91.1

      Here are the corresponding commands:

      printf "1\tuplink0\n" >> /etc/iproute2/rt_tables
      printf "2\tuplink1\n" >> /etc/iproute2/rt_tables

      ip route add 10.60.90.0/24 dev eth0 src 10.60.90.45 table uplink0
      ip route add default via 10.60.90.1 dev eth0 table uplink0
      ip rule add from 10.60.90.45/32 table uplink0
      ip rule add to 10.60.90.45/32 table uplink0

      ip route add 10.60.91.0/24 dev eth1 src 10.60.91.45 table uplink1
      ip route add default via 10.60.91.1 dev eth1 table uplink1
      ip rule add from 10.60.91.45/32 table uplink1
      ip rule add to 10.60.91.45/32 table uplink1

      ip route flush

64. • ben

    • Posted October 6, 2011 at 2:34 pm
    • Permalink
    • Reply
    THANK YOU

65. • Marty M.

    • Posted October 26, 2011 at 4:01 pm
    • Permalink
    • Reply
    Thank you! Your page made it so simple to fix my routing dilemma! I had two NICs on two separate dmz networks and needed to keep traffic on the the same NIC it started on. And have a default gateway for each NIC. This helped! Great Work!

66. • Cliff

    • Posted October 28, 2011 at 7:54 pm
    • Permalink
    • Reply
    Four years on, this is still the clearest guide to this. It just works.

    Thanks.

67. • Birger

    • Posted November 15, 2011 at 6:37 am
    • Permalink
    • Reply
    Thanks for the article.

    Unfortunately it does not work in my setup.
    How can I make sure my kernel has support for policy routing?
    I use Proxmox Virtualization Environment (http://pve.proxmox.com) with kernel 2.6.32-6-pve.

    My setup shows:

    vserver3:~# ip rule show
    0: from all lookup local
    32764: from all to 192.44.37.62 lookup vmbr1
    32765: from 192.44.37.62 lookup vmbr1
    32766: from all lookup main
    32767: from all lookup default

    vserver3:~# ip route show
    192.168.20.0/24 dev vmbr101 proto kernel scope link src 192.168.20.1
    192.44.37.0/24 dev vmbr1 proto kernel scope link src 192.44.37.62
    192.168.19.0/24 dev vmbr100 proto kernel scope link src 192.168.19.1
    153.96.8.0/24 dev vmbr0 proto kernel scope link src 153.96.8.76
    default via 153.96.8.1 dev vmbr0

    vserver3:~# ip route show table vmbr1
    192.44.37.0/24 dev vmbr1 scope link src 192.44.37.62
    default via 192.44.37.101 dev vmbr1

    I have vmbr100 connected via iptables NAT rules to vmbr0
    and vmbr101 via iptables NAT rules to vmbr1.

    Connectivity of vmbr0 and vmbr100 works.
    Connectivity of vmbr1 and vmbr101 is limited to network 192.44.37.0/24.

    Any ideas? Thanks.

    • • Darien

      • Posted November 15, 2011 at 10:22 am
      • Permalink
      • Reply
      Hi Birger,

      "vmbr*" interfaces are not standard network interfaces (http://pve.proxmox.com/wiki/Vlans) - they are bridging interfaces, which is why policy routing does NOT work properly.

      In order to use policy routing, you need to assign one or more NICs (eth*) interfaces to a bridging interface (vmbr*), and then assign the routing to the attached eth* interface, accordingly.

68. • Joel Knight

    • Posted November 15, 2011 at 10:13 pm
    • Permalink
    • Reply
    Hi Darien,

    With respect to multiple default routes on OpenBSD: the virtual tables are primarily meant to create separate logical networks over top of a common base network. For instance you could create a private network for contractors which would keep them logically separated from your employees.

    However within a routing table, OpenBSD actually supports equal-cost multipath route entries. The kernel will choose a path based on the result of a hash and send the packets on their way. It's really easy to setup and does not require the complexity of virtual tables. Details are in the OpenBSD FAQ (http://www.openbsd.org/faq/faq6.html#Multipath).

69. • Birger

    • Posted November 16, 2011 at 3:10 am
    • Permalink
    • Reply
    Thanks for the feedback.

    Special Thanks to Darien for the very valuable blog post.

    Everything works as expected. We tested the configuration with the command "ping -I vmbr1 …", which gave an error "Destination host unreachable". Now we found that ping is buggy.

    "ping -I 192.44.37.62 …" works as expected. "wget -q -O – -bind-address=192.44.37.62 http://whatismyip.org" works also.

    It does work with bridging interfaces and it does work in combination with NAT.

    • • Darien

      • Posted November 16, 2011 at 7:32 am
      • Permalink
      • Reply
      Okay, glad to hear it works with bridging interfaces… I wasn't 100% sure of this, as I had never tested it before.

70. • MyD

    • Posted December 9, 2011 at 2:31 am
    • Permalink
    • Reply
    thx for this. you made my day. if you got 2 nics make sure you add two route tables

    # echo "1 uplink0″ >> /etc/iproute2/rt_tables
    # echo "2 uplink1″ >> /etc/iproute2/rt_tables

    # ip route add 192.168.0.0/24 dev eth0 src 192.168.0.101 table uplink0
    # ip route add default via 192.168.0.1 dev eth0 table uplink0
    # ip rule add from 192.168.0.101/32 table uplink0
    # ip rule add to 192.168.0.101/32 table uplink0

    # ip route add xxx.xxx.xxx.0/24 dev eth3 src xxx.xxx.xxx.219 table uplink1
    # ip route add default via xxx.xxx.xxx.1 dev eth3 table uplink1
    # ip rule add from xxx.xxx.xxx.219/32 table uplink1
    # ip rule add to xxx.xxx.xxx.219/32 table uplink1
    (xxx.xxx.xxx. replace with your settings )

    to make it permanent under gentoo you can use

    # nano /etc/local.d/network.start (http://de.gentoo-wiki.com/wiki/Local.d)
    (add #!/bin/sh & the ip xxx commandos; check with rc-update show if the local is added to the default level)
    # chmod +x /etc/local.d/network.start

    have fun and enjoy!

71. • Luke

    • Posted December 13, 2011 at 12:50 am
    • Permalink
    • Reply
    Hi. Great post and the comment stream is going strong…

    I used your post to set up my system.

    I have a gateway box with 2 NICs.
    One nic sits on the internal network (eth1; 192.168.0.7), the other nic (eth0; 110.xxx.xxx.173) connected to a 10 MBit pipe used for incoming VPN.
    There is an ADSL router (192.168.0.1) sitting on the network as well…
    Everyone on the network uses 192.168.0.1 as their gateway.
    I want people to be able to set 192.168.0.7 as their gateway if they have special speed needs.
    So what happens is that when the user's (say user 192.168.0.10) gateway is set to 192.168.0.7 the traffic still goes via 192.168.0.1… whereas I want it to go through 110.xxx.xxx.173…

    Host Loss% Snt Last Avg Best Wrst StDev
    1. gateway1.e3(192.168.0.7) 0.0% 4 0.1 0.2 0.1 0.2 0.0
    2. firewall.e3(192.168.0.1) 0.0% 4 0.4 0.5 0.4 0.5 0.0
    3. ras8.nap.net.au 0.0% 3 30.0 146.9 30.0 263.7 165.2
    4. g3.cor1.adp.net.au 0.0% 3 280.6 229.0 140.8 280.6 76.8
    5. g1.ap.net.au 50.0% 3 155.1 155.1 155.1 155.1 0.0

    here are my configs:

    /etc/iproute2/rt_tables:

    2 gatewayeth1
    3 gatewayeth0

    /etc/network/interfaces:
    # The primary network interface
    auto eth1
    iface eth1 inet static
    address 192.168.0.7
    netmask 255.255.255.0
    gateway 192.168.0.1

    # 10Mbit interface
    auto eth0
    iface eth0 inet static
    address 110.xxx.xxx.174
    netmask 255.255.255.252
    gateway 110.xxx.xxx.173

    /etc/rc.local:
    ip route add 110.xxx.xxx.172/30 dev eth0 src 110.xxx.xxx.174 table gatewayeth0
    ip route add default via 110.xxx.xxx.173 dev eth0 table gatewayeth0
    ip route add 192.168.0.0/24 dev eth1 src 192.168.0.7 table gatewayeth1
    ip route add default via 192.168.0.1 dev eth1 table gatewayeth1

    ip rule add from 110.xxx.xxx.174/32 table gatewayeth0
    ip rule add to 110.xxx.xxx.174/32 table gatewayeth0
    ip rule add from 192.168.0.7/32 table gatewayeth1
    ip rule add to 192.168.0.7/32 table gatewayeth1

    iptables rules:
    #! /bin/bash

    iptables –flush

    #NAT TABLE
    iptables -t nat –policy PREROUTING ACCEPT
    iptables -t nat –policy POSTROUTING ACCEPT
    iptables -t nat –policy OUTPUT ACCEPT
    iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
    iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

    #FILTER TABLE
    iptables –policy INPUT DROP
    iptables –policy OUTPUT DROP
    iptables –policy FORWARD DROP

    # localhost rules
    iptables -A INPUT -i lo -j ACCEPT
    iptables -A OUTPUT -o lo -j ACCEPT

    # in chain for eth0 (outward facing telstra port on network 110.xxx.xxx.172/30)

    iptables -X IN_RULES_ETH0
    iptables -N IN_RULES_ETH0
    iptables -A INPUT -i eth0 -j IN_RULES_ETH0

    # in chain for eth1 (inward facing internal network 192.168.0.0/24)

    iptables -X IN_RULES_ETH1
    iptables -N IN_RULES_ETH1
    iptables -A INPUT -i eth1 -j IN_RULES_ETH1

    # out chain

    iptables -X OUT_RULES
    iptables -N OUT_RULES
    iptables -A OUTPUT -j OUT_RULES

    # forward chain for eth1

    iptables -X FORWARD_ETH1
    iptables -N FORWARD_ETH1
    iptables -A FORWARD -j FORWARD_ETH1

    # drop stuff for eth0
    # IN_RULES_ETH0
    iptables -A IN_RULES_ETH0 -s 10.0.0.0/8 -j DROP
    iptables -A IN_RULES_ETH0 -s 172.16.0.0/12 -j DROP
    iptables -A IN_RULES_ETH0 -s 192.168.0.0/16 -j DROP
    iptables -A IN_RULES_ETH0 -s 224.0.0.0/4 -j DROP
    iptables -A IN_RULES_ETH0 -s 240.0.0.0/5 -j DROP
    iptables -A IN_RULES_ETH0 -s 127.0.0.0/8 -j DROP
    iptables -A IN_RULES_ETH0 -s 0.0.0.0/8 -j DROP
    iptables -A IN_RULES_ETH0 -d 255.255.255.255 -j DROP
    iptables -A IN_RULES_ETH0 -s 169.254.0.0/16 -j DROP

    # drop stuff for eth1 allow only a few addresses
    # IN_RULES_ETH1
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.2-192.168.0.9 -j DROP
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.11-192.168.0.26 -j DROP
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.27-192.168.0.59 -j DROP
    iptables -A IN_RULES_ETH1 -m iprange –src-range 192.168.0.61-192.168.0.254 -j DROP
    iptables -A IN_RULES_ETH1 -s 169.254.0.0/16 -j DROP

    # drop stuff out rules
    iptables -A OUT_RULES -d 224.0.0.0/4 -j DROP

    # allow previously established / related connections
    iptables -A OUT_RULES -m state –state ESTABLISHED,RELATED -j ACCEPT
    iptables -A IN_RULES_ETH0 -m state –state ESTABLISHED,RELATED -j ACCEPT
    iptables -A IN_RULES_ETH1 -m state –state ESTABLISHED,RELATED -j ACCEPT

    # forward all on eth1
    iptables -A FORWARD_ETH1 -i eth1 -j ACCEPT
    iptables -A FORWARD_ETH1 -o eth1 -j ACCEPT

    ####################################################################
    # out services / ports

    # ICMP
    iptables -A OUT_RULES -p icmp -j ACCEPT
    # Port 22 (SSH)
    iptables -A OUT_RULES -p tcp –dport 22 -j ACCEPT
    # Port 1723 (VPN)
    iptables -A OUT_RULES -p tcp –dport 1723 -j ACCEPT

    # Port 123 (NTP)
    iptables -A OUT_RULES -p udp –dport 123 -j ACCEPT
    iptables -A OUT_RULES -p tcp –dport 123 -j ACCEPT

    iptables -A OUT_RULES -j LOG –log-prefix "BLOCKED OUTPUT "
    iptables -A OUT_RULES -j REJECT

    ####################################################################
    # input services / ports

    # ICMP
    iptables -A IN_RULES_ETH0 -p icmp -j ACCEPT
    iptables -A IN_RULES_ETH1 -p icmp -j ACCEPT

    # Port 22 (SSH)
    iptables -A IN_RULES_ETH0 -p tcp –dport 22 -m state –state NEW -j ACCEPT
    iptables -A IN_RULES_ETH1 -p tcp –dport 22 -m state –state NEW -j ACCEPT

    # Port 1723 (VPN)
    iptables -A IN_RULES_ETH0 -p tcp -m tcp –dport 1723 -j ACCEPT
    iptables -A IN_RULES_ETH0 -p gre -j ACCEPT

    ######################################################################

    # Drop MS NetBIOS scanning early (it spams the log)
    iptables -A IN_RULES_ETH0 -p tcp –dport 137 -j DROP
    iptables -A IN_RULES_ETH1 -p tcp –dport 137 -j DROP
    iptables -A IN_RULES_ETH0 -p udp –dport 137 -j DROP
    iptables -A IN_RULES_ETH1 -p udp –dport 137 -j DROP
    iptables -A IN_RULES_ETH0 -p tcp –dport 138 -j DROP
    iptables -A IN_RULES_ETH1 -p udp –dport 138 -j DROP

    # Log and reject
    iptables -A IN_RULES_ETH0 -j LOG –log-prefix "BLOCKED INPUT "
    iptables -A IN_RULES_ETH1 -j LOG –log-prefix "BLOCKED INPUT "
    iptables -A IN_RULES_ETH0 -j REJECT
    iptables -A IN_RULES_ETH1 -j REJECT

    Thanks.
    Hopefully you can help me…

    Best
    Luke.

    • • Darien

      • Posted December 13, 2011 at 3:30 am
      • Permalink
      • Reply
      > Hopefully you can help me…

      Luke, I need some additional clarification… I assume you posted this information because multiple default routing for you isn't working correctly - is that correct? If so, can you clarify as to what is and is not working correctly? (e.g. which route works properly?)

      Also, remove _all_ the firewall rules and just test to make sure multiple default routing works from the 192.168.0.7 system, first (without any further NAT'ing). If multiple default routes for that host _fail_ to work, then the problem isn't with your iptables rules - however, if multiple default routing _works_ from (just) the host without the iptable rules, then you've isolated the problem to your iptables rules.

      The overall idea here, is to reduce the number variables in your configuration, in order to identify the root cause.

      Regards,
      – Darien

      • • Luke

        • Posted December 13, 2011 at 7:14 pm
        • Permalink
        • Reply
        Hi Darien. Thanks for your reply.
        The problem is that users on the 192.168.0.0 network who set their gateway to 192.168.0.7 should have their traffic go straight out through the 110.xxx.xxx.173 interface, but it goes first to the 192.168.0.1 interface…
        I will try what you suggest and get back here…

      • • Luke

        • Posted December 13, 2011 at 7:33 pm
        • Permalink
        • Reply
        Ive removed the gateway statement from the local NIC for 192.168.0.7 and it seems to have fixed the problem.
        I thought that having routes added later (as in /etc/rc.local) would have overridden the default gateway set in /etc/network/interfaces but apparently not…

      • • Luke

        • Posted December 13, 2011 at 7:35 pm
        • Permalink
        • Reply
        Hi Darien. Lastly is there a way I can edit my posts … I seem to have mentioned my ip address in my original post, although I thought I had edited them all, one still remains, could you replace with xxx… thanks

        • • Darien

          • Posted December 13, 2011 at 9:00 pm
          • Permalink
          • Reply
          Okay, this is fixed..

    • athaullah

    • Posted December 30, 2011 at 7:11 am
    • Permalink
    • Reply
    I nee a help, I m delete device from 'network configuration' and again add it. It may be wrong so I unable to ping eventhough default gateway of my router. so plz guide me to configure properly to access internet via netgare 150 adsl router. plz replay in mail id : [email protected]

72. • Chris Sullivan

    • Posted January 3, 2012 at 6:22 pm
    • Permalink
    • Reply
    Thanks Darien, this has moved things forward for me. I hope that someone can help with the 1 (!) remainng problem.

    I have eth0 on the 10.10.170 subnet and eth1 on the 10.10.177 subnet. There is more than 1 gateway on each subnet, for reasons I won't go into. The suggestion in the initial post, modified for my circumstances, allowed me to ssh and ping into either interface. So far, so good. I wasn't able to do that before. Now however, I'm having trouble transitting from one subnet to the other. I wanted to use the 10.10.177.1 gateway, and that is how I set it up. However, a tracert tries to send me through the 10.10.177.2 gateway. I've searched the system and there is no mention of that ip address anywhere to be found.

    Here's the commands I used.

    ip route add 10.10.177.0/24 dev eth1 src 10.10.177.20 table voice
    ip route add default via 10.10.177.1 dev eth1 table voice
    ip rule add from 10.10.177.20/32 table voice
    ip rule add to 10.10.177.20/32 table voice

    This is under Centos 6

    Cheers,
    Chris

    • • Darien

      • Posted January 3, 2012 at 11:42 pm
      • Permalink
      • Reply
      Hi Chris,

      I'll need more information about your network setup, including the configuration used to setup eth0.

      Regards,
      – Darien

73. • sandm3n

    • Posted January 20, 2012 at 8:08 am
    • Permalink
    • Reply
    Hi, i've follow your step, but when i run /etc/init.d/networking restart , the ip table admin is gone, how to save it?
    thanks

    • • Darien

      • Posted January 20, 2012 at 10:18 am
      • Permalink
      • Reply
      sandm3n: See this earlier comment:
      http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/#comment-43

      • • sandm3n

        • Posted January 20, 2012 at 10:27 am
        • Permalink
        • Reply
        Thanks for the ansewer, but i still confuse about this (i'm new in ubuntu) , i've problem about connection, you can see here (please) http://i41.tinypic.com/nl2as6.jpg

        in that case, i want computer on lan, can connect through squid proxy to the internet, when i setting up without client connect, it can't ping to LAN because of 10.10.7.0 default route (that connect to internet)
        when i disable that default route, i can successfully ping to lan, But link to internet is off.

        How could i make two connection, so i can connect to LAN and the internet, i refer to your site but still not work.

        Many thanks

        • • Darien

          • Posted January 20, 2012 at 10:38 am
          • Permalink
          • Reply
          Hi sandm3n,

          The problem is that your Squid proxy is running inside VMware. I assume you are running VMware Workstation. The problem you are running into is specific to VMware's network configuration.

          1) You need to change eth0 to use a vmnet that is _bridged_ with a physical network interface card (NIC).
          2) You need to change eth1 to use a vmnet that is _bridged_ with a _different_ physical network interface card (NIC).

          Because you are using (host-only) and (NAT) for these interfaces, VMware is not letting the policy routing settings work correctly.

          • • sandm3n

            • Posted January 20, 2012 at 11:14 am
            • Permalink
            i will let you know my whole simulated network, see here http://i40.tinypic.com/2f061hd.jpg , using GNS3 virtual box edition with cisco router and inter vlan on it, i use Virtual Box as client, you can see cloud on the right that is connect to vmware (ubuntu server), client to client between vlan can connect .
            first question is my topology correct? on the cloud (server) i run Samba file sharing and Squid (not running)
            Do i must running samba and squid separately ?
            many thanks!

          • • Darien

            • Posted January 20, 2012 at 11:41 am
            • Permalink
            Hi sandm3n,

            You have enough complex components that I'd recommend testing the multiple default route settings on the Ubuntu server in _isolation_ from the rest of your network. The samba and squid services running on the Ubuntu server do not matter.

            Specifically, if you have two separate clients connected directly to the two interfaces on the Ubuntu server, and you can ping each interface separately from each client, then your multiple default route settings should be working properly.

            Once you have established that this works, you can slowly increase the complexity of your network architecture and re-test these basic checks at each step (e.g., pinging each interface). That way, when the test fails, you will be able to isolate which component is responsible for the problem.

          • • sandm3n

            • Posted January 20, 2012 at 11:47 am
            • Permalink
            Thanks for your nice advice, want to test your suggest first, i've to learn more about this, sorry if i bothering you Darien.

Multihomed Linux server just like my old NetWare...

I have no idea how to explain my environment... It's multi-homed it works in NetWare, breaks in Linux.

Similar to this: http://lartc.org/howto/lartc.rpdb.multiple-links.html

What I did.
1. vi /etc/iproute2/rt_tables
added: 250 eth0
added: 251 eth1

2. vi /etc/sysconfig/network/my-routes

3. added the following lines in the my-routes file
#| /bin/bash
ip route add 192.168.0.0/16 dev eth0 src 192.168.1.5 table eth0
ip route add 10.0.0.0/8 dev eth1 src 10.0.1.80 table eth1
ip route add default via 192.168.0.3 table eth0
ip route add default via 10.0.1.1 table eth1
ip route add 192.168.0.0/16 dev eth0 src 192.168.1.5
ip route add 10.0.0.0/8 dev eth1 src 10.0.1.80
ip route add default via 192.168.0.3
ip rule add from 192.168.1.5 table eth0
ip rule add from 10.0.1.80 table eth1

4. chmod 740 /etc/sysconfig/network/my-routes

5. the last thing I did was add this line to my /etc/crontab file, I'm sure there is a better way but I'm tired:
@boot root /etc/sysconfig/network/my-routes

6. rebooted the box and tested...

Reply packets are sent over an unexpected interface

This document (7000318) is provided subject to the disclaimer at the end of this document.

Novell SUSE Linux Enterprise Server 10
Novell SUSE Linux Enterprise Desktop 10
Novell Open Enterprise Server 2 (OES)

Multiple network interfaces

Symptoms

• Query packets are received over one network interface, but response/reply packets are sent out over a different interface (typically, eth0), or
• Packets destined for an IP address corresponding to a physically disconnected interface are still being received.

Goal

• When a query packet is received over interface X, have the response packets to it be sent out over interface X (source based routing), or
  
• when a network interface is physically disconnected, packets destined for the IP address associated with the interface are to be discarded.

Resolution

Set up split access routing

To route response/reply packets out over the same interface that the packets that they are a response to came in on, separate routing tables (based on the incoming interface) can be set up. For a detailed explanation, refer to the "Split Access" section of the "Rules - routing policy database" chapter in the Linux Advanced Routing & Traffic Control HOWTO (LARTC).

Additional Information

Background information: how Linux routes by default

In Linux, by default, packets are considered individually for routing purposes. Thus, all the routing algorithm considers is where to send a packet based on that packet itself, without taking into consideration that the packet may be a response packet of sorts. In a typical setup, this means that all outgoing traffic is going out over one interface, eth0.

If multiple NICs in Linux are configured with addresses in the same subnet, by default communication into that subnet will always be done by the first NIC in that subnet that is started.

Example setup

To obtain the expected behaviour (as stated under "goal" above), split routing needs to be set up.

Assume a single server with two NICs, one configured for the address 10.105.16.100; the other for 10.105.16.101.

First, create two routing tables, T1 and T2 to be used for packets sent to or from these NICs by adding the lines

252 T1
251 T2

to /etc/iproute2/rt_tables.

Next, set up the routing rules to route incoming and outgoing packets via these tables:

ip route add 10.105.16.0/24 dev eth0 src 10.105.16.100 table T1
ip route add default via 10.105.16.2 dev eth0 src 10.105.16.100 table T1
ip rule add from 10.105.16.100 table T1

ip route add 10.105.16.0/24 dev eth1 src 10.105.16.101 table T2
ip route add default via 10.105.16.2 dev eth1 src 10.105.16.101 table T2
ip rule add from 10.105.16.101 table T2

Note that these rules do not cover the routing of outgoing packets which are not response packets (e.g. outgoing ICMP ECHO ("ping") packets). These packets are still governed by the default route. In some setups, there may be a requirement to use both interfaces for such packets as well (e.g. for load balancing). Refer to the "load balancing" section of LARTC for details on how to implement such requirements.

Recommended Links

Softpanorama Recommended

Top articles

Sites

• http://www.policyrouting.org
• http://www.linuxhorizon.ro/iproute2.html
• http://lartc.org/howto/lartc.rpdb.multiple-links.html
• http://linux-ip.net/html/routing-tables.html
• Linux Advanced Routing & Traffic Control HOWTO
• Configuring Multiple Default Routes in Linux " Darien Kindlund's Blog
• Kezhong's Weblog
• Configuring Multiple Default Routes in Linux
• Linux Route Hacks
• Routing for multiple uplinks-providers

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019