Solaris like most UNIX type hosts can have multiple network cards on their system. It gets tricky when have 2 interfaces on different subnets since you can only have 1 default router.

Consider this example:

A Solaris server has 2 network interfaces, bge0 and bge2. bge0 has an IP of 192.168.1.1, the router on that network is 192.168.1.254. bge2 has an IP of 192.168.100.1, the router on that network is 192.168.100.254. The default route on the system in the /etc/defaultrouter is 192.168.1.254.

When a packet comes in for 192.168.100.1, Solaris will process it and send the answer out to the default router. It knows nothing about the default router on the 2nd network. If you place the 2nd router in /etc/defaultrouter, then Solaris just round-robins the IPs. So a request comes in bge2 and goes out bge0 to the default router, from bge2′s IP. If the router is configured with anti-spoofing rules, then the router will ignore that packet. Thus, the answer never reaches the client.

In comes IPFilter. This is the Solaris firewall that’s built in. After exploring many different options to try to get it to route properly for that interface by checking the ‘route’ command I found this simple rule that allows it to work:

pass out quick on bge0 to bge2:192.168.100.254 from 192.168.100.1 to any

This rule says that any traffic going out bge0 from the IP 192.168.100.1 (bge2′s IP) should be changed to go out bge1 interface and be sent to 192.168.100.254 (the default router on bge2).

Now…introduce the Solaris multipathing. This allows you to use 2 interfaces, a primary and a backup interface. To do the probe-based failure detection, you’ll need to use 3 IPs, the primary IP and 2 test IPs (one for each interface).

Consider this:

Solaris server has 4 networks, bge0 has a primary IP of 192.168.1.1, bge1 is the 2nd backup network, so the test IPs would be 192.168.1.2 (bge0) and 192.168.1.3 (bge1). It look like this with an ifconfig:

bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
groupname backup

bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 192.168.1.2 netmask ffffff00 broadcast 192.168.1.255

bge1: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3
inet 192.168.1.3 netmask ffffff00 broadcast 192.168.1.255
groupname backup

The server has a second network. bge2 has a primary IP of 192.168.100.1, bge3 is the 2nd backup network, so the test IPs would be 192.168.100.2 (bge2) and 192.168.100.3 (bge3). It look like this with an ifconfig:

bge2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.100.1 netmask ffffff00 broadcast 192.168.100.255
groupname backup

bge2:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 192.168.100.2 netmask ffffff00 broadcast 192.168.100.255

bge3: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3
inet 192.168.100.3 netmask ffffff00 broadcast 192.168.100.255
groupname backup

The routing table looks like this:

# netstat -rn
Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
default              192.168.1.254         UG        1      59593
192.168.1.0           192.168.1.1          U         1          9 bge0
192.168.1.0           192.168.1.3          U         1          0 bge0:1
192.168.1.0           192.168.1.3          U         1          3 bge1
192.168.100.0         192.168.100.1         U         1         35 bge2
192.168.100.0         192.168.100.3         U         1          0 bge2:1
192.168.100.0         192.168.100.3         U         1         30 bge3
224.0.0.0            192.168.1.1          U         1          0 bge0
127.0.0.1            127.0.0.1            UH        2      14556 lo0

If the interface bge0 fails, the IP 192.168.1.1 will fail over to the bge1 interface. If bge2 fails, the IP 192.168.100.1 will fail over to the bge3 interface.

There are rules in IPFilter that can be used, but it makes it a little trickier to ensure the failovers still work:

# Normal condition, bge0 and bge2 are primarys
pass out quick on bge0 to bge2:192.168.100.254 from 192.168.100.1 to any

# bge2 has failed
pass out quick on bge0 to bge3:192.168.100.254 from 192.168.100.1 to any

# bge0  has failed
pass out quick on bge1 to bge2:192.168.100.254 from 192.168.100.1 to any

# bge0 and bge2 has failed
pass out quick on bge1 to bge3:192.168.100.254 from 192.168.100.1 to any

These rules in IPFilter should pass the traffic the correct way in the even of any multipath failovers. You’ll need those 4 rules for each IP on the secondary network you want to route correctly. This means any Solaris containers as well. One small thing with containers is that if you have a container on just the second network, you’ll need to add these commands to a startup script in the global zone in order to have a default router for the zone to see:

        /sbin/route add default 192.168.100.254 -ifp bge2
        /sbin/route add default 192.168.100.254 -ifp bge3

Using the ‘route -p’ does not work to keep it persistent in this case, as it only remembers one of the ‘default 192.168.100.254′ routes (it ignores the -ifp part).