I have set up several networks (mid-size) that use dual WAN connections and I have noticed one thing in common...after I set up routing by source IP (policy based routing using modify firewalls and static protocol tables) the bandwidth utilization drops significantly.
Infrastructure/Physical:
I will use my most recent setup as an example, although all three show the same symptoms that have very similar setups. Of the two WANs only one needs NAT translations (it is a school and one of the connections is being NATed by the district and one comes in direct to the school and is NATed locally). The district side uses ATT infrastructure and thus must force-configured to operate at 100mbps full. The other auto negotiates to 1Gbps (although it is limited to 150mbps down by the ISP) and is being NATed by the edge router. The schools use one WAN (the locally NATed WAN) for their student/public wireless network and the other for the school/district owned devices.
Software/Programming
As the networks are separated into VLANs, I am using policy based routing to divert traffic to proper WAN by source IP. I have set up a firewall ruleset and static protocol tables and applied them to the appropriate interfaces. The routing is working perfectly, however (and here is the crux), when I originally had all the traffic being routed by static routes I was able to run at the proper bandwidth (100 down on district WAN and 150 on the Public WAN) but as soon as I started routing using firewalls the rates plummeted. The district runs at about 60% and the public runs at about 40% of the available bandwidth.
I can give specifics if the community wishes, but wonder if I am alone or not with this type of an issue. Is the overhead of package analytics affecting traffic flow? Another (possibly coincidental) thing I noticed is that the combined bandwidth never seems to exceed 100mbps...could the hard-code of 100mbps on the district WAN somehow be throttling both WANs to that rate? Is there a more effective way of routing by source IP that uses less overhead?
Any feedback would be greatly appreciated.
