Hello,
As a WISP, we are using many ER-X(,-SFP) devices on our network as backbone routers (in lightly loaded areas of our network), access routers, and access switches. In general, we are very satisfied with their hardware quality.
The software as a base is very nice as well, but the routing stack and firewall configuration are either inflexible, buggy, or unnecessarily hard to use, depending on each feature needed. As a result we use BIRD for dynamic routing and iptables-(save|restore) based scripts for those parts, and we are very happy that EdgeOS affords us this flexibility.
On access routers, each customer is confined to their own VLAN, and a static route added for their Public IP to that interface.
For example:
# First, set up customer VLANs and enable Proxy ARP on them using the
# EdgeOS configuration system. # Become superuser sudo -i # Reinstate standard Linux kernel ARP behaviour on all customer interfaces echo 0 | tee /proc/sys/net/ipv4/conf/switch0.1*/arp_* # Add a dummy address to the loopback interface - needed to
# demonstrate the bug ip addr add 172.30.30.30 dev lo # Add the source address for locally generated ICMP packets to
# the loopback interface ip addr add 192.0.2.1 dev lo # Add a static route to the first customer's VLAN ip route add 192.0.2.11 dev switch0.11 src 192.0.2.1 # Add a static route to the second customer's VLAN ip route add 192.0.2.12 dev switch0.12 src 192.0.2.1 # Start tcpdump on the ER-X so we can see the
# "source" address of generated ARP packets tcpdump -ni switch0.11 arp # Ping a customer's IP *from another machine connected to the ER-X!!!* ping 192.0.2.11 PING 192.0.2.11 (192.0.2.11) 56(84) bytes of data. ^C --- 192.0.2.11 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2000ms # On the ER-X, we see the following tcpdump output: 15:39:14.091677 ARP, Request who-has 192.0.2.11 tell 172.30.30.30, length 28 15:39:15.091607 ARP, Request who-has 192.0.2.11 tell 172.30.30.30, length 28 15:39:16.091520 ARP, Request who-has 192.0.2.11 tell 172.30.30.30, length 28
As you can see, the "source" IP address selected for ARP packets is not the one specified on the static route for the customer's VLAN.
The customer's CPE in VLAN 11 in this case would be set up with 192.0.2.11/24 as its WAN IP address, and 192.0.2.1 as its default gateway.
Some CPEs reply back to the ARP requests generated by the ER-X's kernel, and everything works as it should.
Other CPEs however, usually bought by the customer themselves, don't reply unless the "source" IP address in the ARP requests falls in the subnet set up on the WAN interface.
In that case, we have to add a static ARP entry for the customer's CPE on the ER-X as follows:
ip neigh add 192.0.2.11 lladdr 00:00:de:ad:be:ef dev switch0.11
This solves the problem until the customer decides to switch CPE. Then it's another escalated support call...