Triple WAN with Fail-Over & PBR => load balance breaks PBR

Hi Community & UBNT-Staff,

After several attempts to setup my desired -quite simple- configuration (no firewall in place except the below) my PBR rules always stop working after adding the load-balancing. Deleting the rules doesn't restore the previous state, it at least takes a reboot to restore the previous state -or even a full reset and restore a backup. The latter actually makes me question if the config is the only "permanent" storage in the system. Which is weird as iptables not saved to file are usually cleared on reboot. In any case firewall modify rules are often not properly deleted in real time. These will be only cleared on reboot. At least I observed this behaviour on earlier releases, which can make live configuration very tricky as the system needs a reboot to correctly apply the rules of the config (read: breaks things after a reboot => bad surprise).

Logically the system consists of 3 SNATed WAN links (one being OVPN TAP) and 2 LAN links (one being VLAN).

• eth0: LAN (192.168.192.x/24 where .254 is the router and .1 the server)
• eth0.100: VLAN for Guest WiFi (irrelevant atm)
• eth1: Primary WAN (DHCP)
• eth2: Fallback WAN (DHCP, but not even connected yet)
• vtun0: WAN Link for Server only (by PBR) with manual static address (10.99.99.1; gateway 10.99.99.2)

Rule 1-3 is what worked before adding WLB. Rule 4-100 where added together with WLB. As things didn't work out I tried marking "route over VPN"-Packets mit Mark 10 and excluding marked packets from WLB. Without success. Clearing rules 4-100 didn't make things work again, as mentioned above I had to go through a hard reset and restoring a backup, which makes try and error troubleshooting extremely tedious...

Also, is there any up2date documentation on what "load-balance" (acc. https://help.ubnt.com/hc/en-us/articles/205145990-EdgeMAX-Dual-WAN-load-balance-feature) actually does compared to manual load balancing according https://help.ubnt.com/hc/en-us/articles/205202690-EdgeMAX-Policy-based-routing-with-WAN-load-balancing ?

Thanks in advance!

group WLB 
     interface eth1 {
     }
     interface eth2 {
         failover-only
     }
     lb-local enable            <<< I assume this is required for the local OVPN tunnel to failover
     sticky {                   <<< I assume this also takes care of DNAT return packets
         dest-addr enable
         dest-port enable
         source-addr enable
     }
 }
___________________________________

modify LAN_IN {
     rule 1 {
         action modify
         description "Restore Mark"
         modify { 
             connmark {
                 restore-mark
             }
         }
     }
     rule 2 {
         action modify
         description "Catch inbound VPN Connections and assign Mark 10"
         destination {
             address 192.168.192.0/24
         }
         mark 10
         modify {
             table 10
         }
         protocol tcp_udp
     }
     rule 3 {
         action modify
         description "outbound serverconnections over VPN"
         modify {
             table 10
         }
         protocol tcp
         source {
             address 192.168.192.1
         }
     }
     rule 4 {        <<< this rule shouldn't be necessary, added for troubleshooting
         action modify
         description "mark outbound servertraffic over VPN to definitely exclude from LB"
         modify {
             mark 10
         }
         protocol tcp
         source {
             address 192.168.192.1
         }
     }
     rule 8 {        <<< 10.99.99.x is the OVPN subnet => no LB for 10.x.x.x
         action modify
         destination {
             address 10.0.0.0/8
         }
         modify {
             table 10         <<< I tried "main" table previously
         }
     }
     rule 9 {                 <<< exclude LAN2LAN-traffic from LB
         action modify
         destination {
             address 192.168.0.0/16
         }
         modify {
             table main
         }
     }
     rule 98 {               <<< exclude traffic to public IP from LB - Why is this necessary? Just followed the guide
         action modify
         destination {
             group {
                 address-group ADDRv4_eth1
             }
         }
         modify {
             table main
         }
     }
     rule 99 {                <<< this rule shouldn't be necessary, added for troubleshooting only
         action modify
         destination {
             group {
                 address-group ADDRv4_eth1
             }
         }
         modify {
             mark 10
         }
     }
     rule 100 {
         action modify
         mark !10             <<< this shouldn't be necessary, added for troubleshooting
         modify {
             lb-group WLB
         }
     }
 }
 modify vtun0_IN {
     rule 1 {
         action modify
         description "Mark new connections with Mark 10"
         modify {
             connmark {
                 set-mark 10
             }
         }
         protocol all
         state {
             new enable
         }
     }
 }

 static {
     table 10 {
         mark 10
         route 0.0.0.0/0 {
             next-hop 10.99.99.2 {
             }
         }
     }
 }

 Thanks in advance"