Hi all,
I have an ERLite-3 that I've connected to AWS via an IPSec VPN. I've followed the AWS-supplied configuration as best I could, but I'm encountering throughput performance issues.
When running iperf3 over the VPN tunnel (with an m4.large instance):
# iperf3 -c 172.31.11.73 -i 1 -t 15 -V
iperf 3.0.11
Linux ubuntu 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Time: Sat, 30 Jul 2016 05:21:11 GMT
Connecting to host 172.31.11.73, port 5201
Cookie: ubuntu.1469856070.948124.1f203e7d33f
TCP MSS: 1367 (default)
[ 4] local 172.16.10.172 port 50240 connected to 172.31.11.73 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 7.06 MBytes 59.2 Mbits/sec 5 454 KBytes
[ 4] 1.00-2.00 sec 7.17 MBytes 60.1 Mbits/sec 0 465 KBytes
[ 4] 2.00-3.00 sec 7.17 MBytes 60.1 Mbits/sec 0 475 KBytes
[ 4] 3.00-4.00 sec 6.62 MBytes 55.5 Mbits/sec 0 486 KBytes
[ 4] 4.00-5.00 sec 6.62 MBytes 55.5 Mbits/sec 0 503 KBytes
[ 4] 5.00-6.00 sec 7.35 MBytes 61.7 Mbits/sec 0 547 KBytes
[ 4] 6.00-7.00 sec 7.17 MBytes 60.1 Mbits/sec 0 609 KBytes
[ 4] 7.00-8.00 sec 7.35 MBytes 61.7 Mbits/sec 0 693 KBytes
[ 4] 8.00-9.00 sec 6.68 MBytes 56.0 Mbits/sec 15 768 KBytes
[ 4] 9.00-10.00 sec 6.86 MBytes 57.6 Mbits/sec 0 774 KBytes
[ 4] 10.00-11.00 sec 6.99 MBytes 58.6 Mbits/sec 0 781 KBytes
[ 4] 11.00-12.00 sec 7.17 MBytes 60.1 Mbits/sec 0 786 KBytes
[ 4] 12.00-13.00 sec 6.37 MBytes 53.5 Mbits/sec 0 805 KBytes
[ 4] 13.00-14.00 sec 6.43 MBytes 54.0 Mbits/sec 0 840 KBytes
[ 4] 14.00-15.00 sec 6.86 MBytes 57.6 Mbits/sec 0 890 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-15.00 sec 104 MBytes 58.1 Mbits/sec 20 sender
[ 4] 0.00-15.00 sec 103 MBytes 57.6 Mbits/sec receiver
CPU Utilization: local/sender 0.6% (0.1%u/0.5%s), remote/receiver 3.0% (0.0%u/3.0%s)When running the same test to another m4.large instance with a public IP:
# iperf3 -c 54.164.16.73 -i 1 -t 15 -V
iperf 3.0.11
Linux ubuntu 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Time: Sat, 30 Jul 2016 05:21:35 GMT
Connecting to host 54.164.16.73, port 5201
Cookie: ubuntu.1469856095.753119.400223fd0aa
TCP MSS: 1448 (default)
[ 4] local 172.16.10.172 port 35908 connected to 54.164.16.73 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 44.2 MBytes 371 Mbits/sec 1 1.63 MBytes
[ 4] 1.00-2.00 sec 51.2 MBytes 430 Mbits/sec 0 1.65 MBytes
[ 4] 2.00-3.00 sec 53.8 MBytes 451 Mbits/sec 0 1.68 MBytes
[ 4] 3.00-4.00 sec 52.5 MBytes 440 Mbits/sec 0 1.70 MBytes
[ 4] 4.00-5.00 sec 53.8 MBytes 451 Mbits/sec 0 1.72 MBytes
[ 4] 5.00-6.00 sec 55.0 MBytes 461 Mbits/sec 0 1.74 MBytes
[ 4] 6.00-7.00 sec 55.0 MBytes 461 Mbits/sec 0 1.79 MBytes
[ 4] 7.00-8.00 sec 57.5 MBytes 482 Mbits/sec 0 1.88 MBytes
[ 4] 8.00-9.00 sec 61.2 MBytes 514 Mbits/sec 0 1.99 MBytes
[ 4] 9.00-10.00 sec 61.2 MBytes 514 Mbits/sec 70 2.03 MBytes
[ 4] 10.00-11.00 sec 55.0 MBytes 461 Mbits/sec 198 1.51 MBytes
[ 4] 11.00-12.00 sec 48.8 MBytes 409 Mbits/sec 118 1.58 MBytes
[ 4] 12.00-13.00 sec 50.0 MBytes 419 Mbits/sec 0 1.60 MBytes
[ 4] 13.00-14.00 sec 51.2 MBytes 430 Mbits/sec 0 1.62 MBytes
[ 4] 14.00-15.00 sec 51.2 MBytes 430 Mbits/sec 0 1.64 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-15.00 sec 802 MBytes 448 Mbits/sec 387 sender
[ 4] 0.00-15.00 sec 800 MBytes 448 Mbits/sec receiver
CPU Utilization: local/sender 2.3% (0.1%u/2.2%s), remote/receiver 0.7% (0.1%u/0.5%s)You can clearly see there is a massive difference in terms of throughput. Additionally, when running the iperf3 tests, my ERLite-3 appears to immediately max the CPU at 100% and remain there for the duration of the test. I'm wondering if there are any performance limitations when it comes to IPSec tunnels on the EdgeRouter platform.
It's worth noting that there are a few items in the AWS-provided configuration, which don't appear to be configurable:
- Clear Don't Fragment (TCP DF) bit - enable
- Fragmentation - before encryption
Here's my configuration (a double-check on this would be nice):
firewall {
all-ping enable
broadcast-ping disable
ipv6-receive-redirects disable
ipv6-src-route disable
ip-src-route disable
log-martians enable
name LOCAL {
default-action accept
description ""
}
name WAN_IN {
default-action reject
description "WAN to internal"
rule 10 {
action accept
description "Allow established/related"
state {
established enable
related enable
}
}
rule 20 {
action reject
description "Reject invalid state"
log disable
state {
invalid enable
}
}
}
name WAN_LOCAL {
default-action reject
description "WAN to router"
rule 1 {
action accept
description "Allow established/related"
state {
established enable
related enable
}
}
rule 2 {
action reject
description "Reject invalid state"
log disable
state {
invalid enable
}
}
rule 3 {
action accept
description "Allow pings"
destination {
}
log disable
protocol icmp
source {
group {
}
}
}
}
options {
mss-clamp {
interface-type vti
mss 1387
}
}
receive-redirects disable
send-redirects enable
source-validation disable
syn-cookies enable
}
interfaces {
ethernet eth0 {
address 10.20.81.2/30
description WAN
duplex auto
firewall {
in {
name WAN_IN
}
local {
name WAN_LOCAL
}
}
speed auto
}
ethernet eth1 {
address 172.16.10.1/23
description "LAN"
duplex auto
speed auto
}
ethernet eth2 {
}
loopback lo {
}
vti vti0 {
address 169.254.44.186/30
mtu 1436
}
vti vti1 {
address 169.254.44.22/30
mtu 1436
}
}
port-forward {
auto-firewall enable
hairpin-nat disable
lan-interface eth1
wan-interface eth0
}
protocols {
static {
route 0.0.0.0/0 {
next-hop 10.20.81.1 {
}
}
route 172.31.0.0/16 {
next-hop 169.254.44.21 {
}
next-hop 169.254.44.185 {
}
}
}
}
service {
dhcp-server {
disabled false
hostfile-update disable
shared-network-name LAN {
authoritative enable
subnet 172.16.10.0/24 {
default-router 172.16.10.1
dns-server 216.52.129.1
dns-server 8.8.8.8
lease 86400
start 172.16.10.20 {
stop 172.16.10.254
}
}
}
}
dns {
}
gui {
http-port 80
https-port 443
older-ciphers enable
}
mdns {
reflector
}
nat {
rule 5010 {
outbound-interface eth0
type masquerade
}
}
snmp {
}
ssh {
port 22
protocol-version v2
}
ubnt-discover {
disable
}
upnp2 {
listen-on eth1
nat-pmp enable
secure-mode enable
wan eth0
}
}
system {
host-name ubnt
login {
user <REDACTED> {<REDACTED>
}
}
name-server 8.8.8.8
name-server 8.8.4.4
ntp {
server 0.ubnt.pool.ntp.org {
}
server 1.ubnt.pool.ntp.org {
}
server 2.ubnt.pool.ntp.org {
}
server 3.ubnt.pool.ntp.org {
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone America/Chicago
traffic-analysis {
dpi disable
export enable
}
}
vpn {
ipsec {
auto-firewall-nat-exclude disable
esp-group esp-aws {
compression disable
lifetime 3600
mode tunnel
pfs enable
proposal 1 {
encryption aes128
hash sha1
}
}
ike-group ike-aws {
dead-peer-detection {
action restart
interval 15
timeout 30
}
ikev2-reauth no
key-exchange ikev1
lifetime 28800
proposal 1 {
dh-group 2
encryption aes128
hash sha1
}
}
site-to-site {
peer <AWS IP1> {
authentication {
id <MY IP>
mode pre-shared-secret
pre-shared-secret <REDACTED>
}
connection-type initiate
default-esp-group esp-aws
description tunnel1
ike-group ike-aws
ikev2-reauth inherit
local-address 10.20.81.2
vti {
bind vti0
esp-group esp-aws
}
}
peer <AWS IP2> {
authentication {
id <MY IP>
mode pre-shared-secret
pre-shared-secret <REDACTED>
}
connection-type initiate
default-esp-group esp-aws
description tunnel2
ike-group ike-aws
ikev2-reauth inherit
local-address 10.20.81.2
vti {
bind vti1
esp-group esp-aws
}
}
}
}
}
Thanks in advance for your help!