L2TP VPN and Site-to-Site?

I feel like I'm either 1) doing this wrong or 2) thinking this wrong, please feel free to mock me as needed.

I have a Site-to-Site set up, works great, no issues.

With the impending PPTP drop from iOS and MacOS, and the fact that PPTP isn't all that secure these days, I wanted to get my employees switched up to L2TP/IPSec via an internal Windows server.

Changing up the Windows server was no issue, works like a champ.  Where I run into trouble is when I set up the port forwarding (1701, 4500, 500) to that windows sever... the Site-to-Site drops.

I realize this seems like a port-forwarding double-up which is no good so how do I make this work?