VOIP QoS with VLANs is affecting all traffic

Here is my setup

EdgeRouter 8-Pro 1.8.0

eth0

eth0.5 VOICE VLAN

eth0.10 SERVERS

eth0.50 COMPUTERS/LAPTOPS

eth1 WAN 10mb/10mb

I applied following configurtion taken and it seems to work too well because all the traffic is being throttled.

For example transfering files between server and personal computer is done over 10mb connection with throttling.

set traffic-policy shaper DownStream description "DownStream QoS policy"
set traffic-policy shaper DownStream bandwidth 10000kbit
set traffic-policy shaper DownStream class 10 description "RTP"
set traffic-policy shaper DownStream class 10 bandwidth 25%
set traffic-policy shaper DownStream class 10 ceiling 100%
set traffic-policy shaper DownStream class 10 match VOIP-RTP ip dscp 46
set traffic-policy shaper DownStream class 20 description "SIP"
set traffic-policy shaper DownStream class 20 bandwidth 5%
set traffic-policy shaper DownStream class 20 ceiling 100%
set traffic-policy shaper DownStream class 20 match VOIP-SIP ip dscp 24
set traffic-policy shaper DownStream default bandwidth 70%
set traffic-policy shaper DownStream default ceiling 100%

set traffic-policy shaper UpStream description "UpStream QoS policy"
set traffic-policy shaper UpStream bandwidth 10000kbit
set traffic-policy shaper UpStream class 10 description "RTP"
set traffic-policy shaper UpStream class 10 bandwidth 50%
set traffic-policy shaper UpStream class 10 ceiling 100%
set traffic-policy shaper UpStream class 10 match VOIP-RTP ip dscp 46
set traffic-policy shaper UpStream class 20 description "SIP"
set traffic-policy shaper UpStream class 20 bandwidth 10%
set traffic-policy shaper UpStream class 20 ceiling 100%
set traffic-policy shaper UpStream class 20 match VOIP-SIP ip dscp 24
set traffic-policy shaper UpStream default bandwidth 40%
set traffic-policy shaper UpStream default ceiling 100%

set interfaces ethernet eth0 traffic-policy out DownStream
set interfaces ethernet eth1 traffic-policy out UpStream

At first my guess was something to do with computers being connected to network through voip phones, but that has been rulled out since laptop is being throttled as well. 

Should I apply traffic-policy to each VLAN for DownStream QoS?