Linksys Router Slowing down my internet connection speed

I have a RV082 Linksys router and at the moment it is slowing down my internet connection by a significant amount. With the router, I get about 3 Mbps and with a modem directly connected to my computer I get about 24 Mbps. Before with the router I used to get 15 Mbps, which is understandable because it is connected to 4 computers total. Maybe my setup has something to do with it. I got a cable that is connected to a modem, the modem is connected to the router, from which one port is connected to a computer, and another is used to connect to a Netgear switch, which in turn is connected to 3 more computers.

Everything was fine until I went on a trip and wasn't home for about 2 weeks, and when I came back, this happened.

I tried restarting the modem, the router and all to no avail. I also tried to do a factory reset on the router and again got slow speeds.