Sounds like something is dragging your network down. I have had slow fire stick restores through WiFi and found I just had too many wireless signals cluttering up my house. I hard wired what I could (reneged a wireless access point) and have not looked back.
When loading to fire sticks I find doing them via usb is more efficient. Of course this won’t fix your WiFi . adbLink and USB