Is this DOCSIS (that is, Internet service from a Cable TV provider)? If so, first look up the real manufacturer and model# of the router (look for a label on the back or bottom of it) and make sure it's not on the badmodems.com list of known-bad modems based on the crappy Intel Puma 6 chipset that has known latency problems. If it's on the list, that's enough reason to have them replace it.
You need to look up the real manufacturer and model# if your ISP has slapped their own branding on it. For example, Comcast uses names like "Xfinity xFi" or "Xfinity XB3" for a number of different routers from different manufacturers, that all more or less meet the same specifications.
The client device you're pinging from…is it on wired Ethernet or Wi-Fi? Ping time on wired Ethernet should be about 0.3ms. Ping time over one Wi-Fi hop (like pinging your router's LAN IP from a wireless client) on a reasonably clean channel and otherwise idle Wi-Fi network should be about 3ms. But Wi-Fi is prone to interference and going off-channel to scan, so spikes into the low hundreds of milliseconds are common.
If this is over Wi-Fi, then to be fair to the ISP, try to eliminate poor radio signal/noise conditions as a factor. Put your client device between 2-5m of the router, with clear line-of-sight. Use a tool like inSSIDer to help you pick the cleanest channels to put your router's Wi-Fi radios on. Make sure everything else on your Wi-Fi network is idle, and that you're not using any kind of repeaters or range extenders for this test.
