You can't without "modifying any files in our system."
Its worth considering a few scenarios though. If its the only website hosted on the server you could just use the IP address. You'd want the server appropriately set up.
Lets assume - for example that you're running a 'simple' web server like the built in python one, on port 80. As far as the web server's concerned a request is a request, whether its to the IP, or any domain name pointed at it.
Practically speaking, you would use things like virtual hostnames and reverse proxies. The critical thing here is you need to tell the server what web page you want served - in theory, if you connected to a server via IP, and, as per the wikipedia article, you had a HOST
http header that pointed at website.site, it'd connect there. I've not personally done that, but this stack overflow answer indicates its possible with some extensions.
Practically its a lot more convoluted than running a 'local' DNS server or modifying a hosts file, and both of these are much better solutions than trying to 'avoid' DNS usage for accessing resources for any reason.