Yes, your network would be slower between two clients than just the one client transferring data using the wireless router. This is because at any given time there can only be one computer/NIC that transfers data on the media. This means that while one computer is transferring data to or from the router all other computers have to wait until the computer already transferring data is done before they can transfer their data.
If you have your server on wire while your clients are on wifi, then the server and one client can communicate with the router at the same time and work more efficiently. This might give you a better transfer speed, but you might not notice the difference.
If you have a relatively new wireless access point and computers that supports the new N standard and that uses the 5GHz band, then there is a good chance you will be satisfied using the wireless for your HTPC. In all other instances this will become very dependent on how many other wireless networks are their close by, what kind of hardware do you have in the computers and server and how much data do you expect to be transferred simultaneous on the media?
In short; If you have equipment that support 5GHz and the N standard, then your network will be on a less crowded band and support networks speed of up to 300Mbps which should be good enough for many private networks. If you have old equipment my personal experience is that in best case scenario you will think that things are OK, but not really be satisfied with the result.
Cheers.