The columns in Task Manager can give misleading figures -- for accurate memory usage per process, you could use perfmon (Performance Monitor in Control Panel) and look at the "private bytes" figures.
To show this, go into perfmon, click to add the counter (I'm a Ctrl+I kind of person). The "Performance object" would be "Process", with "Private bytes" being in the counters list -- obviously you'd need to select the relevant process from the right hand list.