The kernel and the NIC communicate the same way the kernel
(specifically, a device driver, which is part of the kernel)
communicates with any other device (e.g., serial communications device
(like keyboard, mouse, or RS232), disk (or disk-like mass storage device),
display, security token, etc.) – they talk directly to each other.
So, yes, this happens within the NIC driver.
No, driver ↔ device communication does not go through a system call.
Device drivers communicate with devices, generally, through two access methods:
Memory-Mapped I/O (MMIO) and Port-Mapped I/O (PMIO).
These are described and discussed at some length on Super User
in these two questions:
See also What is a driver, and how does it work?
if you need the background information.
BTW, which access method is used is determined by the computer architecture.
For example, Wikipedia says,
“Memory-mapped I/O is preferred in x86-based architectures …”
Getting back to your specific question, if we assume memory-mapped I/O,
the driver could test for collision with C code as simple as
if (nic->error_status & COLLISION)
{
// We get here if there was a collision.
(code to handle collision)
⋮
}
(where nic is a pointer to the NIC’s mapped address)
and it’s a trivial matter then for the driver to cause an error
to be returned to the user process from the write() system call.
Follow-up question:
Does the fact that drivers are part of the kernel mean that
there is no driver ↔ kernel communication method?
That might almost be worth asking as another question
(or at least doing some more research; i.e., web searches).
Originally, the Unix kernel was monolithic
(see also monolithic kernel).
The relationship between the kernel and the drivers
was like the relationship between a human body and its hands —
hands are distinctive,
and would never be confused with, say, elbows or lungs —
but they are an integral part of the body.
Even on machines that had four protection rings, Unix used only two —
the kernel was in ring 0 and userland was ring 3.
There are other operating systems where device drivers
are analogous to knife and fork — one level removed
(e.g., drivers are in one of the intermediate rings (1 and/or 2)).
I haven’t kept up with all the developments in *nix/Linux (e.g., LKMs),
and some unices may have a greater degree of separation
between the base kernel and the drivers than others.
For example, see
As I say, this stuff is at the edge of my knowledge base;
don’t ask me any further follow-up questions on the above.
But also, the question is semantically ambiguous.
Does the body communicate with the hands?
Yes, through the nervous system (and, to an extent, the skeleton).
Does “the kernel” communicate with device drivers
(that are part of the kernel)?
Yes, in the same sense that C programs (e.g., cat, cp, ls, sed, etc.)
communicate with the C library — through subroutine calls
(and perhaps the occasional, judicious use of global variables).
When a user process calls an I/O-related system call (e.g., open(),
close(), read(), write(), ioctl(), poll(), select(), etc.),
the general-purpose syscall-handling code (in the kernel)
calls the appropriate device driver routine
(possibly with some intermediate logic).
Filesystem-related code also calls the appropriate disk driver routines.
And the kernel at least facilitates the invocation
of device-specific interrupt handlers when interrupts are received
(implementations vary).
