Basically, I am exploring Gentoo, and I would like to be able to perform some kind of graphics rendering (open a jpg, or draw basic shapes, or even set the colors of individual pixels if I have to).
I do not have any desktop or window manager, and I would rather not need one, but that is exactly the question. What is the most lightweight/simplest way to render graphics?
You can display graphics using the Linux frame-buffer interface without X11 or Wayland at all.
The fbida package includes the fbi image viewer, which can run directly on the virtual console. You can't get much more light-weight than that.
While not as lightweight as direct framebuffer access, SDL has no dependencies on the X server, and provides a much broader selection of applications, including image viewers and video players.
Check out Is it possible to view media on the console? for software recommendations, many of them SDL-based.
Note that unless we're talking about a very old computer or an embedded system, you won't gain a substantial amount of memory or CPU time by not using X.
