I wrote an earlier answer to the question: "Is true black possible?"earlier answer to the question: "Is true black possible?" which addresses some of the points you mention. It includes this image of "the blackest substance known":
In particular, most "black" objects have a small amount of reflectivity, from which you can deduce clues about their shape. But the sample of really black material which I show is SO black that it looks like there is a hole in space - you can't see any shape in the surface, and it does indeed look "invisible" in the sense that you don't really see the object, you see a "hole". But you don't see what is behind the hole: black is not the same as "transparent", and the black object absorbs all photons. It's not actually possible to make a material that is black in one direction and transparent in the other - there's a simple argument to be made that such a material would violate the laws of thermodynamics. Instead, in this world we're stuck with the "reciprocity theorem" that tells us that blackness is a two-way street.