The effect of getting shaped out-of-focus highlights (bokeh balls) from both non-circular apertures inside the lens, and masks applied in front of it (where an external aperture would presumably also be effective) is well established.
Is the effect because of the aperture acting as a vignette too, just as the external vignette would ... or is it a result of the "magic" that makes an aperture NOT vignette?
Well, the aperture is in the focusing plane which means that every point in 3D reality that is in focus is equally affected in brightness regardless of where you stop light passing through the aperture. Basically in the focusing plane, every in-focus point is presented not by the location in the aperture plane but by the direction to the image sensor it is taking (where the beams from all in-focus points converge to a single point).
That is not the case for points that are out of focus. Their focusing plane is removed from the aperture. If it's outside of the aperture plane (for objects more remote than the in-focus objects), the various points "shine" through in the shape of the aperture, producing an upright image of the aperture (which is the opposite orientation of the upside-down in-focus image). If it's inside of the aperture plane, the image is upright, so you can distinguish whether an out-of-focus highlight is too far or to near, depending on whether it is producing an upside-down image of the aperture shape or not.
Now anywhere else but in the focusing plane we get vignetting. That means that different points of the image are differently affected, even when in-focus, by darkening different point of the opening. Much of the light entering the lens is from much enough of a distance that it counts as mostly parallel, meaning that the vignetting will be very fuzzy and the aperture effect, creating image disks for out-of-focus points, will prevail over seeing the vignetting. However, for wide-angle lenses and/or near focus and/or high depth of focus, you'll likely see the shape of any cutouts before the lens.
This kind of almost-like-an-aperture effect is used when photographing with wide aperture through a wire fence directly in front of the lens. You'll not see the mesh but your bokeh may.
So basically it is a play of different sizes here: as long as your aperture is open enough and your focus far enough that the cutout shape is solidly out of focus, you'll not see much of the vignetting and the cutout shape will instead appear as bokeh.
As you close down the aperture, the cutout shape will come more into focus and will determine less of the bokeh but more of the vignetting.
Any "hole" (like aperture or mask) is "represented" on the sensor. Ray of light will be stopped on the mask or on the aperture, it will pass only through the hole and hit the sensor. Unless it is in focus (which means rays of light will meet in the same spot), light will spread out across the sensor in the same shape. This "magic" is physics, or optics to be more precise. If you really want to dig the topic, there is a nice article about this here: http://www.marcuswinter.de/archives/1703
