Perlin noise's gradient vectors are exactly what you describe: the vector derivative of the continuous noise function.
Specifically, each gradient is the derivative of the noise function at one corner of the underlying integer grid.
Perlin noise is set up so that at each integer point in its domain, it is "locally flat," like a straight line or plane with a particular slope given by that corner's pseudorandomly-selected gradient vector, passing through a value of zero at the corner point itself.
In this diagram of 1D Perlin noise from Scratchapixel, we can see how Perlin noise is constructed by choosing a gradient at each integer coordinate along the X-axis, giving us the slope of a line crossing through that point (the blue arrows).
Very close to this point, the noise function behaves like that straight line, with exact equality in both value and derivative at the integer points themselves (this is what ensures we don't get seams between adjacent blocks of Perlin noise)
In-between these integer points, we blend between these lines/planes through the \$2^n\$ closest integer points in the domain. The dot product of our offset from the integer point with the gradient at that point gives us the value we should have if we continued the flat line/plane through that point out further. Interpolating these \$2^n\$ planar points gives us a compromise value that's generally not on any of the corner planes, but can smoothly bend from one to the other as we move around the domain.
In this in-between region, the gradient of our function is no longer the gradient at any of the corners, but a blend between them that we can calculate the same way we do for the value.
