What would make you think that charges that aren't on the plane (at which my line integral is) doesn't affect the magnetic field. It is true that the LINE INTEGRAL OF B DL is only affected by current that are enclosed by that closed path. this line integral doesn't equal the magnetic field
In fact, "I enclosed" doesn't actually have to be a flat surface, this surface can be a e.g bag shape such that the I enclosed is actually on the wire at a different plane than the boundary curve is on, pretty cool huh?
I'm sure your confusion about I enclosed is the only contribution to the magnetic field comes from the derivation from Ampere's law
$$\int \mathbf B \cdot d\mathbf l = \mu I_{enclosed}$$
For an infinitely straight wire, if I pick a certain circular curve I would like to find the value of $B$ at. the $B$ field along that curve is parallel to the curve
thus
$$\int B dl = \mu I_{enclosed}$$
The $B$ field along that curve is constant for all points along that curve so $B$ is independent of the integral
$$ B \int dl = \mu I_{enclosed}$$
Solving for $B$ in this form is trivial.
Now, why does this integral "only depend on the I enclosed at a single point" and not for other points along the curve. this is due to the fact that I can do the same operation on any current element down the wire and achieve the same result. thus the magnetic field formula must work all points on the wire. another way of putting it is that the formula you have described assumes that I enclosed is THE SAME for all points on the wire which means it applies for any line integral we chose at any point on the wire.
I was going to show you that the magnetic field of a single current element is different than one for a straight wire however using amperes law and symmetry gives the field as being the same as one from an infinite wire along a curve perpendicular to the current. However this is due to the fact that this version of amperes law only works for $\nabla \cdot J = 0$ and a single current element doesn't fit this condition. so the formula breaks down and gives a false result