Let me first start by saying that there is no symmetry between them as "symmetry" has a particular technical meaning to it. There is, however, a very precise relation between the two.
Essentially it all comes down to the equations determining the electrostatic field and the gravitational field. Looking at Maxwell's equations, we can see that in the special case where nothing is time-varying, the determining equations for the electric field are
$$
\nabla\cdot\vec E=\rho/\varepsilon_0,\ \ \ \ \nabla\times\vec E=0,
$$
where $\rho$ is the charge density.
On the other hand, the equations determining the (Newtonian) gravitational field are
$$
\nabla\cdot\vec g=-4\pi G\rho,\ \ \ \ \nabla\times\vec g=0,
$$
where $\rho$ in these equations is the mass density instead.
Clearly these equations are the same if we identify $\vec E\leftrightarrow\vec g$ and $-4\pi G\leftrightarrow 1/\varepsilon_0$. That is, all the solutions for gravity and electrostatics will necessarily "look" the same. This observation also tells us when the analogy between the two breaks down: as soon as things are time-varying (so we are no longer dealing with electrostatics), the rest of Maxwell's equations come into play and the equations for the electric field no longer look like this.
With the mapping between electrostatics and (Newtonian) gravity understood, let me try and clarify some of the other things mentioned in OP. The expression
$$
\vec E=\frac{k q}{r^2}\hat{r}
$$
can be checked to be the solution to the above electrostatic equations when the charge density $\rho$ is just that of a point charge (Dirac delta function). We will keep this in the back of our minds for just a moment.
Now, note that there are two equations, not just one. The second equation, which states that the curl is zero, tells us that there exists a function $V$ such that $\vec E=-\nabla V$ (the minus sign is conventional). This is nothing more than a mathematical fact which can be stated in a number of different ways. For example, by Stokes' theorem the statement $\nabla\times \vec E=0$ implies that the line integral of $\vec E$ is independent of the path (depends only upon the endpoints). In physics, this is often phrased as saying the vector field is "conservative," though that language is typically used in the context of forces.
Recall that if a force $\vec F$ is conservative, then the work done alone any curve $C$, $W_C=\int_C\vec F\cdot d\vec r$ is independent of the path $C$ and depends only upon the end points. This is usually then used to assert the existence of a potential function $U$ such that $\int_C\vec F\cdot d\vec r=-(U(b)-U(a))=-\Delta U$. Using again a general result from vector calculus, this tells us that $\vec F=-\nabla U$.
That's the story that usually gets described, in one way or another, in physics. But all the statements I made are just statements about a vector field $\vec F$ which obeys certain properties (path independence of the line integral). So the same results apply to any other vector field which obeys the same properties. As I mentioned, $\nabla\times \vec E=0$ implies precisely this "path independence" property, so all the same mathematical results must still apply: there exists a function $V$ such that $\Delta V=-\int_C\vec E\cdot d\vec r$ and $\vec E=-\nabla V$. By the way, this is technically correct version of the $E=V/d$ equation you have written. That equation is a special case of the line integral I have written here in which the electric field is a constant along the curve (so essentially we can just factor it out of the integral and the remaining integral is just the length of the curve.
There are many other things that could be said, for example in the same way potential energies are only defined up to an additive constant, the same is true of the electric potential $V$, but I think this is probably enough information to get started sorting things out.
Edit: There was one other thing I planned to say and then promptly forgot. The source of the expression
$$
V=\frac{kq}{r}
$$
itself comes from the things I mentioned above. In particular, if we take the electric field for a point charge (which I mentioned earlier) and compute $-\int_C\vec E\cdot d\vec r$ along a curve which runs from infinity down to some radial location a distance $r$ from the origin (a straight line is simplest for the calculation, and we already argued the path we take doesn't matter, so this choice of integration path is without loss of generality), the result is the expression for $V$ I have just written.
Since the electric field for a point charge and the force due to a point charge differ only by the multiplication of a (constant) test charge, it follows that the electric potential and electric potential energy of a point charge only differ by a factor of the test charge's charge. We can note that this relation does not continue in the way we might hope to more complicated composite configurations (which are composed of many point charges of infinitesimal charge). For example, the potential energy contained in a capacitor is given by $U=\frac{1}{2}QV$ so there's an extra factor of $1/2$ which comes up.
