Think about it this way:
Start with only the magnetic field. The charged particle is on a circular orbit. The force the magnetic field exerts is proportional to its velocity. The faster the particle, the smaller the radius.
Lets say the particle rotates clockwise and is at its most left point in the orbit (i.e. velocity vector points in positive y-direction). Now turn on the electric field in positive y-direction. When the velocity points along the E field, you get maximal acceleration from the E field. So after a some time along this direction, the radius (probably better to call it curvature now) is going to get smaller. The particle's velocity vector then turns away from the direction of the E field and the acceleration decreases, until it points anti-parallel to the E field, where you have the smallest acceleration. Hence the radius is going to decrease.
Then everything starts again. Putting all of this together, you get a net drift in positive x direction.
For why the drift velocity is exactly $E/B$, I can't give you an intuitive answer. But it definitely makes sense that the drift velocity increases with E and decreases with B, so it is intuitive that it is an expression that combines these two thoughts.