First: How did you check the voltage on the 3V3 ? using an oscilloscope ? A multimeter won't do the job here. It won't catch the very narrow voltage drop due to the huge current pulses of the ESP32. And if you are experiencing brownout reset, there is an very high probability that there is such a voltage drop, even extremely short.
You may also have a look at that video : https://www.youtube.com/watch?v=LUB8RWzzLWc&t=2s :
It is related to measuring small currents, but at some point, it shows brownout issues with an ESP32 and measures its associated current peaks. Might be informative for your particular issue.
As said by @Kartman, the kind and placement of the decoupling capacitors can make a big difference here. What kind is it ? Ceramic I hope. Large electrolytic capacitors have poor ESR. They are good for storing energy "slowly", but they are useless for absorbing current peaks of digital chips. Check the datasheet of the various chips. 100nF might not be the way to go, especially with radio chips that may require much lower value to catch very fast transients.
To illustrate the remarks of @Kartman, real capacitors are not like ideal ones. The capacitance derating vs DC voltage can have a very strong impact. Here is a derating curve for a Y5V dielectric capacitors.
Imagine that you are using a Y5V 10uF 4V ceramics caps on your 3V3 power rails. 3.3V is 85% of the 4V rating. Thus looking at the curve, the derating is -90%. Your 10uF capacitor behave like a 1uF !! Thus the exact choice of the component matter. For instance, you could use other dielectric such as X7R which has a much much lower DC voltage derating, or choose a part rated for 25V or more.
With only the schematics given, the only thing that makes me unhappy is the location of Q5. It adds a small resistor between the VCC pin of the GPS and its decoupling capacitor. That is very bad, especially for a device like a GPS that is doing radio stuff.
Please post screenshots of the layout as well and I could extant my answer accordingly. The component placement and the power routing scheme is likely to the the source of your issue, but we can't look at it without a picture of the layout.