There are many standard methods. For example, you can get a proper test jig from Keysight or etc., cut a coupon of the material according to the instructions, and follow the measurement procedure. This is probably most economical when the utmost precision is needed and testing is done regularly; such instruments are not cheap. They detail that here:
https://www.keysight.com/us/en/assets/7018-01284/application-notes/5989-2589.pdf
which amusingly is locked behind a login, but is currently downloadable for free from here:
Basics of Measuring the Dielectric Properties of Materials | Keysight Technologies
On a more basic level, consider a calculator such as:
https://www.chemandy.com/calculators/sample-size-calculator.htm
and the related pages on the same site.
If nothing else, simply make a patch of known geometry, and apply a suitable capacitance formula. Give or take whether you've cut out a cylinder exactly the size of the round electrodes, or left it oversize (fringing in air vs. fringing in dielectric), or what the diameter to thickness ratio is, or choice of frequency, this may have various errors, but you can at least get close; and such a measurement can be done with equipment as simple as an oscilloscope and signal generator, or even just an AC voltmeter. For these, I would suggest using the calculations here,
LC Resonance | Calculators | Seven Transistor Labs (links to my site)
particularly "L or C by Resistor Divider" (scalar measurement, can use an AC (high frequency capable) voltmeter) and "Frequency and Q Factor" (resonate with a known inductor, solve for parameters).
Keep in mind that FR-4 is a material which varies significantly with temperature, and frequency (dispersion, loss); which also factors into antenna gain / efficiency, but more to the point, you should probably measure its properties near the frequency of interest, and this will require high-frequency rather than lumped-element methods.
There are also transmission line and resonator methods, for which I do not have citations offhand, but in which the dielectric constant appears as a parameter. These for example, consider it an introduction; you will have to find exact methods based on the keywords:
Measuring Dielectric Constant | Microwaves101
One of the simplest cases is merely laying down a copper strip of constant width, between a pair of SMA or whatever connectors, with the housings well bonded to the backside ground plane. Run this on a VNA or TDR to find the transmission line impedance, and work backwards from dimensions to find κ.