The Hamiltonian constraint of General relativity has the following form
\begin{align} \frac{(2\kappa)}{\sqrt{h}}\left(h_{ac} h_{bd} - \frac{1}{D-1} h_{ab} h_{cd} \right)p^{ab} p^{cd} - \frac{\sqrt{h}}{(2\kappa)}\left({}^{(D)}\mathscr{R} - 2 \Lambda\right), \end{align} where $D$ is the number of spatial dimensions. Let's define
$$ G_{abcd} \equiv \frac{1}{\sqrt{h}} \left(h_{a(c} h_{d)b} - \frac{1}{D-1}h_{ab} h_{cd} \right). $$
How to define the determinant of $G_{abcd}$?
The Wheeler-DeWitt metric $$\begin{align} G~=~&G_{IJ}(\mathrm{d}y\odot\mathrm{d}y)^I\odot(\mathrm{d}y\odot\mathrm{d}y)^J\cr ~=~&G_{i_1i_2,j_1j_2}(\mathrm{d}y^{i_1}\odot\mathrm{d}y^{i_2})\odot(\mathrm{d}y^{j_1}\odot\mathrm{d}y^{j_2})\tag{1} \end{align}$$ is by definition a covariant (0,4) spatial tensor of a $D=d+1$ dimensional split spacetime ${\cal M}$ with local coordinates $(t,y^1,\ldots, y^d)$. The components $$\begin{align} G_{IJ}~=~&\frac{1}{2}(\gamma_{i_1j_1}\gamma_{i_2j_2}+\gamma_{i_1j_2}\gamma_{i_2j_1})-\frac{1}{d-1}\gamma_I\gamma_J~=~G_{JI},\cr G^{IJ}~=~&\frac{1}{2}(\gamma^{i_1j_1}\gamma^{i_2j_2}+\gamma^{i_1j_2}\gamma^{i_2j_1})-\gamma^I\gamma^J~=~G^{JI}, \end{align} \tag{2}$$ are built from the components $\gamma_I$ of a spatial metric $$\gamma~=~\gamma_I(\mathrm{d}y\odot\mathrm{d}y)^I~=~\gamma_{i_1i_2}\mathrm{d}y^{i_1}\odot\mathrm{d}y^{i_2}.\tag{3}$$ Here $I=i_1i_2=i_2i_1$ is a symmetric double index, i.e. the index $I$ can take $\frac{d(d+1)}{2}$ values, where $i_1,i_2\in\{1,\ldots, d\}$ run over spatial indices.
Due to the positive signature $(+,\ldots,+)$ we can diagonalize the spatial metric as a unit matrix $$ \gamma_{i_1i_2}~=~(m^Tm)_{i_1i_2}~=~m^{k_1}{}_{i_1}m^{k_1}{}_{i_2}, \qquad \det(\gamma_{..})~=~\det(m)^2,\tag{4}$$ where the $d\times d$ matrix $$m~=~\begin{pmatrix} \lambda_1 && \cr &\ddots& \cr && \lambda_d \end{pmatrix} O \tag{5}$$ is a diagonal matrix times an orthogonal matrix. If we define the $\frac{d(d+1)}{2}\times\frac{d(d+1)}{2}$ matrix $$M^I{}_J~:=~\frac{1}{2}(m^{i_1}{}_{j_1}m^{i_2}{}_{j_2}+m^{i_1}{}_{j_2}m^{i_2}{}_{j_1}),\tag{6}$$ then $$G_{IL}~=~(M^T)_I{}^J G^{(\gamma=\delta)}_{JK}M^K{}_L. \tag{7}$$
The determinant $$\det (G_{IJ})~=~\det(M)^2\det(G^{(\gamma=\delta)}_{IJ}) ~=~\frac{\det(\gamma_{..})^{d+1}}{1-d}\tag{8}$$ is over a $\frac{d(d+1)}{2}\times\frac{d(d+1)}{2}$ square matrix $G_{IJ}$, cf. OP's question.
Sketched proof of eq. (8). The matrix $G^{(\gamma=\delta)}_{IJ}$ has eigenvalue $\frac{1}{1-d}$ with multiplicity 1 and all other eigenvalues are 1, so the determinant is $$ \det(G^{(\gamma=\delta)}_{IJ})~=~\frac{1}{1-d}. \tag{9}$$ We can ignore the orthogonal matrix $O$ as it cannot change the determinant. Then $m$ is a diagonal matrix. We calculate: $$ \begin{align}\det(M)^2~=~&\left(\prod_{1\leq i\leq j\leq d} \lambda_i\lambda_j\right)^2 ~=~\left(\prod_{1\leq i,j\leq d} \lambda_i\lambda_j\right) \prod_{1\leq k\leq d}\lambda_k^2\cr ~=~& \left(\prod_{1\leq k\leq d}\lambda_k\right)^{2(d+1)}~=~\det(m)^{2(d+1)}~=~ \det(\gamma_{..})^{d+1}.\end{align}\tag{10}$$ $\Box$
