Diagonal brace with nested parenthesis

Question

I would to obtain a result similar to the image below. So far I only have \bigg( \Big( \big( a^q \big)^q \Big)^{\rotatebox{25}{$\cdots$}} \bigg)^q, without the brace.

I tried to follow this question, but there were no parenthesis and I was not able to circumvent this.

Answer 1

Very easy with horizontal brace; somewhat less easy with rotated brace. I have no time right now to implement a general solution, so I'll go with a brute-force version for now.

\documentclass{article}

\usepackage{mathtools}% loads amsmath and graphicx

\begin{document}

\[
\biggl(\Bigl(\bigl(a
{% extra group for correct spacing
 \overbrace{
     {}^q \bigr)^q \Bigr)^{\rotatebox{20}{$\scriptstyle\dots$}} \biggr)^{\!\!q\mathstrut}
 }^{\text{$n$ times}}
}
\]

\[
\biggl(
\Bigl(
\bigl(a^q\bigr)^{q}
\Bigr)\quad
\biggr)^{
         \!\! q
         \setbox0=\hbox to 0pt{\hss
            \rotatebox{15}{$
              \overbrace{\kern1.8em\scriptstyle\mathstrut\dots\kern1.2em}^{\rotatebox{-15}{\scriptsize$n$ times}}
            $}%
         }%
         \lower1.5ex\box0
        }
\]

\end{document}

Answer 2

As given in my answer at Nested Mathematical Functions

The .4pt stacking gap added with each nesting will determine the growth rate of the parens.

\documentclass{article} 
\usepackage{scalerel,stackengine,graphicx,amsmath}
\stackMath
\ignoremathstyle
\newcommand\Comp[1]{\scaleleftright{(}{\addstackgap[.4pt]{#1}}{)}}
\newcommand\rdots{\ThisStyle{.\raisebox{.1ex}{$\scriptstyle.$}%
  \raisebox{.2ex}{$\scriptstyle.$}}}
\begin{document} 
\[ 
\stackinset{r}{1pt}{t}{-14.5pt}{\rotatebox{6}
  {$\overbrace{\rule{30pt}{0pt}}^{\textrm{\tiny$n$ times}}$}}{%
\Comp{\Comp{\Comp{a^q}^q}^{\rdots}}^q}
\] 
\end{document}

Here's a version where the stackgap is increased to .6pt, to bring out the increasing size of parens:

Answer 3

A solution with pstricks-add:

\documentclass{article}
\usepackage{pstricks-add}

\begin{document}

\[
\biggl(\Bigl(\bigl(a^{\pnode[0,1ex]{E}{q}}\bigr)^{\! \rnode{F}{\scriptstyle q}}\Bigr)\enspace \biggr)^{\!\!\rnode{G}{\scriptstyle q}\pnode[0,1ex]{H}}%
\psset{singleline, braceWidth=1.2pt, braceWidthInner=2pt, braceWidthOuter=2pt, nodesepB=-2pt}
\psbrace[rot=14,ref=c, linewidth=0.3pt](H)(E){\scriptstyle n\:\mathrm{times}}%
\ncline[linestyle=dotted, dotsep=2.pt, nodesepA=4.5pt, nodesepB=3.2pt]{F}{G}
\]

\end{document}