I need a user-defined command to behave differently depending on whether it's invoked inside a \begin{quotation} environment or not. Is that possible? I don't know where to start!
2 Answers
A general solution would be to either define an if switch (aka boolean but I try to avoid this name to avoid mix-up with LaTeX3's boolean type) and (un)set it at the beginning of the environment or redefining the command itself at the beginning of the environment. You can use, e.g., the generic LaTeX environment hooks for this:
\documentclass{article}
\usepackage[svgnames]{xcolor}
\newif\ifinsidequotation
\newcommand*{\foo}[1]{%
\ifinsidequotation\textcolor{LightGreen}{#1}\else\textcolor{DarkGreen}{#1}\fi}
\AddToHook{env/quotation/begin}
{\insidequotationtrue}
\begin{document}
This is \foo{DarkGreen} but
\begin{quotation}
Here you have \foo{LightGreen} and
\begin{minipage}{5em}also \foo{LightGreen} here\end{minipage}.
\end{quotation}
Here it is again \foo{DarkGreen}.
\end{document}
\documentclass{article}
\usepackage[svgnames]{xcolor}
\newcommand*{\foo}[1]{\textcolor{DarkGreen}{#1}}
\AddToHook{env/quotation/begin}
{\renewcommand*{\foo}[1]{\textcolor{LightGreen}{#1}}}
\begin{document}
This is \foo{DarkGreen} but
\begin{quotation}
Here you have \foo{LightGreen} and
\begin{minipage}{5em}also \foo{LightGreen} here\end{minipage}.
\end{quotation}
Here it is again \foo{DarkGreen}.
\end{document}
with the same result.
If the change should only occur if the environment is the innermost one and therefore not inside the minipage, you could also use hooks:
\documentclass{article}
\usepackage[svgnames]{xcolor}
\newcommand*{\foo}[1]{\textcolor{DarkGreen}{#1}}
\AddToHook{cmd/begin/before}
{\renewcommand*{\foo}[1]{\textcolor{DarkGreen}{#1}}}
\AddToHook{env/quotation/begin}
{\renewcommand*{\foo}[1]{\textcolor{LightGreen}{#1}}}
\begin{document}
This is \foo{DarkGreen} but
\begin{quotation}
Here you have \foo{LightGreen} but
\begin{minipage}{5em}also \foo{DarkGreen} here\end{minipage}.
\end{quotation}
Here it is again \foo{DarkGreen}.
\end{document}
or you can define your command with a test to the current environment name:
\documentclass{article}
\usepackage[svgnames]{xcolor}
\makeatletter
\newcommand*{\foo@common}[1]{\textcolor{DarkGreen}{#1}}
\newcommand*{\fooquotation}[1]{\textcolor{LightGreen}{#1}}
\newcommand*{\foo}{%
\@ifundefined{foo\@currenvir}{\foo@common}{\@nameuse{foo\@currenvir}}
}
\makeatother
\begin{document}
This is \foo{DarkGreen} but
\begin{quotation}
Here you have \foo{LightGreen} but
\begin{minipage}{5em}also \foo{DarkGreen} here\end{minipage}.
\end{quotation}
Here it is again \foo{DarkGreen}.
\end{document}
again with the same result as the example immediate before.
Note: All these do not effect environments defined using the begin and end code of the quotation environment:
\documentclass{article}
\usepackage[svgnames]{xcolor}
\makeatletter
\newcommand*{\foo@common}[1]{\textcolor{DarkGreen}{#1}}
\newcommand*{\fooquotation}[1]{\textcolor{LightGreen}{#1}}
\newcommand*{\foo}{%
\@ifundefined{foo\@currenvir}{\foo@common}{\@nameuse{foo\@currenvir}}
}
\makeatother
\newenvironment{notquotation}{\quotation}{\endquotation}
\begin{document}
This is \foo{DarkGreen} but
\begin{notquotation}
Here you have \foo{DarkGreen},too. This is
\begin{minipage}{5em}also \foo{DarkGreen} here\end{minipage}.
\end{notquotation}
Here it is again \foo{DarkGreen}.
\end{document}
If in this case notquotation should also be effected, you would need to add an explicite change for this environment, too. This is what I would recommend. But you as an alternative you could also use the examples but use another hook, e.g., cmd/quotation/before instead of env/quotation/begin:
\documentclass{article}
\usepackage[svgnames]{xcolor}
\newif\ifinsidequotation
\newcommand*{\foo}[1]{%
\ifinsidequotation\textcolor{LightGreen}{#1}\else\textcolor{DarkGreen}{#1}\fi}
\AddToHook{cmd/quotation/before}
{\insidequotationtrue}
\newenvironment{notquotation}{\quotation}{\endquotation}
\begin{document}
This is \foo{DarkGreen} but
\begin{notquotation}
Here you have \foo{LightGreen} and
\begin{minipage}{5em}also \foo{LightGreen} here\end{minipage}.
\end{notquotation}
Here it is again \foo{DarkGreen}.
\end{document}
Note: If a command should change it's behave inside an environment you are defining, you would not need to use hooks, but just can do the redefinition inside the begin code of your definition. This is what LaTeX and classes do when defining lists. For example the description environment in article is defined as:
\newenvironment{description}
{\list{}{\labelwidth\z@ \itemindent-\leftmargin
\let\makelabel\descriptionlabel}}
{\endlist}
As you can see, \makelabel changes the definition to be same as \descriptionlabel inside a description list.
For more information about generic hooks see LaTeX news 34 and LaTeX's hook management. If you are using at least LaTeX 2021-11-15, these documents should also be available inside your TeX installation. If not they are not, you are either using an installation without documentation or a too old LaTeX.
There are two ways to interpret the requirement. I guess you're after \testB, but it's not difficult to do also the first interpretation.
\documentclass[twocolumn]{article}
\ExplSyntaxOn
\NewExpandableDocumentCommand{\ifcurrentenvironmentTF}{mmm}
{% #1 = environment name,
% #2 = code to execute if the current environment equals #1
% #3 = code to execute otherwise
\str_if_eq:eeTF { \use:c {@currenvir} } { #1 } { #2 } { #3 }
}
\ExplSyntaxOff
% this command adds braces only if the current environment is quotation
\newcommand{\testA}[1]{%
\ifcurrentenvironmentTF{quotation}
{\{#1\} used in quotation (testA)}
{[#1] used outside quotation (testA)}%
}
\newif\ifquotation
\AddToHook{env/quotation/begin}{\quotationtrue}
% this command adds braces only if called between \begin{quotation} and \end{quotation}
\newcommand{\testB}[1]{%
\ifquotation
\{#1\} used in quotation (testB)%
\else
[#1] used outside quotation (testB)%
\fi
}
\begin{document}
\section{test A}
\testA{a}
\begin{quotation}
\testA{b}
\begin{enumerate}
\item \testA{c}
\end{enumerate}
\testA{d}
\end{quotation}
\testA{e}
\newpage
\section{test B}
\testB{a}
\begin{quotation}
\testB{b}
\begin{enumerate}
\item \testB{c}
\end{enumerate}
\testB{d}
\end{quotation}
\testB{e}
\end{document}
\begin{quotation} \begin{somethingelse} \command \end{somethingelse} \end{quotation}?quotation, which you've also not shared. Guessing LaTeX, possibly standard class ...).