Through the second partial derivative test, I first confirmed that $(2,3)$ is a local maximum.
Starting from
$$f(x,y)=x^2y^3(6-x-y)$$
the partial derivatives are
$$f_x=x y^3 (-3 x - 2 y + 12) \\ f_y=-x^2 y^2 (3 x + 4 y - 18)\\
f_{xx}=-2 y^3 (3 x + y - 6) \\f_{yy}= -6 x^2 y (x + 2 y - 6) \\ f_{xy}=x y^2 (-9 x - 8 y + 36)
$$
If we set $f_x=0$ then
$$x y^3 (-3 x - 2 y + 12)=0\implies y = 6-\frac{3x}{2} ~\text{ or }~ x=y=0$$
where
$$y=6-\frac{3x}{2}\tag{1}$$
$$x=y=0\tag{2}$$
If we set $f_y=0$ then
$$-x^2 y^2 (3 x + 4 y - 18)=0\implies y = \frac{9}{2}-\frac{3x}{4} ~\text{ or }~ x=y=0$$
where
$$y = \frac{9}{2}-\frac{3x}{4}\tag{3}$$
So, equating $(1)$ and $(3)$ forms
$$6-\frac{3x}{2}=\frac{9}{2}-\frac{3x}{4} \implies x=2$$
which from $(1)$ gives us the critical points $(0,0)$ and $(2,3)$
We now need to apply the second partial derivative test. The test tells us that provided $f(x, y)$ is a differentiable real function of two variables whose second partial derivatives exist we can compute
$$D(x,y)=f_{xx}(x,y)f_{yy}(x,y)-\Big(f_{xy}(x,y)\Big)^2$$
and that
- If $D(a,b)>0$ and $f_{xx}(a,b)>0$ then $(a,b)$ is a local minimum of $f$.
- If $D(a,b)>0$ and $f_{xx}(a,b)<0$ then $(a,b)$ is a local maximum of $f$.
- If $D(a,b)<0$ then $(a,b)$ is a saddle point of $f$.
- If $D(a,b)=0$ then the second derivative test is inconclusive, and the point $(a, b)$ could be any of a minimum, maximum or saddle point.
We found before that
$$f_{xx}=-2 y^3 (3 x + y - 6) \\f_{yy}= -6 x^2 y (x + 2 y - 6) \\ f_{xy}=x y^2 (-9 x - 8 y + 36)$$
Thus, if we compute $D(x,y)$ for our four critical points we have
$$D\big(0,0\big)=(0)(0)-(0)^2=0$$
$$D\big(2,3)=(-162)(-144)-(-108)^2=11664 > 0$$
Therefore, by the second partial derivative test,
- $(0,0)$ is inconclusive.
- $(2,3)$ is a local maximum since $D\big(2,3\big) > 0$ and $f_{xx}(2,3)=-162<0$.
Substituting $(2,3)$ back into $f(x,y)$ forms the local maximum of $f(2,3)=108$. This cannot be a global maximum as $f(-2,3)=540$. Also, $f(0,0)=0$ and $f(-1,-1)=-8$ so $(0,0)$ cannot be a global minimum.
To find the global extremum, I evaluated the limit as $x$ and $y$ approach plus or minus infinity. As $f(x,y)$ isn't bounded by a rectangular region in $\mathbb R^2$, I need to check what happens at the four end points.
$$\lim_{x\to-\infty}f(x,y)=\lim_{x\to-\infty}x^2y^3(6-x-y)=y^3(\infty)$$
$$\lim_{x\to\infty}f(x,y)=\lim_{x\to\infty}x^2y^3(6-x-y)=y^3(-\infty)$$
$$\lim_{y\to-\infty}f(x,y)=\lim_{y\to-\infty}\frac{e^{-y}}{\frac{1}{y}}=x^2(-\infty)$$
$$\lim_{y\to\infty}f(x,y)=\lim_{y\to\infty}\frac{e^{-y}}{\frac{1}{y}}=x^2(-\infty)$$
So, it is clear that our critical points won't be larger or smaller than these and therefore there is no global extremum.