I would think of it without trigonometry and calculus: If the triangle has maximum area, then we can't make the area larger by moving a single corner.
Pick one side of the triangle as base (rotate the circle so that the base is horizontal for clearer view). Then the largest area you can get is by putting the point opposite the base as far away from from the base as possible. This happens to be halfway along the longest arc between the end points of the base. In other words, the opposite point must be exactly in the middle of the two end points of the base for the area to be maximal.
Now, if the triangle is not equilateral, there is at least one point which is not in the middle of the two others, so we can increase the area of the triangle. An equilateral triangle is the only one which cannot be increased this way, so it must be maximal.
To find the area of an equilateral triangle inscribed in a circle, the height of the triangle is $\frac32r$. This is because in an equilateral triangle, heights, medians and side bisectors coincide, and we know that
- The center of the circumcircle of a triangle is where the side bisetors intersect
- The medians partition one another in segmenst of ratio $2:1$
- The $2$ part of the medians is the radius in the circle, so its length is $r$
Now you have an equilateral triangle where you know the height is $\frac32r$, so you can work out that the side length is $\sqrt3r$, which gives the triangle an area of $\frac{3\sqrt3}2r^2$.