Dtyjlui

If $x,y$ have the same sign $f(x,y)>0$

and if they have opposite signs $f(x,y)<0$

With the 4 pairs you have, 2 share the same sign and and 2 do not. That should make it clear wich are the maxima and which are the minima.

But is calculus even necessary?

Make this substitution

$x = u+v\
y = u-v$

Then your objective and constraint become:

$f(u,v) = 3u^2 - 3v^2\
3u^2 + v^2 = 3$

Plugging the constraint into the objective.

$f(u,v) = 3 - 4v^2$

$f$ is maximized when $v = 0$

$u = pm 1\
x = pm 1\
y = pm 1$

$f$ is minimized when $u = 0$

$v = pm sqrt 3\
x = pm sqrt 3\
y = mp sqrt 3$

Let $3xy=k$.

Thus,
$$k(x^2+y^2+xy)=9xy$$ or
$$kx^2+(k-9)xy+ky^2=0$$ has solutions.

Let $kneq0$.

Thus, $$(k-9)^2-4k^2geq0$$ or
$$(k+9)(k-3)leq0$$ or
$$-9leq kleq3.$$
Now, we see that $k=0$ is not relevant and the extreme value of $k$ occurs for $x=-frac{k-9}{2k}y$,

which says that
$$max_{x^2+y^2+xy=3}3xy=3$$ and
$$min_{x^2+y^2+xy=3}3xy=-9$$

If Lagrange multipliers is not mandatory,

$$12=4x^2+4y^2+4xy=(2x+y)^2+(sqrt3y)^2$$

Let $sqrt3y=sqrt{12}cos tiff y=2cos t$

$2x+y=2sqrt3sin tiff 2x=2sqrt3sin t-2cos tiff x=sqrt3sin t-cos t=2sinleft(t-30^circright)$

$3xy=12cos tsinleft(t-30^circright)=6[sin(2t-30^circ)-sin30^circ]$

Now for real $x,y;t$ is real $$-1lesin(2t-30^circ)le1$$

Evaluate your function $f(x,y) = 3xy$ at each of these four points, $(0,0), (1,1), (-1,-1), (sqrt{3},-sqrt{3})$ and compare the results.You will find maximum is achieved at $(1,1)$ and $(-1,-1)$ and minimum is achieved at $ pm (sqrt 3, -sqrt 3)$

As the equations involved are homogeneous, making $y = lambda x$ we have

$$
min(max)3x^2lambda mbox{s. t. } x^2(1+lambda^2+lambda) = 3
$$

which is equivalent to

$$
min_{lambda}(max_{lambda})f(lambda) = frac{9lambda}{lambda^2+lambda+1}
$$

so the stationary points are at

$$
f'(lambda) = -frac{9 left(lambda ^2-1right)}{left(lambda ^2+lambda +1right)^2} = 0Rightarrow lambda^* = {-1,1}
$$

then the solutions are at $y = pm x$ etc.

1) $(x+y)^2- xy= 3;$

$xy = (x+y)^2 -3;$

$f(x,y)=3xy = 3(x+y)^2-9.$

$f_{min}=-9$, at $x=-y$;

$-3x^2= -9; x=pm √3; y=mp √3$;

2) $(x-y)^2+3xy= 3;$

$f(x,y)= 3xy= 3-(x-y)^2$;

$f_{max}= 3$, at $x=y$.

$x^2=1$; $x =pm 1$, $y=pm 1$.

You can determine the "type" of the critical point using the partial derivatives. Suppose $(x,y)$ is a critical point that you are considering. Then,

1. If $f_{xx} f_{yy}- (f_{xx})^2 > 0$ and $f_{xx} > 0$ when evaluated at $(x,y)$, then there is a relative minimum at $(x,y)$.


2. If $f_{xx} f_{yy}- (f_{xx})^2 > 0$ and $f_{xx} < 0$, you have a relative minimum.


3. If $f_{xx} f_{yy}- (f_{xx})^2 < 0$ then you've got a saddle point.


4. If $f_{xx} f_{yy}- (f_{xx})^2 = 0$ its indeterminate.

Edit: Just looked at the function you've provided. Clearly, it's going to indeterminate since the second derivative for $x$ or $y$ is $0$. You'll have to just plug in the values of the critical points you've found and determine it based on that.