Dtyjlui

Find all integer roots of: $x^2(y-1)+y^2(x-1)=1$

Obviously $(2,1)$ and $(1,2)$ are two answers. But I was unable to manipulate the equation algebraically giving a useful form for finding all other possible answers!

I also tried to view it as a quadratic equation in terms of $x$, but forming the Delta of that equation didn't help much other than if $y=1$ then certainly Delta would be a square and equation would have answer for $x$...

Thanks to Piquito for pointing out the negative solution.

As stated in the comments, yours are the only positive integer solutions because you'd get $LHS>1$ with larger $x,y.$ Bounding the discriminant $Delta_x$ (resp. $Delta_y$) with squares of polynomials of $x$ (resp. $y$) is a natural and, in this case, not a bad idea, in order to show there are no negative solutions.

WLOG let us consider the problem as a quadratic equation in $y$. We have $Delta_x=x^4+4x^3-4x^2+4x-4,$ and since $(x^2+2x)^2=x^4+4x^3+4x^2$ we may want to try finding the smallest $m,ninmathbb{Z^+}$ such that for some $a$ and all $x le a$ $$(x^2+2x-m)^2<Delta_x<(x^2+2x-n)^2.$$ It turns out $(m,n)=(8,4)$ works with $a=-6$, and no integer $x$ fulfills $$Delta_{x}=(x^2+2x-k)^2 $$ for any $k=5,6,7.$ Therefore we only need to check $x=-1,-2,-3,-4,-5$ back in the original equation to see if there are other solutions besides yours: and there are, namely $(x,y)=(-5,2),(2,-5).$

Any point on the curve $x^2(y-1)+y^2(x-1)=1$ is either close to the line $x=1$, to the line $y=1$ or to the line $x+y=-2$, since:

$$x^2(y-1)+y^2(x-1)-1 = (x-1)(y-1)(x+y+2)+(x+y-3)$$

$hspace{1.5in}$

These lines are asymptotes and the algebraic curve intersects them only at the solutions you already found. With a crude bound, we just have to test every point in the range $[-6,6]^2$ to be sure there are no other integer solutions.

It is easy to verify that
$$x^2(y-1)+y^2(x-1)=1iff2(xy)^3+(x^2+y^2)(xy)^2=(x^2+y^2+1)^2$$
It follows that $xy$ divides $x^2+y^2+1$, in other words one has
$$frac xy+frac yx+frac{1}{xy}in mathbb Z$$
The obvious solution $x=y=1$ is not solution of the proposed equation. On the other hand the number $3$ given by this $(x,y)=(1,1)$ is given also by $(x,y)=(1,2),(-5,2)$ (as expected, small multiples of $2$ and $5$ are "candidates"). It is not hard to verify that these two (and the corresponding symmetrics respect to the line $y=x$, of course) no other solution exists.

Since O.P.'s equation is symmetric respect to the line $y=x$ then the only solutions are $$(x,y)=(1,2),(2,1),(-5,2),(2,-5)$$

To pull a rabbit from a hat, let's write $y=u-2-x$. When the algebraic dust settles, the equation to solve is now

$$ux^2-u(u-2)x+(u^2-4u+5)=0$$

We see we must have $umid 5$, which leaves four possible values for $u$: $u=pm1$ and $u=pm5$. The values $u=1$ and $u=-5$ lead to quadratics $x^2+x+2=0$ and $x^2+7x-10=0$, which have no integer solutions, while $u=-1$ leads to

$$x^2+3x-10=(x+5)(x-2)=0$$

which gives $(x,y)=(-5,2)$ and $(2,-5)$. Similarly, $u=5$ leads to

$$x^2-3x+2=(x-1)(x-2)=0$$

which gives $(x,y)=(1,2)$ and $(2,1)$.

Remark: The "rabbit" here is the "$-2$" in "$u-2-x$." That is, it made sense initially to let $y=v-x$ and see what the quadratic in $x$ looked like; it turned out it implied $(v+2)mid5$. So at that point it made sense to let $v=u-2$ and redo everything with $y=u-2-x$.

Perhaps it would be easier if you write the equation like this

$$ xy(x+y)-(x+y)^2+2xy = 1$$ and now put $a=x+y$ and $b=xy$. Then you get: $$ ba-a^2+2b=1implies b ={a^2+1over a+2}$$

Write $c=a+2$ and then $$b = {(c-2)^2+1over c} = {c^2-4c+5over c}= c-4 +{5over c}$$

so $cmid 5implies a+2in {-5,-1,1,5} implies ain {-7,-3,-1,3}$ and respectively $bin {-10,-10,2,2}$. This should not be difficult to finsih.