Dtyjlui

When trying to identify a limit cycle of a 2nd order nonlinear DE
$$begin{cases}dot{y}_1=y_2\dot{y}_2=-ky_2-frac{b(a)sin(y_1)}{1+c_1(a)cos(y_1)+c_2(a)sin(y_1)}end{cases},$$
where $b(a)$, $c_1(a)$, and $c_2(a)$ are some coefficients depending on the parameter $a$,
I arrived at the following 1st order PDE:
$$frac{partial V}{partial y_1}y_2-frac{partial V}{partial y_2}ky_2-frac{partial V}{partial y_2}f(y_1)=0.$$
I wish to find a solution to this equation in the form $V(y_1,y_2)=0$ such that the following conditions are satisfied:

1. 
   $V(y_1+2pi,y_2)=V(y_1,y_2)$;


2. 
   $frac{dy_2}{dy_1}bigg|_{y_1=0}=0$ and $frac{d^2y_2}{dy_1^2}bigg|_{y_1=0}>0$;


3. 
   $frac{dy_2}{dy_1}bigg|_{y_1=pi}=0$ and $frac{d^2y_2}{dy_1^2}bigg|_{y_1=pi}<0$.

Item 1 comes from the fact that the DE is defined on a cylinder, while Items 2 and 3 say that the plot $y_2=y_2(y_1)$ of $V(y_1,y_2)=0$ has a minimum at $y_1=0$ and a maximum at $y_1=pi$. The latter conditions were obtained from numerical simulations.

Furthermore, we know that $f(y_1+2pi)=f(y_1)$ and $f(0)=0$.

Question: I wonder whether this problem is well defined and if "yes", how to solve it?

Indeed, I want to find the limit set $V(y_1,y_2)=0$ since the problem -- as it is stated -- can have only one solution that corresponds to the limit cycle.

My attempts. Let $y_1=0$. If $frac{partial V}{partial y_2}bigg|_{y_1=0}neq 0$, Item 2 along with $f(0)=0$ implies that $y_2=0$ which contradicts the fact that the point $(0,0)$ is an isolated equilibrium. So, I conclude that $$frac{partial V}{partial y_2}bigg|_{y_1=0}= 0 mbox{ and } frac{partial V}{partial y_1}bigg|_{y_1=0}= 0.$$
Following the same logic I conclude that the same holds for $y_1=pi$:
$$frac{partial V}{partial y_2}bigg|_{y_1=pi}= 0 mbox{ and } frac{partial V}{partial y_1}bigg|_{y_1=pi}= 0.$$

So, basically, that's it. I tried several candidates for $V(y_1,y_2)$, but couldn't get any meaningful result.

This plot makes me to believe that there is a limit cycle. The red trajectory (approximately) corresponds to the unstable solution of the sadle, two others are just computed for different initial values of $y_2$.

Let me summarize here some ideas from comments and one weird finding of mine. This is not a complete answer, but I think I have found some explanation for what's happening.

As @Dmitry told, numerics show that system has a rotational limit cycle. Standard ways to prove existence of limit cycle for some parameter values include understanding what bifurcation could give rise to this limit cycle. To get a grasp on which bifurcation exactly happens it is reasonable to vary parameters and see what happens. Changing $k$ is a good idea: when $k=0$ the system is particularly simple and some analytics could be done. Naturally I was trying to understand what happens with limit cycle when you decrease $k$ to $0$. This is important because when $k = 0$ no such limit cycle is possible, I'll explain why further. The weird finding is that the more I decrease $k$ to $0$, the higher and higher this rotational limit cycle goes. That's weird and slightly suggests that bifurcation happens at infinity when $k = 0$. I am not good at dealing with bifurcations at infinity, for me it looks like some sort of weird "Andronov-Hopf at infinity", but I've never met myself such bifurcation before.

I think the picture is a bit more tractable when you start increasing $k$: at some value close to $k = 0.215$ it seems that rotational limit cycle collides with a heteroclinic trajectory connecting two saddles. It would be a good idea to figure out how stable separatrices of saddles behave before and after this bifurcation: they are always part of boundaries between different attraction basins and can help figure out multistability. You know about stable focus in this system (which is unique on the cylinder), thus the presence of another attractor might mean that a limit cycle exists.

Here I'll try to explain why rotational closed trajectories exist when $k approx 0$.
I'll start with "unperturbed" version of OP's equations, i.e. when $k = 0$. In that case system takes a form $dot{x} = y, , dot{y} = -f(x)$.
If you consider an equation $$ddot{x} + f(x) = 0,$$ where $f(x)$ is at least continuous, it is well know that all such equations are integrable: the first integral is simply
$frac{dot{x}^2}{2} + F(x)$, where $F(x)$ is such that $F'(x) equiv f(x)$. If $f(x)$ is periodic, then we can consider an equivalent system
$$ dot{x} = y, $$
$$ dot{y} = -f(x),$$
which is naturally a system on a cylinder. As a system on plane it also has a first integral which is $frac{y^2}{2} + F(x)$. Note that $F(x)$ might be not periodic and hence the system on cylinder wouldn't have a first integral. Using the system on the plane and the first integral we can compute Poincaré map for $x = 0$ and $x = 2 pi$ as cross-sections. Namely,
$$ frac{lbrack y(0) rbrack^2}{2} + F(0) = frac{lbrack overline{y} rbrack^2}{2} + F(2pi),$$
or
$$ overline{y} = sqrt{y^2 - 2(F(2pi)-F(0))}$$
if we start with $y > 0$.

For the parameter values that @Dmitry mentioned in comments $F(2pi) - F(0)$ is negative and doesn't depend on $k$. So I'll just write map as $overline{y} = sqrt{y^2 + alpha}$, where $alpha > 0$. Any rotational closed trajectory would correspond to a fixed point of this Poincaré map, i.e. to a solution of $y = sqrt{y^2 + alpha}$. It is quite obvious that there is no solution to this equation because $sqrt{y^2 + alpha} > sqrt{y^2} = y$. However what happens, if we perturb this mapping a bit? For example, does $beta + sqrt{y^2 + alpha} = y$ has solutions for $beta approx 0$? The answer is "no" when $beta > 0$, but when $beta < 0$ the answer is "yes". The function $beta + sqrt{y^2 + alpha}$ has line $y + beta$ as an asymptote, thus $(beta + sqrt{y^2 + alpha}) - (beta + y)$ tends to $0$ as $y rightarrow +infty$. From this follows that $beta+sqrt{y^2 + alpha} - y rightarrow beta$, thus it is negative at some values of $y$, but positive when $y = 0$ it is positive. An existence of fixed point follows from continuity. Note that this fixed point ceases to exist when $beta = 0$, but it exists for small $beta < 0$: smaller $beta$ corresponds to bigger coordinate of fixed point, "it comes from infinity".

My idea is that although the real perturbation of Poincaré map would be much different than my model example, something quite similar happens when $k$ is non-zero in your system. Probably it could be made more rigorous, but I didn't delve much into it.

This is an illustration of level sets for $frac{y^2}{2} + F(x)$. Black level sets contain saddle equilibria. $F(x)$ was evaluated numerically using quad function from SciPy.

Note that the point at which level set intersects $x = 0$ is lower than the point of intersection with $x = 2pi$. It happens due to aperiodicity of $F(x)$; in particular, $F(2pi) - F(0) < 0$.

There is no limit cycle. For $k>0$ the system either moves continuously in one direction towards infinity or spirals down toward a stationary point.

Your system can be put back together as a second order scalar equation of the type
$$
ddot y+kdot y +P'(y)=0
$$
for some potential function $P$. That is, loosely speaking, the system describes a mechanical motion of some object in the landscape of height $P(y)$ with a linear friction with friction coefficient $k$. The intuitive behavior is that this system will continuously lose energy to friction. If it gets trapped in a valley of the potential function it will oscillate inside that valley with continuously descending maxima until it settles at one of the local minima of $P$. Or the initial energy level, that is, the initial velocity, is so high that the object never settles and eventually moves towards infinity.

To derive that result formally, multiply the equation with $dot y$ and integrate to find that the energy function
$$
E(t)=frac12dot y(t)^2+P(y(t))=E(0)-kint_0^tdot y^2(s),ds
$$
is constantly declining (as long as it remains in motion, which is indefinitely) towards one of the points with minimal value of $P(y)$ and $dot y=0$.