Dtyjlui

I am trying to proof that
$$
f : [a,b] rightarrow mathbb{C} mbox{is continuous}
longrightarrow
underset{n rightarrow infty}{lim} int_a^b f(x) e^{-inx} dx = 0
$$
where $ [a,b] $ is a compact interval.

Integrating by parts
$$
int_a^b f(x) e^{-inx} dx =
left[frac{f(x) e^{-inx}}{-in} right]_a^b - frac{1}{in} int_a^b f'(x) e^{-inx} dx
$$
and we have $ underset{n rightarrow infty}{lim} [frac{f(x) e^{-inx}}{-in}]_a^b = 0 $ because $ e^{-inx} $ is bounded.

But I dont know what to do with the expression $ int_a^b f'(x) e^{-inx} dx $, since we have not any hypothesis about continity or boundary of $ f' $.

This is the famous Riemann-Lebesgue lemma - or at least, it's a case of that lemma, with the function $f$ more restricted than it has to be. The general strategy of all the proofs I know: First, prove it for sufficiently nice $f$. Second, show that nice enough $f$ are dense in the space of functions $f$ lives in. Finally, split an arbitrary $f$ in that space into the sum of something nice and something small. Both parts go to zero, so their sum does, and we win.

This integration by parts is one of those proofs I know, and it's the one I prefer. You've got step 1 there; the theorem is proved for $fin C^1[a,b]$. The next step is to show that $C^1$ functions are dense in $C[a,b]$ under the appropriate norm. Here, we can approximate anything continuous on $[a,b]$ uniformly by $C^1$ functions. There are lots of ways to do this - Stone-Weierstrass, fitting splines, smoothing with a convolution - the fact that continuous functions on the closed interval are uniformly continuous gives us lots of flexibility. This is all very standard, of course, and can easily be looked up.

Then, for that final step, we split $f=g+h$ for an arbitrary continuous $f$, where $g$ is $C^1$ and $h$ is small. What, exactly, do we mean by "small"? Integrate
$$int_a^b e^{-inx}(g(x)+h(x)),dx = int_a^b e^{-inx}g(x),dx + int_a^b e^{-inx}h(x),dx$$
The first term there will go to zero as $ntoinfty$, so we need a bound for the second term. For that, we use that $|e^{inx}|=1$, so
$$left|int_a^b e^{-inx}h(x),dxright|le int_a^bleft|e^{-inx}h(x)right|,dx le int_a^b |h(x)|,dx$$
Oh. That's the 1-norm. What we really needed here was for the $C^1$ functions to be dense in the $L^1$ sense. Fortunately, it's a finite interval, and a uniformly convergent sequence also converges in $L^1$, or indeed any $L^p$. So then, choose $gin C^1$ so that $sup_x |f(x)-g(x)|le frac{epsilon}{2(b-a)}$, and $int_a^b |f-g|le frac{epsilon}{2}$. Then choose $N$ large enough that $left|int_a^b e^{-inx}g(x),dxright| <frac{epsilon}{2}$ for all $n>N$, and we get that $int_a^b e^{-inx}f(x),dx < epsilon$ for all $n>N$. That $epsilon$ was arbitrary, and we have the desired convergence to zero.

While the comments indicate solutions that work, here is a solution I find more insightful. First, some reductions. For ease, I will pretend $[a,b] = [0,1]$ and change $e^{-inx}$ to $e^{-2pi i nx}$. Also, by changing $f(x)$ to $f(x)-f(1)x$ (which doesn't alter continuity), we may assume $f(1) = 0$. Now, we may extend $f$ to a function on $mathbb{R}$ by $f(x) = f(x-lfloor x rfloor)$. Let $$I = int_0^1 f(x)e^{-2pi in x}dx.$$ Change $x$ to $x+frac{1}{2n}$ to get $$I = int_{frac{-1}{2n}}^{1-frac{1}{2n}} f(x+frac{1}{2n})e^{-2pi i n(x+frac{1}{2n})}dx = -int_0^1 f(x+frac{1}{2n})e^{-2pi i n x}dx.$$ Now add the two equations together to obtain $$2I = int_0^1 left[f(x)-f(x+frac{1}{2n})right]e^{-2pi i nx}dx.$$ As continuous functions on compact sets are uniformly continuous, the result follows easily.

That $int_a^b f(x)e^{-i nx}dx to 0$ as $n to infty$ is a very nice and important fact. It basically encapsulates the idea that if we want to express a continuous (or really any not too ugly) function as a combination of exponentials, we don't need too many high-phase waves. This statement is intuitive, since smooth functions don't oscillate arbitrarily quickly, and I like the proof above, since it really encodes this fact: the change of $x$ to $x+frac{1}{2n}$ is very slight for $f$ but very dramatic for $e^{-2pi i nx}$, which shows $f$ and $e^{-2pi i nx}$ can't correlate with each other well.