Dtyjlui

$textbf{The question is as follows:}$

Let $H le G$ with $|G : H| = n$ and suppose that $chi in Char(G)$.

$rm (a)$ Show that $[chi ; chi] ge frac{[chi_H; chi_H]}{n}$.

$rm (b)$ Show that, if $H$ is Abelian and $chi in Irr(G)$ then $chi(1) le n$.

$rm (c)$ Show that, if the equality holds in part (b), then $H vartriangleleft G$.

I can show the first part as follows:

$rm (a)$ We have
$$|H| [chi_H; chi_H] = sum_{h in H}^{} |chi(h)|^2 le sum_{g in G}^{}|chi(g)|^2 = |G|[chi, chi] $$

$rm (b)$ For second part I can write

$$chi(1)=chi|_H(1)le [chi|_H,chi|_G]le [G:H][chi,chi]=[G:H]$$
But I am not sure if it is correct?

Can someone correct me please?

$rm (c)$ For the third part I still have no idea so far!

Can someone help me to show this, please?

Thanks!

Any reference here is to the book of Marty Isaacs, Character Theory of Finite Groups. Now for (b) you must be a bit more precise: if $H$ is abelian, and $chi$ is an irreducible character of $G$, then $chi_H=sum_{lambda in Irr(H)}a_{lambda}lambda$, where the $lambda$'s are linear and the integers $a_lambda geq 0$. Hence, $[chi_H,chi_H]=sum_{lambda in Irr(H)}a_{lambda}^2 geq sum_{lambda in Irr(H)}a_{lambda}=chi(1).$ By (a) and the fact that $chi$ is irreducible so $[chi,chi]=1$, you get indeed $chi(1) leq [chi_H,chi_H] leq |G:H|. text{ } (*)$

For (c) we are going to use the Lemma (2.29) of Isaacs' book, which says that if you have a $chi in Irr(G)$ and a subgroup $K$ then $[chi_K,chi_K]=|G:K|$ if and only if $chi$ vanishes outside $K$.

Now assume $H$ is abelian, $chi in Irr(G)$ with $chi(1)=|G:H|$. By $(*)$ we must have that $chi$ vanishes off $H$. Look at the set ${g in G: chi(g) neq 0 }$. This set is normal (since $chi(g)=chi(g^x)$ for any $x in G$) and is contained in $H$. Hence $N=langle g in G: chi(g) neq 0 rangle$ is an abelian normal subgroup of $G$ contained in $H$. Observe that if $h in H-N$, then we must have $chi(h)=0$. So $chi$ vanishes even off $N$, and by the Lemma (2.29) in the other direction and applying (b) to the subgroup $N$, we must have $|G:N|=[chi_N,chi_N]leq chi(1)=|G:H|$. Hence $|H| leq |N|$ and we must have $H=N$ and you are done.

Note (February 6th) Returning to my answer above I discovered a flaw in the argument: it is in the last equation, after “we must have” - specifically $[chi_N,chi_N] leq chi(1)$ is not correct. After some thoughts, I suspected that $(c)$ above is not true at all. As a matter of fact Prof. Derek Holt was very kind to provide me with a counterexample: there is a group of order $32$, SmallGroup(32,6) (notation in Magma or in GAP). It has an abelian subgroup $H cong C_2 times C_4$, hence $|G:H|=4$. This subgroup is not normal and $G$ has a $chi in Irr(G)$ with $chi(1)=4$. In this counterexample $|N|=2$.

My fault analysis led me to the following.

Theorem A Let $H leq G$ be abelian, $chi in Irr(G)$ with $|G:H|=chi(1)$ a prime number, then $H unlhd G$ and $chi$ is monomial (induced by a linear character).

Proof Since $H$ is abelian, $chi(1) leq [chi_H,chi_H] leq |G:H|$ and the last inequality is equality if and only if $chi$ vanishes outside $H$. What about the first inequality? Well, $chi_H=sum_{lambda in Irr(H)}a_lambda lambda$, where the $lambda$'s are all linear and the integers $a_lambda geq 0$. $chi(1)=sum_{lambda in Irr(H)}a_lambda$ and $[chi_H,chi_H]=sum_{lambda in Irr(H)}a^2_lambda$. So we see that $chi(1)=[chi_H,chi_H]$ if and only if $a_lambda in {0,1}$. In that case, $chi_H=lambda_1 + cdots + lambda _{|G:H|}$. Observe that for these $lambda$'s by Frobenius Reciprocity, $[chi_H,lambda_i]=[chi,lambda_i^G]=1$ and it follows that $chi=lambda_i^G$ for all $i=1, cdots ,|G:H|$. Hence, $chi$ is a monomial character.

Next we concentrate on the normal subgroup $N=langle g in G: chi(g) neq 0 rangle$ as defined above, and we apply Clifford Theory to this:
$chi_N=esum_{i=1}^tmu_i$, where $t=|G:I_G(mu)|$, the index of the inertia subgroup of the (linear) $mu_1=mu$ and $e$ is a positive integer. We already remarked above that $chi$ vanishes outside $N$, hence $[chi_N,chi_N]=e^2t=|G:N|$. And also, $chi(1)=et=|G:H|$. It follows that $|H:N|=e$.

Now if $|G:H|=p$, $p$ a prime, then $p=et$, hence we have two cases: $e=1$ and $t=p$, or $e=p$ and $t=1$. In the first case we get $|H:N|=1$, that is $H=N$ and hence $H$ is normal. The latter case yields $|G:N|=p^2$, so $G/N$ is a group of order $p^2$ whence abelian, and again $H$ must be normal.$square$

Note (February 12th) There is a generalization of the above theorem, that I can prove now. It also follows from Udo Riese's results.

Theorem B Let $H leq G$ be an abelian maximal subgroup, $chi in Irr(G)$ with $|G:H|=chi(1)$, then $H unlhd G$ and $chi$ is monomial (induced by a linear character).

Proof The fact that $chi$ is monomial runs exactly in the same way as in the proof of Theorem A.
Now I use the 2.11 Theorem from Marty Isaacs' book Finite Group Theory. The proof of this Theorem is based on the famous Zipper Lemma of Helmut Wielandt. The subgroup $H$ satisfies the conditions of this Theorem (caution: in the aforementioned book our $H$ here is called $A$), namely $H$ and $G$ are the only two subgroups between $H$ and $G$ by the maximality of $H$, and we need to check that $|G:H|^2 leq |G:Z(G)|$. But this trivially true since $|G:H|^2=chi(1)^2 leq |G:Z(G)|$ (for the last inequality, this is easy to prove or see (2.30) Lemma in Character Theory of Finite Groups above). By the 2.11 Theorem, it follows $H subseteq F(G)$, the Fitting subgroup of $G$. Hence either $H=F(G)$, and then $H$ is normal, even characteristic. Or, $G=F(G)$, meaning $G$ is nilpotent and it is well-known that maximal subgroups of nilpotent groups are normal. $square$

Final remark A famous theorem of I.N. Herstein (1957) asserts that if $G$ has an abelian maximal subgroup, $G$ must be solvable, see here.