Solution: We will show the centralizer of $h$ is a subgroup of $G$. We need to show that given any $a,b \in C(h)$, $ab^{-1} \in C(h)$. Note that $a\in G$ so it must have an inverse in $G$. \begin{align*} ha = ah \implies a = h^{-1} ah \implies a^{-1} = h^{-1} a^{-1} h. \end{align*} Now we have \begin{align*} h\left( ab^{-1} \right) = (ha) b^{-1} = a \left( hb^{-1} \right) = a \left( h (h^{-1}b^{-1} h ) \right) = ab^{-1} h. \end{align*} Thus, $ab^{-1} \in C(h)$ and hence it is a subgroup of $G$.

As we know that $\langle h \rangle $ is a subgroup of $G$. In order to show it is a subgroup of $C(h)$ it is enough to show that $\langle h \rangle \subseteq C(h)$. Note that for any $h \in C(h)$ and $C(h)$ is a subgroup, so for any $k \in \mathbb{Z} $, $h^k \in C(h)$.