Problem: A matrix $A\in M(n,\mathbb{R} )$ is said to be positive definite matrix if $=\left\langle A\mathbf{x}, \mathbf{x} \right\rangle = \mathbf{x} ^t A \mathbf{x} \gt 0$ for all non-zero vector $\mathbf{x} $. If $A$ and $B$ are positive definite matrices, then is it true that $AB$ is positive definite matrix?
Solution: This is not true in general. For example, take \[ A = \begin{pmatrix} 2 & 3 \\ 0 & 2 \\ \end{pmatrix} = B. \] Note that the matrix $A$ is positive definite as for every $\mathbf{x} \neq 0$ we have, \begin{align*} \left\langle A \mathbf{x} ,\mathbf{x} \right\rangle & = \mathbf{x} ^t A \mathbf{x} \\ & = \begin{pmatrix} x_1 & x_{2} \\ \end{pmatrix} \begin{pmatrix} 2 & 3 \\ 0 & 2 \\ \end{pmatrix} \begin{pmatrix} x_{1} \\ x_{2} \\ \end{pmatrix} \\ & = \begin{pmatrix} x_{1} & x_{2} \\ \end{pmatrix} \begin{pmatrix} 2x_{1} + 3x_{2} \\ 2x_{2} \\ \end{pmatrix} \\ & = 2x_{1} ^2 + 3x_{1} x_{2} + 2x_{2} ^2 \\ & = \frac{1}{2}\left( x_{1} ^2 + x_{2} ^2 \right) + \frac{3}{2} \left( x_{1} +x_{2} \right) ^2 \gt 0. \end{align*} So $A$ and $B$ both are positive definite matrices.
But, $AB=A^2$ is not positive definite. Take $\mathbf{x} =\begin{pmatrix} 1 \\ -1 \\ \end{pmatrix}$. Then, \begin{align*} \mathbf{x}^t A^2 \mathbf{x} & = \begin{pmatrix} 1 & -1 \\ \end{pmatrix} \begin{pmatrix} 2 & 3 \\ 0 & 2 \\ \end{pmatrix}^2 \begin{pmatrix} 1 \\ -1 \\ \end{pmatrix} \\ & = \begin{pmatrix} 1 & -1 \\ \end{pmatrix} \begin{pmatrix} 4 & 12 \\ 0 & 4 \\ \end{pmatrix} \begin{pmatrix} 1 \\ -1 \\ \end{pmatrix} \\ & = \begin{pmatrix} 1 & -1 \\ \end{pmatrix} \begin{pmatrix} -8 \\ -4 \\ \end{pmatrix} \\ & = -4 \lt 0. \end{align*} Therefore, in general the product of positive definite matrices need not be positive definite.