We have

$\begin{bmatrix} 1 &1 &1 \\1 &2 &4 \\1 &4 &10 \end{bmatrix} \begin{bmatrix} x\\y \\z \end{bmatrix} = \begin{bmatrix} 1\\\lambda \\ \lambda^2 \end{bmatrix} $

By $\begin{matrix} R2 - R1\\ R3 - R1 \end{matrix} \begin{bmatrix} 1 &1 &1 \\0 &1 &3 \\0 &3 &9 \end{bmatrix} \begin{bmatrix} x\\y \\z \end{bmatrix} = \begin{bmatrix} 1\\\lambda - 1 \\ \lambda^2 -1 \end{bmatrix}$

By $R3 - 3R2 \begin{bmatrix} 1 &1 &1 \\0 &1 &3 \\0 &0 &0 \end{bmatrix} \begin{bmatrix} x\\y \\z \end{bmatrix} = \begin{bmatrix} 1\\ \lambda - 1 \\ \lambda^2 - 3\lambda + 2\end{bmatrix} \ \cdots \ \cdots (1)$

Given equation will be consistent if the rank of A=rank of [A.B.].

This requires $\lambda^2 - 3\lambda + 2 = 0$ i.e. $(\lambda - 2) (\lambda - 1 ) =0$

$\therefore \lambda = 1 \text{ or } \lambda = 2$

(i) If λ=2 the equation (1) becomes

$\begin{bmatrix} 1 &1 &1 \\0 &1 &3 \\0 &0 &0 \end{bmatrix} \begin{bmatrix} x\\y \\z\end{bmatrix} = \begin{bmatrix} 1\\1 \\0 \end{bmatrix}$

$\therefore x+ y + z = 1, \ y+3z =1$

Putting z=t, y=1-3t, we get x=2t which is the general solution.

(ii) If λ=1 the equation (1) becomes

$\begin{bmatrix} 1 &1 &1 \\0 &1 &3 \\0 &0 &0 \end{bmatrix} \begin{bmatrix} x\\y \\z\end{bmatrix} = \begin{bmatrix} 1\\0 \\0 \end{bmatrix}$

$\therefore x+ y + z = 1, \ y+3z = 0$

Puttign z=t, y=-3t, we get x=1+2t which is the general solution.