Solution :

We have,

$$\quad \vec{F}=(x^{2}+xy^2) \hat{i}+(y^2+x^2y)\hat{j} $$

$\begin{aligned} \operatorname{Curl} \vec{F} &=\nabla \times \vec{F}\\ &=\left(\hat{i} \frac{\partial}{\partial x}+\hat{j} \frac{\partial}{\partial y}+\hat{k} \frac{\partial}{\partial z}\right) \times\left(y^{2} \hat{i}+2 x y \hat{j}-z^{2} \hat{k}\right) \end {aligned} $

$ \space\space\space\space\space\space\space\space\space\space =\left|\begin{array}{ccc} \hat{i} & \hat{j} & \hat{k} \\ \frac{\partial}{\partial x} & \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\ x^{2}+xy^2 & y^2+x^2y & 0 \end{array}\right|\\ \space\space\space\space\space\space\space\space =\hat{i}(0)-\hat{j}(0) +\hat{k}(2x y-2 xy)\\ \space\space\space\space\space\space\space\space =0 $

Hence, $\vec{F}$ is irrotational.

To find the scalar potential function $\phi$.

$ \begin{aligned} \vec{F} &=\nabla \phi \\ d \phi &=\frac{\partial \phi}{\partial x} d x+\frac{\partial \phi }{\partial y} d y+\frac{\partial \phi }{\partial z} d z \\ \quad &=\left(\hat{i} \frac{\partial \phi }{\partial x}+\hat{j} \frac{\partial \phi}{\partial y}+\hat{k} \frac{\partial \phi }{\partial z}\right)(\hat{i} d x+\hat{j} d y+\hat{k} d z) \\ &=\left(\hat{i} \frac{\partial}{\partial x}+\hat{j} \frac{\partial}{\partial y}+\hat{k} \frac{\partial}{\partial z}\right) \phi \cdot d r \\ &=\nabla \phi \cdot d \vec{r} \\ &=\vec{F} \cdot d r \\ &=\left[(x^{2}+xy^2) \hat{i}+(y^2+x^2y)\hat{j} \right] \cdot(\hat{i} d x+\hat{j} d y+\hat{k} d z) \\ &=(x^{2}+xy^2)d x+ (y^2+x^2y)d y=x^2dx + y^2dy +(xdx)y^2 + (x^2)(ydy) \\ \end{aligned} $

$\begin {aligned} \boldsymbol \phi &=\int [(x^{2}+xy^2)d x+ (y^2+x^2y)d y] \\ &=\int [x^2dx + y^2dy +(xdx)y^2+(x^2)(ydy)] \\ &=\int x^2dx +\int y^2dy +\int (xdx)y^2 + \int (x^2)(ydy) \\ &=\frac{x^3}{3} +\frac{y^3}{3} + \frac{x^2y^2}{2} \end{aligned} $

$$ \therefore \phi =\frac{x^3}{3} +\frac{y^3}{3} + \frac{x^2y^2}{2} $$