Answer:
Option A
Explanation:
Key Idea If any vector x is coplanar with the vector y and z then x=λy +μz Here, u is coplanar with a and b.
$\therefore$ u= λa+µb
Dot product with a, we get
u.a= λ (a.a)+µ (b.a)
$\Rightarrow$ 0= 14λ+2 µ ........(i)
$[\therefore a= 2\hat{i}+3\hat{j}-\hat{k},b=\hat{j}+\hat{k},u.a=0]$
Dot product with b, we get
$u.b=\lambda (a.b)+\mu (b.b)$
$24=2\lambda+2\mu$ .....(ii) $[\because u.b=24]$
Solving Eqs. (i) and(ii), we get
λ =-2, μ = 14
Dot product with u, we get
$\mid u\mid^{2}=\lambda (u.a)+\mu (u.b)$
$\mid u\mid^{2}=-2 (0)+14 (24)$
$\Rightarrow$ $\mid u\mid^{2}=-336$