Answer:
Option A
Explanation:
The general term in the expansion of (ax2+1bx)13
T(p+1)=13Cp(ax2)13−p1(bx)p
= 13Cpa13−pbpx26−3p
For, x5, p=7
So, coefficient of x5 in the expansion of
(ax2+1bx)13 is 13C7a6b7 [∵ p=7]
Similarly, the general term in the expansion of (ax−1bx2)13
Tq+1=13Cq(ax)13−q(−1bx2)q
=13Cq(−1)qa13−qbqx13−q
for x−5,q=6
So, coefficient of x−5 in the expansion of (ax−1bx2)13 is
13C6a7b6
According to the question,
13C7a6b7=13C6a7b6
⇒ ab=1