To find the greatest common factor (it should be divisor) of an expression, you have to take the smallest power of each letter and the smallest power of each prime number that the two expressions have in common. Here you have
[tex]8 {a}^{3} {b}^{2} = {2}^{3} {a}^{3} {b}^{2} [/tex]
and
[tex]12ab^4 = {2}^{2} \cdot 3a {b}^{4} [/tex]
You have 2, a and b in common. Then for each one, you take the smallest power. It gives
[tex] {2}^{2} a {b}^{2} = 4a {b}^{2} [/tex]
