Al-jawab billahi at-taufeeq (the answer with Allah's guidance)
Nail Polish and nail varnish is like paint which forms a barrier on the nails so that water can not get to them, whereas wetting the nails is necessary in wudu and bathing.
Henna, on the hand, does not form a coating but changes the pigment of the skin and nails, hence, does not stop the water from reaching the nails or skin.
In terms of an alternative, the most preferred is Henna, since it has been recommended by the Prophet (Sallallahu Alaihi Wa Sallam).