So I resanded, refilled, reprimed maybe 3-4 times over and it seems a lot smoother now. There are still a couple of nicks and dings but who wants a Brand new standard issued helmet anyway.
I wet sanded, refilled, added details, and reprimed this time in a different primer. Probably wet sand...