It's bloody lovely IMO
For the last few years (sadly no more) there's been the best ever stage race (The Topbike Boomerang) at Mt Beauty the first week of April and it's been superb.
That is the most gorgeous time of year to be up there. We never had anything other than clear blue skies and low 20 degree days. (cold in the mornings!) The trees are doing their autumn thing then too.
Yes there's lots of holiday makers, but there's almost a critical mass of cyclists these days too, so I've found the drivers are pretty bicycle aware. No complaints at all from me. Just have to book your accommodation very early.