How do models in general handle leap days and years with 53 weeks? Shall we remove these dates for the 'ds' column before modeling? If we remove let's say week 53, will this cause any issue for the algorithms since 'ds' column is not continuous anymore?

This in an interesting issue. The models do not actually use the specific dates, they are only used for sorting the dataset. I dont think you should remove them.