In New Orleans, there is a huge public debate about short-term rentals. At the moment there is a partial hiatus on allowing such rentals pending review of some new regulations. Our problem is absentee landlords who do the bare minimum required to bring something up to code and then because they don't live in the area, they don't care to whom they rent. The renters for New Orleans are often party animals who get loud and rowdy, which makes the rest of the neighborhood unpleasant for the "real" residents. It also becomes impossible to decide if someone is "out of place" in the neighborhood, and despite the "profiling" sound of that, it is still the only useful way to watch out for trouble.
The counter argument is that someone might have bought the property for the purpose of rental income because that is the only way they can get by these days. For people with low or fixed income, there is a question of depriving them of the rights to enjoy their property, on which they have paid the appropriate taxes.