Too often gentrification is confused with economic development. There's nothing wrong with economic development, with wealthier people moving in and new businesses starting and buildings being cleaned up.
I would like to urge in particular that there's nothing wrong with neighborhoods changing. Indeed, an urban neighborhood is an organism; if it does not change and grow, it dies. Neighborhoods are not museums. Change and development are not the problem.
The problem is what Jane Jacobs calls cataclysmic change. Put it this way: the problem with gentrification is not new things moving in, but old things, and old neighbors, being pushed out. Cataclysmic change is bad for the same reason making a neighborhood a museum is bad: because the urban neighborhood is an organism, and life means development with continuity.
Concretely, the problem is that a neighborhood full of new people is a neighborhood without culture, without institutions, and without community. For a neighborhood to succeed, new things need to be integrated into the old, so that the newcomers, while bringing change to the neighborhood, also benefit from the neighborhood's accumulated wisdom. New neighbors need to become part of the community, because a community is essentially a set of coordinated parts. It takes time for people to learn what to expect from others, so that they know whom to ask for what kind of help, whom to trust, whom to watch out for. A neighborhood needs particularity: its own feasts and fasts, festivals and memorials, institutions and traditions. Without these, there is no neighborhood.
This is morally important as a duty to the old neighbors, because neighborhoods are people's homes. If cataclysmic change destroys someone's home, it does an injustice. But it is also practically important for the new neighbors, because neighborhoods are neither safe nor interesting without these webs of relationships. And webs take time. Neighborhoods have to change gradually.
There are two causes of cataclysmic change. One is straight-out bigotry combined with government power. In my neighborhoods, for example, the newcomers, who are richer, whiter, and more powerful, don't like neighborhood traditions such as fireworks and having a beer on the front porch, so they mobilize government to make these things illegal. They don't like certain businesses that serve the poor, ranging from some kinds of restaurants to payday loan services to wig shops, so they mobilize government to evict (or "redevelop") these businesses. Note that this can only be done with government power. More on that another time. For now, suffice it to note that the tools of gentrification are a strong police force and government zoning: big government.
But the other, indirect force for gentrification is the property tax. Property tax is interesting because the income tax, and even more, consumption taxes, are based on your own valuation: the money you actually receive or spend. But the property tax is based on how other people value you: what your property is worth on the market, and thus to competitors. If your income is worth more on the market, you can choose to get that income (or choose not, and therefore not pay the higher tax). Not so with property taxes. And the assessment of others is double, because first the market affects it, then a government official makes his own determination of how the market applies to your house.
The biggest factor driving cataclysmic gentrification is rising property taxes: when development comes to your neighborhood, it makes your taxes go up, even though your income stays the same. The dynamic works the same in urban neighborhoods and in the country. According to the Wikipedia article on the property tax, farms are routinely shut down when a development comes next door, because the farm generates the same income but is subjected to higher taxes. And we watch every day in our neighborhood how rising property values push the poor -- read, the people who have always lived here, the culture and institutions of this place -- somewhere else.
But this dynamic has an indirect effect, as well. It incentivizes government-sponsored gentrification. A municipal income tax is a financial incentive for the government to bring jobs to the city. But a municipal property tax is a financial incentive for the government to bring in rich people who make their money elsewhere. It encourages cities to choose business over residents and tourism over locals. (Note that the reverse is not true: the income tax encourages cities to lure residents, but part of the lure for residents is jobs nearby, so there's no incentive to push business out.) The property tax is a perverse incentive.
What's the solution? Just replace it. Cities need revenue, but they should collect that revenue based on the actual earnings or spending of actual citizens, not based on how other people who make their money elsewhere value your home. Income or consumption tax? That's another essay, but I guess the question is whether a city thrives more by bringing in spenders (who pay the consumption tax) or people (the income tax), and then whether raising a certain tax effects the city more directly (by driving out people who pay the tax) or indirectly (by encouraging the government to bring those people in). I guess on this calculation I'd choose a sales tax, but that's another essay.
Of course gentrification of a sort will still happen. When property values in a neighborhood go up, some old residents will choose to take the money and run. But it will be their choice. Rising property values will be an economic incentive to locals instead of a threat to their way of life. That is an incentive we can live with.