During the birth of your town, there will only be 4 animal residents--the original 4, you could call them. As the days go by, more residents will stake their plots in town, until you have exactly 9 villagers.
Once you've reached this point, the game won't give you any more random residents, but you can acquire one more villager, making for a maximum of 10 villagers.
Sway the Camper
Once you've built the Campsite Public Works Project somewhere in town, there's a chance a camper will use the campsite--a tent will appear in the campsite should this occur.
So long as you have 9 villagers or less living in town, when you speak to the camper, there's a chance they'll mention a desire to move into your lovely town.
Convince them to move in and they may do so, but be warned that some campers are stubborn and you'll need to speak to them many times to convince them--worse, some will only live in your town if you win a mini-game.
When one of your villagers has packed their boxes (literally) and is about to leave town, if another player--who has 9 villagers or less in their town--visits your town via Local Play or Online Play, they can convince your villager to go to their town.
Conversely, you can turn the situation around, so you can grab someone else's villager if they're in boxes (and you have 9 or less villagers). In some online communities, you can buy or sell villagers this way (it's a virtual slave trade all right...).
If you ever take your 3DS or 2DS for walkies to StreetPass other folks, know that Animal Crossing is hugely popular and you're almost bound to get tags for this game.
When another player has had a villager leave without a place to go ("into the void", so to speak), there's a chance they will pass on that villager to strangers they've tagged via StreetPass--so, you, basically.
A similar situation occurs during Local Play or Online Play. Be aware of this if you don't want to receive somebody else's junk villager.
We've spoken a bit about villagers leaving already, but once you have at least 9 villagers, there's a chance one of your villagers will want to leave town, permanently.
When this occurs, they'll give you 5 days advance warning and you'll have the option to convince them to stay right on the spot.
Tell them to go, however, and there's a chance they may stay anyway (so if you REALLY hate them, then save beforehand and reset if they opt to stay).
Should both you and villager decide that it's best they leave, you can speak to them before the 5th day to convince them to stay.
You can find out about leaving villagers from either the villager themselves, when they alert you (ping you), or the other villagers, when you speak to them.
On the 5th day, the villager who's leaving will be inside their house, boxes all packed and ready to leave. At this point, you cannot convince them to stay, but you can let another player take the villager off your hands.
In the situation where your 9th villager has left and you're back down to 8 villagers, after a few days, a new random 9th villager will appear, bumping up your villager count up to 9 again.