Agility affects your Small Guns and Sneak skills, and the number of Action Points available for V.A.T.S.

“When a fella’s in a gunfight and shoots the other guy six times before they can get off a shot, it’s cause that fella is agile.”

Agility is more of a mixed bag in New Vegas than it was in Fallout 3. Action Points are still good and all, but since VATS doesn’t make you more-or-less invincible anymore, it can be a liability. It’s often better to use VATS sparingly, when you have time to pull off some safe shots, rather than spam VATS relentlessly. Being able to use VATS more often is better than taking more shots in VATS. Also with the new iron sights, you can kill enemies more effectively at a distance… again, lessening the need for VATS this time around. Still, you should keep a minimum of six points (or five plus the Implant) if you intend to get the Silent Running perk.

Grab seven points of Agility to start with, raising it to eight with the Small Frame trait. It will eventually reach nine when you get the Implant, which is plenty as far as Action Points and reload speed is concerned.