Capturing a monster can be tricky early on, and can still be prone to situational, and accidental, failure further down the line.
And that can be annoying, because not only is capturing a required act in some quests, but it also produces more resources during a free hunt and gives a greater chance to earn different resources from the monsters you capture. To put it simply, capturing is vital.
How to Capture a Monster
The act of capturing a monster can be summed up quite simply: When the monster is low on health - it will be visibly limping or panting - you need to lay down a trap (shock or pitfall) and then, once the monster is in the trap, you need to hit it with two-three tranquillizer bombs to 'catch' it.
The aim is to capture the monster alive, not dead; therefore, killing the monster automatically fails the quest (if it's a capture quest). If you do fail the quest, you will still gain the usual rewards for slaying a monster, so don't despair and try again later.
You're probably sick of hearing this, but you need to prepare before a capture quest. Really, really prepare. Bring spare combine materials type preparing.
It's one thing having a single Shock Trap, but you can bring two Trap Tools and some Thunderbugs , that way you can make at least two more traps should the first fail. Likewise, bring extra Tranq Bombs (Tranquilizer + Bomb Casing. Bomb Casing = Sap Plant + Iron Ore/Stone. Tranq = Sleep Herb + Parashroom).
In multiplayer Capture quests it's advisable that one player choose the Bowgun and bring a salvo of Tranq Shots. This is a no brainer and ensures that you have more Tranq items than if everyone goes melee. Just remember to load the ammo as soon as the trap goes down.
Time Your Trap
Laying a trap makes you defenceless for a few seconds so watch the creature's movements before you choose to lay it down. Even when the creature is panting, those few seconds are easily enough time for the creature to take a massive, desperate bite out of your health.
Also, if the creature is limping away with an aim in mind, avoid the temptation to intercept the monster. Because if you place the trap too late or if the creature's path deviates from your trap, you've just wasted a good trap and valuable minutes to wait for the trap to self-detonate.
Once you've laid the trap and everything looks rosy, remember to stand a good distance away from trap. There's nothing worse than the creature ramming into the trap, while also knocking you out. Then by the time you get back on your free, the monster's broken free.
It seems like a silly thing to say but having your tranq bombs ready means all the difference when it comes to the final moments. The last thing you want is for the creature to move into the trap only for you to be fumbling through your item list to find the bright red icon. Have it selected already.