The Spriggan cave you're looking for is to the southeast of Varnhold, and is marked as the "Overgrown Cavern" on your map. To reach it from Varnhold, follow these directions:
Immediately inside you'll find the probable cause of the Spriggan's displacement in the form of some barbarians. Approach them and wait for them to start up a conversation, and if you have Amiri with you she'll get you past peacefully. Amiri being a diplomatic boon? Guess there's a first time for everything.
Without Amiri, you can still talk your way through the encounter peacefully - and if you want answers, you'd be well-served by doing so. If you claim to be the local baron(ess) they'll believe you outright if your protagonist is a human, half-orc or dwarf. If you're another race you'll need to pass a [Diplomacy 20] check to convince them, otherwise their prejudices will lead them to deny your claim. If you fail this check and have Valerie in your party, she'll talk some sense into them, otherwise you'll need to look for other options. These include claiming to be a traveler (a [Diplomacy 20], which, if failed, has no consequences) try to threaten your way in ([Intimidate 25], and they'll attack if you fail) or you can just bribe them for 100 GP.
However you manage it, get past the barbarians (peacefully!) then continue north until you reach a fork. At the fork, search some ruined stonework to find an Ancient Scrap of Script-Covered Leather. Loot it, then turn northwest to reach a side-cavern where the barbarians are encamped, including this group's leader, Bolga.
Talk to this Bolga and be sure to pick the dialogue options "What are you doing here?", "Who are these sisters?", and finally "Where might I find Dugath and these Sisters?". This will get you the information on the location of the barbarian's camp, which you'll need to get around to exploring when you're done in here.
After this, exhaust the rest of her dialogue options. If you offered to help the Spriggans, pick the dialogue option "The spriggans who lived in this cave are under my protection. Please leave so they can return to their home.", which will lead to a variety of checks. Succeed at an [Intimidate 23] check, a [Diplomacy 22] check or a [Bluff 21] check, the first of which will result in a fight if you fail. If you lose your nerve, you can always back out by picking a [Chaotic Neutral] option to pretend you were joking. If you succeed, the barbarians will leave immediately.
Whether you succeed or fail, loot a crate nearby to secure the Cheese Crostata recipe. If you return southeast to the fork and from there continue northeast you'll find a dead end with two chests and a crate you can loot. The southernmost one [Trickery 28] contains a pair of Bracers of Armor +4, while the northern two contain a variety of gems, jewelry, gold, a scroll and some food.
Note: You can loot the cave freely if the barbarians have been convinced to leave, but if you get greedy while the barbarians are still around, they'll object. You'll either have to pass a [Bluff 20] or [Intimidate 22] check, or pay 100 GP go avoid a fight.