Speak to Karliah and get some answers. The evidence is damning against Mercer and Karliah can fill in a lot of the story. You'll receive some potions, which is nice of her. Now fast-travel/carriage to Winterhold and enter the Frozen Hearth Inn where you'll find Enthir . Enthir will say the journal is in Falmer, and he can't translate it. He'll tell you of a man named Calcelmo who can though and now we know where we need to go. Fast-travel or carriage yourself to Markarth so we can find Calcelmo.
Once here, go all the way west and enter the Understone Keep. Go left at the fork and speak to Calcelmo. Whatever you do, he won't give you access to his research. Dammit. Now we get two more objectives, one optional. The optional one is to get the Museum's key. Umm... to get this you'll need to pickpocket the key that's laying on the table behdind him. Save before doing so and if you equip the Thieves Guild Boots, you'll find this easier. Drink a pickpocketing potion if needed.
Return to the entrance and go north. A guard tells you NOT to go beyond into the Museum. Ignore him, get past him and then Sneak. When he's not looking, use the key you just stole OR lockpick the door and enter the Museum. NOTE that you must be stealthy in this part, just like when we were chasing down Gulum-Ei once more.
Note: Also, if you're a stone cold killer, you could always assassinate all three guards without being caught to freely explore this area. Be careful of getting caught though, I've had it where I still had a bounty even though I killed all the witnesses. Bear in mind that if you are wanted; then you can't fast travel until all of the witnesses/guards have been killed.
If you get caught and don't care, simply kill the three guards and then proceed north to the next area. If not, sneak your way through the area using the right side. It's easier this way, or at least it was for me. In this next room, there are two doors to the east and one to the west. Enter the first room to the east to find a Spider Control Rod.
After grabbing it, a little Dwarven Spider comes out and follows you. You can use the rod to control where it goes and if there's an enemy nearby, it will attack them!
Note: Now that your past the guards, you'll encounter new enemies (who are Calcelmo's guards). It doesn't matter if you take them out or not, but I strongly recommend killing them. Its very difficult to sneak by them, plus you can always profit from the extra loot.
An enemy comes out of the second door on the east side, so kill him and there's another enemy in the room to the west. Kill them and loot their bodies for some keys. Go through the door to the west and continue on. The side doors don't have anything special, but take a look if you so desire. You may find some scrolls that aren't worth much in a bedroom.
In here, listen to the two soldiers and then kill them. Several more will appear. Make your way through them and go south (the area down the ramps on either side doesn't have anything for you). From here, go up the ramp to the northeast and in the next room you'll find several enemies. Kill them all and loot the adjacent rooms. Now proceed east and you'll get to a hall with a gate on the left. Before entering, continue forward and activate the valve to activate the traps in the adjacent room. Oh, you're evil!
Once it kills all or most of the enemies, go inside and kill the remaining enemies. Now check the books on the table to find a SKILLBOOK called "Twin Secrets" that raises your Enchanting level by 1. In this same room, check the southeast corner to find the Dwarven Puzzle Cube...
Now proceed to the Balcony on the east. Go up the stairs up here and into the tower. In here, go southeast, up the stairs and into the room. Read the SKILLBOOK named "Chimarvamidium" on the table to have your Heavy Armor level increase by 1 . In here, also make sure to pick up a piece of Charcoal from the table and a piece of paper . Now go through the double doors and examine the stone.
Some enemies come in and start searching the tower for you, but we can be sneaky here. From the stone, drop to the left and up the little stairs. At the gap, jump on the small pillar and from here, on to the metal catwalk (now don't YOU feel stealthy and bad-ass right about now?). Now simply go to the exit. Out here, go where the waterfall is (not the one by the stone bridge, the one past the small path to the south) and jump on it so you can land safely in the water.
Fast-travel back to Winterhold and go see Enthir and Karliah. Listen to all of the talk here and at the end you'll receive a Nightingale Blade from Karliah. You'll also be done with this quest and will start the next one automatically.