This activity can be acquired from Milan, the chief brawler of Rattay. You can find him wandering the streets of the town. He wears a red tunic and a green cap, as seen below.
Milan will inform you about the rules of the fighting; no weaponry, fists only. He will also only fight for silver, not items. Before you can fight Milan, you must first defeat two other opponents; Ringlet and Stephen, who can be found in Rattay.
Once you have beaten both Stephen and Ringlet in a fist fight, you can approach Milan to let him know that you are ready to fight him. With Milan defeated, you can return to him and the others at any time thereafter for more fights.