At the edge of the cliff is a rope that players can use to jump off the edge, essentially into the Deepwild of Krasarang Wilds. It seems the option is to either use the rope or the buff from the barley sack to get down. The rope is the safer option.

Once down, head north back to the cliff edge, then east to get to the cave.