One problem is that label placement algorithm needs to have font metrics. The possible solution would be to use RVSG for most of the work, and some font lib just to place the labels. It may lead to pretty complicated design.