The Idea to use SVG came as Together can write this as the only vector format. And when producing class diagrams etc. with Together, the PDF result is excellent. Unfortunately Together is no general purpose drawing tool.
After some investigation I found Sodipodi, which is a nice tool with some downsides. But ok, it can produce SVG. When doing some tests, the Fonts always looked ugly (the top line in this pdf). I did some testing around and found that Sodipodi does the following
<tspan>This is some text</tspan>
After removing the <tspan> elements, the text can no longer be moved around in the Sodipodi canvas, but looks like it should (bottom line in this pdf).
As <tspan> is pretty valid in SVG, this seems to be a problem in Batik or FOP, which is used by Batik to produce PDF. Especially as the conversion to PNG works as it should.