"Die andere Theorie besagt, dass Scholli auf eine reale Person zurückgeht, nämlich auf Ferdinand Joly"
According to Wikipedia he was a vagrant in Austria who lived a very own kind of lifestyle.

And to your Question, no it would be in no way offensive to use this. Maybe some people would find it funny to hear a foreigner say this (even for German people this is not a common expression, at least not for my age).