Parsing code snippets from markdown with Go

Posted on March 15, 2018

Those time, I’ve been thinking that choosing the right vendor that fit your needs or will solve your problem has became a pain.

With the growth of platforms like Github and dependencies management tools (whatever the language) more and more people share their work and open source the peace of code they have created to solve a specific problem. That’s great !

But at the end… you find out that for a given problem you want to solve, their are tones of libs that claim to do the same thing but each in a different way.

When I need to choose a third-party component, I usually start by reading code example (hopefully in the README file).

The code bellow is a peace of Go code that parse Github readme file of a given repository and look for code example to show in the terminal.

It uses the markdown parser golang-commonmark/markdown. Again, it took me time to find out the right vendor to do the job. You know… the one that don’t do too much magic and stay simple. But not to simple because I don’t want to waste time. The one that is also well documented….

Anyway, the code bellow can be run easily in command line using the --url flag within you pass the Github url of the repository you want to extract code snippets. It get the content of the README file, parse the markdown and print the extracted peaces of code.

If you combine this with the Go import system, it becomes really easy to quickly know how a third party library may be use :

go run main.go --url https://github.com/{the go lib you want to use}

Here is the code :

packagemainimport("github.com/golang-commonmark/markdown""fmt""net/http""io/ioutil""log""flag""net/url")//snippet represents the snippet we will output.
typesnippetstruct{contentstringlangstring}//getSnippet extract only code snippet from markdown object.
funcgetSnippet(tokmarkdown.Token)snippet{switchtok:=tok.(type){case*markdown.CodeBlock:returnsnippet{tok.Content,"code",}case*markdown.CodeInline:returnsnippet{tok.Content,"code inline",}case*markdown.Fence:returnsnippet{tok.Content,tok.Params,}}returnsnippet{}}//readFromWeb call the given url and return the content of the readme.
funcreadFromWeb(urlstring)([]byte,error){resp,err:=http.Get(url)iferr!=nil{returnnil,err}deferresp.Body.Close()returnioutil.ReadAll(resp.Body)}funcmain(){//Flag management
varurlStringstringflag.StringVar(&urlString,"url","",`The url of the github repository`)flag.Parse()ifurlString==""{log.Fatalln("Please, provide an url for the readme to parse.")}//Parse URL
u,err:=url.Parse(urlString)iferr!=nil{log.Fatalln("Impossible to parse the URL")}//Read the readme file
readMe,err:=readFromWeb(fmt.Sprintf("https://raw.githubusercontent.com%s/master/README.md",u.Path))iferr!=nil{log.Fatalf(err.Error())}//Parse the markdown
md:=markdown.New(markdown.XHTMLOutput(true),markdown.Nofollow(true))tokens:=md.Parse(readMe)//Print the result
for_,t:=rangetokens{snippet:=getSnippet(t)ifsnippet.content!=""{fmt.Printf("##### Lang : %s ###### \n",snippet.lang)fmt.Println(snippet.content)}}}