Microsoft's AI-Driven Dev Tool Turns Whiteboard Sketches into Code

Aptly named Sketch2Code, the tool was introduced by a company AI program manager on the Azure dev site, Tara Jana, who said, "We hope this post helps you get started with AI and motivates you to become an AI developer."

Part of that motivation might be the company's claim that the tool can radically condense the time-consuming process of converting whiteboard designs into development code and provide instant results.

"A layout algorithm uses the spatial information from all the bounding boxes of the predicted elements to generate a grid structure that accommodates all," Microsoft said. Finally, an HTML generation engine creates HTML markup code reflecting the result, using all of the gathered data.

"By combining these two pieces of information, we can generate the HTML snippets of the different elements in the design," the experimental tool's site says. "We then can infer the layout of the design from the position of the identified elements and generate the final HTML code accordingly."

The process is aided by several Azure cloud services. Specifically, Microsoft listed the following components that power the tool:

A Microsoft Custom Vision Model: This model has been trained with images of different handwritten designs tagging the information of most common HTML elements like buttons, text box, and images.

A Microsoft Computer Vision Service: To identify the text written into a design element a Computer Vision Service is used.

An Azure Blob Storage: All steps involved in the HTML generation process are stored, including the original image, prediction results and layout grouping information.

An Azure Function: Serves as the backend entry point that coordinates the generation process by interacting with all the services.

An Azure Web site: User font-end to enable uploading a new design and see the generated HTML results.