Learn everything about Analytics

Home » Sketch2Code Transforms Handwritten Notes into Working Code within Seconds – and it’s Open Source!

Sketch2Code Transforms Handwritten Notes into Working Code within Seconds – and it’s Open Source!


  • Sketch2Code is a web-based offering that uses machine learning to turn handwritten designs into working HTML code
  • The application uses various machine learning elements, including computer vision
  • It’s open source! The code and documentation is available on Microsoft’s GitHub repository



How many times have you brainstormed and come up with solutions on whiteboards, only to then write the entire code to make it workable on your machine? I’ve done this hundreds of times, and it eventually wears on you. How useful would it be if you could simply copy the whiteboard design into your machine without needing to code through it?

This isn’t magic or the stuff of dreams anymore, thanks to the combination of machine learning and Microsoft. Sketch2Code is a web based solution that uses machine learning to transform your handwritten notes from an image to a working HTML markup code. And it’s open source!

The below image, taken from Microsoft’s official blog post announcing this offering, shows the application workflow:

As someone who is interested in data science, you’ll be wondering how machine learning works in this particular application. Wonder no more, because below I’ve listed the elements that went into designing Sketch2Code:

  • Microsoft Custom Vision Model: This model, as one might expect, has been trained using images of different handwritten notes. These notes contain information about HTML elements like buttons, text boxes, and images
  • Microsoft’s Computer Vision Service: This is used to identify the written text
  • Blob Storage: A critical point to note – all the steps that are taken in the HTML generation process are stored, including the image you uploaded and the prediction results
  • Azure Function: The backend which coordinates the HTML generation process
  • Azure Website: This is basically where the user gets to see the final HTML output

You can access all the code and documentation for Sketch2Code on GitHub. This repository contains very detailed steps on each element we saw above, and is more than enough to (at the very least) get your feet wet.


Our take on this

Amazing! I tried this out myself and can attest to how useful this is. Note that this release is still a bit raw and might not give you the perfect result every time, but the concept itself deserves praise. This will save money and time for any business looking for quick web design solutions.

Computer vision just continues to amaze me. We’re living in a glorious age of machine learning and we should take a moment to appreciate that fact. And when you’re done, go ahead and clone the Sketch2Code repository and get started with it!


Subscribe to AVBytes here to get regular data science, machine learning and AI updates in your inbox!


You can also read this article on our Mobile APP Get it on Google Play
This article is quite old and you might not get a prompt response from the author. We request you to post this comment on Analytics Vidhya's Discussion portal to get your queries resolved