Pranav Dar — August 31, 2018
AVbytes

Overview

  • Sketch2Code is a web-based offering that uses machine learning to turn handwritten designs into working HTML code
  • The application uses various machine learning elements, including computer vision
  • It’s open source! The code and documentation is available on Microsoft’s GitHub repository

 

Introduction

How many times have you brainstormed and come up with solutions on whiteboards, only to then write the entire code to make it workable on your machine? I’ve done this hundreds of times, and it eventually wears on you. How useful would it be if you could simply copy the whiteboard design into your machine without needing to code through it?

This isn’t magic or the stuff of dreams anymore, thanks to the combination of machine learning and Microsoft. Sketch2Code is a web based solution that uses machine learning to transform your handwritten notes from an image to a working HTML markup code. And it’s open source!

The below image, taken from Microsoft’s official blog post announcing this offering, shows the application workflow:

As someone who is interested in data science, you’ll be wondering how machine learning works in this particular application. Wonder no more, because below I’ve listed the elements that went into designing Sketch2Code:

  • Microsoft Custom Vision Model: This model, as one might expect, has been trained using images of different handwritten notes. These notes contain information about HTML elements like buttons, text boxes, and images
  • Microsoft’s Computer Vision Service: This is used to identify the written text
  • Blob Storage: A critical point to note – all the steps that are taken in the HTML generation process are stored, including the image you uploaded and the prediction results
  • Azure Function: The backend which coordinates the HTML generation process
  • Azure Website: This is basically where the user gets to see the final HTML output

You can access all the code and documentation for Sketch2Code on GitHub. This repository contains very detailed steps on each element we saw above, and is more than enough to (at the very least) get your feet wet.

 

Our take on this

Amazing! I tried this out myself and can attest to how useful this is. Note that this release is still a bit raw and might not give you the perfect result every time, but the concept itself deserves praise. This will save money and time for any business looking for quick web design solutions.

Computer vision just continues to amaze me. We’re living in a glorious age of machine learning and we should take a moment to appreciate that fact. And when you’re done, go ahead and clone the Sketch2Code repository and get started with it!

 

Subscribe to AVBytes here to get regular data science, machine learning and AI updates in your inbox!

 

About the Author

Pranav Dar

Senior Editor at Analytics Vidhya. Data visualization practitioner who loves reading and delving deeper into the data science and machine learning arts. Always looking for new ways to improve processes using ML and AI.

Our Top Authors

Download Analytics Vidhya App for the Latest blog/Article

2 thoughts on "Sketch2Code Transforms Handwritten Notes into Working Code within Seconds – and it’s Open Source!"

Praneeth
Praneeth says: September 05, 2018 at 8:22 am
The url (https://sketch2code.azurewebsites.net/) is not working to launch it. Reply
Pulkit Sharma
Pulkit Sharma says: September 05, 2018 at 11:05 am
Hi Praneeth, The link works fine at my end. Please refresh the page and try again. Reply

Leave a Reply Your email address will not be published. Required fields are marked *