Sketch2Code Transforms Handwritten Notes into Working Code within Seconds – and it’s Open Source!
- Sketch2Code is a web-based offering that uses machine learning to turn handwritten designs into working HTML code
- The application uses various machine learning elements, including computer vision
- It’s open source! The code and documentation is available on Microsoft’s GitHub repository
How many times have you brainstormed and come up with solutions on whiteboards, only to then write the entire code to make it workable on your machine? I’ve done this hundreds of times, and it eventually wears on you. How useful would it be if you could simply copy the whiteboard design into your machine without needing to code through it?
This isn’t magic or the stuff of dreams anymore, thanks to the combination of machine learning and Microsoft. Sketch2Code is a web based solution that uses machine learning to transform your handwritten notes from an image to a working HTML markup code. And it’s open source!
The below image, taken from Microsoft’s official blog post announcing this offering, shows the application workflow:
As someone who is interested in data science, you’ll be wondering how machine learning works in this particular application. Wonder no more, because below I’ve listed the elements that went into designing Sketch2Code:
- Microsoft Custom Vision Model: This model, as one might expect, has been trained using images of different handwritten notes. These notes contain information about HTML elements like buttons, text boxes, and images
- Microsoft’s Computer Vision Service: This is used to identify the written text
- Blob Storage: A critical point to note – all the steps that are taken in the HTML generation process are stored, including the image you uploaded and the prediction results
- Azure Function: The backend which coordinates the HTML generation process
- Azure Website: This is basically where the user gets to see the final HTML output
You can access all the code and documentation for Sketch2Code on GitHub. This repository contains very detailed steps on each element we saw above, and is more than enough to (at the very least) get your feet wet.
Our take on this
Amazing! I tried this out myself and can attest to how useful this is. Note that this release is still a bit raw and might not give you the perfect result every time, but the concept itself deserves praise. This will save money and time for any business looking for quick web design solutions.
Computer vision just continues to amaze me. We’re living in a glorious age of machine learning and we should take a moment to appreciate that fact. And when you’re done, go ahead and clone the Sketch2Code repository and get started with it!
Subscribe to AVBytes here to get regular data science, machine learning and AI updates in your inbox!