This neural network lets you transform graphic designs into functioning code
The gap between graphic designers and coders just got smaller. Until now, if you wanted to build a graphic user interface, whether for app design or custom websites, you would have to enlist a developer to turn your ideas into code. But designers could soon take this power into their own hands.
Read more: DeepMind's new algorithm adds 'memory' to AI
Copenhagen-based startup UIzard Technologies has developed an app, Pix2Code, which uses a trained neural network to transform screenshots into lines of code. All it needs is an image of a design for a graphic user interface (GUI) and, once recognised, it begins to automatically churn out corresponding code. Even more impressively, it creates code that's compatible with Android, iOS and web-based technologies – a rare functionality across platforms.
The Pix2Code prototype only needs a single input image and has demonstrated an accuracy of 77 per cent. Tony Beltramelli, the founder of UIzard, states that he believes Pix2Code has the potential to "end the need for manually-programmed GUIs".
In order to program the neural network, Beltramelli says that the team had to overcome three main problems. The first being computer vision - that a computer will not automatically understand the given scene and cannot naturally identify the objects present, their positions, and characteristics (i.e. buttons and labels). Secondly, they had to overcome a language problem – the issue of teaching the network to understand text, so it could create accurate samples. Finally, the deep learning programme had to be trained to understand the link between code, text and the corresponding images, so it could connect all three while generating new code.
In the future, Beltramelli writes, Generative Adversarial Networks (GANs) could be used to further update Pix2Code. GANs have shown promising accuracy when generating sequences and images. However, this is a relatively unexplored area of research and the need to train neural networks takes a considerable amount of time – posing an obstacle for the quick progression of this technology. But the obstacle may soon be overcome – thanks to the internet.
"Considering a large number of websites are already available online and the fact that new websites are created every day, the web could theoretically supply an unlimited amount of training data," Beltramelli says.
Merging the fields of design and coding is becoming a trend – Google has recently launched a Data GIF Maker to transform code into colourful animations. It's the natural successor to pie charts and other spreadsheet methods of data presentation – except in this case the figures you enter into the program turn into bold comparisons that literally push against each other on screen.
This article was originally published by WIRED UK