In Part I of this series we will briefly summarize the advantages of Tensorflow.js (TFJS) over the traditional backend-focused Python-based approaches to show you why it’s such a big deal.
Artificial Intelligence, Liberated
Apps are easy to share
The client provides the compute power
Training and predictions are offloaded to the user’s hardware. This eliminates significant cost and effort for the developer. You don’t need to worry about keeping a potentially costly remote machine running, adjusting compute power based on changing usage, or service start-up times. Forget about load balancing, microservices, containerization or provisioning elastic cloud compute capabilities. By removing such backend infrastructure requirements, TensorFlow.js lets you focus on creating amazing user experiences! However, we must still take care that the client’s hardware is powerful enough to provide a satisfying experience given the compute demands of our AI models.
Data never leaves the client’s device!
This is crucially important as users are increasingly concerned about protecting their sensitive information, especially in the wake of massive data scandals and such as . With TensorFlow.js, users can take advantage of AI without sending their personal data over a network and sharing it with a third party. This makes it easier to build secure applications that satisfy data security regulations, e.g. healthcare apps that tap into wearable medical sensors. It also lets you build AI browser extensions for enhanced or adaptive user experiences while keeping user behaviour inherently private. Visit for an example of a Google Chrome extension that uses TensorFlow.js to provide in-browser image recognition.
The TensorFlow.js architecture can be simplified further if a is available. In this case the client downloads the model and skips the training entirely. If further specialization of the model is necessary, considerable effort and training time can be saved by using extended training (transfer learning) on a pre-trained model rather than training from scratch.
Easier access to rich sensor data
Highly interactive and adaptive experiences
Real-time inferencing on the client side lets you make apps that respond immediately to user inputs such as webcam gestures. For example, Google released a webcam game that allows the user to play Pacman by moving their head (try it ). The model is trained to associate the user’s head movements with specific keyboard controls. Apps can also and react to human emotions, providing new opportunities to surprise and delight. Pre-trained models can be loaded and then fine-tuned within the browser using transfer learning to tailor them to specific users.
An image from Google’s WebCam Pacman game, where your head becomes the controller!"
TensorFlow.js also has the following helpful perks:
Use of pre-trained models
Hardware Acceleration for all GPUs
TensorFlow.js leverages onboard GPU devices to speed up model training and inference, thanks to its utilization of WebGL. You don’t need an NVIDIA GPU with CUDA installed to train a model. That’s right -- you can finally train deep learning models on AMD GPUs too!
Unified backend and frontend codebase
A couple of notes on what to consider before going full TFJS. TFJS can use significantly more resources than your typical JS applications, especially when image or video data are involved. For example, the Webcam Pacman app uses approximately 100 MB of in-browser resources and 200 MB of GPU RAM when running in Chrome v71 on MacBook pro with Mojave v10.14.2. *(2.2 GHz Intel Core i7, 16 GB 2400 MHz DDR4, Radeon Pro 555x 4096 Mb, Intel UHD Graphics 630 1536 MB).
Best performance requires browsers with WebGL support plus GPU support. You can check whether WebGL is supported by visiting . Even with WebGL, GPU support may not be available on older devices due to the additional requirement that users have up-to-date video drivers.
The requirements for modern hardware and updated software to achieve optimal performance can cause user experience to vary substantially across devices. Therefore extensive performance testing on a wide variety of devices, especially those on the lower end, should be done before distributing a TFJS app.
There are a couple of strategies to deal with potential performance limitations. Apps can check for available compute capabilities and choose a model size accordingly, with a corresponding tradeoff in accuracy. A 5-10 x reduction in memory requirements can often be achieved by reducing the model parameters without sacrificing more than 20% in accuracy. Reducing the model size will also help save bandwidth in scenarios where a pre-trained model is transferred to the client’s device. Good memory management can help keep the footprint of the app low by avoiding memory leaks (e.g. with TFJS’s tf.tidy()).
Finally, nothing can beat a good understanding of the data and problem domain. Many problems in image recognition, for example, can be solved by using images of surprisingly poor resolution. Some problems can also be simplified by using conventional algorithms with lower resource requirements or by guiding the user to do part of the work, e.g. by centering a to-be detected object.
Sounds great! So how do we get started? Visit Part II of this series to learn how to build your very own TFJS applications!
Here are some more fun examples to get your creative juices flowing!
- A holiday-themed interactive animation created by the Rangle.io team for December 2018.
- by Google. A fun game where you are challenged to seek out and photograph objects that look like emojis, under a time limit!
- by Google, where you can train a neural network to recognize specific gestures and use these to control an output image or sound.
- s by Reiichiro Nakano, a fun twist on the classic schoolyard game.
- by Brendan Sudol, which recognizes emotion from facial expressions in uploaded images, and tags them with the appropriate emoji.
In addition to games and apps, TensorFlow.js can be used to build interactive teaching tools for conveying key concepts or novel methods in machine learning and deep learning:
- : an interactive playground for General Adversarial Networks (GANs) in the browser.
- : a neat demo by Yingtao Tian of a Generative Adversarial Network that ‘dreams’ faces and morphs between them.
- : an in-browser demo of the tSNE algorithm for high-dimensional data analysis.
- : while not technically TFJS, it was built from code that eventually became TFJS, and can be considered the browser app that started it all!