Learn the Different Frameworks of Deep Learning- Keras, Pytorch, Theano, TensorFlow and easily Compare them

By ridhigrg |Email | May 22, 2019 | 7755 Views

Keras: Nice, well-architected API on top of either Tensorflow or Theano and potentially extensible as a shim over other deep learning engines as well. The documentation is of high quality and easy to understand. The fact that Keras is nominally independent of the engine means that it is theoretically possible to reuse Keras code even if you have to swap the engine for performance or other reasons. Its downside is that whenever you want to implement relatively new or esoteric architectures, you usually have to descend below the Keras layer and use Tensorflow or Theano. 

This mostly happens when we need to use advanced NumPy-style indexing, which translates to Tensorflow's gather/scatter and Theano's set/inc_subtensor. Furthermore, Keras is conceptually committed to the automatically optimized computation graph that was pioneered in Theano and continued in Tensorflow. Still, some highly recommend Keras as a high-level interface layer if either Theano or Tensorflow is going to be used.

Theano: Theano has the benefit and drawback of being the oldest and most well established of the three. Because it is older, most of the features requested by users have been implemented. However, because there wasn't a prior example to follow, some of these implementations are a bit too complicated and difficult to use (see the scan function documentation for a good example of what I mean). The documentation is pretty good but is still confusing. It can be very difficult to get a complex project to perform correctly in Theano since there is no simple way to inspect intermediate computations.

Debugging is generally done by inspecting computation graphs or using debuggers. Also, Theano lags Tensorflow in performance and in the pace of development, since the team behind Theano does not have access to either the market clout or the scale of engineering resources that the Tensorflow team has through Google. 

TensorFlow: Tensorflow is basically a streamlined, industry-backed refinement of Theano. It is more efficient and has some support for distributed processing. The function names and terminology used in Tensorflow differs from Theano a bit, which can make it harder to switch from Theano than it should be. Still, the underlying computation graphs in Tensorflow work pretty much the same as in Theano with most of the same benefits and drawbacks. Tensorflow makes it a bit easier to inspect intermediate state thanks to the eval function, but the optimization of the computation graph is still quite opaque, even though the optimizations have a serious effect on performance. 

Tensorflow has essentially taken over the role that Theano and Caffe held a couple of years ago and is now the go-to tool for deep learning, perhaps with frameworks like Keras over top. Recently, Pytorch has given Tensorflow a serious run for its money, forcing the Google team to adapt. There is a stripped down version of Tensorflow in the works that is less dependent on the graph compilation step.

PyTorch: Pytorch is a Python port of Torch, backed by Facebook. As a competitor to the above technologies, it offers just-in-time graph compilation, which means that Pytorch code fits in better with the surrounding Python because it doesn't treat graphs as separate and opaque objects. Instead, you can assemble tensor computations ad hoc in very flexible ways. It is also quite performant. Like Tensorflow, it offers solid multi-GPU support, though Tensorflow still wins for larger distributed systems. 

Pytorch's API is well-documented but less refined than Tensorflow or Keras. Nonetheless, Pytorch wins on flexibility and ease of use without sacrificing performance, and it is definitely forcing Tensorflow to retrench and adapt.

Source: HOB