Data representation is a crucial task in machine learning applications so that their success depends on the representation quality. Recently, Deep learning techniques could achieve the best rank in most machine learning applications due to their power in providing robust and discriminative representations for the data at hand. These methods could significantly improve the power of machine learning techniques in solving the problems which were poorly handled by existing shallow methods such as speech recognition, image colorization, image captioning, etc. The Deep Learning phenomenon is actually nothing more than the well-known neural networks with several number of layers which makes them able to provide several levels of abstractions and improve the generalization ability. There was a drawback in basic Neural networks before 2003 because of the problem in their training process called “vanishing gradient problem” which was solved by Hinton in 2003 and the Deep learning revolution appeared in machine learning field so that large number of last publications are accounted for Deep learning field.