Transfer Learning
Its importance for Machines, not just Humans.
What is Transfer Learning?
The concept of transfer learning lies in imparting knowledge learned for performing a task to another task that is different but similar.
How is Transfer Learning Useful to Me?
In the context of humans, transfer learning is crucial to our lives. Examples include:
- the knowledge of riding a bicycle can be transferred to, and further enhanced, for riding a motorbike
- the knowledge of interviewing for a new job can be transferred, and further adjusted from previous experiences of interviewing for related jobs
- learning to drive a truck is easier for one who has the skill of driving a car than one who has the skill of riding a motorbike
Machine Learning Example
Let us use the CIFAR-10 dataset that contains 10 categories of images — airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck.
Our task of interest is to classify every image to its corresponding category. Of course, the most straightforward and common way to achieve this is to train a model, directly and from scratch, to recognise the 10 categories of images.
On the other hand, imagine we have a pre-existing neural model that has already learned to classify the images into 2 high-level categories — living things (bird, cat, deer, dog, frog, horse) and non-living things (airplane, automobile, ship, truck). Could we then impart the knowledge of this pre-existing model to a new model so that the new model learns faster and better in classifying the original 10 categories?
It turns out that we can! As seen in the following graph, learning with knowledge transferred from the pre-existing model is superior to learning from scratch, that is, learning naively. At any point in time, learning with transferred knowledge always outperforms naive learning. To match up to learning with transferred knowledge, a naive learner has to learn harder for longer.
What are the Implications?
If one provides an intelligent service, one is likely able to provide a better service if it transfer-learns than if it does not.
If one is competing with another who transfer-learns, one is likely to lose out if one simply learns hard.
Last Thoughts
Transfer Learning is undoubtedly an important skill. It is a technique that has allowed us humans to learn faster on newer tasks, based on knowledge acquired from previous tasks that is relevant to the newer tasks. It allows us to take shorter paths to learning new tasks and not to be always learning from scratch.
Consequently, we could even learn new tasks using a few experiences instead of requiring a large amount of experiences, through leveraging related knowledge that we have established previously. Even better, knowledge that we assimilate from the experiences of other people can further enrich our pre-established knowledge — to enable more successful transfer learnings!
