Multi-Task Learning has emerged as a methodology in which multiple tasks arejointly learned by a shared learning algorithm, such as a DNN. MTL is based onthe assumption that the tasks under consideration are related; therefore itexploits shared knowledge for improving performance on each individual task.Tasks are generally considered to be homogeneous, i.e., to refer to the sametype of problem. Moreover, MTL is usually based on ground truth annotationswith full, or partial overlap across tasks. In this work, we deal withheterogeneous MTL, simultaneously addressing detection, classification &regression problems. We explore task-relatedness as a means for co-training, ina weakly-supervised way, tasks that contain little, or even non-overlappingannotations. Task-relatedness is introduced in MTL, either explicitly throughprior expert knowledge, or through data-driven studies. We propose a noveldistribution matching approach, in which knowledge exchange is enabled betweentasks, via matching of their predictions' distributions. Based on thisapproach, we build FaceBehaviorNet, the first framework for large-scale faceanalysis, by jointly learning all facial behavior tasks. We develop casestudies for: i) continuous affect estimation, action unit detection, basicemotion recognition; ii) attribute detection, face identification. We illustrate that co-training via task relatedness alleviates negativetransfer. Since FaceBehaviorNet learns features that encapsulate all aspects offacial behavior, we conduct zero-/few-shot learning to perform tasks beyond theones that it has been trained for, such as compound emotion recognition. Byconducting a very large experimental study, utilizing 10 databases, weillustrate that our approach outperforms, by large margins, thestate-of-the-art in all tasks and in all databases, even in these which havenot been used in its training.