One-sentence Summary

UC Berkeley and ICSI introduce DeCAF, an open-source implementation of deep convolutional activation features extracted from supervised object recognition networks, demonstrating that these features can be repurposed for generic visual tasks such as scene recognition and domain adaptation and significantly outperform the state-of-the-art without task-specific training.

Key Contributions

• The paper releases DeCAF, an open-source implementation of deep convolutional activation features and associated network parameters, enabling reuse of pretrained CNN features for transfer learning on vision tasks with limited data.

• The paper analyzes fixed features from different network levels and visualizes their semantic clustering, showing that pretrained CNN activations can serve as strong generic visual representations when new tasks lack sufficient data to train full deep architectures.

• Experiments using features from a network trained on ImageNet report improvements over conventional representations on Caltech-101, the Office domain adaptation dataset, Caltech-UCSD Birds, and SUN-397, with runtime and resource use not exceptional compared with HOG or KDES.

Introduction

The research context concerns perceptual learning for visual recognition, where the goal is to find representations that capture the semantic properties needed by a task. This matters because many real applications have limited training examples, including user-defined categories, fine-grained recognition, attributes, and domain adaptation, where large-scale supervised learning is impractical.

Earlier visual representations based on hand-engineered flat features, such as gradient histograms and part-based detectors, have been effective but have likely plateaued. Deep convolutional networks can perform very well when trained on large labeled datasets such as ImageNet, but they can dramatically overfit when training data are scarce. Prior transfer-learning and representation-learning methods also struggled to scale, or had only modest success on larger visual recognition problems, especially when dataset bias or domain shift was present.

The authors leverage a deep convolutional network trained on ImageNet as a generic supervised pre-training base and then transfer its learned feature-space representation to new tasks with limited labels. They frame this as a semi-supervised multi-task or transfer-learning approach, or equivalently as a new visual feature called DeCAF. Their key contribution is empirical evidence that such convolutional features outperform conventional representations on several benchmark tasks, including Caltech-101, Office domain adaptation, Caltech-UCSD Birds, and SUN-397, while also showing better semantic clustering and reasonable runtime costs compared with features such as HOG.

Dataset

• The authors use the benchmark Office dataset for domain adaptation experiments.
  • Composition: three image domains covering product photography and office-camera imagery.
  • Sources: Amazon (product images from amazon.com), Webcam, and Dslr (images captured in an office environment with a webcam or digital SLR camera).
• Dataset scale and filtering:
  • The paper does not report exact subset sizes or detailed filtering rules for Amazon, Webcam, and Dslr.
  • It uses the standard Office benchmark setup introduced by Saenko et al. (2010).
• Supervised domain adaptation split:
  • Each experiment treats one domain as the labeled source domain.
  • Another domain is the target domain with either no labels or a small amount of labeled data.
  • Experiments focus on two domain shifts: Amazon to Webcam and Dslr to Webcam.
  • Results are averaged over 5 train/test splits.
• Data usage and processing:
  • Images are used to extract SURF and DeCAF features for classification with SVMs and logistic regression.
  • Models are trained under three label regimes: source only, target only, and source plus target.
  • No special cropping, metadata construction, or dataset reweighting is described beyond the standard Office dataset protocol.

Method

The authors leverage a supervised convolutional neural network as a reusable visual feature extractor. The system first trains a large convolutional model on an ImageNet-style 1000-class classification task using a standard supervised objective. The architecture follows an AlexNet-style design, consisting of $5$5 convolutional stages interleaved with pooling and rectified linear activations, followed by $3$3 fully connected stages. The network input is a mean-centered raw RGB image patch of size $224 \times 224$224×224. During preprocessing, the authors warp images to $256 \times 224$256×224 before cropping, and they do not use random RGB principal component augmentation.

Given an input image $x$x, the forward pass can be viewed as a sequence of layer transformations:

followed by a softmax classifier over 1000 object categories:

Although the full forward pass yields strong classification performance, the main output of the method is not the final prediction. Instead, the authors truncate the forward pass at selected intermediate layers and collect neuron activations as deep convolutional features. These activation vectors are intended to serve as general-purpose representations for other vision tasks.

To make the pipeline practical, the authors implement an open-source framework in Python. Computation-heavy operations are implemented in C and linked to Python, while numerical operations use standard packages such as numpy and scipy. This design allows trained networks to be executed efficiently without a GPU, reaching about 40 images per second on an 8-core commodity machine in minibatch mode. Released network parameters allow direct feature extraction without retraining the large model.

The authors also examine which activation layers provide useful semantic structure. They extract high-dimensional feature vectors from different layers of the trained network and project them into a two-dimensional space using t-SNE. Points are colored according to higher-level semantic categories so that clusters can be compared across feature types. As shown in the figure below, early pooling activations remain less semantically organized, while late hidden-layer activations form clearer semantic groups.

This behavior suggests that later hidden layers encode more transferable, high-level visual information. To test whether this structure persists outside the training dataset, the authors project ImageNet-trained network activations extracted from an independent scene recognition dataset into a t-SNE embedding and color them by broad semantic labels. The figure below shows that high-level activations from the original network still separate coarse semantic groups, supporting the use of these features for general recognition tasks.

The authors also analyze the computational profile of the forward pass inside their framework. The per-layer timing breakdown below shows that convolutional and fully connected operations consume most execution time because they involve large matrix multiplications.

The timing distribution further reveals that, in large networks, the final fully connected layers can dominate runtime because they require large transformation matrices. This observation is relevant when the method is extended to very large category spaces or wider hidden layers, where efficient output computation becomes a system-design concern.

Experiment

Experiments extract frozen activations from later hidden layers of an ImageNet-trained convolutional network and pair them with simple linear classifiers on object recognition, domain adaptation, subcategory recognition, and scene recognition benchmarks. The results show that mid- and late-layer features generalize broadly across tasks, often outperforming hand-engineered feature pipelines and remaining effective with limited labeled data, while dropout further improves robustness. The evaluations also indicate that deeper visual abstractions capture semantic category structure across domains and scenes, and that explicit part-based pooling can further strengthen fine-grained discrimination.

DeCAF features with a simple logistic classifier achieve higher bird subcategory accuracy than prior baselines, showing that generic deep visual features capture fine-grained distinctions. Adding pose-normalized part-based pooling further improves performance and gives the best result among the compared methods. DeCAF features outperform earlier bird subcategory methods without task-specific feature design Part-based pooling improves DeCAF and surpasses the previous best reported baseline

On the Office dataset, DeCAF features lead to markedly higher multi-class accuracy than SURF for both Amazon to Webcam and Dslr to Webcam domain shifts. The improvement appears across logistic regression and SVM classifiers, and is especially strong for Dslr to Webcam, where source-domain DeCAF models perform close to target-domain models. Training with both source and target labels gives the best DeCAF results, but DeCAF already reduces the domain gap substantially with only source labels. DeCAF features produce much higher accuracy than SURF under standard classifier settings. For Dslr to Webcam, source-only DeCAF models are close to target-only accuracy, indicating a small domain shift. Combining source and target labels further improves DeCAF accuracy across domain adaptation settings.

DeCAF features paired with simple linear classifiers achieve scene recognition performance above the traditional-feature multi-kernel baseline on SUN-397. The best configurations reach roughly 41% average accuracy, with DeCAF7 performing strongly and consistently across logistic regression and SVM classifiers. This suggests that generic deep image features can transfer effectively from object recognition to large-scale scene classification. DeCAF with simple classifiers outperforms the multi-kernel traditional-feature baseline of 38.0%. DeCAF7 yields competitive results for both logistic regression and SVM, indicating robust feature quality. Classifier differences are modest, with logistic regression slightly stronger than SVM in several reported settings.

Experiments evaluate generic deep visual features on fine-grained bird subcategory recognition, cross-domain image classification, and scene recognition. Across these tasks, DeCAF features paired with simple classifiers outperform traditional feature baselines, while pose-normalized part-based pooling further improves bird subcategory performance. The results also show that DeCAF reduces domain shift in Office classification, especially when only source labels are available, and that combining source and target labels yields the strongest transfer results. Overall, the findings indicate that generic deep features can transfer effectively to diverse recognition and domain adaptation settings without task-specific feature design.