One-sentence Summary

Researchers at Apple introduce FLAIR, a large-scale annotated image dataset of $429{,}078$429,078 images from $51{,}414$51,414 Flickr users for multi-label classification in cross-device federated learning, capturing heterogeneous user data and long-tailed label distributions to serve as a challenging benchmark for advancing the state of the art in this paradigm.

Key Contributions

• This work introduces FLAIR, a large-scale annotated image dataset for multi-label classification in federated learning, containing 429,078 images from 51,414 Flickr users and reflecting real-world intricacies such as heterogeneous user data and long-tailed label distributions.
• The paper provides reproducible baselines across centralized, federated, and differentially private learning setups, enabling standardized evaluation of methods under varied privacy constraints on this benchmark.
• The dataset and benchmark code are released publicly at the project repository, giving the research community a challenging and accessible resource for advancing federated learning research.

Introduction

Remote devices generate vast amounts of data that could improve on-device machine learning, but privacy, bandwidth, and storage constraints make centralized data collection infeasible. Federated learning addresses this by training models collaboratively while keeping data on user devices, and when combined with differential privacy, it provides strong formal guarantees. However, progress in this field is hindered by a lack of realistic, large-scale benchmark datasets: current image benchmarks either rely on artificial partitions of classic datasets like MNIST or CIFAR, which may not reflect real-world non-IID distributions, or they use realistic user partitions but with limited size, such as the Landmarks dataset with only 1,262 users. To fill this gap, the authors introduce FLAIR, a large-scale multi-label image classification dataset with 429,078 images from 51,414 real Flickr users, featuring both coarse and fine-grained label hierarchies. FLAIR naturally exhibits common federated challenges including imbalanced user contributions, feature skew, and label distribution skew, and the authors provide reproducible benchmarks for centralized, federated, and differentially private settings, as well as with random and pretrained initializations.

Dataset

The authors construct the FLAIR dataset for federated image classification research. Below is a summary of how it is composed, processed, and used.

Sources and composition

• Images are curated via the Flickr API, with original Flickr user IDs preserved so that all images from one user are naturally grouped together.
• Only publicly shared, permissively licensed images are included.

Filtering and annotation

• A two stage filtering process removes personally identifiable information (PII). First, a face detection model automatically removes images containing faces. Then, human annotators review the remaining images, with a two annotator setup where the first flags PII and the second validates the result.
• The initially unlabeled images are annotated with a taxonomy of 1,628 fine-grained classes. The taxonomy also defines 17 coarse-grained classes, and each fine-grained class is mapped to one coarse-grained class.
• Annotation also uses two annotators per image, one to label and one to validate. If the object is ambiguous and cannot be assigned a fine-grained label, a coarse-grained label is used instead.

Scale and statistics

• The finalized dataset contains 429,078 images from 51,414 Flickr users, with 17 coarse-grained and 1,628 fine-grained labels.
• Per-user image counts are heavily skewed. The largest 2.3% of users collectively hold as many images as the bottom 97.7% of users, giving FLAIR the second largest quantity skew among federated image benchmarks, after iNaturalist-User-120k.
• Feature distribution skew is measured using earthmover distance between each user's average pixel histogram and the population average, computed on the most common label (structure) to remove class imbalance effects. The natural non-iid partitioning increases skew compared to simulated non-iid partitioning.
• Labels are also imbalanced. The most common coarse-grained class, structure, appears 228,923 times and is present on 87% of users. The least common, religion, appears 866 times across 1.4% of users. Among fine-grained labels, 1,255 of the 1,628 classes appear on fewer than 0.1% of users.

Dataset split and usage

• The authors provide a fixed train, validation, and test split based on Flickr user IDs, so that no user appears in more than one partition.
• The split assigns 80% of users to training, 10% to validation, and 10% to test. This yields 345,879 training images, 39,239 validation images, and 43,960 test images.
• The preserved user grouping and natural skew are meant to reflect realistic federated learning conditions, where each user's local data distribution differs from the global distribution.

Method

The authors consider a federated learning setting in which a central server coordinates model training over a distributed cohort of users while keeping each user's data on their own device. In each communication round, the server samples a subset of users and sends the current global model to them. Each sampled user performs several steps of local SGD on their private data and then transmits only the resulting gradient update back to the server. The server aggregates these per-user updates and applies them as a pseudo-gradient to update the global model, following an optimizer such as SGD or Adam.

To mitigate the risk that shared gradient updates leak sensitive information about individual users, the authors integrate differential privacy (DP) into the federated training procedure. A randomized mechanism $\mathcal{M}:\mathcal{D}\mapsto\mathcal{R}$M:D↦R is said to satisfy $(\epsilon,\delta)$(ϵ,δ)-differential privacy if for any two adjacent datasets $d, d'$d,d′ and any subset of outputs $S \subseteq \mathcal{R}$S⊆R, it holds that

In the federated context, two datasets are adjacent if one can be obtained by adding or removing all examples associated with a single user. This definition ensures that the presence or absence of any one user's entire data contribution has a bounded effect on the model distribution.

To achieve this guarantee in practice, the authors modify the standard federated averaging algorithm in two key ways. First, each user's model update is clipped so that its $L_2$L2​ norm is bounded by a predetermined constant, limiting the sensitivity of the aggregated update. Second, Gaussian noise calibrated to the clipping bound and the desired privacy parameters is added to the aggregated update from the sampled cohort before the server applies it. For privacy accounting, the authors assume that each user is sampled uniformly and independently, and that the exact set of sampled users is kept hidden from any adversary.

In parallel with the algorithmic design, the authors collect and curate a dataset tailored to evaluate the performance of privacy-preserving federated learning. Images are sourced through the Flickr API, and the associated Flickr user IDs are preserved so that the data is naturally grouped by user, reflecting the non-IID distribution typical of real federated deployments. To eliminate personally identifiable information, a strict two-stage filtering pipeline is applied. First, a face detection model automatically removes any image containing a face. Second, human annotators manually inspect the remaining images and flag those that still contain PII, with a second annotator validating each flag to ensure accuracy. The images are then annotated with a taxonomy of 1,628 fine-grained object classes, further organized into 17 coarse-grained categories. Each image is labeled by one annotator and validated by another; if an object is ambiguous and cannot be assigned a fine-grained label, a coarse-grained label is used instead. This curation process yields a user-specific image dataset that supports realistic federated learning experiments under a formal privacy guarantee.

Experiment

The experiments benchmark the FLAIR dataset for multi-label classification using a ResNet-18 model across centralized, non-private federated, and differentially private federated learning settings, with metrics including precision, recall, F1, and averaged precision. Results show that federated learning substantially degrades performance relative to centralized training, with the largest drops occurring for fine-grained and infrequent classes, and that applying differential privacy further worsens these gaps, particularly for rare categories. Larger cohort sizes improve private federated learning convergence by reducing noise, while fine-tuning from pretrained models mitigates but does not eliminate the performance losses. The findings highlight class imbalance, long-tailed distributions, and DP noise as key challenges in heterogeneous private federated settings, motivating future research on few-shot learning, personalization, and noise-robust algorithms.

On the FLAIR benchmark, federated learning consistently underperforms centralized training, with larger gaps on per-class metrics than overall metrics. Differential privacy further degrades performance, and the negative impact is amplified for rare classes in both coarse and fine-grained taxonomies. Fine-grained labels are especially challenging, where federated models show drastic drops relative to centralized baselines regardless of initialization. Federated learning from scratch lags centralized training by about 20% on per-class metrics but only 6% on overall metrics, and these gaps shrink to 8% and 2% when fine-tuning from a pretrained model. Adding differential privacy to federated learning causes a roughly 40% per-class and 24% overall metric drop from scratch, while fine-tuning reduces these losses to about 30% and 10%. For fine-grained taxonomy, federated learning performs far worse than centralized, with gaps around 90% for per-class and 50% for overall metrics, independent of initialization. Rare classes suffer more under federated and private federated learning; the performance gap between frequent and infrequent classes widens with DP, as seen in coarse-grained class-wise AP differences. Larger cohort sizes in private federated learning reduce noise on aggregated updates and lead to faster, better generalization, especially when training from scratch or fine-tuning.

The table reports per-class average precision for coarse-grained categories under centralized, federated, and private federated learning, each with training from scratch or fine-tuning. Performance varies widely across classes and is consistently higher for more frequent classes; federated learning and differential privacy degrade performance, with a more pronounced impact on rarer classes, while fine-tuning mitigates some of the loss. Class frequency shows a positive correlation with precision across all training settings. The performance gap between frequent and infrequent classes widens under federated learning and becomes even larger when differential privacy is applied. Fine-tuning from a pretrained model reduces the precision drop caused by federated learning and differential privacy compared to training from scratch.

On the FLAIR benchmark, federated learning consistently underperforms centralized training, with larger gaps on per-class metrics than overall metrics, and differential privacy further degrades performance, especially for rare classes and fine-grained taxonomies. Training from scratch amplifies these gaps, while fine-tuning from a pretrained model mitigates some of the loss. Class frequency correlates positively with precision across all settings, and the performance gap between frequent and infrequent classes widens under federated learning and even more so with privacy applied. Larger cohort sizes in private federated learning lead to faster, better generalization by reducing noise on aggregated updates.