Handbook of Digital Face Manipulation and Detection : : From DeepFakes to Morphing Attacks.

• Superior document: Advances in Computer Vision and Pattern Recognition Series
• Author: Rathgeb, Christian.
• TeilnehmendeR: Tolosana, Ruben.; Vera-Rodriguez, Ruben.; Busch, Christoph.
• Place / Publishing House: Cham : : Springer International Publishing AG,, 2022.; ©2022.
• Year of Publication: 2022
• Edition: 1st ed.
• Language: English
• Series: Advances in Computer Vision and Pattern Recognition Series
• Physical Description: 1 online resource (483 pages)

Table of Contents

• Intro
• Preface
• Contents
• Part I Introduction
• 1 An Introduction to Digital Face Manipulation
• 1.1 Introduction
• 1.2 Types of Digital Face Manipulations
• 1.2.1 Entire Face Synthesis
• 1.2.2 Identity Swap
• 1.2.3 Face Morphing
• 1.2.4 Attribute Manipulation
• 1.2.5 Expression Swap
• 1.2.6 Audio-to-Video and Text-to-Video
• 1.3 Conclusions
• References
• 2 Digital Face Manipulation in Biometric Systems
• 2.1 Introduction
• 2.2 Biometric Systems
• 2.2.1 Processes
• 2.2.2 Face Recognition
• 2.3 Digital Face Manipulation in Biometric Systems
• 2.3.1 Impact on Biometric Performance
• 2.3.2 Manipulation Detection Scenarios
• 2.4 Experiments
• 2.4.1 Experimental Setup
• 2.4.2 Performance Evaluation
• 2.5 Summary and Outlook
• References
• 3 Multimedia Forensics Before the Deep Learning Era
• 3.1 Introduction
• 3.2 PRNU-Based Approach
• 3.2.1 PRNU Estimation
• 3.2.2 Noise Residual Computation
• 3.2.3 Forgery Detection Test
• 3.2.4 Estimation Through Guided Filtering
• 3.3 Blind Methods
• 3.3.1 Noise Patterns
• 3.3.2 Compression Artifacts
• 3.3.3 Editing Artifacts
• 3.4 Learning-Based Methods with Handcrafted Features
• 3.5 Conclusions
• References
• Part II Digital Face Manipulation and Security Implications
• 4 Toward the Creation and Obstruction of DeepFakes
• 4.1 Introduction
• 4.2 Backgrounds
• 4.2.1 DeepFake Video Generation
• 4.2.2 DeepFake Detection Methods
• 4.2.3 Existing DeepFake Datasets
• 4.3 Celeb-DF: the Creation of DeepFakes
• 4.3.1 Synthesis Method
• 4.3.2 Visual Quality
• 4.3.3 Evaluations
• 4.4 Landmark Breaker: the Obstruction of DeepFakes
• 4.4.1 Facial Landmark Extractors
• 4.4.2 Adversarial Perturbations
• 4.4.3 Notation and Formulation
• 4.4.4 Optimization
• 4.4.5 Experimental Settings
• 4.4.6 Results
• 4.4.7 Robustness Analysis
• 4.4.8 Ablation Study.
• 4.5 Conclusion
• References
• 5 The Threat of Deepfakes to Computer and Human Visions
• 5.1 Introduction
• 5.2 Related Work
• 5.3 Databases and Methods
• 5.3.1 DeepfakeTIMIT
• 5.3.2 DF-Mobio
• 5.3.3 Google and Jigsaw
• 5.3.4 Facebook
• 5.3.5 Celeb-DF
• 5.4 Evaluation Protocols
• 5.4.1 Measuring Vulnerability
• 5.4.2 Measuring Detection
• 5.5 Vulnerability of Face Recognition
• 5.6 Subjective Assessment of Human Vision
• 5.6.1 Subjective Evaluation Results
• 5.7 Evaluation of Deepfake Detection Algorithms
• 5.8 Conclusion
• References
• 6 Morph Creation and Vulnerability of Face Recognition Systems to Morphing
• 6.1 Introduction
• 6.2 Face Morphing Generation
• 6.2.1 Landmark Based Morphing
• 6.2.2 Deep Learning-Based Face Morph Generation
• 6.3 Vulnerability of Face Recognition Systems to Face Morphing
• 6.3.1 Data Sets
• 6.3.2 Results
• 6.3.3 Deep Learning-Based Morphing Results
• 6.4 Conclusions
• References
• 7 Adversarial Attacks on Face Recognition Systems
• 7.1 Introduction
• 7.2 Taxonomy of Attacks on FRS
• 7.2.1 Threat Model
• 7.3 Poisoning Attacks on FRS
• 7.3.1 Fast Gradient Sign Method
• 7.3.2 Projected Gradient Descent
• 7.4 Carlini and Wagner (CW) Attacks
• 7.5 ArcFace FRS Model
• 7.6 Experiments and Analysis
• 7.6.1 Clean Dataset
• 7.6.2 Attack Dataset
• 7.6.3 FRS Model for Baseline Verification
• 7.6.4 FRS Baseline Performance Evaluation
• 7.6.5 FRS Performance on Probe Data Poisoning
• 7.6.6 FRS Performance on Enrolment Data Poisoning
• 7.7 Impact of Adversarial Training with FGSM Attacks
• 7.8 Discussion
• 7.9 Conclusions and Future Directions
• References
• 8 Talking Faces: Audio-to-Video Face Generation
• 8.1 Introduction
• 8.2 Related Work
• 8.2.1 Audio Representation
• 8.2.2 Face Modeling
• 8.2.3 Audio-to-Face Animation
• 8.2.4 Post-processing
• 8.3 Datasets and Metrics.
• 8.3.1 Dataset
• 8.3.2 Metrics
• 8.4 Discussion
• 8.4.1 Fine-Grained Facial Control
• 8.4.2 Generalization
• 8.5 Conclusion
• 8.6 Further Reading
• References
• Part III Digital Face Manipulation Detection
• 9 Detection of AI-Generated Synthetic Faces
• 9.1 Introduction
• 9.2 AI Face Generation
• 9.3 GAN Fingerprints
• 9.4 Detection Methods in the Spatial Domain
• 9.4.1 Handcrafted Features
• 9.4.2 Data-Driven Features
• 9.5 Detection Methods in the Frequency Domain
• 9.6 Learning Features that Generalize
• 9.7 Generalization Analysis
• 9.8 Robustness Analysis
• 9.9 Further Analyses on GAN Detection
• 9.10 Open Challenges
• References
• 10 3D CNN Architectures and Attention Mechanisms for Deepfake Detection
• 10.1 Introduction
• 10.2 Related Work
• 10.2.1 Deepfake Detection
• 10.2.2 Attention Mechanisms
• 10.3 Dataset
• 10.4 Algorithms
• 10.5 Experiments
• 10.5.1 All Manipulation Techniques
• 10.5.2 Single Manipulation Techniques
• 10.5.3 Cross-Manipulation Techniques
• 10.5.4 Effect of Attention in 3D ResNets
• 10.5.5 Visualization of Pertinent Features in Deepfake Detection
• 10.6 Conclusions
• References
• 11 Deepfake Detection Using Multiple Data Modalities
• 11.1 Introduction
• 11.2 Deepfake Detection via Video Spatiotemporal Features
• 11.2.1 Overview
• 11.2.2 Model Component
• 11.2.3 Training Details
• 11.2.4 Boosting Network
• 11.2.5 Test Time Augmentation
• 11.2.6 Result Analysis
• 11.3 Deepfake Detection via Audio Spectrogram Analysis
• 11.3.1 Overview
• 11.3.2 Dataset
• 11.3.3 Spectrogram Generation
• 11.3.4 Convolutional Neural Network (CNN)
• 11.3.5 Experimental Results
• 11.4 Deepfake Detection via Audio-Video Inconsistency Analysis
• 11.4.1 Finding Audio-Video Inconsistency via Phoneme-Viseme Mismatching
• 11.4.2 Deepfake Detection Using Affective Cues
• 11.5 Conclusion.
• References
• 12 DeepFakes Detection Based on Heart Rate Estimation: Single- and Multi-frame
• 12.1 Introduction
• 12.2 Related Works
• 12.3 DeepFakesON-Phys
• 12.4 Databases
• 12.4.1 Celeb-DF v2 Database
• 12.4.2 DFDC Preview
• 12.5 Experimental Protocol
• 12.6 Fake Detection Results: DeepFakesON-Phys
• 12.6.1 DeepFakes Detection at Frame Level
• 12.6.2 DeepFakes Detection at Short-Term Video Level
• 12.7 Conclusions
• References
• 13 Capsule-Forensics Networks for Deepfake Detection
• 13.1 Introduction
• 13.2 Related Work
• 13.2.1 Deepfake Generation
• 13.2.2 Deepfake Detection
• 13.2.3 Challenges in Deepfake Detection
• 13.2.4 Capsule Networks
• 13.3 Capsule-Forensics
• 13.3.1 Why Capsule-Forensics?
• 13.3.2 Overview
• 13.3.3 Architecture
• 13.3.4 Dynamic Routing Algorithm
• 13.3.5 Visualization
• 13.4 Evaluation
• 13.4.1 Datasets
• 13.4.2 Metrics
• 13.4.3 Effect of Improvements
• 13.4.4 Feature Extractor Comparison
• 13.4.5 Effect of Statistical Pooling Layers
• 13.4.6 Capsule-Forensics Network Versus CNNs: Seen Attacks
• 13.4.7 Capsule-Forensics Network Versus CNNs: Unseen Attacks
• 13.5 Conclusion and Future Work
• 13.6 Appendix
• References
• 14 DeepFakes Detection: the DeeperForensics Dataset and Challenge
• 14.1 Introduction
• 14.2 Related Work
• 14.2.1 DeepFakes Generation Methods
• 14.2.2 DeepFakes Detection Methods
• 14.2.3 DeepFakes Detection Datasets
• 14.2.4 DeepFakes Detection Benchmarks
• 14.3 DeeperForensics-1.0 Dataset
• 14.3.1 Data Collection
• 14.3.2 DeepFake Variational Auto-Encoder
• 14.3.3 Scale and Diversity
• 14.3.4 Hidden Test Set
• 14.4 DeeperForensics Challenge 2020
• 14.4.1 Platform
• 14.4.2 Challenge Dataset
• 14.4.3 Evaluation Metric
• 14.4.4 Timeline
• 14.4.5 Results and Solutions
• 14.5 Discussion
• 14.6 Further Reading
• References.
• 15 Face Morphing Attack Detection Methods
• 15.1 Introduction
• 15.2 Related Works
• 15.3 Morphing Attack Detection Pipeline
• 15.3.1 Data Preparation and Feature Extraction
• 15.3.2 Feature Preparation and Classifier Training
• 15.4 Database
• 15.4.1 Image Morphing
• 15.4.2 Image Post-Processing
• 15.5 Morphing Attack Detection Methods
• 15.5.1 Pre-Processing
• 15.5.2 Feature Extraction
• 15.5.3 Classification
• 15.6 Experiments
• 15.6.1 Generalisability
• 15.6.2 Detection Performance
• 15.6.3 Post-Processing
• 15.7 Summary
• References
• 16 Practical Evaluation of Face Morphing Attack Detection Methods
• 16.1 Introduction
• 16.2 Related Work
• 16.3 Creation of Morphing Datasets
• 16.3.1 Creating Morphs
• 16.3.2 Datasets
• 16.4 Texture-Based Face Morphing Attack Detection
• 16.5 Morphing Disguising
• 16.6 Experiments and Results
• 16.6.1 Within Dataset Performance
• 16.6.2 Cross Dataset Performance
• 16.6.3 Mixed Dataset Performance
• 16.6.4 Robustness Against Additive Gaussian Noise
• 16.6.5 Robustness Against Scaling
• 16.6.6 Selection of Similar Subjects
• 16.7 The SOTAMD Benchmark
• 16.8 Conclusion
• References
• 17 Facial Retouching and Alteration Detection
• 17.1 Introduction
• 17.2 Retouching and Alteration Detection-Review
• 17.2.1 Digital Retouching Detection
• 17.2.2 Digital Alteration Detection
• 17.2.3 Publicly Available Databases
• 17.3 Experimental Evaluation and Observations
• 17.3.1 Cross-Domain Alteration Detection
• 17.3.2 Cross Manipulation Alteration Detection
• 17.3.3 Cross Ethnicity Alteration Detection
• 17.4 Open Challenges
• 17.5 Conclusion
• References
• Part IV Further Topics, Trends, and Challenges
• 18 Detecting Soft-Biometric Privacy Enhancement
• 18.1 Introduction
• 18.2 Background and Related Work
• 18.2.1 Problem Formulation and Existing Solutions.
• 18.2.2 Soft-Biometric Privacy Models.