Deepfake (DF) is a kind of forged image or video that is developed to spread misinformation and facilitate vulnerabilities to privacy hacking and truth masking with advanced technologies, including deep learning and artificial intelligence with trained algorithms. This kind of multimedia manipulation, such as changing facial expressions or speech, can be used for a variety of purposes to spread misinformation or exploitation. This kind of multimedia manipulation, such as changing facial expressions or speech, can be used for a variety of purposes to spread misinformation or exploitation. With the recent advancement of generative adversarial networks (GANs) in deep learning models, DF has become an essential part of social media. To detect forged video and images, numerous methods have been developed, and those methods are focused on a particular domain and obsolete in the case of new attacks/threats. Hence, a novel method needs to be developed to tackle new attacks. The method introduced in this article can detect various types of spoofs of images and videos that are computationally generated using deep learning models, such as variants of long short-term memory and convolutional neural networks. The first phase of this proposed work extracts the feature frames from the forged video/image using a sparse autoencoder with a graph long short-term memory (SAE-GLSTM) method at training time. The first phase of this proposed work extracts the feature frames from the forged video/image using a sparse autoencoder with a graph long short-term memory (SAE-GLSTM) method at training time. The proposed DF detection model is tested using the FFHQ database, 100K-Faces, Celeb-DF (V2) and WildDeepfake. The evaluated results show the effectiveness of the proposed method.

Department of Applied Cybernetics Faculty of Science University of Hradec Králové Czech Republic

Department of Mathematics University of Hradec Králové Hradec Králové Czech Republic

See more in PubMed

Afchar D, Nozick V, Yamagishi J, Echizen I. MesoNet: a compact facial video forgery detection network. 2018 IEEE International Workshop on Information Forensics and Security (WIFS); Piscataway: IEEE; 2018. pp. 1–7.

Agarwal S, Farid H, Fried O, Agrawala M. Detecting deep-fake videos from phoneme-viseme mismatches. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops; Piscataway: IEEE; 2020. pp. 660–661.

Chesney R, Citron D. Deepfakes and the new disinformation war: the coming age of post-truth geopolitics. Foreign Affairs. 2019;98(1):147–155.

Chintha A, Thai B, Sohrawardi SJ, Bhatt K, Hickerson A, Wright M, Ptucha R. Recurrent convolutional structures for audio spoof and video deepfake detection. IEEE Journal of Selected Topics in Signal Processing. 2020;14(5):1024–1037. doi: 10.1109/JSTSP.2020.2999185. DOI

Fernandes S, Raj S, Ewetz R, Pannu JS, Jha SK, Ortiz E, Vintila I, Salter M. Detecting deepfake videos using attribution-based confidence metric. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops; Piscataway: IEEE; 2020. pp. 308–309.

Güera D, Delp EJ. Deepfake video detection using recurrent neural networks. 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS); Piscataway: IEEE; 2018. pp. 1–6.

Hoq M, Uddin MN, Park S-B. Vocal feature extraction-based artificial intelligent model for Parkinson’s disease detection. Diagnostics. 2021;11(6):1076. doi: 10.3390/diagnostics11061076. PubMed DOI PMC

Howcroft E. How faking videos became easy and why that’s so scary. New York, NY, USA: Bloomberg; 2018.

Hsu C-C, Zhuang Y-X, Lee C-Y. Deep fake image detection based on pairwise learning. Applied Sciences. 2020;10(1):370. doi: 10.3390/app10010370. DOI

Kang M, Ji K, Leng X, Xing X, Zou H. Synthetic aperture radar target recognition with feature fusion based on a stacked autoencoder. Sensors. 2017;17(1):192. doi: 10.3390/s17010192. PubMed DOI PMC

Kaur S, Kumar P, Kumaraguru P. Deepfakes: temporal sequential analysis to detect face-swapped video clips using convolutional long short-term memory. Journal of Electronic Imaging. 2020;29(3):33013. doi: 10.1117/1.JEI.29.3.033013. DOI

Kipf TN, Welling M. Semi-supervised classification with graph convolutional networks. Proceedings of the 5th International Conference on Learning Representations, ICLR ’17; 2017. pp. 1–14.

Korshunova I, Shi W, Dambre J, Theis L. Fast face-swap using convolutional neural networks. Proceedings of the IEEE International Conference on Computer Vision; Piscataway: IEEE; 2017. pp. 3677–3685.

Kwok AOJ, Koh SGM. Deepfake: a social construction of technology perspective. Current Issues in Tourism. 2021;24(13):1798–1802. doi: 10.1080/13683500.2020.1738357. DOI

Leng J, Jiang P. A deep learning approach for relationship extraction from interaction context in social manufacturing paradigm. Knowledge-Based Systems. 2016;100(12):188–199. doi: 10.1016/j.knosys.2016.03.008. DOI

Li Y, Chang M-C, Lyu S. In ictu oculi: exposing AI created fake videos by detecting eye blinking. 2018 IEEE International Workshop on Information Forensics and Security (WIFS); Piscataway: IEEE; 2018. pp. 1–7.

Li Y, Lyu S. Exposing deepfake videos by detecting face warping artifacts. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); Piscataway: IEEE; 2019.

Nataraj L, Mohammed TM, Manjunath B, Chandrasekaran S, Flenner A, Bappy MJH, Roy-Chowdhury A. Detecting GAN generated fake images using co-occurrence matrices. Electronic Imaging. 2019;2019(5):532-1–532-7. doi: 10.2352/ISSN.2470-1173.2019.5.MWSF-532. DOI

Ng A. Sparse autoencoder. CS294A Lecture Notes. 2011;72(2011):1–19.

Nguyen HH, Yamagishi J, Echizen I. Capsule-forensics: using capsule networks to detect forged images and videos. ICASSP, 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); Piscataway: IEEE; 2019. pp. 2307–2311.

Olshausen BA, Field DJ. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature. 1996;381(6583):607–609. doi: 10.1038/381607a0. PubMed DOI

Sabir E, Cheng J, Jaiswal A, AbdAlmageed W, Masi I, Natarajan P. Recurrent convolutional strategies for face manipulation detection in videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops; Piscataway: IEEE; 2019. pp. 80–87.

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C. MobileNetV2: inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Piscataway: IEEE; 2018. pp. 4510–4520.

Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G. The graph neural network model. IEEE Transactions on Neural Networks. 2008;20(1):61–80. doi: 10.1109/TNN.2008.2005605. PubMed DOI

Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015; San Diego, CA, USA: 2015. May 7-9, 2015, Conference Track Proceedings.

Soare SR, Burton J. Cyber Threats and NATO 2030: Horizon Scanning and Analysis. Tallinn, Estonia: NATO CCDCOE Publications; 2020. Smart cities, cyber warfare and social disorder; pp. 108–124.

Soni H, Sankhe D. Image restoration using adaptive median filtering. International Research Journal of Engineering IT & Scientific Research. 2019;6(10):841–844.

Wang S-Y, Wang O, Zhang R, Owens A, Efros AA. CNN-generated images are surprisingly easy to spot… for now. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Piscataway: IEEE; 2020. pp. 8695–8704.

Westerlund M. The emergence of deepfake technology: a review. Technology Innovation Management Review. 2019;9(11):39–52. doi: 10.22215/timreview/1282. DOI

Xuan X, Peng B, Wang W, Dong J. On the generalization of GAN image forensics. In: Sun Z, He R, Feng J, Shan S, Guo Z, editors. Biometric Recognition. CCBR 2019. Lecture Notes in Computer Science. Vol. 11818. Cham: Springer; 2019. pp. 134–141. DOI

Yang X, Li Y, Lyu S. Exposing deep fakes using inconsistent head poses. ICASSP, 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); Piscataway: IEEE; 2019. pp. 8261–8265.

Zagoruyko S, Komodakis N. Wide residual networks. ArXiv preprint. 2016. DOI

Zhang K, Zhang Z, Li Z, Qiao Y. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters. 2016;23(10):1499–1503. doi: 10.1109/LSP.2016.2603342. DOI

Zhou P, Han X, Morariu VI, Davis LS. Two-stream neural networks for tampered face detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); Piscataway: IEEE; 2017. pp. 1831–1839.

Zhou J, Xiang J, Huang S. Classification and prediction of typhoon levels by satellite cloud pictures through GC-LSTM deep learning model. Sensors. 2020;20(18):5132. doi: 10.3390/s20185132. PubMed DOI PMC

Zi B, Chang M, Chen J, Ma X, Jiang Y-G. WildDeepfake: a challenging real-world dataset for deepfake detection. Proceedings of the 28th ACM International Conference on Multimedia; New York: ACM; 2020. pp. 2382–2390.