Written by | Zheng Shanshan and Zhang Xiangyu
Description | This article is submitted by the author (research group).
With the rapid development of mathematical optimization and computing hardware, deep neural network（Deep Neural Networks, DNN）(noun explanation >) has become a powerful tool to solve many challenging problems in various fields, including decision-making, computational imaging, holographic technology and so on.
Since its development in 1947, holography（holography）(noun explanation >) has become an important and extensive technology, and has applications in various fields of optical engineering, including optical imaging, microscopy, metrology, three-dimensional (3D) display and so on. Holography means containing all information of light waves, which is physically divided into two processes: recording and reconstruction of wavefront, both of which can be realized by optical or digital methods.
At present, the two different fields of holography and deep learning have merged with each other, forming a new interdisciplinary field-depth holography.
Recently, from Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of SciencesSitu guohaiResearcher with Deep holography For the topic of Light: Advanced Manufacturing In this paper, a review article was published, and the progress of depth holography was comprehensively reviewed.holographic(holography)andDeep neural network(DNN)The mutual promotion relationship between them.
Dnn has been proved in holographic reconstruction and computer generated holography.（computer-generated Holography, CGH）Almost all aspects of the application;
● Holography has the ability of interconnection and parallel processing of light speed, which is an optical realization of DNN, that is, optical neural network.（optical neural networks, ONN）(noun explanation >).
In this paper, the basic theory and architecture of DNN are briefly introduced, then some important progress and research trends of depth holography are discussed, and finally its future prospect is prospected. I hope this review can promote the further development of this field.
Deep Neural Networks: A Concise Introduction
The conceptual framework of DNN is shown in Figure 1. Feedforward neural network, or multilayer perceptron.（MLP）There is an input layer, an output layer, and one or more hidden layers. Each layer can have a different number of neurons, called perceptron; And each unit receives the weighted sum of its inputs from the upper layer.(Except for the first layer, its input is the original image to be processed by NN). The goal of feedforward neural network is to optimize an NN model to approximate a continuous function.trainAft learning that network parameter,The mapping from the input space x to the output space y can be realized.
Figure 1. Conceptual architecture of DNN (Source: Light: Advanced Manufacturing)
Compared with the traditional holographic technology based on physics, DNN method has been proved to have unique advantages. As long as there is enough data, DNN can learn features from the higher-level structure composed of lower-level features, but it can’t clearly express the exact physics of the system, which leads to great challenges to the generalization and interpretability of the method.
Therefore, we very much hope that we canGive full play to the advantages of DNN and physical model at the same time, develop DNN based on physical information.（Physics-informed DNN）.
● Physical integration（physical fusion）, or a single physical DNN.
It directly inputs the solution of physical model as DNN model.(part).
● Residual physics（residual physics）, or recursive physical information DNN
Add physical solutions to the output of DNN, so that DNN model only needs to learn the mismatch between model-based solutions and the real situation.
● Regularized physics（physical regularization）, or concatenated physical information DNN.
The regularization term in a set of physical constraints is used to punish the network solution. The regularization term can be used as part of the loss function or through the physical reconstruction process.
● Embedded physics（embedded physics）DNN
As shown in Figure 2, the central idea is to bring the physical model into the network optimization cycle.
Figure 2. Typical architecture of DNN embedded in physics (Source: Light: Advanced Manufacturing)
● Network approximate physics
As the name implies, the physical model is approximated by using DNN.
Holographic reconstruction
Hologram can be regarded as the superposition of object light and reference beam. The intuitive method of holographic reconstruction is based on the physical model of diffraction, that is, the numerical calculation of wave diffraction process. Off-axis holography can be solved by spatial filtering, but spatial filtering will bring the loss of high-frequency components, which greatly hinders the quality of reconstructed images. For coaxial holography, the removal of zero order is relatively simple, so most of the research on coaxial holography reconstruction is to deal with twins.
Physically, twin images are actually out-of-focus copies of reconstructed object images, which can be eliminated by subtracting out-of-focus images. At present, the most widely used strategy is to adjust some physical parameters of the optical system and obtain the corresponding holograms, so as to establish a small linear equation set, and to link the recorded holograms with the tuning parameters, and then solve the object wavefront.
Mathematically, the twin image artifact is caused by the lack of phase when the hologram is recorded. This shows that if the missing phase of the hologram can be solved, the twin image artifacts can be solved.
Holography inspired by DNN
1) The most direct method: "end to end" DNN
EHoloNet developed by ResNet and U-Net: receiving the original digital hologram as input, and generating the phase object wavefront without artifacts;
Y-network architecture: simultaneously reconstruct intensity and phase from a single digital hologram;
It is also possible to generate a countermeasure network（GAN）Implementation, training data does not need to be paired.
2) Physical fusion, or DNN of single physics.
The traditional numerical free space is used to propagate back to the target plane, and then the amplitude and phase of the reconstructed wavefront are sent to DNN respectively, and all these artifacts are removed by training.
3) physically embedded DNN
As shown in Figure 3, this strategy has two obvious advantages by incorporating the physical imaging model into the traditional DNN:First of all, it does not need any data for pre-training; Secondly, the reconstructed image satisfies the constraints imposed by the physical model, which is interpretable.
DNN model in PhysenNet can be replaced by other neural networks, depending on the task at hand. For example, researchers have successfully incorporated the phase imaging model into the GAN network.
Figure 3. Schematic diagram of DNN incorporating physical model: phase imaging (Source: Light: Science & Applications, 9: 77 (2020).)
In addition, DNN method can also be used to solve various problems related to holography.Such as phase unwrapping, autofocus, phase distortion compensation, speckle suppression, computer generated holography (CGH).
DNN inspired by holographic technology
Holographic technology is to realize optical neural network.（ONN）One of the important ways.
The early research includes the optical realization of fully connected neural network and Hopfield model, which is a cyclic neural network.（RNN）The foundation of. Holographic neural network does not use logical neurons like digital neural network, but relies on the interconnection of hologram itself. In a fully connected holographic neural network, the weights are stored in the pixels of the hologram.(neuron)Medium. a layer or deck(hologram)Each neuron simply modulates the light shining on it from the upstream layer and then illuminates the downstream layer. Holographic neural network can be realized by photorefractive crystal, which is a three-dimensional device in itself and may store billions of weight data. Therefore, it is hopeful to solve large-scale reverse problems in principle.
Figure 4. Schematic diagram of diffraction depth neural network
(a) DNN contains a plurality of transmission (or reflection) layers, in which each point on the layers acts as a neuron with a complex transmission (or reflection) coefficient, and the transmission or reflection coefficient of each layer can be trained using deep learning, thereby performing a function between the input and output planes of the network. After the learning stage, the design of D2NN is fixed; Once it is manufactured or printed in 3D, it can perform the learned functions at the speed of light. Different types of diffraction depth neural networks: (b) classification and (c) imaging. (Source: Science,361(2018))
The modern realization of holographic neural network takes advantage of diffraction, so it is named diffraction depth neural network.（D2NN）, as shown in Figure 4. In hardware implementation, the hologram in D2NN is actually a diffractive optical element.（DOE）It can be realized by 3D printing, nanotechnology, or electrically addressable digital micromirror devices.（DMD）Get. Like many other optical neural networks, the original D2NN was trained offline. In order to implement better training strategies, researchers have made a lot of efforts, such as back propagation technology to update unit weights through gradient transformation from Euclidean space to Riemannian space, and methods to realize optical grouping. People have also studied the applications of D2NN in logic operation, optical information processing, holographic reconstruction, pulse shaping, single pixel machine vision of spectral coding and so on.
Summary and prospect
The integration of holography and neurocomputing has greatly promoted each other’s development, and also provided us with a wonderful field to explore.
For holography inspired by DNN, this paper discusses five different ways to incorporate physical information, and shows their applications in solving various problems related to holography.
● In addition to trying different DNN architectures,An important trend is to incorporate physical models into DNN models.. This idea has been paid close attention by researchers in different fields.
● PhysenNet embedded in physical model has obvious advantages, but its optimization speed is slow.For lightweight network structure and more effective training algorithmMade a request.
● Comprehensive utilization of the advantages of physical prior and traditional data-driven methodsIt can make the training more effective, improve generalization and increase interpretability.
For DNN inspired by holographic technology, most of the research published so far focuses on optical reasoning. The parallel processing ability of the holographic neural network at the speed of light does guarantee powerful reasoning ability, even surpassing the performance of Tesla V100 tensor core GPU, the top product of Nvidia, in some tasks.
In order to give full play to the advantages of optics, it is necessary toOn-line training of network by optical means.There are many possibilities to effectively implement most basic functions in DNN.
● Relying on the development of new optical materials and equipmentThe wave can be modulated at high speed at sub-wavelength scale to improve the performance of D2NN.
● The rapid development of neural computing provides us with more powerful algorithms., such as spike neural synaptic network (SNN), can simulate the behavior and learning potential of the brain. The applicability and potential of these new algorithms in holographic technology need to be tapped.
Thesis information
Situ. Light: Advanced Manufacturing (2022)3:13
http://doi.org/10.37188/lam.2022.013
Welcome the submission of the research group-press release
Reprinted/cooperated/contributed by the research group, WeChat:447882024
Take you to read 1 document every day!Join >Light reading club