About Me

i) We demonstrate, for the first time, the capability to reconstruct volumetric RI of semi-transparent biological samples from diffracted fluorescence images with limited angles and sparse views, validated through both simulations and real-world experiments. This work opens up a new avenue in diffraction-informed neural volumetric representations.

ii) We demonstrate the effectiveness and efficiency of NVP in optical tomography, reducing the required number of images by nearly 50-fold and processing time by 3-fold compared to previous methods in our demonstrated experiments.

iii) We leverage the physical prior of light diffraction to achieve physically accurate rendering and quantitative RI reconstruction of volumetric objects, overcoming the limitations of ray-optics models at microscale imaging and broadening the applicability of neural fields in microscopic imaging.

Simultaneous imaging of fluorescence-labeled and label-free phase objects in the same sample can provide distinct and complementary information. Most multimodal fluorescence-phase imaging operates in transmission mode, capturing fluorescence images and phase images separately or sequentially, which limits their practical use for in vivo applications. Alternatively, simultaneous fluorescence-phase imaging in reflection mode, which captures diffracted fluorescence and then reconstructs phase information from fluorescence images, has been demonstrated with fluorescent beads and label-free single-layer cells. However, reconstructing the 3D refractive index (RI) of thick samples from fluorescence images over a large volume and at high resolution remains challenging. To tackle this challenge, we develop fluorescence diffraction tomography (FDT) with explicit neural fields to reconstruct the 3D RI of thick samples from diffracted fluorescence images captured on a defocused image plane. The successful reconstruction of 3D RI using FDT relies on four key components: coarse-to-fine structure, self-calibration, a differential multi-slice rendering model, and partial coherent masks. The explicit representation integrates with coarse-to-fine structure for high-speed, high-resolution reconstruction, while the differential multi-slice rendering model enables self-calibration of system parameters, ensuring accurate forward image prediction and RI reconstruction. Partial coherent masks efficiently resolve discrepancies between the coherent light model and partial coherent light data. FDT successfully reconstructed the RI of 3D cultured label-free 3D MuSCs tube in a 530×530×300 µm³ volume at 1024×1024 pixels across 24 z-layers from fluorescence images, demonstrating high fidelity 3D RI reconstruction of bulky and heterogeneous biological samples in vitro.

Two-photon microscopy is a powerful tool for in vivo imaging, but tissue scattering typically limits its depth to a few hundred microns, and most active scattering-correction methods are restricted to a small region by the optical memory effect. C-FOCUS is an active scattering-correction approach that integrates Fourier-domain intensity modulation with compressive sensing.

Using C-FOCUS, we demonstrate high-resolution imaging of YFP-labeled neurons and FITC-labeled blood vessels at depths exceeding 900 µm in the intact mouse brain in vivo, and transcranial imaging of YFP-labeled dendritic structures through the intact adult mouse skull.

DeepFocus combines deep learning with Fourier-domain intensity coupling to rapidly optimize illumination patterns and suppress multiple scattering, achieving ~1 mm-deep, high-contrast imaging in the mouse brain. Compared with uncorrected two-photon microscopy at the same average power, it delivers ~20-fold higher signal-to-noise ratio.

The purely intensity-based strategy avoids complex phase-modulation hardware, enabling millisecond pattern updates with a DMD — opening a practical route to longitudinal studies of deep neural circuits.

The light field camera has significantly advanced conventional imaging methods and microscopy over the past decades, providing high-dimensional information in 2D images and enabling a variety of applications. However, inherent shortcomings persist, mainly due to the complex optical setup and the trade-off between resolution.

In this work, we propose a Neural Defocus Light Field (NDLF) rendering method, which constructs the light field without a micro-lens array but achieves the same resolution as the original image. The basic unit of NDLF is the 3D point spread function (3D-PSF), which extends the 2D-PSF by incorporating the focus depth axis. NDLF can directly solve the distribution of PSFs in 3D space, enabling direct manipulation of the PSF in 3D and enhancing our understanding of the defocus process. NDLF achieves the focused images rendering by redefining the focus images as slices of the NDLF, which are superpositions of cross-sections of the 3D-PSFs.

NDLF modulates the 3D-PSFs using three multilayer perceptron modules, corresponding to three Gaussian-based models from coarse to fine. NDLF is trained on 20 high-resolution (1024 × 1024) images at different focus depths, enabling it to render focused images at any given focus depth. The structural similarity index between the predicted and measured focused images is 0.9794. Moreover, we developed a hardware system to collect the high resolution focused images with corresponding focus depth, and depth maps.

NDLF achieves high-resolution light field imaging with a single-lens camera and also resolves the distribution of 3D-PSFs in 3D space, paving the way for novel light-field synthesis techniques and deeper insights into defocus blur.

Motivation: Explored the inverse process of the Point Spread Function (PSF) to decode the defocused and depth maps from focused images

We propose a multi-task learning network for depth and defocus estimation, which can efficiently unite the depth and defocus map estimation.

We set up the ALL-in-3D dataset which is the firtst all real image dataset consisting of all-in-focus image, focused image with focus depth, depth map and defocus map. The ALL-in-3D dataset is of high resolution and precision, and contains 100K sets of images.

We explore the implied defocus and the depth information in the focused images with the All-in-3D dataset and the new network structure. The experiment results demonstrate that the depth and defocus map promote each other by the multi-task architecture.

100,000 sets of the high-resolution all-in-focus images, focused images, depth maps and defocus maps with the size of 2452 × 2056 are provided.

The depth map in our dataset do not require extra alignment or interpolation with the RGB images, since the depth is solved by pixel. Compared to the depth maps obtained by laser, the depth maps in our dataset are dense and have the same FOV as the RGB images.

It provides pixel-level annotated defoucs maps where the defocus is calculated by the CoC size. Some datasets just provide the binary defocus map labeled by subjective perception or clearness analysis, and others use the simulated focus depth to calculate the CoC size of the defocus maps.

More general. The dataset can be applied in SFF/SFDF and optical deblur domain.

Motivation: Investigated the process of generating focused image and proposed a new method to determine PSF, from the perspective of computational photography

Developed a precise mathematical model of the camera’s point spread function to describe the defocus process and solved the parameters which cannot be solved by the optimization algorithm, including one optical composite parameter and one mechanical parameter

Built a hardware system consisting of a focusing system and a structured light system to acquire the all-in-focus image, the focused image with corresponding focus depth, and the depth map in the same view

Designed experiments on both standard planes with synthetic patterns and actual objects to solve and evaluate the efficiency and effectiveness of the proposed algorithm. Implemented the model using the open-source PyTorch framework and the CUDA toolkits with an Nvidia Titan RTX 24G

Optimized focused image generation algorithms. Utilized CUDA toolkit instead of SCIPY tool library on CPU to synthesize the focused images. Improved image processing speed from 5s to 1ms

Designed a novel metric based on the defocus histogram to evaluate the difference between the simulated focused image and the actual focused image to obtain 40% higher accuracy than the previous blurred image generation method

Course Learning: (1) Learned the structured light depth estimation method to find the depth of an object by projecting a stripe onto the object and then photographing it with a camera. (2) Studied the shape from focus method to solve for object depth by using defocus information. Built the hardware system for the method and wrote the corresponding code

Hardware System Building: (1) Built focus system with Arduino, stepper motor, industrial infrared sensor, industrial camera. (2) Used Arduino to control the motor to drive the camera and trigger the camera to take pictures. (3) Utilized Arduino to connect with IR sensor through TTL to RS485 adapter to trigger sensor measurement and get measured data. (4) Deployed the camera to connect to the computer via RJ-45 to transfer the image data. (5) Implemented Arduino to connect to computer via USB to transfer sensor measurement data and obtained distance data on computer

Software Building: (1) Built a development environment for Arduino using C/C++. (2) Used Python and OpenCV to process the captured images and focal length. (3) Proposed a novel algorithm to improve the Shape from Focus accuracy using gradient of focus measure curve