# Fileset

[Image__process_inspired_by_WT_submit_R1_20230303_FU.docx](https://mdr.nims.go.jp/filesets/297b9710-7f7e-4e17-ae35-e208109edc28/download)

## Creator

[Fumihiko Uesugi](https://orcid.org/0000-0003-3346-4218)

## Rights

[Creative Commons BY-NC-ND Attribution-NonCommercial-NoDerivs 4.0 International](https://creativecommons.org/licenses/by-nc-nd/4.0/)

## Other metadata

[Novel image processing method inspired by wavelet transform](https://mdr.nims.go.jp/datasets/980e3c0a-d578-4bb3-984c-acdf7b7e17bf)

## Fulltext

Novel image processing method inspired by wavelet transform Fumihiko Uesugi* National Institute for Materials Science, 1-2-1 Sengen, Tsukuba, Ibaraki 305-0047, Japan*Corresponding author.AbstractI developed a novel image processing method inspired by wavelet transform that uses a concentric mother wavelet. This method can be used for noise filtering, feature extraction, image enhancement, and other applications and is effective for images with low spatial frequencies. When filtering small-sized images, setting masks for Fourier filtering is often difficult; however, the proposed method does not require mask setting, thus facilitating filtering. In practice, many concentric mother wavelets of different frequencies are prepared in advance, and the cross-correlation between the input image and prepared wavelets is calculated. Reconstruction or image processing can be easily performed by combining the obtained results for each frequency. I believe that the present method can aid image processing related to material analysis.[footnoteRef:1] STEM: scanning electron transmission microscope; FT: Fourier transform; WFT: window-Fourier transformation; WT; Wavelet transform; 2D: two-dimensional; CC: cross-correlation; DM: DigitalMicrograph; FF: Fourier filtering; ADF: annular dark field; EDS: energy dispersive X-ray spectroscopy.Keywords: Wavelet transform, Fourier transform, Image processing 1 IntroductionIn recent years, the amount of data that can be acquired has increased significantly owing to the development of (scanning) transmission electron microscope [(S)TEM] systems, faster computer systems to control (S)TEM systems, and increase in storage capacity. For analyzing the acquired data, automation systems like scripting [1] and statistical methods [2–4] such as principal component analysis and non-negative matrix factorization have been used. Accordingly, software packages have been developed for general-purpose [5] and dedicated STEM systems [6]. Additionally, image processing of the acquired data is important. Image processing [7–9] such as noise filtering, feature extraction, and image and edge enhancement, are some of the most effective options available for the extraction of additional information from (S)TEM data. The Fourier transform [8] (FT) has been traditionally and presently used for the aforementioned processes in electron microscopy. One-dimensional FT is expressed as follows:, (1)where  is the input function, and  is the special frequency. The FT outcomes determine the frequency content and major or minor spectral contributions of specific frequencies. However, it cannot provide information about the origin of these contributions in the input data. Therefore, window-Fourier transformation (WFT) was developed to determine the origin of specific frequencies in the input signal data. In WFT, FT is performed with a moving window function and is expressed as:,  (2)where  is the window position in real space, and  is the window function; in many cases, a Gaussian function is used [Eq. (3)].  (3)Although the window size in WFT is fixed, it can be improved and made adaptable for wavelet transforms (WT) [10]. WT is defined as a transformation that uses a kernel function that can be scaled and translated. The kernel function is referred to as the “mother wavelet” or simply “wavelet.” The transformed data is in the reciprocal and real space for FT and WT, respectively. The WT results indicate the spatial (or temporal) origin of a specific frequency in the input data in real space. Let the input data be  and the mother wavelet be ; wavelet transformation  can then be expressed as:   (4)where parameter  is “scaling” and expresses the degree of localization or expansion, and  is “translation” and expresses position. Consequently, mother wavelets are constructed by adjusting  and . Equation (4) is a one-dimensional expression, but two-dimensional (2D) expressions can also be defined and used on 2D data such as images. Numerous mother wavelets have been proposed in 2D-WT [11–14]. Since most of the wavelets have direction, the results have direction and bias. However, a mother wavelet without direction is more useful for image analysis or processing. Image processing techniques using centrosymmetric wavelets, such as the Mexican hat wavelet [15], have been proposed [16,17]. The image processing method typically use contrast inversion and unnatural intensity modification such as intensity reduction at the edges due to the first minimum of the Mexican hat wavelet.  When this method is applied to circles aligned like a lattice, circles that should not exist may arise between the original circles in the processed image. Accordingly, the wavelet with maximum intensity at the concentric loci, referred to as the "concentric wavelet", is more useful than that with a single maximum at the center locus, since it can extract information regarding elemental arrangement effectively. This study reports image processing using the concentric wavelet and its application outcomes. 2 Materials and methodsThe details of the proposed method are listed below. The notation for cross-correlation (CC) is:.  (5)A comparison of CC with the WT equation shows that the moving direction of the wavelet [or ] is opposite to each other. In this case, the obtained results were considered to be the same if the mother wavelet is centrosymmetric. The present method replaces  in the concentric wavelet and executes the wavelet using CC. The concentric wavelet is expressed by a sine function for frequency and Gauss function for locality and is prepared at the center of an image with the same size as that of the input image. Both functions are dependent on  (the distance from the center () of the concentric wavelet) and are defined as follows.The sine function () is (6)where   is the image size.Additionally, the window function () is: (7)where  is constant.The kth concentric wavelet (Fig. 1) can be expressed as:.  (8)The reconstructed image  is obtained with a suitable choice of  is:  (9)For small l, the full width half maximum of the window function becomes larger, and the number of the concentric circles increases. However, locality is lost and the first peak effect becomes lower.  The wavelets were adjusted to prevent more than 2 positive concentric circles to avoid the locality lost and the first peak effect weakened. The constant l is set to 0.75 for all applications below.   Figure 1. Presented wavelet．(a) Series of wavelets, (b) bird’s eye view of the wavelet, and (c) central intensity profile of the sine function, window function and mother wavelet ().  The flow of this method is shown in Fig. 2. The actual process was conducted on Gatan DigitalMicrograph (DM, above GMS 3.4.x) [18]. CC calculations used the intrinsic DM function. The preparation of concentric wavelets and CC calculations were performed using the DM script[19–22]. The number of concentric wavelets was adjusted depending on the input image size. The "Select & Sum" process in Fig. 2 can be manually performed on DM very easily.Figure 2.　Flow of the proposed method.3 Results and discussionThe experimental results are presented below. 3.1 Graphene TEM imagesFig. 3 shows the results for TEM imaging of graphene. Fig. 3(b) is the raw TEM image and Figs. 3(a) and 3(c) are the processed results obtained via the proposed method and Fourier filtering (FF), respectively. The FF result was obtained by applying the FT (Fig. 3(d)), masking FT [Fig. 3(e)], and inverse FT. In both methods, the image is smooth and its quality is improved. The edges of the stacked graphene contrast around both corners at the lower parts of the original image disappear in the Fourier-transformed image are retained by the proposed method; furthermore, the overall shading is also clearer. Hence, I can conclude that this method is more suitable for noise filtering than FF. Figure 3. Graphene transmission electron microscopy images. (a) Processed image using the proposed method, (b) original image, (c) Fourier-filtered image, (d) Fourier transform (FT) of image of the original image in (b), and the masked image of (d).3.2 Annular dark field (ADF)-STEM image of Pt atoms dispersed in grapheneFig. 4 shows the ADF-STEM image of Pt atoms dispersed in the multilayered graphene. Figs. 4(a) and 4(b) show images acquired before and after applying the proposed method, respectively. In the processed image, the smoothness and visibility of Pt atoms over the entire surface improved. Owing to the high intensities of Pt particles, small contrast differences, such as in the number of graphene layers and stains, are barely noticeable in the unprocessed image. The processed image indicates that the contrast difference between the two is more emphasized, thus making it easier to distinguish them. Conversely, if the processed image is viewed first and a contrast difference identified, the original image will have notable contrast difference as well. Fig. 4(c) indicates the intensity profiles of Pt particles from the region between the arrowheads in Figs. 4(a) and 4(b). The profile from Fig. 4(b) also shows that the profile is smoother than the unprocessed image, and the peak positions of Pt particles are also preserved. Applying image processing using the Mexican Hat wavelet to a point-like contrast, such as that of Pt particles, results in unnatural intensity reduction around the particles. This does not arise in the proposed method. It can be observed that smoothing (or noise processing) has been performed and the characteristics have been preserved in the case of the present method. Hence, the proposed method is effective when applied to dot-like objects. Figure 4. Annular dark field (ADF)-scanning TEM (STEM) image obtained from Pt atoms dispersed in the multilayered graphene. (a) Original and (b) processed images, and (c) the intensity profiles between the blue and red arrowheads.3.3 ADF-STEM image of SrAlSi4N7The following application processes ADF-STEM images of SrAlSi4N7. The compound’s structure is complex and comprises of Sr atoms and a chain-like structure of Al and N, as shown in the inset of Fig. 5. The input image size is 512 × 512. In total, 256 concentric wavelets were prepared and processed. When all slices are used, the reconstructed image is similar to the input image, as shown in Fig. 5(c). Conversely, the chain-like structure of Al and N can be emphasized via the proposed method and some of the concentric wavelets (27th to 62nd). The boundary between the chain-like structure and Sr, which was unclear in the preprocessed image, is now clearly identifiable. Although the individual atoms in the chain-like parts cannot be separated, the same complexity as that of the structural model can be reproduced. The FT result of the input image shows that the information in the input image is concentrated in the center of the Fourier space. Accordingly, extracting the necessary periodic information from the FT outcome is difficult because filtering is sensitive to masking position and size. In this example, the desired features can be easily emphasized by summing the processed results using specific wavelets.Figure 5. ADF-STEM image of SrAlSi4N7 (a) before and (c, d) after processing with the proposed method, and (b) central region of the FT results of the original image (magnified by a factor of two).3.4 Small-sized EDS dataVisibility enhancement of the EDS mapping results from a-SiN is shown in Fig. 6. The mapping data are acquired (size: 64 × 64 pixels) using a small EDS signal. For small-sized images, similar to the case described earlier, feature extraction or filtering using FT is often difficult. Further, visibility can also be enhanced using the proposed method can also be conducted by adding the transformed images. The nitrogen map [Fig. 6(c)] before processing contains only noise and no useful information; however, the nitrogen sites are emphasized after processing [Fig. 6(f)]. After reviewing the processed images, we can recognize that the signal in the original image was strong. In particular, the area indicated by the arrow is a site containing N atoms only and is easily recognizable. As demonstrated, the proposed method is effective for small-sized images.Figure 6. Application results from EDS mapping of a-SiN. (a) ADF-STEM image, (b) raw data of Si map, (c) raw data of N map, (d) color map of the processed data (Si-red, N-green), (e) processed Si map, and (f) processed N map. Arrowheads show the locations of N atoms.4 ConclusionAn overview and examples of WT-inspired image processing techniques were presented. Noise processing, feature extraction, image enhancement, and other applications can be performed using the concentric wavelet. As the proposed method calculates a CC function between an input image data and a localized periodic function with a window function, it is different from using exact WT; however, the concept is the same as it extracts the basis contribution at a point by moving the basis localized by the window function and calculating to input data. Contrast inversion and unnatural artefacts due to the Mexican Hat wavelet did not arise when the concentric wavelet was used. Hence, the concentric wavelet is more suitable for image processing than the wavelet with single maximum at the center locus, such as the Mexican hat wavelet. A processed image can often help us identify image features that were difficult to observed in the unprocessed image. In fact, it is possible that the features that appear to be created by processing are actually intrinsic features. I believe that the concentric wavelet method proposed herein can be utilized for many microscopic observation scenarios because it can be used in situations where it was difficult to use conventionally.Author informationAffiliationsThis study was fully supported by the National Institute for Materials Science, 1-2-1 Sengen, Tsukuba, Ibaraki 305-0044, Japan.Contributions The concept of this study was initially developed by Fumihiko Uesugi. Competing interests The author declares no competing financial and non-financial interests.Corresponding authors All correspondence must be directed to Fumihiko Uesugi.References[1] F. Uesugi, A. Hokazono, S. Takeno, Ultramicroscopy Evaluation of two-dimensional strain distribution by STEM / NBD, Ultramicroscopy. 111 (2011) 995–998. https://doi.org/10.1016/j.ultramic.2011.01.035.[2] M.G. Burke, M. Watanabe, D.B. Williams, J.M. Hyde, Quantitative characterization of nanoprecipitates in irradiated low-alloy steels: Advances in the application of FEG-STEM quantitative microanalysis to real materials, in: J Mater Sci, 2006: pp. 4512–4522. https://doi.org/10.1007/s10853-006-0084-x.[3] M. Shiga, K. Tatsumi, S. Muto, K. Tsuda, Y. Yamamoto, Ultramicroscopy Sparse modeling of EELS and EDX spectral imaging data by nonnegative matrix factorization, Ultramicroscopy. 170 (2016) 43–59. https://doi.org/10.1016/j.ultramic.2016.08.006.[4] F. Uesugi, S. Koshiya, J. Kikkawa, T. Nagai, K. Mitsuishi, K. Kimoto, Non-negative matrix factorization for mining big data obtained using four-dimensional scanning transmission electron microscopy, Ultramicroscopy. 221 (2021). https://doi.org/10.1016/j.ultramic.2020.113168.[5] F. Pedregosa FABIANPEDREGOSA, V. Michel, O. Grisel OLIVIERGRISEL, M. Blondel, P. Prettenhofer, R. Weiss, J. Vanderplas, D. Cournapeau, F. Pedregosa, G. Varoquaux, A. Gramfort, B. Thirion, O. Grisel, V. Dubourg, A. Passos, M. Brucher, M. Perrot andÉdouardand, andÉdouard Duchesnay, Fré. Duchesnay EDOUARDDUCHESNAY, Scikit-learn: Machine Learning in Python Gaël Varoquaux Bertrand Thirion Vincent Dubourg Alexandre Passos PEDREGOSA, VAROQUAUX, GRAMFORT ET AL. Matthieu Perrot, 2011. http://scikit-learn.sourceforge.net.[6] B.H. Savitzky, S.E. Zeltmann, L.A. Hughes, H.G. Brown, S. Zhao, P.M. Pelz, T.C. Pekin, E.S. Barnard, J. Donohue, L. Rangel Dacosta, E. Kennedy, Y. Xie, M.T. Janish, M.M. Schneider, P. Herring, C. Gopal, A. Anapolsky, R. Dhall, K.C. Bustillo, P. Ercius, M.C. Scott, J. Ciston, A.M. Minor, C. Ophus, Py4DSTEM: A Software Package for Four-Dimensional Scanning Transmission Electron Microscopy Data Analysis, Microscopy and Microanalysis. 27 (2021) 712–743. https://doi.org/10.1017/S1431927621000477.[7] E.J. Kirkland, B.M. Siegel, N. Uyeda, Y. Fujiyoshi, Improved high resolution image processing of bright field electron micrographs, Ultramicroscopy. 17 (1985) 87–103. https://doi.org/10.1016/0304-3991(85)90002-6.[8] Y. Yokota, M. Tomita, H. Hashimoto, H. Endoh, Construction of an on-line system for FFT processing and analysis of atomic resolution microscopic images and its applications, Ultramicroscopy. 6 (1981) 313–321. https://doi.org/10.1016/S0304-3991(81)80218-5.[9] J.J. Hu, F.H. Li, Maximum entropy image deconvolution in high resolution electron microscopy, Ultramicroscopy. 35 (1991) 339–350. https://doi.org/10.1016/0304-3991(91)90086-L.[10] A. Grossmann, J. Morlet, T. Paul, Transforms associated to square integrable group representations. I. General results, J Math Phys. 26 (1985) 2473–2479. https://doi.org/10.1063/1.526761.[11] T.S. Lee, Using 2D Gabor Wavelets, IEEE Trans Pattern Anal Mach Intell. 18 (1996) 1–13. https://ieeexplore.ieee.org/document/541406 (accessed November 9, 2022).[12] D. Barina, Gabor Wavelets in Image Processing, (2016) 2–6. http://arxiv.org/abs/1602.03308.[13] B.A. Olshausen, D.J. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images, Nature. 381 (1996) 607–609. https://doi.org/10.1038/381607a0.[14] Y. Yoon, S. Noh, J. Jeong, K. Park, An accuracy improvement method for the topology measurement of an atomic force microscope using a 2D wavelet transform, Ultramicroscopy. 188 (2018) 70–76. https://doi.org/10.1016/j.ultramic.2018.02.009.[15] I. Daubechies, The Wavelet Transform, Time-Frequency Localization and Signal Analysis, IEEE Trans Inf Theory. 36 (1990) 961–1005. https://doi.org/10.1109/18.57199.[16] L.B. del Río, A. Gómez, M. José-Yacamán, Image processing in TEM using the wavelet transform, Ultramicroscopy. 38 (1991) 319–324. https://doi.org/10.1016/0304-3991(91)90165-3.[17] L.B. del Río, A. Gómez, M. José-Yacamán, Analysis of scanning probe microscope images using wavelets, Ultramicroscopy. 106 (2006) 389–397. https://doi.org/10.1016/j.ultramic.2005.11.006.[18] I. Gatan, Gatan Microscopy Suite Software, (2018). https://www.gatan.com/products/tem-analysis/gatan-microscopy-suite-software.[19] B. Schaffer, How to script.... Digital Micrograph Scripting Handbook, http://www.gatan.com/.[20] D. Mitchell, Dave Mitchell’s DigitalMicrographTM Scripting Website, http://www.dmscripting.com/ .[21] B. Schaffer, “How to script...” - Digital Micrograph Scripting handbook, http://digitalmicrograph-scripting.tavernmaker.de/HowToScript_index.htm.[22] D. R. G. Mitchell, B. Schaffer, Scripting-customised microscopy tools for Digital MicrographTM, Ultramicroscopy. 103 (2005) 319–332. https://doi.org/10.1016/j.ultramic.2005.02.003.   13 / 13  image3.pngimage4.pngimage5.pngimage6.pngimage1.pngimage2.png