# Fileset

[article.pdf](https://mdr.nims.go.jp/filesets/0ccbab6e-02bc-4939-8142-12f5c82ee543/download)

## Creator

Zhuo Diao, Keiichi Ueda, Linfeng Hou, Hayato Yamashita, [Oscar Custance](https://orcid.org/0000-0001-7931-603X), Masayuki Abe

## Rights

This article may be downloaded for personal use only. Any other use requires prior permission of the author and AIP Publishing. This article appeared in Zhuo Diao, Keiichi Ueda, Linfeng Hou, Hayato Yamashita, Oscar Custance, Masayuki Abe; Automatic drift compensation for nanoscale imaging using feature point matching. Appl. Phys. Lett. 20 March 2023; 122 (12): 121601 and may be found at https://doi.org/10.1063/5.0139330.[In Copyright](http://rightsstatements.org/vocab/InC/1.0/)

## Other metadata

[Automatic drift compensation for nanoscale imaging using feature point matching](https://mdr.nims.go.jp/datasets/15adbe74-0767-42f4-970c-8a741fa4018f)

## Fulltext

Automatic Drift Compensation for Nanoscale Imaging Using FeaturePoint MatchingZhuo Diao,1 Keiichi Ueda,1, 2 Linfeng Hou,1 Hayato Yamashita,1 Oscar Custance,3 and Masayuki Abe11)Graduate School of Engineering Science, Osaka University, 1-3 Machikaneyama, Toyonaka, Osaka 560-0043,Japan2)Tokyo Metropolitan Industrial Technology, Research Institute, 2-4-10 Aomi, Kotoku, Tokyo, 135-0064,Japan3)National Institute for Materials Science (NIMS), 1-2-1 Sengen, Tsukuba, Ibaraki 305-0047,Japan(Dated: 19 December 2022)A novel implementation of drift compensation for imaging at the nanoscale is presented. The method is based on com-puter vision techniques, and hence applicable to any microscope that generates images through a computer interface.The algorithm extracts and matches pairs of feature points from consecutive images to compute and compensate forprobe-sample misalignments over time. The protocol also applies selection rules that make it robust against changesin image contrast. We demonstrate our fully-automatic implementation by continuously imaging the same area of aSi(100) surface at the atomic scale with scanning probe microscopy over a period of 25 hours at room temperature,showing that the method is robust even under the presence of non-linear drift or spontaneous changes of the probe apex.We apply our method to study the movement of pairs of tin atoms confined within a half-unit cell of the Si(111)-(7×7)surface and estimate the energy barrier for their diffusion at room temperature.Advances in nanoscience and nanotechnology are linkedto imaging with microscopes at the nanoscale. When ex-ploring nano-structures at room temperature with either scan-ning transmission electron microscopy, scanning electron mi-croscopy, or scanning probe microscopy, the ever-presentthermal drift compromises the time frame for reliable dataacquisition. Typical thermal drift in vacuum at room tem-perature (assuming thermal equilibrium conditions) is, for in-stance, between 0.3 and 1 Å/min. These values considerablyincrease for experiments carried out in air or liquid environ-ment. Such thermal drift causes a three-dimensional mis-alignment between the imaging source and the object underobservation, which leads to distorted images1. These distor-tions become critical when exploring specimens of just a fewnanometer size2–4. To reduce these detrimental effects, oneis compelled to either use high-speed data acquisition5–7 orto implement schemes for the real-time compensation of thethermal drift in three dimensions8,9. Similar to thermal drift,piezoelectric creep can also produce image distortions10,11.Here, we introduce a novel method for the compensation ofprobe-sample misalignments that is based on computer visiontechniques and therefore universally applicable to any instru-ment that provides images of objects through a computer in-terface. Although we demonstrate this method using scanningprobe microscopy, it is also possible to apply it to scanningtransmission electron microscopy and scanning electron mi-croscopy.Operation of microscopes in a cryogenic environment —inwhich sample and probe are cooled down close to the boil-ing temperature of liquid helium— has been the traditionalway to cancel out thermal drift. In such cryogenic condi-tions landmark feats on scanning probe microscopy (SPM)have been achieved12–14. However, on-surface reactions, dif-fusion of atomic and molecular species and other processesthat are in some degree thermally assisted, are more likelyto occur well above cryogenic temperatures. To study suchprocess, it is pivotal to compensate the misalignment betweenthe imaging probe and the sample caused by the thermal drift.In SPM, the use of atom tracking to measure the drift and tocompensate it by using feedforward techniques15 has enabledto temporarily keep the relative position of probe and surfacewith a precision close to the one obtained at cryogenic temper-atures so that atom manipulation3 and reliable spectroscopicmeasurements4 were accomplished even at room temperature.The use of feedforward technics requires the thermal drift tobe a well-defined function of time so that the probe-surfacemisalignment can be predicted and compensated over a giventime window. For a microscope in thermal equilibrium withthe ambient temperature, the thermal drift can be consideredalmost a linear function of time (i.e., constant drift velocity)over a few minutes, but this linearity is lost over longer peri-ods due to small variations of the ambient temperature. There-fore, the drift velocity must be measured and updated period-ically. The method we present here, however, allows imag-ing the same spot of a sample at the nanometer scale overseveral days with atomic-level accuracy under the presenceof non-linear thermal drift, long-standing piezo creep16, andeven upon changes in the imaging contrast.The experiments were performed with a home-built SPMoperated in ultra-high vacuum and at room temperature. Themicroscope was controlled by a National Instruments PXIe-7857R unit using a home-built scan and data acquisition soft-ware based on LabVIEW and Python. Data and image pro-cessing algorithms were implemented using OpenCV17 anda personal Python library18. Atomically clean silicon sur-faces were prepared by standard flashing-annealing cycles,and Pt/Ir probes were used to acquire the scanning tunnelingmicroscopy (STM) data presented in this work.Figure 1 outlines an experiment representative of applyingour algorithm. It shows a selection of 9 images extracted froma set of 444 STM topographies that were continuously mea-sured on the same area of the Si(100) surface over a period of2start 1 start 2 start 32 hour24 hour8 hour16 hour 20 hour12 hourFIG. 1. Selection of STM topographies extracted from a total of 444images continuously measured over a period of 25 hours tracking thesame area of the Si(100) surface using the algorithm presented here(the whole process is presented in a video). The dotted line in the im-age taken at the 8th hour of the experiment highlights the position atwhich the image contrast significantly changed due to a spontaneousmodification of the probe apex. Image acquisition time, scan area,tunneling current set-point and sample bias were: t=200 s, (11.49× 11.49) nm2, It=300 pA and Vs=1.5 V, respectively. The order inwhich the images were measured and the time since the experimentstarted are used to label each image.25 hours. The first three images (1, 2, 3) were acquired with-out drift compensation, as they are used for the first estimationof the drift. In the image taken at the 8th hour of the experi-ment, the dotted line highlights the position at which a spon-taneous modification of the tip apex changed the topographiccontrast. Even upon this significant alteration, the algorithmwas still able to precisely track the same spot of the surfaceuntil the end of the experiment.Figure 2 illustrates the basic implementation of our fullyautomated drift correction system. The algorithm comprisestwo stages: in the first stage, it extracts and matches pairs offeature points from three consecutive images; in the secondstage, it groups the matched feature points, calculates the driftvelocity, and compensates for the drift. These processes arecyclically repeated every three images. The first three consec-utive images are measured without any drift compensation.Once the drift correction is applied, subsequent three-imagecycles of the algorithm are carried out with the drift correc-tion on, and the variation of the drift velocity is adjusted withtime.In the first stage of the algorithm, the images are automat-ically pre-processed by applying a plane fit subtraction fol-lowed by a 2D convolution smoothing based on a GaussianHann filter with a (11 × 11) pixel kernel and a σ=1 stan-dard deviation. Feature points from the images (multicolor!"#$%XC&!"'% &!"'%A1A2A1A3(e)A1-A2 A1-A3 (!')*+&!"#$%,!')*+&!"#$%,!')*+&!"#$%(b)(d)(a)(c)!"#$%YC0 100 200 300 400 0 100 200 300 4001.35 1.401.300.240.260.282.4 2.52.3 2.60.4750.4500.4250.400FIG. 2. Illustration of one cycle of our automatic drift correctionalgorithm. (a) and (c), Acquisition of three consecutive images(A1,A2,A3), extraction of feature points from them (multicolor dots),and matching of pairs of feature points between A1 and A2, and A1and A3 (lines connecting dots). (b) and (d), Clustering of the (x, y)coordinates of the relative position between pairs of matched points,and application of a "majority rule" to select a centroid (blue trian-gle) representative of the relative displacement between images A1and A2, and A1 and A3. Yellow lines and points highlight pairs ofmatched points assigned to the same cluster, and red ones representmatched pairs excluded from the centroid calculation. (e), Estima-tion of the drift velocity for X and Y scan directions from the XC andYC coordinates of the centroids.dots in the STM topographies of Fig. 2) are extracted usingthe AKAZE detector19, which offers a precision as good asthe KAZE Features detector20 and a faster calculation speed.Feature points from the first image are matched to the sec-ond [Fig. 2(a)] and third images [Fig. 2(c)], respectively. Thefeature point matching —lines in Figs. 2 (a) and (c)— is doneby using OpenCV’s Brute-Force matcher with crossCheck en-abled: Brute-Force matcher compares the vectors of all fea-ture points of the two images and pairs them by proximity;and crossCheck verifies that the pairing is reciprocal if theimage order is exchanged. The relative position between pairsof matched points is calculated and stored on a list of (x, y)coordinates.In the second stage, the (x, y) coordinates of the relativeposition between pairs of matched points are grouped by us-ing k-means clustering21 [see Figs. 2 (b) and (d)]. To definean a priori unknown number (n) of clusters, we use an iter-ative approach. A two-cluster initialization (n = 2) is givento the k-means algorithm. For each of the clusters providedby k-means, we check that the average separation of each ofthe points of the cluster (g j) with respect to the cluster’s cen-troid (gc; the arithmetic mean of all points within the cluster)3!"#!$#%&%'()*+%',"-).)/0%!/.#%1%'()*+%',"-).)/0%!/.#234254364748-"/%9"/:)%22;6</.0(.)%!4#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%=9(>0%-+99)-0(+/%".+?/0%!/.@.(/#FIG. 3. (a) Trajectory of the drift compensation for the experimentsummarized in Fig. 1. Representative time laps are indicated by redspots. The inset denotes the scan area relative to the total trajectory.(b) Drift velocity correction in the X and Y scan directions for the ex-periment displayed in Fig. 1. The insets represent a zoom-in over thesignal excluding the initial stages of the experiment, in which piezocreep dominated the correction. Dotted vertical red lines pinpointthe timing at which the image contrast changed due to spontaneousatomic rearrangements at the probe apex.is smaller than a given distance d, which is empirically deter-mined according to the image acquisition parameters22:d ≥ 1m ∑1≤ j≤m√(g j −gc)2 (1)with m being the number of points contained within thecluster. If this condition is not fulfilled, n is increased in aunit and fed back into k-means. The iteration on n stops wheneq. (1) is satisfied by each of the supplied clusters. Once thenumber of clusters is defined, a "majority rule" is applied, andthe centroid of the cluster containing the largest number ofpoints is chosen to characterize the relative displacement ofimages A2 and A3 with respect to A1. In case the number ofpairs of matched featured points is lower than five, the algo-rithm selects all the matched points to calculate the centroid.The slope of a linear regression9 fitting of the XC and YC co-ordinates of the centroids with respect to the correspondingimage timestamp (t) [Fig. 2(e)] supplies the drift velocity inthe X and Y scan directions, respectively. The drift is thencompensated by applying a linear feedforward signal into theX and Y channels, and the acquisition of three new imagesto start the next loop of the algorithm is performed. Each ofthese loops produces a correction to the drift velocity, dvdxand dvdy, that is supplied to the feedforward protocol to lockon the same surface area for long periods of time.Figure 3 displays the drift compensation report for the ex-periment summarized in Fig. 1. The trace depicted in Fig. 3(a)reveals a strong non-linear behavior of the drift over the 25hours the algorithm was tracking the same surface spot. Thisnon-linearity over long periods does not represent a challengeto our method, because the drift correction is updated everythree images. The graphs in Fig. 3(b) show the correctionto the drift velocity in the X and Y scan directions during the25 hours experiment. At the initial stages of the tracking, bothcomponents display a spike, which is more prominent over theX scan direction. These rapid changes in the drift velocity aredue to piezo creep originated from moving the scanner to thedesired surface area before starting the algorithm. After sixloops (i.e, approximately one hour) the creep effect becamenegligible, and the drift velocity stabilized with a standard de-viation of 3.11 pm/min in the X direction and 2.79 pm/minin the Y direction, respectively, see insets in Fig. 3(b). Thesevalues demonstrate that our method has enough precision andstability to perform sub-angstrom order drift compensation.It is possible to pause the imaging loop to carry out avariety of experiments, such as point spectroscopy4, biaspulse manipulations23 and, depending on the drift condition,even three-dimensional spectroscopic measurements24. Inthis case, the last drift velocity calculated will be used to com-pensate for the drift until the imaging loop is restored25.Unlike other approaches to compensate for the drift, suchas using cross-correlation between images26, our method isquite robust against changes in image contrast and resolu-tion. This characteristic is demonstrated in Fig. 4, wherethe feature point matching from the three-image loops be-fore (A139,A140,A141) and during (A142,A143,A144) the spon-taneous probe modification displayed in Fig. 1 is presented.The atomic reconfiguration occurring at the probe apex dur-ing the acquisition of image A142 led to a change in the atomiccontrasts and to a different set of pairs of featured points, seeFigs. 4(e) and (g). Even upon this dramatic probe change,the algorithm was able to cluster the newly detected pairs ofpoints, apply the "majority rule" [points and lines in yellowin Fig. 4 (e) and (g)] and update the drift velocity. In theinsets of Fig. 3(b), the dotted vertical red lines indicate theoccurrence of spontaneous probe modifications similar to theone described in Fig. 4. So far, we have not found any sit-uation in which a probe change was catastrophic enough forthe algorithm to fail: there are always enough pairs of featurepoints produced to ensure the correct execution of the algo-rithm. However, as a safety measurement, in case no feature4(c)9.04.50.03 3.5!"#$%&'"()*+!"#$%&'"()*+18.09.00.04.5 6.752.25,"#$%&'"()*+""""""""""A139A140A139A141A142A143A142A1441.80.90.0−0.9 0.0−1.80.90.450.00.0 0.45−0.45,"#$%&'"(-*+"""""""""""""!"#$%&'"(-*+!"#$%&'"(-*+(a) (b)(c) (d)(e) (f)(g) (h)FIG. 4. Feature point matching and clustering of pairs of matchedpoints before (a) to (d) and during (e) to (h) the probe-induced imagecontrast change displayed in Fig.1.points are matched, the algorithm will not compensate for thedrift and it will restart the acquisition of three new images torecalculate the drift velocity.We applied our drift compensation algorithm to study theswitching of pairs of Sn atoms between two adsorption con-figurations on the Si(111)-(7×7) surface27. The samples wereprepared following the procedure described in Ref. 2727. Weexplored this surface system in several experimental sessions,in each of them imaging an area of interest at the surface forlong periods of time. We have been able to track the samesurface spot as long as for three consecutive days. Figure 5summarizes one of these measurement sessions, in which weperformed an 18-hour observation of the same surface area,generating a total of 927 images, with an acquisition time of66 sec per frame plus an idle time of 5 sec between imagesto reduce piezo creep10. During this 18-hour experiment, wecharacterized the switching behavior of a pair of Sn atomstrapped within a half-unit cell of the Si(111)-(7×7) surface.This pair of Sn atoms was seen in two adsorption configura-tions: P and P∗ [Fig. 5(a)]. We identified reversible transitionsbetween P and P∗ configurations that occur at a relatively lowfrequency, as it is shown in the six panels of Fig. 5(a). Wealso detected jumps of the pair in a P∗ configuration betweenthree equivalent adsorption sites with an occurrence close toour fastest image speed, see images 2 to 5 in Fig. 5(a). There-fore, we focused our analysis on the reversible transitionsPP*!"#$%&'()*"+%!,)$-&.(P P* P*P* P* P/ 01 2 3&*(FIG. 5. (a) Series of consecutive images demonstrating the reversibleswitching of a pair of Sn atoms between two types of adsorption con-figurations, P (1 and 6) and P∗ (2 to 5), on a faulted half-unit cell ofthe Si(111)-(7×7) surface. Images 2 to 5 show the diffusion of thepair of Sn atoms in a P∗ configuration between three equivalent ad-sorption positions within the half-unit cell. (g) Telegraphic switchingbehavior of the pair of Sn atoms displayed in (a) between the P andP∗ configurations characterized over a period of 18 hours. The redarrow points to the event at which the series of images in (a) wasacquired. Image acquisition time, tunneling current set-point andsample bias were t=66 s, It=200 pA and Vs=1.5 V, respectively. Theimage size is (2.43 × 3.80) nm2.P ↔ P∗, because at the image speed we conducted the exper-iment jumps of the pair in a P∗ configuration can be missedduring imaging. In Fig. 5(b), we have quantified the transi-tions P → P∗ and P∗ → P during the 18 hours we wereimaging the same surface area. Our results suggest that the Snatoms prefer to stay in a P configuration longer than in a P∗one, thus the P configuration seems to be more stable than theP∗ one.We have made a rough estimation of the energy barrier forboth transitions assuming that they are thermally activated andfollow an Arrhenius equation (Γ = Γ0 exp[−Eg/kT ]), withΓ0 being the attempt frequency, Eg the energy barrier for thetransition, k the Boltzmann constant and T the absolute tem-perature. In the absence of a variable temperature STM thatwould allow us to measure the switching rate at several tem-peratures to estimate both the attempt frequency and the en-ergy barrier, we have adopted values for Γ0 from the liter-ature to make a rough estimation of the energy barrier. Inthe case of the rotation of pairs of Pb atoms within a half-5unit cell of the Si(111)-(7×7) surface, an attempt frequency of1012 jumps/pair/second was experimentally obtained28. Con-sidering this value of Γ0 and that the experiments were doneat room temperature (293 K), we obtain an energy barrierof 0.91±0.01 eV for the P → P∗ transition and a value of0.86±0.03 eV for the P∗ → P one. The error in these val-ues is provided by the statistical standard deviation of the timespent by the Sn pair in a given configuration. This experimentis a showcase for the potential of our drift compensation algo-rithm to obtain the physical properties of surface systems.We have put forward a new algorithm for the real-time au-tomatic compensation of drift at the atomic scale using fea-ture point detection in images. Because we deploy matchingand clustering of feature points between three consecutive im-ages, our approach is robust against contrast changes by slightmodifications of both probe and surface. At variance withpreviously reported approaches15, our method can track thesame surface area for days with sub-angstrom precision evenunder a no-linear drift behavior just by imaging. Addition-ally, since our method is based on computer vision technics,its application is not limited to scanning probe microscopy,but it can also be implemented in other microscopy setupssuch as scanning electron microscopy or transmission elec-tron microscopy. Our algorithm is open source and availableelsewhere29.This work was supported in part by a Grant-in-Aid for Sci-entific Research (19H05789, 21H01812, 21K18876) from theMinistry of Education, Culture, Sports, Science and Technol-ogy of Japan (MEXT) and by NIMS grants (PF2010, PF3130and QN3510). A part of this work was also supported by JSTSPRING (JPMJSP2138).1C. T. Herbschleb et al., Review of Scientific Instruments 85, 083703 (2014).2E. Hill, B. Freelon, and E. Ganz, Phys. Rev. B 60, 15896 (1999).3Y. Sugimoto et al., Science 322, 413 (2008).4Y. Sugimoto et al., Nature 446, 64 (2007).5L. M. Picco et al., Nanotechnology 18, 044030 (2006).6L. Gura et al., Applied Physics Letters 119, 251601 (2021).7L. Gura et al., Phys. Rev. B 105, 035411 (2022).8M. Abe, Y. Sugimoto, O. Custance, and S. Morita, Nanotechnology 16,3029 (2005).9P. Rahe et al., Review of Scientific Instruments 82, 063704 (2011).10M. S. Rana, H. R. Pota, and I. R. Petersen, IEEE/ASME Transactions onMechatronics 20, 1458 (2015).11S. B. Park, S. S. Park, and G. P. Carman, Linear and nonlinear behavior ofpiezoelectric materials, in Proc.SPIE, volume 2715, 1996.12D. M. Eigler and E. K. Schweizer, Nature 344, 524 (1990).13M. A. Lantz et al., Phys. Rev. Lett. 84, 2642 (2000).14L. Gross, F. Mohn, N. Moll, P. Liljeroth, and G. Meyer, Science 325, 1110(2009).15M. Abe, Y. Sugimoto, O. Custance, and S. Morita, Appl. Phys. Lett. 87,173503 (2005).16D. Croft, G. Shed, and S. Devasia, Journal of Dynamic Systems, Measure-ment, and Control 123, 35 (1999).17G. Bradski, Dr. Dobb’s Journal of Software Tools (2000).18Spmutil, see https://github.com/DIAOZHUO/SPMUtil for installationand basic usage.19P. F. Alcantarilla, J. Nuevo, and A. Bartoli, Fast explicit diffusion foraccelerated features in nonlinear scale spaces, in British Machine VisionConference, BMVC 2013, Bristol, UK, September 9-13, 2013, edited byT. Burghardt, D. Damen, W. W. Mayol-Cuevas, and M. Mirmehdi, BMVAPress, 2013.20P. F. Alcantarilla, A. Bartoli, and A. J. Davison, Kaze features, in Proceed-ings of the 12th European Conference on Computer Vision - Volume PartVI, ECCV’12, pages 214–227, Berlin, Heidelberg, 2012, Springer-Verlag.21D. Arthur and S. Vassilvitskii, K-means++: The advantages of careful seed-ing, in Proceedings of the Eighteenth Annual ACM-SIAM Symposium onDiscrete Algorithms, SODA ’07, pages 1027–1035, USA, 2007, Society forIndustrial and Applied Mathematics.22The value of the distance d is empirically tuned depending on the size andresolution of the images. In this work, we used a value d=1nm for (256×256) pixel images of (20×20) nm2 scan area.23N. Pavlicek et al., Nature Nanotech. 12, 308 (2017).24B. J. Albers et al., Nature Nanotech. 4, 307 (2009).25Our algorithm is not rotation or scale-invariant: if the image orientation orits scale is changed dur- ing the experiment, the feature point matching willnot provide a reliable result upon restoring the imaging loop.26I. Horcas et al., Review of Scientific Instruments 78, 013705 (2007).27O. Custance, I. Brihuega, J. Gómez-Rodríguez, and A. Baró, Surface Sci-ence 482-485, 1406 (2001).28O. Custance et al., Phys. Rev. B 67, 235410 (2003).29The source code and an example are available at https://github.com/DIAOZHUO/drift_compensation_by_feature_point_matching.