# Fileset

[Prediction_and_optimization_of_epoxy_adhesive_strength_from_a_small_dataset_through_active_learning.pdf](https://mdr.nims.go.jp/filesets/cb79e4f4-bdb7-4180-8ba8-b4c69d29cd60/download)

## Creator

[Lambard, Guillaume](https://orcid.org/0000-0003-0275-4079), [Naito, Masanobu](https://orcid.org/0000-0001-7198-819X), [Samitsu, Sadaki](https://orcid.org/0000-0002-4139-1656), [Pruksawan, Sirawit](https://orcid.org/0000-0002-9380-1872), [Sodeyama, Keitaro](https://orcid.org/0000-0002-9228-0729)

## Rights



## Other metadata

[Prediction and optimization of epoxy adhesive strength from a small dataset through active learning](https://mdr.nims.go.jp/datasets/3531cb90-075f-4b92-9922-17ba4ba6d95e)

## Fulltext

Prediction and optimization of epoxy adhesive strength from a small dataset through active learningFull Terms & Conditions of access and use can be found athttps://www.tandfonline.com/action/journalInformation?journalCode=tsta20Science and Technology of Advanced MaterialsISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/tsta20Prediction and optimization of epoxy adhesivestrength from a small dataset through activelearningSirawit Pruksawan , Guillaume Lambard , Sadaki Samitsu , KeitaroSodeyama & Masanobu NaitoTo cite this article: Sirawit Pruksawan , Guillaume Lambard , Sadaki Samitsu , Keitaro Sodeyama& Masanobu Naito (2019) Prediction and optimization of epoxy adhesive strength from a smalldataset through active learning, Science and Technology of Advanced Materials, 20:1, 1010-1021,DOI: 10.1080/14686996.2019.1673670To link to this article:  https://doi.org/10.1080/14686996.2019.1673670© 2019 The Author(s). Published by NationalInstitute for Materials Science in partnershipwith Taylor & Francis Group.View supplementary material Published online: 21 Oct 2019. Submit your article to this journal Article views: 3352 View related articles View Crossmark data Citing articles: 6 View citing articles https://www.tandfonline.com/action/journalInformation?journalCode=tsta20https://www.tandfonline.com/loi/tsta20https://www.tandfonline.com/action/showCitFormats?doi=10.1080/14686996.2019.1673670https://doi.org/10.1080/14686996.2019.1673670https://www.tandfonline.com/doi/suppl/10.1080/14686996.2019.1673670https://www.tandfonline.com/doi/suppl/10.1080/14686996.2019.1673670https://www.tandfonline.com/action/authorSubmission?journalCode=tsta20&show=instructionshttps://www.tandfonline.com/action/authorSubmission?journalCode=tsta20&show=instructionshttps://www.tandfonline.com/doi/mlt/10.1080/14686996.2019.1673670https://www.tandfonline.com/doi/mlt/10.1080/14686996.2019.1673670http://crossmark.crossref.org/dialog/?doi=10.1080/14686996.2019.1673670&domain=pdf&date_stamp=2019-10-02http://crossmark.crossref.org/dialog/?doi=10.1080/14686996.2019.1673670&domain=pdf&date_stamp=2019-10-02https://www.tandfonline.com/doi/citedby/10.1080/14686996.2019.1673670#tabModulehttps://www.tandfonline.com/doi/citedby/10.1080/14686996.2019.1673670#tabModulePrediction and optimization of epoxy adhesive strength from a small datasetthrough active learningSirawit Pruksawana,b, Guillaume Lambardc, Sadaki Samitsua, Keitaro Sodeyamac and Masanobu Naitoa,b,daData-driven Polymer Design Group, Research and Services Division of Materials Data and Integrated System (MaDIS), National Institutefor Materials Science (NIMS), Tsukuba, Japan;bProgram in Materials Science and Engineering, Graduate School of Pure and Applied Sciences, University of Tsukuba, Tsukuba, Japan;cEnergy Materials Design Group, Research and Services Division of Materials Data and Integrated System (MaDIS), National Institute forMaterials Science (NIMS), Tsukuba, Japan;dDepartment of Advanced Materials Science, Graduate School of Frontier Sciences, The University of Tokyo, Kashiwa, JapanABSTRACTMachine learning is emerging as a powerful tool for the discovery of novel high-performancefunctional materials. However, experimental datasets in the polymer-science field are typicallylimited and they are expensive to build. Their size (< 100 samples) limits the development ofchemical intuition from experimentalists, as it constrains the use of machine-learning algo-rithms for extracting relevant information. We tackle this issue to predict and optimizeadhesive materials by combining laboratory experimental design, an active learning pipelineand Bayesian optimization. We start from an initial dataset of 32 adhesive samples that wereprepared from various molecular-weight bisphenol A-based epoxy resins and polyetheraminecuring agents, mixing ratios and curing temperatures, and our data-driven method allows us topropose an optimal preparation of an adhesive material with a very high adhesive jointstrength measured at 35.8 ± 1.1 MPa after three active learning cycles (five proposed prepara-tions per cycle). A Gradient boosting machine learning model was used for the successiveprediction of the adhesive joint strength in the active learning pipeline, and the modelachieved a respectable accuracy with a coefficient of determination, root mean square errorand mean absolute error of 0.85, 4.0 MPa and 3.0 MPa, respectively. This study demonstratesthe important impact of active learning to accelerate the design and development of tailoredhighly functional materials from very small datasets.ARTICLE HISTORYReceived 26 July 2019Revised 25 September 2019Accepted 25 September 2019KEYWORDSMaterials informatics; activelearning; adhesive jointstrength; epoxy resin;crosslink network structureCLASSIFICATION6001. IntroductionIn recent decades, interest in machine-learning (ML)techniques has increased in various research fieldsbecause of their outstanding efficiency to extract salientinformation [1]. More recently in the field of materialsscience, ML techniques have begun to play an impor-tant role in the design and development of novel mate-rials [2,3]. ML usually requires a large amount of data,that is, > 1000 samples, to build accurate models [1].Themain goal ofML inmaterials science is to search forhighly functional materials with properties that aretailored to fit the requirements of a specific application[2]. Recent studies demonstrate the potential of ML-based experimental design to discover various newfunctional materials in different fields within an activelearning framework. The active learning strategy istypically efficient in improving prediction model. Theexamples of this include finding very low thermalCONTACT Guillaume Lambard LAMBARD.Guillaume@nims.go.jp; Masanobu Naito NAITO.Masanobu@nims.go.jp Data-driven PolymerDesign Group, Research and Services Division of Materials Data and Integrated System (MaDIS), National Institute for Materials Science (NIMS),Tsukuba, JapanSupplemental data for this article can be accessed here.SCIENCE AND TECHNOLOGY OF ADVANCED MATERIALS2019, VOL. 20, NO. 1, 1010–1021https://doi.org/10.1080/14686996.2019.1673670© 2019 The Author(s). Published by National Institute for Materials Science in partnership with Taylor & Francis Group.This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permitsunrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.https://doi.org/10.1080/14686996.2019.1673670http://www.tandfonline.comhttps://crossmark.crossref.org/dialog/?doi=10.1080/14686996.2019.1673670&domain=pdf&date_stamp=2020-02-02hysteresis NiTi-based shape memory alloys using adap-tive experimental design [4], discovery of large electro-strains in BaTiO3-based piezoelectrics using activelearning [5], searching high-temperature ferroelectricperovskites by two-step machine learning [6], findingBaTiO3-based ceramics with large energy storage at lowfields using machine learning and experimental design[7] and discovery of new metallic glasses through itera-tion of machine learning and high-throughput experi-ments [8]. However, a ML based approach has not beenwidely applied to the field of polymer science. Onemajor constraint is that experimental datasets in poly-mer science are typically limited and expensive to con-struct. A huge and comprehensive source ofinformation on polymer properties is not easily obtain-able. Sometimes, the experimental dataset is scattered[2]. Datasets that are collected from various literaturesources may be noisy and inconsistent because severalexperimental factors affect any obtained sample andmeasurements, such as process conditions, the sourceand purity of used chemicals and environmental con-ditions [9,10]. Particularly, if a material requiresa specific design, only few data are available. Thus, itis challenging to obtain a sufficiently large curated data-set, which limits the use of ML for polymer research.The development of high-strength adhesives forjoint bonding is one of the cases where application-specific design is needed. In addition to adhesive prop-erties, several other factors influence the adhesive jointstrength (σad), such as: substrate properties, substratesurface preparation, joint configuration, measurementconditions and environmental factors. Hence, anadhesive will behave differently under different joint-design and bonding conditions [11]. In consequence,the experimental dataset for an adhesive for one spe-cific joint cannot be acquired easily because differentstudies usually use different conditions, such as anadhesive thickness, substrate surface treatment andthe joint configuration. Furthermore, no theoreticaland empirical knowledge exists to predict precisely theσad from a modified adhesive system.Various approaches have been exploited in the litera-ture to modify the mechanical properties of adhesives,such as the fracture toughness, elastic modulus and ten-sile strength [12–15]. The modification of epoxy adhe-sives by adjusting their network structure is one of themost effective ways to provide a wide diversity ofmechanical properties. Using this approach, we can tailorthe adhesive properties tomeet a specific requirement forjoint bonding. In the case of adhesively bonded joints,that is, when two substrates are bonded via an adhesive,several properties are required to achieve a high σad.A good resistance to crack growth as reflected by a highfracture toughness and high flexibility of adhesives isa desirable property to withstand the tensile stress con-centration of joints [11]. The adhesive needs a reasonablyhigh elastic modulus to obtain a high-shear fracturestress [11]. The interaction between an adhesive andthe substrates is important for controlling the fracturebehaviour of joints [16]. Because several factors influenceadhesively bonded joint properties, the development ofhigh-performance adhesives for joint bonding is morecomplicated than that for the bulk form, and requiresfurther advanced techniques for achieving an exception-ally high σad [11,17]. In the metal joining process, espe-cially in structural bonding, an adhesive with high σad ishighly desired to resist joint failure and impactforces [18].Therefore, we propose a combination of the designof experimental techniques with an active learning (asknown as optimal experimental design [19]) pipelineand a Bayesian optimization to model and maximizethe σad from various mixtures to overcome the issuespresented above. Compared to other machine-learning-based materials’ design approaches [4–8], our two-stagedata-driven approach allows us to propose an optimalcondition for achieving target property from a verysmall dataset with designing controlled experiments,and does not require data from previous literatures.The first stage, active learning, aims to construct anaccurate ML model with a particular focus only ona specific range of high σad. By refining the experimentalconditions in the second stage, the Bayesian optimiza-tion is refined to search for the adhesive materials withextremely high adhesive strength. This approach is fore-seen to acceleratematerials design and reduce the devel-opment cost and time, especially for which initialnumber of samples is limited compared to the numberof combinations of free parameters for theirformulations.We use an initial small experimental dataset that webuilt and controlled. This dataset is focused ona model adhesive system that is composed of conven-tional bisphenol A-based epoxy resin and an amine-terminated poly(propylene glycols) curing agent thatis described in Section 2.1. The use of these types ofepoxy resins and curing agents with different linearchain lengths allows us to tailor the adhesive proper-ties. Throughout this paper, σad is measured througha single-lap shear test presented in Section 2.2. Toobtain epoxy adhesives with various network struc-tures, 32 samples of epoxy adhesives were preparedfrom different epoxy resin molecular weights (MWE),curing agent molecular weights (MWC), amine-to-epoxide ratios (r) and curing temperatures (Tcure)according to conditions suggested by a Graeco–Latinsquare design as shown in Section 2.3. The experi-mental results are reported in Table S2 of the supple-mental materials and they are used as our initialcurated dataset. Then, various ML models weretrained on this dataset to predict the σad. To enhancethe prediction accuracy of the most promising MLmodel and to increase the dataset size (ns) iteratively,an active learning pipeline was applied as detailed inSci. Technol. Adv. Mater. 20 (2019) 1011 S. PRUKSAWAN et al.Section 2.4. Therefore, specifically targeted experi-ments for reaching a high σad were conducted. Afterachieving experimental-like accuracy on σad predic-tions, the obtained ML model was fixed. Finally,a Bayesian optimization was used to optimize anepoxy network structure in greater processing detailand achieve the reported extreme high σad in thisstudy. Indeed, the Bayesian optimization highlydepends on its forward ML model for making propo-sals. Then, avoiding the active learning step would beequivalent to reduce the Bayesian optimization toa naive random sampling of our features space. Thiskind of strategy is here proposed in the case that theinitial dataset is very small, which is often found in thefield of polymer science. We present the promisingresults in Sections 3.1 and 3.2 to accelerate the discov-ery of new application-specific materials by usinga very small experimental dataset (few tens of sam-ples). An understanding of those predictions, as dis-cussed in Section 3.3, should provide valuableknowledge for the future development of adhesivematerials. Finally, we conclude and discuss furtherpossible improvements in Section 4.2. Experimental and ML methods2.1. MaterialsDiglycidyl ether of bisphenol A-based epoxy resin(DGEBA) and amine-terminated poly(propylene gly-col) curing agent (Jeffamine) with four different mole-cular weights were used: MWE 2 {370, 1650, 2900,3800} g/mol for the DGEBA (Mitsubishi Chemical,Japan) and MWC 2 {230, 400, 2000, 4000} g/molfor the Jeffamine (Sigma-Aldrich, Japan). The chemi-cal structures of the DGEBA and Jeffamine are shownin Figure 1. All chemicals were used as received with-out further purification. Aluminium alloy A6061P-T6(100 mm × 25 mm × 2 mm) was used as a substrate.Prior to the adhesive joint fabrication, the substratesurfaces were sandblasted and cleaned with ethanoland acetone.2.2. Preparation of adhesive joint specimens andsingle-lap shear testADGEBA epoxy resin (5.0 g) was preheated at 190°C for30 min to melt crystals. The Jeffamine curing agent wasadded to the liquid epoxy resin at a specific ratio r 2{0.75, 1.0, 1.25, 1.5}, where r < 1.0 indicates an epoxyexcess, r = 1.0 indicates a stoichiometricmixture betweenthe amine and epoxide and r > 1.0 indicates an amineexcess. For example, an r of 1.25 means 25% excessamine. The epoxy resin and curing agent were mixedby hand at 190°C for a few seconds to achievea homogeneous blend. This adhesive precursor wasspread over a 25 mm × 12.5 mm area on one face ofa pair of substrates. The two substrates were bondedtogether and the overlapping area was fixed by metalclamps as described previously [20]. An illustration ofthe adhesive joint specimen is provided in Figure 2. Theprepared specimen was cured in an oven at a specifictemperature Tcure 2 {90, 130, 170, 210}°C for one hour.The adhesive thickness wasmaintained at ~100 μmusing0.1 parts per hundred resin of spherical glass bread(Fujiseisakujo, Japan) as spacers. The four variable para-meters used later as input features for the ML models(see Section 2.4.2) are summarized in Table 1. The para-meter values in Table 1 are typical values ofMWE,MWC,r and Tcure for adhesive preparation. To be specific,MWEand MWC were selected on the basis of commerciallyavailable source material, and the values of r and Tcurewere chosen within a range that allow samplepreparation.The single-lap shear test of the adhesive joint speci-men was carried out by using a 10-kN AG-X plusseries universal tensile testing machine (Shimadzu,Japan). All tests were performed at a 2-mm/min cross-head speed at room temperature. The σad was calcu-lated by dividing the maximum tension load by theFigure 1. Chemical structures of diglycidyl ether of bisphenol A-based epoxy resin (DGEBA) and amine-terminated poly(propyleneglycols) curing agent (Jeffamine) and their curing reaction.Sci. Technol. Adv. Mater. 20 (2019) 1012 S. PRUKSAWAN et al.area of overlap (25 mm × 12.5 mm). At least twospecimens were used for each measurement and theaverage value was reported with the standard devia-tion. Indeed, the maximum tension load that wasreached by the developed epoxy resin of the highestσad exceeded 10 kN. Therefore, a second 50-kN AG-Xplus series universal tensile testing machine(Shimadzu, Japan) was needed at the final stage ofour design study. The use of this second machinewas required only when we had reached the measure-ment limitation of the first one.2.3. Selection of experimental conditions for theinitial datasetThe experimental conditions in this study consisted of256 possible conditions that were provided bya combination of four molecular weights for the epoxyresin and the curing agent, four amine-to-epoxide ratiosand four Tcure values (see Table 1). An initial set of ns = 32samples was collected according to the conditions thatwere suggested by a Graeco–Latin square design [21].The Graeco–Latin square design is a design of experi-mental techniques that can generate a uniform sample ofscattered data points [22]. By conducting two replicatedfour-by-four Graeco–Latin square designs, 32 experi-mental conditions were obtained.2.4. ML methodData pre-processing, data splitting and the applicationof the ML algorithms was performed using the Pythonpackage Scikit-learn (version 0.21) [23], and theBayesian optimization was executed using thePython package GPyOpt [24].2.4.1. Data pre-processing and splittingThe four variable parameters in this study (see Table 1)were standardized following a standard Gaussian dis-tribution of mean zero and standard deviation of one[17]. A k-fold cross-validation of different ML algo-rithms was performed [25]. The dataset was split ran-domly into k folds of equal size. Each fold was used asa training set by an ML algorithm with one other foldkept as a test set. The process was repeated k times.Their mean absolute error (MAE), root mean squareerror (RMSE) and coefficient of determination (R2) ofthe property predictions versus observations were aver-aged across all k folds to evaluate the MLmodels. Whena validation set was required for early stopping (e.g. forGradient boosting), the training set was split so that80% of the original training set was retained for trainingand 20% was used for validation.2.4.2. ML algorithmsThree supervised ML algorithms were applied asa regression tool to our dataset: Elastic Net, Randomforest and Gradient boosting [23]. Elastic Net isa linear regression model, whereas Random forestand Gradient boosting are ensemble learning methodsthat make predictions by combining the outputs fromindividual regression trees. The Random forest buildseach regression tree independently and merges themto obtain accurate and stable predictions, andGradient boosting builds regression trees sequentiallyto minimize residual errors from the previous trees.XGboost in Scikit-learn library was used to trainGradient boosting model [23]. During Gradient boost-ing training, early stoppage was applied to minimizethe overfit on the training set [26]. The accuracy ofan ML model was accessed through their RMSE (alower value is better), MAE (a lower value is better)and R2 (a value closer to one is better) on the predic-tions versus observations via a k-fold cross-validation.2.4.3. ML model and active learningThe best ML model that was chosen for its accuracy topredict the σad was trained on the initial dataset of ns = 32samples. The model predicted the σad of all (256–32)possible experimental conditions (see Table 1) from theinitial dataset. The predicted σad were ranked in descend-ing order. The top-five ranked experimental conditionswere selected as proposals for the next measurements toFigure 2. Schematic illustration of adhesive joint specimen for single-lap shear test.Table 1. Summary of variable parameters for adhesive formu-lation used at the active learning stage. Variable parametersinclude the molecular weight of the epoxy resin MWE (g/mol),the molecular weight of the curing agent MWC (g/mol), theamine-to-epoxide ratio r and the curing temperature Tcure (°C).Variable parameterNo MWE (g/mol) MWC (g/mol) r Tcure (°C)1 370 230 0.75 902 1650 400 1 1303 2900 2000 1.25 1704 3800 4000 1.5 210Sci. Technol. Adv. Mater. 20 (2019) 1013 S. PRUKSAWAN et al.be performed in the laboratory to increase the σad. Thesenew measurements were added to the initial dataset ofnow ns = (32 + 5) samples. Then, the ML model for σadprediction was trained again on this improved dataset.The ML model improved its σad prediction with addi-tional data, especially for a range of high σad, and pro-posed again the experimental conditions to follow for thenext measurements. This type of iterative supervisedlearning, or so-called active learning, was repeated cycleafter cycle until a preliminary goal of a sufficiently highaccuracy of the ML model was reached. In this study,active learning was stopped if the prediction error wascomparable to the experimental error of the σad that wasmeasured by a single-lap shear test. The final ML modelwas kept fixed and used as a forward model fora subsequent Bayesian optimization. The availableexperimental data at this stage of active learning werefed to theBayesian optimization as initial data points. Theflowchart of the active learning method is shown inFigure 3. Compared to conventional ML approaches,we use an initial experimental dataset that we built andcontrolled by design of experiments techniques. Thistechnique would generate a highly uniform set of samplepoints (Figure S1). In addition, all of the sample prepara-tion and measurements is carried out under the sameexperimental environment resulting in accurate and con-sistent data.2.4.4. Bayesian optimizationA Bayesian optimization [27] was used to search for thehighest σad by refining the variable conditions fromTable 1 once the coarse optimization through activelearning had been terminated. The ExpectedImprovement (EI) was used as an acquisition functionto propose new experimental conditions tomaximize theσad. In this step, two experimental conditions wererefined: r and Tcure. The r could vary from 0.75 to 1.50with an increment of 0.01, and the Tcure could vary from90 to 210°C by an increment of 1°C. TheMWE andMWCwere kept as four possible discrete values because theseare difficult to control precisely. Thus, the proposedexperimental conditions from the Bayesian optimizationwere ranked in descending order with respect to thepredicted σad. A series of experiments was carried outstarting from rank 1 until a new highest σad was observed.3. Results and discussions3.1. Experimental results from the initial datasetExperimental measurements of σad that compose ourinitial curated dataset are reported in Table S2 of thesupplemental materials. Figure 4 shows the distribu-tion of σad experimental values. σad was distributedfrom 0.0 MPa (no bond strength) to 31.9 MPa with anaverage at 10 ± 9 MPa.3.2. ML model3.2.1. Assessment and selection of an σadprediction modelGradient boosting, Random forest and Elastic Netperformance were checked through a 32-fold cross-validation. The comparison of predicted againstFigure 3. Flowchart of our proposed approach for modelling and optimization. Note that ns indicates the dataset size andi indicates the number of cycles.Figure 4. Distribution of adhesive joint strength σad (MPa)from the initial dataset of size ns = 32 samples.Sci. Technol. Adv. Mater. 20 (2019) 1014 S. PRUKSAWAN et al.measured σad for each algorithm is shown in Figure 5.A dashed straight line indicates an exact matchbetween the predicted and measured values. TheRandom forest and Gradient boosting algorithmscould capture non-linear relationships among thevariable parameters that cannot be accessed viaa linear regressive model, such as Elastic Net. Theirindicated RMSE and MAE in Figure 5 were averagedover the 32 folds, and the R2 was calculated to evaluatetheir prediction accuracy. A comparison of the accu-racy for each algorithm is shown in Figure 5 (top-right). The Elastic Net model showed the lowest accu-racy of R2, RMSE and MAE, and therefore, was dis-carded. The Gradient boosting model showeda slightly better accuracy than the Random forestmodel in terms of a higher R2 value, and lowerRMSE and MAE values. Hence, the Gradient boostingalgorithm was selected to predict the σad in furthersteps.3.2.2. Active learning and ML model performanceIn Section 3.2.1, the Gradient boosting model wasselected to predict the σad based ondifferent experimentalconditions. The σad of all remaining (256–32) possibleexperimental conditions were predicted and ranked indescending order. The top-5 experimental conditionswith the highest σad were proposed for measurements.The new measurements were re-used in the Gradientboosting model to improve the accuracy. This processfrom the prediction phase to the re-injection phase sum-marizes one cycle of the active learning pipeline. Table 2lists the top-five proposed experiments for each threecycles of active learningwith the corresponding predictedand measured σad. The measured σad in Table 2 that areabove ~20 MPa show that the Gradient boosting modelallows us to classify experimental conditions witha potentially high outcome compared with the others.These additional data of high strength adhesives are verybeneficial to further maximization with BayesianFigure 5. Distribution of predicted versus measured adhesive joint strength σad (MPa) from successive test sets used in the 32-foldcross-validation using different ML algorithms: (a) Gradient boosting, (b) Random forest and (c) Elastic Net. A dashed straight lineindicates equal measured and predicted σad. Hyperparameters used for these runs are shown in Table S4 of the Supplementalmaterial.Sci. Technol. Adv. Mater. 20 (2019) 1015 S. PRUKSAWAN et al.Table 2. Proposed experimental conditions during the active learning stage via Gradient boosting with related experimentalresults. Predicted adhesive joint strength σad (MPa) was calculated by averaging the predictions over the 32 folds via cross-validation. Hyperparameters used for these runs are shown in Table S4 of the Supplemental material.Proposed experimental conditionCycle Rank MWE (g/mol) MWC (g/mol) r Tcure (°C) Predicted σad (MPa) Measured σad (MPa)Initial dataset(ns = 32 samples)1 2900 400 1.00 210 25.6 ± 0.9 24.0 ± 1.12 3800 400 1.00 210 25.5 ± 1.4 21.2 ± 1.23 370 400 1.00 210 25.4 ± 1.2 29.0 ± 0.14 1650 400 1.00 170 25.4 ± 1.2 22.4 ± 1.75 1650 400 1.00 210 25.4 ± 1.2 27.3 ± 1.61(ns = 37 samples)1 370 400 1.25 210 25.4 ± 1.1 27.8 ± 0.52 370 400 1.25 170 25.3 ± 1.1 28.3 ± 0.93 370 400 1.50 210 25.1 ± 1.9 23.1 ± 0.44 370 400 1.50 170 25.0 ± 1.9 22.4 ± 1.85 1650 400 1.25 210 24.9 ± 0.5 24.6 ± 0.02(ns = 42 samples)1 2900 400 1.00 170 23.9 ± 0.4 20.5 ± 3.52 370 230 1.00 210 23.7 ± 1.1 24.6 ± 2.03 370 230 1.00 170 23.7 ± 1.1 27.9 ± 0.24 1650 400 1.25 170 23.5 ± 1.4 23.5 ± 1.05 2900 400 1.25 210 23.4 ± 1.1 25.7 ± 0.9Figure 6. Correlation scatter plots (test data) of predicted and measured adhesive joint strength σad (MPa) using different datasetsizes ns (samples): (a) initial dataset, (b) cycle 1, (c) cycle 2 and (d) cycle 3. Grey and orange dots indicate data from existing andnew measurements, respectively, at cycle i. All proposed experimental conditions are summarized in Table 2. Hyperparametersused for these runs are shown in Table S4 of the Supplemental material.Sci. Technol. Adv. Mater. 20 (2019) 1016 S. PRUKSAWAN et al.optimization. Without this strategy, the use of Bayesianoptimization on the initial dataset with the model inFigure 6(a) would outcome less relevant proposals andwouldn’t be beneficial compared to a simple randomsampling. In addition, 90% of proposed experimentsrequire a MWC of ~400 g/mol, a high Tcure of 170 and210°C, and an excess of amine (r > 1), when theMWE canevolve widely across its specific range (see Table 1).Therefore, a high σad can be achieved regardless of theMWE. However, it is premature to make any furtherconclusion about optimal adhesive preparations beforethe r and Tcure parameters are relaxed in Section 3.3.To show the improvement in accuracy of theGradient boosting model along the cycles of activelearning, Figure 6 presents scatter plots of the predictedversus measured σad from the initial dataset to the lastcycle. Grey and orange dots indicate existing and newmeasurements, respectively, at each cycle. As expected,an increase in the dataset size improves the correspon-dence between the predicted and measured σad as sum-marized in Figure 7 for the corresponding R2, RMSEand MAE for the predictions of the σad at each cyclebeginning with the initial dataset. The R2 increases, andthe RMSE andMAEdecrease gradually with an increasein ns. For a dataset of 47 samples, the Gradient boostingmodel reaches an R2, RMSE and MAE of 0.85, 4.0 MPaand 3.0 MPa, respectively. An improvement of 25%,~26% and ~19%, respectively, was achieved comparedwith the Gradient boosting model that had trained onlyon the initial dataset. At cycle three of this active learn-ing pipeline, the prediction performance of theGradientboostingmodel became comparable with the maximumstandard deviation from experiments (3.5 MPa).Therefore, the active learning procedure was stoppedat this stage and the Gradient boosting model was keptfixed based on existing data.3.2.3. Bayesian optimizationAt the Bayesian optimization stage (see Section 2.4.4),theMWE andMWCwere kept fixed at the four differentvalues used in Table 1, whereas the r and Tcure werevaried in steps of 0.01 and 1°C, respectively. The sug-gested experimental conditions with the highestexpected improvement from Bayesian optimizationwere selected, and a series of experiments was con-ducted starting from ranking number 1 (Table 3). Thenewhighest σad of 35.8MPawas observed. The σad valuewas considered as a very high σad compared with pre-vious studies on epoxy-aluminium joints, whichreported a typical σad range from ~10 MPa up to25 MPa [11,28]. Furthermore, this σad value was com-parable to the commercial epoxy adhesives likeHuntsman Araldite 2000+ (26 MPa) and 3M Scotch-Weld DP420 (31 MPa) [29,30], characterized by single-lap shear test. For this sample, the 50-kN tensilemachinewas used tomeasure the σad because the sampledid not break under a 10-kN applied force, i.e. the failurestress of the adhesive joint exceeded the maximumcapacity of a 10-kN tensile machine. The suggestedexperimental conditions from Bayesian optimizationshowed that a low MWE and a high Tcure werea promising condition to reach a high σad. The MWCand r should be in themiddle of their defined range (seeTable 1). The σad improved for the sample that wasprepared with a slight excess of epoxide because otherconditions (MWE, MWC and Tcure) in the samplesshown in Table 3 were only slightly different. Thislarge improvement in σad indicates the suitable balancebetween strength and flexibility of adhesives [31].Because excess epoxide (lower r than the stoichiometricratio) leads to a higher tensile strength but a lowerflexibility of adhesives [14], an optimum combinationof high strength and good flexibility would be achievedby adjusting the r precisely through Bayesianoptimization.In summary, Figure 8 illustrates the distribution ofσad from the initial dataset alone (grey), after threeactive learning cycles (blue), and after a Bayesian opti-mization (red). The values of σad from the initial datasetwere spread randomly from 0 to 31.9 MPa. In contrast,all samples that followed an active learning cycle exhib-ited a high value of σad (> 20 MPa), and one samplefrom the Bayesian optimization dataset showed anFigure 7. Comparison of the accuracy of the Gradient boostingmodel to predict the adhesive joint strength σad (MPa) fordifferent dataset sizes ns of the dataset.Table 3. Proposed preparations of an epoxy adhesive atBayesian optimization stage with the related experimentaladhesive joint strength σad (MPa).Suggested experimental conditionsRank MWE (g/mol)MWC(g/mol) rTcure(°C)Predictedσad (MPa)Measuredσad (MPa)1 370 400 1.11 199 26.9 28.0 ± 0.72 370 400 1.24 194 26.9 27.4 ± 1.23 370 400 1.30 191 26.9 18.8 ± 1.44 370 400 0.89 209 26.9 35.8 ± 1.1Sci. Technol. Adv. Mater. 20 (2019) 1017 S. PRUKSAWAN et al.exceptionally high σad. The spread in measurementsfrom the Bayesian optimization was wider than thatfrom the active learning cycles. A Bayesian optimiza-tion balances the exploitation (surrogate model predictsa high objective) and exploration (sampling of regionswhere the prediction uncertainty is high) of the epoxyadhesive preparation parameters space, where ouractive learning pipeline based on the ML model pre-dictions only exploits the parameters. These resultsdemonstrate the potential of our method for the designand development of new functional materials when theinitial number of samples is reduced compared with thenumber of combinations of free parameters involved.3.3. Interpretation of ML model for adhesivedesignWe explore the influence of epoxy network structure onσad of the joints through the developed ML model(Figure 9). The epoxy network structure was alteredby varying the MWE, MWC, r and Tcure used to cross-link the adhesives. The predicted σad were calculated byaveraging the predictions over the 47 folds of cross-validation and their standard deviations are shown.The plots show a step change in the value of predictedσad. This step change corresponds to the decision-treeformation process in Gradient boosting within limiteddiscrete input values. The experimental σad values wereplotted with their standard deviations. Although thebulk properties of various epoxy network structureshave been studied extensively and reported previously[11], no comprehensive study focuses on their adhesivejoint property, which is related more closely to thepractical application of epoxy resin.As shown in Figure 9(a), the σad decreased slightly(i.e. less than 5 MPa) with an increase in MWE. Thisslight decrease of σad for a high-MW epoxy resin mostlikely originates from an increased epoxy-resin viscos-ity. Because a higher MWE possesses a higher viscosity,it is observed in the experiment that an adhesive that isprepared from a solid-type epoxy resin (MWE = 1650,2900 and 3800 g/mol) cannot spread well on the sub-strates, which results in a lowered adhesion strengthbetween the adhesive and the substrates.In the case of a curing agent, the σad first increaseswith an increasing MWC, reaches a maximum of~26MPa at ~380–1200 g/mol, and then decreases shar-ply to less than 5 MPa (Figure 9(b)). The increase in σadcould be attributed to an enhanced flexibility within thecrosslinked epoxy-amine network when the aminechain length is increased [13]. However, at a higherMWC (> 1200 g/mol), the adhesive is too flexible toresist a high applied force, which results in a low σad. Asobserved in the experiment, the adhesives that wereprepared with a MWC above 2000 g/mol are extremelysoft, which implies a much lower adhesive elastic mod-ulus and tensile strength. This result is consistent withprevious studies in which the elastic modulus of curedepoxies was reduced significantly from 2 GPa to1.9 MPa when the molecular weight of Jeffamine wasincreased from 400 to 2000 g/mol [32,33].For the amine-to-epoxide ratio effect, σad increasesfirst then it reaches a maximum, and then decreasesslightly with an increase in r (Figure 9(c)). The high σadfrom ~0.87 to 1.37 is attributed to the appropriate bal-ance between flexibility and strength of adhesives [14,34].The σad increased gradually as Tcure increased andappears to be almost constant for a Tcure of 150–210°C(Figure 9(d)). Fully cured adhesives were obtained ata Tcure of 150–210°C because there is no significantdifference in σad and because of the physical appearancein this range. The low σad region at a lowTcure between 90and 150°C may indicate incomplete curing because theincomplete network structures of a partially cured adhe-sive result in a remarkably lower elastic modulus [35].The experimental evidence shows that an adhesive curedat 90°C is relatively soft and/or the resin componentremains liquid (uncured) compared with that cured at170–210°C.4. ConclusionsThe design of experimental techniques combined withan active learning pipeline and Bayesian optimizationwas proposed to predict and optimize the adhesivejoint strength (σad) of an epoxy-amine adhesive com-prised of bisphenol A-based epoxy resin and amine-terminated poly(propylene glycol) curing agent with var-ious molecular weights (MWE, MWC), mixing ratios (r)and curing temperatures (Tcure). From an initial datasetof only 32 measured σad with related epoxy-amine mix-ture preparation parameters {MWE, MWC, r, Tcure}, ouractive leaning pipeline was able to propose preferredFigure 8. Distribution of adhesive joint strength σad (MPa)measurements from the initial dataset alone (grey), afteractive learning cycles (blue) and after Bayesian optimization(red). Lines are used to guide the eye only.Sci. Technol. Adv. Mater. 20 (2019) 1018 S. PRUKSAWAN et al.experimental conditions to build a predictive Gradientboosting model of σad with an experimental-like errorlevel, and to maximize the likelihood to design epoxy-amine adhesives with a high σad, along three cycles ofactive learning. An extremely high σad of 35.8 ± 1.1 MPawas achieved using the experimental conditions thatwere refined by Bayesian optimization. Because the pre-diction model was built using a very small dataset(e.g. < 50 samples), and the efficiency of prediction wasreasonably high (e.g. R2 > 0.8), our proposed approach isforeseen to reduce materials design and developmenttime and cost, especially for which experimental datasetsare rare.Our predictivemodel also provides a physical under-standing of adhesive systems over a wide range ofparameters for preparation. A quantitative analysisindicates that high-strength adhesives require a MWCof ~380–1200 g/mol, an r of ~0.87–1.37 and a Tcureabove 150°C. However, a MWE of 370–3800 g/mol hasa slight effect on σad. Qualitatively, we emphasize that:(i) a balance between flexibility and strength of adhe-sives (by adjusting MWC, r) influences σad significantly,(ii) a complete curing (high Tcure) is compulsory toobtain a high σad and (iii) an increase in epoxy viscosity(MWE) degrades the adhesive–substrate adhesion.Future work on this topic should target multiple-objective optimization of an adhesive (e.g. adhesivejoint strength, glass transition temperature and che-mical resistance). Other molecular weights or epoxyresin and curing agent types can be added to thedataset to increase the design freedom of advancedhigh-strength adhesives. From an experimental per-spective, structural and mechanical characterizations(e.g. crosslink density, dynamic mechanical analysisand fracture morphology) of the extremely high-strength adhesive achieved in this study are essentialFigure 9. Predicted adhesive joint strength σad (MPa) as a function of (a) molecular weight of epoxy resin MWE (g/mol), (b)molecular weight of epoxy resin MWC (g/mol), (c) amine-to-epoxide ratio r and (d) curing temperature Tcure (°C). The predicted σadwas calculated by averaging the predictions over the 47 folds of cross-validation. The blue line consisted of the predicted values ofσad (blue). Triangles represent experimental results (red).Sci. Technol. Adv. Mater. 20 (2019) 1019 S. PRUKSAWAN et al.and will be conducted to elucidate the source of theexceptional properties, to guide experimentalists inthe design of an epoxy-amine system for adhesive-bonding applications.AcknowledgmentsS.P. acknowledges the Research Fellowship of the NIMS JuniorResearcher (2018–2019). We are grateful to Dr. SusumuTakamori of NIMS for his instrumental support on the 50-kN universal tensile testing machine.Disclosure statementNo potential conflict of interest was reported by the authors.FundingThis work was supported by the Japan Science andTechnology Agency [Mirai Program JPMJMI18A2].References[1] Rahman Minar M, Naher J. Recent advances in deeplearning: an overview: arXiv.org; [cited 2019 April16]. Available from: https://arxiv.org/abs/1807.08169[2] Butler KT, Davies DW, Cartwright H, et al. Machinelearning for molecular and materials science. Nature.2018;559(7715):547–555.[3] Pilania G, Wang C, Jiang X, et al. Accelerating mate-rials property predictions using machine learning. SciRep. 2013;3:2810.[4] Xue D, Balachandran PV, Hogden J, et al. Acceleratedsearch for materials with targeted properties by adap-tive design. Nat Commun. 2016;7:11241.[5] Yuan RLZ, Balachandran PV, Xue D, et al.Accelerated discovery of large electrostrains inBaTiO3-based piezoelectrics using active learning.Adv Mater. 2019;30(7):1702884.[6] Balachandran PV, Kowalski B, Sehirlioglu A, et al.Experimental search for high-temperature ferroelec-tric perovskites guided by two-step machine learning.Nat Commun. 2018;9:1668.[7] Yuan R, Tian Y, Xue D, et al. Accelerated search forBaTiO3-based ceramics with large energy storage atlow fields using machine learning and experimentaldesign. Adv Sci. 2019;1901395.[8] Ren F, Ward L, Williams T, et al. Accelerated discov-ery of metallic glasses through iteration of machinelearning and high-throughput experiments. Sci Adv.2018;4(4):eaaq1566.[9] JhaA, ChandrasekaranA, KimC, et al. Impact of datasetuncertainties on machine learning model predictions:the example of polymer glass transition temperatures.Modell Simul Mater Sci Eng. 2019;27(2):24002.[10] Bojarski AD, Álvarez CR, Puigjaner L. Dealing withuncertainty in polymer manufacturing by using linearregression metrics and sensitivity analysis. In:Jeżowski J, Thullie J, editors. 19th european sympo-sium on computer aided process engineering,Cracow, Poland. Vol. 26; Elsevier; 2009. p. 725–730.[11] Kinloch A. Adhesion and adhesives: science and tech-nology. 1st ed. Basel: Springer; 1987.[12] Prozonic TM. The effect of epoxy network structureon toughenability [dissertation]. Pennsylvania (PA):Lehigh university; 2012.[13] Nakka JS, Jansen KMB, Ernst LJ. Effect of chain flex-ibility in the network structure on the viscoelasticityof epoxy thermosets. J Polym Res. 2011;18(6):1879–1888.[14] Li YF, Xiao MZ, Wu Z, et al. Effects of epoxy/hard-ener Stoichiometry on structures and properties of adiethanolamine-cured epoxy encapsulant. IOP ConfSer Mater Sci Eng. 2016;137:12012.[15] Sinclair JW. Effects of cure temperature on epoxyresin properties. J Adhes. 1992;38(3):219–234.[16] Prolongo SG, Del Rosario G, Ureña A. Comparativestudy on the adhesive properties of different epoxyresins. Int J Adhes Adhes. 2006;26(3):125–132.[17] Comyn J. Adhesion Science. 1st ed. London: RoyalSociety of Chemistry; 1997.[18] Abbott S. Adhesion science: principles and practice.1st ed. Pennsylvania: DEStech Publications; 2015.[19] Olsson F A literature survey of active machine learn-ing in the context of natural language processing.SICS Technical Report. T2009:06; 2009[20] Pruksawan S, Samitsu S, Yokoyama H, et al.Homogeneously dispersed polyrotaxane in epoxyadhesive and its improvement in the fracturetoughness. Macromolecules. 2019;52(6):2464–2475.[21] Cooper BE. Experimental design. In: Cooper BE, edi-tor. Statistics for experimentalists. Surrey: Pergamon;1969. p. 177–193.[22] Keppel G. Design and analysis: A researcher’s hand-book. 1st ed. New Jerse: Prentice-Hall, Inc; 1991.[23] Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in Python. J Mach Learn Res.2011;12:2825–2830.[24] The GPyOpt authors. GPyOpt: A Bayesian optimiza-tion framework in Python; [cited 2019 May 4].Available from: https://github.com/SheffieldML/GPyOpt[25] Refaeilzadeh P, Tang L, Liu H. Cross-validation. In:Liu L, ÖZsu MT, editors. Encyclopedia of databasesystems. Boston, MA: Springer; 2009. p. 532–538.[26] Zhang T, Yu B. Boosting with early stopping: conver-gence and consistency. Ann Statist. 2005;33(4):1538–1579.[27] Packwood D. Theory of Bayesian optimization. In:Packwood D, editor. Bayesian optimization for mate-rials science. Singapore: Springer; 2017. p. 11–28.[28] Meng Q, Araby S, Saber N, et al. Toughening polymeradhesives using nanosized elastomeric particles.J Mater Res. 2014;29(5):665–674.[29] Product information of araldite 2000+ structuraladhesives; [cited 2019 Aug 29]. Available from:https://www.intertronics.co.uk/product/araldite-2000-structural-adhesive/[30] Product information of 3M™ Scotch-Weld™ epoxyadhesive DP420; [cited 2019 Aug 29]. Availablefrom: https://www.3m.com/3M/en_US/company-us/all-3m-products/~/3M-Scotch-Weld-Epoxy-Adhesive-DP420[31] Da Silva LFM. Design rules and methods to improvejoint strength. In: Da Silva LFM, Öchsner A,Adams RD, editors. Handbook of adhesion technol-ogy. Berlin, Heidelberg: Springer; 2011. p. 689–723.[32] Liu L, Wagner HD. Rubbery and glassy epoxy resinsreinforced with carbon nanotubes. Compos SciTechnol. 2005;65(11):1861–1868.Sci. Technol. Adv. Mater. 20 (2019) 1020 S. PRUKSAWAN et al.https://arxiv.org/abs/1807.08169https://github.com/SheffieldML/GPyOpthttps://github.com/SheffieldML/GPyOpthttps://www.intertronics.co.uk/product/araldite-2000-structural-adhesive/https://www.intertronics.co.uk/product/araldite-2000-structural-adhesive/https://www.3m.com/3M/en_US/company-us/all-3m-products/~/3M-Scotch-Weld-Epoxy-Adhesive-DP420https://www.3m.com/3M/en_US/company-us/all-3m-products/~/3M-Scotch-Weld-Epoxy-Adhesive-DP420https://www.3m.com/3M/en_US/company-us/all-3m-products/~/3M-Scotch-Weld-Epoxy-Adhesive-DP420[33] Park J-M, Kim D-S, Han S-B. Properties of interfacialadhesion for vibration controllability of compositematerials as smart structures. Compos Sci Technol.2000;60(10):1953–1963.[34] Alhabill FN, Ayoob R, Andritsch T, et al. Effect ofresin/hardener stoichiometry on electrical behavior ofepoxy networks. IEEE Trans Dielectr Electr Insul.2017;24(6):3739–3749.[35] Gong M, Zhou D, editors. Effect of curing temperatureand curing degree on elastic recovery of conductiveparticles. Asia-Pacific Energy Equipment EngineeringResearchConference; 2015; Zhuhai: Atlantis Press; 2015.Sci. Technol. Adv. Mater. 20 (2019) 1021 S. PRUKSAWAN et al. Abstract 1. Introduction 2. Experimental andML methods 2.1. Materials 2.2. Preparation of adhesive joint specimens and single-lap shear test 2.3. Selection of experimental conditions for the initial dataset 2.4. ML method 2.4.1. Data pre-processing and splitting 2.4.2. ML algorithms 2.4.3. ML model and active learning 2.4.4. Bayesian optimization 3. Results and discussions 3.1. Experimental results from the initial dataset 3.2. ML model 3.2.1. Assessment and selection of an σ<sub>ad</sub> prediction model 3.2.2. Active learning andML model performance 3.2.3. Bayesian optimization 3.3. Interpretation ofML model for adhesive design 4. Conclusions Acknowledgments Disclosure statement Funding References