# Fileset [Automatic_extraction_of_materials_and_properties_from_superconductors_scientific_literature-v1.pdf](https://mdr.nims.go.jp/filesets/9d16673e-5ee2-4732-a0ae-344b28c4209b/download) ## Creator [Foppiano Luca](https://orcid.org/0000-0002-6114-6164), [Baptista de Castro, Pedro](https://orcid.org/0000-0001-8673-2840), [Ortiz Suarez, Pedro](https://orcid.org/0000-0003-0343-8852), [Terashima, Kensei](https://orcid.org/0000-0003-0375-3043), [Takano, Yoshihiko](https://orcid.org/0000-0002-1541-6928), [Ishii, Masashi](https://orcid.org/0000-0002-7721-9735) ## Rights In Copyright[In Copyright](http://rightsstatements.org/vocab/InC/1.0/) ## Other metadata [Automatic Extraction of Materials and Properties from Superconductors Scientific Literature ](https://mdr.nims.go.jp/datasets/e269bb03-6a2a-4b29-b368-ce253c961337) ## Fulltext RESEARCH PAPERAutomatic Extraction of Materials and Properties fromSuperconductors Scientific LiteratureLuca Foppianoa, Pedro Baptista de Castrob, Pedro Ortiz Suarezc, KenseiTerashimab, Yoshihiko Takanob, Masashi IshiiaaMaterial Database Group, MaDIS, NIMS, Tsukuba, JP; bNano Frontier SuperconductingMaterials Group, MANA, NIMS, Tsukuba, JP; cData and Web Science Group, University ofMannheim, Mannheim, DEARTICLE HISTORYCompiled September 13, 2022ABSTRACTThe automatic extraction of materials and related properties from the scientific lit-erature is gaining attention in data-driven materials science (Materials Informatics).In this paper, we discuss Grobid-superconductors, our solution for automatically ex-tracting superconductor material names and respective properties from text. Builtas a Grobid module, it combines machine learning and heuristic approaches in amulti-step architecture that supports input data as raw text or PDF documents.Using Grobid-superconductors, we built SuperCon2, a database of 40324 materialsand properties records from 37700 papers. The material (or sample) information isrepresented by name, chemical formula, and material class, and is characterised byshape, doping, substitution variables for components, and substrate as adjoined in-formation. The properties include the Tc superconducting critical temperature and,when available, applied pressure with the Tc measurement method.KEYWORDSmaterials informatics, superconductors, machine learning, nlp, tdmCLASSIFICATION: Machine learning, Text mining, NLP1. IntroductionIn recent years, with the creation of computational databases, such as the MaterialsProject (MP) [1] and the Open Quantum Materials Database (OQMD) [2], and thenexperimental data repositories such as NIMS MDR (http://mdr.nims.go.jp) [3],focus has been steadily shifting towards a data-driven design of materials, which isoften called Materials Informatics (MI). Such an approach is expected to acceleratethe exploration of functional materials because it is not limited to the intuition orexperience of very little genius researchers. In this new paradigm, the efficient use ofdata to guide experiments and material property prediction through the use of machinelearning methods takes center stage. For example, data-driven methods have beenused to search/design magneto-caloric materials [4–6], photo-catalysts for hydrogensplitting [7], thermoelectrics [8], and superconductors [9]. In such a data-driven search,Corresponding authors: Luca Foppiano (luca@foppiano.org) and Masashi Ishii (ISHII.Masashi@nims.go.jp)one of the most important keys lies in the availability of the data, which should atleast should consist of compositions of materials and their physical properties. In thespecific case of superconductivity, most of the data-driven works [9–11] rely on a singledatabase: SuperCon (http://supercon.nims.go.jp).SuperCon is a structured database of superconductor materials and properties;itwas developed at the National Institute for Materials Science (NIMS) in Japan. Atthe time of writing this paper, SuperCon contained about 33000 inorganic and 600organic materials and is the “de-facto” standard in data-driven research for supercon-ductors materials (about 4400 articles contain the mention “SuperCon database” inGoogle Scholar). However, the SuperCon harvesting process is currently fully manual“from scratch”: humans have to read the human-readable printed matter such as PDFdocuments and enter the information into the system. The efficiency is directly pro-portional to the number of available human curators. Considering the cost of databaseconstruction, it is necessary to consider an assisted or alternative system that improvesthroughput while ensuring data quality equivalent to that of manual extraction.As a solution, we are developing a hybrid data extraction method from scientificliterature that combines automation using text data mining and manual curation.The automated system extracts and formats potential data and proposes them to thecurator as “pre-cooked” structured data by (1) highlighting the relevant entities on theoriginal document and (2) pre-filling the extracted information in a tabular format.In building the automated part of this hybrid system (training, evaluation) we usedSuperMat, an annotated linked dataset from scholarly documents in superconductorresearch, which we recently constructed [12].In this work, we present Grobid-superconductors: a system that automatically ex-tracts structured information related to superconductor materials and properties fromscientific literature. The tool is a specialised module of Grobid (Generation of Biblio-graphic Data) [13], a machine learning library designed to parse and structure scientificdocuments. Grobid provides an open-source platform for building specialised mod-ules including astronomical entities recognition[14], dictionaries [15], software men-tions [16], and physical measurements extraction [17]. Grobid provides several built-infeatures including access to PDF document layout information, citation resolution,bibliographic information consolidation through biblio-glutton [18] (a fast open-sourcereference matching service for CrossRef data), and a diverse set of machine learning(ML) architectures from a fast linear Conditional Random Field (CRF) to the lateststate-of-the-art deep learning implementations.Using Grobid-superconductors and other sub-tools, we established a pipeline toprocess a large number of documents and obtain an automated database of super-conductor materials and properties. We processed 37770 papers from ArXiv (https://arxiv.org) and obtained a database of 40324 records. This new database, namedSuperCon2, can become an automated staging area for SuperCon, when bridged bya curation interface. During the project, there was also an opportunity to focus onproperties that that have recently become of greater interest and that are underrepre-sented in SuperCon. For example, the “pressure” applied to obtain superconductivity(about 20 records in SuperCon), has gained attention because it can radically changethe physical structure of a material. In addition, the “method” used to measure thesuperconducting transition temperature Tc (about 600 records in SuperCon) can beused to semantically recognise multiple Tc’s obtained from the same material or sample(e.g., distinguish calculated and experimental values of Tc).22. Grobid-superconductorsWe developed Grobid-superconductors as a Grobid module following principles (multi-step, sentence-based, full-text-based) discussed in a previous preliminary study [19].Grobid has several advantages: 1) it can be integrated with pdfalto (https://github.com/kermitt2/pdfalto), a specialised tool for converting PDF to XML, which mit-igates extraction issues such as the resolution of embedded fonts, invalid characterencoding, and the reconstruction of the correct reading order, 2) it allows access toPDF document layout information for both machine learning and document decora-tion (e.g., coordinates in the PDF document); and, 3) it provides access to a set ofhigh-quality, pre-trained machine learning models for structuring documents. Grobid-superconductors is structured as a three-steps process illustrated in Figure 1 anddescribed in the Sections 2.1, 2.2, and 2.3.Abstract versus full-text At the time of writing this paper, we are aware of relatedworks that utilise text from abstracts as training data for machine learning. The mainreason for using abstracts is that they are usually freely available as text [20], and con-tain condensed information [21,22]. Accurately parsing the full-text presents more chal-lenges, however, but they are mitigated by Grobid and, the full-text contain a broaderrange of information, including the sample preparation process, negative results (e.g.,absence of superconductivity for certain samples), and background information (e.g.,reports on other materials from referenced works). Thus, grobid-superconductors isbuilt to support full-text documents.Paragraphs versus sentences Another question related to natural language pro-cessing (NLP) is whether to use sentence-based or paragraph-based text. While para-graphs can be extracted as part of the layout of PDF documents, obtaining sentencesadds an additional step in which text is processed with a sentence segmenter. How-ever, sentences are almost always shorter by definition, and in deep learning, this hasadvantages. In training and prediction, sentences will likely be shorter than the “maxsequence length” limitation (e.g., 512 tokens for transformers). During training, sen-tences also use less memory and allow us to train models with a larger “batch size”,which has been shown to improve efficiency and obtain better results [23].We chose to use sentence-based text in Grobid-superconductors after performingpreliminary experiments on our tasks typologies, but on a smaller scale. For the NamedEntities Recognition (NER) task we trained and evaluated a sequence labelling modelfor each version (paragraph-based and sentence-based) on four annotated documents(3/1 document partition for training/evaluation) from SuperMat [12]. As indicated inTable 1, the F1-score increased by 17.94 percentage points when the sentence-basedtext was used.For the Entity Linking (EL) task, we want to maximise precision. In our previouswork [24] we noticed that limiting linking entities within the same sentence (versusparagraph) would obtain higher precision (68.7% versus 57%) at the expense of lowerrecall (6.5% versus 10.7%), and F1-score (11.87% versus 18.01%). Therefore, in bothour tasks we found evidence that a sentence-based dataset is more beneficial thanparagraph-based dataset.32.1. Document structuring and pre-processingIn the first step of our process, the PDF document is converted into an internal modelbased on a list of text statements, tokens, and features. The input document is pro-cessed using the Grobid original models, where we apply customised processes fordocument header and content. We select a subset of bibliographic information fromthe header: title, authors, DOI, publisher, journal, and year of publication, and weconsolidate them via Grobid to match the publisher’s quality (even by processing the“preprint version” of the publication). The superconductors entities extraction is ap-plied to the content, only on relevant text items: title, abstract, text content frombody or annexes, text content from figure and table captions (Figure 2).We use the collected reference markers (also called reference callouts) from the textas features for improving the paragraph segmentation in sentences: the segmentationis cancelled if the end of sentence falls within the boundaries of a reference marker.For example, a sentence containing a reference in the form “Foppiano et. al.” may bemistakenly segmented in the middle at the token “et.”.2.2. Named Entity RecognitionIn the second step, the Named Entity Recognition (NER) task is performed on thepreviously extracted text.2.2.1. OverviewAs illustrated in Figure 3 the “superconductor parser” extracts the mainsuperconductor-related information by aggregating the resulting entities from twoML models. The Superconductors ML model was developed based on the SuperMatschema [12], and the Quantity ML model was developed in a separated Grobid modulefor measurement extraction [17] and the output is limited to only temperatures andpressures. Overlapping entities are merged, exacted duplicates are removed, and thelargest entities (in terms of string length) are preserved. The resulting entities aresummarised in Table 2.Entities of type , which may contain mixed heterogeneous information,are passed in the cascade to the “Material parser” which aggregates ML and othertools. First, the entity is passed through a Material ML model to segment and iden-tify its content (Table 3). Then, different processes are applied, depending on whichinformation is available. These processes include the following:• Formulas are decomposed into a structured composition. We identify eachelement-stoichiometry pair (e.g., “O”: 7.0) using mat2chem [20] and Pymat-gen [25]; if only the material name is available, we lookup its formula (e.g.,hydrogen to H ),• Using heuristics, we classify the formula by assigning multiple classes as theyare understood from superconductor researchers, for example cuprate, oxides,alloys, etc.• Using the variables and values extracted, we substitute them into partial formu-las. For example, in La 4 Fe 2 A 1-x O 7 (A=Mg,Co; x=0.1,0.2), we substi-tute A and x using their parsed values, and applying permutations, we obtainfour resolved formulas: La 4 Fe 2 Mg 0.9 O 7, La 4 Fe 2 Mg 0.8 O 7, La 4Fe 2 Co 0.9 O 7, and La 4 Fe 2 Co 0.8 O 7.4Finally, after all entities are extracted, the post-processing aggregates different men-tions of the same materials using the parsed formulas at the document-level. For ex-ample, formula with partial substitutions such as La 2 Fe 1-x O 7 (x = 0.1, 0.2)will be aggregated with materials like La 2 Fe 0.9 O 7 appearing in other sectionsof the same document.2.2.2. Machine Learning studyIn this section we discuss the novel ML models we have trained for extracting spe-cialised entities: the Superconductor ML model and the Material ML model (Figure 3).SuperMat [12], our training dataset, contains 164 papers as of the time of writing andis composed of annotated full-text and layout features from PDF documents.For both ML models we trained and evaluated the following four architec-ture/implementations: linear CRF (CRF), bidirectional LSTM with CRF [26](BidLSTM CRF), bidirectional LSTM with CRF with Features [26] (the sameas (BidLSTM CRF) with an additional input channel for features; BidL-STM CRF FEATURES), and SciBERT [27] using a CRF as the activation layer (Scib-ert).The ML models are interfaced by Grobid, which uses the Wapiti[28] implementationfor linear CRF, and DeLFT (Deep Learning For Text) [29] for deep learning models.The architectures CRF and BidLSTM CRF FEATURES make use of the orthogonalfeatures we have summarised in Table A5.Superconductor ML modelHoldout set The holdout set evaluation consists in using a fixed part of a datasetfor validation. The selection must be performed to reproduce the same distribution ofentities of the original dataset. We assembled the holdout set by manually selecting 32documents (24%) from SuperMat, making sure they had a similar ratio of examples,entities and unique entities with the remaining 76% (132 documents) which was usedas training set (Figure 4a). Maintaining the same rate for entity type distributionbetween the two sets was more challenging: on average, we obtained about 15-18% oflabels of each type in the holdout set (Figure 4b), except for the label(23%).We defined the “out-of-domain” ratio as the number of unique entities from theholdout set that were not in the training set. The holdout set “out-of-domain” ratio wason average around 72%, which challenge the model generalisation (every 100 entitiesin the holdout set, 72 were never seen before during training). Most of the labelshad an “out-of-domain” ratio above 50% (Figure 5); , the most importantlabel, had the highest ratio (82%) while and have the lowest(25% and 33%). The low ratio of can be explained by their low entityvariability (11.44%).Positive sampling We trained the model with positive sampling by removing theexamples without entities (negative examples, Figure 4a). This approach provided animprovement of 2% in both precision and recall as compared to the result withoutsampling when testing against the holdout set. Additional experiments with activeand random sampling [16] with ratios of negative examples of 0.1, 0.25, 0.5 and 1.0did not provide stable evidence suggesting scoring improvements when testing against5the holdout set.Evaluation The best results were obtained by Scibert with an F1 of 77.03% and arecall of around 80.69% (Table 4). The features did not provide any improvements withRNN models: BidLSTM CRF and BidLSTM CRF FEATURES resulted in the sameF1 score. This result comes as a surprise because features such as superscript/subscriptwere expected to be determinants for recognising material sequences.The label had the lowest performance scores in all architectures. Webelieve that 274 training examples are not a sufficient large number considering thatpressure expressions can be dependent on the context because they can refer to differ-ent types of pressures (e.g., annealing pressure). The label with the highest score was, with F1 values of 80.77% and 78.06% for Scibert and BidLSTM CRF,respectively. In addition, had the highest “out-of-domain” ratio in theholdout set (greater than 75%, Figure 5) and the highest “label variability” (the ratiobetween unique entities and total entities, about 42%), which suggests that the modelrecognises correctly materials that has not been “seen” during the training. On theother hand, the label, which has lower “label variability” (around 11%)and a low “out-of-domain” ratio, had an F1 score of 66.56% with Scibert and 65.92%with BidLSTM CRF. For , the CRF outperformed the other architectures (F1score of 83.96%), especially Scibert (78.35%). This outcome can be explained by theextremely low variability (12.69%) of entities labelled as .Scibert shows good generalisation capacity for unseen examples or examples ap-pearing in different contexts. For example, in Figure 6a, only Scibert correctly extracts“above 100K”, while CRF misses it completely and BidLSTM CRF misses “above”.In the training data, “above 100K” is not present, but “below 100K” and “1̃00K”are present, and several other entities contain the token “above” and Scibert can un-derstand that the token “above” is relevant to the temperature. In a second example(Figure 6b), only Scibert can correctly extract “W-C nanowire” which is not presentin the SuperMat training data. Unfortunately, we cannot check whether “above 100K”or “W-C nanowire” are also present in the dataset used in the pre-train of SciBERTby their authors [27] because the data are not available.Material ML model To train the Material ML model we created a special datasetwith an additional layer of labels (Table 3), which included the material informationrepresented by entities annotated as in the SuperMat documents.Holdout set In this model we created an independent holdout set because the man-ual annotation work is performed on smaller chunks of text and requires less effortthan annotating sentences as when we developed SuperMat. We used material dataextracted from a dataset of 500 documents (500-papers) from three publishers: Amer-ican Institute of Physics (AIP), American Physical Society (APS) and Institute ofPhysics (IOP) [24]. The resulting holdout set has a average coverage greater than 25%(Figure 7) and an average “out-of-domain” ratio of 83.93% (Figure 8).Evaluation Scibert obtained the best results, with F1 at 84.15% (Table 5). The in-clusion of features in the BidLSTM CRF architecture only improved results by lessthan 1% (from 83.13 to 83.76%). The label did not perform well withany architecture, most likely because it is too generic (Table 3), and the content is tooheterogeneous. Another label, has only one-third of the training exam-6ples of but obtained results that were three times higher with Scibert,suggesting that should be split into separate and more homogeneouslabels.2.3. Entity LinkingEntity linking (EL) links materials and their corresponding properties.We use a rule-based algorithm, but there are other approaches such as the use ofdependency parsing [30–33]. We did not use these because it was difficult to find asuitable dependency parser for scientific texts, and complementary methods based oncomplex rule sets were needed to compensate for the poor performance of the parser.In our algorithm, pairs of entities are linked focusing on three types of link:• material-tcValue: The link between a material and its corresponding Tc.• tcValue-pressure: The link between Tcand its related critical pressure.• me method-tcValue: The link between Tcand its corresponding measurementmethod.Entities of type are pre-processed through a classifier that establisheswhether or not they temperatures related to the superconductivity. This is used toexclude other temperatures (e.g., annealing, transition, Curie) which might be incor-rectly extracted by the previous step. This rule-based classifier combines the extractedentities of Tcexpressions (label ) with a set of predefined standard terms. If atemperature is not considered a Tc, it is excluded from the list of possible linkingcandidates.Two scenarios are considered. First, if entities to be linked in the sentence areonly two they are linked automatically, else further rules are applied. If the word“respectively” appears in the sentence, we apply “order-linking”. For example, considerthe following sentence:P-or Ba-122 and Co-doped Ba-122 have lower Tc’s of about 30 K and 24 K, respectively,which makes helium free operation questionable.It contains the word “respectively”, and by applying “order-linking”, P-or Ba122 isassigned to 30 K and Co-doped Ba-122 to 24 K.If the word “respectively” does not appears in the sentence, we apply “distance-linking” which works by defining the distance measurement d as a value calculated asthe numbers of characters between the centroid of each entity. Entities surrounded byparenthesis are expanded to the whole parenthesis, and its centroid is updated. As anexample, in the sentenceWe tested two materials MgB2 (Tc = 39 K) and FeSe (Tc = 16 K).39 K is closer to FeSe (d=10) than to MgB2 (d=11). In this example, however, bothtemperatures entities would be expanded to their containing parenthesis (e.g. “39 K”to “(Tc = 39 K)”. In this case the centre of the entity “39 K” is shifted toward theleft, from the initial value of 38 to 35 and the distance from MgB2 is reduced fromd=11 to d=8. As a result, the MgB2 entity is correctly linked to “39 K”.The distance calculation is also adjusted with the addition of “penalties” by dou-bling the calculated distance when certain keywords or punctuations (“,”, “.”, “;”,“and”, “but’, “while’, “whereas’, “which”, “although”) appear between two entitiesbecause they represent a logical separation of predicates [34]. In the above example,the distance between 39 K and FeSe would be doubled (d=20) and the link would not7be made.This rule-based linking was evaluated using the linked entities from Super-Mat [12] (Table 6) and is divided considering each link type. The F1 score for thematerial-tcValue was about 80% with a precision of 88.40%. tcValue-pressureF1 score was 3% lower than material-tcValue considering much less data available(support was 118 compared with 726).2.4. End to end evaluationEnd-to-end evaluation (E2EE) measures the capacity of the system from the PDFdocuments until the final linked results. We limited the scope of the E2EE to thetriplet ‘material-Tc-pressure’ which, at the moment, is the backbone upon which thedatabase is built. We performed the E2EE on the “500-papers” dataset where wemanually examined the resulting database as follows: 1) we marked invalid recordsand 2) we identified the cause of failure from a predefined set of five error types(Figure 9):• From table: the extracted text is wrongly extracted from a table. Althoughtable content is ignored, the error rate from the Grobid library is still relevantdue to the lack of training data.• Extraction: entities are not recognised, wrongly recognised, or partially recog-nised.• Quantity extraction: quantity entities (pressure, temperature) are not cor-rectly extracted. We measured this error separately to identify the failure thatcould be shared with the Quantity ML model.• Tcclassification: the temperature is wrongly classified as superconducting Tc.• Linking: given the initial steps were performed correctly, the resulting entitiesare not linked correctly.The E2EE scores are summarised in Table 7. Recall is omitted because it is lessrelevant and difficult to calculate manually. The precision score (micro average) was72.60% for all the subsections, although the error rates of figure captions (59.28%) andunknown subsections (57.14%) were clearly lower than those of the other subsections(> 70%). The ‘unknown‘ subsections indicate that the extracted text’s structure wasnot well identified by Grobid but it was nevertheless aggregated. The overall scoreincreases to 73% when excluding unknown subsections, 75.24% when excluding figurecaptions, and 79.14% when excluding both. Excluding these two subsections will notimpact the amount of text, because both account for less than 20% of the total numberof subsections.The error types are summarised in Figure 10. The most common failures originatefrom Tc classification (40%), Linking (32%), and Extraction (20%). The most commonTcclassification failures are as incorrect recognition of 1) relative values of Tc(e.g., 1 Khigher than material X); 2) values indicating the transition temperature width (∆Tc);3) temperature values that are not Tc, for example, material synthesis temperatures(T ), other critical transition temperatures that are not superconducting (e.g., TCurie);and 4) values of temperature at which there is no superconductivity (e.g., “at 70 Kthere is no superconductivity”). “Linking errors” mainly occur when the text comparerelative values of Tc using materials as the basis for comparison (e.g., “The Tc = 38K is similar to the one of MgB2”). Finally, “Extraction” issues mainly originate from:1) implicit mention of the main material when experimented using different “sub-strates” combination, and 2) mismatches between and which,8by definition, overlap.3. Supercon2We created SuperCon2 by processing 37770 research papers belonging to the cate-gory cond-mat.supr-cond in ArXiv. Currently SuperCon2 contains 40324 records in-cluding 2052 triplets with applied pressure (material-Tc-pressure), and 3602 recordswith explicit measurement method (material-Tc-measurement method). The schemaof SuperCon2 is summarised with examples in Table 8.The data is processed and ingested through the asynchronous Map-Reduce ap-proach [35]. The “extraction task” (Map) processes the PDF documents by access-ing Grobid-superconductors via REST API and stores their processed representationtogether with the original PDF document. Furthermore, the “aggregation task” (Re-duce) reduces the document information into a synthesised tabular format. We storethe processed document representation in JSON format. The processed documents arekept separately and used for displaying the enhanced PDF document (Figure 11). Thepipeline uses a persistence layer for storage and reporting (logger).We built a visualisation interface to exploit the extracted information. Users cansearch in the synthesised tabular data, access the PDF document enriched with theextracted information (Figure 11), and export locally in CSV, TSV and MicrosoftExcel formats.4. ConclusionIn this work, we present our solution for automatically building a database of ma-terials and properties from scientific literature. Our contribution is composed of: 1)Grobid-superconductors, a specialised open-source system that processes PDF docu-ments combining ML and rule-based methods to extract and link relevant informationin superconductors research; 2) a pipeline allowing large-scale document processing;and 3) a visualisation interface for rapid data exploration, which includes PDF docu-ment information enrichment.We made SuperCon2, a database with 40324 records of superconductors materi-als and properties, including the applied pressure and the Tc measurement method.SuperCon2 is available in text format at https://github.com/lfoppiano/supercon.In the future, we plan to improve our tools by 1) extracting more properties, suchas crystal structure type, space groups type, and lattice structure; 2) training super-vised models for the “Linking step”; and 3) extending the interface to support datacorrection toward efficient curation. We confirmed the good generalisation ability ofthe Scibert architecture for the entity extraction task. Although we hope to obtainbetter results using materials science pre-trained BERT, such as MatSciBERT [36],the gain might be just minimal for relatively larger models [37].AcknowledgementOur warmest thanks to Patrice Lopez, the author of Grobid [13], DeLFT [29], andmany other interesting open-source projects.9Data and code availabilityGrobid-superconductors is available on Github at https://github.com/lfoppiano/grobid-superconductors and the code is released under license Apache 2.0.SuperCon2 is available in text format at https://github.com/lfoppiano/supercon.Competing interestsThe authors declare no competing interests.References[1] Jain A, Ong SP, Hautier G, et al. Commentary: The materials project: A materialsgenome approach to accelerating materials innovation. APL Materials. 2013;1(1):011002.Available from: https://doi.org/10.1063/1.4812323.[2] Kirklin S, Saal JE, Meredig B, et al. The open quantum materials database (oqmd):assessing the accuracy of dft formation energies. npj Computational Materials. 2015;1(1):1–15.[3] Ranganathan A, Matsuda A, Tanifuji M, et al. The Development of an Integrated NextGeneration Data Repository for Materials Science ; 2019. Available from: https://doi.org/10.5281/zenodo.3553963.[4] Bocarsly JD, Levin EE, Garcia CA, et al. A simple computational proxy for screeningmagnetocaloric compounds. Chemistry of Materials. 2017 2;29:1613–1622.[5] de Castro PB, Terashima K, Yamamoto TD, et al. Machine-learning-guided discovery ofthe gigantic magnetocaloric effect in hob2 near the hydrogen liquefaction temperature.NPG Asia Materials. 2020 12;12:35.[6] Court CJ, Jain A, Cole JM. Inverse design of materials that exhibit the magnetocaloriceffect by text-mining of the scientific literature and generative deep learning. Chemistryof Materials. 2021;33(18):7217–7231.[7] Xiong Y, Campbell QT, Fanghanel J, et al. Optimizing accuracy and efficacy in data-driven materials discovery for the solar production of hydrogen. Energy & EnvironmentalScience. 2021;14(4):2335–2348.[8] Iwasaki Y, Takeuchi I, Stanev V, et al. Machine-learning guided discovery of a new ther-moelectric material. Scientific reports. 2019;9(1):1–7.[9] Stanev V, Oses C, Kusne A, et al. Machine learning modeling of superconducting criticaltemperature. npj Computational Materials. 2017;4.[10] Le TD, Noumeir R, Quach HL, et al. Critical temperature prediction for a supercon-ductor: A variational bayesian neural network approach. IEEE Transactions on AppliedSuperconductivity. 2020;30(4):1–5.[11] Hamlin JJ. Superconductivity near room temperature. Nature. 2019;569:491–492.[12] Foppiano L, Dieb S, Suzuki A, et al. Supermat: construction of a linked annotated datasetfrom superconductors-related publications. Science and Technology of Advanced Mate-rials: Methods. 2021;1(1):34–44. Available from: https://doi.org/10.1080/27660400.2021.1918396.[13] GROBID contributors. Grobid [https://github.com/kermitt2/grobid]; 2008–2022.[14] Grobid Astro contributors. Grobid-astro: A machine learning software for extract-ing astronomical entities from scholarly documents [https://github.com/kermitt2/grobid-astro]; 2017–2022.[15] Khemakhem M, Foppiano L, Romary L. Automatic Extraction of TEI Structuresin Digitized Lexical Resources using Conditional Random Fields. In: electronic lexi-10cography, eLex 2017; Sep.; Leiden, Netherlands; 2017. Available from: https://hal.archives-ouvertes.fr/hal-01508868.[16] Lopez P, Du C, Cohoon J, et al. Mining software entities in scientific literature: Document-level ner for an extremely imbalance and large-scale task. New York, NY, USA: Associa-tion for Computing Machinery; 2021. p. 3986–3995. Available from: https://doi.org/10.1145/3459637.3481936.[17] Foppiano L, Romary L, Ishii M, et al. Automatic identification and normalisation ofphysical measurements in scientific literature. In: Proceedings of the ACM Symposium onDocument Engineering 2019; New York, NY, USA. Association for Computing Machinery;2019. DocEng ’19. Available from: https://doi.org/10.1145/3342558.3345411.[18] Biblio Glutton Lookup contributors. Biblio glutton lookup [https://github.com/kermitt2/biblio-glutton]; 2018–2022.[19] Foppiano L, Dieb TM, Suzuki A, et al. Proposal for Automatic Extraction Framework ofSuperconductors Related Information from Scientific Literature. In: The Institute of Elec-tronics, Information and Communication (IEICE) - Services Computing Research Group- 2019, First Research Meeting , The 43rd Joint Meeting of MaDIS Research ExchangeGroup; May; Tsukuba, Japan; 2019. Available from: https://hal.archives-ouvertes.fr/hal-02870896.[20] Kononova O, Huo H, He T, et al. Text-mined dataset of inorganic materials synthesisrecipes. Scientific Data. 2019 Oct;6(1):203. Available from: https://doi.org/10.1038/s41597-019-0224-1.[21] Yamaguchi K, Asahi R, Sasaki Y. SC-CoMIcs: A superconductivity corpus for materialsinformatics. In: Proceedings of the 12th Language Resources and Evaluation Conference;May; Marseille, France. European Language Resources Association; 2020. p. 6753–6760.Available from: https://www.aclweb.org/anthology/2020.lrec-1.834.[22] Court CJ, Cole JM. Magnetic and superconducting phase diagrams and transition temper-atures predicted using text mining and machine learning. npj Computational Materials.2020 Mar;6(1):18. Available from: https://doi.org/10.1038/s41524-020-0287-8.[23] Liu Y, Ott M, Goyal N, et al. Roberta: A robustly optimized bert pretraining approach; 2019. Available from: https://arxiv.org/abs/1907.11692.[24] Foppiano L, Dieb TM, Suzuki A, et al. Proposal for automatic extraction frameworkof superconductors related information from scientific literature. THE INSTITUTE OFELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS. 2019;.[25] Ong SP, Richards WD, Jain A, et al. Python materials genomics pymatgen : A ro-bust open-source python library for materials analysis. Computational Materials Science.2013 2;68:314–319. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0927025612006295.[26] Lample G, Ballesteros M, Subramanian S, et al. Neural architectures for named entityrecognition. In: Proceedings of the 2016 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technologies; Jun.; SanDiego, California. Association for Computational Linguistics; 2016. p. 260–270. Availablefrom: https://aclanthology.org/N16-1030.[27] Beltagy I, Lo K, Cohan A. SciBERT: A pretrained language model for scientific text. In:Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process-ing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Nov.; Hong Kong, China. Association for Computational Linguistics; 2019. p.3615–3620. Available from: https://aclanthology.org/D19-1371.[28] Lavergne T, Cappé O, Yvon F. Practical very large scale CRFs. In: Proceedings the48th Annual Meeting of the Association for Computational Linguistics (ACL); July.Association for Computational Linguistics; 2010. p. 504–513. Available from: http://www.aclweb.org/anthology/P10-1052.[29] DeLFT contributors. Delft [https://github.com/kermitt2/delft]; 2018–2022.[30] Yoshikawa M, Noji H, Matsumoto Y. A* ccg parsing with a supertag and dependencyfactored model. In: Proceedings of the 55th Annual Meeting of the Association for Compu-11tational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics;2017. p. 277–287. Available from: http://aclweb.org/anthology/P17-1026.[31] Tiktinsky A, Goldberg Y, Tsarfaty R. pybart: Evidence-based syntactic transformationsfor ie. In: ACL; 2020.[32] Swayamdipta S, Thomson S, Dyer C, et al. Frame-Semantic Parsing with Softmax-MarginSegmental RNNs and a Syntactic Scaffold. arXiv preprint arXiv:170609528. 2017;.[33] Zhou J, Zhao H. Head-driven phrase structure grammar parsing on Penn treebank. In:Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics;Jul.; Florence, Italy. Association for Computational Linguistics; 2019.[34] Oka H, Yoshizawa A, Shindo H, et al. Automatic extraction of solvent-names on polymer’ssolubility from academic articles. In: The 66th JSAP Spring Meeting 2019; Mar.; Tokyo,Japan. Japan Society of Applied Physics; 2019.[35] Dean J, Ghemawat S. Mapreduce: Simplified data processing on large clusters. CommunACM. 2008 jan;51(1):107–113. Available from: https://doi.org/10.1145/1327452.1327492.[36] Gupta T, Zaki M, Krishnan NMA, et al. MatSciBERT: A materials domain languagemodel for text mining and information extraction. npj Computational Materials. 2022May;8(1):102. Available from: https://doi.org/10.1038/s41524-022-00784-w.[37] Hong Z, Ajith A, Pauloski G, et al. Scholarbert: Bigger is not always better ; 2022.Available from: https://arxiv.org/abs/2205.11342.[38] Gajda G, Morawski A, Rogacki K, et al. Ag-doped FeSe0.94 polycrystalline samples ob-tained through hot isostatic pressing with improved grain connectivity. SuperconductorScience and Technology. 2016 jul;29(9):095002. Available from: https://doi.org/10.1088/0953-2048/29/9/095002.[39] Shibata Y, Nomura S, Ishiguro R, et al. Magnetic field imaging of a tungsten carbide filmby scanning nano-SQUID microscope. Superconductor Science and Technology. 2016 aug;29(10):104004. Available from: https://doi.org/10.1088/0953-2048/29/10/104004.[40] Durajski AP, Szczśniak R. Structural, electronic, vibrational, and superconducting prop-erties of hydrogenated chlorine. The Journal of Chemical Physics. 2018;149(7):074101.Available from: https://doi.org/10.1063/1.5031202.[41] Swain MC, Cole JM. Chemdataextractor: A toolkit for automated extraction of chemicalinformation from the scientific literature. Journal of Chemical Information and Modeling.2016;56(10):1894–1904. PMID: 27669338; Available from: https://doi.org/10.1021/acs.jcim.6b00207.Figures & TablesFigure 1. Processing pipeline for extracting superconductors materials and properties.12Table 1. Results from cross-validation for sentence-based andparagraphs-based text.Label Precision Recall F1Paragraph-based micro avg. 44.44 27.21 33.76Sentence-based micro avg. 48.41 50.00 51.70Figure 2. Grobid-superconductors extraction processes (bibliographic information, superconductor entityextraction and sentence segmentation) within the Grobid cascade data flow.Table 2. Entities extracted by the superconductors parser.Entity (tag) DescriptionMaterial ()Materials and samples names, formulas (including stochiometric for-mulas), substitution variables of values and elements, shape, doping,and substrateClass ()Groups of materials having similar characteristics or common strate-gic compounds that define their natureTcvalue () The value of the superconductor critical temperatureTcexpressions ()Expressions in the text that provide information about the phe-nomenon of superconductivity related to a value, interval or variationof the TcMeasurement method ()Technique used to measure or calculate the presence of superconduc-tivityApplied pressure () Applied pressure when superconductivity is recordedTable 3. Entities extracted by the material parser.Entity (tag) DescriptionName () The canonical name of a material (e.g., hydrogen, PCCO, carbon)Formula ()Chemical formula of the material (e.g., Pr1.869Ce0.131CuO 4-, MgB2,La 2-x Sr x CuO 4)Doping ()Doping ratio and doping materials that are adjoined to the materialname (e.g., Zn-doped, 2% Zn-doped)Shape ()shape of the material (e.g. single crystal, polycrystalline, thin film,powder, film)Substitution variables () Variables that can be substituted in the formula.Substitution values () Values expressed in the doping.Substrate () Substrates as defined in the material nameFabrication ()Additional information that does not belong to any of the previoustags (e.g., intercalated, electron-doped)13Figure 3. Cascade architecture in the NER step. The white rectangles indicate the extracted information(described in Tables 2 and 3).Figure 4. Holdout/training set distribution for (a) general metrics and (b) entity labels; entities and uniqueentities indicate the number of labelled entities with and without value duplicates, respectively, and positiveexamples (+) and negative examples (-) indicate the number of sentences with at least one entity and with noentities, respectively.14Figure 5. Holdout “out-of-domain” rates. The entities from the holdout set that are also in the training setare “in-domain” and the entities that are not in the training set are “out-of-domain”.Figure 6. Examples taken from two sources [38,39] of results from three different architectures: CRF, BidL-STM CRF and, Scibert. The boxes annotating the text represent the extracted entities (material are indicatedin light blue, Tcin green, and Tcexpressions in yellow).Table 4. Evaluation scores (%) for the Superconductor ML model in the four architectures. For the DLarchitecture the results are averaged over 5 runs. Support (Supp) indicates the number of labels in the trainingdata. Values in bold indicate the highest score. P: precision, R: recall.Label CRF BidLSTM CRFBidLSTM CRFFEATURESScibert SuppP R F1 P R F1 P R F1 P R F1 79.74 66.79 72.69 79.01 72.62 75.66 77.84 72.40 74.97 72.95 75.28 74.09 1646 79 72.15 75.42 79.25 76.94 78.06 81.07 75.10 77.94 80.15 81.42 80.77 6943 60.25 68.73 64.21 56.41 79.49 65.92 55.86 80.45 65.90 56.26 81.52 66.56 1883 46.15 29.27 35.82 49.45 58.05 52.53 50.25 60.49 54.36 41.72 52.68 46.51 274 84.36 83.57 83.96 78.61 82.54 80.48 79.19 82.07 80.60 74.46 82.66 78.35 3741 69.8 66.24 67.97 70.36 75.16 72.67 68.95 76.56 72.52 70.90 79.74 75.06 1099All (micro avg) 76.88 72.77 74.77 74.59 77.67 76.09 75.17 76.79 75.96 73.69 80.69 77.03Figure 7. Holdout/training set for the Material ML model: (a) general metrics and (b) entity labels.15Figure 8. Holdout “out-of-domain” rates for the Material ML model. The entities from the holdout set thatare also in the training set are the in-domain, and the entities that are not in the training set are the out-of-domain.Table 5. Evaluation scores (%) of the Material ML model with holdout set. Support (Supp) indicates thenumber of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.Label CRF BidLSTM CRFBidLSTM CRFFEATURESSciBERT SuppP R F1 P R F1 P R F1 P R F1 60.41 55.85 58.04 67.98 62.42 64.95 69.00 62.34 65.43 63.58 62.79 63.16 792 40.00 4.55 8.16 23.61 5.91 9.24 37.33 9.09 14.48 22.51 13.18 16.52 94 80.81 82.29 81.54 82.59 84.14 83.35 83.83 85.14 84.47 84.53 86.56 85.53 6301 72.2 63.75 67.71 76.29 78.76 77.43 74.51 80.38 77.33 77.18 81.86 79.44 1930 90.89 92.51 91.69 90.93 95.79 93.29 90.33 95.74 92.96 89.67 97.20 93.28 809 37.04 6.76 11.43 54.31 32.43 40.44 60.08 33.38 42.82 56.32 41.22 47.59 32 80.21 83.15 81.65 84.81 89.33 86.99 85.16 90.15 87.58 83.14 85.92 84.50 1895 96.85 95.98 96.41 95.19 97.77 96.46 96.32 97.90 97.10 96.22 96.52 96.37 1795All (micro avg) 81.15 78.09 79.59 82.76 83.50 83.13 83.20 84.33 83.76 83.11 85.23 84.15Table 6. Evaluation scores for the Linking. Support (Supp)indicates the number of labels in the training data. Values inbold indicate the highest score. P: precision, R: recall.Relationship type P R F1- Suppmaterial-tcValue 88.40 74.52 80.87 726tcValue-pressure 85.71 71.52 77.98 118me method-tcValue 62.28 65.74 63.96 151Figure 9. Error types in the context of the data flow.16Table 7. Summary of the E2EE evaluation scores. Support indicates the numberof labels in the training data.Subsection Precision SupportTitle 100 2Abstract 80.32 61Paragraph 75.2 623Figure captions 59.28 140Unknown 57.14 21Micro avg. 72.60 847Micro avg. (excl. figures) 75.24 707Micro avg. (excl. unknown sections) 73.00 603Micro avg. (excl. figures and unknown sections) 79.14 657Figure 10. Error type distribution in the E2EE of the 500-papers dataset.Figure 11. Example of a superconductors research PDF document [40] enriched with extracted annotations.Materials information (class, formula) and properties (Tc) are summarised in the information box when theusers click on the highlighted annotated entity in the text.17Table 8. Summary and description of the SuperCon2 schema. “Internal information” is technical infor-mation not accessible to the users.Field name Description ExamplesMaterial informationRawmaterialThe material or sample as it appears in the textName Canonical name of a materialPCCO, PCO, Metal diboride,hydrogen, carbonFormulaMaterial expressed as chemical formula. Thisincludes also formulas with stochiometric variablesPr1.869Ce0.131CuO4 − δ,MgB2, La2−xSrxCuO4DopingDoping ratio and doping materialsthat might be adjoined to the materialOverdoped, underdopded,optimally doped,bulk, pure, 1% Zn, Zn(from Zn-doped XYZ)Shape The shape of the material or the sampleSingle crystal, polycrystal,wire, powder, filmVariables Variables that can be substituted in the formula x = 0, RE=Ln,StClassMaterial classification accordingto the domain-experts taxonomycuprates, oxides, and alloysFabricationAll the information that does notbelong to any of the previous tagsIntercalated,synthesized by MBE method,electron-doped, hole-dopedSubstrate Substrate material described in the raw materialPCCO films ontoPr2CuO4(PCO)/SrT iO3PropertiesCriticalTemperatureSuperconducting critical temperatureAppliedPressurePressure applied when measuringthe superconducting critical temperatureMeasurementMethodMethod for measurement of thesuperconducting critical temperatureMagnetic susceptibility,specific heat, calculation,prediction, resistivityDocument bibliographic informationSection The main body section of the paper Header, body, annexSubsection The secondary segmentation area of the paperParagraph, table caption,figure caption, title, abstractAuthors,Title, DOI,Publisher,Journal, YearBibliographic information of the documentInternal informationHash,TimestampHash calculated on the binary content of the original PDFdocument and the timestamp when the document was processed.18Appendix A. Additional informationTable A1. Holdout/Training set distribution (%) between training andholdout sets for the Superconductor ML model. Positive examples indicatethe number of sentences with at least one entity, and negative examples thenumber of sentences with no entities.training holdout % holdout/trainingdocuments 132 32 24.24%examples 16902 3905 23.10%entities 15586 3112 19.97%unique entities 6699 1372 20.48%positive examples 8380 1776 21.19%negative examples 8522 2129 24.98%Table A2. Holdout/Training set distribution (%) between train-ing and holdout sets on different labels for the Superconductors MLmodel.label training holdout % holdout/training 3741 639 17.08% 1646 271 16.46% 6943 1649 23.75% 1883 355 18.85% 1099 157 14.29% 274 41 14.96%Table A3. Holdout/Training set distribution (%) training and holdoutsets for the Material ML model.training holdout % holdout/trainingexamples 13648 5728 41.97%entities 4512 2817 62.43%unique entities 9268 3292 35.52%19Table A4. Holdout/Training set distribution (%) training and hold-out sets on different labels for the Material ML model.label training holdout % holdout/training 94 44 46.81% 6301 2569 40.77% 809 841 103.96% 32 148 462.50% 1930 949 49.17% 1795 449 25.01% 1895 463 24.43% 792 265 33.46%Table A5. Summary of the features used in the superconductors and material ML models. All under Architectureindicate only BidLSTM CRF FEATURES and CRF.# Feature Model Architecture1 current token all all2 current token lower cased all all3-6 (four features) current token, prefix characters 1 to 4 all CRF7-10 (four features) current token, suffix characters 1 to 4 all CRF11information about capitalisation: first character (INITCAP), all char-acters (ALLCAPS), none (NOCAPS)all all12digits content: all (ALLDIGIT), some digits (CONTAINDIGIT), nodigits (NODIGIT)all all13 (boolean) the token is composed by a single character all all14punctuaction information and normalisation to placeholders: no punc-tuation (NOPUNCT), open or end brackets (OPENBRACKET,ENDBRACKET), various punctuation (DOT, COMMA, HYPHEN,QUOTE), open or close quotes (OPENQUOTE, ENDQUOTE), any-thing else (PUNCT)all all15 Shadow the numbers all CRF16Shadow any characters: “x” for lowercase, “X” for uppercase, “d” fordigitsall CRF17 As the previous but compressed all CRF18 Font name superconductors all19 Font size superconductors all20Font style: standard (BASELINE), superscript (SUPERSCRIPT) orsubscript (SUBSCRIPT)superconductors all21 (boolean) if the token style is bold superconductors all22 (boolean) if the token style is italic superconductors all23(boolean) the token is identified as a chemical compound byChemDataExtractor[41]superconductors all20