# Fileset

[DCMI2024.pdf](https://mdr.nims.go.jp/filesets/447d85ba-21f3-47e7-b6d0-f783718f8bd3/download)

## Creator

[Kosuke Tanabe](https://orcid.org/0000-0002-9986-7223), [Asahiko Matsuda](https://orcid.org/0000-0001-5989-027X), [Masashi Ishii](https://orcid.org/0000-0003-0357-2832)

## Rights

[Creative Commons BY Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/)

## Other metadata

[Cross-searching of datasets by linking repository and vocabulary management in material data platform](https://mdr.nims.go.jp/datasets/9210b5a3-3c69-4c61-b5d0-fcd65d819952)

## Fulltext

NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORMCross-searching of datasets by linking repository and vocabulary management in materials data platformKosuke Tanabe, Asahiko Matsuda, Masashi Ishiihttps://orcid.org/0000-0002-9986-7223National Institute for Materials Science (NIMS), JapanOctober 23rd, 2024DCMI 2024 Toronto Best practices sessionhttps://orcid.org/0000-0002-9986-7223https://orcid.org/0000-0002-9986-7223NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM1• Research data platform for materials science• Developed and operated by National Institute for Materials Science (NIMS), Japan• Consists of several research software and services◦ Materials databases◦ Research data sharing service◦ User authentication / authorizationNIMS Materials Data Platform “DICE”NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM2•Materials Data Repository (MDR)•MDR XAFS DB•MatVoc / MatVoc ExplorerSelected in this presentationNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM• Data repository in the DICE◦ https://mdr.nims.go.jp• released in June 2020• About 1,000 articles and 12,800 datasets as of October 2024• provides facet search UI and REST API• Built on Samvera Hyrax (OSS)• Project leader: Kosuke Tanabe3NIMS Materials Data Repository (MDR)https://mdr.nims.go.jp/NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM• Collection of X-ray absorption fine structure (XAFS) datasets◦ One of the most popular collections in MDR• Includes approximately 2600 XAFS spectra datasets collected by six research institutions in Japan• Project leader: Masashi Ishii• https://doi.org/10.48505/nims.14474MDR XAFS DBhttps://doi.org/10.48505/nims.1447NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM5NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM6NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM7• MatVoc: https://matvoc.nims.go.jp◦ Vocabulary service for materials science◦ Built on Wikibase◦ Includes more than 2400 entities◦ Provides SPARQL endpoint• MatVoc Explorer: https://matvoc.nims.go.jp/explore/en/home◦ Provides browse and search UI◦ Provides a link to MDR and other external databases associated with each vocabulary• Project leader: Asahiko MatsudaMatVoc / MatVoc Explorerhttps://matvoc.nims.go.jphttps://matvoc.nims.go.jp/explore/en/homeNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM8NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM9NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM10Link to MDR and other DICE services associated with the entityLink to Wikipedia and WikidataLink to relative MatVoc entitiesNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORMMetadata schemas aroundMDR and MDR XAFS DB11NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM12◦XAFS Metadata◦DICE common messaging format◦MDR Schema◦DataCite Metadata Schema◦JPCOAR SchemaMetadata (schemas) in MDR and MDR XAFS DBScientific metadataBibliographic metadataNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM• https://github.com/xafs-db/xafs-schema• Metadata schema developed by the Japan XAFS Society◦ https://www.jxafs.org/xafs-database/• Includes domain-specific properties (e.g., instruments, measurement)13XAFS Metadatahttps://github.com/xafs-db/xafs-schemahttps://github.com/xafs-db/xafs-schemahttps://www.jxafs.org/xafs-database/NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM14• https://doi.org/10.48505/nims.3240• Designed for entire datasets and research areas on the data platform DICE• Supports many scientific entities and properties including instruments, specimens and experimental methods• Consists of many deeply-nested properties• “We have defined a common metadata schema for research dataset distribution/exchange/storage, with features for not only system-to-system communication but also description of the datasets. The metadata were initially defined by a XML Schema, and has been converted into JSON Schema. The schema has a complex structure, which includes the dataset's bibliographic metadata as well as description of the dataset from the materials science viewpoint.”DICE common messaging formathttps://doi.org/10.48505/nims.3240NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM15The structure of DICE common messaging formatMandatoryDomain-specificParameters (uncontrolled)Arbitrary dataS. Kikuchi et al., IEICE Tech. Rep. vol. 119, no. 66, SC2019-2, pp. 7-17, 2019, in Japanesehttps://ken.ieice.org/ken/paper/20190531k1nc/(also in Ranganathan et al., 14th Int. Conf. Open Repositories, 2019. https://doi.org/10.5281/zenodo.3553963 )https://ken.ieice.org/ken/paper/20190531k1nc/https://doi.org/10.5281/zenodo.3553963NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM16An example of the DICE common messaging formatNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM17• All research datasets and applications are expected to use the DICE common messaging format to describe and share its metadataInitial plan in depositing XAFS datasets to MDRS. Kikuchi et al., IEICE Tech. Rep. vol. 119, no. 66, SC2019-2, pp. 7-17, 2019, in Japanesehttps://ken.ieice.org/ken/paper/20190531k1nc/https://ken.ieice.org/ken/paper/20190531k1nc/https://ken.ieice.org/ken/paper/20190531k1nc/https://ken.ieice.org/ken/paper/20190531k1nc/NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM18• Some of the XAFS datasets were deposited to MDR using the schema, but it took much more efforts than expected• It was extremely difficult for researchers and application engineers to understand its complex specification◦ Encountered implementation incompatibilities in handling the metadata and schema between the application software◦ Encountered many limitations and bugs in the software (customized OSS)• After some operations, it was officially abandoned in 2022Difficulties in applying the huge metadata schemahttps://dice.nims.go.jp/about.htmlhttps://dice.nims.go.jp/about.htmlNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM19• MDR basically supports only keyword search• MDR doesn’t fully support nested metadata structure in the DICE common messaging format◦ e.g., cannot update a value in a nested property• These came from technical limitations in MDR’s software stack• “…What should we do?”Technical limitations in MDRNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM20• https://github.com/nims-dpfc/mdr-schema• Simplified version of the DICE common messaging format• Solely designed for depositing datasets to MDR◦ Mostly compatible with the existing metadata on MDR• Eliminates deeply-nested hierarchy structure• Expects to be written in YAMLMDR Schemahttps://github.com/nims-dpfc/mdr-schemaNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM21An example of metadata in MDR SchemaNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM22• Using Yamale to create and validate the metadata schema◦ https://github.com/23andMe/YamaleValidating MDR Schema YAMLhttps://github.com/23andMe/YamaleNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM23Schema definition of MDR SchemaNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM24Schema definition of MDR SchemaNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM25• human-readable and (almost) writable◦ Easy to create and distribute a metadata template file since it supports single-line and inline comments◦ It is painful for human to read and write a spreadsheet that has a lot of columns• It can be converted to JSON or other formats easilyWhy YAML?NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM• “instruments”->”function”->”category” is moved to “function_category”• “instruments”->“managing_organization”is moved to “instrument_management_organization”• Assigns local identifier “instrument_00001” to the instrument and refers itfrom the “instrument_management_ organization”26An example of MDR metadata migrationDICE common messaging format(converted to YAML)MDR SchemaNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM27Comparison of metadata schemas around MDR XAFS DBXAFS MetadataDICE common messaging formatMDR SchemaSupport scientific metadata properties?Yes Yes Partially yesResearch domain specific?Yes No NoFormat YAML, JSON or TSV XML, JSON YAML, JSONMetadata structure deeply-nested deeply-nested single-nestedHuman-friendly? ? No Yes(?)NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORMadvanced case study:International XAFS DB Portal28NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM29• Cross-search for MDR XAFS DB and other three XAFS databases in the world◦ SSHADE/FAME (France)◦ XASLIB (US)◦ LISA XAS Database (Italy)• All datasets are linked with MatVoc vocabularies• https://ixdb.jxafs.org/International XAFS DB PortalXASLIBMDR XAFS DBSSHDE/FAME LISAhttps://ixdb.jxafs.org/NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM30NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM31NATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM32Link to MDR XAFS DB and other XAFS databases associated with the entityNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM33Link to Crystallography Open Databasehttps://www.crystallography.net/cod/1530508.htmlhttps://www.crystallography.net/cod/1530508.htmlNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM34Linking metadata using MatVocZinc oxide10.48505/nims.351610.48505/nims.359410.48505/nims.366610.48505/nims.366710.48505/nims.282110.48505/nims.2726SPring-8 BL14B2SPring-8 BL14B2KEK BL-9CKEK BL12CHokkaido-Univ.KEK BL-9CNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM35XAFS knowledge baseXAFSknowledge base(RDF store)International XAFS DB PortalXASLIBSSHDE/FAMELISAannotationDatabaseusersearch resultquery stringURIsquerydataset URLsNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM36• One big metadata schema does not fit all actually• Everyone says, “Metadata is important”, but the meaning depends on the context◦ Researchers, engineers, and librarians have different viewpoints on metadata and its use◦ Especially we should consider researcher-friendly metadata creation environment• Browse and search features are essential in research data management even if they are basic implementations and have basic metadata◦ It is difficult for even researchers and data curators to grasp the whole picture of their datasets• Identifier (for datasets and vocabularies) is the key to tame diverse research data and their domain-specific metadataLessons learnedNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORM37• Support querying MatVoc URIs in MDR REST API• Add MatVoc URIs to datasets previously deposited on MDR• Add MatVoc URIs to datasets on other DICE services• Expand MatVoc vocabulary for a focused domain of materials and enhance findability of those data• Linking materials to their crystallographic information through MatVoc• Cross-linking more XAFS databases around the worldFurther plansNATIONAL INSTITUTE FOR MATERIALS SCIENCE   |  MATERIALS DATA PLATFORMTANABE.Kosuke@nims.go.jp38 スライド 0: Cross-searching of datasets by linking repository and vocabulary management in materials data platform スライド 1: NIMS Materials Data Platform “DICE” スライド 2: Selected in this presentation スライド 3: NIMS Materials Data Repository (MDR) スライド 4: MDR XAFS DB スライド 5 スライド 6 スライド 7: MatVoc / MatVoc Explorer スライド 8 スライド 9 スライド 10 スライド 11: Metadata schemas around MDR and MDR XAFS DB スライド 12: Metadata (schemas) in MDR and MDR XAFS DB スライド 13: XAFS Metadata スライド 14: DICE common messaging format スライド 15: The structure of DICE common messaging format スライド 16: An example of the DICE common messaging format スライド 17: Initial plan in depositing XAFS datasets to MDR スライド 18: Difficulties in applying the huge metadata schema スライド 19: Technical limitations in MDR スライド 20: MDR Schema スライド 21: An example of metadata in MDR Schema スライド 22: Validating MDR Schema YAML スライド 23: Schema definition of MDR Schema スライド 24: Schema definition of MDR Schema スライド 25: Why YAML? スライド 26: An example of MDR metadata migration スライド 27: Comparison of metadata schemas around MDR XAFS DB スライド 28: advanced case study: International XAFS DB Portal スライド 29: International XAFS DB Portal スライド 30 スライド 31 スライド 32 スライド 33 スライド 34: Linking metadata using MatVoc スライド 35: XAFS knowledge base スライド 36: Lessons learned スライド 37: Further plans スライド 38: TANABE.Kosuke@nims.go.jp