SlideShare a Scribd company logo

Earth Science Markup Language (ESML) - A Tutorial

•
•
The HDF-EOS Tools and Information Center
The HDF-EOS Tools and Information Center

This document provides an overview of the Earth Science Markup Language (ESML). ESML is an XML-based interchange format that allows applications and services to access heterogeneous earth science data regardless of the underlying data format. It provides syntactic, semantic, and content metadata that describe data structures, meanings, and contents in a machine-readable way. The document outlines the need for such an interchange format, describes the components of ESML including the schema, libraries, and tools, and provides examples of writing ESML descriptions for different types of data files.

Related slideshows

Seed block algorithm
Seed block algorithmSeed block algorithm
Seed block algorithm
 
Vikram Andem Big Data Strategy @ IATA Technology Roadmap
Vikram Andem Big Data Strategy @ IATA Technology Roadmap Vikram Andem Big Data Strategy @ IATA Technology Roadmap
Vikram Andem Big Data Strategy @ IATA Technology Roadmap
 
Dr.Hadoop- an infinite scalable metadata management for Hadoop-How the baby e...
Dr.Hadoop- an infinite scalable metadata management for Hadoop-How the baby e...Dr.Hadoop- an infinite scalable metadata management for Hadoop-How the baby e...
Dr.Hadoop- an infinite scalable metadata management for Hadoop-How the baby e...
 
1 of 45
Download to read offline
EARTH SCIENCE MARKUP LANGUAGE (ESML) “Define once, use anywhere” A TUTORIAL HDF and HDF-EOS Workshop VI DECEMBER 4, 2002 STEVE TANNER INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE
Interoperability: Accessing Heterogeneous Data The Problem DATA DATA FORMAT 11 FORMAT DATA DATA FORMAT 22 FORMAT DATA DATA FORMAT 3 FORMAT 3 FORMAT CONVERTER READER 1 READER 2 APPLICATION The Solution DATA DATA DATA DATA DATA DATA FORMAT 11 FORMAT 22 FORMAT 33 FORMAT FORMAT FORMAT ESML ESML FILE FILE ESML ESML FILE FILE ESML LIBRARY APPLICATION ESML ESML FILE FILE • Earth science data comes in: • Different formats, types and structures • Different states of processing (raw, calibrated, derived, modeled or interpreted) • Enormous volumes • One approach: Standard data formats • Difficult to implement and enforce • Can’t anticipate all needs • Some data can’t be modeled or is lost in translation • The cost of converting legacy data • A better approach: Interchange technologies • Earth Science Markup Language
Interchange Technology: Earth Science Markup Language (ESML) Data Applications Interchange Technology Integration • Facilitate effective utilization of distributed, heterogeneous data products • Enable interchangeable tools and services
Background on eXtensible Markup Language • What is a Markup? – Anything on a document which adds special meaning – Rules are defined so the others can understand • Types of Markups – Stylistic – presentation – Structural – document structure – Semantic – provide meaning about the content in the document – Content – describe the content of the document
Difference between HTML and XML • HTML – Fixed format (rules are defined) – Used for presentation • XML – Allows users to create their own rules – Can be utilized in different applications – Allows data to be used across different platforms and applications – Allows non human interaction • Generate rules in XML using: – DTD – had limitations – Schema – improvement over DTD
What is ESML • Specialized markup language for Earth Science metadata based on XML • ESML is NOT a new data format - it is a “Smart” README file that both scientist and the application can understand • External metadata files that can be generated by either data producer or data consumer (at collection, data set, and/or granule level) • Machine-readable and -interpretable representation of the structure and content of any data file, regardless of data format • Beyond traditional metadata to include both structural and semantic information needed to effect a practical runtime interpretation of a data set • ESML will provide the benefits of a standard, self-describing data format (like HDF, HDF-EOS, netCDF, geoTIFF, …) without the cost of data conversion
Benefits of ESML • Enables independently developed applications and services to effectively utilize distributed, heterogeneous data products • Allows the end-user to integrate data sets of differing structures to aid in data fusion and analysis without having to write a special reader for each data set • Is simple enough that end-users can create their own ESML for on-hand datasets (new or legacy) • • • • Single data decoder can read heterogeneous data sets Scientists can manipulate data decoder using ESML Easy to use programming interface ESML content metadata allows search capabilities – Promotes data sharing and research • ESML provides basic preprocessing capabilities
Example: ESML file for an Image Mitch.esml Hurricane Mitch (981027) Binary format (32 bits/pixel) 300 512 <a:ESML> <SyntacticMetaData> <Binary dataSetName="mitch"> <BinaryStructure name="raster" instances="1" geoInfo="NoGeoInfo"> <Array name="track" occurs="300" dimName="Y"> <Array name="Crosstrack" occurs="512" dimName="X"> <Field name="Channel 1" type="Int32"> <Data/> </Field> … </SyntacticMetaData> </a:ESML>
ESML Vision: Data Usability DATA ARCHIVING CENTERS SCIENTIST WITH ESML DATA SET A DATA SET B WITHOUT ESML OTHER OTHER SOURCES SOURCES APPLICATIONS ESML Libraries VENDORS ESML ESML
Components of the Interchange Technology DATA FORMAT1 DATA FORMAT2 DATA FORMAT3 OTHER FORMATS ESML FILE ESML FILE ESML FILE ESML LIBRARY ESML SCHEMA ESML EDITOR ESML CONSISTS OF: MARKUPS ESML DATA BROWSER ADaM DATA MINING SYSTEM RULES FOR THE MARKUPS OTHER APPLICATIONS MIDDLEWARE FOR AUTOMATION
Components of the Interchange Technology DATA FORMAT1 DATA FORMAT2 DATA FORMAT3 OTHER FORMATS ESML FILE ESML FILE ESML FILE ESML SCHEMA ESML LIBRARY ESML DATA BROWSER ADaM DATA MINING SYSTEM OTHER APPLICATIONS INTERCHANGE TECHNOLOGY ESML EDITOR
Components of the Interchange Technology DATA FORMAT1 ESML FILE DATA FORMAT2 ESML FILE DATA FORMAT3 ESML FILE OTHER FORMATS ESML SCHEMA ESML LIBRARY ESML DATA BROWSER APPLICATION DEVELOPERS ADaM DATA MINING SYSTEM OTHER APPLICATIONS INTERCHANGE TECHNOLOGY ESML EDITOR DATA PRODUCERS OR CONSUMERS
ESML Schema • Content metadata describe the contents of a file in human-readable and machine-readable terms. An example of this would be ECS or FGDC metadata. • Syntactic metadata describe the structure of the file in machinereadable and -interpretable terms. HDF-EOS and HDF provide this mechanism for HDF-EOS files. • Semantic metadata, describe the contents of a file in machine-readable terms such that an application can interpret the data in an intelligent manner. Semantic metadata is embedded in the Syntactic
ESML Content Metadata • Content metadata are mainly used for human knowledge and websearching • Content metadata in ESML are derived from FGDC and ECS metadata sets • Tools are planned for converting existing ECS metadata into ESML
Syntactic Metadata - Binary Data ESML supports descriptions of the structures of data files in two basic data formats, Binary and ASCII
Syntactic Metadata - ASCII Data ESML supports descriptions of the structures of data files in two basic data formats, Binary and ASCII
Syntactic Metadata - HDF-EOS • Self-Describing Formats: –Descriptions of the structures of self-describing data files through their internal metadata • HDF, HDF-EOS, CDF, netCDF, etc • Additional data formats can be added as necessary
Semantic Metadata in ESML • Goals – Get the right data – Navigating the data • Figure out within the data what is latitude, longitude, time and/or coordinate system – Reading the data in actual scientific units, for example: • Data stored as value to save space • Equation is applied on processing • Embedded in Syntactic metadata • Provides semantic descriptions for Time, Geo (Lat/Lon) and Data fields • Provides capabilities to define equation for data conversion for Time, Geo and Data fields • e.g. Y=mx+b where m is scale and b is offset
Semantic Metadata Examples - <BinaryStructure geoInfo="ByProjection" occurs="1"> <Projection LowRight_X="360" LowRight_Y="-40" UpLeft_X="0" UpLeft_Y="40"> <Geographic latOffset="-0.125" lonOffset="-0.125" /> </Projection> <Syntac icMetaData> t <HdfEos> <HdfEosSt t ruc ure name="Day 1998335"> <Datum name="Lat_ar ray"> <Lattude/> i </Datum > <Datum name="Lon_ar ray"> <Long t i ude/> </Datum > <Datum name="50299.91 M Hz"> <Data > / <SyntacticMetaData> </Datum > <HdfEos /> </HdfEosSt ructu re> </SyntacticMetaData> </HdfEos> </Syntact icMetaData>
Need For Semantic Metadata: Example TMI Data Product README File The data values between 0 and 250 need to be scaled to obtain meaningful geophysical data. To scale the data, multiply by the scale factors listed below: T: multiply by 6.0 to get minute of day (0 – 1440) S: multiply by 0.15 AND subtract 3 to get SST (-3 - 34.5 C) W: multiply by 0.15 to get 10 m winds (0 - 37.5 m/s) V: multiply by 0.3 to get water vapor (0 - 75 mm) L: multiply by 0.01 to get cloud liquid water (0 - 2.5 mm) R: multiply by 0.1 to get rain rate (0 - 25 mm/hr)
Semantic Metadata for TMI Products <SyntacticMetaData> <Binary> <BinaryStructure geoInfo="ByProjection" instances="1"> <Projection LowRight_X="360" LowRight_Y="-40" UpLeft_X="0" UpLeft_Y="40"> <Geographic latOffset="-0.125" lonOffset="-0.125"/> </Projection> <Array occurs="320"> <Array occurs="1440"> <Field name="SST" type="UInt8" order="LittleEndian" size="8"> <Data unit="C " equation="0.15 * X -3"/> </Field> </Array> </Array> ... </SyntacticMetaData>
Writing an ESML File (1) ESML MARKUP FOR THE DATA FILE 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field> <Array occurs="4"> <Array occurs="5"> <Field name="BrightnessTemp" format="%f"> <Data unit="Degrees Kelvin"/> </Field> </Array> </Array> </AsciiStructure> </Ascii> </SyntacticMetaData> </a:ESML>
Writing an ESML File (2) ONLY THE STRUCTURE 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData>
Writing an ESML File (3) DESCRIBE THE FORMAT 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii>
Writing an ESML File (4) ENTIRE FILE CONTENTS INTO 1 LOGICAL STRUCTURE 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1">
Writing an ESML File (5) DEFINE THE FIRST FIELD IN THE FILE: HEADER INFORMATION 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field>
Writing an ESML File (6) DEFINE THE SECOND FIELD IN THE FILE: HEADER INFORMATION 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field>
Writing an ESML File (7) DEFINE THE DATA FIELD IN THE FILE: PROVIDE SIZE AND FORMAT INFORMATION 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field> <Array occurs="4"> <Array occurs="5"> <Field name="BrightnessTemp" format="%d"> <Data unit="Degrees Kelvin"/> </Field> </Array> </Array>
Writing an ESML File (8) CLOSE ALL THE TAGS: ESML FILE IS READY 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field> <Array occurs="4"> <Array occurs="5"> <Field name="BrightnessTemp" format="%d"> <Data unit="Degrees Kelvin"/> </Field> </Array> </Array> </AsciiStructure> </Ascii> </SyntacticMetaData> </a:ESML>
Demonstration of the Interchange Technology • Read multiple formats – NLDN in ASCII – GOES Binary – AMSU-A HDF-EOS • Single ESML file – multiple data files • Equation to preprocess the data • Semantic capabilities – slice and dice the data
Hands On Tutorial Outline 1. Writing an ESML file using the ESML Editor – Simple ASCII file – Simple Binary file – HDF-EOS file 2. Utilizing the ESML descriptions to browse the data files using the ESML Data Browser 3. Overview of the ESML library – Writing a simple reader – Writing a generic reader
ESML Editor 100% Java version prototype Utilizes Expert System principles to give users correct options Hides the XML tags from the users Future Features: Allow text editing of the XML tags also Incorporate feedback from users
ESML Data Browser Hybrid product (Java Client/C++ Library backend/JNI Interface) Features Browse and view data file using a ESML file Browse the metadata for each data field Future Features: Allow format conversion with automatic generation of ESML metadata file Allow selection of multiple fields Additional functionality such as Subsetting
ESML Editor • ESML Schema Window – Displays the ESML Schema in tree structure • ESML File Window – Displays the user generated ESML description • ESML Attribute Window – Provides the means to set the attributes for the selected ESML element
Writing an ESML File: Simple ASCII File (1) ASCII data file: 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Writing an ESML File: Simple ASCII File (2) • Understand the structure of the data file to be described by ESML file • Open ESML editor (ready to create new ESML file) and describe this structure using the rules established in the ESML schema • Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available • Add “Ascii” as child of “SyntacticMetadata” and modify its attributes if available
Writing an ESML File: Simple ASCII File (3) • Add “AsciiStructure” as child of “Ascii” and modify its attribute • Add “Field” element as one child of “AsciiStructure” and modify its attributes if available. This element is used to describe the first datum in the data file • Add ‘semantic information’ to describe this “Field” • Add “Field” element as one child of “AsciiStructure” and modify its attributes if available. This element is used to describe the second datum in the data file • Add “Array” element as one child of “AsciiStructure” to describe the outer dimension of the 2D array, and modify its attributes if available
Writing an ESML File: Simple ASCII File (4) • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available
Writing an ESML File: Simple Binary File (1) • The structure and content are the same as those of ASCII file that we just worked on, but stored in Binary format • Open ESML editor • Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available • Add “Binary” as child of “SyntacticMetadata” and modify its attributes if available
Writing an ESML File: Simple Binary File (2) • Add “BinaryStructure” as child of “Binary” and modify its attribute • Add “Field” element as one child of “BinaryStructure” and modify its attributes if available. This element is used to describe the first datum in the data file • Add “Field” element as one child of “BinaryStructure” and modify its attributes if available. This element is used to describe the second datum in the data file • Add “Array” element as one child of “BinaryStructure” to describe the outer dimension of the 2D array, and modify its attributes if available
Writing an ESML File: Simple Binary File (3) • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available
Writing an ESML File: HDFEOS File (1) • Since HDF-EOS is self-describing format and metadata is contained in the data file, ESML file for HDF-EOS data file is simple • Here are the steps to create ESML file for HDF-EOS: • Open ESML editor • Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available and needed • Add “HdfEos” as child of “SyntacticMetadata” • You are done!!!
Writing an ESML File: HDFEOS File (2) • HDF-EOS format has rules that Latitude and Longitude fields have to have field names of “Latitude” and “Longitude”, respectively • If a HDF-EOS file does not follow this rule, ESML file has to be more detailed • Here are the follow-on from the ESML file before: • Add “HdfEosStructure” as child of “HdfEos” and modify its attributes if available and needed
Writing an ESML File: HDFEOS File (3) • Add “Field” as child of “HdfEosStructure” • Add ‘semantic information’ as child to describe this “Field” element. If this field is latitude, then add a child element “Latitude” to indicate that this field is latitude field. If longitude field, add “Longitude” child element to describe the “Field” element.
ESML Web Page URL: esml.itsc.uah.edu Post latest products, news, presentations, papers Schema and related documents available to all ESML Library, ESML Editor and ESML Data Browser available

More Related Content

Earth Science Markup Language (ESML) - A Tutorial

  • 1. EARTH SCIENCE MARKUP LANGUAGE (ESML) “Define once, use anywhere” A TUTORIAL HDF and HDF-EOS Workshop VI DECEMBER 4, 2002 STEVE TANNER INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE
  • 2. Interoperability: Accessing Heterogeneous Data The Problem DATA DATA FORMAT 11 FORMAT DATA DATA FORMAT 22 FORMAT DATA DATA FORMAT 3 FORMAT 3 FORMAT CONVERTER READER 1 READER 2 APPLICATION The Solution DATA DATA DATA DATA DATA DATA FORMAT 11 FORMAT 22 FORMAT 33 FORMAT FORMAT FORMAT ESML ESML FILE FILE ESML ESML FILE FILE ESML LIBRARY APPLICATION ESML ESML FILE FILE • Earth science data comes in: • Different formats, types and structures • Different states of processing (raw, calibrated, derived, modeled or interpreted) • Enormous volumes • One approach: Standard data formats • Difficult to implement and enforce • Can’t anticipate all needs • Some data can’t be modeled or is lost in translation • The cost of converting legacy data • A better approach: Interchange technologies • Earth Science Markup Language
  • 3. Interchange Technology: Earth Science Markup Language (ESML) Data Applications Interchange Technology Integration • Facilitate effective utilization of distributed, heterogeneous data products • Enable interchangeable tools and services
  • 4. Background on eXtensible Markup Language • What is a Markup? – Anything on a document which adds special meaning – Rules are defined so the others can understand • Types of Markups – Stylistic – presentation – Structural – document structure – Semantic – provide meaning about the content in the document – Content – describe the content of the document
  • 5. Difference between HTML and XML • HTML – Fixed format (rules are defined) – Used for presentation • XML – Allows users to create their own rules – Can be utilized in different applications – Allows data to be used across different platforms and applications – Allows non human interaction • Generate rules in XML using: – DTD – had limitations – Schema – improvement over DTD
  • 6. What is ESML • Specialized markup language for Earth Science metadata based on XML • ESML is NOT a new data format - it is a “Smart” README file that both scientist and the application can understand • External metadata files that can be generated by either data producer or data consumer (at collection, data set, and/or granule level) • Machine-readable and -interpretable representation of the structure and content of any data file, regardless of data format • Beyond traditional metadata to include both structural and semantic information needed to effect a practical runtime interpretation of a data set • ESML will provide the benefits of a standard, self-describing data format (like HDF, HDF-EOS, netCDF, geoTIFF, …) without the cost of data conversion
  • 7. Benefits of ESML • Enables independently developed applications and services to effectively utilize distributed, heterogeneous data products • Allows the end-user to integrate data sets of differing structures to aid in data fusion and analysis without having to write a special reader for each data set • Is simple enough that end-users can create their own ESML for on-hand datasets (new or legacy) • • • • Single data decoder can read heterogeneous data sets Scientists can manipulate data decoder using ESML Easy to use programming interface ESML content metadata allows search capabilities – Promotes data sharing and research • ESML provides basic preprocessing capabilities
  • 8. Example: ESML file for an Image Mitch.esml Hurricane Mitch (981027) Binary format (32 bits/pixel) 300 512 <a:ESML> <SyntacticMetaData> <Binary dataSetName="mitch"> <BinaryStructure name="raster" instances="1" geoInfo="NoGeoInfo"> <Array name="track" occurs="300" dimName="Y"> <Array name="Crosstrack" occurs="512" dimName="X"> <Field name="Channel 1" type="Int32"> <Data/> </Field> … </SyntacticMetaData> </a:ESML>
  • 9. ESML Vision: Data Usability DATA ARCHIVING CENTERS SCIENTIST WITH ESML DATA SET A DATA SET B WITHOUT ESML OTHER OTHER SOURCES SOURCES APPLICATIONS ESML Libraries VENDORS ESML ESML
  • 10. Components of the Interchange Technology DATA FORMAT1 DATA FORMAT2 DATA FORMAT3 OTHER FORMATS ESML FILE ESML FILE ESML FILE ESML LIBRARY ESML SCHEMA ESML EDITOR ESML CONSISTS OF: MARKUPS ESML DATA BROWSER ADaM DATA MINING SYSTEM RULES FOR THE MARKUPS OTHER APPLICATIONS MIDDLEWARE FOR AUTOMATION
  • 11. Components of the Interchange Technology DATA FORMAT1 DATA FORMAT2 DATA FORMAT3 OTHER FORMATS ESML FILE ESML FILE ESML FILE ESML SCHEMA ESML LIBRARY ESML DATA BROWSER ADaM DATA MINING SYSTEM OTHER APPLICATIONS INTERCHANGE TECHNOLOGY ESML EDITOR
  • 12. Components of the Interchange Technology DATA FORMAT1 ESML FILE DATA FORMAT2 ESML FILE DATA FORMAT3 ESML FILE OTHER FORMATS ESML SCHEMA ESML LIBRARY ESML DATA BROWSER APPLICATION DEVELOPERS ADaM DATA MINING SYSTEM OTHER APPLICATIONS INTERCHANGE TECHNOLOGY ESML EDITOR DATA PRODUCERS OR CONSUMERS
  • 13. ESML Schema • Content metadata describe the contents of a file in human-readable and machine-readable terms. An example of this would be ECS or FGDC metadata. • Syntactic metadata describe the structure of the file in machinereadable and -interpretable terms. HDF-EOS and HDF provide this mechanism for HDF-EOS files. • Semantic metadata, describe the contents of a file in machine-readable terms such that an application can interpret the data in an intelligent manner. Semantic metadata is embedded in the Syntactic
  • 14. ESML Content Metadata • Content metadata are mainly used for human knowledge and websearching • Content metadata in ESML are derived from FGDC and ECS metadata sets • Tools are planned for converting existing ECS metadata into ESML
  • 15. Syntactic Metadata - Binary Data ESML supports descriptions of the structures of data files in two basic data formats, Binary and ASCII
  • 16. Syntactic Metadata - ASCII Data ESML supports descriptions of the structures of data files in two basic data formats, Binary and ASCII
  • 17. Syntactic Metadata - HDF-EOS • Self-Describing Formats: –Descriptions of the structures of self-describing data files through their internal metadata • HDF, HDF-EOS, CDF, netCDF, etc • Additional data formats can be added as necessary
  • 18. Semantic Metadata in ESML • Goals – Get the right data – Navigating the data • Figure out within the data what is latitude, longitude, time and/or coordinate system – Reading the data in actual scientific units, for example: • Data stored as value to save space • Equation is applied on processing • Embedded in Syntactic metadata • Provides semantic descriptions for Time, Geo (Lat/Lon) and Data fields • Provides capabilities to define equation for data conversion for Time, Geo and Data fields • e.g. Y=mx+b where m is scale and b is offset
  • 19. Semantic Metadata Examples - <BinaryStructure geoInfo="ByProjection" occurs="1"> <Projection LowRight_X="360" LowRight_Y="-40" UpLeft_X="0" UpLeft_Y="40"> <Geographic latOffset="-0.125" lonOffset="-0.125" /> </Projection> <Syntac icMetaData> t <HdfEos> <HdfEosSt t ruc ure name="Day 1998335"> <Datum name="Lat_ar ray"> <Lattude/> i </Datum > <Datum name="Lon_ar ray"> <Long t i ude/> </Datum > <Datum name="50299.91 M Hz"> <Data > / <SyntacticMetaData> </Datum > <HdfEos /> </HdfEosSt ructu re> </SyntacticMetaData> </HdfEos> </Syntact icMetaData>
  • 20. Need For Semantic Metadata: Example TMI Data Product README File The data values between 0 and 250 need to be scaled to obtain meaningful geophysical data. To scale the data, multiply by the scale factors listed below: T: multiply by 6.0 to get minute of day (0 – 1440) S: multiply by 0.15 AND subtract 3 to get SST (-3 - 34.5 C) W: multiply by 0.15 to get 10 m winds (0 - 37.5 m/s) V: multiply by 0.3 to get water vapor (0 - 75 mm) L: multiply by 0.01 to get cloud liquid water (0 - 2.5 mm) R: multiply by 0.1 to get rain rate (0 - 25 mm/hr)
  • 21. Semantic Metadata for TMI Products <SyntacticMetaData> <Binary> <BinaryStructure geoInfo="ByProjection" instances="1"> <Projection LowRight_X="360" LowRight_Y="-40" UpLeft_X="0" UpLeft_Y="40"> <Geographic latOffset="-0.125" lonOffset="-0.125"/> </Projection> <Array occurs="320"> <Array occurs="1440"> <Field name="SST" type="UInt8" order="LittleEndian" size="8"> <Data unit="C " equation="0.15 * X -3"/> </Field> </Array> </Array> ... </SyntacticMetaData>
  • 22. Writing an ESML File (1) ESML MARKUP FOR THE DATA FILE 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field> <Array occurs="4"> <Array occurs="5"> <Field name="BrightnessTemp" format="%f"> <Data unit="Degrees Kelvin"/> </Field> </Array> </Array> </AsciiStructure> </Ascii> </SyntacticMetaData> </a:ESML>
  • 23. Writing an ESML File (2) ONLY THE STRUCTURE 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData>
  • 24. Writing an ESML File (3) DESCRIBE THE FORMAT 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii>
  • 25. Writing an ESML File (4) ENTIRE FILE CONTENTS INTO 1 LOGICAL STRUCTURE 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1">
  • 26. Writing an ESML File (5) DEFINE THE FIRST FIELD IN THE FILE: HEADER INFORMATION 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field>
  • 27. Writing an ESML File (6) DEFINE THE SECOND FIELD IN THE FILE: HEADER INFORMATION 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field>
  • 28. Writing an ESML File (7) DEFINE THE DATA FIELD IN THE FILE: PROVIDE SIZE AND FORMAT INFORMATION 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field> <Array occurs="4"> <Array occurs="5"> <Field name="BrightnessTemp" format="%d"> <Data unit="Degrees Kelvin"/> </Field> </Array> </Array>
  • 29. Writing an ESML File (8) CLOSE ALL THE TAGS: ESML FILE IS READY 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 SIMPLE ASCII DATA FILE <a:ESML> <SyntacticMetaData> <Ascii> <AsciiStructure geoInfo="NoGeoInfo" instances="1"> <Field name="SizeX" format="%d"> <Attribute/> </Field> <Field name="SizeY" format="%d"> <Attribute/> </Field> <Array occurs="4"> <Array occurs="5"> <Field name="BrightnessTemp" format="%d"> <Data unit="Degrees Kelvin"/> </Field> </Array> </Array> </AsciiStructure> </Ascii> </SyntacticMetaData> </a:ESML>
  • 30. Demonstration of the Interchange Technology • Read multiple formats – NLDN in ASCII – GOES Binary – AMSU-A HDF-EOS • Single ESML file – multiple data files • Equation to preprocess the data • Semantic capabilities – slice and dice the data
  • 31. Hands On Tutorial Outline 1. Writing an ESML file using the ESML Editor – Simple ASCII file – Simple Binary file – HDF-EOS file 2. Utilizing the ESML descriptions to browse the data files using the ESML Data Browser 3. Overview of the ESML library – Writing a simple reader – Writing a generic reader
  • 32. ESML Editor 100% Java version prototype Utilizes Expert System principles to give users correct options Hides the XML tags from the users Future Features: Allow text editing of the XML tags also Incorporate feedback from users
  • 33. ESML Data Browser Hybrid product (Java Client/C++ Library backend/JNI Interface) Features Browse and view data file using a ESML file Browse the metadata for each data field Future Features: Allow format conversion with automatic generation of ESML metadata file Allow selection of multiple fields Additional functionality such as Subsetting
  • 34. ESML Editor • ESML Schema Window – Displays the ESML Schema in tree structure • ESML File Window – Displays the user generated ESML description • ESML Attribute Window – Provides the means to set the attributes for the selected ESML element
  • 35. Writing an ESML File: Simple ASCII File (1) ASCII data file: 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
  • 36. Writing an ESML File: Simple ASCII File (2) • Understand the structure of the data file to be described by ESML file • Open ESML editor (ready to create new ESML file) and describe this structure using the rules established in the ESML schema • Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available • Add “Ascii” as child of “SyntacticMetadata” and modify its attributes if available
  • 37. Writing an ESML File: Simple ASCII File (3) • Add “AsciiStructure” as child of “Ascii” and modify its attribute • Add “Field” element as one child of “AsciiStructure” and modify its attributes if available. This element is used to describe the first datum in the data file • Add ‘semantic information’ to describe this “Field” • Add “Field” element as one child of “AsciiStructure” and modify its attributes if available. This element is used to describe the second datum in the data file • Add “Array” element as one child of “AsciiStructure” to describe the outer dimension of the 2D array, and modify its attributes if available
  • 38. Writing an ESML File: Simple ASCII File (4) • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available
  • 39. Writing an ESML File: Simple Binary File (1) • The structure and content are the same as those of ASCII file that we just worked on, but stored in Binary format • Open ESML editor • Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available • Add “Binary” as child of “SyntacticMetadata” and modify its attributes if available
  • 40. Writing an ESML File: Simple Binary File (2) • Add “BinaryStructure” as child of “Binary” and modify its attribute • Add “Field” element as one child of “BinaryStructure” and modify its attributes if available. This element is used to describe the first datum in the data file • Add “Field” element as one child of “BinaryStructure” and modify its attributes if available. This element is used to describe the second datum in the data file • Add “Array” element as one child of “BinaryStructure” to describe the outer dimension of the 2D array, and modify its attributes if available
  • 41. Writing an ESML File: Simple Binary File (3) • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available • Add “Array” element as child of “Array” to describe the inner dimension of the 2D array, and modify its attributes if available • Add “Field” element as child of “Array” to describe the individual element and modify its attributes if available
  • 42. Writing an ESML File: HDFEOS File (1) • Since HDF-EOS is self-describing format and metadata is contained in the data file, ESML file for HDF-EOS data file is simple • Here are the steps to create ESML file for HDF-EOS: • Open ESML editor • Add “SyntacticMetadata” as child of “ESML” and modify its attributes if available and needed • Add “HdfEos” as child of “SyntacticMetadata” • You are done!!!
  • 43. Writing an ESML File: HDFEOS File (2) • HDF-EOS format has rules that Latitude and Longitude fields have to have field names of “Latitude” and “Longitude”, respectively • If a HDF-EOS file does not follow this rule, ESML file has to be more detailed • Here are the follow-on from the ESML file before: • Add “HdfEosStructure” as child of “HdfEos” and modify its attributes if available and needed
  • 44. Writing an ESML File: HDFEOS File (3) • Add “Field” as child of “HdfEosStructure” • Add ‘semantic information’ as child to describe this “Field” element. If this field is latitude, then add a child element “Latitude” to indicate that this field is latitude field. If longitude field, add “Longitude” child element to describe the “Field” element.
  • 45. ESML Web Page URL: esml.itsc.uah.edu Post latest products, news, presentations, papers Schema and related documents available to all ESML Library, ESML Editor and ESML Data Browser available