COBWEB A quality assurance workflow authoring tool for citizen science and crowdsourced data

Editor's Notes

1. 1/ QAWAT is the quality assurance tool designed by the university of Nottingham with the FP7 project COBWEB. It is used for citizen science by the stakeholder who designed the survey campaign using the data capture tool.
2. 2/ Quality assurance via the QAQC processing comes during the data capture or after the data capture of observations from each Volunteering citizen and aims at producing metadata on data quality for the captured observation
3. 4/ Added to the ISO standard for quality: the producer model, the QAQC uses a quality model to qualify the volunteers: the stakeholder model as well as a very simplified version of the consumer model designed by GeoViQUA. The Stakeholder quality model evaluate or calibrate the properties of the volunteer seen as a sensor using 6 dimensions related to its accuracy, consistency, and trust
4. 3/ The COBWEB platform and the Quality Assurance tool are based on interoperability standards for geoprocessing, workflow and quality information encoding
5. 5/ Each Quality Control produces or updates a number of quality elements from the three models; and is seen as a single task within the workflow evaluating the quality from different type of controls: the pillars. The whole QA workflow will be composed a a series of QC’s belonging to these 7 pillars. The 7+ pillars deal with security and privacy when necessary for a specific QC which would belong to one of the 7 pillars
6. 6/ The categorisation of the QCs in 7 pillars is to help in the development of geoprocesses and in the composition of the workflow. The 7 pillars represent the top an ontology of the QCs for VGI ,citizen science and crowdsourcing. A similar QC can exist in different pillars but the quality elements generated or the rule to assign their values will be different due to the semantic of the pillars.
7. 7/ Any code wrapped up into a WPS is registered as part of a particular pillar and the workflow web editor is the tool used to compose the workflow. In its algorithm, each QC will have a processing or geoprocessing part followed by a logic reasoning part depending on the results of the first part and the semantic attached to it and to the pillar description.
8. 8/ From the list of QCs in each pillar, any tool to compose a BPMN workflow can be used. The conceptual aspect of the workflow captured by the graphical analytic of the BPMN can be annotated and shared within the different stakeholders. QAwAT is to compose and execute the workflow using the workflow engine linked to the WPS.
9. 9/ As an example we have here a quality Control in pillar1 helping to assess the position of the aimed point when taking a photo with the smartphone. Besides reporting the obs point as being the LoS point one can also from the distance to the point estimate a Topological Consistency, with the position of the observer to identify properly what he/she is observing. From the uncertainty of the GPS of the phone and of the DEM and of the bearings parameters. The LoS point, distance to it and its uncertainty can be computed then the probability for a Normal distribution to be less than a stakeholder given’s threshold of a reasonable distance to make a proper observation.
10. 10/ We used JBPM suite which has a workflow engine and an editor either within Eclipse or as a web editor. The workflow engine has been modified to accept WPS services as tasks.
11. 11/ Using the online editor. The editor enables one to drag and drop the tasks. Then the input URL and output names are filled in. The QA workflow can be run from the web interface or later from a WPS interface once the whole workflow is wrapped into a process of the WPS.
12. 12/ The QAQC starts by postulating 50% quality uncertainty for Classification Correctness (for example)
13. 13/ The quality elements evolves through the workflow as different QC update their values. The results therefore depend on the choice of QCs and set of parameters used in that workflow. The BPMN of the workflow is stored as metaquality element so encoding the provenance of the metadata values on data quality. This is a qualifying not a validating step or verifying step. It indicates how uncertain we are about each observation considering the rules put in the QA workflow. Low and high quality values being less uncertain, here of being a true Japanese Knotweed observation or not. After verification using the ground truth, even though the tendency is that the QAQC helps to identify correct observations, you can see that some high quality were given to some wrong JKW observations and vice versa. Quite a number of citizen’s data still have uncertainty attached, i.e. still around 50%.
14. 14/ and here are the results for the Snowdonian National Park and the initial survey for Japanese knotweed performed during May-July 2015, with some the data source used in the pillars as well. In summary, the ones close to managed lands or woods but not to the EO identified areas as at risk of having JKW have a lower quality (higher uncertainty).
15. 15/ Some of the research on data quality and workflows at the Nottingham Geospatial Institute
16. 1/ QAWAT is the quality assurance tool designed by the university of Nottingham with the FP7 project COBWEB. It is used for citizen science by the stakeholder who designed the survey campaign using the data capture tool. 2/ Quality assurance comes during the data capture or after the data capture of observations from each Volunteering citizen and aims at producing metadata on data quality for the captured observation 3/ The COBWEB platform and the Quality Assurance tool are based on interoperability standards for geoprocessing, workflow and quality information encoding 4/ Added to the ISO standard for quality: the producer model, the QAQC uses a quality model to qualify the volunteers: the stakeholder model as well as a very simplified version of the consumer model designed by GeoViQUA. The Stakeholder quality model evaluate or calibrate the properties of the volunteer seen as a sensor using 6 dimensions related to its accuracy, consistency, and trust 5/ Each Quality Control or QC for short produces or updates a number of quality elements from the three models; and is seen as a single task within workflow evaluating the quality from different type of controls: the pillars. The whole QA workflow will be composed a a series of QC belonging to these 7 pillars. The 7+ pillar deals with security and privacy when necessary for a specific QC which would belong to one of the 7 pillars 6/ The categorisation of the QCs in 7 pillars is to help in the development of geoprocesses and in the composition of the workflow. The 7 pillars represent the top an ontology of the QCs for VGI ,citizen science and crowdsourcing. A similar QC can exist in different pillars but the quality elements generated or the rule to assign their values will be different due to the semantic of the pillars. 7/ Any code wrapped up into a WPS is registered as part of a particular pillar and the workflow web editor is the tool used to compose the workflow. In its algorithm, each QC will have a processing or geoprocessing part followed by a logic reasoning part depending on the results of the first part and the semantic attached to it and to the pillar description. 8/ From the list of QCs in each pillar, any tool to compose a BPMN workflow can be used. The conceptual aspect of the workflow captured by the graphical analytic of the BPMN can be annotated and shared within the different stakeholders. QAwAT is to compose and execute the workflow using the workflow engine linked to the WPS. 9/ As an example we have here a quality Control in pillar1 helping to assess the position of the aimed point when taking a photo with the smartphone. Besides reporting the obs point as being the LoS point one can also from the distance to the point estimate a Topological Consitency, with the position of the observer to identify properly what he/she is observing. From the uncertainty of the GPS of the phone and of the DEM and of the bearings parameters. The LoS point, distance to it and its uncertainty can be computed then the probability for a Normal distribution to be less than a stakeholder given’s threshold of a reasonable distance to make a proper observation. 10/ We used JBPM suite which has a workflow engine and an editor either within Eclipse or as a web editor. The workflow engine has been modified to accept WPS services as tasks. 11/ Using the online editor. The editor allows to drag and rop the tasks. Then the input URL and ouput names are filled in. The QA workflow can be run from the web interface or later from a WPS interface once the whole workflow is wrapped into a process of the WPS. 12/ The QAQC starts by postulating 50% quality uncertainty for Classification Correctness (for example) 13/ The quality elements evolves through the workflow as different QC update their values. The results therefore eon the choice of QCs and set of parameters used in that workflow. The BPMN of the workflow is stored as metaquality element so encoding the provenance of the metadata values on data quality. This is a qualifying not a validating step or verifying step. It indicates how uncertain we are about each observation considering the rules put in the QA workflow. Low and high quality values being less uncertain, here of being a true Japanese Knotweed observation or not. After verification using the ground truth, even though the tendency is that the QAQC helps to identify correct observations, you can see that some high quality were given to some wrong JKW observations and vice versa. Quite a number of citizen’s data still have uncertainty attached, i.e. still around 50%. 14/ … and here is the map of the results for the Snowdonian National Park and the initial survey for Japanese knotweed performed during May-July 2015, with some the data source used in the pillars as well. Basically the ones too close to ‘managed lands’ or ‘woods’ and not enough close to the area identified from EO as at risk of having JKW have a lower quality. 15/ Some of the research on data quality, error propagation and workflows at the Nottingham Geospatial Institute