EO Training

FRESHWATER QUALITY MONITORING FROM SPACE

Categories: Hydrology,

FRESHWATER QUALITY MONITORING FROM SPACE

Introduction Print

Freshwater is a fundamental resource for human civilization, the availability or unavailability of it is directly linked to water security, food security, economic productivity and conflicts. Moreover, freshwater ecosystems play a major role in driving global biodiversity and ecological stability, however they are also exceptionally vulnerable to human mismanagement and climate change. With the progress of climatic change and growing world population, the pressure on our freshwater resources is growing and with it the need for careful monitoring and management, to preserve them for current and future generations.

Remote sensing enables us to monitor a number of water-quality indicators that reflect the state of the entire freshwater ecosystems. These indicators observable from space help us to estimate among others the concentrations of Chlorophyll-a, Total suspended matter (TSM), Colored dissolved organic matter (CDOM), as well as the density of cyanobacteria and water turbidity.

NOTE: This exercise was developed by the Serco EO Training Team within the framework of the RUS contract. The final outcome of this tutorial has not been validated and the information contained in this tutorial does not purport to constitute professional advice.

Data used

12 low-cloud Sentinel-2A Level 1A tile (Tile ID: T32TQN) acquired from 16 April to 15 November 2018 [downloadable @ https://scihub.copernicus.eu/]

  • S2A_MSIL1C_20180416T100031_N0206_R122_T32TQN_20180416T120852.zip
  • S2B_MSIL1C_20180511T100029_N0206_R122_T32TQN_20180511T134849.zip
  • S2B_MSIL1C_20180531T100029_N0206_R122_T32TQN_20180531T120906.zip
  • S2B_MSIL1C_20180630T100029_N0206_R122_T32TQN_20180630T134105.zip
  • S2B_MSIL1C_20180710T100029_N0206_R122_T32TQN_20180710T135302.zip
  • S2A_MSIL1C_20180718T101031_N0206_R022_T32TQN_20180718T171829.zip
  • S2B_MSIL1C_20180809T100019_N0206_R122_T32TQN_20180809T140850.zip
  • S2B_MSIL1C_20180822T101019_N0206_R022_T32TQN_20180822T142412.zip
  • S2B_MSIL1C_20180829T100019_N0206_R122_T32TQN_20180829T190727.zip
  • S2A_MSIL1C_20180906T101021_N0206_R022_T32TQN_20180906T122259.zip
  • S2B_MSIL1C_20180928T100019_N0206_R122_T32TQN_20180928T170858.zip
  • S2A_MSIL1C_20181115T101251_N0207_R022_T32TQN_20181115T103245.zip
Software used

Internet browser, SNAP + Sentinel-2 Toolbox, aria2 (download utility), text editor (e.g. Mousepad, Notepad, etc..)

NOTE: This tutorial was prepared on a Linux Ubuntu 16.04 OS, steps may differ for other OS (Windows version under development)

Reference folder structure

Folder structure referenced in the exercise is as follows:

  • /HYDR02_FreshWaterQuality_Trasimeno
    • /AuxData – includes auxiliary data needed for the processing – you can re-create them yourself or download them below
    • /Original – should contain your downloaded input products
    • /Processing – should contains all intermediate and final results of the processing organized in subfolders
      • /Pre-processed
      • /C2RCC
      • /Code

Data download – ESA SciHUB

First, we will download the 12 Sentinel-2 products which we will use for the exercise, from the Copernicus Open Access Hub using the online interface. Launch your Internet Browser and go to https://scihub.copernicus.eu

Go to “Open HUB”, if you do not have an account please register by going to “Sign-up” in the LOGIN menu in the upper right corner. (Note that the position of the buttons may change over time since the creation of this tutorial)

If you had to fill-in the registration form, you will receive an activation link by e-mail. Once your account is activated or if you already have an account, “LOGIN”.
Navigate to Lago Trasimeno in central Italy (approximate area – blue rectangle).

Switch to drawing mode (marked with green circle below) and draw a search rectangle approximately as indicated above in blue. Open the search menu by clicking to the left part of the search bar and specify the parameters below. Then click on the “Search” icon.
Sensing period: From 2018/04/16 to 2018/11/15
Check Mission: Sentinel-2
Product type: S2MSIL1C

Press “Search” (red arrow below).

Using the icon, import only the following products to Cart (in red you can see the acquisition date and in blue the tile identifier):
S2A_MSIL1C_20180416T100031_N0206_R122_T32TQN_20180416T120852.zip
S2B_MSIL1C_20180511T100029_N0206_R122_T32TQN_20180511T134849.zip
S2B_MSIL1C_20180531T100029_N0206_R122_T32TQN_20180531T120906.zip
S2B_MSIL1C_20180630T100029_N0206_R122_T32TQN_20180630T134105.zip
S2B_MSIL1C_20180710T100029_N0206_R122_T32TQN_20180710T135302.zip
S2A_MSIL1C_20180718T101031_N0206_R022_T32TQN_20180718T171829.zip
S2B_MSIL1C_20180809T100019_N0206_R122_T32TQN_20180809T140850.zip
S2B_MSIL1C_20180822T101019_N0206_R022_T32TQN_20180822T142412.zip
S2B_MSIL1C_20180829T100019_N0206_R122_T32TQN_20180829T190727.zip
S2A_MSIL1C_20180906T101021_N0206_R022_T32TQN_20180906T122259.zip
S2B_MSIL1C_20180928T100019_N0206_R122_T32TQN_20180928T170858.zip
S2A_MSIL1C_20181115T101251_N0207_R022_T32TQN_20181115T103245.zip

Then click on the Profile icon in the upper left corner (marked with blue circle above) and go to Cart. You should now have six products in your cart. Click on the arrow to Download the Cart.

The grey products are currently placed in an Offline (Long Term) Archive: to learn how to request them please follow the steps outlined here: https://scihub.copernicus.eu/userguide/LongTermArchive

TIP: Alternatively, you can try to retrieve the products from other repositories such as: PEPS (https://peps.cnes.fr/rocket/#/home) or ONDA DIAS Catalogue (https://catalogue.onda-dias.eu/catalogue/) or others. Registering for a free account is usually necessary, but archived data retrieval will be faster than with Open Access Hub.

A products.meta4 file will be downloaded to your Downloads folder. To download our data, we will use aria2 tool. To use the tool, we first need to place the products.meta4 file to the folder where we wish our data to be downloaded to e.g.: …/HYDR02_FreshWaterQuality_Trasimeno/Original

First let’s test our aria2 installation. To do this we open the Command Line and type:

aria2c

The correct response should be as follows (example from Linux):

NOTE 1: If (and only if) the response is “-bash aria2c: command not found” or similar. Then we have to install the tool.
For Linux OS command line type: sudo apt-get install aria2
When requested type: Y
Then you can test your installation again.
For Windows and other OS see: https://aria2.github.io/

If the response is shows an error see   NOTE 1. If you have received the correct response (may differ for other OS), then we can run the tool by typing following commands in the command line (replace <username> and <password> with your login credentials for Copernicus Open Access Hub):

cd .../HYDR02_FreshWaterQuality_Trasimeno/Original 
aria2c --http-user='<username>' --http-passwd='<password>' --check-certificate=false --max-concurrent-downloads=2 -M products.meta4

The first line changes our directory to the target directory (replace the path with the full path to where you wish to place the downloaded products). The second line runs the download tool. All twelve products will be downloaded to the /Original folder automatically, two products in parallel. This might take some time.

TIP: Note that the constraint of maximum 2 parallel downloads at a time is imposed by the Copernicus Access Hub, if you increase the number, your download will fail.

Data exploration

Launch SNAP (icon on desktop ). When the SNAP window opens click Open product  and navigate to:  …/HYDR02_FreshWaterQuality_Trasimeno/Original/

Open the first S2 product (from 16 April 2018). We can first investigate the structure of the Sentinel-2 Level 1C products. Click on the dot next to the product name to expand the structure. The L1C products contain (among others):

  • 13 TOA (top-of-atmosphere) reflectance bands
  • Quality flags

Now, let’s visualize the product. We will visualize it in true (natural) colors and as a false color composite which is better for distinguishing water surfaces. Right-click the product and click Open RGB image window, a new window will open. From the drop-down menu select:

Profile : Sentinel 2 MSI Natural Colors

Click OK. the image opens, repeat the same but choose the false colour Profile. Profile : Sentinel 2 MSI False-color Infrared

Finally, go to Window Tile Horizontally.

NOTE 2: The Sentinel-2 data are distributed as 100×100 km tiles resampled to a common grid in WGS84 UTM projection at two levels of processing:
• Level-1C – Top-Of-Atmosphere reflectances, systematically generated since the start of the mission.
• Level-2A – Bottom-Of-Atmosphere reflectances (atmospherically and radiometrically corrected), systematically produced for products acquired over Europe since the spring of 2017, the coverage has increased through 2018 to reach global coverage in the beginning of 2019. Can be produced on user side by applying the Sen2Cor algorithm.

STEP 1 – Pre-processing

Processing the data one by one would not be very time effective. However, we can use either the Batch Processing tool available in SNAP or the GPT tool (command line) to process all images automatically.

To use either method, we first need to define the process we want to apply and all its steps. We can do this using the GraphBuilder tool. Another advantage of the GraphBuilder is that only the final product will be physically saved, and we save valuable disk space. 

So, let’s build our graph. First, in the Product Explorer select the first loaded product (so it is highlighted), then go to Tools GraphBuilder.

At the moment, the graph only has two operators: Read (to read the input) and Write (to write the output).

The 13 bands in Sentinel-2 products do not all have the same resolution (therefore size) as mentioned in   NOTE 3. Many operators do not support products with bands of different sizes so first we need to resample the bands to equal resolution. 

To add the appropriate operator, right-click the white space between existing operators and go to Add Raster Geometric Resample.

NOTE 3: The input product contains 13 spectral bands in three different spatial resolutions (The surface area measured on the ground and represented by an individual pixel).  When we open the RGB view all our input bands have 20 m resolution, however, the view is displayed in the full 10 m resolution.

Credits: ESA 2015

A new operator rectangle appears in our graph and new tab appears below. Now connect the new Resample operator with the Read operator by clicking to the right side of the Read operator and dragging the red arrow towards the Resample operator.

Next step will be to subset the images to the area of interest, we do this by right-clicking the white space somewhere right of the resample operator and going to Add Raster Geometric Subset. Connect the Subset operator with the Resample operator.

Finally, connect the Subset operator to the Write operator.

Now, let’s set the parameters. In Resample tab, we set: 

Under Define size of resampled product: Select the “By reference band from source product” and choose the B2 band(we will resample all the bands to 10m resolution).

In the Subset tab, select Geographic Coordinates, in the text-window paste the subset polygon coordinates in Well-Known-Text format (WKT) from Expressions_WQ.txt file in: …/HYDR02_FreshWaterQuality_Trasimeno/

POLYGON ((12.009554862976 43.20463943481, 12.198662757873 43.20463943481, 12.198662757873 43.0686149597, 12.009554862976 43.0686149597, 12.009554862976 43.20463943481, 12.009554862976 43.20463943481))

We can leave the Read and Write tabs unchanged and save the graph as STEP1_Graph_Prep.xml to: …/HYDR02_FreshWaterQuality_Trasimeno/Processing/ by clicking Save at the bottom of the window and then close the GraphBuilder window.

In this tutorial, we will use the GPT and shell script to automatize the processing of our 12 products. To do this, we must first edit the saved graph file.

To edit the file, we leave SNAP and in the file explorer, navigate to the …/Processing folder and right-click on the saved graph file and open it with text editor e.g. Mousepad or Notepad. When the file is open, go to View Line Numbers for easier navigation (if not available you can use the Edit → Go to … to find lines ).

Now, we need to replace the input and output file path with a pattern that will allow the script (See   NOTE 4) to recognize and replace it with appropriate value for each file automatically.

NOTE 4: A shell script is a computer program designed to be run by the Unix shell, a command-line interpreter. The various dialects of shell scripts are considered to be scripting languages. The one used in this exercise is called Bash script (https://en.wikipedia.org/wiki/Bash_(Unix_shell)). Typical operations performed by shell scripts include file manipulation, program execution, and printing text. If you wish to run this exercise on Windows, you can replicate the steps in any programming language you are familiar with, such as Python or R.

In line 7 (in the node with id=’Read’) replace the path to input file with (between <file> and </file>): $Input (“$” signifies that what follows is a parameter).

In line 51 (in the node with id=’Write’) replace the path to output file with (between <file> and </file>):  $Output

The next step will be different depending on your OS:


Linux

Create new file named STEP1_script_gpt.sh in the following folder: …/Processing/Code/ and copy-paste and edit the following code:

#!/bin/bash

output_folder=.../HYDR02_FreshWaterQuality_Trasimeno/Processing/Pre-processed/
nameStart=Subset_S2_MSIL2A_
nameEnd=.dim

for i in $(ls -d -1 .../HYDR02_FreshWaterQuality_Trasimeno/Original/*.zip)
do
	product_basename=$(basename $i)
	ac_date=$(echo $product_basename | cut -d '_' -f 3)
	Output_pathname=$output_folder$nameStart$ac_date$nameEnd
	gpt .../HYDR02_FreshWaterQuality_Trasimeno/Processing/STEP1_Graph_Prep.xml -PInput=$i -POutput=$Output_pathname
done

On line 1, we specify that the script should be run with Bash scripting language. (See   NOTE 4) On line 3, 4 and 5, we specify our desired output folder path (replace the path with your desired output path), and the pattern beginning and ending of the output name respectively.

Line 7 signifies the start of the loop. The script will loop over all files in the specified folder ending with “.zip”. In each iteration, the path to one input file is denoted by “i” (replace the path with full path to your downloaded input products). “do” on line 8 and “done” on line 13 enclose the steps that should be performed on each input product. On line 9, we extract the name of the input product; on line 10, we extract the date of the acquisition from the name. On line 11, we create the final path and name of our output, combining our specified output folder path, the beginning of the output name, the acquisition date and specified name end including the format suffix.

On line 12, we call GPT and specify the path to our saved and edited graph file. Then with -P{variable name} we specify the parameters to be replaced in the graph file (marked with $ in the graph file). We have set the variables when we edited the .xml graph file – Input, Output. On line 13, the loop is closed.

To run the script, we navigate to the folder where it is saved (…/HYDR02_FreshWaterQuality_Trasimeno/Processing/Code/) and right-click on the white space and go to Open Terminal Here. In the terminal type:

sh STEP1_script_gpt.sh

where STEP1_script_gpt.sh is the name of the Bash script file. Click ENTER to run the script.

The processing time will depend on your PC/VM setup. (approx. 3 mins with 30GB ram). You can check the …/HYDR02_FreshWaterQuality_Trasimeno/Processing/Pre-processed for the output products. Close the terminal window.


Windows

Under development


Now, let’s have a look at the pre-processed images. Go back to SNAP and go to File Open Product. Navigate to …/HYDR02_FreshWaterQuality_Trasimeno/Processing/Pre-processed and open all the pre-processed images in SNAP.

Once all products are loaded, right-click the first and go to Open RGB image window. Select the Sentinel 2 MSI Natural Colors profile and click OK. You can repeat this step for all products and the go to Window Tile Evenly.

Even by visual assessment we can determine that increase in biological production occurs throughout the summer and fall and decreases again towards colder months. However, to be able to quantify the concentration of phytoplankton and other substances we need to process the data further.

STEP 2 – Atmospheric correction & bio-optical model

As mentioned previously, atmospheric correction is crucial, in order to be able to separate different optically active constituents in water (chl-a, TSM, CDOM, phosphorus etc.). Number of different atmospheric correction algorithms exist for Case 2 waters. Here we will use the Case 2 Regional Coast Color Processor.

While available in SNAP graphical interface and in the command line interface (GPT), it is not available in the Graph builder tool. We cannot therefore create a graph in the GUI as we have done in the previous step, however we can use the GUI to create a parameter file which we can then use in combination with GPT and bash script (or python or another scripting language) to run it in batch.

The C2RCC processor allows many different parameters to be set, to accommodate more accurate atmospheric correction and extraction of the optically active constituents.

Let’s go to SNAP and set our parameters. Go to Optical Thematic Water Processing C2RCC Processors S2-MSI. In the I/O Parameter tab, we do not need to change anything.

In the Processing Parameters tab, we can set following:
Salinity: 0.0001 PDU
Elevation: 257 m

The atmospheric parameters – Temperature (Water), Ozone and Air Pressure at Sea Level – will be set in our bash script, because here we can not specify different value for each image.

We will leave all other parameters as default, including the thresholds and factors used for TSM and Chl-a extraction. Finally, we can select which data we wish to include in the output. Here we select:
• Output normalized water leaving reflectances
• Output irradiance attenuation coefficients
• Output uncertainties

Then click File Save Parameters and save them as c2rcc_param.xml (XML format as the SNAP graphs) to: …/HYDR02_FreshWaterQuality_Trasimeno/Processing/

Go to the file location (in file explorer) and open it. It will automatically open in the internet browser window. Now, we will use another Bash processing script to run the C2RCC algorithm on all our images and assign the respective water temperature, ozone and pressure values. These parameters will naturally change over time. The values we will use, were manually acquired from ECMWF ERA-Interim global atmospheric reanalysis dataset (https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim) for each image acquisition date. Create a CSV file named Met_Aux.csv and paste the following tab-delimited values (can be created in Excel or text editor e.g. Notepad or Mousepad and saved as CSV). Save the CSV in the …/AuxData folder.

Date	Ozone[DU]	MeanSeaLevelPressure[hPa]	Temperature[C]
20180416	375.9	1010.342	10
20180511	333.5	1015.939	18.2
20180531	326.83	1016.981	20
20180630	345.41	1011.56	25
20180710	335.42	1013.863	26
20180718	323.14	1014.703	26
20180809	314.5	1016.832	27
20180822	321.41	1015.507	27
20180829	310.06	1016.216	27
20180906	299.32	1013.29	22.8
20180928	293.53	1017.787	20
20181115	289.1	1024.182	10

Now we can go to the script we will use to run the processor on all the images automatically. It is similar to the one used previously for pre-processing.


Linux

Create a new empty file named STEP2_script_C2RCC.sh and place it in: …/HYDR02_FreshWaterQuality_Trasimeno/Processing/Code/

Open the file with text editor and copy-paste the following code:

#!/bin/bash

output_folder=.../HYDR02_FreshWaterQuality_Trasimeno/Processing/C2RCC/
metData=.../HYDR02_FreshWaterQuality_Trasimeno/AuxData/Met_aux.csv
paramFile=.../HYDR02_FreshWaterQuality_Trasimeno/Processing/c2rcc_param.xml
oldEnd=.dim
newEnd=_C2RCC.dim

for i in $(ls -d -1 .../HYDR02_FreshWaterQuality_Trasimeno/Processing/Pre-processed/*.dim)
do
	name=$(basename $i) 
	idate=$(basename $i | cut -d '_' -f 4 | cut -d 'T' -f 1)
	ozone=$(grep -F $idate $metData | cut -d ',' -f 2)
	press=$(grep -F $idate $metData | cut -d ',' -f 3)
	temp=$(grep -F $idate $metData | cut -d ',' -f 4)
	gpt c2rcc.msi -SsourceProduct=$i -p $paramFile -Pozone=$ozone -Ppress=$press -Ptemperature=$temp -t $output_folder${name%$oldEnd}$newEnd 
done

On lines 3 to 7 we specify input parameters, such as the path to out output directory, path to the meteorological ancillary data (table above), the path to the C2RCC parameter file saved from SNAP GUI and the input file end and output file end, respectively.

Line 9 signifies the start of the loop. The script will iterate over all files in the specified folder ending with “.dim”. In each iteration the path to one input file is denoted by “i”. “do” on line 10 and “done” on line 17 enclose the steps that should be performed on each input product. On line 11, we extract the input file name, on line 12 we extract the image acquisition date. On 13 to 15 we extract from the Met_aux.csv file the ozone, air pressure at sea level and water temperature for the appropriate date. 

Finally, on line 16 we call GPT (SNAP command line utility) and the c2rcc.msi tool. Then, we load the parameter file we have created in the SNAP GUI. However, in this file the ozone pressure and temperature parameters were not set in the parameter file, instead we overwrite the values in the parameter file with the values extracted from the Met_aux.csv file. “done” at line 17 closes the loop.

To run the script, we navigate to the folder where it is saved (…/HYDR02_FreshWaterQuality_Trasimeno/Processing/Code/) and right-click on the white space and go to Open Terminal Here. In the terminal type:

sh STEP2_script_C2RCC.sh

where STEP2_script_C2RCC.sh is the name of the Bash script file. Click ENTER to run the script.


Windows

Under development


Visualization

Finally, we can go back to SNAP and go to File Session Open Session, then click Yes in the dialog that opens. Then open all the processed files from your output folder in the order of acquisition – you may need to open them one-by-one.

Now, let’s investigate the results. In Product Explorer open the structure of one product and go to Bands. Here we choose the product from 22 August 2018 and opened the conc chl_conc & tsm_conc and the iop iop_agelp (gelbstoff == CDOM attenuation coefficient).

For each opened view you can go to the Color Manipulation tab and if you have requested the auxiliary data for this exercise, click on Import color palette from text file. In …/HYDR02_FreshWaterQuality_Trasimeno/AuxData you can find the appropriate color map for Chlorophyll, TSM and aCDOM. To keep the pre-set values, click No in the Import Color Palette dialog. Alternatively you can recreate the color palette yourself by dragging and the sliders and adding desired color to them.

To visualize the images next to one another you can go to Window Tile Horizontally.

Finally you can also use the Time Series tool to visualize the development of chl-a for specific observation point over the study period.

Further reading and references

Bresciani, Mariano, Monica Pinardi, Gary Free, Giulia Luciani, Semhar Ghebrehiwot, Marnix Laanen, Steef Peters, Valentina Della Bella, Rosalba Padula, and Claudia Giardino. 2020. “The Use of Multisource Optical Sensors to Study Phytoplankton Spatio-Temporal Variation in a Shallow Turbid Lake.” Water 12(1):284.

Dörnhöfer, Katja, and Natascha Oppelt. 2016. “Remote Sensing for Lake Research and Monitoring – Recent Advances.” Ecological Indicators 64:105–22.

Giardino, C., V. E. Brando, P. Gege, N. Pinnel, E. Hochberg, E. Knaeps, I. Reusen, R. Doerffer, M. Bresciani, F. Braga, S. Foerster, N. Champollion, and A. Dekker. 2019. “Imaging Spectrometry of Inland and Coastal Waters: State of the Art, Achievements and Perspectives.” Surveys in Geophysics 40(3):401–29.

Koponen, Sampsa. 2006. Remote Sensing of Water Quality for Finnish Lakes and Coastal Areas. Helsinki University of Technology.

Toming, Kaire, Tiit Kutser, Alo Laas, Margot Sepp, Birgot Paavel, and Tiina Nõges. 2016. “First Experiences in Mapping Lake Water Quality Parameters with Sentinel-2 MSI Imagery.” Remote Sensing 8(8):640. Warren, M. A., S. G. H. Simis, V. Martinez-Vicente, K. Poser, M. Bresciani, K. Alikas, E. Spyrakos, C. Giardino, and A. Ansper. 2019. “Assessment of Atmospheric Correction Algorithms for the Sentinel-2A MultiSpectral Imager over Coastal and Inland Waters.” Remote Sensing of Environment 225:267–89.

Doing business with us

EO expertise at your service

We are used to prepare and release training materials, but also to work in partnership with the Earth Observation industry and we may complement your business case with training or consultancy services. Do not hesitate to get in touch with us to get an offer for a customized training event or EO consultancy services…