Detecting Technical Anomalies in Water-Quality Data From River Networks

# Detecting Technical Anomalies in Water-Quality Data From River Networks
## Priyanga Dilini Talagala prital.netlify.appMay 2022

---

## Hello from the team!

---
class: inverse, middle, center

# What is an anomaly?

<!--High dimentio al context: This deviation can be defined in terms of either distance, density

time series context: historical data"-->

---
class: inverse, middle, center

# Statistical anomaly (outlier, novelty) is an observation that deviates markedly from other members of the dataset.

<!--High dimentio al context: This deviation can be defined in terms of either distance, density

time series context: historical data"-->

---
class: middle, center

]

---
class: middle, center

].pull-right[
#### Technical issues in the sensor equipment

<img src="fig/15_definition_b.png" width="90%" style="display: block; margin: auto;" />
]

---

## Motivation

---
## Motivation

<!-- Water quality breaches associated with real events

### Features of nature-based anomaly types

- Double peaks in turbidity: peaks which are abrupt and clear out of the typical range. If two (or potentially more) peaks are observed in a short time window, may lead to suspect real event (contamination, dredging, etc).

<!--Double peaks in turbidity: peaks which are abrupt and clear out of the typical range. Would be classified as sensor-based error A-type if it was single, but the fact that two (or potentially more) peaks are observed in a short time window (say less than one hour), may lead to suspect real event (contamination, dredging, etc)

Dredging is the act of removing silt and other material from the bottom of bodies of water.

-  Massive input of SPM load, possibly starting by a strong peak. This is the typical signature of a flood event

but there is no direct relationship between the discharge and the incoming SPM load (history effect, hysteresis, etc).

- Very large resuspension event. These turbidity pulses of few hours can be related to the resuspension of recently deposited sediment, or interaction with the tide.

- Non-flushing periods. In low discharge and low tidal range conditions, the salty marine waters are not expelled out of the estuary. The salinity remains high for days, which can have consequences on the local ecosystems and the aquifer.
.-->

<!--
## Technical issues in the sensor equipment

## Technical issues in the sensor equipment

- Low battery power

- Biofouling of the probes

- Errors in calibration

- Rust

- Sensor maintenance activities

- Sensor breakdowns -->

---
## Study area and water-quality data

- Pringle Creek, NEON (National Ecological Observatory Network) aquatic sites located in Wise County, Texas, USA
- NEON usually has two sensor locations on any river in the network where water quality is measured using in-situ sensors.
- [https://data.neonscience.org/data-products](https://data.neonscience.org/data-products)

---

# In this work we define an anomaly as an observation that has an unexpectedly low conditional probability density.

<!--conditioning means incorporating new information

form th upstream behavioir we expect something, what we observe in the downstream is very different from what we expected then it become an anomaly. -->

---
## What is "conditioning"?

---

## What is "conditioning"?

---
## What is "conditioning"?

---
## What is "conditioning"?

---
## What is "conditioning"?

---

## What is "conditioning"?

---

# Different types of "conditioning" (New information)

---

**Only one sensor**

lagged (past) downstream observations

].pull-right[

Contemporaneous (simultaneous) downstream observations

]
---
**Only one sensor**

lagged (past) downstream observations

<img src="fig/21_oddwater_logo.png" width="30%" style="display: block; margin: auto;" />
<h1 style="font-size: 1rem">devtools::install_github("pridiltal/oddwater")</h1>

].pull-right[

Contemporaneous (simultaneous) downstream observations

<h1 style="font-size: 1rem">Talagala, Priyanga Dilini, et al. "A Feature-Based Procedure for Detecting Technical Outliers in Water-Quality Data From In Situ Sensors." Water Resources Research 55.11 (2019): 8547-8568.</h1>

]

---

## Two sensors in close proximity with connected flow

---

## Two sensors in close proximity with connected flow

<img src="fig/20_idea_b.png" width="100%" style="display: block; margin: auto;" />
---
## Two sensors in close proximity with connected flow

<img src="fig/20_idea_c.png" width="100%" style="display: block; margin: auto;" />
---

## Two sensors in close proximity with connected flow

- We combine all these conditioning (new information) to differentiate "technical issues" from "real events".

---
class: inverse, center, middle

## Our conditional cross-correlation based algorithm (conduits based approach)

devtools::install_github("PuwasalaG/conduits")
---
#### Pringle Creek data from NEON

---

.pull-left[
## Lag time estimation
- Assume the lag time between two sensor locations depends on the upstream river behavior

- Use conditional cross-correlations to estimate the lag time

- let `$x_t$` : Turbidity upstream, `$y_t$` : Turbidity downstream and `$z_t$` : {level upstream, temperature upstream}

`$x^*_t = \frac{x_t - \text{E}[x_t|\mathbf{z}_t]}{\sqrt{\text{V}[x_t|\mathbf{z}_t]}}$` and `$y^*_t = \frac{y_t - \text{E}[y_t|\mathbf{z}_t]}{\sqrt{\text{V}[y_t|\mathbf{z}_t]}}$`

].pull-right[

```r
library(conduits)

fit_mean_y <- data %>%
 conditional_mean(turbidity_downstream ~
 s(level_upstream, k = 5) +
 s(temperature_upstream, k = 5))

fit_var_y <- data %>%
 conditional_var(turbidity_downstream ~
 s(level_upstream, k = 4) +
 s(temperature_upstream, k = 4),
 family = "Gamma",
 fit_mean_y )

fit_mean_x <- data %>%
 conditional_mean(turbidity_upstream ~
 s(level_upstream, k = 5) +
 s(temperature_upstream, k = 5))

fit_var_x <- data %>%
 conditional_var(turbidity_upstream ~
 s(level_upstream, k = 4) +
 s(temperature_upstream, k = 4),
 family = "Gamma",
 fit_mean_x)
```
]
---
.pull-left[

## Conditional cross-correlations

`$$r_k(\mathbf{z}_t) = \text{E}[x_t^*y^*_{t+k}|\mathbf{z}_t] \quad \text{for} \quad k = 1,2,...$$`

- To estimate `$r_k(\mathbf{z}_t)$` we fit the following GAMs

- Estimating time delay

`$$\hat{d}_{t}(\mathbf{z}_t) = \underset{k}{\operatorname{argmax}}\quad \hat{r}_{k}(\mathbf{z}_t)$$`
].pull-right[

```r
fit_c_ccf <- data %>%
 conditional_ccf(
 formula = I(turbidity_upstream*turbidity_downstream) ~
 splines::ns(level_upstream, df = 3) +
 splines::ns(temperature_upstream, df = 3),
 lag_max = 10,
 fit_mean_x, fit_var_x, fit_mean_y, fit_var_y,
 df_correlation = c(3,3))

new_data <- fit_c_ccf %>% estimate_dt()
```

]
---

### Outlier detection based on Conditional cross-correlation and Extreme value theory

`$$\text{turbidity_down}_t = \phi_0 + \sum_{i=1}^pg_i(z_{i,t_l}) + \sum_{j=1}^qh_j(\text{turbidity_down}_{t-j}) + \varepsilon_t$$`
.pull-left[
- If `$z_i$` is a contemporaneous variables measured at downstream sensor, then `$t_l = t$`.

- If it is a lag downstream variable at lag l, then `$t_l = t-l$`.

- If `$z_i$` is a predictor from the upstream sensor, then `$t_l = t-dt$` where `$dt$` is the estimated time delay.

].pull-right[

]
---

### Outlier detection based on Conditional cross-correlation and Extreme value theory

<img src="fig/25_An3.png" width="55%" style="display: block; margin: auto;" />
---
class: center, middle, inverse

# Thank You

Slides available at: prital.netlify.app

priyangad@uom.lk

pridiltal

@pridiltal

#### Acknowledgement

Australian Research Council (ARC) Linkage project (grant number: LP180101151) "Revolutionising high resolution water-quality monitoring in the information age".