Bad data versus big data (or big bad data!!)

There's currently quite a buzz about "big data" and how water utilities might dig into all the data they collect in order to be "smarter." Several of my colleagues are investigating ways to do this under the banner of Smart Integrated Infrastructure (SII) and Smart Water Analytics. Pretty cool stuff. In a couple of conversations on the topic I half-jokingly said that wastewater doesn't have big data, it has crap data!  To avoid misunderstanding, I should clarify that by "crap" I'm referring to it being bad data and not just data describing the fecal material we treat!

Over the years I've been involved with various projects and discussions on generating and handling data in wastewater treatment. A few years ago I was involved in a couple of WERF Projects focused on developing Decision Support Systems (DSS) to prevent plant upsets, along with Dr Nancy Love and Advanced Data Mining (ADMi). The folks at ADMi did some nice data analytics to pick out anomalies that might indicate toxins in the plant influent, but one of the major hurdles we ran into was distinguishing anomalies due to toxins and anomalies due to measurement problems. This reminded me of what my ex-boss and mentor, Dr John Watts, used to drill into me which is you need to focus on good primary measurements in order to have confidence in your data. Wastewater is a tough place to try to do that! As I said, a lot of our data is bad.

So, here is my brain dump on some of the keys to making big data work in wastewater, and avoiding the pitfalls of bad big data (there's a tongue-twister there somewhere...)!

5 keys to making big data work

1. Focus on data quality rather than quantity

Starting from Dr Watt's sage advice to me years ago, and written up in one of his rare papers here, no amount of fancy analytics can overcome measurement errors, whether that's noise, drift or interferences.  You need to have confident in your primary sensors and analyzers otherwise your big data analytics will be crunching numbers that are meaningless and therefore any results you'll get will be useless.  Crap data = crap analytics!

In order to gain confidence in your data, you need to do 3 things with your sensors/analyzers:

1. Clean them - wastewater is an extremely fouling environment an not the best place to put scientific equipment.  My experience has been that everyone underestimates how quickly sensors become fouled.  Go for auto-cleaning whenever possible and avoid installing anything in raw sewage or primary effluent unless you really need the measurement (see Key #2!) as these areas are particularly prone to fouling. Mixed liquor is actually an easier place to take measurements and final effluent the easiest of all!
2. Calibrate them - this is generally understood, though the frequency of calibration, particularly for sensors that tend to drift, is generally shorter than ideal.
3. Validate them - this is the piece that's overlooked by most instrumentation suppliers, I think. Analytics to validate the measurements, particularly during calibration is an area that needs much more attention.

Much of the work that Dr Watts did at Minworth Systems was focused on automating these 3 things and I've seen very few instruments come close to what he did 20 years ago!

2. Measure what matters most

I could probably make this blog an ode to John Watts and fill it with his anecdotes.  One of my favorites was one where a customer asked him to install a dissolved oxygen (DO) probe in an anoxic zone. He suggested it would be cheaper to install a wooden probe and write 0 mg/L on a fake display!  Maybe that's a little harsh, but the point is that we should only measure things that are useful to help us to run the plant and that we're actually going to use to make some decision. Generally we're lacking many important and basic measurements in our treatment plants (e.g. dissolved oxygen in the aerated basins, airflow to each aeration zone and electricity use by blowers), but we need to be careful in our enthusiasm not to swing to the other extreme and start measuring stuff that's interesting but not useful. You can spend some serious money measuring ammonia and nitrate all over a treatment plant, but unless you're actually using it for control, the measurements will eventually be ignored and the instrument neglected.  It's much better to have a handful of good instruments, positioned in locations where you're actually measuring something you can control, then there's motivation to keep those sensors running well (see Key#1!)

3. Think dynamics, not steady state

A lot of the design and operational guidance in text books and training materials have simple equations into which you plug a single number to get your answer (e.g. sludge age calculation or removal efficiency). Similarly, influent and effluent samples are usually flow-weighted or time-averaged composites (worse-still, grab samples!).  All this means that we're used to thinking and talking about average daily conditions.

Graphic showing difference between composite
 sample and continuous measurement
(Courtesy Dr. Leiv Rieger/WEF,
taken from WEF Modeling 101 Webcast)

However, the reality is that our treatment plants see significant daily variations in flows and concentrations and therefore we need to look at them as a dynamic system. This was first brought home to me when I was working on a plant in the UK doing biological phosphorus removal back in the late 1990's. We had an online phosphate analyzer taking measurements at the end of the aeration basin just prior to the clarifiers and we would see daily phosphate peaks of 1 or 2 mg/L every afternoon for just an hour or so, but the effluent composite sample measurements would be pretty consistently below 0.2 mg/L. To understand our wastewater treatment systems we need to measure their dynamics and then analyze that good data (having adhered to Keys #1 and #2, of course!!)  

4. Recognize different timescales

Hand-in-hand with dynamics is the need to think about different timescales:

• Diurnal (daily) variations
• Weekly trends (especially weekend versus weekday differences)
• Seasonal shifts

For each of these, the data analytics needs are quite different and need to be thought through properly. For diurnal variations, it's useful to compare one day to the next by maybe overlaying the dynamic data; for weekly trends we can do something similar over a 7-day horizon; for seasonal shifts we need to plot out long-term trends and compare them to temperature and maybe rainfall shifts.

5. Consider how to handle outliers and extraordinary events

This blog is getting long, so I'll try to wrap up this 5th key quickly!  In data analytics it's common practice to identify and eliminate outliers, assuming they're either "bad" measurements or not typical and therefore we can ignore them.  However, thinking back to my involvement in the WERF projects on DSS, a lot of what is done at wastewater treatment plants is trying to keep the process stable in response to abnormal events such as upsets from shock loads or toxins, or more typically responding to wet weather.  This means we need to identify these "outliers" but rather than throw them away, we need to decide how to respond. Maybe this is a topic for another blog?!!

WWTmod2014 - The Process Design and Optimization Seminar

I'm sitting in Brussels airport the day after the last day of the latest in a series of seminars focused on process modeling under the moniker "wwtmod" for "waste-water treatment modeling." The previous 3 biannual wwtmod's were held at Mont-Sainte-Anne, just outside Quebec City in Canada, but the latest in the series - WWTmod2014 - was held in the lovely European town of Spa, Belgium. In this blog I'd like to summarise a few of the many highlights of WWTmod2014, but first I'd like to digress a little to ponder the name and the focus of this seminar series.

WWTmod2008 was first concieved to be a follow-up to a series of modeling-focused seminars called "Kolle-kolle" which I'm told were an excellent set of seminars in the 1990's to discuss wastewater treatment models, from which the IWA's "ASM" activated sludge models were developed. Process models have come a long way since the 1990's, shifting from the realm of academia to mainstream design and operations also. Any consultancy worth it's salt will use process models to develop or refine their designs. In fact, it's all but impossible to design a good nutrient removal facility doing nitrogen and phosphorus removal without the use of models. A few stalwart old-scholers might argue differently but they're a fading voice. This has meant that the WWTmod seminars, from the very start, have pulled together academia and practitioners to discuss process models and these models are now at the heart of process design. Despite the name including "mod" in the title, the seminars very quickly shifted into discussions about the mechanisms and processes that go into the models and so I'd argue that it is now the premiere seminar for process designers and those wanting optimize their waste water plant operations (maybe using models to help!) I can't think of another conference that draws together the top researchers from universities with experts from practice, the way that it's done at WWTmod. If you're a university researcher wanting to understand the challenges of practice, come to a WWTmod seminar. If you're a practitioner wanting to discover the latest tools and techniques to overcome your challenges, come to a WWTmod seminar!

So, what were the highlights of the latest, WWTmod2014? The program covered a broad range of topics in wastewater treatment, from screenings and grit to model extensions for trace organics; the latest in nitrogen reaction pathways and current thinking on phosphorus removal; primary settler and secondary settler performance; integrated modeling and LCA. At the closing session, the chairman of the scientific committee, Ingmar Nopens, had a slide with a few "Useful Quotes and Concepts", so maybe I'll focus on those:

"Wipers versus Washers" - there's nothing like the topic of bum wiping to get a partly humorous and partly serious discussion going on wastewater treatment. It's not pleasant to discuss the fate of toilet paper but there are serious considerations when it comes to understanding the biodegradability of the material when it gets to the treatment plant. If it's not biodegradable, it uses up treatment capacity and increases the mass of residuals that have to be handled. If it is biodegradable, it's a valuable source of material for biogas generation. Ultimately there wasn't a consensus on its biodegradability, but on the whole it was thought that it wasn't too degradable within the wastewater treatment plant (despite what it says on roll!)

The "Drinker-Barman Concept" - trying to be a little radical, I suggested that the Monod equation, near and dear to the hearts of all who model biological processes, may be past its sell-by date and that consideration of diffusion may be key to shifting to simpler kinetics where the "intrinsic" half-saturation coefficient can be ignored. I'm already getting too much into the details here but needless to say I had to use a nice analogy involving beer to make my point and hopefully make it more palitable!

"Communism/socialism versus capitalism in biological processes" - Dr Zhiguo Yuan discussed how the models we use assume that substrate is shared evenly amongst the different organisms that use it (like socialism shares out everything) but in fact there is competition for the substrate that is akin to biological capitalism.

"Being within the law is not always the good thing" - Dr George Ekama has produced many of the most memorable quotes at these seminars. A few years back he said (I'll get it wrong, but hopefully close enough): "The main problem is to keep the main problem the main problem" and he's also the source of the wipers quote above. His latest quote may sound revolutionary but it's really meant to highlight that some of regulations that govern treatment standards don't always result in the best overall environmental solution. In particular we discussed the focus on carbon emissions at wastewater treatment facilities and legislation pushing them to reduce energy use and carbon footprint when in fact the carbon emissions per person served by a treatment plant is peanuts - maybe less than 1% of their overall emissions - and they'd be better driving less or using less hot water.

Beyond these highlights another fun part of the conference was the use of twitter to post comments. I confess I was a twittering fool but it was fun! Check out the hashtag #wwtmod2014 to see!

Posted with Blogsy

Thinking About Resource Recovery

I'm helping with a new WEF special publication entitled "Moving Toward Resource Recovery Facilities" in which we're looking at wastewater as less of a disposal option and more as a source of materials that can be recovered. The book will follow the general themes of looking to recover Nutrients, Energy and Water which gives the handily memorable acronym N-E-W! (see WERF workshop for more background on N-E-W)

Ostara Prills

Give me an N

So, let's start with "N" for Nutrients. Way back at the start of the 20th Century, wastewater inventors and innovators were looking for ways to recover nitrogen from sewage and saw the benefit of using sewage sludge as a fertilizer, but then along came the Haber-Bosch process that revolutionized fertilizer production by fixing nitrogen from the atmosphere and took away some of the urgency for nitrogen fertilizers. Fast forward to the start of the 21st Century and now the focus has shifted to phosphorus recovery. Unlike nitrogen, phosphorus does not exist naturally in a gaseous form and so the natural phosphorus cycle involves rocks, water and plants shifting phosphorus around over several millenia. On a global scale, rock phosphorus is mined for fertilizer and the majority of it ends up being eventually washed off to rivers and oceans with no easy way to recover it. Wastewater contributes a small but significant load to the global phosphorus footprint and therefore opens the opportunity for us to tap that resource. (I've written more about this elsewhere.)

Give me an E

Woodman Point Digesters

Energy generation is getting a lot of attention in the North American wastewater sector right now. It has been common practice worldwide for many years to use anaerobic digesters to generate biogas to heat the digesters and in some instances to use that biogas in generators to produce electricity. In North America and elsewhere the economics haven't always worked out to enable the the biogas to be used this way (darned siloxanes from beauty products making things considerably worse), but an increased focus on energy and new technologies are moving more utilities to consider anaerobic digestion. WERF recently completed a study on the barriers to biogas use that showed that there are many misconceptions about digesters and biogas and that it's much more attractive than many people thought.

Give me a W

Singapore PUB NEWater Visitors Centre

So, with all this focus on Nutrients and Energy it's easy to forget that the single biggest resource that can be recovered from wastewater is the obvious one... Water. Water reclamatation and reuse is well-established throughout the world, especially in those regions where water is scarce and there is no alternative to re-using water as effectively as possible. My favorite example is the NEWater scheme in Singapore (and not just because I end up with "NEW" as the "W" in my "N-E-W" acronym - everything is New!!!). There they are serious about catching as much water as possible and re-using what they term "used water" to provide up to 30% of their water requirements now and plans to provide more than half of their water needs in the future. It's pretty cool.

These are exciting and interesting times in the wastewater field. The move to recycle and reuse materials to improve the sustainability are self-evident to most of us in the industry. The main hurdle, as with most things, is the financing. Adding technology and infrastructure to recover nutrients, energy and water costs money and if it's cheaper to just let all of the resources to go untapped into rivers, oceans and landfills then that's the default we'll fall back to.

Posted with Blogsy

Nitrogen Removal 3.0

There's a buzz in the wastewater community right now. It's not often someone discovers a whole new mechanism for treatment and it is causing quite a stir. Not since Dr James Barnard formalized biological phosphorus removal in the 1970s has there been so much excitement. The reason? Anaerobic ammonia oxidation or "Anammox". On paper, this process removes nitrogen from wastewater using less than half the energy of traditional nitrification/denitrification, no carbon addition and little sludge production. In the early part of this century several process configurations and patents have been issued that use anammox to remove nitrogen from high strength wastes, including ANAMMOX by Paques, DEMON by Cyklar-Stulz and ANITA Mox by AnoxKaldnes. These processes are proving to be robust and very successful for high ammonia concentration wastes such as the "sidestreams" from sludge treatment at domestic wastewater treatment plants. The trick now is to take these ideas and make them work on the "mainstream" where the wastewater is much weaker and the temperatures lower than sidestreams. WERF is coordinating research projects to this very end. Could 2013 be the year when we see anammox go mainstream? I'm hopeful that it is!

 

Posted with Blogsy