FlowBot: Going beyond existing best practice to deliver measurable benefit
Spreadsheets used to manage repetitive data analysis tasks takes time – too much time. This inefficiency results in less time available for engineers to work with their clients to support informed, sustainable decision making on their projects.
18 Mar 2020
A common industry challenge
When managing flow surveys for wastewater modelling projects there are a variety of ways that data can be assessed to deliver understanding of the quality of data; monitoring equipment performance; and the recorded size of rainfall events. These assessments mainly comprise the copying of the recorded flow, depth and velocity into multiple spreadsheets, of varying quality, which are then used to produce many graphs and data quality reports.
RPS was about to embark on a project that would require the need to install and assess 3,568 monitors and 801 rain gauges across 143 catchments as part of the Anglian Water Integrated Urban Drainage Framework. Needless to say, we were highly motivated to find a more efficient, digital solution to this data challenge – to save time and our sanity! FlowBot was the result.
So, what is FlowBot? FlowBot is an RPS software solution that we created using Python programming language that optimises data visualisation, analysis and machine learning methods. It provides the user complete control over a range of assessments designed to help us understand monitoring equipment performance in a user-friendly environment - and delivers 80% time savings.
Commitment to delivery
Initially, a proof of concept was developed to demonstrate how a range of statistical assessments and plotting of recorded data could be completed using a more efficient, single application. Following an successful demonstration, funding was granted for the next stage of the applications development. Ultimately this would be a resource that RPS could use to assess flow survey data for the benefit of all our water sector clients.
The team collated a range of assessments and translated the statistical processes into the Python programming language. Regular product testing was conducted throughout the development process, the feedback provided was invaluable in ensuring the end product would meet the output objective.
Going beyond existing best practice
A sizeable flow survey can generate over 10,000 cells of data, which in turn creates a 100+ graphs that must be reviewed – previously a very manual task that could take an engineer weeks to complete. It’s also an extremely repetitive activity which increases the risk of error.
One of FlowBot’s features was designed to allow a machine learning algorithm to complete this tiresome and time-consuming process. We used a random forest decision tree classifier trained with 30,000 days of human classifications extracted from data mined from previous RPS flow surveys. Once trained, the algorithm completed a process that previous might have taken weeks in just minutes, and with a high level of accuracy.
For the first time, one application i.e. FlowBot, was available to deliver a wide range of statistical assessments needed to effectively and quickly assess flow survey data in a standardised way – reducing time and money, and the sanity of engineers.
Measurable benefit
A measurable benefit of the FlowBot application is time saving, and lots of it. To quantify this, we made a comparison using the manual assessment of a 7 monitor, 5 rain gauge, flow survey conducted over a 9-week period. The result was a time saving of 86% using FlowBot.
Additionally, a range of assessment that were previously housed in multiple spreadsheets and applications were now available in one single package – making life much easier for the users completing daily assessments by removing the need for the individual to perform the task of categorising recorded data quality.
An ongoing benefit that reaches beyond this specific application is that we’ve been able to demonstrate how machine learning algorithms can be deployed to reduce manual data assessment. This has given RPS a springboard to automate other time-consuming data analysis tasks.
Paper
Related services
Flooding
Find out more »
Leakage
Find out more »
Pollution
Find out more »
Innovation
Find out more »