Deep Deterministic Policy Gradient for Simulation-Based Control of Water Quality in Nano Aquatic Systems

Abstract

This study proposes a simulation-based Deep Deterministic Policy Gradient (DDPG) framework for water-quality control in nano aquatic systems. Nano tanks are highly sensitive to small disturbances because their limited water volume reduces buffering capacity and causes rapid changes in dissolved oxygen, ammonia, pH, temperature, biological oxygen demand, and chemical oxygen demand. To represent these coupled dynamics, a nonlinear simulation model adapted from the Continuously Stirred Tank Reactor concept is developed and implemented as an OpenAI Gym-compatible reinforcement learning environment. The DDPG agent learns continuous control actions related to aeration, feeding, and filtration through repeated interaction with the simulated nano-tank environment. The proposed nonlinear CSTR-DDPG framework is evaluated against a linear-model DDPG baseline using RMSE, cumulative reward, and closed-loop control performance. Simulation results show that the nonlinear model reduced RMSE by 45.2% for dissolved oxygen, 64.0% for ammonia, and 61.3% for pH compared with the linear baseline. The DDPG agent also achieved a 41.7% higher cumulative reward under the same reward structure. These findings indicate that nonlinear simulation can provide a more informative training environment for DDPG-based water-quality control. However, the present study remains limited to simulation-based evaluation, and physical validation using calibrated sensors, actuators, communication-delay analysis, and real nano-tank experiments is required in future work.