The vulnerability was a critical Remote Code Execution (RCE) due to a failure to properly sandbox user-submitted code. The analysis of the patch reveals that the application had multiple code execution endpoints. The most evident vulnerability was in backend/routes/compiler.py, where the execute_python function would write arbitrary code from a user to a file and then execute it using subprocess.run. This provides a direct and unrestricted way to execute commands on the host system. Another endpoint in backend/routes/coding.py also used an insecure method for code execution. The vulnerability was not just a simple command injection but a fundamental flaw in the sandboxing mechanism for the code execution environment.
The fix involved a major refactoring of the code execution logic. The vulnerable functions that used subprocess.run directly were removed. A new centralized service, RealCompilerService, was introduced (and hardened) to handle all code execution requests. This new service uses Docker containers to create a strongly isolated sandbox for each execution, with defined timeouts, memory limits, and CPU limits. This ensures that any user-submitted code, even if malicious, is contained within the Docker environment and cannot affect the host system, effectively mitigating the RCE vulnerability.