The vulnerability is an uncontrolled resource consumption issue in the Bouncy Castle FIPS Java API (BC-FJA). The root cause is the behavior of the NativeLoader class, which is responsible for loading native libraries required for FIPS-certified cryptographic operations. In the vulnerable version (2.1.0), the loadLibrary method in this class would create a new temporary directory and extract the native library from the JAR file every time it was called. In an environment with multiple JVMs using the Bouncy Castle FIPS provider, this would lead to the creation of a large number of directories, each containing a copy of the native library. This behavior could exhaust disk space or inode limits on the server, leading to a denial of service.
The fix, introduced in version 2.1.1, modifies the loadLibrary method to prevent this excessive resource allocation. The patched version now checks for the existence of a pre-extracted library in a shared location. If the library is found and its checksum matches the one in the JAR, it is reused. Otherwise, the library is extracted once and a checksum file is created to be used for future checks. This ensures that the native library is extracted only once per version, regardless of how many times the loadLibrary method is called or how many JVMs are running.
The vulnerable function is org.bouncycastle.crypto.fips.NativeLoader.loadLibrary. A runtime profiler would show this function being called during the initialization of the Bouncy Castle FIPS provider. In a vulnerable environment, this function would be called frequently, and its execution would involve file system operations to create directories and write files, which would be the primary indicators of the vulnerability being triggered.