Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Fix response body (based on #84) #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
airween merged 1 commit into owasp-modsecurity:master from g00g1:fix_response_body_v1.0.3
Jan 10, 2025

Conversation

Copy link
Contributor

@g00g1 g00g1 commented Jun 18, 2024

This PR is an attempt to revive previously developed fix for #41. It seems like its' developer abandoned PR, therefore I would like to try my best to finish it.

I have updated the code base to a current master, therefore it is possible to continue discussion regarding this patch.

Copy link

Is there any update on when this PR will be reviewed/merged? This solution also solves another issue, wherein if the response size is large, even if the blocking rule is triggered, the response still goes through to the client.

Copy link
Member

airween commented Oct 31, 2024

Hi @g00g1, @Dr-Lazarus-V2,

although all tests were passed, it would be nice to explain the exact problem. @Dr-Lazarus-V2 mentioned this PR in his issue, but unfortunately I wasn't able to reproduce the problem - and then I can't try the fix.

Also this comment mentions that the requested feature is not available yet - this PR adds it?

Copy link
Contributor Author

g00g1 commented Oct 31, 2024

@airween how did you try to reproduce it? Have you seen my comment here: #41 (comment) ?

Copy link

Hey @airween, yes I believe this PR resolves the issue. As the response body is fully examined before the response is sent out.

Copy link
Member

airween commented Nov 5, 2024

@g00g1: I saw your comment, but I'm afraid it's not enough to reproduce the issue. Could you share some more details, eg. (relevant) configuration, and a curl request?

@Dr-Lazarus-V2: I'm sure the PR solves the problem, but first I would like to see the problem itself 😃

Copy link
Member

airween commented Nov 7, 2024

Hi guys,

finally I had some time to reproduce the issue. Coreruleset issue #3277 helped me.

I tried your patch, seems like it solves the problem - but unfortunately the audit log messages key is empty, even though the triggered rule is in the error.log. The other parts are there in audit.log: transaction, response and producer. The response shows the status was 403.

Could you take a look this issue?

Copy link
Contributor Author

g00g1 commented Nov 7, 2024 •
edited
Loading

I am still not very familiar with modsecurity code, it might take some time before I figure out how this should work properly and what is lacking to fill messages key currently, unfortunately I couldn't figure out the reason fast.

If anyone wished to help or guide me regarding this, I will be very grateful.

Am I right that this pull request somehow impacts the codepath taken at https://github.com/owasp-modsecurity/ModSecurity/blob/4a720004ddf37963ad63deb1f1a2fdd8113f7d41/src/rule_with_actions.cc#L495 ?

Copy link
Member

airween commented Nov 7, 2024

Sure, I'm happy to help you. You can join to OWASP's Slack server, and choose #project-modsecurity channel. You can reach me there, feel free to ask.
g00g1 and Dr-Lazarus-V2 reacted with thumbs up emoji

Copy link
Contributor Author

g00g1 commented Nov 7, 2024

@airween thanks, but I don't have an account there nor email at owasp.org domain.

Copy link
Member

airween commented Nov 7, 2024

@airween thanks, but I don't have an account there nor email at owasp.org domain.

As I know you don't need any owasp.org email address. But if you send me your email me in private, I can send you an invitation.

Copy link
Member

airween commented Nov 7, 2024

Or you can use this link.

Copy link

Hey, if it's all right, I can also have a look at the PR and try to debug what the issue might be

Copy link

Dr-Lazarus-V2 commented Nov 12, 2024 •
edited
Loading

I have noticed a weird behaviour:

[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "POST /api/v1/*/* HTTP/1.1" (Variable: REQUEST_LINE)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920160) Executing operator "Rx" with param "^\d+$" against REQUEST_HEADERS:Content-Length.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "150" (Variable: REQUEST_HEADERS:Content-Length)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920170) Executing operator "Rx" with param "^(?:GET|HEAD)$" against REQUEST_METHOD.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "POST" (Variable: REQUEST_METHOD)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920171) Executing operator "Rx" with param "^(?:GET|HEAD)$" against REQUEST_METHOD.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "POST" (Variable: REQUEST_METHOD)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920180) Executing operator "Within" with param "HTTP/2 HTTP/2.0 HTTP/3 HTTP/3.0" against REQUEST_PROTOCOL.

[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "GET /api/v1/*/* HTTP/1.1" (Variable: REQUEST_LINE)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920160) Executing operator "Rx" with param "^\d+$" against REQUEST_HEADERS:Content-Length.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "150" (Variable: REQUEST_HEADERS:Content-Length)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 0.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars cleaned.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 920170) Executing operator "Rx" with param "^(?:GET|HEAD)$" against REQUEST_METHOD.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Target value: "GET" (Variable: REQUEST_METHOD)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [9] Matched vars updated.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Rule returned 1.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] Executing chained rule.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/*/*] [4] (Rule: 0) Executing operator "Rx" with param "^0?$" against

Within a single transaction (identified by a unique ID), the same rule is triggered twice, but with inconsistent results: once with the HTTP method identified as POST and then as GET. This means that during the same request, ModSecurity is evaluating the method as both POST and GET at different points. This unusual behavior may indicate an issue with how ModSecurity is processing the request, possibly due to a configuration quirk or rule interpretation that causes the request method to appear inconsistent during evaluation.

Copy link

Dr-Lazarus-V2 commented Nov 12, 2024 •
edited
Loading

As an addition to the previously mentioned setup, I created the following custom rule to block responses containing the word "leak":

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
 "id:187, \
 phase:4,\
 deny,\
 status:403,\
 log,\
 t:none,\
 msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

However, at Debug Level 9, when this rule is triggered and blocks the response, the transaction appears to restart unexpectedly. Below are the logs capturing this issue:

[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] (Rule: 187) Executing operator "Rx" with param "(?i)(\n|\''|\:|\W*)leak" against RESPONSE_BODY.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Target value: "{"id":"chatcmpl-fd274e7b","created":1731393695,"model":"meta.llama3-8b-instruct- (792 characters omitted)" (Variable: RESPONSE_BODY)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Matched vars updated.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] Rule returned 1.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Saving msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Running action: status
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Running action: log
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [9] Saving transaction to logs
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] Running (disruptive) action: deny.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [8] Running action deny
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [8] Skipping this phase as this request was already intercepted.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Initializing transaction
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Transaction context created.
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Starting phase CONNECTION. (SecRules 0)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [9] This phase consists of 29 rule(s).
[ee71a9fe8ef2c5bed4b88784fe1ef990] [] [4] Starting phase URI. (SecRules 0 + 1/2)
[ee71a9fe8ef2c5bed4b88784fe1ef990] [/api/v1/chat/completions] [4] Starting phase REQUEST_HEADERS. (SecRules 1)

Copy link
Member

airween commented Nov 13, 2024

Hey, if it's all right, I can also have a look at the PR and try to debug what the issue might be

Hi @Dr-Lazarus-V2, this would be awesome! Thanks!

Copy link
Member

airween commented Nov 13, 2024

Within a single transaction (identified by a unique ID), the same rule is triggered twice, but with inconsistent results: once with the HTTP method identified as POST and then as GET.

Could you show your request with that I can reproduce this? Also please provide the used CRS version too.

Copy link

Hey @airween, the CRS Version I used is v4.7.0. In the application I am trying to protect, I had an endpoint which returns a JSON response with the word leak. I had used this custom rule to detect it:

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
 "id:187, \
 phase:4,\
 deny,\
 status:403,\
 log,\
 t:none,\
 msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

If you'd like to reproduce this, I think you would need a simple POST endpoint in the Web Application which is protected by the WAF. The POST endpoint needs to return a JSON response with the word "leak". If you perform a curl to this endpoint with this custom rule present in your application. You will notice that the transaction terminates and restarts when the custom rule is triggered (evaluates to true and implements action deny).

Here is my modsecurity configuration for reference:

SecRuleEngine On
SecRequestBodyAccess on
SecRule REQUEST_HEADERS:Content-Type "^(?:application(?:/soap\+|/)|text/)xml" \
 "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type "^application/json" \
 "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
SecRequestBodyLimit 10485760
SecRequestBodyNoFilesLimit 1048576
SecRequestBodyLimitAction ProcessPartial
SecRequestBodyJsonDepthLimit 512
SecArgumentsLimit 1000
SecRule &ARGS "@ge 1000" \
"id:'200007', phase:2,t:none,log,deny,status:400,msg:'Failed to fully parse request body due to large argument count',severity:2"
SecRule REQBODY_ERROR "!@eq 0" \
"id:'200002', phase:2,t:none,log,deny,status:400,msg:'Failed to parse request body.',logdata:'%{reqbody_error_msg}',severity:2"
SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:400, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"
SecPcreMatchLimit 100000
SecPcreMatchLimitRecursion 100000
SecRule TX:/^MSC_/ "!@streq 0" \
 "id:'200005',phase:2,t:none,deny,msg:'ModSecurity internal error flagged: %{MATCHED_VAR_NAME}'"
SecResponseBodyAccess On
SecResponseBodyMimeType text/plain text/html text/xml application/json
SecResponseBodyLimit 537600
SecResponseBodyLimitAction Reject
SecTmpDir /tmp/
SecDataDir /tmp/
SecAuditEngine On
SecAuditLogType Serial
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLog /var/log/modsec_logs/modsec_audit_waf.theviscousweb.com.log
SecAuditLogFormat JSON
SecAuditLogParts ABFHZ
SecArgumentSeparator &
SecCookieFormat 0
SecUnicodeMapFile unicode.mapping 20127
SecStatusEngine Off
SecGeoLookupDb /var/GeoLite2-Country/GeoLite2-Country.mmdb
SecDebugLog /var/log/modsec_logs/modsec_debug.log
SecDebugLogLevel 9

Copy link

@airween, I have tested and realized the following:

I have setup an endpoint: "/healthz" which returns "leak!"

Bug Case:

 location /healthz {
 add_header Content-Type text/plain;
 return 200 'leak!';
 }

Now if I perform the following curl command:

 curl --location --request POST 'http://waf.theviscousweb.com/healthz' \
--data ''

It seems to return the following log:

{
 "transaction": {
 "client_ip": "10.103.253.191",
 "time_stamp": "Thu Nov 14 03:29:54 2024",
 "server_id": "88a6b3d6a5b5bc6b73061e112bad4a1309e1d4b2",
 "client_port": 36616,
 "host_ip": "192.168.32.3",
 "host_port": 8080,
 "unique_id": "bdcd2324b71e0518896ea843b61e488b",
 "request": {
 "method": "GET",
 "http_version": 1.1,
 "uri": "/healthz",
 "headers": {
 "Host": "waf.theviscousweb.com",
 "User-Agent": "curl/7.81.0",
 "Accept": "*/*",
 "Content-Length": "0",
 "Content-Type": "application/x-www-form-urlencoded"
 }
 },
 "response": {
 "http_code": 403,
 "headers": {
 "Server": "nginx/1.27.2",
 "Date": "2024年11月14日 03:29:54 GMT",
 "Content-Type": "text/html",
 "Connection": "keep-alive",
 "ETag": "\"66dfdff2-93d6\""
 }
 },
 "producer": {
 "modsecurity": "ModSecurity v3.0.13 (Linux)",
 "connector": "ModSecurity-nginx v1.0.3",
 "secrules_engine": "Enabled",
 "components": [
 "OWASP_CRS/4.7.0\""
 ]
 },
 "messages": [
 {
 "message": "GET Request Detected",
 "details": {
 "match": "Matched \"Operator `StrEq' with parameter `GET' against variable `REQUEST_METHOD' (Value: `GET' )",
 "reference": "v0,3",
 "ruleId": "191",
 "file": "/var/opt/modsecurity.d/custom-ruleset/Data-Leak/Data-Leak-RULES.conf",
 "lineNumber": "18",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "Inbound Score: 0 Outbound Score: 0",
 "details": {
 "match": "",
 "reference": "",
 "ruleId": "980145",
 "file": "/var/opt/modsecurity.d/owasp-crs/waf.theviscousweb.com.crs_setup.conf",
 "lineNumber": "40",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 }
 ]
 }
}

You can see that even though the curl command is POST, in the logs method is GET. Now some further pointers, this only occurs if the following custom rule is triggered.

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
 "id:187, \
 phase:4,\
 deny,\
 status:403,\
 log,\
 t:none,\
 msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

Bug Free Case:

If I modify the /healthz endpoint to return something else:

 location /healthz {
 add_header Content-Type text/plain;
 return 200 'test!';
 }

It does not cause any issues and the request is logged as a POST.

 {
 "transaction": {
 "client_ip": "10.103.253.191",
 "time_stamp": "Thu Nov 14 03:36:23 2024",
 "server_id": "88a6b3d6a5b5bc6b73061e112bad4a1309e1d4b2",
 "client_port": 43622,
 "host_ip": "192.168.32.3",
 "host_port": 8080,
 "unique_id": "13fdd4d10d426b9e653cc22a833bd3df",
 "request": {
 "method": "POST",
 "http_version": 1.1,
 "uri": "/healthz",
 "headers": {
 "Host": "waf.theviscousweb.com",
 "User-Agent": "curl/7.81.0",
 "Accept": "*/*",
 "Content-Length": "0",
 "Content-Type": "application/x-www-form-urlencoded"
 }
 },
 "response": {
 "http_code": 200,
 "headers": {
 "Server": "nginx/1.27.2",
 "Date": "2024年11月14日 03:36:23 GMT",
 "Content-Length": "5",
 "Content-Type": "application/octet-stream",
 "Content-Type": "text/plain",
 "Connection": "keep-alive"
 }
 },
 "producer": {
 "modsecurity": "ModSecurity v3.0.13 (Linux)",
 "connector": "ModSecurity-nginx v1.0.3",
 "secrules_engine": "Enabled",
 "components": [
 "OWASP_CRS/4.7.0\""
 ]
 },
 "messages": [
 {
 "message": "Inbound Score: 0 Outbound Score: 0",
 "details": {
 "match": "",
 "reference": "",
 "ruleId": "980145",
 "file": "/var/opt/modsecurity.d/owasp-crs/waf.theviscousweb.com.crs_setup.conf",
 "lineNumber": "40",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 }
 ]
 }
}

Hence I see that when a rule is triggered in the response body, somehow the REQUEST_METHOD changes from POST to GET.

Copy link

Dr-Lazarus-V2 commented Nov 14, 2024 •
edited
Loading

Hello, everyone. Apologies for the confusion regarding the recent PR. I want to clarify that the bug I observed – where malicious POST requests are being logged as GET requests – appears to be a redirection issue, which affects the logging. I had a redirect in my nginx code. This specific issue is not introduced by the recent code changes, as I found it to be present in the current active version of the ModSecurity-Nginx connector. This bug has already been raised and is still open in: #182.
@airween , I attempted to reproduce the core issue concerning the unavailability of messages in the audit logs, and I was not able to reproduce it. I can successfully view the messages in the audit logs.

Here is my audit logs under:

{
 "transaction": {
 "client_ip": "10.103.253.191",
 "time_stamp": "Thu Nov 14 06:18:37 2024",
 "server_id": "135e057d99ba46945f45e685b2bae7652ad9cc73",
 "client_port": 54524,
 "host_ip": "192.168.32.3",
 "host_port": 8080,
 "unique_id": "349dd1ee0e6e742bfc604e2732173015",
 "request": {
 "method": "POST",
 "http_version": 1.1,
 "uri": "/",
 "headers": {
 "Host": "waf.theviscousweb.com",
 "User-Agent": "curl/7.81.0",
 "Accept": "*/*",
 "Content-Type": "application/x-www-form-urlencoded",
 "Content-Length": "89"
 }
 },
 "response": {
 "http_code": 403,
 "headers": {
 "Server": "nginx/1.27.2",
 "Date": "2024年11月14日 06:18:37 GMT",
 "Content-Length": "153",
 "Content-Type": "text/html",
 "Connection": "keep-alive"
 }
 },
 "producer": {
 "modsecurity": "ModSecurity v3.0.13 (Linux)",
 "connector": "ModSecurity-nginx v1.0.3",
 "secrules_engine": "Enabled",
 "components": [
 "OWASP_CRS/4.7.0\""
 ]
 },
 "messages": [
 {
 "message": "OS File Access Attempt",
 "details": {
 "match": "Matched \"Operator `PmFromFile' with parameter `lfi-os-files.data' against variable `ARGS:cmd' (Value: `;cat /etc/passwd;' )",
 "reference": "o6,10v220,17t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin",
 "ruleId": "930120",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf",
 "lineNumber": "97",
 "data": "Matched Data: etc/passwd found within ARGS:cmd: ;cat /etc/passwd;",
 "severity": "2",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "application-multi",
 "language-multi",
 "platform-multi",
 "attack-lfi",
 "paranoia-level/1",
 "OWASP_CRS",
 "capec/1000/255/153/126",
 "PCI/6.5.4"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "Remote Command Execution: Unix Shell Code Found",
 "details": {
 "match": "Matched \"Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:cmd' (Value: `;cat /etc/passwd;' )",
 "reference": "o5,10v220,17t:cmdLine,t:normalizePath",
 "ruleId": "932160",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf",
 "lineNumber": "596",
 "data": "Matched Data: etc/passwd found within ARGS:cmd: cat/etc/passwd ",
 "severity": "2",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "application-multi",
 "language-shell",
 "platform-unix",
 "attack-rce",
 "paranoia-level/1",
 "OWASP_CRS",
 "capec/1000/152/248/88",
 "PCI/6.5.2"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "XSS Attack Detected via libinjection",
 "details": {
 "match": "detected XSS using libinjection.",
 "reference": "v186,29t:utf8toUnicode,t:urlDecodeUni,t:htmlEntityDecode,t:jsDecode,t:cssDecode,t:removeNulls",
 "ruleId": "941100",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
 "lineNumber": "82",
 "data": "Matched Data: XSS data found within ARGS:password: <script>alert('XSS')</script>",
 "severity": "2",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "application-multi",
 "language-multi",
 "platform-multi",
 "attack-xss",
 "xss-perf-disable",
 "paranoia-level/1",
 "OWASP_CRS",
 "capec/1000/152/242"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "XSS Filter - Category 1: Script Tag Vector",
 "details": {
 "match": "Matched \"Operator `Rx' with parameter `(?i)<script[^>]*>[\\s\\S]*?' against variable `ARGS:password' (Value: `<script>alert('XSS')</script>' )",
 "reference": "o0,8v186,29t:utf8toUnicode,t:urlDecodeUni,t:htmlEntityDecode,t:jsDecode,t:cssDecode,t:removeNulls",
 "ruleId": "941110",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
 "lineNumber": "108",
 "data": "Matched Data: <script> found within ARGS:password: <script>alert('XSS')</script>",
 "severity": "2",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "application-multi",
 "language-multi",
 "platform-multi",
 "attack-xss",
 "xss-perf-disable",
 "paranoia-level/1",
 "OWASP_CRS",
 "capec/1000/152/242"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "NoScript XSS InjectionChecker: HTML Injection",
 "details": {
 "match": "Matched \"Operator `Rx' with parameter `(?i)<[^0-9<>A-Z_a-z]*(?:[^\\s\\x0b\\\"'<>]*:)?[^0-9<>A-Z_a-z]*[^0-9A-Z_a-z]*?(?:s[^0-9A-Z_a-z]*?(?:c[^0-9A-Z_a-z]*?r[^0-9A-Z_a-z]*?i[^0-9A-Z_a-z]*?p[^0-9A-Z_a-z]*?t|t[^0-9A-Z_a-z]*?y[^0-9A-Z_a-z]*?l[^0-9A (4378 characters omitted)' against variable `ARGS:password' (Value: `<script>alert('XSS')</script>' )",
 "reference": "o0,7v186,29t:utf8toUnicode,t:urlDecodeUni,t:htmlEntityDecode,t:jsDecode,t:cssDecode,t:removeNulls",
 "ruleId": "941160",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
 "lineNumber": "200",
 "data": "Matched Data: <script found within ARGS:password: <script>alert('XSS')</script>",
 "severity": "2",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "application-multi",
 "language-multi",
 "platform-multi",
 "attack-xss",
 "xss-perf-disable",
 "paranoia-level/1",
 "OWASP_CRS",
 "capec/1000/152/242"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "Javascript method detected",
 "details": {
 "match": "Matched \"Operator `Rx' with parameter `(?i)\\b(?:eval|set(?:timeout|interval)|new[\\s\\x0b]+Function|a(?:lert|tob)|btoa|prompt|confirm)[\\s\\x0b]*\\(' against variable `ARGS:password' (Value: `<script>alert('XSS')</script>' )",
 "reference": "o8,6v186,29t:htmlEntityDecode,t:jsDecode",
 "ruleId": "941390",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf",
 "lineNumber": "713",
 "data": "Matched Data: alert( found within ARGS:password: <script>alert('XSS')</script>",
 "severity": "2",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "application-multi",
 "language-multi",
 "attack-xss",
 "xss-perf-disable",
 "paranoia-level/1",
 "OWASP_CRS",
 "capec/1000/152/242"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "SQL Injection Attack Detected via libinjection",
 "details": {
 "match": "detected SQLi using libinjection.",
 "reference": "v157,19",
 "ruleId": "942100",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf",
 "lineNumber": "46",
 "data": "Matched Data: s&sos found within ARGS:username: admin' OR '1'='1;--",
 "severity": "2",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "application-multi",
 "language-multi",
 "platform-multi",
 "attack-sqli",
 "paranoia-level/1",
 "OWASP_CRS",
 "capec/1000/152/248/66",
 "PCI/6.5.2"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "Inbound Anomaly Score Exceeded (Total Score: 35)",
 "details": {
 "match": "Matched \"Operator `Ge' with parameter `20' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `35' )",
 "reference": "",
 "ruleId": "949110",
 "file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf",
 "lineNumber": "222",
 "data": "",
 "severity": "0",
 "ver": "OWASP_CRS/4.7.0",
 "rev": "",
 "tags": [
 "anomaly-evaluation",
 "OWASP_CRS"
 ],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "Inbound Score: 35 Outbound Score: 0",
 "details": {
 "match": "",
 "reference": "",
 "ruleId": "980145",
 "file": "/var/opt/modsecurity.d/owasp-crs/waf.theviscousweb.com.crs_setup.conf",
 "lineNumber": "40",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 }
 ]
 }
}

Here is the curl command:

curl -X POST http://waf.theviscousweb.com \
 -H "Content-Type: application/x-www-form-urlencoded" \
 -d "username=admin' OR '1'='1;--&password=<script>alert('XSS')</script>&cmd=;cat **/etc/passwd;"

Copy link

Dr-Lazarus-V2 commented Nov 14, 2024 •
edited
Loading

@airween, I have also tested it with the custom rule mentioned in: #3277. I was not able to reproduce the bug raised earlier.

Custom Rule:

SecRule RESPONSE_BODY "@rx ^}" "id:911,phase:4, deny, status:403, capture,log, t:none, msg:'multiline active'"

I can see the message key & value in the audit logs

{
 "transaction": {
 "client_ip": "10.103.253.240",
 "time_stamp": "Thu Nov 14 06:33:40 2024",
 "server_id": "135e057d99ba46945f45e685b2bae7652ad9cc73",
 "client_port": 53601,
 "host_ip": "192.168.32.3",
 "host_port": 8080,
 "unique_id": "bdcc99b802f390fe8f0f0d7119e3a0d5",
 "request": {
 "method": "POST",
 "http_version": 1.1,
 "uri": "/check",
 "headers": {
 "Content-Type": "application/json",
 "User-Agent": "PostmanRuntime/7.42.0",
 "Postman-Token": "90021de8-7a7e-4a2d-a78b-1339ea7a1061",
 "Accept": "*/*",
 "Host": "weborion.licensing.portal",
 "Accept-Encoding": "gzip, deflate, br",
 "Connection": "keep-alive",
 "Content-Length": "21"
 }
 },
 "response": {
 "http_code": 403,
 "headers": {
 "Server": "nginx/1.27.2",
 "Date": "2024年11月14日 06:33:40 GMT",
 "Content-Length": "11",
 "Content-Type": "text/plain; charset=utf-8",
 "Connection": "keep-alive"
 }
 },
 "producer": {
 "modsecurity": "ModSecurity v3.0.13 (Linux)",
 "connector": "ModSecurity-nginx v1.0.3",
 "secrules_engine": "Enabled",
 "components": [
 "OWASP_CRS/4.7.0\""
 ]
 },
 "messages": [
 {
 "message": "multiline active",
 "details": {
 "match": "Matched \"Operator `Rx' with parameter `^}' against variable `RESPONSE_BODY' (Value: `} triggered' )",
 "reference": "o0,1v259,11",
 "ruleId": "189",
 "file": "/var/opt/modsecurity.d/custom-ruleset/TestRukeset/TestRukeset-RULES.conf",
 "lineNumber": "2",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "Inbound Score: 0 Outbound Score: 0",
 "details": {
 "match": "",
 "reference": "",
 "ruleId": "980145",
 "file": "/var/opt/modsecurity.d/owasp-crs/weborion.licensing.portal.crs_setup.conf",
 "lineNumber": "40",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 }
 ]
 }
}

Copy link

Outside of that, I do not see any critical bugs in this PR and it looks stable to me.

Copy link
Member

airween commented Nov 14, 2024

Hi @Dr-Lazarus-V2,

thanks for the very detailed explanation.

I tried - I think - the same config as you have:

  • I replaced my modsecurity.conf with yours, changed only the logs' paths
  • I use CRS 4.9-dev, but added your rule with id:187 into EXCLUSION-RULES-BEFORE-CRS.conf
  • created a new server with hostname waf.test
  • used the same curl request that you gave here

But I still get empty messages key in the audit.log:

{
 "transaction": {
 "client_ip": "127.0.0.1",
 "time_stamp": "Thu Nov 14 16:05:58 2024",
 "server_id": "0d4d434e96996020298a3e094b066f66e6634d3b",
 "client_port": 53650,
 "host_ip": "127.0.0.1",
 "host_port": 8080,
 "unique_id": "173159675881.617821",
 "request": {
 "method": "GET",
 "http_version": 1.1,
 "uri": "/healthz",
 "headers": {
 "Host": "waf.test:8080",
 "User-Agent": "curl/8.10.1",
 "Accept": "*/*",
 "Content-Length": "0",
 "Content-Type": "application/x-www-form-urlencoded"
 }
 },
 "response": {
 "http_code": 403,
 "headers": {
 "Server": "nginx/1.26.0",
 "Date": "2024年11月14日 15:05:58 GMT",
 "Content-Length": "101",
 "Content-Type": "text/html",
 "Last-Modified": "2024年10月17日 11:36:28 GMT",
 "Connection": "keep-alive",
 "ETag": "\"6710f6bc-65\""
 }
 },
 "producer": {
 "modsecurity": "ModSecurity v3.0.13 (Linux)",
 "connector": "ModSecurity-nginx v1.0.3",
 "secrules_engine": "Enabled",
 "components": [
 "OWASP_CRS/4.9.0-dev\""
 ]
 },
 "messages": []
 }
}

In the error.log I get this:

2024年11月14日 16:05:58 [error] 43687#43687: *1 [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Rx' with parameter `(?i)(\n|\''|\:|\W*)leak' against variable `RESPONSE_BODY' (Value: `leak!' ) [file "/home/airween/src/coreruleset/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf"] [line "205"] [id "187"] [rev ""] [msg " 'UNAUTHORIZED RESPONSE DATA ACCESS'"] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "127.0.0.1"] [uri "/healthz"] [unique_id "173159675897.213267"] [ref "o0,4o0,0v145,5"], client: 127.0.0.1, server: waf.test, request: "POST /healthz HTTP/1.1", host: "waf.test:8080"

I continue the investigation.

Copy link
Member

airween commented Nov 14, 2024

@airween , I attempted to reproduce the core issue concerning the unavailability of messages in the audit logs, and I was not able to reproduce it. I can successfully view the messages in the audit logs.

With the given curl request I also see the messages in audit.log - but that's okay, probably because the intervention happens in phase:2, not in phase:4. So it's okay.

The question is to me why can't I see the messages in audit.log if the intervention happens in phase:4 (with this patch, of course).

Copy link

That's interesting. In my system, I use the OWASP Modsecurity-CRS Docker Image and I modify the nginx connector in the Dockerfile. I change the following lines in this file nginx/Dockerfile:

Old Line
RUN set -eux; \ git clone -b master --depth 1 https://github.com/owasp-modsecurity/ModSecurity-nginx.git; \

New Line
RUN set -eux; \ git clone -b fix_response_body_v1.0.3 --depth 1 https://github.com/g00g1/ModSecurity-nginx.git; \

I've set up this configuration as my WAF, and based on your audit logs, it’s evident that our configurations are largely similar, except for the Nginx server version. I'm running version 1.27.2 on my end, which I don’t believe should cause any issues.

I have used the same rule:

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)leak" \
 "id:10, \
 phase:4,\
 deny,\
 status:403,\
 log,\
 t:none,\
 msg: 'UNAUTHORIZED RESPONSE DATA ACCESS'"

Here is the audit log:

{
 "transaction": {
 "client_ip": "10.103.253.240",
 "time_stamp": "Fri Nov 15 01:43:06 2024",
 "server_id": "6eaa3f3dd6bdf64b45eb7031a0aeddb38c480a84",
 "client_port": 62070,
 "host_ip": "192.168.32.3",
 "host_port": 8080,
 "unique_id": "000f40896fa2781e7e9aba6f05b81d09",
 "request": {
 "method": "GET",
 "http_version": 1.1,
 "uri": "/check",
 "headers": {
 "User-Agent": "PostmanRuntime/7.42.0",
 "Postman-Token": "2a09f719-9cf5-48f7-ba1c-52e5e1e89572",
 "Accept": "*/*",
 "Host": "weborion.licensing.portal",
 "Accept-Encoding": "gzip, deflate, br",
 "Connection": "keep-alive"
 }
 },
 "response": {
 "http_code": 403,
 "headers": {
 "Server": "nginx/1.27.2",
 "Date": "Fri, 15 Nov 2024 01:43:06 GMT",
 "Content-Length": "4",
 "Content-Type": "text/plain; charset=utf-8",
 "Connection": "keep-alive"
 }
 },
 "producer": {
 "modsecurity": "ModSecurity v3.0.13 (Linux)",
 "connector": "ModSecurity-nginx v1.0.3",
 "secrules_engine": "Enabled",
 "components": [
 "OWASP_CRS/4.7.0\""
 ]
 },
 "messages": [
 {
 "message": " 'UNAUTHORIZED RESPONSE DATA ACCESS'",
 "details": {
 "match": "Matched \"Operator `Rx' with parameter `(?i)(\\n|\\''|\\:|\\W*)leak' against variable `RESPONSE_BODY' (Value: `leak' )",
 "reference": "o0,4o0,0v208,4",
 "ruleId": "10",
 "file": "/var/opt/modsecurity.d/custom-ruleset/TestRukeset/TestRukeset-RULES.conf",
 "lineNumber": "2",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 },
 {
 "message": "Inbound Score: 0 Outbound Score: 0",
 "details": {
 "match": "",
 "reference": "",
 "ruleId": "980145",
 "file": "/var/opt/modsecurity.d/owasp-crs/weborion.licensing.portal.crs_setup.conf",
 "lineNumber": "40",
 "data": "",
 "severity": "0",
 "ver": "",
 "rev": "",
 "tags": [],
 "maturity": "0",
 "accuracy": "0"
 }
 }
 ]
 }
}

Here is the error log:

2024年11月15日 01:30:46 [error] 90#90: *5 [client 10.103.253.240] ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Rx' with parameter `(?i)(\n|\''|\:|\W*)leak' against variable `RESPONSE_BODY' (Value: `leak' ) [file "/var/opt/modsecurity.d/custom-ruleset/TestRukeset/TestRukeset-RULES.conf"] [line "2"] [id "10"] [rev ""] [msg " 'UNAUTHORIZED RESPONSE DATA ACCESS'"] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "192.168.32.3"] [uri "/check"] [unique_id "4bd7fa0fac010feeb629092cd4c416d4"] [ref "o0,4o0,0v259,4"] while sending to client, client: 10.103.253.240, server: weborion.licensing.portal, request: "POST /check HTTP/1.1", upstream: "http://10.103.253.191:5000/check", host: "weborion.licensing.portal"

Modsecurity-CRS Docker Image
I have also attached the docker image as a tar file, if it is of any assistance. Here is the modified docker file to build the image. You can reference it from here, https://github.com/coreruleset/modsecurity-crs-docker/blob/main/nginx/Dockerfile. I have made the line change for the modsecurity-nginx connector. Here:

ARG NGINX_VERSION="n/a"
FROM nginxinc/nginx-unprivileged:${NGINX_VERSION} AS build
ARG MODSEC3_VERSION="n/a"
ARG LMDB_VERSION="n/a"
ARG LUA_VERSION="n/a"
USER root
# Note: libpcre3-dev (PCRE 1) is required by the build description,
# even though the build will use PCRE2.
RUN set -eux; \
 echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections; \
 apt-get update -qq; \
 LD_LIBRARY_PATH="" apt-get install -y -qq --no-install-recommends --no-install-suggests \
 automake \
 cmake \
 doxygen \
 g++ \
 git \
 libcurl4-gnutls-dev \
 libfuzzy-dev \
 liblua${LUA_VERSION}-dev \
 libpcre3-dev \
 libpcre2-dev \
 libtool \
 libxml2-dev \
 libmaxminddb-dev \
 libyajl-dev \
 make \
 patch \
 pkg-config \
 ruby \
 zlib1g-dev; \
 apt-get clean; \
 rm -rf /var/lib/apt/lists/*
WORKDIR /sources
RUN set -eux; \
 git clone https://github.com/LMDB/lmdb --branch LMDB_${LMDB_VERSION} --depth 1; \
 make -C lmdb/libraries/liblmdb install; \
 strip /usr/local/lib/liblmdb*.so*
RUN set -eux; \
 git clone https://github.com/owasp-modsecurity/ModSecurity --branch "v${MODSEC3_VERSION}" --depth 1 --recursive; \
 cd ModSecurity; \
 ARCH=$(gcc -print-multiarch); \
 sed -ie "s/i386-linux-gnu/${ARCH}/g" build/ssdeep.m4; \
 sed -ie "s/i386-linux-gnu/${ARCH}/g" build/pcre2.m4; \
 ./build.sh; \
 ./configure --with-yajl --with-ssdeep --with-pcre2 --with-maxmind --enable-silent-rules; \
 make install; \
 strip /usr/local/modsecurity/lib/lib*.so*
# We use master
RUN set -eux; \
 git clone -b fix_response_body_v1.0.3 --depth 1 https://github.com/g00g1/ModSecurity-nginx.git; \
 curl -sSL https://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz -o nginx-${NGINX_VERSION}.tar.gz; \
 tar -xzf nginx-${NGINX_VERSION}.tar.gz; \
 cd ./nginx-${NGINX_VERSION}; \
 ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx; \
 make modules; \
 strip objs/ngx_http_modsecurity_module.so; \
 cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/; \
 mkdir /etc/modsecurity.d; \
 curl -sSL https://raw.githubusercontent.com/owasp-modsecurity/ModSecurity/v3/master/unicode.mapping \
 -o /etc/modsecurity.d/unicode.mapping
# Generate/Download Diffie-Hellman parameter files
RUN set -eux; \
 mkdir -p /usr/share/TLS; \
 curl -sSL https://ssl-config.mozilla.org/ffdhe2048.txt -o /usr/share/TLS/dhparam-2048.pem; \
 curl -sSL https://ssl-config.mozilla.org/ffdhe4096.txt -o /usr/share/TLS/dhparam-4096.pem
FROM nginxinc/nginx-unprivileged:${NGINX_VERSION} AS crs_release
ARG CRS_RELEASE
USER root
# hadolint ignore=DL3008,SC2016
RUN set -eux; \
 apt-get update; \
 apt-get -y install --no-install-recommends \
 ca-certificates \
 curl \
 gnupg; \
 mkdir /opt/owasp-crs; \
 chown nginx:nginx /opt/owasp-crs
USER nginx
WORKDIR /sources
RUN curl -sSL https://github.com/coreruleset/coreruleset/releases/download/v${CRS_RELEASE}/coreruleset-${CRS_RELEASE}-minimal.tar.gz -o v${CRS_RELEASE}-minimal.tar.gz; \
 curl -sSL https://github.com/coreruleset/coreruleset/releases/download/v${CRS_RELEASE}/coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc -o coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc; \
 gpg --fetch-key https://coreruleset.org/security.asc; \
 gpg --verify coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc v${CRS_RELEASE}-minimal.tar.gz; \
 tar -zxf v${CRS_RELEASE}-minimal.tar.gz --strip-components=1 -C /opt/owasp-crs; \
 rm -f v${CRS_RELEASE}-minimal.tar.gz coreruleset-${CRS_RELEASE}-minimal.tar.gz.asc; \
 mv -v /opt/owasp-crs/crs-setup.conf.example /opt/owasp-crs/crs-setup.conf
FROM nginxinc/nginx-unprivileged:${NGINX_VERSION}
ARG MODSEC3_VERSION
ARG LMDB_VERSION
ARG LUA_VERSION
ARG LUA_MODULES
LABEL maintainer="Felipe Zipitria <felipe.zipitria@owasp.org>"
ENV \
 ACCESSLOG=/var/log/nginx/access.log \
 BACKEND=http://localhost:80 \
 DNS_SERVER= \
 ERRORLOG=/var/log/nginx/error.log \
 KEEPALIVE_TIMEOUT=60s \
 LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib \
 LOGLEVEL=warn \
 METRICS_ALLOW_FROM='127.0.0.0/24' \
 METRICS_DENY_FROM='all' \
 METRICSLOG=/dev/null \
 MODSEC_ARGUMENT_SEPARATOR="&" \
 MODSEC_ARGUMENTS_LIMIT=1000 \
 MODSEC_AUDIT_ENGINE="RelevantOnly" \
 MODSEC_AUDIT_LOG=/dev/stdout \
 MODSEC_AUDIT_LOG_FORMAT=JSON \
 MODSEC_AUDIT_LOG_PARTS='ABIJDEFHZ' \
 MODSEC_AUDIT_LOG_RELEVANT_STATUS="^(?:5|4(?!04))" \
 MODSEC_AUDIT_LOG_TYPE=Serial \
 MODSEC_COOKIE_FORMAT=0 \
 MODSEC_AUDIT_STORAGE_DIR=/var/log/modsecurity/audit/ \
 MODSEC_DATA_DIR=/tmp/modsecurity/data \
 MODSEC_DEBUG_LOG=/dev/null \
 MODSEC_DEBUG_LOGLEVEL=0 \
 MODSEC_DEFAULT_PHASE1_ACTION="phase:1,pass,log,tag:'\${MODSEC_TAG}'" \
 MODSEC_DEFAULT_PHASE2_ACTION="phase:2,pass,log,tag:'\${MODSEC_TAG}'" \
 MODSEC_DISABLE_BACKEND_COMPRESSION="Off" \
 MODSEC_PCRE_MATCH_LIMIT=100000 \
 MODSEC_PCRE_MATCH_LIMIT_RECURSION=100000 \
 MODSEC_REQ_BODY_ACCESS=on \
 MODSEC_REQ_BODY_JSON_DEPTH_LIMIT=512 \
 MODSEC_REQ_BODY_LIMIT=13107200 \
 MODSEC_REQ_BODY_LIMIT_ACTION="Reject" \
 MODSEC_REQ_BODY_NOFILES_LIMIT=131072 \
 MODSEC_RESP_BODY_ACCESS=on \
 MODSEC_RESP_BODY_LIMIT=1048576 \
 MODSEC_RESP_BODY_LIMIT_ACTION="ProcessPartial" \
 MODSEC_RESP_BODY_MIMETYPE="text/plain text/html text/xml" \
 MODSEC_RULE_ENGINE=on \
 MODSEC_STATUS_ENGINE="Off" \
 MODSEC_TAG=modsecurity \
 MODSEC_TMP_DIR=/tmp/modsecurity/tmp \
 MODSEC_TMP_SAVE_UPLOADED_FILES="on" \
 MODSEC_UNICODE_MAPPING=20127 \
 MODSEC_UPLOAD_DIR=/tmp/modsecurity/upload \
 MODSEC_UPLOAD_FILE_MODE=0600 \
 MODSEC_UPLOAD_KEEP_FILES=Off \
 NGINX_ALWAYS_TLS_REDIRECT=off \
 NGINX_ENVSUBST_OUTPUT_DIR=/etc/nginx \
 PORT=8080 \
 PROXY_SSL_CERT=/etc/nginx/conf/proxy.crt \
 PROXY_SSL_CERT_KEY=/etc/nginx/conf/proxy.key \
 PROXY_SSL_CIPHERS="ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" \
 PROXY_SSL=off \
 PROXY_SSL_PROTOCOLS="TLSv1.2 TLSv1.3" \
 PROXY_SSL_VERIFY_DEPTH=1 \
 PROXY_SSL_VERIFY=off \
 PROXY_TIMEOUT=60s \
 REAL_IP_HEADER="X-REAL-IP" \
 REAL_IP_PROXY_HEADER="X-REAL-IP" \
 REAL_IP_RECURSIVE="on" \
 SERVER_NAME=localhost \
 SERVER_TOKENS=off \
 SET_REAL_IP_FROM="127.0.0.1" \
 SSL_CERT=/etc/nginx/conf/server.crt \
 SSL_CERT_KEY=/etc/nginx/conf/server.key \
 SSL_CIPHERS="ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" \
 SSL_DH_BITS=2048 \
 SSL_OCSP_STAPLING=on \
 SSL_PORT=8443 \
 SSL_PREFER_CIPHERS=off \
 SSL_PROTOCOLS="TLSv1.2 TLSv1.3" \
 SSL_VERIFY_DEPTH=1 \
 SSL_VERIFY=off \
 WORKER_CONNECTIONS=1024 \
 # CRS specific variables
 PARANOIA=1 \
 ANOMALY_INBOUND=5 \
 ANOMALY_OUTBOUND=4 \
 BLOCKING_PARANOIA=1
COPY --from=build /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/
COPY --from=build /etc/nginx/modules/ngx_http_modsecurity_module.so /etc/nginx/modules/ngx_http_modsecurity_module.so
COPY --from=build /usr/local/lib/liblmdb.so /usr/local/lib/
COPY --from=build /usr/share/TLS/dhparam-* /etc/ssl/certs/
COPY --from=build /etc/modsecurity.d/unicode.mapping /etc/modsecurity.d/unicode.mapping
COPY --from=crs_release /opt/owasp-crs /opt/owasp-crs
COPY src/etc/modsecurity.d/modsecurity.conf /etc/nginx/templates/modsecurity.d/modsecurity.conf.template
COPY src/etc/modsecurity.d/modsecurity-override.conf /etc/nginx/templates/modsecurity.d/modsecurity-override.conf.template
COPY src/etc/modsecurity.d/setup.conf /etc/nginx/templates/modsecurity.d/setup.conf.template
COPY nginx/docker-entrypoint.d/*.sh /docker-entrypoint.d/
COPY src/opt/modsecurity/activate-plugins.sh /docker-entrypoint.d/94-activate-plugins.sh
COPY src/opt/modsecurity/activate-rules.sh /docker-entrypoint.d/95-activate-rules.sh
# We use the templating mechanism from the nginx image here.
COPY nginx/templates /etc/nginx/templates/
COPY src/bin/* /usr/local/bin/
USER root
RUN set -eux; \
 echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections; \
 apt-get update -qq; \
 LD_LIBRARY_PATH="" apt-get install -y -qq --no-install-recommends --no-install-suggests \
 ca-certificates \
 curl \
 libcurl4-gnutls-dev \
 libfuzzy2 \
 liblua${LUA_VERSION} \
 ${LUA_MODULES} \
 libxml2 \
 libyajl2 \
 libmaxminddb-dev \
 moreutils; \
 rm -rf /var/lib/apt/lists/*; \
 apt-get clean; \
 mkdir /etc/nginx/ssl; \
 # Comment out the SecDisableBackendCompression option since it is not supported in V3
 sed -i 's/^\(SecDisableBackendCompression .*\)/# 1円/' /etc/nginx/templates/modsecurity.d/modsecurity-override.conf.template; \
 ln -s /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/libmodsecurity.so.3.0; \
 ln -s /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/libmodsecurity.so.3; \
 ln -s /usr/local/modsecurity/lib/libmodsecurity.so.${MODSEC3_VERSION} /usr/local/modsecurity/lib/libmodsecurity.so; \
 ln -sv /opt/owasp-crs /etc/modsecurity.d/; \
 chown nginx:nginx /opt/owasp-crs /etc/modsecurity.d
USER nginx
RUN mkdir -p /tmp/modsecurity/data; \
 mkdir -p /tmp/modsecurity/upload; \
 mkdir -p /tmp/modsecurity/tmp
HEALTHCHECK CMD /usr/local/bin/healthcheck

Copy link
Member

airween commented Nov 15, 2024

Hi @Dr-Lazarus-V2,

thanks again for this very detailed report.

I try to review that soon and reconstruct the expected behavior.

Copy link
Member

airween commented Nov 17, 2024

Guys,

seems like I found the root cause of the issue why isn't there any lines under messages key.

If I have a custom configuration like this:

 error_page 403 /403.html;
 location /403.html {
 root /usr/share/nginx/html;
 internal;
 }

then (I think) this block starts a new transaction, which clears all collected messages inside of the transaction. In the second transaction there is no rule match so the messages in audit log will be empty.

Could you confirm this?

Copy link

Dr-Lazarus-V2 commented Nov 18, 2024 •
edited
Loading

Yes, I can confirm this. With the following configuration, I also noticed that in the second transaction (after the new transaction) there is no rule match and the audit logs are empty. I believe it is linked to this comment: #326 (comment)

Copy link

Dr-Lazarus-V2 commented Nov 18, 2024 •
edited
Loading

I believe this issue relates to the previously raised bug in #182, as the problem appears to be recurring here. Since the rule is in Phase 4, there is no body due to the error redirect, which seems to prevent the rule from being triggered. Based on this, I do not believe this PR introduces the bug. As metioned in #182, this seems to be a potential workaround:

Hi,
@martinhsv thanks for providing those workarounds as I am running into the same issue.
I think I found one more way that might help.
It is to use a named location for the error page instead of a URL, like so:
error_page 403 @error;
location @error {
 ...
}
Then NGINX won't change the request method to GET and full audit log will appear with correct messages in part H.
If there is no need to change URI and method during internal redirection it is possible to pass error processing into a named location: [...]
(http://nginx.org/en/docs/http/ngx_http_core_module.html#error_page)
EDIT: I just noticed the reference to https://github.com/owasp-modsecurity/ModSecurity-nginx/issues/152#issuecomment-593259386 which seems to use the same solution (although in a little more complicated way)

However from a deeper investigation, it is clear that the this is not a full workaround but a temporary one. There does seem to be a PR relating to it: #204

Copy link
Member

airween commented Nov 18, 2024

I believe this issue relates to the previously raised bug in #182, as the problem appears to be recurring here. Since the rule is in Phase 4, there is no body due to the error redirect, which seems to prevent the rule from being triggered. Based on this, I do not believe this PR introduces the bug. As metioned in #182, this seems to be a potential workaround:

Hi,
@martinhsv thanks for providing those workarounds as I am running into the same issue.
I think I found one more way that might help.
It is to use a named location for the error page instead of a URL, like so:
error_page 403 @error;
location @error {
 ...
}
Then NGINX won't change the request method to GET and full audit log will appear with correct messages in part H.
If there is no need to change URI and method during internal redirection it is possible to pass error processing into a named location: [...]
(http://nginx.org/en/docs/http/ngx_http_core_module.html#error_page)
EDIT: I just noticed the reference to https://github.com/owasp-modsecurity/ModSecurity-nginx/issues/152#issuecomment-593259386 which seems to use the same solution (although in a little more complicated way)

However from a deeper investigation, it is clear that the this is not a full workaround but a temporary one. There does seem to be a PR relating to it: #204

Yes, I saw this and other redirect related fixes. Now I try to make a new PR which merges all of these.

Copy link

Is there anything I can do to help?

Copy link
Member

airween commented Nov 18, 2024

Is there anything I can do to help?

If you have any C experiences, you can try to pick up any of related PR from the pending ones and try to apply those on this one. Or later (if I can progress with the patch) you can help to test that.

Copy link
Member

airween commented Nov 19, 2024

I tried to add these modifications.

The good news is that there is no new transaction, I see only one transaction in debug.log.

The bad news that there is no audit.log at all. (Audit.log had created but it's empty.)

Copy link
Member

airween commented Nov 19, 2024

Okay, picked up the modifications from #273 (that's an old PR and it's not able to build) the audit.log is generated, no new transaction started and this patch also works as well.

PR #204 is not necessary.

I'm going to add a new PR soon.

Copy link

Hey was caught up in other work, I will have a look at it now!
airween reacted with thumbs up emoji

Copy link
Member

airween commented Dec 22, 2024

@Dr-Lazarus-V2 could you test this patch?

Copy link

Okiee! Will do!
airween reacted with thumbs up emoji

@airween airween merged commit 62639fa into owasp-modsecurity:master Jan 10, 2025
airween added a commit that referenced this pull request Jan 10, 2025
airween added a commit to airween/ModSecurity-nginx that referenced this pull request Feb 5, 2025
airween added a commit that referenced this pull request Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /