8
24
Fork
You've already forked nuMatrix
4

Make referrer-spoofing more granular #1153

Closed
opened 2023年11月08日 23:54:31 +01:00 by arek · 19 comments
Owner
Copy link

Submitted by: madduck
Submitted at: 2019年07月05日T11:10:05Z


Prerequisites

  • I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • This is not a support issue or a question
    • Support issues and questions are handled at /r/uMatrix
  • I tried to reproduce the issue when...
    • uMatrix is the only extension
    • uMatrix with default lists/settings
    • using a new, unmodified browser profile
  • I am running the latest version of uMatrix
  • I checked the documentation to understand that the issue I report is not a normal behavior
  • I used the logger to rule out that the issue is caused by my ruleset

Description

I have referrer spoofing turned on globally.

referrer-spoof: * True

In the case of Trademe, I found I need to disable the spoofing for the login to work. In this very example, trademe.nz is the website, and the login process causes a XHR to api.trademe.co.nz. This request is denied unless the HTTP referrer specifies trademe.nz.

Without any changes to the ruleset, the XHR does not contain any HTTP referrer
field, which is weird, because I thought spoofing would have it send the
target domain's root document, i.e. https://api.trademe.co.nz, but there is
no header whatsoever. Consequently, the XHR yields a 401 response.

Since api.trademe.co.nz needs the referrer data for security decisions
(let's not discuss this...), I decided that I need to turn off referrer
spoofing, and naively thought I'd just add:

referrer-spoof: api.trademe.co.nz False

This, however, did not have any effect.

It turns out that instead I need to add

referrer-spoof: trademe.nz False

I.e. I need to turn off referrer-spoofing for all of trademe.nz, which also means that the site can now send referrer information to all the other 3rd party servers it contacts, even though I don't want that to happen.

While the rules syntax, i.e. specifying the rule, the context, and the verdict, makes sense in the case of e.g. web workers (I want to disable web workers in the context of trademe.nz and there is no 3rd party involved), referrer spoofing is different, as it involves a context, and a 3rd party.

In some ways, I think that a different rules syntax is required, i.e.:

* * referrer block
* api.trademe.co.nz referrer allow

this may be too much of an ask at this stage.

And even though I think the behaviour should be different, i.e. specifying referrer-spoof: trademe.nz True should cause trademe.nz to always see fake/spoofed referrer information, the current behaviour (trademe.nz is spoofed when used in a referrer header) should be better documented.

What do you think?

A specific URL where the issue occurs

https://trademe.nz login calling out to https://api.trademe.co.nz

Steps to Reproduce

  1. Disable referrer spoofing globally
  2. Enable referrer spoofing for api.trademe.co.nz
  3. Log in to https://trademe.nz and watch it fail
  4. Use network console to inspect the request and witness lack of HTTP referrer header
  5. Change referrer spoofing rule to trademe.nz and watch the request to api.trademe.co.nz receive a spoofed referrer, which makes the login work.

Ruleset

referrer-spoof: * True
referrer-spoof: trademe.nz False
#referrer-spoof: api.trademe.co.nz False

Supporting evidence

Request with referrer-spoofing enabled, as well as disabled for api.trademe.co.nz:

POST https://api.trademe.co.nz/v1/authentication/token.json HTTP/1.1
Host: api.trademe.co.nz
[...]
Origin: https://trademe.nz
[...]

Request with referrer-spoofing disabled for trademe.nz:

POST https://api.trademe.co.nz/v1/authentication/token.json HTTP/1.1
Host: api.trademe.co.nz
[...]
Referer: https://trademe.nz/(modal:login)
Origin: https://trademe.nz
[...]

Your environment

  • uMatrix version: 1.3.16
  • Browser Name and version: Firefox 67.0.4
  • Operating System and version: Debian sid
Submitted by: [madduck](https://github.com/madduck) Submitted at: [2019年07月05日T11:10:05Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171) ----- <!-- Do NOT delete this template or any part of it when submitting your issue --> ### Prerequisites <!-- Check the appropriate boxes after you submit your issue --> <!-- Speculated performance issues will be marked as invalid and closed if they do not come with actual profiling data + analysis supporting the claim --> - [x] I performed a cursory search of the issue tracker to avoid opening a duplicate issue - Your issue may already be reported. - [x] I also searched the existing issues at <https://github.com/gorhill/uMatrix/issues> - [x] This is not a support issue or a question - Support issues and questions are handled at [/r/uMatrix](https://old.reddit.com/r/uMatrix/) - I tried to reproduce the issue when... - [X] uMatrix is the only extension - [X] uMatrix with default lists/settings - [x] using a new, unmodified browser profile - [x] I am running the latest version of uMatrix - [x] I checked the [documentation](https://github.com/gorhill/uMatrix/wiki) to understand that the issue I report is not a normal behavior - [x] I used the logger to rule out that the issue is caused by my ruleset ### Description I have referrer spoofing turned on globally. ``` referrer-spoof: * True ``` In the case of Trademe, I found I need to disable the spoofing for the login to work. In this very example, `trademe.nz` is the website, and the login process causes a XHR to `api.trademe.co.nz`. This request is denied unless the HTTP referrer specifies `trademe.nz`. Without any changes to the ruleset, the XHR does not contain any HTTP referrer field, which is weird, because I thought spoofing would have it send the target domain's root document, i.e. `https://api.trademe.co.nz`, but there is no header whatsoever. Consequently, the XHR yields a 401 response. Since `api.trademe.co.nz` needs the referrer data for security decisions (let's not discuss this...), I decided that I need to turn off referrer spoofing, and naively thought I'd just add: ``` referrer-spoof: api.trademe.co.nz False ``` This, however, did not have any effect. It turns out that instead I need to add ``` referrer-spoof: trademe.nz False ``` I.e. I need to turn off referrer-spoofing for all of `trademe.nz`, which also means that the site can now send referrer information to all the other 3rd party servers it contacts, even though I don't want that to happen. While the rules syntax, i.e. specifying the rule, the context, and the verdict, makes sense in the case of e.g. web workers (I want to disable web workers in the context of `trademe.nz` and there is no 3rd party involved), referrer spoofing is different, as it involves a context, and a 3rd party. In some ways, I think that a different rules syntax is required, i.e.: ``` * * referrer block * api.trademe.co.nz referrer allow ``` this may be too much of an ask at this stage. And even though I think the behaviour should be different, i.e. specifying `referrer-spoof: trademe.nz True` should cause `trademe.nz` to always see fake/spoofed referrer information, the current behaviour (`trademe.nz` is spoofed when used in a referrer header) should be better documented. What do you think? ### A specific URL where the issue occurs `https://trademe.nz` login calling out to `https://api.trademe.co.nz` ### Steps to Reproduce 1. Disable referrer spoofing globally 2. Enable referrer spoofing for `api.trademe.co.nz` 3. Log in to https://trademe.nz and watch it fail 4. Use network console to inspect the request and witness lack of HTTP referrer header 5. Change referrer spoofing rule to `trademe.nz` and watch the request to `api.trademe.co.nz` receive a spoofed referrer, which makes the login work. ### Ruleset ``` referrer-spoof: * True referrer-spoof: trademe.nz False #referrer-spoof: api.trademe.co.nz False ``` ### Supporting evidence Request with referrer-spoofing enabled, as well as disabled for `api.trademe.co.nz`: ``` POST https://api.trademe.co.nz/v1/authentication/token.json HTTP/1.1 Host: api.trademe.co.nz [...] Origin: https://trademe.nz [...] ``` Request with referrer-spoofing disabled for `trademe.nz`: ``` POST https://api.trademe.co.nz/v1/authentication/token.json HTTP/1.1 Host: api.trademe.co.nz [...] Referer: https://trademe.nz/(modal:login) Origin: https://trademe.nz [...] ``` ### Your environment * uMatrix version: 1.3.16 * Browser Name and version: Firefox 67.0.4 * Operating System and version: Debian sid
Author
Owner
Copy link

Submitted by: uBlock-user
Submitted at: 2019年07月05日T11:33:09Z


Log in to https://trademe.nz and watch it fail

Username and password ?

Submitted by: [uBlock-user](https://github.com/uBlock-user) Submitted at: [2019年07月05日T11:33:09Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508729722) ----- > Log in to https://trademe.nz and watch it fail Username and password ?
Author
Owner
Copy link

Submitted by: uBlock-user
Submitted at: 2019年07月05日T11:35:59Z


Also FYI -- https://github.com/uBlockOrigin/uMatrix-issues/wiki/Per-scope-switches#spoof-referer-header

Why not login via https://www.trademe.co.nz/Members/Login.aspx?url=%2fdefault.aspx ?

Submitted by: [uBlock-user](https://github.com/uBlock-user) Submitted at: [2019年07月05日T11:35:59Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508730305) ----- Also FYI -- https://github.com/uBlockOrigin/uMatrix-issues/wiki/Per-scope-switches#spoof-referer-header Why not login via `https://www.trademe.co.nz/Members/Login.aspx?url=%2fdefault.aspx` ?
Author
Owner
Copy link

Submitted by: gorhill
Submitted at: 2019年07月05日T11:41:29Z


The hostname part in the referrer spoofing switch is matched against the context, aka the hostname of the root document, not the hostname of outgoing network requests. The context is the top left cell in the popup panel.

Submitted by: [gorhill](https://github.com/gorhill) Submitted at: [2019年07月05日T11:41:29Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508731513) ----- The hostname part in the referrer spoofing switch is matched against the context, aka the hostname of the root document, not the hostname of outgoing network requests. The context is the top left cell in the popup panel.
Author
Owner
Copy link

Submitted by: gorhill
Submitted at: 2019年07月05日T11:46:55Z


What do you think?

The reason the per-site switches are designed the way they are is simply because of the UI. To have a referrer-spoof switch which takes into consideration context and destination hostnames would require a new column in the matrix, which I decided was too much in the past. As a consequence per-site switches are somewhat blunt, and for a more precise control, the advice is to install a more specialezed extension for either referrer-spoofing, cookie control, etc.

Submitted by: [gorhill](https://github.com/gorhill) Submitted at: [2019年07月05日T11:46:55Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508732783) ----- > What do you think? The reason the per-site switches are designed the way they are is simply because of the UI. To have a referrer-spoof switch which takes into consideration context _and_ destination hostnames would require a new column in the matrix, which I decided was too much in the past. As a consequence per-site switches are somewhat blunt, and for a more precise control, the advice is to install a more specialezed extension for either referrer-spoofing, cookie control, etc.
Author
Owner
Copy link

Submitted by: madduck
Submitted at: 2019年07月05日T11:53:25Z


Why not login via https://www.trademe.co.nz/Members/Login.aspx?url=%2fdefault.aspx ?

The mobile site is under the other URL.

Submitted by: [madduck](https://github.com/madduck) Submitted at: [2019年07月05日T11:53:25Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508734342) ----- > Why not login via `https://www.trademe.co.nz/Members/Login.aspx?url=%2fdefault.aspx` ? The mobile site is under the other URL.
Author
Owner
Copy link

Submitted by: madduck
Submitted at: 2019年07月05日T12:02:05Z


The hostname part in the referrer spoofing switch is matched against the context, aka the hostname of the root document, not the hostname of outgoing network requests. The context is the top left cell in the popup panel.

Yes, this is what I found as well. And I appreciate your explanation, aiming at simplicity. I guess what I am suggesting foremost is that the behaviour be properly documented, as https://github.com/uBlockOrigin/uMatrix-issues/wiki/Per-scope-switches#spoof-referer-header doesn't make this obvious.

I personally wouldn't think another column would be too much, and it's not like it'll open a can of worms, because I can't think of any other columns that would then also need to be added.

Adding such a column would give me the freedom to really express what I want and expect to happen.

I, for one, find it rather weird that in order for a website to be named as referrer in a specific 3rd party request, I need to whitelist the website itself, thereby allowing it to be named as referrer in all outgoing requests. Generally, I want to inhibit someone receiving information, not someone sending it, and then I want to punch specific holes through this ruleset. In the case of the referrer, some sites rely on it. I want to whitelist those sites, not all sites that refer to the arguably broken ones relying on such data.

Submitted by: [madduck](https://github.com/madduck) Submitted at: [2019年07月05日T12:02:05Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508736396) ----- > The hostname part in the referrer spoofing switch is matched against the context, aka the hostname of the root document, not the hostname of outgoing network requests. The context is the top left cell in the popup panel. Yes, this is what I found as well. And I appreciate your explanation, aiming at simplicity. I guess what I am suggesting foremost is that the behaviour be properly documented, as https://github.com/uBlockOrigin/uMatrix-issues/wiki/Per-scope-switches#spoof-referer-header doesn't make this obvious. I personally wouldn't think another column would be too much, and it's not like it'll open a can of worms, because I can't think of any other columns that would then also need to be added. Adding such a column would give me the freedom to really express what I want and expect to happen. I, for one, find it rather weird that in order for a website to be named as referrer in a specific 3rd party request, I need to whitelist the website itself, thereby allowing it to be named as referrer in *all* outgoing requests. Generally, I want to inhibit someone receiving information, not someone sending it, and then I want to punch specific holes through this ruleset. In the case of the referrer, some sites rely on it. I want to whitelist those sites, not all sites that refer to the arguably broken ones relying on such data.
Author
Owner
Copy link

Submitted by: gorhill
Submitted at: 2019年07月05日T12:05:50Z


I will consider adding referrer as a new type (for matrix-filtering purpose), but adding a column to the matrix is too much of a big change for now, so whoever wants finer control of referrer spoofing can always manually craft them.

Submitted by: [gorhill](https://github.com/gorhill) Submitted at: [2019年07月05日T12:05:50Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508737239) ----- I will consider adding `referrer` as a new type (for matrix-filtering purpose), but adding a column to the matrix is too much of a big change for now, so whoever wants finer control of referrer spoofing can always manually craft them.
Author
Owner
Copy link

Submitted by: uBlock-user
Submitted at: 2019年07月05日T12:10:04Z


new type

So what would be the rules be like ?

Submitted by: [uBlock-user](https://github.com/uBlock-user) Submitted at: [2019年07月05日T12:10:04Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508738254) ----- > new type So what would be the rules be like ?
Author
Owner
Copy link

Submitted by: gorhill
Submitted at: 2019年07月05日T12:13:08Z


Like any matrix filtering rule: source destination type action, example

trademe.nz * referrer block
trademe.nz api.trademe.co.nz referrer allow
Submitted by: [gorhill](https://github.com/gorhill) Submitted at: [2019年07月05日T12:13:08Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508739053) ----- Like any matrix filtering rule: `source destination type action`, example trademe.nz * referrer block trademe.nz api.trademe.co.nz referrer allow
Author
Owner
Copy link

Submitted by: uBlock-user
Submitted at: 2019年07月05日T12:14:13Z


Would * * referrer block work ?

Submitted by: [uBlock-user](https://github.com/uBlock-user) Submitted at: [2019年07月05日T12:14:13Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508739295) ----- Would `* * referrer block` work ?
Author
Owner
Copy link

Submitted by: gorhill
Submitted at: 2019年07月05日T12:15:12Z


Sure, matrix-filtering rules are all handled with the same core algorithm.

Submitted by: [gorhill](https://github.com/gorhill) Submitted at: [2019年07月05日T12:15:12Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508739522) ----- Sure, matrix-filtering rules are all handled with the same core algorithm.
Author
Owner
Copy link

Submitted by: uBlock-user
Submitted at: 2019年07月05日T12:16:03Z


and would we able to configure it from the uMatrix's popup panel via the referrer switch ?

Submitted by: [uBlock-user](https://github.com/uBlock-user) Submitted at: [2019年07月05日T12:16:03Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508739738) ----- and would we able to configure it from the uMatrix's popup panel via the referrer switch ?
Author
Owner
Copy link

Submitted by: gorhill
Submitted at: 2019年07月05日T12:17:32Z


The switch would just be converted to a matrix-filtering rule from a per-site switch rule.

Submitted by: [gorhill](https://github.com/gorhill) Submitted at: [2019年07月05日T12:17:32Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508740096) ----- The switch would just be converted to a matrix-filtering rule from a per-site switch rule.
Author
Owner
Copy link

Submitted by: uBlock-user
Submitted at: 2019年07月05日T12:19:35Z


No I meant if I have to disable referrrer spoofing for an embedded domain as an iframe on some website, would I need to do from the My rules tab manually or will I able to do it via the popup panel itself ?

Submitted by: [uBlock-user](https://github.com/uBlock-user) Submitted at: [2019年07月05日T12:19:35Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508740602) ----- No I meant if I have to disable referrrer spoofing for an embedded domain as an iframe on some website, would I need to do from the My rules tab manually or will I able to do it via the popup panel itself ?
Author
Owner
Copy link

Submitted by: gorhill
Submitted at: 2019年07月05日T12:24:49Z


All matrix rules use the hostname of the root context as source and always have, never that of an embedded document (iframe), so the answer is you will be able to do it through the popup panel.

Submitted by: [gorhill](https://github.com/gorhill) Submitted at: [2019年07月05日T12:24:49Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508741962) ----- All matrix rules use the hostname of the root context as `source` and always have, never that of an embedded document (`iframe`), so the answer is you will be able to do it through the popup panel.
Author
Owner
Copy link

Submitted by: madduck
Submitted at: 2019年07月05日T12:41:13Z


I will consider adding referrer as a new type, but adding a column to the matrix is too much of a big change for now, so whoever wants finer control of referrer spoofing can always manually craft them.

Perfect. Thank you!

Submitted by: [madduck](https://github.com/madduck) Submitted at: [2019年07月05日T12:41:13Z](https://github.com/uBlockOrigin/uMatrix-issues/issues/171#issuecomment-508746118) ----- >I will consider adding referrer as a new type, but adding a column to the matrix is too much of a big change for now, so whoever wants finer control of referrer spoofing can always manually craft them. Perfect. Thank you!
Author
Owner
Copy link

already fixed

already fixed

already fixed

To be clear, this doesn't work (source destination referrer action just get removed when one saves their rules).

> already fixed To be clear, this doesn't work (`source destination referrer action` just get removed when one saves their rules).
arek added this to the (deleted) project 2026年06月19日 10:25:42 +02:00
arek added this to the v1.0 project 2026年06月21日 11:25:19 +02:00
arek added this to the v1.0 milestone 2026年06月21日 11:25:21 +02:00
Author
Owner
Copy link

Thank you @acracadabra . I believe now it has been implemented. Rules can be written as:

trademe.nz api.trademe.co.nz referrer allow

Thank you @acracadabra . I believe _now_ it has been implemented. Rules can be written as: > trademe.nz api.trademe.co.nz referrer allow
arek 2026年06月29日 12:38:43 +02:00
  • closed this issue
  • added
    done
    and removed
    backlog
    labels
Sign in to join this conversation.
No Branch/Tag specified
master
dev
v0.1.2
v0.1
0.0.0.8b
0.0.0.7b
0.0.0.5b
0.0.0.4b
0.0.0.3b
0.0.0.2b
0.0.0.1b
1.4.3b0
1.4.2
1.4.1b6
1.4.1b5
1.4.1b4
1.4.0
1.3.17b2
1.3.17b1
1.3.16
1.3.14
1.3.12
1.3.10
1.3.8
1.3.6
1.3.4
1.3.3b9
1.3.3b8
1.3.2
1.3.0
1.2.0
1.1.20
1.1.18
1.1.16
1.1.14
1.1.12
1.1.11b0
1.1.10
1.1.8
1.1.7rc0
1.1.7b0
1.1.6
1.1.4
1.1.0
1.0.0
0.9.3.6
0.9.3.4
0.9.3.3
0.9.3.2
0.9.3.1
0.9.3.0
0.9.2.1
0.9.2.0
0.9.1.2
0.9.1.1
0.9.1.0
0.9.0.1
0.9.0.0
0.8.1.4
0.8.1.3
0.8.1.1
0.8.1.0
0.8.0.1
0.8.0.0
0.8.0.0-rc.2
0.8.0.0-rc.1
0.8.0.0-rc.0
0.8.0.0-alpha.19
0.8.0.0-alpha.18
0.8.0.0-alpha.17
0.8.0.0-alpha.16
0.8.0.0-alpha.15
0.8.0.0-alpha.14
0.8.0.0-alpha.11
0.8.0.0-alpha.10
0.8.0.0-alpha.9
0.8.0.0-alpha.8
0.8.0.0-alpha.7
0.8.0.0-alpha.6
0.8.0.0-alpha.4
0.8.0.0-alpha.3
0.8.0.0-alpha.2
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
arek/nuMatrix#1153
Reference in a new issue
arek/nuMatrix
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?