Reporting API

1. Introduction

[INTRODUCTION GOES HERE]

1.1. Guarantees

This specification aims to provide a best-effort report delivery system that executes out-of-band with website activity. The user agent will be able to do a better job prioritizing and scheduling delivery of reports, as it has an overview of cross-origin activity that individual websites do not, and can deliver reports based on error conditions that would prevent a website from loading in the first place.

The delivery is not, however, guaranteed in a strict sense. We spell out a reasonable set of retry rules in the algorithms below, but it’s quite possible for a report to be dropped on the floor if things go badly.

Reporting can generate a good deal of traffic, so we allow developers to set up groups of endpoints in order to distribute load. Each of these endpoints will receive a subset of the generated reports which target that group. The user agent will do its best to deliver a particular report to at most one endpoint in a group. That is, reports will not fan-out to all the endpoints in a group, but the user agent will attempt delivery to one endpoint, and fallback to another upon failure.

1.2. Examples

Report-To: https://example.com/reports; group=endpoint-1; max-age=10886400,
           https://backup.com/reports;group=endpoint-1; max-age=10886400

Content-Security-Policy: ...; report-to=endpoint-1
Public-Key-Pins: ...; report-to=endpoint-1

Report-To: https://example.com/csp-violations; group=csp-endpoint; max-age=10886400,
           https://example.com/hpkp-reports; group=hpkp-endpoint; max-age=10886400

Content-Security-Policy: ...; report-to=csp-endpoint
Public-Key-Pins: ...; report-to=hpkp-endpoint

2. Concepts

2.1. Endpoints

An endpoint is location to which reports may be sent.

Each endpoint has a url, which is a URL.

Each endpoint has a failures, which is a non-negative integer representing the number of consecutive times this endpoint has failed to respond to a request.

Each endpoint has a retry-after, which is either null, or a timestamp after which delivery should be retried.

Each endpoint has a clients list, which is a list of clients.

An endpoint is expired if every client in its list of clients is expired.

An endpoint is pending if its retry-after is not null, and represents a time in the future.

2.2. Clients

A client represents a particular origin’s relationship to an endpoint.

Each client has an origin, which is an origin.

Each client has a subdomains flag (which is either "include" or "exclude").

Each client has a group which is an ASCII string.

Each client has a ttl representing the the number of seconds the client remains valid for an endpoint.

Each client has a creation which is the timestamp at which the client was added to an endpoint.

A client is expired if its creation plus its ttl represents a time in the past.

2.3. Reports

A report is a collection of arbitrary data which the user agent is expected to deliver to a specified endpoint.

Each report has a body, which is either null or an object which can be serialized into a JSON text.

Each report has an url, which is the address of the Document or Worker from which the report was generated.

Note: We strip the username, password, and fragment from this serialized URL. See §7.1 Capability URLs.

Each report has an origin, which is an origin representing the report’s initiator.

Each report has an group, which is a string representing the group to which this report will be sent.

Each report has a type, which is a non-empty string specifying the type of data the report contains.

Each report has a timestamp, which records the time at which the report was generated, in milliseconds since the unix epoch.

Each report has a attempts, which is a non-negative integer representing the number of times the user agent attempted to deliver the report.

2.4. Storage

A conformant user agent MUST provide a reporting cache, which is a storage mechanism that maintains a set of endpoints that websites have instructed the user agent to associate with their origins, and a set of reports which are queued for delivery.

This storage mechanism is opaque, vendor-specific, and not exposed to the web, but it MUST provide the following methods which will be used in the algorithms this document defines:

1. Insert, update, and remove endpoints.

2. Enqueue and dequeue reports for delivery.

3. Retrieve a list of endpoint objects.

4. Retrieve a list of queued report objects.

5. Clear the cache.

3. Endpoint Delivery

A server MAY define a set of reporting endpoints for an origin it controls via the Report-To HTTP response header field. This mechanism is defined in §3.1 The Report-To HTTP Response Header Field, and its processing in §3.2 Process reporting endpoints for response to request.

3.1. The Report-To HTTP Response Header Field

The Report-To HTTP response header field instructs the user agent to store a reporting endpoints for an origin. The header is represented by the following ABNF grammar [RFC5234]:

Report-To = absolute-URI *( OWS ";" [ OWS parameter ] )
parameter     = key [ BWS "=" BWS value ]
key           = token
value         = token / quoted-string

; The #rule, OWS, BWS, token, and
; quoted-string productions are all part of RFC 7230.
; absolute-uri is defined in RFC 3986.

The header’s value is an absolute URL, followed by a semicolon-delimited list of "[key]=[value]" pairs that govern specified endpoint’s behavior.

The URL MUST be potentially trustworthy [SECURE-CONTEXTS]. Non-secure endpoints will be ignored.

The following subsections define the initial of known parameter component values. Future versions of this document may defined additional such values. The ABNF is intentionally generic and extensible to make room for these future values, and user agents MUST ignore unknown values when parsing the header.

3.1.1. The group parameter

The OPTIONAL group parameter is an ASCII string that associates a name with the reporting endpoint.

parameter-name  = "group"
parameter-value = token

The name is not unique and multiple endpoints may use the same name to create a group of reporting endpoints that can be used for backup and failover purposes. If the parameter is omitted, the endpoint MUST be associated with the "default" group name.

Note: If a group resolves to multiple endpoints, the user agent will deliver a particular report to at most one endpoint in that group on a best-effort basis.

3.1.2. The includeSubdomains parameter

The OPTIONAL includeSubdomains parameter, if present, enables an endpoint for all subdomains of the current origin’s host.

parameter-name  = "includeSubdomains"

The parameter’s value will be ignored, if present.

3.1.3. The max-age parameter

The REQUIRED max-age parameter defines the reporting endpoint’s lifetime, as a non-negative integer number of seconds. The ABNF grammar for the directive is:

parameter-name  = "max-age"
parameter-value = delta-seconds

; delta-seconds is defined in Section 1.2.1 of RFC 7234.

A value of "0" will cause the endpoint to be removed from the user agent’s reporting cache.

3.2. Process reporting endpoints for response to request

Given a response (response) and a request (request), this algorithm extracts a list of endpoint objects, and updates the reporting cache accordingly.

Note: This algorithm is called from around step 13 of main fetch [FETCH], and only updates the reporting cache if the response has been delivered securely.

Fetch monkey patching. Talk to Anne.

1. Abort these steps if any of the following conditions are true:

   1. response’s HTTPS state is not "modern", and the origin of response’s url is not potentially trustworthy.

   2. response’s header list does not contain a header whose name is "Report-To".

2. Let header be the value of the header in response’s header list whose name is "Report-To".

3. Let origin be the origin of response’s url.

4. Let tuples be the result of executing §3.3 Parse reporting endpoint tuples from value on header.

5. For each tuple in tuples:

   1. Let new be a new client whose properties are set as follows:

      origin

      origin

      subdomains

      tuple’s subdomains

      group

      tuple’s group

      ttl

      tuple’s ttl

      creation

      The current timestamp

   2. If there exists an endpoint (endpoint) in the reporting cache whose url is tuple’s URL:

      1. For each client in endpoint’s clients, delete client if new’s origin is the same as client’s origin.

      2. If new’s ttl is greater than 0, append new to endpoint’s clients list.

      3. Skip to the next tuple.

   3. Otherwise, there is no existing endpoint, so let endpoint be a new endpoint whose properties are set as follows:

      url

      tuple’s URL

      failures

      0

      retry-after

      null

      clients

      A new list, containing tuple

   4. Insert endpoint into the reporting cache for origin.

3.3. Parse reporting endpoint tuples from value

Given a string (value) this algorithm will return a list of tuples containing a URL, subdomains flag, ttl, and a group. This list will be empty if no valid endpoints could be parsed.

Do the parsing here once we decide whether we’re going to run with a JSON format, a la [RESCHKE]. <https://github.com/mikewest/error-reporting/issues/4>

4. Report Delivery

Over time, various features will queue up a list of reports in the user agent’s reporting cache. The user agent will periodicially grab the list of currently pending reports, and deliver them to the associated endpoints. This document does not define a schedule for the user agent to follow, and assumes that the user agent will have enough contextual information to deliver reports in a timely manner, balanced against impacting a user’s experience.

That said, a user agent SHOULD make a effort to deliver reports as soon as possible after queuing, as a report’s data might be signficantly more useful in the period directly after its generation than it would be a day or a week later.

4.1. Queue data as type for endpoint group on settings

Given a serializable object (data), a string (type), another string (endpoint group), and an environment settings object (settings), the following algorithm will create a report, and add it to reporting cache’s queue for future delivery.

1. Let report be a new report object with its values initialized as follows:

   body

   data

   origin

   settings’s origin

   group

   endpoint group

   type

   type

   timestamp

   The current timestamp.

   attempts

   0

2. Let url be settings’s creation URL.

3. Set url’s username to the empty string, and its password to null.

4. Set report’s url to the result of executing the URL serializer on url with the exclude fragment flag set.

5. Add report to the reporting cache.

Note: We strip the username, password, and fragment from the serialized URL in the report. See §7.1 Capability URLs.

Note: The user agent MAY reject reports for any reason. This API does not guarantee delivery of arbitrary amounts of data, for instance.

4.2. Does endpoint match report?

Given an endpoint (endpoint) and a report (report), this algorithm returns "Match" if report should be sent to endpoint, and "Does Not Match" otherwise:

1. For each client in endpoint’s clients:

   1. If client is expired, skip to the next client.

      Note: In this case, the user agent MAY remove client from endpoint, or it may wait and collect garbage en masse at some point in the future as described in §5.2 Garbage Collection.

   2. Return "Match" if each of the following criteria is met:

      1. report’s group is an ASCII case-insensitive match for client’s group

      2. If client’s subdomains is "exclude", report’s origin is the same as client’s origin

         Otherwise, report’s origin's host is either a superdomain match or congruent match for client’s origin's host [RFC6797]

2. Return "Does Not Match".

4.3. Send reports

A user agent sends reports by executing the following steps:

1. Let reports be a copy of the list of queued report objects in reporting cache.

2. Let endpoint map be an empty map of endpoint objects to lists of report objects.

3. For each report in reports:

   1. For endpoint in the reporting cache:

      1. If endpoint is expired, skip to the next endpoint.

         Note: In this case, the user agent MAY remove endpoint from the reporting cache, or it may wait and collect garbage en masse at some point in the future as described in §5.2 Garbage Collection.

      2. If endpoint is pending, skip to the next endpoint.

      3. If §4.2 Does endpoint match report? returns "Match" when executed upon endpoint and report:

         1. Append report to endpoint map’s list of reports for endpoint.

         2. Skip to the next report.

            Note: This ensures that each report is assigned to a single endpoint, even if it matches multiple. In order to ensure an even distribution across endpoints, the user agent SHOULD randomize the order in which it walks through endpoints.

   2. If we reach this step, the report did not match any endpoint and the user agent MAY remove report from the reporting cache directly. Depending on load, the user agent MAY instead wait for §5.2 Garbage Collection at some point in the future.

4. For each (endpoint, reports) pair in endpoint map, execute the following steps asynchronously:

   1. Let result be the result of executing §4.4 Attempt to deliver reports to endpoint on endpoint and reports.

   2. If result is "Success":

      1. Set endpoint’s failures to 0, and its retry-after to null.

      2. Remove each report in reports from the reporting cache.

      Otherwise:

      1. Increment endpoint’s failures.

      2. Set endpoint’s retry-after to a point in the future which the user agent chooses.

         Note: We don’t specify a particular algorithm here, but user agents are encouraged to employ some sort of exponential backoff algorithm which increases the retry period with the number of failures, with the addition of some random jitter to ensure that temporary failures don’t lead to a crush of reports all being retried on the same schedule.

         Add in a reasonable reference describing a good algorithm. Wikipedia, if nothing else.

Note: User agents MAY decide to attempt delivery for only a subset of the collected reports or endpoints (because, for example, sending all the reports at once would consume an unreasonable amount of bandwidth, etc). As reports are only removed from the cache when they’re successfully delivered, skipped reports will simply be delivered later.

4.4. Attempt to deliver reports to endpoint

Given a list of reports (reports) and an endpoint (endpoint), this algorithm will construct a request, and attempt to deliver it to endpoint. It returns "Success" if that delivery succeeds, "Remove Endpoint" if the endpoint explicitly removes itself as a reporting endpoint by sending a 410 response, and "Failure" otherwise.

1. Let collection be a new ECMAScript Array object [ECMA-262].

2. For each report in reports:

   1. Let data be a new ECMAScript Object with the following properties [ECMA-262]:

      age

      The number of milliseconds between report’s timestamp and the current time.

      type

      report’s type

      url

      report’s url

      report

      report’s body

      Note: Client clocks are unreliable and subject to skew. We therefore deliver an age attribute rather than an absolute timestamp. See also §8.2 Clock Skew

   2. Increment report’s attempts.

   3. Append data to collection.

3. Let request be a new request with the following properties [FETCH]:

   url

   endpoint’s url

   header list

   A new header list containing a header named "Content-Type" whose value is "application/report"

   client

   null

   window

   "no-window"

   skip-service-worker flag

   Set.

   initiator

   ""

   type

   "report"

   destination

   ""

   mode

   "cors"

   credentials

   "include"

   body

   The string resulting from executing the JSON.stringify() algorithm on collection [ECMA-262]

   The "report" type does not exist in Fetch. Talk to Anne.

4. Queue a task to fetch request.

5. Wait for a response (response).

6. If response’s status is an OK status (200-299), return "Success".

7. If response’s status is 410 Gone [RFC7231], return "Remove Endpoint".

8. Return "Failure".

5. Implementation Considerations

5.1. Delivery

The user agent SHOULD attempt to deliver reports as soon as possible to provide feedback to developers as quickly as possible. However, when this desire is balanced against the impact on the user, the user wins. With that in mind, the user agent MAY delay delivery of reports based on its knowledge of the user’s activities and context.

For instance, the user agent SHOULD prioritize the transmission of reporting data lower than other network traffic. The user’s explicit activities on a website should preempt reporting traffic.

The user agent MAY choose to withhold report delivery entirely until the user is on a fast, cheap network in order to prevent unnecessary data cost.

The user agent MAY choose to prioritize reports from particular origins over others (perhaps those that the user visits most often?)

5.2. Garbage Collection

Periodically, the user agent SHOULD walk through the cached reports and endpoints, and discard those that are no longer relevant. These include:

• endpoints which are expired

• endpoints which have not been used in some arbitrary period of time (perhaps a ~week?)

• endpoints whose failures exceed some user-agent-defined threshold (~5 seems reasonable)

• reports whose attempts exceed some user-agent-defined threshold (~5 seems reasonable)

• reports which have not been delivered in some arbitrary period of time (perhaps ~2 days?)

6. Sample Reports

POST / HTTP/1.1
Host: example.com
...
Content-Type: application/report

[{
  type: "csp",
  age: 10,
  url: "https://example.com/vulnerable-page/",
  report: {
    "blocked": "https://evil.com/evil.js",
    "directive": "script-src",
    "policy": "script-src 'self'; object-src 'none'",
    "status": 200,
    "referrer": "https://evil.com/"
  }
}, {
  type: "hpkp",
  age: 32,
  url: "https://www.example.com/",
  report: {
    "date-time": "2014-04-06T13:00:50Z",
    "hostname": "www.example.com",
    "port": 443,
    "effective-expiration-date": "2014-05-01T12:40:50Z"
    "include-subdomains": false,
    "served-certificate-chain": [
      "-----BEGIN CERTIFICATE-----\n
      MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT\n
      ...
      HFa9llF7b1cq26KqltyMdMKVvvBulRP/F/A8rLIQjcxz++iPAsbw+zOzlTvjwsto\n
      WHPbqCRiOwY1nQ2pM714A5AuTHhdUDqB1O6gyHA43LL5Z/qHQF1hwFGPa4NrzQU6\n
      yuGnBXj8ytqU0CwIPX4WecigUCAkVDNx\n
      -----END CERTIFICATE-----",
      ...
    ]
  }
}, {
  type: "nel",
  age: 29,
  url: "https://example.com/thing.js",
  report: {
    "referrer": "https://www.example.com/",
    "server-ip": "234.233.232.231",
    "protocol": "",
    "status-code": 0,
    "elapsed-time": 143,
    "age": 0,
    "type": "http.dns.name_not_resolved"
  }
}]

7. Security Considerations

7.1. Capability URLs

Some URLs are valuable in and of themselves. To mitigate the possibility that such URLs will be leaked via this reporting mechanism, we strip out credential information and fragment data from the URL we store as a report’s originator. It is still possible, however, for a feature to unintentionally leak such data via a report’s body. Implementers SHOULD ensure that URLs contained in a report’s body are similarly stripped.

8. Privacy Considerations

8.1. Network Leakage

Because this reporting mechanism is out-of-band, and doesn’t rely on a page being open, it’s entirely possible for a report generated while a user is on one network to be sent while the user is on another network, even if they don’t explicitly open the page from which the report was sent.

Consider mitigations. For example, we could drop reports if we change from one network to another. <https://github.com/WICG/BackgroundSync/issues/107>

8.2. Clock Skew

Each report is delivered along with an age property, rather than the timestamp at which it was generated. We do this because each user’s local clock will be skewed from the clock on the server by an arbitrary amount. The difference between the time the report was generated and the time it was sent will be stable, regardless of clock skew, and we can avoid the fingerprinting risk of exposing the clock skew via this API.

8.3. Cross-origin correlation

If multiple origins all use the same reporting endpoint, that endpoint may learn that a particular user has interacted with a certain set of websites, as it will receive origin-tagged reports from each. This doesn’t seem worse than the status quo ability to track the same information from cooperative origins, and doesn’t grant any new tracking ability above and beyond what’s possible with <img> today.

8.4. Subdomains

This specification allows any resource on a host to declare a set of reporting endpoints for that host and each of its subdomains. This doesn’t have privacy implications in and of itself (beyond those noted in §8.5 Clearing the reporting cache), as the reporting endpoints themselves don’t take any real action, as features will need to opt-into using these reporting endpoints explicitly. Those features certainly will have privacy implications, and should carefully consider whether they should be enabled across origin boundries.

8.5. Clearing the reporting cache

A user agent’s reporting cache contains data about a user’s activity on the web, and user agents ought to handle this data carefully. In particular, if a user agent gives users the ability to clear their site data, browsing history, browsing cache, or similar, the user agent MUST also clear the reporting cache. Note that this includes both the pending reports themselves, as well as the endpoints to which they would be sent. Both MUST be cleared.

9. IANA Considerations

The permanent message header field registry should be updated with the following registration: [RFC3864]

9.1. Report-To

Header field name

Report-To

Applicable protocol

http

Status

standard

Author/Change controller

W3C

Specification document

This specification (see §3.1 The Report-To HTTP Response Header Field)