This is a draft that could be altered, removed or replaced by other documents. It is not a recommendation approved by TO.
Conformance
As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.
The key words MAY, MUST, and SHOULD in this document
are to be interpreted as described in
BCP 14
[RFC2119] [RFC8174]
when, and only when, they appear in all
capitals, as shown here.
Abstract
This document is part of the Nederlandse API Strategie.
Resources in REST APIs are usually represented as a JSON or XML body.
Some payloads are not suitable for such a format, such as media files or bulk data exports.
This module provides rules for describing such files as a metadata resource and transferring them reliably using two patterns: pull and push.
1.1 Pull pattern
The sender (server) makes the file available at a dedicated URI (contentUri). The receiver (client) fetches the file with GET. Interrupted transfer can be resumed.
1.2 Push pattern
The sender (client) requests an upload URI from the receiver (server), using POST. Subsequently, the sender uploads the file to the provided URI (contentUri), using PUT.
2. Design rules
Summary
Design rules are technical rules, which should be tested automatically, and functional rules, which should be considered when designing and building the API.
/transfer/upload: Request an upload location, then use HTTP PUT to push
List of functional rules
/transfer/threshold: Agree on a size threshold above which this module applies
Rules
functional
/transfer/threshold: Agree on a size threshold above which this module applies
Statement
Sender and receiver MAY agree bilaterally on a payload size threshold above which this module applies. In lieu of such an agreement, this module SHOULD be applied for payloads over 20 MiB.
Rationale
Size limits depend on the infrastructure of the sender and receiver. Agreeing on a limit is therefore preferred. A default is provided to not force a negotiation for each integration.
A large file MUST be described as a JSON metadata resource containing: fileName, contentType, size (bytes), and contentUri. It MAY also contain createdAt and expiresAt.
When receiving a file, the API MUST verify that the size of the received content matches size. A mismatch MUST be treated as an unsuccessful transfer.
Rationale
A receiver needs to know what it is about to transfer so it can commit storage, bandwidth, and processing time. The optional availability window avoids transfer attempts for content that has been removed.
How to test
Retrieve the metadata resource and validate the presence of the required fields.
Transfer content to a push contentUri with an incorrect size value and verify it is rejected.
technical
/transfer/range: Support HTTP range requests when retrieving large files
Statement
When presenting a file for retrieval, the API MUST support range requests with the range unit set to bytes, as defined in [RFC9110] (section 14). This MUST be indicated with Accept-Ranges: bytes in the response header.
The response header MUST also include a strong entity-tag (ETag), so the receiver can reliably resume the transfer with If-Range.
Note
This rule only applies to retrieval, even though RFC 9110 mentions partial PUT via Content-Range. According to the Internet-Draft Resumable Uploads for HTTP: "there are caveats that affect its deployability". The aforementioned draft offers a method for resumable uploads. It is a work in progress and therefore not included as a rule in this module.
Rationale
Large transfers take longer and are therefore more likely to be interrupted. With range support, the transfer can be resumed without having the previously received data go to waste.
How to test
Send a GET request without a Range header.
Validate the response is 200 and contains the Accept-Ranges: bytes and ETag headers.
Send a GET request with a Range header.
Validate the response is 206 and with a correct Content-Range header.
Send a GET request with a Range header and an If-Range that does not match the ETag.
Requests and responses transferring large files (to or from contentUri) MUST include Content-Digest [RFC9530] with sha-256 or stronger.
Responses transferring large files MUST also include Repr-Digest [RFC9530].
When receiving a file, the API MUST compute the digest of the received content and verify it matches Content-Digest. A mismatch MUST be treated as an unsuccessful transfer.
Note
A response may contain only a fragment of the file (see /transfer/range), in which case Content-Digest covers only the content of that message. Repr-Digest covers the complete file, regardless of range used.
Rationale
Large transfers are more likely to be corrupted. Validating the arrived content alleviates this concern.
If the file is retrieved in parts, the receiver needs a digest that covers the whole file instead of a part.
How to test
Verify the response of a pull contentUri contains Content-Digest and Repr-Digest using sha-256 or stronger.
Transfer content to a push contentUri with an incorrect Content-Digest value and verify it is rejected.
technical
/transfer/upload: Request an upload location, then use HTTP PUT to push
Statement
An API accepting large files MUST publish an endpoint to which the sender can POST a partial metadata resource containing fileName, contentType, size. Upon acceptance, the API MUST respond with 201 Created with the Location header pointing to the metadata resource and a body with the completed metadata resource.
The API MUST support HTTP PUT requests to the contentUri.
Rationale
The receiver should be the one to decide the location of a large file sent to their server. Through POST the sender can request a location in an automated fashion. This also gives the receiver a chance to refuse the transfer based on content type or size before time and bandwidth are wasted.
Large transfers take longer and are therefore more likely to be interrupted. Resending the file should not create another file, but overwrite the previous attempt. That's why PUT with its idempotent nature is chosen over POST for the file transfer.
How to test
Validate contentUri accepts PUT.
Verify a 201 response is defined including a Location header and a body in the metadata resource format.
