Chunked encoding and content-length header

Question

Is it possible to set the content-length header and also use chunked transfer encoding? and does doing so solve the problem of not knowing the length of the response at the client side when using chunked?

the scenario I'm thinking about is when you have a large file to transfer and there's no problem in determining its size, but it's too large to be buffered completely. (If you're not using chunked, then the whole response must get buffered first? Right??)

thanks.

Answer 1

1. No:

"Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding. If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored." (RFC 2616, Section 4.4)

2. And no, you can use Content-Length and stream; the protocol doesn't constrain how your implementation works.

Answer 2

Well, you can always send a header stating the size of the file. Something like response.addHeader("File-Size","size of the file");
And ignore the Content-Length header.

The client implementation has to be tweaked to read this value, but hey you can achieve both the things you want :)

Answer 3

You have to use either Content-Length or chunking, but not both.

If you know the length in advance, you can use Content-Length instead of chunking even if you generate the content on the fly and never have it all at once in your buffer.

However, you should not do that if the data is really large because a proxy might not be able to handle it. For large data, chunking is safer.

Answer 4

This headers can be cause of Postman Parse Error:

"Content-Length" and "Transfer-Encoding" can't be present in the response headers together.

Using parametrized ResponseEntity<?> except raw ResponseEntity in controller can fixed the issue.

Answer 5

The question asks:

Is it possible to set the content-length header and also use chunked transfer encoding?

The RFC HTTP/1.1 spec, quoted in Julian's answer, says:

Messages MUST NOT include both a Content-Length header field and a non-identity transfer-coding.

There is an important difference between what's possible, and what's allowed by a protocol. It is certainly possible, for example, for you to write your own HTTP/1.1 client which sends malformed messages with both headers. You would be violating the HTTP/1.1 spec in doing so, and so you'd imagine some alarm bells would go off and a bunch of Internet police would burst into your house and say, "Stop, arrest that client!" But that doesn't happen, of course. Your request will get sent to wherever it's going.

OK, so you can send a malformed message. So what? Surely on the receiving end, the server will detect the HTTP/1.1 protocol client-side violation, vanquish your malformed request, and serve you back a stern 400 response telling you that you are due in court the following Monday for violating the protocol. But no, actually, that probably won't happen. Of course, it's beyond the scope of HTTP/1.1 to prescribe what happens to misbehaving clients; i.e. while the HTTP/1.1 protocol is analogous to the "law", there is nothing in HTTP/1.1 analogous to the judicial system.

The best that the HTTP/1.1 protocol can do is dictate how a server must act/respond in the case of receiving such a malformed request. However, it's quite lenient in this case. In particular, the server does not have to reject such malformed requests. In fact, in such a scenario, the rule is:

If the message does include a non-identity transfer-coding, the Content-Length MUST be ignored.

Unfortunately, though, some HTTP servers will violate that part of the HTTP/1.1 protocol and will actually give precedence to the Content-Length header, if both headers are present. This can cause a serious problem, if the message visits two servers in sequence in the same system and they disagree about where one HTTP message ends and the next one starts. It leaves the system vulnerable to HTTP Desync attacks a.k.a. Request Smuggling.