Minifying content and using gzip compression returns empty response.

[rniedosmialek]

• With issues:
  • Use the search tool before opening a new issue.
  • Please provide source code and commit sha if you found a bug.
  • Review existing issues and provide feedback or react to them.

Description

I am trying to minify the HTML/JS/CSS output before finally being gzipped using GIN's internal compression process. I have verified my middleware is working and outputting correctly for the minifying process. When I enable compression (which is the step after minifying), it completes the response with the proper coding type, but the body of the request is empty. If I turn it off, I once again see my minified output. Any help understanding what I am doing wrong or if this is a bug is appreciated.

How to reproduce

package middleware

import (
	"bytes"
	"log"
	"regexp"

	"github.com/gin-gonic/gin"
	"github.com/tdewolff/minify/v2"
	"github.com/tdewolff/minify/v2/css"
	"github.com/tdewolff/minify/v2/html"
	"github.com/tdewolff/minify/v2/js"
	"github.com/tdewolff/minify/v2/json"
	"github.com/tdewolff/minify/v2/svg"
	"github.com/tdewolff/minify/v2/xml"
)

func MinifyHTML() gin.HandlerFunc {
	return func(c *gin.Context) {
		log.Println("MinifyHTML middleware invoked")

		// Intercept the response with a buffer
		var buffer bytes.Buffer
		writer := &captureWriter{
			ResponseWriter: c.Writer,
			Buffer:         &buffer,
		}
		c.Writer = writer

		// Process the request
		c.Next()

		// Check if the response is HTML
		contentType := c.Writer.Header().Get("Content-Type")
		log.Printf("MinifyHTML: Content-Type: %s", contentType)

		if contentType != "text/html; charset=utf-8" {
			log.Println("MinifyHTML: Skipping non-HTML response")
			writer.FlushBufferToResponse() // Write original response to the client
			return
		}

		// Minify the HTML
		log.Println("MinifyHTML: Minifying HTML response")
		m := minify.New()
		m.AddFunc("text/css", css.Minify)
		m.AddFunc("text/html", html.Minify)
		m.AddFunc("image/svg+xml", svg.Minify)
		m.AddFuncRegexp(regexp.MustCompile("^(application|text)/(x-)?(java|ecma)script$"), js.Minify)
		m.AddFuncRegexp(regexp.MustCompile("[/+]json$"), json.Minify)
		m.AddFuncRegexp(regexp.MustCompile("[/+]xml$"), xml.Minify)

		minified, err := m.String("text/html", buffer.String())
		if err != nil {
			log.Printf("MinifyHTML: Minification failed: %v", err)
			writer.FlushBufferToResponse() // Write original response to the client
			return
		}

		// Replace the buffered content with the minified content
		log.Printf("MinifyHTML: Writing minified content of size: %d bytes", len(minified))
		writer.ReplaceBuffer([]byte(minified))
	}
}

// captureWriter intercepts and buffers the response
type captureWriter struct {
	gin.ResponseWriter
	Buffer *bytes.Buffer
}

func (w *captureWriter) Write(data []byte) (int, error) {
	return w.Buffer.Write(data)
}

func (w *captureWriter) FlushBufferToResponse() {
	if w.Buffer.Len() > 0 {
		_, err := w.ResponseWriter.Write(w.Buffer.Bytes())
		if err != nil {
			log.Printf("captureWriter: Failed to flush buffer: %v", err)
		}
	}
}

func (w *captureWriter) ReplaceBuffer(data []byte) {
	w.Buffer.Reset()
	w.Buffer.Write(data)
	_, err := w.ResponseWriter.Write(w.Buffer.Bytes())
	if err != nil {
		log.Printf("captureWriter: Failed to write replaced buffer: %v", err)
	}
}

//Middleware setup:
r.Use(middleware.MinifyHTML())
r.Use(gzip.Gzip(gzip.DefaultCompression))

Expectations

$ curl -v https://localhost:8080/ -k
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Texas; L=Lufkin; O=Magniedo; OU=IT; CN=www.ihadnoclue.com; emailAddress=Magniedo@proton.me
*  start date: Sep 24 15:22:14 2024 GMT
*  expire date: Sep 22 15:22:14 2034 GMT
*  issuer: C=US; ST=Texas; L=Lufkin; O=Magniedo; OU=IT; CN=www.ihadnoclue.com; emailAddress=Magniedo@proton.me
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x57fec623eeb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: localhost:8080
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< content-security-policy: default-src * data: blob: 'unsafe-inline' 'unsafe-eval'; base-uri 'self'; script-src 'unsafe-inline' 'unsafe-eval' *; style-src 'self' 'unsafe-inline' *; img-src * data:; connect-src *;font-src * data:;object-src 'none';media-src *;frame-src *;
< content-type: text/html; charset=utf-8
< referrer-policy: same-origin
< set-cookie: lang=en; Path=/; Max-Age=31536000; HttpOnly; Secure
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-robots-tag: all
< x-xss-protection: 1; mode=block
< date: Sat, 11 Jan 2025 14:56:14 GMT
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
<!doctype html><html lang=en><meta charset=utf-8><meta content="width=device-width,initial-scale=1" name=viewport>...</script>(base)

Actual result

$ curl -v https://localhost:8080/ --compressed -k
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Texas; L=Lufkin; O=Magniedo; OU=IT; CN=www.ihadnoclue.com; emailAddress=Magniedo@proton.me
*  start date: Sep 24 15:22:14 2024 GMT
*  expire date: Sep 22 15:22:14 2034 GMT
*  issuer: C=US; ST=Texas; L=Lufkin; O=Magniedo; OU=IT; CN=www.ihadnoclue.com; emailAddress=Magniedo@proton.me
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x64d606073eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: localhost:8080
> user-agent: curl/7.81.0
> accept: */*
> accept-encoding: deflate, gzip, br, zstd
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< content-encoding: gzip
< content-security-policy: default-src * data: blob: 'unsafe-inline' 'unsafe-eval'; base-uri 'self'; script-src 'unsafe-inline' 'unsafe-eval' *; style-src 'self' 'unsafe-inline' *; img-src * data:; connect-src *;font-src * data:;object-src 'none';media-src *;frame-src *;
< content-type: text/html; charset=utf-8
< referrer-policy: same-origin
< set-cookie: lang=en; Path=/; Max-Age=31536000; HttpOnly; Secure
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< vary: Accept-Encoding
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-robots-tag: all
< x-xss-protection: 1; mode=block
< date: Sat, 11 Jan 2025 14:56:44 GMT
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Error while processing content unencoding: invalid distance too far back
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* stopped the pause stream!
* Connection #0 to host localhost left intact
curl: (61) Error while processing content unencoding: invalid distance too far back

Environment

• go version: 1.23.2
• gin version (or commit ref): v1.10.0
• operating system: Ubuntu Jammy

[pscheid92]

Hello @rniedosmialek, I think the problem arises from a confusing middleware ordering. As the gzip compression and the minifying middleware process the outgoing response, meaning it works after the controller logic runs, the ordering is switched: The last middleware you register is the first to get called after the controller logic.

So, switching the order solved it for me (using your reproduction example):

r.Use(gzip.Gzip(gzip.DefaultCompression))
r.Use(middleware.MinifyHTML())

Could you verify and mark this issue as resolved if it helped?