Skip to content

Varnish with brotli -> missing ESI handling/filtering

I’ve compiled brotli modules for nginx and added following (from https://www.getpagespeed.com/server-setup/varnish/varnish-brotli-done-right) to varnish default.vcl:

sub vcl_recv {
    if(req.http.Accept-Encoding ~ "br" && req.url !~
            ".(jpe?g|png|webp|gif|gz|mp3|mov|avi|mpg|mp4|swf|wmf)$") {
        set req.http.X-brotli = "true";
    }
}

# The data on which the hashing will take place
sub vcl_hash {
    if(req.http.X-brotli == "true" && req.http.X-brotli-unhash != "true") {
        hash_data("brotli");
    }
}

sub vcl_backend_fetch {
    if(bereq.http.X-brotli == "true") {
        set bereq.http.Accept-Encoding = "br";
        unset bereq.http.X-brotli;
    }
}

sub vcl_purge {
    # repeat purge for brotli or gzip object 
    # (force hash/no hash on "brotli" while doing another purge)
    # set Accept-Encoding: gzip so that we don't get brotli-encoded response upon purge
    if (req.url !~ ".(jpg|png|gif|gz|mp3|mov|avi|mpg|mp4|swf|wmf)$" && 
            !req.http.X-brotli-unhash) {
        if (req.http.X-brotli == "true") {
            set req.http.X-brotli-unhash = "true";
            set req.http.Accept-Encoding = "gzip";
        } else {
            set req.http.X-brotli-unhash = "false";
            set req.http.Accept-Encoding = "br";
        }        
        return (restart);
    } 
}

All would be perfect (loading speed increased significantly) but top menu isn’t displayed. Looks like ESI handling has been changed.
ESI itself is enabled in Varnish and without brotli code there are no problems.

[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd 
          -a localhost:6081 
          -a localhost:8443,PROXY 
          -S /etc/varnish/secret 
          -f /etc/varnish/default.vcl 
          -s Cache=malloc,2G 
          -s Transient=malloc,512m 
          -p http_req_size=256k 
          -p http_resp_hdr_len=128k 
          -p http_resp_size=512k 
          -p workspace_backend=512k 
          -p workspace_client=256k 
          -p feature=+esi_ignore_other_elements 
          -p feature=+esi_disable_xml_check 
          -p feature=+esi_ignore_https 
          -p thread_pool_min=200 
          -p thread_pool_max=2000 
          -p thread_pool_add_delay=2

With brotli code, ESI is gone from varnish filter (and esi block is just sent over http and visible in browser dev tools)…

-   VCL_return     deliver
-   Timestamp      Process: 1667168951.732238 0.000115 0.000115
-   Filters        
-   RespHeader     Content-Length: 141996
-   RespHeader     Connection: close
-   Timestamp      Resp: 1667168951.732649 0.000526 0.000411
-   ReqAcct        2492 0 2492 5906 141996 147902
-   End            

Without brotli code:

--  VCL_return     deliver
--  Timestamp      Process: 1667090732.937994 1.547312 0.000076
--  Filters         esi
--  Storage        malloc Transient
--  Fetch_Body     2 chunked -
--  BackendClose   47 default recycle
--  Timestamp      BerespBody: 1667090733.003310 1.612627 0.065315
--  Length         124527
--  BereqAcct      1811 0 1811 6828 124527 131355
--  End            

Nginx proxy settings for SSL termination:

        proxy_pass http://127.0.0.1:6081;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Ssl-Offloaded "1";
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Port 443;
        proxy_http_version 1.1;
        proxy_buffer_size                   64k;
        proxy_buffers                       32 16k;
        proxy_busy_buffers_size             64k;

and Varnish backend:

server {
   server_name localhost example.com;
   listen 127.0.0.1:8080;
   set $MAGE_ROOT <magento_dir>;
   access_log off;
   error_log /var/log/nginx/error_mag_var.log warn;
   set_real_ip_from 127.0.0.1;
   real_ip_header X-Forwarded-For;
   real_ip_recursive on;
  include <magento_dir>/nginx.mag.conf;
}

I’m not very familiar with Varnish. Is it possible to restore ESI handling with brotli code in default.vcl?