Full request/response body logging in nginx

nginx.conf

http {
  log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

  lua_need_request_body on;

  set $resp_body "";
  body_filter_by_lua '
    local resp_body = string.sub(ngx.arg[1], 1, 1000)
    ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
    if ngx.arg[2] then
      ngx.var.resp_body = ngx.ctx.buffered
    end
  ';

}
 
server {
  listen 1.2.3.4;
 
  location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://127.0.0.1:3000;
    access_log  /var/log/nginx/server.log bodylog;
  }
}

I'm using nginx < 1.11.8 and escape=json is disable. Is there any other solution to decode json? Maybe something with lua

Looks like this isn't working anymore? Just gave it a try with a current distribution of OpenResty (it already has the lua module baked in) but even moving the set directive into the server block will just have the parser complain [emerg] unknown "resp_body" variable.

Update: I managed to get it to work after all. The problem seemed to originate from the missing nesting.
Once I moved the server block into the http block, it started to work.

I'm trying to get the response body but it always returns empty, I have tried both ways below:

  set $resp_body "";
#    body_filter_by_lua '
#                       local resp_body = string.sub(ngx.arg[1], 1, 1000)
#                       ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
#                       if ngx.arg[2] then
#                               ngx.var.resp_body = ngx.ctx.buffered
#                       end
#                       ';
   body_filter_by_lua 'ngx.log(ngx.ERR, ngx.var.resp_body, ngx.var.request_body )';

Any ideas how to capture the response body?

@jimmyss04

A complete example and works as of 7/25/2018 on Ubuntu 16.04. Some APIs have changed, and the concept of contexts and phases is confusing working with nginx/openresty.

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 768;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    gzip on;
    gzip_disable "msie6";

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" $request_time '
    '\n\n"$req_headers" \n"$req_body" \n>"$resp_body"';


    server {
        listen 80 default_server;
        access_log  /var/log/nginx/server.log bodylog;
     
        location / {
        }

        lua_need_request_body on;

        set $resp_body "";
        set $req_body "";
        set $req_headers "";

        client_body_buffer_size 16k;
        client_max_body_size 16k;

        rewrite_by_lua_block {
            local req_headers = "Headers: ";
            ngx.var.req_body = ngx.req.get_body_data();
            local h, err = ngx.req.get_headers()
            for k, v in pairs(h) do
                req_headers = req_headers .. k .. ": " .. v .. "\n";
            end

            ngx.var.req_headers = req_headers;
        }

        body_filter_by_lua '
        local resp_body = string.sub(ngx.arg[1], 1, 1000)
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
          ngx.var.resp_body = ngx.ctx.buffered
        end
        ';
     }
}

Does this require lua-nginx-module?

@simonecogno yes, it requires.

$req_headers

nginx: [emerg] unknown "req_headers" variable
the version of my Nginx on Ubuntu 16.04 is 1.10.3, installed with 'apt-get install nginx-extras'

Thanks Eugene, your working example helped me figure out what my issue was. Basically, I had a leftover content_by_lua_block { …} which was overwriting the response from the remote server

@morhekil I am running Red Hat Enterprise Linux release 8.8 (Ootpa)

`# nginx -t
nginx: [emerg] unknown directive "log_by_lua_block" in /etc/nginx/conf.d/microservice.conf:8
nginx: configuration file /etc/nginx/nginx.conf test failed

nginx -v

nginx version: nginx/1.24.0

nginx -V

nginx version: nginx/1.24.0
built by gcc 8.5.0 20210514 (Red Hat 8.5.0-4) (GCC)
built with OpenSSL 1.1.1k FIPS 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
#`

Hi,
First of all thanks for the code, it helped me a lot. Second, how would we tackle logging gzipped response bodies? I tried lua-zlib, but for some reason I can't make it work. Any suggestions? Thanks!