Nginx and Proxy 2019(1)Nginx Enable Lua and Parse JSON

Nginx and Proxy 2019(1)Nginx Enable Lua and Parse JSON

Set Up Nginx
> wget http://nginx.org/download/nginx-1.16.1.tar.gz
> tar zxvf nginx-1.16.1.tar.gz
> ./configure --prefix=/home/carl/tool/nginx-1.16.1
> make
> make install
> sudo ln -s /home/carl/tool/nginx-1.16.1 /opt/nginx-1.16.1
> sudo ln -s /opt/nginx-1.16.1 /opt/nginx

Start the Service
> sudo sbin/nginx

Visit the page
http://ubuntu-master/

Check this and enable lua
https://docs.nginx.com/nginx/admin-guide/dynamic-modules/lua/#
https://github.com/openresty/lua-nginx-module#installation
https://github.com/openresty/lua-nginx-module#nginx-compatibility

Install LuaJIT
http://luajit.org/install.html
> wget http://luajit.org/download/LuaJIT-2.0.5.tar.gz
> tar zxvf LuaJIT-2.0.5.tar.gz
> cd LuaJIT-2.0.5/
> make
> make install PREFIX=/home/carl/tool/luajit-2.0.5
> sudo ln -s /home/carl/tool/luajit-2.0.5 /opt/luajit-2.0.5
> sudo ln -s /opt/luajit-2.0.5 /opt/luajit

Add to PATH
PATH="/opt/luajit/bin:$PATH"

Check version
> luajit -v
LuaJIT 2.0.5 -- Copyright (C) 2005-2017 Mike Pall. http://luajit.org/

Install NDK (nix_devel_kit)
https://github.com/simplresty/ngx_devel_kit
Download the latest codes from here https://github.com/simplresty/ngx_devel_kit/tags
> wget https://github.com/simplresty/ngx_devel_kit/archive/v0.3.1.tar.gz
> tar zxvf v0.3.1.tar.gz
> mv ngx_devel_kit-0.3.1 ~/tool/
> sudo ln -s /home/carl/tool/ngx_devel_kit-0.3.1 /opt/ngx_devel_kit-0.3.1
> sudo ln -s /opt/ngx_devel_kit-0.3.1 /opt/ngx_devel_kit

Instal ngx_lua
https://github.com/openresty/lua-nginx-module
> wget https://github.com/openresty/lua-nginx-module/archive/v0.10.15.tar.gz
> tar zxvf v0.10.15.tar.gz
> mv lua-nginx-module-0.10.15 ~/tool/
> sudo ln -s /home/carl/tool/lua-nginx-module-0.10.15 /opt/lua-nginx-module-0.10.15
> sudo ln -s /opt/lua-nginx-module-0.10.15 /opt/lua-nginx-module

Compile and Install Nginx
Check version here https://github.com/openresty/lua-nginx-module#nginx-compatibility
> wget https://nginx.org/download/nginx-1.17.6.tar.gz
> tar zxvf nginx-1.17.6.tar.gz
> cd nginx-1.17.6/

> export LUAJIT_LIB=/opt/luajit/lib
> export LUAJIT_INC=/opt/luajit/include/luajit-2.0

>  ./configure --prefix=/home/carl/tool/nginx-1.17.6 --with-ld-opt="-Wl,-rpath,/opt/luajit/lib" --add-module=/opt/ngx_devel_kit --add-module=/opt/lua-nginx-module
> make
> make install
> sudo ln -s /home/carl/tool/nginx-1.17.6 /opt/nginx-1.17.6
> sudo ln -s /opt/nginx-1.17.6 /opt/nginx

> sudo sbin/nginx
nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)
nginx: [error] lua_load_resty_core failed to load the resty.core module from https://github.com/openresty/lua-resty-core; ensure you are using an OpenResty release from https://openresty.org/en/download.html (rc: 2, reason: module 'resty.core' not found:
no field package.preload['resty.core']
no file './resty/core.lua'

I may find the release from here LUAJIT2
https://github.com/openresty/luajit2/releases
> wget https://github.com/openresty/luajit2/archive/v2.1-20190912.tar.gz
> tar zxvf v2.1-20190912.tar.gz
> cd luajit2-2.1-20190912/
> make
> make install PREFIX=/home/carl/tool/luajit-2.1.0.3
> sudo ln -s /home/carl/tool/luajit-2.1.0.3 /opt/luajit-2.1.0.3
> sudo ln -s /opt/luajit-2.1.0.3 /opt/luajit

> export LUAJIT_LIB=/opt/luajit/lib
> export LUAJIT_INC=/opt/luajit/include/luajit-2.1

>  ./configure --prefix=/home/carl/tool/nginx-1.17.6 --with-ld-opt="-Wl,-rpath,/opt/luajit/lib" --add-module=/opt/ngx_devel_kit --add-module=/opt/lua-nginx-module
> make
> make install

Solution:
https://github.com/openresty/lua-nginx-module/issues/1533
https://superuser.com/questions/1426961/error-while-installing-lua-nginx-module
Add Confguration
> vi conf/nginx.conf
http {
    include       mime.types;
    default_type  application/octet-stream;
    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';
    #access_log  logs/access.log  main;
    lua_load_resty_core off;

> sudo sbin/nginx


Check my implementation
https://github.com/openresty/lua-nginx-module#rewrite_by_lua_file
https://www.cnblogs.com/tinywan/p/6526191.html

        location /DWH {
            rewrite_by_lua_file /opt/nginx/lua/DWH_proxy.lua;
        }

> cat lua/DWH_proxy.lua
ngx.log(ngx.ERR, "hello, lua working");
ngx.req.set_uri("/",true);

URL rewrite
There are 2 commands to support that. ngx.exec and ngx.redirect
Try this library
https://github.com/ledgetech/lua-resty-http
> wget https://github.com/ledgetech/lua-resty-http/archive/v0.15.tar.gz
> cp /home/carl/install/lua-resty-http-0.15/lib/resty/* /opt/nginx/lua/

It does not work like that, it load the dependencies like this
2019/11/25 22:49:04 [error] 28551#0: *1 lua entry thread aborted: runtime error: /opt/nginx/lua/DWH_proxy.lua:2: module 'resty.http' not found:
no field package.preload['resty.http']
no file './resty/http.lua'
no file '/usr/local/share/luajit-2.1.0-beta3/resty/http.lua'
no file '/usr/local/share/lua/5.1/resty/http.lua'
no file '/usr/local/share/lua/5.1/resty/http/init.lua'
no file './resty/http.so'
no file '/usr/local/lib/lua/5.1/resty/http.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file './resty.so'
no file '/usr/local/lib/lua/5.1/resty.so'
no file '/usr/local/lib/lua/5.1/loadall.so'

Solution:
Install and Setup luarocks
Check https://luarocks.org/
https://luarocks.org/#quick-start

> wget https://luarocks.org/releases/luarocks-3.2.0.tar.gz
> tar zxvf luarocks-3.2.0.tar.gz
> cd luarocks-3.2.0/

> ./configure --prefix=/opt/luajit --with-lua=/opt/luajit --lua-suffix=jit-2.1.0-beta3 --with-lua-include=/opt/luajit/include/luajit-2.1

> make build
> make install

> luarocks --version
/opt/luajit/bin/luarocks 3.2.0
LuaRocks main command-line interface

Install package
> luarocks install md5

> luarocks install lua-resty-http
Installing https://luarocks.org/lua-resty-http-0.15-0.src.rock
lua-resty-http 0.15-0 is now installed in /opt/luajit (license: 2-clause BSD)

Stop the nginx
> sudo sbin/nginx -s stop

Error load package
no file '/usr/local/share/luajit-2.1.0-beta3/luarocks/loader.lua'
Looking for  /opt/luajit/share/luajit-2.1.0-beta3

Reinstall that
> ./configure --prefix=/home/carl/tool/luarocks-3.2.0
> make
> make install
> sudo ln -s /home/carl/tool/luarocks-3.2.0 /opt/luarocks-3.2.0
> sudo ln -s /opt/luarocks-3.2.0 /opt/luarocks
Add to Path
PATH="/opt/luarocks/bin:$PATH"

> luarocks --version
/opt/luarocks/bin/luarocks 3.2.0

Add this to make it working
http {
    lua_package_path 'lua/?.lua;/opt/luarocks/share/lua/5.1/?.lua;;';

My testing LUA file works fine
> cat lua/DWH_proxy.lua
ngx.log(ngx.ERR, "hello, lua working");
local http = require "resty.http";
local httpc = http.new();
local res, err = httpc:request_uri("http://www.baidu.com",{
   method="GET"
});
if res.status == ngx.HTTP_OK then
       ngx.exec("/");
else
       ngx.exit(res.status);
end

This seems working fine for me
        location /DWH2 {
            resolver 8.8.8.8 8.8.4.4;
            set $proxy "";
            set $proxy_uri "";
            rewrite_by_lua '
                ngx.log(ngx.ERR, "hello, lua22 working");
                local http = require "resty.http";
                local httpc = http.new();
                local res, err = httpc:request_uri("http://www.baidu.com",{
                        method="GET"
                });
                if res.status == ngx.HTTP_OK then
                        ngx.var.proxy = "stagingdwhapi.sillycatcloudbeta.com:9000";
                        ngx.var.proxy_uri = "/api/1.0/accounts/01616dae-f6d6-4eda-b818-6991a9144685/streams?dateEnd=2019-11-22&dateStart=2019-11-08&filter=list";
                else
                        ngx.status=ngx.HTTP_UNAUTHORIZED;
                        ngx.exit(ngx.status)
                end
            ';
            proxy_pass http://$proxy$proxy_uri;
        }

Download the exactly the same version and compile with SSL
> wget https://nginx.org/download/nginx-1.14.0.tar.gz

> export LUAJIT_LIB=/opt/luajit/lib
> export LUAJIT_INC=/opt/luajit/include/luajit-2.1

Add Open SSL Library
> sudo apt-get install -y libssl-dev

>  ./configure --prefix=/home/carl/tool/nginx-1.14.2 --with-ld-opt="-Wl,-rpath,/opt/luajit/lib" --add-module=/opt/ngx_devel_kit --add-module=/opt/lua-nginx-module --with-http_ssl_module
> make
> make install

1.14.0 Not make well there.
Change to use 1.14.2, works well.

My previous working copy
cat /opt/nginx-1.17.6/conf/nginx.conf
#user  nobody;
worker_processes  1;
error_log  logs/error.log  notice;
#pid        logs/nginx.pid;
events {
    worker_connections  1024;
}
http {
    lua_package_path 'lua/?.lua;/opt/luarocks/share/lua/5.1/?.lua;;';
    include       mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  logs/access.log  main;
    lua_load_resty_core off;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  localhost;
        location /DWH {
    resolver 8.8.8.8 8.8.4.4;
            set $proxy "";
            set $proxy_uri "";
            rewrite_by_lua '
               ngx.log(ngx.ERR, "lua working start---------");
local h = ngx.req.get_headers()
                for k, v in pairs(h) do
                    if k == "cookie" then
                    ngx.log(ngx.ERR, k,": ",v);
                    end
                end
local http = require "resty.http";
local httpc = http.new();
local res, err = httpc:request_uri("https://login.sillycatcloudbeta.com/ls/validate",{
   ssl_verify=true,
method="GET",
                        headers = h
});
                ngx.log(ngx.ERR, res);
if res.status == ngx.HTTP_OK then
       ngx.var.proxy = "stagingdwhapi.sillycatcloudbeta.com:9000";
        ngx.var.proxy_uri = "/api/1.0/accounts/01616dae-f6d6-4eda-b818-6991a9144685/streams?dateEnd=2019-11-22&dateStart=2019-11-08&filter=list";
else
       ngx.status=ngx.HTTP_UNAUTHORIZED;
            ngx.exit(ngx.status)
end
            ';
            proxy_pass http://$proxy$proxy_uri;
        }
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Install cjson to parse the JSON response
> luarocks install lua-cjson

The installation is not working
https://www.kyne.com.au/~mark/software/lua-cjson-manual.html
> wget https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz
> tar zxvf lua-cjson-2.1.0.tar.gz

This is a working copy
https://github.com/jiyinyiyong/json-lua


References:
https://github.com/Ermlab/nginx-lua-proxy
https://stackoverflow.com/questions/22788236/how-can-i-manipulate-the-json-body-of-a-post-request-using-nginx-and-lua
https://openresty-reference.readthedocs.io/en/latest/Directives/

https://serverfault.com/questions/946228/nginx-openresty-how-to-reverse-proxy-a-call-to-2-different-servers-based-bas
https://yos.io/2016/01/28/building-an-api-gateway-with-lua-and-nginx/
https://openresty-reference.readthedocs.io/en/latest/Lua_Nginx_API/
https://github.com/upyun/slardar/issues/19