#metabase #superset #nginx #postgres #graphana #neo4j

shiny proxy

Debug mode:

logging:
  requestdump: true
  level:
    root: TRACE

→ sp server client call

we might fetch the user groups from the client and generate a form on the fly. we need a userInfo endpoint. Or we xan enrich the prepared map here.

→ Superset

Overall superset does not support natively base url, so it's a pain to integrate with SP.

A good solution might be considered, since it allows sharing public dashboards to someone not logged in, and we have two docker-compose starters:

→ Metabase

→ disable CSP

  1. It is very easy to build a custom metabase and removing that security
  2. Leverage nginx reverse proxy to hide the CSP headers

The second option looks better:

→ skip login

One idea is to call the metabase login api and create a cookie, transfered by nginx.

OpenResty is an nginx distribution which includes the LuaJIT interpreter for Lua scripts

FROM openresty/openresty:buster-fat
RUN opm install ledgetech/lua-resty-http thapakazi/lua-resty-cookie
COPY default.conf /etc/nginx/conf.d/
COPY *.lua /usr/local/openresty/nginx/
COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf

server {
  listen 8080;
  server_name your.metabase.domain;

  location / {
    access_by_lua_file gen_token.lua;
    proxy_pass http://127.0.0.1:3000;
  }

}

local cjson = require("cjson")
local httpc = require("resty.http").new()
local ck = require("resty.cookie")

local cookie, err = ck:new()
if not cookie then
	ngx.log(ngx.ERR, err)
	return
end

local field, err = cookie:get("metabase.SESSION")
if not field then
	local res, err = httpc:request_uri("http://127.0.0.1:3000/api/session", {
		method = "POST",
		body = cjson.encode({
			username = os.getenv("METABASE_USERNAME"),
			password = os.getenv("METABASE_PASSWORD"),
		}),
		headers = {
			["Content-Type"] = "application/json",
		},
	})
	if not res then
		ngx.log(ngx.ERR, "request failed:", err)
		return
	end
	local data = cjson.decode(res.body)
	local ok, err = cookie:set({
		key = "metabase.SESSION",
		value = data["id"],
		path = "/",
		domain = ngx.var.host,
		httponly = true,
		-- max_age = 1209600,
		samesite = "Lax",
	})
	if not ok then
		ngx.log(ngx.ERR, err)
		return
	end
end

→ enable concurrent connections

Sounds like we could run multiple instances of MT having the same db. For example sharing the db in the team folder, so that team members share their dashboards.

→ resources management

→ volume access

Goal:

  1. In the user directory, files folders are rw across applications
  2. In the team directory, files and folders are rw across applications and members of the team
FROM ubuntu:22.04
RUN mkdir -p '/foo' ; chown  '1001':'1001' '/foo'
# then
docker  build -t nico:latest .
docker run -it --rm  --user=1001:1001 --mount='source=volumeName,target=/foo,readonly=false' nico:latest ls -alrth /|grep foo
drwxr-xr-x   2 1001 1001 4.0K Sep 10 22:26 foo

→ LDH folder design

This allows both research projects on HDS infra and courses/misc projects to work with the same design.

Three level of groups:

  1. Project: access to personal folder related to their projects plus a shared folder for all members
  2. Project-Admin: same as above plus the project-personal folders of every members
  3. Admin: same as above plus all personal and shared folders

The three structure can work that way:

Admin would mount:

Admin-project1 would mount:

User2 would mount:

Notes:

  1. If a user has no project, it has no mount point.
  2. Access to the apps could be require a a sandbox project
  3. For admin, the access is read only: it avoids mistakes

The volume expression can work that way:

- #{listToCsv('./data/<repl>/' + userId + ':/root/<repl>/personal', projects}
- #{listToCsv('./data/<repl>-shared/:/root/<repl>/shared', projects}
# for admin project
- #{listToCsv('./data/<repl>/:/root/<repl>/users:ro', projects}
# for admin
- #{listToCsv('./data/:/root/projects:ro', projects}

→ grafana

We can provide a graphana instance per category: server metrics, logs, postgres metrics... and set a hme dashboard in anonymous mode.

depening on the user role, they would have access to more or less container info.

→ access

User acces

Project admin access

Admin access

→ postgres metrics

→ Onlyoffice

→ Databases

→ Neo4j

→ Postgres

→ extensions

→ access management

Needs:

Proposal:

How to maintain the db:

→ mongodb

→ rocksdb

→ tantivy

→ neo4j

→ Jupyter

→ Custom image

→ Accessing postgres

→ Extensions

→ vscode

→ rstudio

→ databases

So it is possible to have predefined connections, that can be navigated from the connection panel.

  1. write this into /etc/rstudio/connections/Postgres\ parisni.R 
    library(connections)
library(RPostgres)
con <- connection_open(RPostgres::Postgres(), dbname = "postgres", host = "postgres", port = 5432, user = "parisni", password = "pwd")

→ airflow

→ Installation

we have two options:

  1. run one webserver per user within SP and all the other services in compose. BTW use remote_user auth + enable iframe
  2. run all services in compose, and provide a link within SP. BTW configure airflow with the user's auth (keycloak, ldap...)
  3. same as 2. but starts an nginx in SP to redirect to the unique webserver. BTW uses a identity proxy to log the user

option 1. consumes more resources since there is a webserver per user, but the authent part is managed by SP. Starting the webserver is about 1min option 2. shares one webserver among all users, but the auth part is way more complicated to setup option 3. has all advantages

In all case we will need to register users within airflow

Ideas:

Proxy_redirect to replace with https plus some sub_filters did fix the UI.

→ Usage

React ?

This page was last modified: