#programming

Python

→ Create a cli executable

→ Use click

• use setuptools

→ Environments

→ virualenv

it is limited to the python version present on the host linux distribution. it handles the pip package manager.

→ anaconda

anaconda allows to get multiple version of python even if the host linux distribution does not. it handles both conda and pip package managers, but conda is the prefered way.

→ nix

nix proposes an alternate way of packaging python environments. It is not compatible with pip.

→ guix

it has less packages than nix and has the same drawbacks: python packages has to be packaged the guix way to be accessible.

→ pyenv/direnv

This activates the virtualenv when you go into the folder.

Installation:

curl https://pyenv.run | bash

# add to your bashrc
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

source ~/.bahsrc
curl -sfL https://direnv.net/install.sh | bash
eval "$(direnv hook bash)"

Then install the python version you want:

source ~/.bahsrc
pyenv install 3.8.11

Then in any python project:

echo "layout pyenv 3.8.11" > .envrc
direnv allow

→ Data

→ Pandas

→ Koalas

→ Map/Reduce

→ Dask

Dask peut cohabiter avec spark sur un cluster hadoop via dask-yarn. Il peut lire/ecrire sur hdfs des fichiers parquet/orc.

Le fonctionnement est largement inspiré de spark: dask s'appuie sur la résolution d'un DAG lazy, avec possibilité de caching des étapes.

La finalité de dask est de transformer des big-data en small-data pour pouvoir les traiter avec pandas.

→ API

→ Flask

→ Dashboard

→ Dash

→ Database

→ ZODB

ZODB uses “object traversal” to retrieve information from the persistance layer. It supports ACID transaction and is ideal for complex datamodel.

→ Compilation

→ numba

numba

→ cython

→ One liners

Read file content:

## Reading file contents into a list: one-liner
file_lines = [line.strip() for line in open(filename)]

→ Remote debugging fastapi on kubernetes with idea

requirements:

pydevd-pycharm~=232.10227.8

add endpoint:

@app.get("/debug")
def debug():
    pydevd_pycharm.settrace("localhost", port=4321, stdoutToServer=True, stderrToServer=True, suspend=False)
    return "connected!"

deploy your app on k8s w/ a deployment (here example-app)

attach a side container with ktunnel: Note: the app.kubernetes.io/instance must match example-app

ktunnel inject -v -n <namespace> deployment example-app 4321

• port forward example-app to localhost
• debug your app with idea
• call curl localhost:8000/debug -> return connected

→ Virtualenv + direnv + pyenv

first create a pyenv version. Then a virtualenv :

~/.pyenv/versions/3.8.18/bin/python3 -m venv /tmp/venv

.envrc

VIRTUAL_ENV=/tmp/venv
PATH=/tmp/venv/bin:$PATH

direnv allow