#programming
Python
→ Create a cli executable
→ Use click
→ Environments
→ virualenv
it is limited to the python version present on the host linux distribution. it handles the pip package manager.
→ anaconda
anaconda allows to get multiple version of python even if the host linux distribution does not. it handles both conda and pip package managers, but conda is the prefered way.
→ nix
nix proposes an alternate way of packaging python environments. It is not compatible with pip.
→ guix
it has less packages than nix and has the same drawbacks: python packages has to be packaged the guix way to be accessible.
→ pyenv/direnv
This activates the virtualenv when you go into the folder.
Installation:
curl https://pyenv.run | bash
# add to your bashrc
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
source ~/.bahsrc
curl -sfL https://direnv.net/install.sh | bash
eval "$(direnv hook bash)"
Then install the python version you want:
source ~/.bahsrc
pyenv install 3.8.11
Then in any python project:
echo "layout pyenv 3.8.11" > .envrc
direnv allow
→ Data
→ Pandas
→ Koalas
→ Map/Reduce
→ Dask
Dask peut cohabiter avec spark sur un cluster hadoop via dask-yarn. Il peut lire/ecrire sur hdfs des fichiers parquet/orc.
Le fonctionnement est largement inspiré de spark: dask s'appuie sur la résolution d'un DAG lazy, avec possibilité de caching des étapes.
La finalité de dask est de transformer des big-data en small-data pour pouvoir les traiter avec pandas.
→ API
→ Flask
→ Dashboard
→ Dash
→ Database
→ ZODB
ZODB uses “object traversal” to retrieve information from the persistance layer. It supports ACID transaction and is ideal for complex datamodel.
→ Compilation
→ numba
→ cython
→ One liners
Read file content:
## Reading file contents into a list: one-liner
file_lines = [line.strip() for line in open(filename)]
