Software Lobotomy — blog

Concourse

2023-09-11T14:42:00+00:00

Concourse

Commands

# create/update a pipeline
fly -t tutorial set-pipeline -c pipeline.yml -p hello-world 

# trigger a job
fly -t tutorial trigger-job -j  hello-world/job-hello-world

# ask for pipeline builds
fly -t tutorial builds
id  name                           status     start                     end                       duration  team  created by
60  hello-world/job-hello-world/4  succeeded  2023-08-28@16:34:11+0200  2023-08-28@16:34:14+0200  3s        main  admin
59  hello-world/job-hello-world/3  succeeded  2023-08-28@16:26:01+0200  2023-08-28@16:26:04+0200  3s        main  admin
15  hello-world/job-hello-world/2  failed     2023-08-28@15:34:40+0200  2023-08-28@15:34:43+0200  3s        main  admin
13  hello-world/job-hello-world/1  succeeded  2023-08-28@14:50:49+0200  2023-08-28@14:50:52+0200  3s        main  admin

# watch a pipeline / job / build-number
fly -t tutorial watch -j hello-world/job-hello-world --build 4

# destroy pipelines
fly -t tutorial destroy-pipeline -p hello-world

Parameter> Values in params sections in turn become environment variables within the task:

React ?

This page was last modified: 2023-09-11 14:42

Hive Reflexions

2021-08-20T15:30:00+00:00

Date:2018-05-10

Estimated Time:less than a minute

Hive Reflexions

Hive has several advantages over other Datawarehouse solutions.

it is java based. This allows it to take advantage of UDF. The best and the moste complicate way to write them as generic UDFs. The other important aspect is to make them blessed UDF by mean they are global and accessible to any user. Generic UDF can produce any hive type from integer to complexe map structure.
it can read/write into hbase and take advantage of a key-value store.
it is ACID compliant, and provide both insert/update. However, this new feature is not well integrated with other tools such apache spark or facebook presto.
it supports multiple level of performance tunning such partitionning, bucketting, bloom filters, predicate pushdown.

Still hive has several disavantages:

lack or support for sequences. The workaround right now is to either use a UUID - this has several drawbacks such performances/storage issue. A second approach is to use the sequence_nb + (row_number() over()) function where sequence_nb is maintained into a table - the count of the inserted rows need to be added to the sequence. The last solution is a long, but it cannot be used concurrently. The solution cannot be used in production.

hive configuration tips

Hive can have OOM. Then this parameter might help - set hive.exec.orc.memory.pool = 1.0

Predicate Push Down is better when INSERT STMT use a SORT BY "orc.bloom.filter.columns"="col1,col2"

It is also possible to update the bloom filter by running: ANALYZE TABLE Table1 COMPUTE STATISTICS FOR COLUMNS;

show progress bar in beeline

React ?

This page was last modified: 2021-08-20 15:30

PostgreSQL Reflexions

2021-08-20T15:30:00+00:00

Date:2018-05-13

Estimated Time:less than a minute

PostgreSQL Reflexions

After 4 years of postgresql management, I d'like to relate my experience.

Postgreql is incredible:

its community is wonderful, the mailing list is very active and structured
the recent years have been very productive for postgres, with many analytics improvements and perspectives
it has one of the best support for transactions, and for example deal with DDL in transaction
it's stable, we never had any crash in 4 years, or need to reboot the server
it manage bot structured and unstructured data
free text indexes and features are very advanced

However, sadly postgresql has some drawnbacks:

it does not deal great with large volumne when comes analytics
because it may be used for all batch ingesting, batch exporting, OLTP, dumps, analytics, the overall performances might be affected
it does not scale horizontaly
it has no fail over, or this lead to complex setup

Complementing postgresql with hadoop:

For this reason, it has been decided to externalize some responsibilities such batch exporting and analytics to an other technology that scale horizontally: hadoop.

The good point is synchronizing data from postgresql to hive daily is easy thanks to incremental sqoop import on hdfs, based on indexed timestamps in postgreql. This allows pushing the data from postgresql to hive ready to analyse. The two tools interface well where hive does not handle sequence, or triggers, and postgresql does not manage huge data historisation, and columnar aggregation.

While postgreqsl does not handle well aggregations on bilions rows, it's children greenplum is supposed to.: Let's envisage a greenplum post soon.

React ?

This page was last modified: 2021-08-20 15:30

Hive Merge

2021-08-20T15:30:00+00:00

Date:2018-05-13

Estimated Time:2 minutes

Hive Merge

Hive introduced ACID statement: INSERT, UPDATE, DELETE & MERGE. It turns-out HIVE 1.2.x is not optimised enough to support MERGE efficiently. An in depth documentation can be found there

The code below :

is idempotent in someway. Suppose for some reason, the merge insert statement fails. Then running again all statements won't be an issue since no update at all will append again. Same case if the update statement fails.
Simple version: makes the assumption that any data both present in the staging table and the target table needs to be updated.
SCD1 and SCD2 have slighly difference that are mentionned if existing.
SCD2 has been tested (upsert version) on a large target table ( 1.3 Bilion rows * 100 columns) and a source table of 100 Milion rows. As a result, the total runtime was les than 3 hours. (50% resource on a 5 computer hadoop cluster).

0) Staging step

Some data was put on hdfs in the <stg_table> to be merged with a target table. This might be done by scoop incrementally. Also, this might be the whole table. All the steps below will work the same in both case.

1) Preparation step

The 3 table above are necessary for any of SCD1 or SCD2. They are respectively the transformation, insert, update candidate tables.

-- create transformation table
CREATE TEMPORARY TABLE <trf_table> LIKE <target_table>

-- create insert table
CREATE TEMPORARY TABLE <ins_table> LIKE <target_table>

-- create update table
CREATE TEMPORARY TABLE <upd_table> LIKE <target_table>

2) Transformation step

This step loads the trf table (transformed) from the stg table (staging). It adds the hash column, and is the place to add other transformations/columns if needed.

-- example without transformations
INSERT INTO <trf_table>
SELECT * , HASH(*) as hash
FROM <stg_table>;

-- example with transformations
-- t1 and t2 are new columns added
INSERT INTO <trf_table>
SELECT * , HASH(stg.*) as hash
FROM (SELECT * , t1(), t2()
      FROM <stg_table>) as stg;

3) Dispatching step

Simplified version:

-- variation when you already know that rows are all different
FROM <trf_table>
INSERT INTO <upd_table> SELECT * WHERE <maj_date> IS NOT NULL
INSERT INTO <ins_table> SELECT * WHERE <maj_date> IS NULL

Complete version:

-- feed both update & insert table
FROM (SELECT *
      FROM  <trf_table> trf
      JOIN <target_table> targ
      ON (
      targ.<end_date> IS NULL
      AND trf.<primary_key> = targ.<primary_key>
      )
      WHERE TRUE
      AND trf.hash <> targ.hash) as updateable
INSERT INTO TABLE <upd_table> SELECT * ;

-- feed the insert table with new rows
INSERT INTO <ins_table>
SELECT *
FROM <trf_table>
WHERE TRUE
AND <primary_key> NOT IN (
     SELECT <primary_key>
     FROM <target_table>
);

4.1) Upsert version

SCD 1:

Notice the delete is used in place of an update. The reason is an update cannot join an other table and thus apply updates where needed.

-- first delete update rows
DELETE FROM <target_table>
WHERE TRUE
AND <primary_key> IN (
     SELECT <primary_key>
     FROM <upd_table>);

-- second, insert all new rows
INSERT INTO <target_table>
SELECT *
FROM <ins_table>
UNION ALL
SELECT *
FROM <upd_table>;

SCD 2:

-- first update old rows
UPDATE <target_table>
SET <end_date> = current_date()
WHERE TRUE
AND <end_date> IS NULL
AND <primary_key> IN (
     SELECT <primary_key>
     FROM <upd_table>);

-- second, insert all new rows
INSERT INTO <target_table>
SELECT *
FROM <ins_table>
UNION ALL
SELECT *
FROM <upd_table>;

4.2) Merge version

SCD 1:

-- first update old rows
MERGE INTO <target_table>
USING
(
SELECT null AS join_key, upd.* FROM <upd_table>
UNION ALL
SELECT <primary_key> AS join_key, ins.* FROM <ins_table>
) AS sub
ON (sub.join_key = <target_table>.join_key
    AND <target_table>.<end_date> IS NULL)
WHEN MATCHED
     THEN UPDATE SET col1=sub.col1, ... , hash=sub.hash
WHEN NOT MATCHED
     THEN INSERT VALUES (sub.col1, ... , sub.hash) ;

SCD 2:

-- first update old rows
MERGE INTO <target_table>
USING
(
SELECT null AS join_key, upd.* FROM <upd_table>
UNION ALL
SELECT null AS join_key, ins.* FROM <ins_table>
UNION ALL
SELECT <primary_key> AS join_key, ins.* FROM <upd_table>
) AS sub
ON (sub.join_key = <target_table>.join_key
    AND <target_table>.<end_date> IS NULL)
WHEN MATCHED
     THEN UPDATE SET <end_date> = current_date()
WHEN NOT MATCHED
     THEN INSERT VALUES (sub.col1, ... , sub.hash) ;

React ?

This page was last modified: 2021-08-20 15:30

Spark Reflexions

2021-08-20T15:30:00+00:00

Date:2018-05-13

Estimated Time:less than a minute

Spark Reflexions

Spark is a Swiss Army Knife of a hadoop stack. It makes glue between tools and is a de-facto interface for them.

Spark has a lot of advantages:

it's a wonderfull connector: if you look at a way to connect to a tool, spark already has a connector for you. Special mention to csv: spark is also able to read multilined csv with quote, empty lines and special character in it.
it can manipulate very huge dataset: in case the dataset does not feat in memory, the spark partitionning feature allows spliting the tasks in working pipelines
its parallel feature can take advantage of a hadoop cluster, but is a great solution for a laptop too
the scala spark-shell is really handy for development. It's auto-completion features are superior to any shell I've dealt with before
it has SQL syntax, SPARKQL compliance, and also machine learning distributed implementation.
it can be programmed in scala, python, R, java and julia.

Spark as some disadvantages:

time of writing, spark does not support hive ACID tables. The workaround is to do a major compaction of the table before reading it with spark. This means it cannot yet be integrated into a hive ETL process based on ACID tables.
spark programming needs a lot of skills. It is then not a good choice for intermediate enterprise in comparison to hive programming. Again, hive ETL looks a better fit for a maintainability reason.

React ?

This page was last modified: 2021-08-20 15:30

Hbase Reflexions

2021-08-20T15:30:00+00:00

Date:2018-05-13

Estimated Time:less than a minute

Hbase Reflexions

hbase looks a powerfull tool in complement of hive. While hive suits well for ETL and data historisation, it cannot offer sub-second access to thousand of concurrent users.

Hbase comes with a powerful compagnion aka Phoenix that provides many features, while keeping the hbase features intact. Phoenix provides a jdbc driver to hbase. This distinction adds many improvements:

it offers sql, joins, secondary indexes, transactions, sequences on top of hbase.
it simplifies hive, spark, solr, and general programming access to hbase data.

JDBC overview

jdbc access can be done with kerberos on the form

"jdbc:phoenix:thin:url=<scheme>://<server-hostname>:<port>;authentication=SPNEGO;principal=my_user;keytab=/home/my_user/my_user.keytab"

Phoenix connection thought JDBC needs : - a jdbc jar file - the hbase-site.conf - a kerberos ticket

Phoenix does not provide a way for connection pooling. The costly part of the connection is the hbase link. However it is cached within the region servers and a new phoenix jdbc connection can be rebuild for every query since it is a lightweight object.

Hive integration

Phoenix tables can be mounted into hive thanks to a recent plugin. In comparaison to hbase plugin, this allows fast join to hive table (to be tested). While this plugin needs phoenix 4.8.0+ HDP ships with phoenix 4.7.0. However HDP Phoenix is a fork of Phoenix, and it integrates this feature.

Because phoenix allows to mount a hbase table, hive a phoenix or a hbase table, there is many ways to make hive querying hbase. Moreover hive can create a regular table stored as a hbase or phoenix table wich is slightly different. The best way is to mount phoenix as a hive external table. First advantage is phoenix allows salted tables by mean there is no need to manually handle the key to be well distributed. Second, is loading directly into phoenix allows it to manage its secondary indexes and also some other tools such solr.

Finally, hive can access an external phoenix table and allow both upsert and select. The plugin also provides a way to delete/update from hive, however I haven't been able to reproduce yet since transactional tables are only compatible with orc based tables.

Spark integration

Spark can read and write from phoenix here on SOF or here on official. The good point is it can be done from scala or python.

Solr integration

Solr integration with hbase is a good idea: hbase contains the truth and solr allows fast lookup to discover it. However, setting up solr on top of hbase is a pain. Phoenix simplifies drastically that : here

React ?

This page was last modified: 2021-08-20 15:30

PrestoDB reflexions

2021-08-20T15:30:00+00:00

Date:2018-05-13

Estimated Time:less than a minute

PrestoDB reflexions

Prestodb is a cool distributed engine over hadoop. It can be compared to hive LLAP, impala or apache DRILL for fast in-memory OLAP analysis and data discovery.

The main disadvantages it has right now are:

it is not ACID compliant and far from being: this means tables in presto shall be created from scratch for analytics. Because we are speaking about big data, this implies copying
it does not and never will support hive views: this would have been an opportunity to create pseudo-acid workflows
it does not integrate with hbase
it does not integrate with YARN, or at least the attempt looks discontinued on github.

The main advantage it has over hive LLAP are:

many more build-in functions
has been proven to scale well with tons of users
integrates with RDBMS such postgres or NOSQL stores such mongodb

Sor far, the best feet for our internal arhitecture is to go with hive llap in replacement to prestodb. This will brings us the ability to better integrate with hbase/phoenix/solr from one part, and also give us the ability to query ACID version of the DWH without data deduplication.

React ?

This page was last modified: 2021-08-20 15:30

Yarn Reflexions

2021-08-20T15:30:00+00:00

Date:2018-05-16

Estimated Time:less than a minute

Yarn Reflexions

Here are some Apache Yarn consideratoins.

Yarn is so great

Queue modifications: Some application needs extra cores/memory and need to be boosted somehow? Yarns allows on the fly adjustement of queues. In our case, we have nightly ETL and online users the day. It's then easy to dispatch the resources where needed, and without turning off running application.
GPU management: Hadoop 3 will provide a new dimension in yarn responsibility. Yarn will be able to allocate GPU resources for users. This will be of incredible help for machine learning programs and users to share and optimize access to them.

Yarn has some drawbacks

Applications logs: They are stored locally. Same case for temporary files on the local filesystem such map results. And there is no trivial way to store point to hdfs. This is an essential point when configuring the cluster, because the disk allocated for storing yarn user files (hive, spark....) is a real bottleneck. Depending on the cluster workload, keep in mind having a minimum of 2TO for this stuff, per machine.
Yarn superuser: There is now way to kill a process if you are not the owner of the process. This makes things a bit more complex for an admin to supervise such platform.

React ?

This page was last modified: 2021-08-20 15:30

Best terminal Emulator

2021-08-20T15:30:00+00:00

Date:2018-05-27

Estimated Time:less than a minute

Best terminal Emulator

Termite is way better than my previous terminals. It's sensibly based on vim paradigm. It's able to look ahead in the history, copy from it and back to insert mode.

An other great aspect is it notifies when something happens in it, such end of process, new mail an so on.

keybinding

ctrl+space : insert mode
escape : normal mode

React ?

This page was last modified: 2021-08-20 15:30

Jupyter Notebooks and Conda

2021-08-20T15:30:00+00:00

Date:2018-06-02

Estimated Time:less than a minute

Jupyter Notebooks and Conda

After creating a environment with conda and installing packages, the jupyter notebook environment may not see them. The method below adds a new kernel based on the later environment and allow to benefit from conda:

conda activate my-env
conda  create -n my-env pip setuptools ipykernel python=3.5
python -m ipykernel install --user --name my-env --display-name "Python (my-env)"

React ?

This page was last modified: 2021-08-20 15:30

Managing local jars with maven

2021-08-20T15:30:00+00:00

Date:2018-06-02

Estimated Time:less than a minute

Managing local jars with maven

Maven has for me a non user friendly documentation, with 80% of xml content. For long time I have been stuck in how easily manage both dependency from maven central and from local jars.

I finally got something quite general and functional. Here is how to make this working fine:

The pom.xml file:

<!-- Add the manifest -->
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-jar-plugin</artifactId>
  <version>2.4</version>
  <configuration>
    <archive>
      <manifest>
        <mainClass>your.class.Main</mainClass>
      </manifest>
    </archive>
  </configuration>
</plugin>

<!-- Make a standalone build -->
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>2.2</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <outputFile>
        ${project.build.directory}/${artifactId}-${version}-standalone.jar
        </outputFile>
      </configuration>
    </execution>
  </executions>
</plugin>

<!--  import your local jar-->
<dependency>
  <groupId>org.hsqldb</groupId>
  <artifactId>jdbcDriver</artifactId>
  <version>2.4.1</version>
</dependency>

<!-- create a local respository  -->
<repositories>
  <repository>
    <id>my-local-repo</id>
    <url>file://${basedir}/repos</url>
  </repository>
</repositories>

By the way one can add dependency jars by such command line. This will populate the repos folder:

mvn org.apache.maven.plugins:maven-install-plugin:2.3.1:install-file
-Dfile=lib/hsqldb.jar
-DgroupId='org.hsqldb'
-DartifactId='jdbcDriver'
-DgeneratePom=true
-Dpackaging=jar
-Dversion=2.4.1
-DlocalRepositoryPath=repos

As a result, the compilation will result as a standalone jar, containing both local and central dependencies.

React ?

This page was last modified: 2021-08-20 15:30

Loading Postgres from Sqoop

2021-08-20T15:30:00+00:00

Date:2018-06-02

Estimated Time:less than a minute

Loading Postgres from Sqoop

PostgreSQL and Hadoop are complementary technologies. PostgreSQL outperforms Hadoop to retrieve small amount of information from multiple tables. Hadoop outperforms PostgreSQL to transform multiple tables into simplfied and aggregated tables. So, how to load PostgreSQL efficiently from Hadoop ?

Sqoop as a bridge

Hadoop stores data on its distributed filesystem: HDFS. At the end of the day, the resulting procecced data is stored into a hdfs table and may need to be loaded into a PosgreSQL table.

Copy is a efficient a robust PostgreSQL bulkloading function allowing to load among from others CSV files. It is way faster than a bunch of parallel prepared insert statements.

ORC is a efficient and compressed Hadoop data format. It is both faster to build and to read compared to CSV files.

As the bridge for Hadoop and relational databases, Sqoop manages all ORC, CSV, COPY and INSERT methodologies. When dealing with PostgreSQL, there is no perfect solution.

Sqoop best approach

There is two different scenario to load PostgreSQL.

structured columns: Sqoop does not allow to use COPY together with ORC tables but only with CSV. It splits them and run multiple COPY statements from each part in parallel from HDFS. The CSV format needs to be defined in the sqoop parameters.
unstructured columns: Text information may contain new lines. In this situation, Sqoop is stuck when it splits the CSV into multiple parts. Yet it is not possible to use the sqoop direct mode, and the parallel inserts is the only way to succeed. By chance, it is not necessar to transform the result as CSV, and it is finally possible to exploit ORC tables as-is.

Writing this post makes me thinking about a potential workaround to load textual CSV from COPY by using only one mapper in this situation. Since COPY is highly optimized it is possible that it won't degrade the performances while making the overall process totally robust.

EDIT:

Indeed, using only one mapper allows to handle multiline CSV. Since COPY is very fast and has been optimized to import dataset into postgreSQL, in the general case, the method can be generalized to any kind of CSV to work in any situation. That's the ultimate way to load PostgreSQL from Hadoop:

sqoop export\
--connect "jdbc:postgresql://<postgres_host>/<postgres_db>"\
--username <postgres_user>\
--password-file file:///home/$USER/.password\
--export-dir /apps/hive/warehouse/<my_db>/<my_csv_table>\
--table <my_postgres_table>\
--columns "id, text_column"\
-m 1\
--direct\
--lines-terminated-by '\n'\
--fields-terminated-by ','\
--input-null-string "\\\\N"\
--input-null-non-string "\\\\N"\
-- --schema <my_schema>

In order to prepare the table ready to be loaded by sqoop, it can be transformed with a hive sql query:

CREATE TABLE <my_db>.<my_csv_table>
STORED AS textfile
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
AS
SELECT *
FROM <my_hive_table>

React ?

This page was last modified: 2021-08-20 15:30

Eclipse Tips

2021-08-20T15:30:00+00:00

Date:2018-06-03

Estimated Time:less than a minute

Eclipse Tips

darkest theme: simply go throught install menu from this link

emacs mode: select it as a key binding in the preferences. The good point is emacs shortcuts are also used in most linux terminals. The bad point is when managing emacs and vi is a kind of challenge in term of reflexes. The next step would be managing both azerty and qwerty keyboards.

React ?

This page was last modified: 2021-08-20 15:30

Nvidia & ubuntu 18.04

2021-08-20T15:30:00+00:00

Date:2018-08-19

Estimated Time:less than a minute

Nvidia & ubuntu 18.04

My new gigabyte aero 14 does ship a nvidia GTX1050. I had several problem to make it working correctly.

Installation from scratch

When installing ubuntu from my usb key it simply froze after setup. I had to enter in maintenance mode, log as root and the resume to get a VGA screen. After that I was able to install the nvidia drivers by folowing the steps the folowing steps

Turning in nvidia mode

The command sudo prime-select nvidia followed by a reboot makes use of the nvidia chipset. As a result, the powertop tool show a use of 15W.

Turning intel mode

The command sudo prime-select intel followed by a reboot makes use of the intel chipset. However a bug introduced with 18.04 makes both nvidia and intel running and it uses 20W and worst did not allow to shutdown/reboot/suspend/hibernate. I ended up by using that solution which reduce to 7W. Note that I had to make sure the grub was nouveau.modeset=0 nouveau.runpm=0 in the /etc/default/grub file. After installed the tlp tool I ended up using 5W.

Note that I tried that other solution but this resulted in system freeze at reboot.

React ?

This page was last modified: 2021-08-20 15:30

Encrypt & Sign Mails

2021-08-20T15:30:00+00:00

Date:2018-08-25

Estimated Time:less than a minute

Encrypt & Sign Mails

Cacert.org offers validated certificates with a network of trusted people. That's fun !

Get a certificate (eg: CAcert.org)

Client Certificate > new
add it to Firefox

The key pair can then be extracted in p12 format from Firefox and it is possible to transform it to rsa format:

openssl pkcs12 -in ~/Downloads/cacert.p12 -out /home/$USER/.ssh/id_rsa_cacert

Installing neomutt from source on Ubuntu 18.04

sudo apt install libncursesw5-dev
sudo apt install libidn11-dev
git clone https://github.com/neomutt/neomutt
cd neomutt
./configure
make
sudo make install

Loading the key into mutt

It is necessary to setup and use smime_key command to manage keys. It will generate a KeyID for the certificate (something like xxxxxx.0)

cp /home/$USER/git/neomutt/contrib/smime.rc ~/.smime.rc
/usr/libexec/neomutt/smime_keys init
mkdir -p ~/.smime/certificates
touch ~/.smime/certificates/.index
mkdir -p ~/.smime/keys
touch ~/.smime/keys/.index
/usr/libexec/neomutt/smime_keys add_p12 ~/Download/cacert.p12
/usr/libexec/neomutt/smime_keys refresh

Add to source /home/$USER/.smime.rc to .muttrc
Setup the KeyID in the .smime.rc file

Send a signed mail

After having written the mail in neomutt type shift-S and choose the s to sign the mail. It will choose the KeyID by default, and ask for the key password if exists.

Loading a public key

Right now, I don't know

Encrypt a mail

Choose e for encryption, and then enter to list the existing public keys.

React ?

This page was last modified: 2021-08-20 15:30

Mutt & Sidebar

2021-08-20T15:30:00+00:00

Date:2018-08-26

Estimated Time:less than a minute

When receiving many mails a day, having a single mail pool makes easy to loose important emails. Using mailboxes (ie: sub folders) with rules becomes a need. The good point is all the mailboxes are regularly updated to alert when incoming email are received. The few steps are generating a mailbox tree, mooving mails manually, mooving them automatically, and finally mooving the history in batch.

Generating a mailbox tree

Mailboxes consists of folder{cur, new, old} where folder represents its name. The easy way to build them is from the mutt interface, by pressing s on an email and choosing the new mailboxe name. The root of the mailboxe is defined as the folder variable.

Mooving mails manually

It is possible to redirect an email into a mailboxe by pressing s and then =myMailBoxeChoice. In case you want to put it back to the general spool just type = with an empty mailboxe.

Mooving mails automatically

I have been able to write rules based on email features such from with the helpful tutorial. In my setup, getmail fetches the mails and invokes procmail to redirect the mail somewhere. The idea is to parse the email address with a regex and depending on the result redirect it in the right new mailboxe folder.

Mooving mails history

I haven't yet a solution for that.

React ?

This page was last modified: 2021-08-20 15:30

Mutt Search on Steroids

2021-08-20T15:30:00+00:00

Date:2018-08-30

Estimated Time:less than a minute

Mutt Search on Steroids

Mutt has a powerful feature to dig into the history: the limit feature. By pressing l it is then possible to write a query to filter the mails. It is comparable to the WHERE part of a SQL statement and allow combining AND operators. In particular the ~b predicate allow to search within the email's body and the ~X lets filter attachments.

However that feature acts as a grep command and parse all the mails one by one. Searching in multiple dozen of thousand emails can last dozen of minutes. Here comes the notmuch program, that improves greatly the limit feature with instant search and some other goodies (and also drawbacks). The secret of this speed is it scan emails and produces a full text index to be used later.

The installation is straightforward on ubuntu with sudo apt install notmuch. The configuration is also piece of cake: notmuch config will setup the maildir folder. Later the notmuch new command allows to the mails and can be run again when new mails arrive. There is also a way to automatize the indexation by setup system notifications right there.

The integration in mutt to replace the limit feature only need the two rows below in the .muttrc configuration file. Pressing L will propose to write a query to filter the mails based on the index more information here. To get the full details on how to search within notmuch: man notmuch-search-index describes the syntax of the search.

# 'L' performs a notmuch query, showing only the results
macro index L "<enter-command>unset wait_key<enter><shell-escape>read -p 'notmuch query: ' x; echo \$x >~/.cache/mutt_terms<enter><limit>~i \"\`notmuch search --output=messages \$(cat ~/.cache/mutt_terms) | head -n 600 | perl -le '@a=<>;chomp@a;s/\^id:// for@a;s/\\+/\\\\+/g for@a;s/=/\\\\=/g for@a;$,=\"|\";print@a'\`\"<enter>" "show only messages matching a notmuch pattern"
# 'a' shows all messages again (supersedes default <alias> binding)
macro index a "<limit>all\n" "show all messages (undo limit)"

There is some caveats with notmuch however. Its search within attachment is not as powerfull as the built-in mutt. In particular, the later allows to specify the number of attachments. Last but not least, notmuch does only provide a stemmer for english. That's might be possible to extend it with an existing implementation of the french snowball parser.

EDIT:

mu is an alternative to notmuch . It covers the features described above, and has a better syntax.

Here is the full documentation there mu documentation

# Install
sudo apt install maildir-utils

# Index Maildir
mu index -m my/maildir

# setup in crontab or use inotify service

# to add to .muttrc
# Mode 1.
#
# this allows to search into all mailbowes at once
# however, it is a readonly mailbox because off symlinks
macro index <F8> "<shell-escape>mu find --clearlinks --format=links --linksdir=~/Maildir/search "   "mu find"
macro index <F9> "<change-folder-readonly>~/Maildir/search"      "mu find results"

# Mode 2.
#
# this allows to show the 600 first emails
# it is
macro index L "<enter-command>unset wait_key<enter><shell-escape>read -p 'mu query: ' x; echo \$x >~/.cache/mutt_terms<enter><limit>~i \"\`mu find --fields i --quiet 2> /dev/null \$(cat ~/.cache/mutt_terms) | head -n 600 | perl -le '@a=<>;chomp@a;s/\^id:// for@a;s/\\+/\\\\+/g for@a;s/=/\\\\=/g for@a;$,=\"|\";print@a'\`\"<enter>" "show only messages matching a mu pattern"
# 'a' shows all messages again (supersedes default <alias> binding)
macro index a "<limit>all\n" "show all messages"

React ?

This page was last modified: 2021-08-20 15:30

Spelling, mutt & vim

2021-08-20T15:30:00+00:00

Date:2018-09-03

Estimated Time:less than a minute

Spelling, mutt & vim

Having a correct spelling is important in society. This is true specially when come time to write emails.

Here is a trick to activate vim spelling while editing email in mutt. Add the following line into your .muttrc configuration file:

set editor="vim -c 'set textwidth=72' -c':set spell' -c':setlocal spell spelllang=en_us'"

This allows to check English spelling correctly.

Enjoy !

React ?

This page was last modified: 2021-08-20 15:30

Email Best Practices

2021-08-20T15:30:00+00:00

Date:2018-09-03

Estimated Time:less than a minute

Email Best Practices

I will post here all best practices about writing emails. A lot of useful information can be found there

Writing email

Choose an email object

For example this pattern is great: [Action] Topic. Action to help your recipient to understand what you want from him. topic is a small synthesis of the topic in which the action is supposed to happen. This pattern has been advised by Michael Kahn.
Keep the body narrow

The email should not size more than 72 character width. This improve readability specially on small screen. In vim, the width can be setup, and the shorkey g+q on a visual selection makes things easy.

Reply to email

Reply below your recipient

Bottom Posting (BP) is answering bellow the previous email as opposite to Top Posting (TP) aka gmail way of doing. BP has many advantages over TP. BP allows to read the emails in the natural order ; specially for people who discover the thread. more information here Keep an empty row between the previous answer and yours for readability.
Remove information

Remove the recipient signature, out of context content. In particular, do not keep the recipient email within the email body for security reasons.
Append your signature

A signature is generally on the bottom of the email, preceded by a row containing: --
Sign the email

Sign the email with your AES private key as a proof of your identity.
Mask your mailing system

Email can store the User-agent field. This can be found in the header of the email and may share precious information such your operating system and its version. eg: User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0

Mailing List

Email the mailing list only

Remove the previous writer. In any cases, he will receive the email since he subscribed to the list.
Use a dedicated alias

Mailing list are usually public and this you will probably end up with spam. A good trick is to use an alias combined with a rule such: "message directed to the alias goes into the trash". Riseup.net provides alias email.

React ?

This page was last modified: 2021-08-20 15:30

Hive Llap

2021-08-20T15:30:00+00:00

Date:2018-09-07

Estimated Time:less than a minute

Hive Llap

Resource consumption

Hive llap is quite complicated to tune.

hive.llap.daemon.task.scheduler.enable.preemption=true

a great article

React ?

This page was last modified: 2021-08-20 15:30

Postgresql 11 on Archlinux

2021-08-20T15:30:00+00:00

Date:2018-09-27

Estimated Time:less than a minute

Postgresql 11 on Archlinux

I am fairly new to archlinux. Postgresql 11 is fairly new too. Here are the steps to install that version.

# get the lattest postgres in the repos
# avoid checking for sha fingerprint, because not the same version
yaourt --m-arg "--skipchecksums --skippgpcheck" -Sb postgresql
# 1. edit the PKGBUILD file, change to 11.0 version
# 2. comment the check() part
# 3. after building and installing, change to 11 the PGMAJORVERSION within:
sudo vim /usr/bin/postgresql-check-db-dir
# make the link for psql command working
sudo ln -s /usr/lib/libreadline.so.7 /usr/lib/libreadline.so.6
# install that old version to make posgres command work
yaourt -S icu54

That's all. By the way, I do not have any idea right now on how to upgrade minor versions in the future, but I will employ myself to update this post.

React ?

This page was last modified: 2021-08-20 15:30

Deploying Nikola with Githup Pages

2021-08-20T15:30:00+00:00

Date:2018-11-11

Estimated Time:less than a minute

Deploying Nikola with Githup Pages

The simple way

Its about using the docs github pages method.

create a new github repository
create a /docs/ folder with the website in it
enable github pages and specify the docs folder as html source

Why the simple way

The alternate way is to use a gh-pages branch to store the website. The main problem is the method leads to highly complex deployment method with git subtree or other way

A second problem is this method make either duplicate website source code, or makes building the website for each post.

React ?

This page was last modified: 2021-08-20 15:30

A decent python configuration

2021-08-20T15:30:00+00:00

Date:2018-11-11

Estimated Time:less than a minute

A decent python configuration

Use Anaconda

Anaconda allows to have multiple python versions. This is much easier than using virtualenv

conda env list # lists all envs
conda env create py3.5 pip setuptools python=3.5 # create a python env
conda activate py3.5 # activate the latter
conda deactivate # deactivate it
conda env remove -n py3.5 # remove it completely

Use pip together with anaconda

The package will be installed within the environement. pip has a much broader list of packages than the anaconda channel.

pip install <my-package> # install a package
pip install --index-url https://pypi.org/simple/ <my-package> # specify the repository
pip install . # a local package
pip remove <my-package>  # remove it

React ?

This page was last modified: 2021-08-20 15:30

Extracting raw data from hdfs

2021-08-20T15:30:00+00:00

Date:2018-11-22

Estimated Time:less than a minute

Extracting raw data from hdfs

I have tested multiple ways to get data from hdfs. There is two situations:

The data fits in a local node RAM memory:

# will produce a local csv
sql("select * from my_table where true")
    .toPandas()
    .to_csv("myLocalFile.csv", encoding="utf8")
# otherwise spark shell keeps running
exit(0)

# run the python script:
PYTHONSTARTUP=my/python/path/prog.py pyspark --master yarn [...]

The dataset does not fit into RAM memory:

# will produce some csv on hdfs
sql("select * from my_table where true")
    .write()
    .format("csv")
    .save("/output/dir/on/hdfs/")
# otherwise spark shell keeps running
exit(0)

# run the python script:
PYTHONPATH=my/python/path/prog.py pyspark --master yarn [...]
# merge the files on the local filesystem
hadoop fs -getmerge /output/dir/on/hdfs/ /desired/local/output/file.csv

React ?

This page was last modified: 2021-08-20 15:30

Rendering Patients in SolR (part-1)

2021-11-07T15:49:00+00:00

Date:2018-12-02

Estimated Time:less than a minute

Rendering Patients in SolR (part-1)

Patient medical records contain a broad variety of data such numeric, coding, time-series or medical reports. It can be translated as a json document with a nested structure.

SolR allows to handle highly nested objects to be queried efficiently. It provides pagination, and also faceting features wich are very powerful to provide a patient centered search engine.

Patients in electronic health records:

Consider the two patients below:

It yet possible to ask relevant questions to SolR:

# Count and give me one patient having during the same visit:
# - one XX1 code
# - one report containing "popeye"
curl http://localhost:8983/solr/patient/query -d '
q=*:*
&fq={!parent which="content_type_s:visite"}
    ccam_ss:XX1
&fq={!parent which="content_type_s:visite"}
    {!complexphrase inOrder=true}
        doc_text_t:"popeye marche"~3
        AND doc_type_ss:APHP.CRH-H
&fl=id
&rows=1'

Using blockJoin queries has some limitations: - it is not possible to highlight matching text from a parent or a child document - it has some impact on performances

The particular model used in this post has some limitations: - it is not possible to highlight matching text at the patient level - it is not possible to play with a sequence of events, such of encounter - queries are complicated

Let's see a other approach in a next post.

React ?

This page was last modified: 2021-11-07 15:49

Rendering Patients in SolR (part-2)

2021-11-07T15:49:00+00:00

Date:2018-12-04

Estimated Time:less than a minute

Rendering Patients in SolR (part-2)

A patient search engine needs to meet several criteria: #. blazing fast #. handle both structured and unstructured data #. for free text retrieval: include NLP pipelines results #. define complex sequence of events, on patient, and/or encounter level #. highlight results

The previous post was a first attempt to modelize a patient in SolR. While it offered solution for the three first crieteria, it was ineficient to provide event based reasonning and also highlighting results. The below method tries to answer more elegantly the question.

The idea is to modelize the patient as a list of nested encounters with the last encounter beeing the first and vice-versa. Each encounter object being a flat and denormalized document. The design allows to reason sequence of events and also to highlight the part of the texts that match the query.

The denormalization has several advantages in SolR such speading up the queries and also simplifying them. By combining multivalued texts fields and path based text values the design pattern offers power and simplicity. An other aspect of denormalization is to repeat the patient demographic information in each encounter, to enable filtering on this level, and also faceting on patients details.

Having nested encounters allows reasoning on sequence of event by using blockJoin. Indeed this offers the opportunity to add query filter based on childs. The idea is to add a level in each encounter by counting the number of days from the last encounter. It is yet possible to ask for: A patient having a breast cancer and whoom had a chirurgical mamal implant within the 2 years.

One drawback of this design is it does not bring back patients but encounters. One solution is to use faceting capabilities to count the number of unique patients within the cohort.

EDIT: The method has been disapointing:

#. a parent object shall have a different nature than childs. It is then not possible make a be a parent at any level of the chain. #. only a parent object can be highlighted. While it is possible to reason on childs content, it is not possible to highlight them

Let's invent a better method

React ?

This page was last modified: 2021-11-07 15:49

Rendering Patients in SolR (part 3)

2021-08-20T15:30:00+00:00

Date:2018-12-13

Estimated Time:less than a minute

Rendering Patients in SolR (part 3)

The two previous attempts have failed to bring enough power and flexibility in searching within patient history.

The current approach allows:

highlighting all the relevant information at the encounter level
mixing logical operators (OR,AND,NOT) between encounters
mixing temporal order between encounters

First the process of building cohorts changes to:

define the constraints on one encounter
eventually add patient level constraints
look for those matching encounters
loop 1. to 3. to define other encounters
add logical links between encounters/groups
add temporal links between encounters/groups

A constraint on a encounter might be:

a date
a delay
a number
a code
a code and a number
a code and a date
a complex natural langage query
a sequence of events
a combination of all of them (AND,OR,NOT)

Constrained encounters can be then linked together. For example:

encounterA: (37°C) => temp:37
encounterB: (36°C) => temp:36
encounterC: (35°C) => temp:35
encounterA AND encounterB (where encounterA = encounterB is possible)
- fq={!parent which="type:encounter"}(temp:37)&fq={!parent which="type:encounter"}(temp:36)
encounterA OR encounterB
- fq={!parent which="type:encounter"}(temp:37) OR (temp:36)
encounterA BEFORE encounterB
- fq={!parent which="type:encounter"}(temp:37 AND old.temp:36)&fq=(temp:36)
(encounterA AND encounterB) BEFORE encounterC
- fq={!parent which="type:encounter"}(temp:37)&fq={!parent which="type:encounter"}(temp:36)&fq={!parent which="type:encounter"}(temp:37 AND old.temp:35)&fq={!parent which="type:encounter"}(temp:36 AND old.temp:35)&fq={!parent which="type:encounter"}(temp:35)
encounterA AND encounterB (where encounterA != encounterB)
- fq={!parent which="type:encounter"}(temp:37 AND old.temp:36) OR (temp:36 AND old.temp:37)&fq={!parent which="type:encounter"}(temp:36)&fq={!parent which="type:encounter"}(temp:37)
(encounterA AND encounterB) BEFORE encounterC (where encounterA != encounterB)
- fq={!parent which="type:encounter"}(temp:37 AND old.temp:36) OR (temp:36 AND old.temp:37)&fq={!parent which="type:encounter"}(temp:36)&fq={!parent which="type:encounter"}(temp:37)&fq={!parent which="type:encounter"}(temp:37 AND old.temp:35)&fq={!parent which="type:encounter"}(temp:36 AND old.temp:35)&fq={!parent which="type:encounter"}(temp:35)

Sacrifices

To enable the temporal/disjoint feature the documents are repeated in a factorial way. This may lead to terrible situations. Only the textual data should be stored. Moreover in each document, the historical information should only be indexed and not stored. Even by doing this the dimention of the index might be unreasonnable. An other problem is the complexity to produce such documents: the serialization to json might be a nightmare.

Less Ambitions

By not enabling temporal/disjoint, what append's?

encounterA: (37°C) => temp:37
encounterB: (36°C) => temp:36
encounterC: (35°C) => temp:35
encounterA AND encounterB (where encounterA = encounterB is possible)
- fq={!parent which="type:encounter"}(temp:37)&fq={!parent which="type:encounter"}(temp:36)
encounterA OR encounterB
- fq={!parent which="type:encounter"}(temp:37) OR (temp:36)
(encounterA OR encounterB) AND encounterC
- fq={!parent which="type:encounter"}(temp:37) OR (temp:36)&fq={!parent which="type:encounter"}(temp:35)
(encounterA OR encounterB) NOT encounterC
- fq={!parent which="type:encounter"}(temp:37) OR (temp:36)&fq=-_query:"{!parent which="type:encounter"}(temp:35)"

React ?

This page was last modified: 2021-08-20 15:30

Multiword synomyms in SolR

2021-08-20T15:30:00+00:00

Date:2018-12-14

Estimated Time:3 minutes

Multiword synomyms in SolR

Medical domain has multiple ways to tell about the same thing. It becomes handy to use and maintain muliword synonyms thesauri.

SolR recently released a built-in solution to handle multi-word synonyms and I have tested it successfully:

synonyms are computed at query time only.
using multi-word synonyms are only compatible with simple/lucene/eDismax query parsers right now. complexphrase is broken with such dictionary.
multi-word synonyms are supposed to be used before stemming
adding new synonyms needs to delete the collection and reindex it. (to be confirmed)

Some questions remains: 1) How many synonyms can solr handle and still perform well ? 2) How to make synonyms addition/modifications without rebooting ? 3) Which is the better query parser together with multi-word synonyms ?

The simple query parser

Given this field: text:"say me bye or say hello world" And the given configuration:

<example>

one word text:(hello) matches "say me bye or say hello world"
one regex text:(/hel./)** matches "say me bye or say hello world"
not one word text:(hello) matches "say me bye or say hello world"
one tailing wildcard text:(o)** matches "say me bye or say hello world"
one fuzzy word text:(word~) matches "say me bye or say hello world"
two words text:(hello AND bye) matches "say me bye or say hello world"
two words exact text:("hello world") matches "say me bye or say hello world"
two words with distance text:("say world"~2) matches "say me bye or say hello world"
one word AND two words with distance text:(bye AND "hello world"~2) matches "say me bye or say hello world"
two words with distance AND two words with distance text:("say bye"~2 AND "say world"~2) matches "say me bye or say hello world"
one multi-word synonym: text:("see you soon"~2 AND "hello world"~2) matches "say me bye or say hello world"

Note: The simple query parser has some limations compared to the complexphrase parser:

it only keeps order for exact phrases
fuzzy, joker are not allowed for phrases

However it has some advantages:

it allows to define multiple groups of words
it allows regex on one word

How to handle negation

SolR allows to look for the absence of a word or phrase, but not for negations of them. One solution I found is to exploit the multi-word synonyms, by linking all the various ways to express negation to the neg word. Then it is possible to ask for the absence of a negation.

never had,never had any, has no, not, no anymore => neg

This can be used in this way:

"text": "I never had any trouble"
searching for people with trouble: text:((trouble) AND NOT ("neg trouble"))

Generalisation of negation

For a given simple full-text low level filter such a word or a phrase:

simple => (simple AND NOT "neg simple")
"very simple" => ("very simple" AND NOT "neg very simple)
less simple => ( (less AND NOT "neg less") OR (simple AND NOT "neg simple) )
less AND simple => ( (less AND NOT "neg less") AND (simple AND NOT "neg simple) )

Simplifying interface for user

The more user friendly interface in my mind would allo people to write group of words and articulate them with logical operator such AND/OR/NOT. You don't want them to manage phrase and word too. So when a user look for:

see you soon => "see you soon"

They might check for ordered and compacted word, but by default we the tolerate some distance:

see you soon => "see you soon" => "see you soon"~4

In order to remove negated pattern, we apply the above method:

see you soon => "see you soon" => "see you soon"~4 => "see you soon"~4 AND NOT "neg see you soon"~4

People might add other groups and articulate them together:

("see you soon"~4 AND NOT "neg see you soon"~4) NOT ("an other example"~4 AND NOT "neg an other example"~4)

The above example would match a text containing see you soon and not containing an other example. By the way any multi-word synonym would be translated. This also means the user not able to use jocker or fuzzy search. However, the jocker can be replaced with the stemming process, which has the advantage of not breaking the performances in case of very narrow joker query and also being transparent for the end user. There is also a possible replacement for fuzzy feature transparent to user.

Fuzzy search with synonyms

Word Embedding offers the opportunity to freely produce typo synonyms. I have tested succesfully a quite large list of them (~200k entries) and the performances where not impacted. So most common error are transparently integrated to end users.

Dated, Delay and other structured informations

The simple query parser let add some structured informations within the free-text to be queried. For example the below enriched text allows:

"text":"delay0002 dateCreat20181214 my example free-text"
text:([delay0001 TO delay0003] AND [dateCreat201812 TO NOW] AND "example") this makes possible to filter based on dates.

It is also possible to add many coded informations within the text to be queried as structured data with full-text structured capabilities. This will be very useful to get NLP pipelines results.

Dealing with multivalued fields (MV)

While the MVs look temptating for storing multiple occurence of the same concept, they are not a good choice when dealing with full text queries. Indeed the multiple values are not considered as independant values but as a whole with some defined distance between values.

So how to modelize multiple occurences of the same concept within SolR ? One alternate solution is to use multiple single fields for textual datatype. For example it the encounter has two physician notes both with a adverse event section, the resulting encounter document will have two fields "pn.adverse1" and "pn.averse2". This implies that when the user asks for something present into this section, the resulting query should be modified accordingly

"pn.adverse1":"hello world"
"pn.adverse2":"my example"
"pn.adverse:(hello AND example)" will be replaced by:
"pn.adverse1:(hello AND example) OR pn.adverse2:(hello AND example)"

This makes also possible to mix full text queries with structured queries based on the same field. Also there is some drawbacks. The replacement mecanism makes mandatory to know in advance the maximum number of fields of every documents. An other drawback is when the user wants to look into every section of the same document. One solution is to copyField

Conclusion

The final method offers:

multi-word synonyms
monoword synonyms
monoword typos (fuzzy match)
stemming
negation
AND/OR/NOT operators

Still some aspects are missing:

no weighted queries (but not needed for my use case)
no multivalued textual fields (this parser does not make difference between them)

This lets envisage a simple but powerful interface. Let's now see how to transform a medical relational database and populate SolR.

React ?

This page was last modified: 2021-08-20 15:30

Mixing Spark and SolR

2021-08-20T15:30:00+00:00

Date:2018-12-17

Estimated Time:2 minutes

Mixing Spark and SolR

The spark-solr library makes it possible to interact with SolR-Cloud - it is not compatible with SolR standalone. It allows both reading and writing within SolR.

Reading from SolR

The library implements the export features which is fast to retrieve DocValues enabled columns.

// configure the access
val options = Map( "collection" -> "gettingstarted", "zkhost" -> "localhost:9983")
// use export to boost the retrieval
// choose fields and the query
spark.read.format("solr").options(options).option("request_handler", "/export").option("fields","id").option("query","id:/.*.pdf/").load.count

Writing from Spark

csvDF.write.format("solr").options(options).mode(org.apache.spark.sql.SaveMode.Overwrite).save

Creating JSON from Spark

Many useful sql functions to build json from spark exist. So far, I have been using to_json, named_stuct, map, map_from_entries and concat_map with some success.

Generate a json object with zero to many items, from zero to many rows:

with
tmp as (select 1 id,'sec1' sec,'a' cont
    union all
    select 1,'sec2', 'b'
    union all
    select 2, 'sec2','c')
select id, to_json(map_from_entries(collect_list(struct(sec, cont)))) as t
from tmp
group by id;

-- +---+-----------------------+
-- |id |t                      |
-- +---+-----------------------+
-- |1  |{"sec1":"a","sec2":"b"}|
-- |2  |{"sec2":"c"}           |
-- +---+-----------------------+

Add an element within the resulting object, use concat_map to merge them. By using the coalesce function to replace null with empty map, this avoids the collateral effect of nulls that would result into null in any case (null + x = null)

with
tmp as (select 1 id,'sec1' sec,'a' cont
    union all
    select 1,'sec2', 'b'
    union all
    select 2, 'sec2','c')
select id, to_json(
        concat_map(
            coalesce(map("id",id),map()),
            coalesce(map_from_entries(
                collect_list(
                    struct(sec, cont))),map()))) as t
from tmp
group by id

-- +---+---------------------------------+
-- |id |t                                |
-- +---+---------------------------------+
-- |1  |{"id":"1","sec1":"a","sec2":"b"}|
-- |2  |{"id":"1","sec2":"c"}           |
-- +---+---------------------------------+

Create nested objects and place them into one parent object. The use of a named_struct allows to nest objects within a first parent object:

with
secs as (select 1 as p_id, 'sec1' as sec, 'hello' as cont
    union all
    select 1 as p_id, 'sec2' as sec, 'world' cont),
tmp as (select p_id, map_from_entries(collect_list( struct(sec, cont) ) ) as t
    from secs
    group by p_id)
select to_json( named_struct('id', p_id, '_childDocuments_', collect_list(t))) as t
from tmp
group by p_id

-- +-------------------------------------------------------------+
-- |t                                                            |
-- +-------------------------------------------------------------+
-- |{"id":1,"_childDocuments_":[{"sec1":"hello","sec2":"world"}]}|
-- +-------------------------------------------------------------+

Good practice for building json from SQL within spark

Since it is possible to create json elements and concatenate them into objects with joins, it improves readibility to create each kind of element step by step and at the end concatenate them into larger objects. This should not degrade the performance thanks to the spark lazy evaluation design.

As a result, it is possible to begin with encounter information one by one. Sections, then laboratories, medications and so on. Then it is possible to build the encounter by combining all this stuff. Afterwards comes the patient related informations. At the end, it is possible include the encounters as nested childrens within the patients.

By using this design it will be quite easy to enrich any element, or moove some into other, repeat, rename, remove them. It will be also easy to debug. The overall prototype would look like the below steps:

-- encounter labs step
-- encounter diags step
-- encounter sections step

-- patient step and nesting encounters into them
-- good point is: named_struct automatically removes empty fields
with
secs as (select 1 as p_id, 'sec1' as sec, 'hello' as cont
    union all
    select 1 as p_id, 'sec2' as sec, 'world' cont),
tmp as (select p_id, map_from_entries(collect_list( struct(sec, cont) ) ) as t
    from secs
    group by p_id)
select to_json( named_struct(
            'id', patient_id,
            'birthDate', birthDate,
            '_childDocuments_', collect_list(t)
            )
        ) as t
from patients
left join encounters using (patient_id)
group by patient_id

Limitations of spark-solr lib

it does not support solr standalone (only solr-cloud)
it does not handle multi-valued for childDocuemnt yet (fixed with a patch)
it does not handle two levels of childDocuments yet

React ?

This page was last modified: 2021-08-20 15:30

What's the MESH

2021-08-20T15:30:00+00:00

Date:2018-12-19

Estimated Time:5 minutes

What's the MESH

What's the French MESH

This is basically a fat XML without documentation(?)

What is the data structure

The databricks spark-xml lib makes easy to transform an xml file into a sparks dataframe.

import com.databricks.spark.xml._

spark.read.format("com.databricks.spark.xml")
.option("rowTag", "DescriptorRecord")
.load("fredesc2017.xml")
.persist
.registerTempTable("b").printSchema

The resulting dataframe schema is quite complicated.

#root
# |-- AllowableQualifiersList: struct (nullable = true)
# |    |-- AllowableQualifier: array (nullable = true)
# |    |    |-- element: struct (containsNull = true)
# |    |    |    |-- Abbreviation: string (nullable = true)
# |    |    |    |-- QualifierReferredTo: struct (nullable = true)
# |    |    |    |    |-- QualifierName: struct (nullable = true)
# |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |-- QualifierUI: string (nullable = true)
# |-- Annotation: string (nullable = true)
# |-- ConceptList: struct (nullable = true)
# |    |-- Concept: array (nullable = true)
# |    |    |-- element: struct (containsNull = true)
# |    |    |    |-- CASN1Name: string (nullable = true)
# |    |    |    |-- ConceptName: struct (nullable = true)
# |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |-- ConceptRelationList: struct (nullable = true)
# |    |    |    |    |-- ConceptRelation: array (nullable = true)
# |    |    |    |    |    |-- element: struct (containsNull = true)
# |    |    |    |    |    |    |-- Concept1UI: string (nullable = true)
# |    |    |    |    |    |    |-- Concept2UI: string (nullable = true)
# |    |    |    |    |    |    |-- _RelationName: string (nullable = true)
# |    |    |    |-- ConceptUI: string (nullable = true)
# |    |    |    |-- RegistryNumber: string (nullable = true)
# |    |    |    |-- RelatedRegistryNumberList: struct (nullable = true)
# |    |    |    |    |-- RelatedRegistryNumber: array (nullable = true)
# |    |    |    |    |    |-- element: string (containsNull = true)
# |    |    |    |-- ScopeNote: string (nullable = true)
# |    |    |    |-- TermList: struct (nullable = true)
# |    |    |    |    |-- Term: array (nullable = true)
# |    |    |    |    |    |-- element: struct (containsNull = true)
# |    |    |    |    |    |    |-- DateCreated: struct (nullable = true)
# |    |    |    |    |    |    |    |-- Day: long (nullable = true)
# |    |    |    |    |    |    |    |-- Month: long (nullable = true)
# |    |    |    |    |    |    |    |-- Year: long (nullable = true)
# |    |    |    |    |    |    |-- EntryVersion: string (nullable = true)
# |    |    |    |    |    |    |-- SortVersion: string (nullable = true)
# |    |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |    |    |-- TermNote: string (nullable = true)
# |    |    |    |    |    |    |-- TermUI: string (nullable = true)
# |    |    |    |    |    |    |-- ThesaurusIDlist: struct (nullable = true)
# |    |    |    |    |    |    |    |-- ThesaurusID: array (nullable = true)
# |    |    |    |    |    |    |    |    |-- element: string (containsNull = true)
# |    |    |    |    |    |    |-- _ConceptPreferredTermYN: string (nullable = true)
# |    |    |    |    |    |    |-- _IsPermutedTermYN: string (nullable = true)
# |    |    |    |    |    |    |-- _LexicalTag: string (nullable = true)
# |    |    |    |    |    |    |-- _RecordPreferredTermYN: string (nullable = true)
# |    |    |    |-- TranslatorsScopeNote: string (nullable = true)
# |    |    |    |-- _PreferredConceptYN: string (nullable = true)
# |-- ConsiderAlso: string (nullable = true)
# |-- DateCreated: struct (nullable = true)
# |    |-- Day: long (nullable = true)
# |    |-- Month: long (nullable = true)
# |    |-- Year: long (nullable = true)
# |-- DateEstablished: struct (nullable = true)
# |    |-- Day: long (nullable = true)
# |    |-- Month: long (nullable = true)
# |    |-- Year: long (nullable = true)
# |-- DateRevised: struct (nullable = true)
# |    |-- Day: long (nullable = true)
# |    |-- Month: long (nullable = true)
# |    |-- Year: long (nullable = true)
# |-- DescriptorName: struct (nullable = true)
# |    |-- String: string (nullable = true)
# |-- DescriptorUI: string (nullable = true)
# |-- EntryCombinationList: struct (nullable = true)
# |    |-- EntryCombination: array (nullable = true)
# |    |    |-- element: struct (containsNull = true)
# |    |    |    |-- ECIN: struct (nullable = true)
# |    |    |    |    |-- DescriptorReferredTo: struct (nullable = true)
# |    |    |    |    |    |-- DescriptorName: struct (nullable = true)
# |    |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |    |-- DescriptorUI: string (nullable = true)
# |    |    |    |    |-- QualifierReferredTo: struct (nullable = true)
# |    |    |    |    |    |-- QualifierName: struct (nullable = true)
# |    |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |    |-- QualifierUI: string (nullable = true)
# |    |    |    |-- ECOUT: struct (nullable = true)
# |    |    |    |    |-- DescriptorReferredTo: struct (nullable = true)
# |    |    |    |    |    |-- DescriptorName: struct (nullable = true)
# |    |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |    |-- DescriptorUI: string (nullable = true)
# |    |    |    |    |-- QualifierReferredTo: struct (nullable = true)
# |    |    |    |    |    |-- QualifierName: struct (nullable = true)
# |    |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |    |-- QualifierUI: string (nullable = true)
# |-- HistoryNote: string (nullable = true)
# |-- NLMClassificationNumber: string (nullable = true)
# |-- OnlineNote: string (nullable = true)
# |-- PharmacologicalActionList: struct (nullable = true)
# |    |-- PharmacologicalAction: array (nullable = true)
# |    |    |-- element: struct (containsNull = true)
# |    |    |    |-- DescriptorReferredTo: struct (nullable = true)
# |    |    |    |    |-- DescriptorName: struct (nullable = true)
# |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |-- DescriptorUI: string (nullable = true)
# |-- PreviousIndexingList: struct (nullable = true)
# |    |-- PreviousIndexing: array (nullable = true)
# |    |    |-- element: string (containsNull = true)
# |-- PublicMeSHNote: string (nullable = true)
# |-- SeeRelatedList: struct (nullable = true)
# |    |-- SeeRelatedDescriptor: array (nullable = true)
# |    |    |-- element: struct (containsNull = true)
# |    |    |    |-- DescriptorReferredTo: struct (nullable = true)
# |    |    |    |    |-- DescriptorName: struct (nullable = true)
# |    |    |    |    |    |-- String: string (nullable = true)
# |    |    |    |    |-- DescriptorUI: string (nullable = true)
# |-- TreeNumberList: struct (nullable = true)
# |    |-- TreeNumber: array (nullable = true)
# |    |    |-- element: string (containsNull = true)
# |-- _DescriptorClass: long (nullable = true)

The folowing code extracts the first element of the term list. There is 54 one of them in the 2017 mesh version. It is possible to get all of them by union.

sql("""with
tmp as (select
  treenumberlist.TreeNumber as tree,
  descriptorui as descriptor,
  explode(conceptlist.concept.termlist.term[0]) as t
  from b)
select tree,
  descriptor,
  t.string,
  t.termui,
  t._ConceptPreferredTermYN,
  explode(t.thesaurusidlist.thesaurusid) as dict
from tmp""")
.show(30,true)
// +--------------------+----------+--------------------+----------+-----------------------+----------------+
// |                tree|descriptor|              string|    termui|_ConceptPreferredTermYN|            dict|
// +--------------------+----------+--------------------+----------+-----------------------+----------------+
// |[D03.633.100.221....|   D000001|          Calcimycin|   T000002|                      Y|  FDA SRS (2014)|
// |[D03.633.100.221....|   D000001|          Calcimycin|   T000002|                      Y|      NLM (1975)|
// |[D02.705.400.625....|   D000002|             Temefos|   T000008|                      Y|  FDA SRS (2014)|
// |[D02.705.400.625....|   D000002|             Temefos|   T000008|                      Y|      INN (19XX)|
// |[D02.705.400.625....|   D000002|             Temefos|   T000008|                      Y|     USAN (1974)|
// |[D02.705.400.625....|   D000002|            Temephos|   T000007|                      N|      NLM (1996)|
// |[J01.576.423.200....|   D000003|           Abattoirs|   T000009|                      Y|      NLM (1966)|
// |[J01.576.423.200....|   D000003|    Slaughter Houses|T000901742|                      N|      NLM (2017)|
// |[J01.576.423.200....|   D000003|     Slaughter House|T000901743|                      N|      NLM (2017)|
// |[J01.576.423.200....|   D000003|     Slaughterhouses|   T000010|                      N|      UNK (19XX)|
// |[L01.143.506.598....|   D000004|Abbreviations as ...|   T698652|                      Y|      NLM (2008)|
// |       [A01.923.047]|   D000005|             Abdomen|   T000012|                      Y|      NLM (1966)|
// |[C10.597.617.044....|   D000006|      Abdomen, Acute|   T000013|                      Y|      NLM (1966)|
// |[C10.597.617.044....|   D000006| Abdomen chirurgical|fre0138059|                      N|French thesaurus|
// |[C10.597.617.044....|   D000006|      Abdomen urgent|fre0138060|                      N|French thesaurus|
// |           [C26.017]|   D000007|  Abdominal Injuries|   T000015|                      Y|      NLM (1966)|
// |           [C26.017]|   D000007| Injuries, Abdominal|   T000014|                      N|      UNK (19XX)|
// |           [C26.017]|   D000007|Traumatismes de l...|fre0037777|                      Y|French thesaurus|
// |           [C26.017]|   D000007|Blessures abdomin...|fre0072220|                      N|French thesaurus|
// |           [C26.017]|   D000007|Blessures de l'ab...|fre0103940|                      N|French thesaurus|
// |           [C26.017]|   D000007| Lésions abdominales|fre0113338|                      N|French thesaurus|
// |           [C26.017]|   D000007|Lésions de l'abdomen|fre0113337|                      N|French thesaurus|
// |           [C26.017]|   D000007|Lésions traumatiq...|fre0103942|                      N|French thesaurus|
// |           [C26.017]|   D000007|Lésions traumatiq...|fre0103941|                      N|French thesaurus|
// |           [C26.017]|   D000007|Traumatismes abdo...|fre0047606|                      N|French thesaurus|
// |       [C04.588.033]|   D000008| Abdominal Neoplasms|   T000016|                      Y|      NLM (1966)|
// |   [A02.633.567.050]|   D000009|   Abdominal Muscles|   T000018|                      Y|      NLM (1966)|
// |   [A02.633.567.050]|   D000009|    Muscle abdominal|fre0042987|                      N|French thesaurus|
// |   [A02.633.567.050]|   D000009|Muscles de l'abdomen|fre0042988|                      N|French thesaurus|
// |[A08.800.800.120....|   D000010|      Abducens Nerve|   T000019|                      Y|      NLM (1966)|
// +--------------------+----------+--------------------+----------+-----------------------+----------------+
// only showing top 30 rows

React ?

This page was last modified: 2021-08-20 15:30

Solr joins thought Spark

2021-08-20T15:30:00+00:00

Date:2019-01-13

Estimated Time:2 minutes

Solr joins thought Spark

Context

SolR is great to search within complexes documents. However it is weak to make complexes joins between documents and lack for expressive languages. Combining both solr and spark let envisage having best of two worlds.

Dataset

patient_id	visit_id
1	1
1	2
2	1
2	2

R1

patient_id	visit_id
1	2
2	1
2	2

R2

patient_id	visit_id
1	2
2	2

Use case: Q1 AND Q2

Let's say you have defined two queries Q1 and Q2 which have respectively R1 and R2. Now you d'like to know which patients have both kind of visits.

SELECT R1.patient_id, R1.visit_id
FROM R1
JOIN R2 USING (patient_id)
WHERE AND
R2.visit_id != R1.visit_id

Use Case: Q1 OR Q2

(SELECT patient_id, visit_id FROM R1)
UNION
(SELECT patient_id, visit_id FROM R2)

Use Case: Q1 AND NOT Q2

(SELECT patient_id, visit_id FROM R1)
MINUS
(SELECT patient_id, visit_id FROM R2)

Use Case: Q1 THEN Q2

Say we also have date along with data.

Say you want all Patients who had Q2 >1> Q1. You need to have the dates.

SELECT R1.patient_id, R1.visit_id, R1.date
FROM R1
JOIN R2 USING (patient_id)
WHERE TRUE
AND R2.visit_id != R1.visit_id
AND R2.date > R1.date + 1

Operations

Given this formula: (Q1 AND Q2) <2< Q3

WITH
Q1 as  (SELECT 1 as patient_id, 2 as visit_id, 3 as date
    UNION ALL
       SELECT 2,1,2
    UNION ALL
       SELECT 2,2,4),
Q2 as  (SELECT 1 as patient_id, 2 as visit_id, 3 as date
    UNION ALL
       SELECT 2,2,4),
Q3 as  (SELECT 1 as patient_id, 2 as visit_id, 5 as date
    UNION ALL
       SELECT 2,2,6),
T1 as  (SELECT Q1.patient_id
    FROM Q1
    JOIN Q2 USING(patient_id)
    WHERE TRUE
    AND Q2.visit_id != Q1.visit_id),
T2 as  (SELECT Q1.patient_id
    FROM Q1
    JOIN Q3 USING(patient_id)
    WHERE TRUE
    AND Q3.visit_id != Q1.visit_id
    AND Q3.date > Q1.date + 2),
T3 as  (SELECT Q2.patient_id
    FROM Q2
    JOIN Q3 USING(patient_id)
    WHERE TRUE
    AND Q3.visit_id != Q2.visit_id
    AND Q3.date > Q2.date + 2)
(SELECT * FROM T1)
INTERSECT
(SELECT * FROM T2)
INTERSECT
(SELECT * FROM T2)

Given this formula: (Q1 OR Q2) AND Q3

WITH
Q1 as  (SELECT 1 as patient_id, 2 as visit_id, 3 as date
    UNION ALL
       SELECT 2,1,2
    UNION ALL
       SELECT 2,2,4),
Q2 as  (SELECT 1 as patient_id, 2 as visit_id, 3 as date
    UNION ALL
       SELECT 2,2,4),
Q3 as  (SELECT 1 as patient_id, 2 as visit_id, 5 as date
    UNION ALL
       SELECT 2,2,6),
T1 as  (SELECT Q1.patient_id
    FROM Q1
    UNION
    SELECT Q2.patient_id
    FROM Q2),
T2 as  (SELECT Q3.patient_id
    FROM Q3)
(SELECT * FROM T1)
INTERSECT
(SELECT * FROM T2)

Given this formula: Q1 AND Q2 AND Q3

WITH
Q1 as  (SELECT 1 as patient_id, 1 as visit_id),
Q2 as  (SELECT 1 as patient_id, 2 as visit_id),
Q3 as  (SELECT 1 as patient_id, 3 as visit_id),
T1 as  (SELECT Q1.patient_id
    FROM Q1
    JOIN Q2 USING(patient_id)
    WHERE TRUE
    AND Q2.visit_id != Q1.visit_id),
T2 as  (SELECT Q2.patient_id
    FROM Q2
    JOIN Q3 USING(patient_id)
    WHERE TRUE
    AND Q3.visit_id != Q2.visit_id),
T3 as  (SELECT Q1.patient_id
    FROM Q1
    JOIN Q3 USING(patient_id)
    WHERE TRUE
    AND Q3.visit_id != Q1.visit_id)
(SELECT * FROM T1)
INTERSECT
(SELECT * FROM T2)
INTERSECT
(SELECT * FROM T3)

Given this formula: Q1 <1< Q2 <1< Q3

WITH
Q1 as  (SELECT 1 as patient_id, 1 as visit_id, 3 as date),
Q2 as  (SELECT 1 as patient_id, 2 as visit_id, 4 as date),
Q3 as  (SELECT 1 as patient_id, 3 as visit_id, 5 as date),
T1 as  (SELECT Q1.patient_id
    FROM Q1
    JOIN Q2 USING(patient_id)
    WHERE TRUE
    AND Q2.visit_id != Q1.visit_id
    AND Q2.date >= Q1.date + 1),
T2 as  (SELECT Q2.patient_id
    FROM Q2
    JOIN Q3 USING(patient_id)
    WHERE TRUE
    AND Q3.visit_id != Q2.visit_id
    AND Q3.date >= Q2.date + 1)
(SELECT * FROM T1)
INTERSECT
(SELECT * FROM T2)

Given this formula: Q1 <1< Q2 AND NOT Q3

WITH
Q1 as  (SELECT 1 as patient_id, 1 as visit_id, 3 as date),
Q2 as  (SELECT 1 as patient_id, 2 as visit_id, 4 as date),
Q3 as  (SELECT 1 as patient_id, 3 as visit_id, 5 as date),
T1 as  (SELECT Q1.patient_id
    FROM Q1
    JOIN Q2 USING(patient_id)
    WHERE TRUE
    AND Q2.visit_id != Q1.visit_id
    AND Q2.date >= Q1.date + 1)
(SELECT * FROM T1)
MINUS
(SELECT * FROM Q3)

Isolating matching records

For each Qn it would be interesting to get the matching visit. This allows to run the solr queries afterwards.

React ?

This page was last modified: 2021-08-20 15:30

Earth sana in corpore sano

2021-11-07T15:49:00+00:00

Date:2019-03-31

Estimated Time:less than a minute

Earth sana in corpore sano

Society produces more and more Energy

Human have gradualy been less and less physical: hunter/trapper, farmer, workers and novadays digitalized works. In the meantime, it turns out food has been more and more easy to get. We saturate of energy.

While sport is a fairly ealthy activity, the lost of energy is a pity. Worst, going to the gym room does not collect that energy but in addition needs energy to make those sport machine active. This man-power could be use for other purposes.

Society produces more and more Waste

Waste appeared in the begining of 20th century. Before that, most of human production were reused: from animal bones and skin to human feces. However novodays, the waste is far from being a profitable industry.

We all are aware that industry produces way too much plastic packaging. While most people pay attention to put thoses wastes into trash, some do not. The reasons of this lack of care of packaging is unknown and might be related to education, mood or loss of attention. However what is clearly identified is the reason why industry uses such amount of packaging: this ease customers experience.

Eco-Sport: Best of Two Worlds

Let's say every jogger would gather every waste he finds in a plastic bag. This would make it's practice way more fun, and also it's energy would be used efficiently. In addition let's say people would see that jogger gather those things, I bet a few proportion (children in particular) would adopt this practice. Worst case, they would not be so prone to leave rubbish liying anymore !

There is so many healthy sports to practice in town, forest, sea, river and so on and each might be turned in eco-sport! I have right now no idea what eco-friendly gesture a boxer might have...

React ?

This page was last modified: 2021-11-07 15:49

Going Barefoot

2021-08-20T15:30:00+00:00

Date:2019-04-07

Estimated Time:less than a minute

Going Barefoot

Every body has unbalanced legs

Humans are not symetric. As a result the legs are not exactly the same size. For some the size diffence leads to pathologies particularly for sportsmen. Thurty percent of running people have the iliotibial band syndrome which is a consequence of such unbalenced body. As a chronic disease it is complicated to heal and sometimes changing shoes or environment is sufficient to relapse and a podiastrist treatment is often necessary.

Podiastrists advice orthopaedic insoles

Most of them will sell to you their insoles in order to rebalance your posture properly. Insoles have multiple drawbacks: - they costs a lot, are not reimbursed, and need to be replaced every year - they imply a dependency to your podiastrist - your body will adapt to the insole, and it is discuradged to walk without

The minimal shoes awareness

Being overly dependant of a piece of cardboard was innapropriate to me. Since almost every body is asymetric and almost 30% of runner are using insoles, how did humans were hunting in the 200 million past years without our highly skilled podiastrists, and even without shoes ? Am I an not supposed to travel by myself ? The minimal shoes were an elegant solution to all that doubt.

Walking with minimal shoes

I bought a pair of minimal shoes, and tried to walk as usual, but lighter on my feets. Sadly this has not improved a lot my condition an my knee pain. After a while I realized this was not the right way of doing. When walking barefoot I was naturally walking with forefoot before. I have been walking this way for about 2 weeks, and my knee is far better. Also I can feel my muscle adapting: I am much more using my calves, and my foot toes and so on. This muscle reinforcement will be needed when I will be able to run this way soon.

Running with minimal shoes

While walking forefoot is far from obvious, running with minimal shoes does not allow other attitude. I have tested multiple series of few distance and I felt the tendons of my feets not ready to support more distance right now. Also the Achylles tendons is very requested that way.

Conclusion after two month

My iliac band syndrome looks far better. Even after walking hours, I do not feel pain by night in the knee. I fell my feets being more and more robusts, and my body balance more stable. I am thinking this transition would be useful for people with back of hip pain too.

Better than having orthopeadic insoles, walking barefoot is possible and advised. The physiologic changes are huge and the transition must be slow.

There is no possible concession with using minimal shoes: there is no return. It's over with regular shoes.

React ?

This page was last modified: 2021-08-20 15:30

Importance of Teaching

2021-11-07T15:49:00+00:00

Date:2019-07-07

Estimated Time:less than a minute

Importance of Teaching

Today knowledge exists. Search-engine and recomendation-algorithms makes knowledge accessible. We live in a knowledge opulence. Does this mean we make a good use of the knowledge ?

Opulence is counter productive

For multiple reason among reducing consumption from predatory, fish use schooling: they regroup and swim in the same direction. An interpretation is that the predator is less able to make choice in this context of opulence.

Paradoxally, living in a megalopole usually increase lonely and people decrease their activity. Opulence of offer limit the usage.

I am pretty sure the opulence of information on the web acts in the same manner. The potential is huge, but the usage is low.

Faster, but less emotion

Before internet, knowledge was physically shared among people or books. The knowledge came together with an emotional context. Some memory method use emotion to ehnance performances.

Internet knowledge does not provide such emotion together with the information. Therefore, the memory processus cannot be enhanced naturally. The is a need to reinforce the internet knowledges.

Teaching

There is multiple ways of teaching: explaining to others, writing a blog-post, self explaining by mind. In every cases, teaching need a synthesis step, and structure the informations to improve understanding and finally ease the learning.

Teaching is a powerful way of consolidating the knowledges. It helps to raise error or superficial understanding. It has interesting advantages: the teaching materials can be used to re-activate knowledges, and also reduce the cognitive load.

Conclusion

In the days of knowledge opulence, there is a numeric hygiene to fix the informations and take advantage of it.

React ?

This page was last modified: 2021-11-07 15:49

Benefits of functional linux distro

2020-06-06T22:05:00+00:00

Date:2019-08-17

Estimated Time:less than a minute

Benefits of functional linux distro

Operating System is a tool to build and create other tools. OS is secondary and the perfect one should be less present as possible. Less time fighting with installation, less time with configuration, less time with frustration.

My first contact with linux has been ubuntu, and I am still using it at work: using a mainstream tool, is a security and the promize to find out solution fast on forum, blogs. When comes new version of ubuntu LTS, this is a pain to migrate, and always a lost of energy so it is usual to be frustrated and keeping and old OS with old software dependency.

Then I gave a try to rolling-release distribution: Archlinux. The good point is you never late and get the latest version on anything. However I felt that OS was taking more and more importance in my daily practice ; any package upgrade was a critical action. I finally become very quiet with upgrade, and paradoxally late again with software release.

By chance I recently discovered functional programming in general, and functional OS in particular. Nixos and Guix are both exploiting the idea of immutability to build a very robust OS design. This allows to rollback any bad situation very easily. The cost of this power is to learn how to master the beast: haskell or scheme programming language respectively. An I d'say that's an opportunity to learn new languages of interest.

While guix is more appealing for its full-GNU support I choosed nixos. The reason is the packet manager called nix can be installed and used on any distribution. Therefore I can take my time to learn haskell and keep ubuntu as a security for daily primary objective: having a transparent and robust OS.

React ?

This page was last modified: 2020-06-06 22:05

Easy neovim with nix

2020-06-06T22:05:00+00:00

Date:2019-08-22

Estimated Time:less than a minute

Easy neovim with nix

In this post, I will show how to maintain a neovim configuration with the nix package manager. Nix can be installed on any linux distribution with a single sudo bash script.

One of the good point is it ease the way to install and configure softwares: below is an example on how to install neomutt and several vim packages. In case a package does not yet exists or is not up-to-date, the nixpkgs github repository is the place to make pull requests.

In case the merge is accepted, the nix unstable channel will get the modification few hours later, and any body is able to install the package.

Nix is well documented and the community looks extraordinary active. However the syntax and what happens behind the scene is for now a source of curiosity for me: I have not yet switch to nixos, and only be using the nix manager on few servers.

Still some packages are bogus, but things looks moving fast and I such active community makes the invest confident.

# .config/nixpkgs/config.nix
{
  packageOverrides = pkgs: with pkgs; rec {
    myNeovim = neovim.override {
        vimAlias = true;
      configure = {
        customRC = ''
    colo murphy
    set number
    set encoding=utf-8
    au FileType python setlocal formatprg=autopep8\ -

    function! DoPrettyXML()
      let l:origft = &ft
      set ft=
      1s/<?xml .*?>//e
      0put ='<PrettyXML>'
      $put ='</PrettyXML>'
      silent %!xmllint --format -
      2d
      $d
      1
      exe "set ft=" . l:origft
    endfunction
    command! PrettyXML call DoPrettyXML()
        '';
        packages.myVimPackage = with pkgs.vimPlugins; {
          start = [ 
        airline
        vim-nix
        molokai
        tagbar
        easymotion
        nerdtree
        readline-vim
       ];
          opt = [ ];
        };      
    };
  };
    myPackages = pkgs.buildEnv {
      name = "myPackages";
      paths = [
        myNeovim
        vlc
      ];
    };
  };
}

# it is possible to refresh the channels 
nix-channel --update
# it is possible to install an apply both vimrc/packages from this command:
nix-env -i myPackages
# it is possible to remove this by doing
nix-env -e myPackages

React ?

This page was last modified: 2020-06-06 22:05

Spark interactions with Postgres

2020-06-06T22:05:00+00:00

Date:2019-08-24

Estimated Time:less than a minute

Spark interactions with Postgres

Both spark and sqoop allow nativelly to ingest/load relational databases such postgresql. After being disapointed by both, I have finally developped an optimized library to better bridge spark and postgres: reading from postgres, writing to postgres from spark.

The spark-postgres library

Comparison with Sqoop

Automation with yaml

React ?

This page was last modified: 2020-06-06 22:05

FHIR Search Engine - Resource Operators

2020-06-06T22:05:00+00:00

Date:2019-10-12

Estimated Time:2 minutes

FHIR Search Engine - Resource Operators

Technical Consideration

solr de-normalization faster selection query performances and expressivity
solr de-normalization slower update perforamces and add complexity
solr-join can be used in solr-cloud - the joined collection SHALL be replicated over every shard.
solr-join performances are linear among the size of the joined collection (local parameter: score=none).
solr-join query time is up to 1 sec on a 1M sized collection.
solr-join only implements inner joins without access to the joined collection fields
spark-solr provides fast access to solr-cloud fields with docValues=true or type string, long, timestamp
spark-solr produces a dataframe from a solr-cloud collection thought the /export streaming expression handler
spark-solr supports any query filters to subset the collection
spark-solr offers a large set of joins, intersect, union, aggregation and windowing
fhir _query search, allows top down search - Observation from patient with identifier
fhir _has allows bottom-up search - Patient having Observation with coding
fhir _graphql does not allow temporal operators (?)

Technical Conclusion

solr-join are not compatible with large tables
top-down-search can be implemented with solr de-normalized / solr-join
top-down-search prefers solr-denormalized since the joined collection are huge (patient/encounter)
the de-normalization can cover patient/encounter details - spark can fetch them from the patient/encounter delta files (including the _lastUpdated)
the _has covers and operator but lacks or, exept, after, before, not, count and operator scope ( and )
the _has could be implemented by creating a dedicated collection with all resources in it.
the _has could be implemented with spark-solr with and operators only (patients semi-left joins)
adding a long patient field to the fhir patient resource will be usefull
faceting, and unique faceting fields and denormalized fields (extension in the bundle)
[Base]/Condition?patient.identifier=xxx&encounter.lenght=10

Implementation

It is possible from spring-data-solr to build the solr query. This query can be passed to spark-solr.

fhir-syntax -> solr-syntax

(Condition?active=false) AFTER(24, recorded-date, created-date) (Composition?status=canceled&_text=patient&type:below=letter)
ConditionAphp(q=*:&active=true) AFTER(encounter, 24, recorded-date, created-date) CompositionAphp(q=:*&status=canceled)
ConditionAphp(q=*:&active=true) AND(encounter) CompositionAphp(q=:*&status=canceled)
AFTER(AND(Condition(query), Composition(query), encounter), Medication(query), patient, "start-date", "end-date")

AND(Dataframe, Dataframe, Column) OR(Dataframe, Dataframe, Column) EXCEPT(Dataframe, Dataframe) AFTER(Dataframe, Dataframe, column, column, column) OCCURRENCE(Dataframe, number) NOT_IN(Dataframe, Dataframe)

import org.springframework.data.solr.core.DefaultQueryParser
DefaultQueryParser b = new DefaultQueryParser(new SimpleSolrMappingContext());
String z = b.createQueryStringFromNode(search.getCriteria(), null);

Things go like this:

fhir-client: fhir-syntax-operators -> fhir-backend: fhir-syntax-operator => solr-syntax-operator -> spark-backend: parser,joins, results -> create a patient list within solr

Group versus List

it is possible to filter resources by Group thanks to reverse chaining
[Base]/Patient?_has:Group:member:_id=xxxx
[Base]/Encounter?patient._has:Group:member:_id=xxxx
[Base]/Observation?patient._has:Group:member:_id=xxxx
[Base]/Composition?patient._has:Group:member:_id=xxxx
it is possible to filter resources by List thanks to _list parameter:
[Base]/Patient?_list=xxxx where _list is a list of patients
[Base]/Encounter?_list=xxxx where _list is a list of encounters

Results Informations

the resulting json should contain:
- the results on the whole
- the results on the practitioner ward
- the faceting results

Most relevant FHIR ressources have two, three or four of those elements:

id
patient
encounter
date

Patient Level

Every filtered ressource being linked to a patient it is possible to operate on them all:

val r0 = (1::2::3::6::7::Nil).toDF("patient")
val r1 = (1::2::3::Nil).toDF("patient")
val r2 = (2::3::6::Nil).toDF("patient")
val r3 = (1::3::7::Nil).toDF("patient")

`r1 AND r2 AND NOT r3`: r1.intersect(r1).join(r3,"patient"::Nil,"leftanti").dropDuplicates.show
`r1 OR r2 OR NOT r3`: r1.union(r2).union(r0.join(r3,"patient"::Nil,"leftanti")).dropDuplicates.show
`(r1 OR r2) AND r3`: r3.intersect(r1.union(r2)).dropDuplicates.show

Encounter Level

val r1 = ((1,10)::(1,11)::(3,12)::Nil).toDF("patient", "encounter")
val r2 = ((1,11)::(3,12)::Nil).toDF("patient", "encounter")
val r3 = (1::3::7::Nil).toDF("patient")

`r1 SAME_ENC r2`: r1.join(r2, "encounter"::Nil, "leftsemi").select("patient")
`r1 SAME_ENC NOT r2`: r1.join(r2, "encounter"::Nil, "leftanti").select("patient")
`r1 SAME_ENC r2 OR_PAT r3`: r1.join(r2, "encounter"::Nil, "leftsemi").select("patient").union(r3)

Date Level

val r1 = ((1,new Date(1L))::(1,new Date(2L))::Nil).toDF("patient", "date")
val r2 = ((1,new Date(1L))::(3,new Date(2L))::Nil).toDF("patient", "date")

`r1 AFTER r2`: r1.join(r2, "encounter"::Nil, "leftsemi").select("patient")
`r1 AFTER_24h r2`: r1.as("r1").join(r2.as("r2"), "patient"::Nil, "left").filter(col("r1.date").>=(expr("r2.date + INTERVAL 24 HOURS")))
`r1 NOT AFTER_24h r2`: r1.as("r1").join(r2.as("r2"), "patient"::Nil, "full").filter(col("r1.date").isNull || !col("r1.date").>=(col("r2.date")))

All Together

Number Of Occurrence

React ?

This page was last modified: 2020-06-06 22:05

Postgres SCD1 and SCD2 strategy comparison

2020-06-06T22:05:00+00:00

Published on 2019-11-17 5 minutes

Postgres SCD1 and SCD2 strategy comparison

In this experimentation, we will demonstrate that using one CTE which differentiate both update and insert improve significantly the merge processus for both SCD1 and SCD2

Experiment tables

drop table if exists target;
drop table if exists tmp;
create table target(id int, cd text, hash int, end_datetime timestamp);
create table tmp(id int, cd text, hash int);
-- TARGET
insert into target(id,cd,hash) SELECT g.* as id, 'jim' as cd, g.* as hash FROM generate_series(1,10000000) g;

-- TMP
insert into tmp(id,cd,hash) SELECT g.* as id, 'jim' as cd, g.* as hash FROM generate_series(4000000,5000000) g;
insert into tmp(id,cd,hash) SELECT g.* as id, 'john' as cd, g.* as hash FROM generate_series(4000000,5000000) g;
insert into tmp(id,cd,hash) SELECT g.* as id, 'john' as cd, g.* as hash FROM generate_series(14000000,15000000) g;

Atomic implementation scd1

BEGIN;
LOCK TABLE target IN EXCLUSIVE MODE;
explain analyse 
with fj as (
   select 
   tmp.*
   , case when targ.id is null then false else true end as upd
   from target targ
   full outer join tmp on (targ.id = tmp.id)
   where targ.id is null or targ.hash != tmp.hash
),
upd as (
update target targ set id=fj.id, cd=fj.cd, hash=fj.hash 
from fj 
where fj.id = targ.id 
and fj.upd is true)
insert into target select id, cd, hash from fj where fj.upd is false;
ROLLBACK;
--                                                                   QUERY PLAN                                                                  
-- ----------------------------------------------------------------------------------------------------------------------------------------------
--  Insert on target  (cost=1139239.61..1364236.44 rows=49999 width=48) (actual time=6858.761..6858.761 rows=0 loops=1)
--    CTE fj
--      ->  Hash Full Join  (cost=318113.85..515190.96 rows=9999859 width=44) (actual time=3340.633..5629.141 rows=1000001 loops=1)
--            Hash Cond: (tmp.id = targ.id)
--            Filter: ((targ.id IS NULL) OR (targ.hash <> tmp.hash))
--            Rows Removed by Filter: 11000001
--            ->  Seq Scan on tmp  (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.007..227.686 rows=3000003 loops=1)
--            ->  Hash  (cost=154053.60..154053.60 rows=9999860 width=8) (actual time=2919.111..2919.111 rows=10000000 loops=1)
--                  Buckets: 131072  Batches: 256  Memory Usage: 2553kB
--                  ->  Seq Scan on target targ  (cost=0.00..154053.60 rows=9999860 width=8) (actual time=0.042..1421.153 rows=10000000 loops=1)
--    CTE upd
--      ->  Update on target targ_1  (cost=337644.85..624048.65 rows=49999 width=118) (actual time=109.975..109.975 rows=0 loops=1)
--            ->  Hash Join  (cost=337644.85..624048.65 rows=49999 width=118) (actual time=109.974..109.974 rows=0 loops=1)
--                  Hash Cond: (fj_1.id = targ_1.id)
--                  ->  CTE Scan on fj fj_1  (cost=0.00..224996.83 rows=49999 width=104) (actual time=109.973..109.973 rows=0 loops=1)
--                        Filter: (type = 'update'::text)
--                        Rows Removed by Filter: 1000001
--                  ->  Hash  (cost=154053.60..154053.60 rows=9999860 width=18) (never executed)
--                        ->  Seq Scan on target targ_1  (cost=0.00..154053.60 rows=9999860 width=18) (never executed)
--    ->  CTE Scan on fj  (cost=0.00..224996.83 rows=49999 width=48) (actual time=3340.639..6016.017 rows=1000001 loops=1)
--          Filter: (type = 'insert'::text)
--  Planning time: 0.518 ms
--  Execution time: 6974.737 ms

BEGIN;
explain analyse 
with u as (
UPDATE target
SET id=tmp.id, cd=tmp.cd, hash=tmp.hash
FROM tmp
WHERE tmp.id = target.id
)
INSERT INTO target
SELECT tmp.*
FROM tmp
LEFT OUTER JOIN target ON (target.id = tmp.id)
WHERE target.id IS NULL;
ROLLBACK;
--                                                                         QUERY PLAN                                                                        
-- ----------------------------------------------------------------------------------------------------------------------------------------------------------
--  Insert on target  (cost=945906.04..1113139.12 rows=1 width=20) (actual time=5206.659..5206.659 rows=0 loops=1)
--    CTE u
--      ->  Update on target target_2  (cost=371406.47..595983.57 rows=3000003 width=32) (actual time=8586.434..8586.434 rows=0 loops=1)
--            ->  Hash Join  (cost=371406.47..595983.57 rows=3000003 width=32) (actual time=2899.041..5459.650 rows=2000002 loops=1)
--                  Hash Cond: (tmp_1.id = target_2.id)
--                  ->  Seq Scan on tmp tmp_1  (cost=0.00..46217.03 rows=3000003 width=18) (actual time=0.024..403.061 rows=3000003 loops=1)
--                  ->  Hash  (cost=169457.54..169457.54 rows=10999754 width=18) (actual time=2896.588..2896.588 rows=10000000 loops=1)
--                        Buckets: 65536  Batches: 256  Memory Usage: 2195kB
--                        ->  Seq Scan on target target_2  (cost=0.00..169457.54 rows=10999754 width=18) (actual time=0.012..1319.621 rows=10000000 loops=1)
--    ->  Hash Anti Join  (cost=349922.47..517155.54 rows=1 width=20) (actual time=3003.185..4367.119 rows=1000001 loops=1)
--          Hash Cond: (tmp.id = target_1.id)
--          ->  Seq Scan on tmp  (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.027..220.187 rows=3000003 loops=1)
--          ->  Hash  (cost=169457.54..169457.54 rows=10999754 width=4) (actual time=2591.285..2591.285 rows=10000000 loops=1)
--                Buckets: 131072  Batches: 256  Memory Usage: 2400kB
--                ->  Seq Scan on target target_1  (cost=0.00..169457.54 rows=10999754 width=4) (actual time=0.035..1162.839 rows=10000000 loops=1)
--  Planning time: 0.206 ms
--  Execution time: 13793.154 ms

BEGIN;
explain analyse
with u as (
UPDATE target
SET id=tmp.id, cd=tmp.cd, hash=tmp.hash
FROM tmp
WHERE tmp.id = target.id
returning target.id
),
i as (
select tmp.*
from tmp
left join u using (id)
where u.id is null
)
INSERT INTO target
SELECT i.*
FROM i
LEFT OUTER JOIN target ON (target.id = i.id)
WHERE target.id IS NULL;
ROLLBACK;
--                                                                         QUERY PLAN                                                                        
-- ----------------------------------------------------------------------------------------------------------------------------------------------------------
--  Insert on target  (cost=2111532.64..2235532.95 rows=750001 width=48) (actual time=17158.676..17158.676 rows=0 loops=1)
--    CTE u
--      ->  Update on target target_2  (cost=438935.84..682731.95 rows=3000003 width=32) (actual time=2858.907..8387.049 rows=1000001 loops=1)
--            ->  Hash Join  (cost=438935.84..682731.95 rows=3000003 width=32) (actual time=2858.892..5487.211 rows=2000002 loops=1)
--                  Hash Cond: (tmp.id = target_2.id)
--                  ->  Seq Scan on tmp  (cost=0.00..46217.03 rows=3000003 width=18) (actual time=0.026..376.514 rows=3000003 loops=1)
--                  ->  Hash  (cost=200268.26..200268.26 rows=12999726 width=18) (actual time=2856.471..2856.471 rows=10000000 loops=1)
--                        Buckets: 65536  Batches: 256  Memory Usage: 2195kB
--                        ->  Seq Scan on target target_2  (cost=0.00..200268.26 rows=12999726 width=18) (actual time=0.014..1352.460 rows=10000000 loops=1)
--    CTE i
--      ->  Merge Anti Join  (cost=936289.71..1015254.85 rows=1500002 width=12) (actual time=10701.594..10942.156 rows=1000001 loops=1)
--            Merge Cond: (tmp_1.id = u.id)
--            ->  Sort  (cost=471508.34..479008.35 rows=3000003 width=12) (actual time=1143.032..1468.744 rows=3000003 loops=1)
--                  Sort Key: tmp_1.id
--                  Sort Method: external merge  Disk: 72464kB
--                  ->  Seq Scan on tmp tmp_1  (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.075..266.799 rows=3000003 loops=1)
--            ->  Sort  (cost=464781.37..472281.38 rows=3000003 width=4) (actual time=8993.435..9113.745 rows=1000001 loops=1)
--                  Sort Key: u.id
--                  Sort Method: external merge  Disk: 13800kB
--                  ->  CTE Scan on u  (cost=0.00..60000.06 rows=3000003 width=4) (actual time=2858.909..8670.080 rows=1000001 loops=1)
--    ->  Hash Anti Join  (cost=413545.84..537546.15 rows=750001 width=48) (actual time=13088.538..15120.417 rows=1000001 loops=1)
--          Hash Cond: (i.id = target_1.id)
--          ->  CTE Scan on i  (cost=0.00..30000.04 rows=1500002 width=40) (actual time=10701.598..11209.117 rows=1000001 loops=1)
--          ->  Hash  (cost=200268.26..200268.26 rows=12999726 width=4) (actual time=2386.139..2386.139 rows=10000000 loops=1)
--                Buckets: 131072  Batches: 256  Memory Usage: 2400kB
--                ->  Seq Scan on target target_1  (cost=0.00..200268.26 rows=12999726 width=4) (actual time=0.025..1091.998 rows=10000000 loops=1)
--  Planning time: 0.474 ms
--  Execution time: 17204.783 ms

BEGIN;
explain ANALYSE with u as (
INSERT INTO target
SELECT tmp.*
FROM tmp
LEFT OUTER JOIN target ON (target.id = tmp.id)
WHERE target.id IS NULL
)
UPDATE target
SET id=tmp.id, cd=tmp.cd, hash=tmp.hash
FROM tmp
WHERE tmp.id = target.id;
ROLLBACK;
--                                                                        QUERY PLAN                                                                        
-- ---------------------------------------------------------------------------------------------------------------------------------------------------------
--  Update on target  (cost=1181490.50..1444504.63 rows=3000003 width=32) (actual time=8281.352..8281.352 rows=0 loops=1)
--    CTE u
--      ->  Insert on target target_1  (cost=477168.21..675026.30 rows=1 width=20) (actual time=5430.576..5430.576 rows=0 loops=1)
--            ->  Hash Anti Join  (cost=477168.21..675026.30 rows=1 width=20) (actual time=2954.507..4443.851 rows=1000001 loops=1)
--                  Hash Cond: (tmp_1.id = target_2.id)
--                  ->  Seq Scan on tmp tmp_1  (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.019..256.940 rows=3000003 loops=1)
--                  ->  Hash  (cost=231078.98..231078.98 rows=14999698 width=4) (actual time=2520.272..2520.272 rows=10000000 loops=1)
--                        Buckets: 131072  Batches: 256  Memory Usage: 2400kB
--                        ->  Seq Scan on target target_2  (cost=0.00..231078.98 rows=14999698 width=4) (actual time=0.014..1175.381 rows=10000000 loops=1)
--    ->  Hash Join  (cost=506464.21..769478.33 rows=3000003 width=32) (actual time=2964.008..5611.788 rows=2000002 loops=1)
--          Hash Cond: (tmp.id = target.id)
--          ->  Seq Scan on tmp  (cost=0.00..46217.03 rows=3000003 width=18) (actual time=0.071..429.298 rows=3000003 loops=1)
--          ->  Hash  (cost=231078.98..231078.98 rows=14999698 width=18) (actual time=2961.366..2961.366 rows=10000000 loops=1)
--                Buckets: 65536  Batches: 256  Memory Usage: 2195kB
--                ->  Seq Scan on target  (cost=0.00..231078.98 rows=14999698 width=18) (actual time=0.034..1409.287 rows=10000000 loops=1)
--  Planning time: 0.424 ms
--  Execution time: 13712.030 ms

\timing
BEGIN;
UPDATE target
SET id=tmp.id, cd=tmp.cd, hash=tmp.hash
FROM tmp
WHERE tmp.id = target.id;
INSERT INTO target
SELECT tmp.*
FROM tmp
LEFT OUTER JOIN target ON (target.id = tmp.id)
WHERE target.id IS NULL;
ROLLBACK;
-- Time: 5329,939 ms (00:05,330)
-- Time: 12554,331 ms (00:12,554)

Atomic implementation scd2

begin;
explain analyse
with i as (
INSERT INTO target
SELECT tmp.*
FROM tmp
LEFT OUTER JOIN target ON (target.id = tmp.id)
WHERE target.id IS NULL
), 
u as (
INSERT INTO target
SELECT tmp.*
FROM tmp
JOIN target ON (target.id = tmp.id)
WHERE target.hash is distinct from tmp.hash
returning target.id
)
UPDATE target
SET end_datetime = now()
FROM u
WHERE u.id = target.id;
rollback;
--                                                                        QUERY PLAN                                                                       
-- --------------------------------------------------------------------------------------------------------------------------------------------------------
--  Update on target  (cost=1560116.39..1800721.53 rows=3000003 width=54) (actual time=6156.396..6156.396 rows=0 loops=1)
--    CTE i
--      ->  Insert on target target_1  (cost=400062.40..552387.47 rows=1 width=20) (actual time=8926.725..8926.725 rows=0 loops=1)
--            ->  Hash Anti Join  (cost=400062.40..552387.47 rows=1 width=20) (actual time=3602.728..5597.313 rows=1000001 loops=1)
--                  Hash Cond: (tmp.id = target_2.id)
--                  ->  Seq Scan on tmp  (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.450..300.704 rows=3000003 loops=1)
--                  ->  Hash  (cost=250713.29..250713.29 rows=9103129 width=4) (actual time=3092.523..3092.523 rows=10000000 loops=1)
--                        Buckets: 131072  Batches: 128  Memory Usage: 3781kB
--                        ->  Seq Scan on target target_2  (cost=0.00..250713.29 rows=9103129 width=4) (actual time=0.010..1519.702 rows=10000000 loops=1)
--    CTE u
--      ->  Insert on target target_3  (cost=400062.40..589887.51 rows=3000003 width=20) (actual time=6156.392..6156.393 rows=0 loops=1)
--            ->  Hash Join  (cost=400062.40..589887.51 rows=3000003 width=20) (actual time=6156.391..6156.391 rows=0 loops=1)
--                  Hash Cond: (tmp_1.id = target_4.id)
--                  Join Filter: (target_4.hash IS DISTINCT FROM tmp_1.hash)
--                  Rows Removed by Join Filter: 2000002
--                  ->  Seq Scan on tmp tmp_1  (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.013..334.453 rows=3000003 loops=1)
--                  ->  Hash  (cost=250713.29..250713.29 rows=9103129 width=8) (actual time=3359.346..3359.346 rows=10000000 loops=1)
--                        Buckets: 131072 (originally 131072)  Batches: 256 (originally 128)  Memory Usage: 4087kB
--                        ->  Seq Scan on target target_4  (cost=0.00..250713.29 rows=9103129 width=8) (actual time=0.014..1677.709 rows=10000000 loops=1)
--    ->  Hash Join  (cost=417841.40..658446.55 rows=3000003 width=54) (actual time=6156.394..6156.395 rows=0 loops=1)
--          Hash Cond: (u.id = target.id)
--          ->  CTE Scan on u  (cost=0.00..60000.06 rows=3000003 width=32) (actual time=6156.394..6156.394 rows=0 loops=1)
--          ->  Hash  (cost=250713.29..250713.29 rows=9103129 width=18) (never executed)
--                ->  Seq Scan on target  (cost=0.00..250713.29 rows=9103129 width=18) (never executed)
--  Planning time: 0.220 ms
--  Execution time: 15083.304 ms

begin;
explain analyze
with tmp as (
select 
tmp.*
, case when targ.id is null then false else true end as upd
from target targ
full outer join tmp on (targ.id = tmp.id)
where targ.id is null or targ.hash != tmp.hash
 ), 
 u as (
 UPDATE target
 SET end_datetime = now()
 FROM tmp
 where 
 target.id = tmp.id
 AND tmp.upd is true
 )
 INSERT INTO target (id, cd, hash)
 SELECT id, cd, hash
 FROM tmp;
rollback;
--                                                                   QUERY PLAN                                                                  
-- ----------------------------------------------------------------------------------------------------------------------------------------------
--  Insert on target  (cost=1166655991509.07..1166656191396.43 rows=9994368 width=48) (actual time=8666.204..8666.204 rows=0 loops=1)
--    CTE tmp
--      ->  Hash Full Join  (cost=438165.30..635221.42 rows=9994368 width=13) (actual time=4531.084..7278.771 rows=1000001 loops=1)
--            Hash Cond: (tmp_1.id = targ.id)
--            Filter: ((targ.id IS NULL) OR (targ.hash <> tmp_1.hash))
--            Rows Removed by Filter: 11000001
--            ->  Seq Scan on tmp tmp_1  (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.140..307.256 rows=3000003 loops=1)
--            ->  Hash  (cost=274194.69..274194.69 rows=9994369 width=8) (actual time=3974.719..3974.719 rows=10000000 loops=1)
--                  Buckets: 131072  Batches: 256  Memory Usage: 2553kB
--                  ->  Seq Scan on target targ  (cost=0.00..274194.69 rows=9994369 width=8) (actual time=0.059..2009.929 rows=10000000 loops=1)
--    CTE u
--      ->  Update on target target_1  (cost=0.00..1166655356287.65 rows=49943700856896 width=50) (actual time=98.574..98.574 rows=0 loops=1)
--            ->  Nested Loop  (cost=0.00..1166655356287.65 rows=49943700856896 width=50) (actual time=98.573..98.573 rows=0 loops=1)
--                  ->  CTE Scan on tmp tmp_2  (cost=0.00..199887.36 rows=4997184 width=24) (actual time=98.572..98.572 rows=0 loops=1)
--                        Filter: (upd IS TRUE)
--                        Rows Removed by Filter: 1000001
--                  ->  Materialize  (cost=0.00..382727.54 rows=9994369 width=18) (never executed)
--                        ->  Seq Scan on target target_1  (cost=0.00..274194.69 rows=9994369 width=18) (never executed)
--    ->  CTE Scan on tmp  (cost=0.00..199887.36 rows=9994368 width=48) (actual time=4531.092..7663.395 rows=1000001 loops=1)
--  Planning time: 0.137 ms
--  Execution time: 8769.648 ms

React ?

This page was last modified: 2020-06-06 22:05

First contact with spark-jobserver

2020-06-20T19:02:00+00:00

Published on 2019-11-24 less than a minute

First contact with spark-jobserver

This describe how to test locally a spark-jobserver instance. This derive from the readme. First impressions are great.

Server side

git clone git@github.com:spark-jobserver/spark-jobserver.git

Activate Basic Auth

# add this at the end config/local.conf
shiro {
authentication = on
# absolute path to shiro config file, including file name
config.path = "config/shiro.ini.basic"
}
# specify the number of query per slot (default 8)
spark {
jobserver.max-jobs-per-context = 2
}

More detail in the source code

Increase the jar size

# in spark section
short-timeout = 60 s

# in root section
spray.can.server {
    parsing.max-content-length = 300m
    idle-timeout = 400s
    request-timeout = 300s
}

Start the server

sbt
# Run this
job-server-extras/reStart config/local.conf --- -Xmx8g

Client

# compile the jar
sbt job-server-tests/package

# add the jar and give it the "test-binary" name
curl --basic --user 'user:pwd' X POST localhost:8090/binaries/test-binary -H "Content-Type: application/java-archive" --data-binary @job-server-tests/target/scala-2.11/job-server-tests_2.11-0.9.1-SNAPSHOT.jar

# pass the basic auth
curl localhost:8090/contexts --basic --user 'user:pwd' -d "" "localhost:8090/contexts/test-context?num-cpu-cores=4&memory-per-node=512m"

# get the contexts
curl -k --basic --user 'user:pw' https://localhost:8090/contexts

# get the jobs
curl -k --basic --user 'user:pw' https://localhost:8090/jobs

# run a job based on the test-binary file
# also make it syncronized and increase the timeout
curl --basic --user 'user:pwd' -d "input.string = a b c a b see" "localhost:8090/jobs?appName=test-binary&classPath=spark.jobserver.WordCountExample&context=test-context&sync=true&timeout=100000"

Async Client

curl --basic --user user:pw' -d "input.string = a b c a b see" "localhost:8090/jobs?appName=omop-spark-job&classPath=io.frama.parisni.spark.omop.job.WordCountExample&context=test-context&sync=false&timeout=100000"
{
  "duration": "Job not done yet",
  "classPath": "io.frama.parisni.spark.omop.job.WordCountExample",
  "startTime": "2020-06-12T16:43:46.980+02:00",
  "context": "test-context",
  "status": "STARTED",
  "jobId": "d492de45-dc1e-4d08-996c-19b47fdb986f",
  "contextId": "47607883-d90c-456e-8884-d160f4c41480"
}

curl -X GET --basic --user 'user:pw' "localhost:8090/jobs/d492de45-dc1e-4d08-996c-19b47fdb986f"
{
  "duration": "0.157 secs",
  "classPath": "io.frama.parisni.spark.omop.job.WordCountExample",
  "startTime": "2020-06-12T16:43:46.980+02:00",
  "context": "test-context",
  "result": {
    "b": 2,
    "a": 2,
    "see": 1,
    "c": 1
  },
  "status": "FINISHED",
  "jobId": "d492de45-dc1e-4d08-996c-19b47fdb986f",
  "contextId": "47607883-d90c-456e-8884-d160f4c41480"
}

Enabling sparkContext with hive support

By extending the SparkSessionJob class and using the below code, you get the hive support:

context-create:
        curl -X POST -k --basic --user '${USER}:${PWD}' "${HOST}:${PORT}/contexts/test-context?num-cpu-cores=4&memory-per-node=512m&context-factory=spark.jobserver.context.SessionContextFactory"

Complex result (use array/maps as object returned):

{
  "jobId": "4e85111f-98aa-4736-8153-11f937d4fd2f",
  "result": [{
    "list": 1,
    "count": "800",
    "resourceType": "Patient"
  }, {
    "list": 2,
    "count": "100",
    "resourceType": "Condition"
  }]
}

Passing config parameters

One way is to store them into the env.conf file, in the passthough section. They are then accessible from the runtime.contextConfigvariable within the runJob method.

They can be accessed from code thought val ZK = runtime.contextConfig.getString("passthrough.the.variable")

Passing extra options

This can be done with the env.sh file. In this example, we configure the basic auth to solr cloud.

MANAGER_EXTRA_SPARK_CONFS="spark.executor.extraJavaOptions=-Dsolr.httpclient.config=<path-to>/auth_qua.txt -Dhdp.version=2.6.5.0-292|spark.driver.extraJavaOptions=-Dhdp.version=2.6.5.0-292,-Dsolr.httpclient.config=<path-to>/auth_qua.txt,spark.yarn.submit.waitAppCompletion=false|spark.files=$appdir/log4jcluster.properties,$conffile"

React ?

This page was last modified: 2020-06-20 19:02

Introducing the "Readings" section

2020-08-04T14:43:00+00:00

Published on 2020-03-08 less than a minute

Introducing the "Readings" section

Principle

Books are meant to be read. Also books should be reviewed and criticized. I am introducing a new book section with several goals in mind:

read more books
read carefuly
keep notes, and summary

Content

Books, comics, novel, essay, philosophy and also research papers will be covered. In term of content, I can imagine:

reference
citation
synthesis
form analysis
and also cover picture

I wonder I it's legal to share citations, and pictures of comics. This is something I will check out.

Form

I guess I could distinguish several categories, and provide views per categories.

Entry preview

Each entry preview would have:

title
summary
thumbnail
stars

Entry content

Depending on the category, each book entry might have:

summary
screenshots
citations

Next Steps

I will need to explore existing things on the web. Still the content and reading can start with this goal. I can think of:

React ?

This page was last modified: 2020-08-04 14:43

Postgresql Versus Greenplum

2021-11-07T11:53:00+00:00

Published on 2020-12-04 4 minutes

Postgresql Versus Greenplum

Server configuration

a dataset of 30M/60M and 100M rows
16GO RAM
500GO SSD
8 core

Postgresql

version 13.1
madlib 1.17

configured with pgtune

max_connections = 20
shared_buffers = 4GB
effective_cache_size = 12GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 500
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 26214kB
min_wal_size = 4GB
max_wal_size = 16GB
max_worker_processes = 8
max_parallel_workers_per_gather = 4
max_parallel_workers = 8
max_parallel_maintenance_workers = 4

Greenplum

version 6.12
madlib 1.17
4 segments
default configurations

Overall results

conclusion

From this preliminary results, Greeplum is superior to postgreSQL on a standalone server, which is very surprining because it is supposed to work on a distrinbuted cluster.

Greenplum is way faster than postgres in the context of datawarhousing and managing large tables in every domains: insert/update/delete/select. It provides a set of tools such gptext, madlib, map-reduce, pl/python-R and PFX which looks relevant. Also it provides 'queues' to makes multi-users collaborate better.

While postgreSQL is effective on a 30M rows table, it struggles to handle larger tables.

SQL

Postgresql

create table labevents(
   row_id      integer
 , subject_id  integer
 , hadm_id     integer
 , itemid      integer
 , charttime   timestamp(0) without time zone
 , value       character varying(200)
 , valuenum    double precision
 , valueuom    character varying(20)
 , flag        character varying(20)
 , mimic_id    integer
 ) ;
CREATE TABLE

warehouse=# \copy labevents from 'labevents.csv' csv ;
COPY 27854055
Time: 353750.640 ms (05:53.751)

warehouse=# select count(*) c, c1.flag from labevents as c1   group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 18009889 |
  9779594 | abnormal
    64572 | delta
(3 rows)

Time: 939.223 ms

warehouse=# select c1.row_id, row_number() over(partition by c1.itemid order by c1.row_id desc) rk from labevents as c1   limit 10;
  row_id  | rk
----------+----
 27906665 |  1
 27906621 |  2
 27906612 |  3
 27906561 |  4
 27906550 |  5
 27906507 |  6
 27906370 |  7
 27906325 |  8
 27906270 |  9
 27906257 | 10
(10 rows)

Time: 13714.732 ms (00:13.715)

warehouse=# insert into labevents select * from labevents;
INSERT 0 27854055
Time: 423631.155 ms (07:03.631)

warehouse=# select count(*) c, c1.flag from labevents as c1   group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 36019778 |
 19559188 | abnormal
   129144 | delta
(3 rows)

Time: 5295.608 ms (00:05.296)

warehouse=# select c1.row_id, row_number() over(partition by c1.itemid order by c1.row_id desc) rk from labevents as c1   limit 10;
  row_id  | rk
----------+----
 27906665 |  1
 27906665 |  2
 27906621 |  3
 27906621 |  4
 27906612 |  5
 27906612 |  6
 27906561 |  7
 27906561 |  8
 27906550 |  9
 27906550 | 10
(10 rows)

Time: 126347.700 ms (02:06.348)

warehouse=# select count(*) c, c1.flag from labevents as c1 join labevents c2 on c1.row_id = c2.row_id  group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 72039556 |
 39118376 | abnormal
   258288 | delta
(3 rows)

Time: 234570.663 ms (03:54.571)

warehouse=# insert into labevents select * from labevents;
INSERT 0 55708110
Time: 1201961.089 ms (20:01.961)

warehouse=# select count(*) c, c1.flag from labevents as c1   group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 72039556 |
 39118376 | abnormal
   258288 | delta
(3 rows)

Time: 7277.858 ms (00:07.278)

warehouse=# select c1.row_id, row_number() over(partition by c1.itemid order by c1.row_id desc) rk from labevents as c1   limit 10;
  row_id  | rk
----------+----
 27906665 |  1
 27906665 |  2
 27906665 |  3
 27906665 |  4
 27906621 |  5
 27906621 |  6
 27906621 |  7
 27906621 |  8
 27906612 |  9
 27906612 | 10
(10 rows)

Time: 465615.775 ms (07:45.616)

warehouse=# select count(*) c, c1.flag from labevents as c1 join labevents c2 on c1.row_id = c2.row_id  group by c1.flag order by c  desc limit 10 ;
     c     |   flag
-----------+----------
 288158224 |
 156473504 | abnormal
   1033152 | delta
(3 rows)

Time: 494615.953 ms (08:14.616)

warehouse=# delete from labevents where row_id < 5000000;
DELETE 19952316
Time: 157689.975 ms (02:37.690)

warehouse=# update  labevents set flag = 'new flag' where row_id < 10000000;
UPDATE 19949040
Time: 810058.857 ms (13:30.059)

create unlogged table tbl_range (data1 text, data2 text, randomnumber double precision);
insert into tbl_range
( select
    ('{a,b,c,d,e,f}'::text[])[1 + floor(random() * 6)] as data1
    ,('{g,h,i,j,k,l}'::text[])[1 + floor(random() * 6)] as data2
    ,random() * 1000 as randomnumber
from generate_series(1,400000000));
SELECT madlib.pivot('tbl_range', 'pivout', 'data1', 'data2', 'randomnumber', 'sum,max');

Greenplum

warehouse=# create table labevents(
 row_id      integer
 ,subject_id  integer
 ,hadm_id     integer
 ,itemid      integer
 ,charttime   timestamp(0) without time zone
 ,value       character varying(200)
 ,valuenum    double precision
 ,valueuom    character varying(20)
 ,flag        character varying(20)
 ,mimic_id    integer
 ) with (appendonly=true, orientation=column, compresstype=ZSTD );
NOTICE:  Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'row_id' as the Greenplum Database data distribution key for this table.
HINT:  The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE

warehouse=# \copy labevents from '/home/nparis/labevents.csv' csv ;
COPY 27854055
Time: 80878,637 ms

warehouse=# select count(*) c, c1.flag from labevents as c1   group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 18009889 |
  9779594 | abnormal
    64572 | delta
(3 rows)

Time: 1164,303 ms

warehouse=# select c1.row_id, row_number() over(partition by c1.itemid order by c1.row_id desc) rk from labevents as c1   limit 10;
  row_id  | rk
----------+----
 27907576 |  1
 27907382 |  2
 27901097 |  3
 27901054 |  4
 27900783 |  5
 27897288 |  6
 27896669 |  7
 27894567 |  8
 27892281 |  9
 27884982 | 10
(10 rows)

Time: 12569,391 ms

warehouse=# insert into labevents select * from labevents;
INSERT 0 27854055
Time: 79361,173 ms

warehouse=# select count(*) c, c1.flag from labevents as c1   group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 36019778 |
 19559188 | abnormal
   129144 | delta
(3 rows)

Time: 2353,157 ms

warehouse=# select c1.row_id, row_number() over(partition by c1.itemid order by c1.row_id desc) rk from labevents as c1   limit 10;
  row_id  | rk
----------+----
 27906665 |  1
 27906665 |  2
 27906621 |  3
 27906621 |  4
 27906612 |  5
 27906612 |  6
 27906561 |  7
 27906561 |  8
 27906550 |  9
 27906550 | 10
(10 rows)

Time: 25409,642 ms

warehouse=# select count(*) c, c1.flag from labevents as c1 join labevents c2 on c1.row_id = c2.row_id  group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 72039556 |
 39118376 | abnormal
   258288 | delta
(3 rows)

Time: 101654,716 ms

warehouse=# insert into labevents select * from labevents;
INSERT 0 55708110
Time: 112203,521 ms

warehouse=# select count(*) c, c1.flag from labevents as c1   group by c1.flag order by c  desc limit 10 ;
    c     |   flag
----------+----------
 72039556 |
 39118376 | abnormal
   258288 | delta
(3 rows)

Time: 3941,551 ms

warehouse=# select c1.row_id, row_number() over(partition by c1.itemid order by c1.row_id desc) rk from labevents as c1   limit 10;
  row_id  | rk
----------+----
 27906665 |  1
 27906665 |  2
 27906665 |  3
 27906665 |  4
 27906621 |  5
 27906621 |  6
 27906621 |  7
 27906621 |  8
 27906612 |  9
 27906612 | 10
(10 rows)

Time: 65279,232 ms

warehouse=# select count(*) c, c1.flag from labevents as c1 join labevents c2 on c1.row_id = c2.row_id  group by c1.flag order by c  desc limit 10 ;
     c     |   flag
-----------+----------
 288158224 |
 156473504 | abnormal
   1033152 | delta
(3 rows)

Time: 188816,067 ms

warehouse=# delete from labevents where row_id < 5000000;
DELETE 19952316
Time: 8930,371 ms

warehouse=# update  labevents set flag = 'new flag' where row_id < 10000000;
UPDATE 19949040
Time: 32977,965 ms

nparis@gnubuntu:~/bin$ /opt/greenplum-db-6-6.12.1/bin/gpload -f gpload.yaml
2020-12-04 18:16:38|INFO|gpload session started 2020-12-04 18:16:38
2020-12-04 18:16:38|INFO|started gpfdist -p 8081 -P 8082 -f "/home/nparis/labevents.csv" -t 30
2020-12-04 18:17:49|INFO|running time: 71.18 seconds
2020-12-04 18:17:49|INFO|rows Inserted          = 27854055
2020-12-04 18:17:49|INFO|rows Updated           = 0
2020-12-04 18:17:49|INFO|data formatting errors = 0
2020-12-04 18:17:49|INFO|gpload succeeded

warehouse=# create table tbl_range (data1 text, data2 text, randomnumber double precision) with (appendonly=true, orientation=column, compresstype=ZLIB );
NOTICE:  Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'data1' as the Greenplum Database data distribution key for this table.
HINT:  The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
Time: 352,343 ms
warehouse=#                                                                                                                                    insert into tbl_range
( select
    ('{a,b,c,d,e,f}'::text[])[1 + floor(random() * 6)] as data1
    ,('{g,h,i,j,k,l}'::text[])[1 + floor(random() * 6)] as data2
    ,random() * 1000 as randomnumber
from generate_series(1,100000000)) ;
INSERT 0 100000000
Time: 145408,190 ms
warehouse=# SELECT madlib.pivot('tbl_range', 'pivout', 'data1', 'data2', 'randomnumber', 'sum,max');
 pivot
-------

(1 row)

Time: 67074,359 ms

create table tbl_range (data1 text, data2 text, randomnumber double precision) with (appendonly=true, orientation=column, compresstype=ZLIB );
insert into tbl_range
( select
    ('{a,b,c,d,e,f}'::text[])[1 + floor(random() * 6)] as data1
    ,('{g,h,i,j,k,l}'::text[])[1 + floor(random() * 6)] as data2
    ,random() * 1000 as randomnumber
from generate_series(1,400000000));
SELECT madlib.pivot('tbl_range', 'pivout', 'data1', 'data2', 'randomnumber', 'sum,max');

React ?

This page was last modified: 2021-11-07 11:53

An archeology mobile application

2021-03-28T22:16:00+00:00

Published on 2021-03-28 less than a minute

An archeology mobile application

There is a tool I need

I regularly find ancient artefacts (mostly prehistoric silex). I would find highly valuable to store the GPS coordinate of those, and get a map.

I know this practice is subject to regulation. However, most of the time, I found the stuff in ploughed fields or in the forest trails. My guess is having the tool to store and share the information could help the research by building an open-database.

Such a tool could be free software, accesible to anybody. Some feature will allow collaboration (annotation) and also contact between users. The dataset (including photos) will be released as open-data.

Features

create an account
upload a geologalized photo
show artifacts on a map
- filter by category
- filter by user
add metadata
suggest metadata
export open-data

Implemention

python API (fastapi)
postgresql database, postgis
object storage (store the images)
imagemagic tools
android client application
website client application

React ?

This page was last modified: 2021-03-28 22:16

Self hosting emails in the 21th centory

2021-06-13T16:43:00+00:00

Published on 2021-06-13 2 minutes

Self hosting emails in the 21th centory

Why self hosting emails

Mailing technology still active after decades of production. It's one of the simpliest and universal way of sharing information. There is out there a lot of very efficient solutions such gmail or protonmail.

But mail is FUN with a very broad spectrum of technology to master. For example, mailu is built over containers and targets kubernetes, it has a flask api/cli, is maintained by a prolific community using great automation tools on the continuous delivery and also bots on matrix. It has a machine learning based spam filter.

Also mail is serious with personal informations, that shall be kept secret. Hacking ofter takes place on the mailing area, and understanding how this stuff works is a great skill.

Available mailing selfhosted solutions

Mail In A Box (aka MIAB) looks the most advanced solution.

Mailu is quite new and it naturally comes with less features. One major distinction compared to MIAB is its configurability. Most of the modules are optional or sereval choices availables.

Ham & Spam

The smtp port 25 shall be opened. However most internet box close it by defaul. It is often possible to open it from the box web ui.

The reverse dnsis a huge obstacle to self host email. The domain name associated with the home ip server has to be the same as the mail domain name. Sadly most internet providers does not allow clients to change the value of rDNS (which is trivial on the cloud).

Using a relay

Because sending trusted email from a home server is so tricky one can use a mail provider (such riseup) to deliver the email. Also the provider can route the email to your selfhosted server. In this case, you still use your existing email, but nothing is stored on the provider side but on your own instance.

The main advantage with this is you manage your storage yourself. Also in term of privacy, your provider does not store your email history. By the way the email only transit on his side.

Email client

I have been using in the past multiple cli email clients such mutt, neomutt and mu4e together with local indexing tools such notmuch or mu. Those tools are not designed to work with imap but with pop. This means the emails are pulled locally and indexed.

Having a self-hosted email solution such mailu or mail in a box let use the dovecot full text searching feature based on xapian or solr. Together with the imap protocol, it leverages server search with great performances. Moreover imap is far better than pop because all your client will have the same view on the maildir, and there is no need to store locally the emails which improves security.

In the terminal, aerc is a great cli email client focused on imap and it supports server side FTS (filter -a).

On desktop, evolution also is a great software, and it provides imap push in order to get real time notifications and decrease server load.

On mobile, k9mail is great, and provides draft email encryption, autocrypt. Sadly the FTS is not yet supported. I would be glad to help on this. I have also tested fairemail, but most interesting features are limited to the paid version.

Calendar, contacts and tasks

Having our contacts and calendar together with the email is important. Mailu provides a simple python package radicale which brings this features. It does not come with a visual calendar, but can export vcal and ics formats. Mobile applications such etar calendar, tasks, can connect to radicale and synchronize content thought davx5.

Mail-in-a-box comes with next-cloud calendar which is a more complete solution but too heavy for my taste.

Roundcube webmail is able to connect to radicale carddav feature only. Sadly it does not provide any decent calendar plugin.

React ?

This page was last modified: 2021-06-13 16:43

Relax with python

2021-08-18T22:32:00+00:00

Published on 2021-07-15 2 minutes

Relax with python

Most programmer knows python and like it. I have been working with it a lot recently, and there is few important caveat to really have good time.

Forget vim or emacs: Use a tool called an IDE

I have much affection for both vim and emacs and being currently writing this post with the latter. I tried my best to setup a full fledged python with it and I won't do it again. I lost myself in few tutorials and when in the end I got a result it was far from the feature a python IDE can offer.

Intellij IDE has a python plugin which turn it into Pycharm. Unlike vim and emacs, they parse the python code with an AST, which provide some powerful feature:

a powerful python debugger !
find where is used the given object in the project
batch rename object in the project
refactor piece of code

It also come with feature we like:

an authentic vim key binding, including search/replace, tab navigation
an integrated git manager (as good as magit for emacs)
search for files and search within files
a nice dark mode
pytest integration
last but not least, it manages most programming languages out of the box

Manage your python versions with pyenv

Managing multiple versions of python is not trivial, and there is many ways for this. You can go with the packaged versions of the OS (such apt), but it won't have new versions and you will end up adding weird source list and repository. I did use anaconda which is a huge reptilian mess.

Pyenv is really the silver bullet for this. What it does is provide a command line for installing (download and compile) every python version that exist including minors and beta. You can then choose for each project the one you want, without ANY friction. Pyenv is not a replacement with virtualenv at all, they actually combine: you build venv with a given python version provided by pyenv.

Manage you project environment with direnv

Python is so popular, you end up with many virtualenv. Activating/deactivating them becomes really painful. And you will forget to do so, and compromise your pip environment easily.

direnv automates the management of virtualenvs when you navigate in the linux projects, it activate / deactivate them accordingly. You never have to do this again.

It has a pyenv binding and it's super easy to specify the python version you d'like in your venv. This produce a virtualenv folder called .direnv at the root of your project. In order to make intellij not considering this folder's python sources as part of the project, you simply have to "exclude" it with a right click.

Format your code with black

Python syntax rules are great, but it is not perfect. There is still freedom in the way every one format there code. Black takes care of everything. Its main advantage is it formats the same source code the same way, independently of how it is formatted at the beginning.

Commit your code with pre-commit

There is many small details that everyone wants: format python, markdown, xml, yaml... All this can be added BEFORE committing to git with a git-hook. pre-commit takes care of all, and let you add those step within a descriptive configuration.

Test your code with tox

When you test the code you don't only check the source code acts correctly, you also test its environment works. Tox encapsulates the tests in order they will work on your computer but also on the other's one. With tox, you avoid the "it works on my laptop" syndrome.

Build your changelog with Towncrier

Maintaining a changelog is important and there is several tools out there. Towncrier simplifies this process, and with few efforts, you get a clear and maintained changelog.md file.

Publish your package on pypi with twine

Publishing on pypi can be straightforward when you use the right tools. Both setuptools and twine are fine for this. You also have to get several files present. This blog is helpful for this.

React ?

This page was last modified: 2021-08-18 22:32

Aujourd'hui: je déménage demain

2021-12-01T23:43:00+00:00

Published on 2021-11-25 6 minutes

Aujourd'hui: je déménage demain

Je déménage dans 3 mois, disons sur la planete Mars. Dans ce billet, je biffe les sociétés en question, appelons les:

FRIC: Une société fournissant téléphonie et internet
CPAFER: Une société fournissant téléphonie et internet, mais aussi l'infrastructure réseau
Abrasif: Une société revendant de la téléphonie et internet pour le compte de FRIC et CPAFER
SUM: Une société fournissant électricité
OEUF-DES-HAIES: Une société fournissant électricité
CanalPlouf: Une société de télévision
Clitoriche: Une société fournissant téléphonie et internet, partenaire de CanalPlouf
Le service des Empotés: Une administration responsable de la récolte des finances
PapierQ: Cette société m'a été conseillée par l'agence immobilière, et se présente comme un organisme financé par des acteurs divers, pour simplifier les aspects administratifs des déménagements, en particulier l'energie, l'internet et la téléphonie. De plus, ils ont des contrats avec des offres exclusives
STOP-DIRECT: Mon ancienne banque: j'ai dû en changer pour obtenir un prêt immobilier ailleurs.

Dans cette aventure, je cherche à me doter d'internet, du téléphone fixe, et de l'electricité de manière optimale. En quête secondaire, je tente de changer de banque sans faire de vagues.

JOUR 1

Clitoriche

Le site internet de la planete Mars donne des détails sur la mise en place de la fibre dans la ville ou j'emmenage dans 3 mois. Je commence par signaler au webmaster un lien mort qui m'empeche de confirmer par moi même l'eligibilté de mon domicile. Je vois cependant le contact de Clitoriche, société mettant en œuvre la fibre dans ma ville.

Je les contacte donc avec pour objectif de vérifier mon éligibilité. J'indique habiter dans un immeuble au numéro 666. La personne me répond que le logement ne contient qu'un seul habitant et que cela doit être une erreur dans leur fichier, et qu'une demande de rectification doit être faite avant de souscrire. Je me frotte les mains avec le sentiment de gagner du temps en évitant des complications à m'y prendre à l'avance.

JOUR 2

Clitoriche

Entre temps, en feuilletant les papiers OEUF-DES-HAIES, je réalise que j'habite au 666B. Pour éviter d'engager une procédure inutile, je rapelle Clitoriche pour leur signifier mon erreur. Après avoir expliqué le contexte, la personne au bout du fil me demande mon code de souscription que j'ai dû recevoir par mail. Je suis formel: je n'ai pas reçu de mail. Pour retrouver mon dossier, je donne mon adresse email. Il ne lui est pas possible de changer mon addresse pour le 666B. La personne me conseille alors de rappeler, et de demander le service de souscription pour résilier mon compte en fournissant mon code. Je lui fais remarquer que je n'ai justement pas ce code. La personne me demande de patienter. Après 10 minutes d'attente, la ligne est coupée.

Je rappelle donc Clitoriche avec une stratégie différente: je demande à souscrire un nouveau contrat. Je vais donc au bout de la démarche en précisant le 666B cette fois, et je reçois enfin l'email, avec le code. C'est un forfait fibre à 33€/mois, sans télévision, avec une téléphonie payante à la minute pour les appels sortants: ce n'est pas donné mais après tout, ce sont les seuls à fournir de la fibre ?

Je ne réalise pas à ce moment là que je ne souhaitais pas prendre de contrat si tôt mais simplement vérifier mon éligibilité à la fibre.

PapierQ

Avec un sentiment d'accomplissement, j'enchaine avec PapierQ.

Je souhaite gérer l'electricité avec eux. Très cordial, mon interlocuteur me conseille de prendre un contrat à taux fixe au plus vite étant donné l'augmentation du prix de l'energie dûe à la crise du gaz avec les Russes. Il me fait l'éloge de SUM énergie, et sa filiale Verte. Je refrenne mon envie de tempérer son engouement vis à vis de SUM, et nous passons à la simulation. Après vérification, le verdict tombe: le forfait que SUM me propose est de 160€ par mois. Je tombe de si haut, que je ne ressens rien tant le choc est violent et sec. Nous tombons d'accord pour considérer une autre option, à savoir payer en fonction de ce qui est consommé. Cela est plus raisonnable, 9€ pour le forfait, 5€ pour l'assurance 24/24 panne électrique, et le kilowat/heure utilisé. Plutôt enjoué par la tournure des évenements, je lui demande néammoins si l'assurance est optionnelle, ce qu'il me confirme avec une comme une gène dans la voix. Je ne me gène pas pour lui dire que j'aimerais autant ne pas prendre cette assurance.

Avant de se séparer bons amis, il me fait remarquer qu'il peut aussi me faire une comparaison qui n'engage à rien sur les offres internet. Je suis déjà engagé avec Clitoriche, mais finalement, pourquoi ne pas bénéficier de leurs bons conseils voire de leurs offres spéciales ?

Lui n'est pas habilité sur la partie internet et il me passe son collègue pour lequel il a une estime et une confiance sans borne. Rapidement celui-ci m'oriente vers FRIC. Après plusieurs confirmations de sa part: au 666B, je suis bien éligible. Il m'annonce que le forfait disponible est a 20€/mois pendant 1an puis 44€ - mais que je pourrai changer - avec plus de 200 chaines de télé et la téléphonie complète. Je lui fais part de mon désintéret pour la télé, mais il n'a pas d'autre forfait à me proposer. Fort bien, cela me fait tout de même gagner près de 14€ par mois par rapport à Clitoriche. C'est une belle victoire, et je me réjouis d'annoncer mes prouesses à ma femme quand elle va rentrer.

Après avoir reçu de moi des explications liriques mais confuses et quelques recherches de son côté, mon épouse me fait remarquer qu'un forfait FRIC semble plus adapté et surtout moins cher que ce que j'ai souscrit. Immédiatement, j'envoie un mail mesuré à PapierQ pour leur indiquer qu'un forfait plus interessant que ce qu'ils m'ont fait souscrire semble exister, et que je leur serais gré de me basculer dessus.

JOUR 3

Empotés

Je reçois un email de mon ancienne banque STOP-DIRECT qui m'indique que mon solde est désormais de 0€ et que tout à été envoyé à ma nouvelle banque. Je suis étonné car le courrier que j'avais reçu quelques jours avant m'indiquait que je devais leur envoyer un RIB et des justificatifs pour qu'ils procèdent à ce transfert. Mais soit, ce qui m'inquiète davantage est ma taxe d'habitation, qui devait être débitée sur mon ancien compte. Ce n'est pas par choix, j'avais fait le changement de coordonnées banquaires sur le site du service des empotés, mais apparement cela n'a pas été pris en compte. Je tente donc de me connecter aus site de STOP-DIRECT, mais mon compte à été gelé. Au téléphone, le conseiller m'indique qu'en effet, un débit du service des Empotés est en cours mais que celui-ci sera refusé, et que cela n'est pas aussi grâve que si il ne pouvait pas être honoré.

Sur le site du service des Empotés, je cherche par tous les diables la manière de changer de coordonnées banquaires, et je finis par constater que la prise en compte des nouveau RIB se fait au moins un mois après son changement. Je les contacte pour 1. me plaindre du fait de ne pas avoir été prévenu au moment de changer de RIB de la lenteur de ce processus et 2. leur demander comment procéder pour que le prélèvement soit effectué sur ma nouvelle banque.

Joie: je reçois une réponse dans la journée ! Je me rends sur le site pour consulter ma messagerie sécurisée. En quelques mots laconiques on me confirme "qu'en effet le changement de RIB prend un certain temps". Je dois me concentrer avec énergie pour prendre la sage décision de poliement reformuler ma question pour cette fois obtenir la marche à suivre pour qu'ils soient payés, et que je ne devienne pas interdit banquaire.

PapierQ

Je reçois comme réponse que trop tard, si je veux changer de forfait, je dois appeler un numéros FRIC (pour la modique somme de 0.35€/min).

JOUR 5

Clitoriche

Comme prévu, je contacte Clitoriche avec cette fois, mon numéros de souscription. Je reçois d'abord des excuses car leurs techniciens on indiqué un problème sur mon dossier. J'en viens à expliquer mes problèmes à savoir que 1. l'option téléphone n'est pas suffisante 2. j'ai vu un contrat 40% moins cher chez eux et 3. je souhaite débuter mes mensualités dans 3 mois.

Concernant l'option téléphone, si je débourse 5€ supplémentaires je pourrai téléphoner sur les fixes en illimité. Si je débourse 4€ de plus par mois, c'est aussi vers les mobiles que pourrais appeler sans limite. Cela nous s'élève désormais à près de 45€/mois.

Concernant le contrat moins cher, j'ai précisé que c'est sur "le réseau CPAFER" que cette offre s'appuie. En entendant le mot clé CPAFER, mon interlocuteur ne cache plus sa gène et m'indique que Clitoriche n'est pas CPAFER mais une filiale de CanalPlouf. Je précise que l'offre est visible en première page sur leur site. Après vérification, je dois noter le numéro de Clitoriche-CPAFER, une autre filiale.

Concernant la question du début de mes mensualités, on me répond qu'il faut attendre que mon compte soit débloqué, je pourrai les rappeler pour avoir des détails. Je fais remarquer que c'est une question générale mais je réalise que cette considération abstraite n'est pas à la porté de mon interlocuteur, et je le remercie pour ses réponses précises.

Le site que j'ai sous les yeux propose une simulation d'éligibilité à la fibre. Je tente ma chance et je vois que je suis éligible à Bouygues, FRIC et une vente privée, chacun avec un numéro de téléphone. Je décroche mon téléphone pour appeler le numéro indiquant FRIC. La messagerie automatique m'indique que je suis sur le service: Abrasif. Un conseiller me répond, je commence par lui faire part de mon désarois: je pensais appeler FRIC. On me répond qu'Abrasif est affiliée à FRIC. J'explique mon déménagement, ma prochaine addresse. La personne vérifie mon éligibilité et le verdicte tombe: Je ne suis pas éligible à la fibre, encore moins celle de FRIC.

Le service des Empotés

Je suis notifié par mail qu'une réponse du service des Empotés m'a été faite. Une ligne laconique m'indique en substance que je peux leur envoyer un chèque. Cette réponse déclenche une série de questions: quel chéquier (je ne l'ai pas encore reçu), quelle addresse, quel ordre, quelle guarantie que mon nouveau RIB ne va pas être prélevé aussi ? Je pense que cette conversation ne fait que commencer.

FRIC

Après avoir reçu un mail et un sms de FRIC m'engageant à prendre rendez-vous avec un technicien pour l'installation de ma ligne, je décide de consulter mon espace client sur le site internet. À nouveau, je constate que le processus est figé à la première étape : vérification de l'éligibilité.

J'envisage alors de téléphoner à FRIC au numéro présent dans le mail: le 4242. Très méfiant lorsque je lis "Gratuit depuis une ligne FRIC", je vérifie sur plusieurs sites, et je suis convaincu: ce numéro est au prix d'un appel local, c'est à dire gratuit dans le cadre de mon forfait mobile. En revanche, je risque d'attendre très longtemps.

La messagerie m'indique mon numéro de client (déduit de mon numéros de mobile). Un conseiller me répond instantanément. Je lui indique que 1. je me suis trompé d'offre 2. j'ai un doute sur mon éligibilité fibre et 3. j'aimerais débuter mes mensualités dans 3 mois.

Concernant mon offre, étant donné que rien n'a été déclenché de leur côté, il me conseille de me rétracter et pour m'éviter d'avoir à rédiger une lettre, me propose de le faire de son côté: voilà, c'est fait.

Pour l'éligibilité à la fibre, il vérifie et me confirme que c'est le cas. Je lui fais part de ma réserve étant donné que le réseau CPAFER n'est pas présent, il m'explique qu'il s'agit d'un autre réseau: planete Mars fibre qui est associatif, probablement mis en œuvre par la région. En outre, il m'indique que l'immeuble d'à côté, le 666B ne l'est pas. Je réalise que le conseiller PapierQ m'a inscrit par erreur au 666 avec FRIC, et je lui fais part qu'il est probable que j'habite au 666B. Il me donne un détail sur le 666 qui compte 3 habitant: cela me conforte dans l'idée que j'habite finalement au 666 et que:

l'adresse sur le contrat d'electricité est éronné (ce n'est pas le 666B)
Clitoriche n'a pas le bon nombre d'habitant au 666

À suivre...

React ?

This page was last modified: 2021-12-01 23:43