https://blog.parisni.com/atom.xmlSoftware Lobotomy2024-03-27T23:00:29.767008+00:00Nicolas Parisparisni@riseup.netpython-feedgenhttps://blog.parisni.com/images/favicon.webpparisni's bloghttps://blog.parisni.com/wiki/Tmuxtmux2023-11-02T12:24:00+00:00<div id="generated-toc"> </div><hr> <p>#linux</p>
<h1>tmux</h1>
<pre><code>setw -g mode-keys vi
# in order to use xclip as a proxy for copy
# then idea works w/ tmux
bind -T copy-mode-vi y send-keys -X copy-pipe-and-cancel 'xclip -in -selection clipboard'
# Use v to trigger selection (instead of space)
bind-key -T copy-mode-vi v send-keys -X begin-selection
# don't rename windows automatically
# I like to give my tmux windows custom names using the , key.
set-option -g allow-rename off
# see https://github.com/tmux-plugins/tmux-sensible
# Address vim mode switching delay (http://superuser.com/a/252717/65504)
set -s escape-time 0
# Increase scrollback buffer size from 2000 to 50000 lines
set -g history-limit 50000
# Increase tmux messages display duration from 750ms to 4s
set -g display-time 4000
# Upgrade $TERM
set -g default-terminal "screen-256color"
# new panes to be open in the same current folder
# yeah confusing new-window action but works
bind c new-window -c "#{pane_current_path}"
bind % split-window -h -c "#{pane_current_path}"
bind '"' split-window -v -c "#{pane_current_path}"
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=tmux — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-11-02 12:24</time> </div>https://blog.parisni.com/wiki/gnulinuxGnu-Linux2020-09-24T14:41:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span></p>
<h1>Gnu-Linux</h1>
<h2>Make user change password next connection</h2>
<pre><code class="language-bash">chage <span class="hl kwb">-d</span> <span class="hl num">0</span> the_user
</code></pre>
<h2>Search in the history of command</h2>
<ul>
<li><code>ctrl+r</code>: reverse search within the history</li>
<li><code>ctrl+s</code>: search in the history (add <code>stty -ixon</code> in the .bashrc)</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Gnu-Linux — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-09-24 14:41</time> </div>https://blog.parisni.com/wiki/sparkSpark2024-01-14T18:33:00+00:00<div id="generated-toc"> </div><hr> <p>#big-data
#database
#jvm
#video</p>
<h1>Spark</h1>
<h2>Statistics optimisations</h2>
<p>Par defaut, spark n'exploite pas les statistiques:</p>
<pre><code class="language-scala">spark<span class="hl opt">.</span><span class="hl kwd">config</span><span class="hl opt">(</span><span class="hl sng">"spark.sql.cbo.joinReorder.enabled"</span><span class="hl opt">,</span> <span class="hl sng">"true"</span><span class="hl opt">)</span>
spark<span class="hl opt">.</span><span class="hl kwd">config</span><span class="hl opt">(</span><span class="hl sng">"spark.sql.cbo.enabled"</span><span class="hl opt">,</span> <span class="hl sng">"true"</span><span class="hl opt">)</span>
spark<span class="hl opt">.</span><span class="hl kwd">config</span><span class="hl opt">(</span><span class="hl sng">"spark.sql.cbo.joinReorder.dp.star.filter"</span><span class="hl opt">,</span> <span class="hl sng">"true"</span><span class="hl opt">)</span>
spark<span class="hl opt">.</span><span class="hl kwd">config</span><span class="hl opt">(</span><span class="hl sng">"spark.sql.statistics.histogram.enabled"</span><span class="hl opt">,</span> <span class="hl sng">"true"</span><span class="hl opt">)</span>
</code></pre>
<h2>Analyser les statistiques pour les tables</h2>
<pre><code class="language-scala">spark<span class="hl opt">.</span><span class="hl kwd">sql</span><span class="hl opt">(</span>s<span class="hl sng">"ANALYZE TABLE <table> COMPUTE STATISTICS"</span><span class="hl opt">)</span>
spark<span class="hl opt">.</span><span class="hl kwd">sql</span><span class="hl opt">(</span>s<span class="hl sng">"ANALYZE TABLE <table> COMPUTE STATISTICS FOR COLUMNS <col1>,<col2>"</span><span class="hl opt">)</span>
</code></pre>
<p>By the way when analyzing for columns, the whole table statistics are
also updated. So the first one is not relevant in case the second one
is ran.</p>
<h2>Récupérer les statistiques dans le metastore</h2>
<pre><code class="language-sql"><span class="hl kwa">select</span> <span class="hl opt">*</span> <span class="hl kwa">from</span> <span class="hl sng">"TABLE_PARAMS"</span> <span class="hl kwa">where</span> <span class="hl sng">"TBL_ID"</span> <span class="hl opt">=</span> <span class="hl num">34900</span><span class="hl opt">;</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.colStats.person_id.<span class="hl kwa">max</span> | <span class="hl num">9454641234</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.colStats.person_id.maxLen | <span class="hl num">8</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.colStats.person_id.avgLen | <span class="hl num">8</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.colStats.person_id.<span class="hl kwa">version</span> | <span class="hl num">1</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.colStats.person_id.distinctCount | <span class="hl num">8119231</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.colStats.person_id.<span class="hl kwa">min</span> | <span class="hl num">185429806</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.colStats.person_id.nullCount | <span class="hl num">0</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.totalSize | <span class="hl num">1738777345</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">statistics</span>.numRows | <span class="hl num">39720020</span>
<span class="hl num">34900</span> | numFiles | <span class="hl num">0</span>
<span class="hl num">34900</span> | totalSize | <span class="hl num">0</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.sources.<span class="hl kwa">schema</span>.numParts | <span class="hl num">1</span>
<span class="hl num">34900</span> | <span class="hl kwa">EXTERNAL</span> | <span class="hl kwa">TRUE</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.<span class="hl kwa">create</span>.<span class="hl kwa">version</span> | <span class="hl num">2.4.3</span>
<span class="hl num">34900</span> | spark.<span class="hl kwa">sql</span>.sources.provider | delta
<span class="hl num">34900</span> | transient_lastDdlTime | <span class="hl num">1583162364</span>
</code></pre>
<p>The transient_lastDdlTime represents the last updated statistics timestamp. It can be retrieved with:</p>
<pre><code class="language-sql"><span class="hl kwa">select</span> <span class="hl kwd">to_timestamp</span><span class="hl opt">(</span><span class="hl kwa">cast</span><span class="hl opt">(</span><span class="hl num">1583162364</span> <span class="hl kwa">as</span> <span class="hl kwb">bigint</span><span class="hl opt">)):</span><span class="hl kwc">:timestamp</span>
to_timestamp
<span class="hl slc">---------------------</span>
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">03</span><span class="hl opt">-</span><span class="hl num">02 15</span><span class="hl kwc">:19:24</span>
</code></pre>
<h2>Joins</h2>
<p><a href="https://towardsdatascience.com/the-art-of-joining-in-spark-dcbd33d693c">good article about joins</a></p>
<h2>Range join</h2>
<p><a href="http://zachmoshe.com/2016/09/26/efficient-range-joins-with-spark.html">efficient range join</a>
The idea is to bucket both columns and add this join condition.</p>
<h2>Register delta table in hive</h2>
<p>you can simply save the table like :</p>
<pre><code class="language-scala">spark<span class="hl opt">.</span>write<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"delta"</span><span class="hl opt">).</span><span class="hl kwd">saveAsTable</span><span class="hl opt">(</span><span class="hl sng">"the_table"</span><span class="hl opt">)</span>
</code></pre>
<p>you can also use an external table :</p>
<pre><code class="language-scala">spark<span class="hl opt">.</span><span class="hl kwd">range</span><span class="hl opt">(</span><span class="hl num">10</span><span class="hl opt">).</span>write<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"delta"</span><span class="hl opt">).</span><span class="hl kwd">save</span><span class="hl opt">(</span><span class="hl sng">"/tmp/events"</span><span class="hl opt">)</span>
spark<span class="hl opt">.</span><span class="hl kwd">sql</span><span class="hl opt">(</span><span class="hl sng">"create table events using delta location '/tmp/events'"</span><span class="hl opt">)</span>
spark<span class="hl opt">.</span><span class="hl kwd">sql</span><span class="hl opt">(</span><span class="hl sng">"select * from events"</span><span class="hl opt">).</span><span class="hl kwd">show</span><span class="hl opt">(</span><span class="hl num">100</span><span class="hl opt">)</span>
</code></pre>
<p>It is also possible to directly write a delta table into the metastore:</p>
<pre><code class="language-scala">df<span class="hl opt">.</span>write<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"delta"</span><span class="hl opt">).</span><span class="hl kwd">saveAsTable</span><span class="hl opt">(</span><span class="hl sng">"myDeltaTable"</span><span class="hl opt">)</span>
</code></pre>
<h2>Partitionning</h2>
<p>This will create a folder per enumerated column content. Later, any filter (in, =, rlike) will allow to skip partitions.</p>
<blockquote>
<p>WARNING: The partition column shall not contain special spark column character such upper case, dot... Otherwize, the
filter will throw an exception.</p>
</blockquote>
<h2>Bucketing</h2>
<p><a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-bucketing.html">see</a></p>
<p>It has two benefits and one drawback. The drawback is it needs to shuffle and sort the data before writing it, hence
increasing a lot the creation of the table. The two benefits are:</p>
<ul>
<li>speed-up filters on the column (skips unused buckets)</li>
<li>speed-up joins on the column</li>
</ul>
<p>Best is when both tables are bucketed (again, same bucket number and same bucket column).</p>
<p>It is also possible to join a single bucketed table to a <code>on the fly</code> repartitionned table. The latter has to be
repartitioned into the same number bucket and on the same column.</p>
<pre><code class="language-scala">table1<span class="hl opt">.</span><span class="hl kwd">repartition</span><span class="hl opt">(</span><span class="hl num">100</span><span class="hl opt">)</span> <span class="hl slc">// 20 * 100 parquet files</span>
<span class="hl opt">.</span>write<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"parquet"</span><span class="hl opt">)</span>
<span class="hl opt">.</span><span class="hl kwd">bucketBy</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">,</span> <span class="hl sng">"encounter"</span><span class="hl opt">)</span> <span class="hl slc">// 20 buckets</span>
<span class="hl opt">.</span><span class="hl kwd">sortBy</span><span class="hl opt">(</span><span class="hl sng">"encounter"</span><span class="hl opt">)</span> <span class="hl slc">// sorted</span>
<span class="hl opt">.</span><span class="hl kwd">saveAsTable</span><span class="hl opt">(</span><span class="hl sng">"bucketTable"</span><span class="hl opt">)</span>
<span class="hl kwa">val</span> notBucketTable <span class="hl opt">=</span> table2
<span class="hl opt">.</span><span class="hl kwd">repartition</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">,</span> $<span class="hl sng">"encounter"</span><span class="hl opt">)</span>
<span class="hl opt">.</span><span class="hl kwd">sortWithinPartitions</span><span class="hl opt">(</span><span class="hl sng">"encounter"</span><span class="hl opt">)</span>
spark<span class="hl opt">.</span><span class="hl kwd">table</span><span class="hl opt">(</span><span class="hl sng">"bucketTable"</span><span class="hl opt">)</span>
<span class="hl opt">.</span><span class="hl kwd">join</span><span class="hl opt">(</span>notBucketTable<span class="hl opt">,</span> <span class="hl kwd">Seq</span><span class="hl opt">(</span><span class="hl sng">"encounter"</span><span class="hl opt">))</span>
</code></pre>
<h2>Partitionning + Bucketing</h2>
<p>It is possible to use both partitionnning and bucketing.</p>
<pre><code class="language-scala">table1<span class="hl opt">.</span><span class="hl kwd">repartition</span><span class="hl opt">(</span><span class="hl num">100</span><span class="hl opt">).</span>write<span class="hl opt">.</span><span class="hl kwd">mode</span><span class="hl opt">(</span><span class="hl sng">"overwrite"</span><span class="hl opt">).</span><span class="hl kwd">partitionBy</span><span class="hl opt">(</span><span class="hl sng">"typesimple"</span><span class="hl opt">).</span><span class="hl kwd">bucketBy</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">,</span> <span class="hl sng">"encounter"</span><span class="hl opt">).</span><span class="hl kwd">sortBy</span><span class="hl opt">(</span><span class="hl sng">"encounter"</span><span class="hl opt">).</span><span class="hl kwd">saveAsTable</span><span class="hl opt">(</span><span class="hl sng">"edsomop.encounterAphp"</span><span class="hl opt">)</span>
</code></pre>
<h1>Spark Streaming</h1>
<p>A streaming context can be created from a sparkContext:</p>
<pre><code class="language-scala"><span class="hl kwa">import</span> org<span class="hl opt">.</span>apache<span class="hl opt">.</span>spark<span class="hl opt">.</span>streaming<span class="hl opt">.{</span>Seconds<span class="hl opt">,</span> StreamingContext<span class="hl opt">}</span>
<span class="hl kwa">val</span> ssc <span class="hl opt">=</span> <span class="hl kwa">new</span> <span class="hl kwd">StreamingContext</span><span class="hl opt">(</span>spark<span class="hl opt">.</span>sparkContext<span class="hl opt">,</span> <span class="hl kwd">Seconds</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">))</span>
<span class="hl kwa">val</span> lines <span class="hl opt">=</span> ssc<span class="hl opt">.</span><span class="hl kwd">textFileStream</span><span class="hl opt">(</span><span class="hl sng">"hdfs:///user/nparis/streamingRepository/*"</span><span class="hl opt">)</span>
lines<span class="hl opt">.</span>foreachRDD <span class="hl opt">{</span> rdd <span class="hl opt">=> {</span>
<span class="hl kwa">val</span> df <span class="hl opt">=</span> rdd<span class="hl opt">.</span>toDF<span class="hl opt">;</span> df<span class="hl opt">.</span>show
<span class="hl opt">}</span>
<span class="hl opt">}</span>
ssc<span class="hl opt">.</span>start
</code></pre>
<p>The <code>textFileStream</code> gets the texts files within a given repository,
and produces a RDD with a row per files rows.</p>
<p>A DStream object, supports transformations such <code>filter</code>,<code>count</code>, or
<code>map</code> which returns back a new DStream.</p>
<p>The streaming process can only begin when an "output operation"
happens on the stream, such <code>print</code>, <code>foreachRDD</code>, <code>saveAsTextFile</code>...</p>
<h2>Structured streaming</h2>
<p><a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.streaming.DataStreamReader">scaladoc of structured streaming</a>
<a href="https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-structured-streaming.html">wikibooks</a></p>
<p>Structured Streaming allows spark streaming to manipulate structured
input/output such <code>parquet</code>, <code>csv</code>, <code>json</code> or <code>delta</code>. It can also
handle any output datasource thanks to <code>writeStream.foreach</code>and
<code>foreachBatch</code> methods.</p>
<pre><code class="language-scala"><span class="hl slc">// schema is necessary</span>
<span class="hl kwa">val</span> schema <span class="hl opt">=</span> spark<span class="hl opt">.</span>read<span class="hl opt">.</span><span class="hl kwd">json</span><span class="hl opt">(</span><span class="hl sng">"structuredStreaming"</span><span class="hl opt">).</span>schema
<span class="hl kwa">val</span> c <span class="hl opt">=</span> spark<span class="hl opt">.</span>readStream<span class="hl opt">.</span><span class="hl kwd">schema</span><span class="hl opt">(</span>schema<span class="hl opt">).</span><span class="hl kwd">json</span><span class="hl opt">(</span><span class="hl sng">"structuredStreaming"</span><span class="hl opt">)</span>
<span class="hl slc">// checkpointing is necessary</span>
spark<span class="hl opt">.</span>conf<span class="hl opt">.</span><span class="hl kwd">set</span><span class="hl opt">(</span><span class="hl sng">"spark.sql.streaming.checkpointLocation"</span><span class="hl opt">,</span> <span class="hl sng">"structuredStreamingCheckpoint"</span><span class="hl opt">)</span>
c<span class="hl opt">.</span>writeStream<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"delta"</span><span class="hl opt">).</span><span class="hl kwd">option</span><span class="hl opt">(</span><span class="hl sng">"path"</span><span class="hl opt">,</span> <span class="hl sng">"structuredStreamingDelta"</span><span class="hl opt">).</span>start
<span class="hl slc">// this alternative allows to apply any thing on the stream</span>
c<span class="hl opt">.</span>writeStream<span class="hl opt">.</span><span class="hl kwd">foreachBatch</span><span class="hl opt">((</span>b<span class="hl opt">,</span> i<span class="hl opt">) => {</span>
<span class="hl kwd">println</span><span class="hl opt">(</span>i<span class="hl opt">);</span> b<span class="hl opt">.</span>show
<span class="hl opt">}).</span>start
</code></pre>
<p>Each batch add a new parquet file with the content.</p>
<p><a href="https://docs.databricks.com/delta/delta-streaming.html">delta structured streaming</a></p>
<pre><code class="language-scala"><span class="hl kwa">val</span> in <span class="hl opt">=</span> spark<span class="hl opt">.</span>readStream<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"delta"</span><span class="hl opt">).</span><span class="hl kwd">option</span><span class="hl opt">(</span><span class="hl sng">"ignoreChanges"</span><span class="hl opt">,</span> <span class="hl sng">"true"</span><span class="hl opt">).</span><span class="hl kwd">option</span><span class="hl opt">(</span><span class="hl sng">"path"</span><span class="hl opt">,</span> <span class="hl sng">"structuredStreamingDelta"</span><span class="hl opt">).</span>load
<span class="hl slc">// update the input table with an other process</span>
DeltaTable<span class="hl opt">.</span><span class="hl kwd">forPath</span><span class="hl opt">(</span><span class="hl sng">"structuredStreamingDelta"</span><span class="hl opt">).</span><span class="hl kwd">updateExpr</span><span class="hl opt">(</span><span class="hl sng">"query = '91400+11457491401+54+rue+alphonse+daudet'"</span><span class="hl opt">,</span> <span class="hl kwd">Map</span><span class="hl opt">(</span><span class="hl sng">"query"</span> <span class="hl opt">-></span> <span class="hl sng">"'daudet'"</span><span class="hl opt">))</span>
<span class="hl slc">// this will resend all the data of every parquet file containing the updated row.</span>
</code></pre>
<p>In case of <code>delta</code> source, it does not handle <code>delete</code>. Hewever it
handles <code>updates</code>: every parquet files which are rewritten by the
delta acid method are sent again to the streamWriter. This will cause
duplicated record and they have to be handled by the datasource.</p>
<p>The main(?) advantage of using delta as a datasource for strucuted
streaming <code>vacuum</code> processes allow to reduce the number of files that
would increase batch after batch.</p>
<h2>RDD Operations</h2>
<p><a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions">pairRDD scaladoc</a>:
<a href="https://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations">RDD transformations</a>
Consider the below RDD:</p>
<pre><code class="language-scala"><span class="hl kwa">val</span> keysWithValuesList <span class="hl opt">=</span> <span class="hl kwd">Array</span><span class="hl opt">(</span><span class="hl sng">"foo=A"</span><span class="hl opt">,</span> <span class="hl sng">"foo=A"</span><span class="hl opt">,</span> <span class="hl sng">"foo=A"</span><span class="hl opt">,</span> <span class="hl sng">"foo=A"</span><span class="hl opt">,</span> <span class="hl sng">"foo=B"</span><span class="hl opt">,</span> <span class="hl sng">"bar=C"</span><span class="hl opt">,</span> <span class="hl sng">"bar=D"</span><span class="hl opt">,</span> <span class="hl sng">"bar=D"</span><span class="hl opt">)</span>
<span class="hl kwa">val</span> data <span class="hl opt">=</span> sc<span class="hl opt">.</span><span class="hl kwd">parallelize</span><span class="hl opt">(</span>keysWithValuesList<span class="hl opt">)</span>
</code></pre>
<h3>groupByKey</h3>
<pre><code class="language-scala">data
<span class="hl opt">.</span><span class="hl kwd">flatMap</span><span class="hl opt">(</span>line <span class="hl opt">=></span> line<span class="hl opt">.</span><span class="hl kwd">split</span><span class="hl opt">(</span><span class="hl sng">"="</span><span class="hl opt">))</span>
<span class="hl opt">.</span><span class="hl kwd">map</span><span class="hl opt">(</span>word <span class="hl opt">=> (</span>word<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">))</span>
<span class="hl opt">.</span><span class="hl kwd">groupByKey</span><span class="hl opt">()</span>
<span class="hl opt">.</span><span class="hl kwd">mapValues</span><span class="hl opt">(</span>_<span class="hl opt">.</span>sum<span class="hl opt">)</span>
<span class="hl opt">+---+---+</span>
<span class="hl opt">|</span> _1 <span class="hl opt">|</span> _2 <span class="hl opt">|</span>
<span class="hl opt">+---+---+</span>
<span class="hl opt">|</span> bar <span class="hl opt">|</span>
<span class="hl num">3</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> A <span class="hl opt">|</span>
<span class="hl num">4</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> B <span class="hl opt">|</span>
<span class="hl num">1</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> foo <span class="hl opt">|</span>
<span class="hl num">5</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> C <span class="hl opt">|</span>
<span class="hl num">1</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> D <span class="hl opt">|</span>
<span class="hl num">2</span> <span class="hl opt">|</span>
<span class="hl opt">+---+---+</span>
</code></pre>
<h3>reduceByKey</h3>
<pre><code class="language-scala">data
<span class="hl opt">.</span><span class="hl kwd">flatMap</span><span class="hl opt">(</span>line <span class="hl opt">=></span> line<span class="hl opt">.</span><span class="hl kwd">split</span><span class="hl opt">(</span><span class="hl sng">"="</span><span class="hl opt">))</span>
<span class="hl opt">.</span><span class="hl kwd">map</span><span class="hl opt">(</span>word <span class="hl opt">=> (</span>word<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">))</span>
<span class="hl opt">.</span><span class="hl kwd">reduceByKey</span><span class="hl opt">((</span>x<span class="hl opt">,</span> y<span class="hl opt">) => (</span>x <span class="hl opt">+</span> y<span class="hl opt">))</span>
<span class="hl opt">+---+---+</span>
<span class="hl opt">|</span> _1 <span class="hl opt">|</span> _2 <span class="hl opt">|</span>
<span class="hl opt">+---+---+</span>
<span class="hl opt">|</span> bar <span class="hl opt">|</span>
<span class="hl num">3</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> A <span class="hl opt">|</span>
<span class="hl num">4</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> B <span class="hl opt">|</span>
<span class="hl num">1</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> foo <span class="hl opt">|</span>
<span class="hl num">5</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> C <span class="hl opt">|</span>
<span class="hl num">1</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> D <span class="hl opt">|</span>
<span class="hl num">2</span> <span class="hl opt">|</span>
<span class="hl opt">+---+---+</span>
</code></pre>
<h3>aggregateByKey</h3>
<pre><code class="language-scala"><span class="hl kwa">def</span>
aggregateByKey<span class="hl opt">[</span>U<span class="hl opt">](</span>zeroValue<span class="hl opt">:</span> U<span class="hl opt">)(</span>seqOp<span class="hl opt">: (</span>U<span class="hl opt">,</span> V<span class="hl opt">)</span> ⇒ U<span class="hl opt">,</span> combOp<span class="hl opt">: (</span>U<span class="hl opt">,</span> U<span class="hl opt">)</span> ⇒ U<span class="hl opt">)(</span><span class="hl kwa">implicit</span> arg0<span class="hl opt">:</span> ClassTag<span class="hl opt">[</span>U<span class="hl opt">]):</span> RDD<span class="hl opt">[(</span>K<span class="hl opt">,</span> U<span class="hl opt">)]</span>
Aggregate the values of each key
<span class="hl opt">,</span> <span class="hl kwe">using</span> <span class="hl kwa">given</span> combine functions and a
neutral
<span class="hl sng">"zero value"</span><span class="hl opt">.</span>This function can
<span class="hl kwa">return</span> a different result
<span class="hl kwa">type</span><span class="hl opt">,</span> U
<span class="hl opt">,</span> than the
<span class="hl kwa">type</span> of
the values in
<span class="hl kwa">this</span> RDD
<span class="hl opt">,</span> V<span class="hl opt">.</span>Thus
<span class="hl opt">,</span> we need one
operation
<span class="hl kwa">for</span> merging a V into a U and one operation
<span class="hl kwa">for</span> merging two
U
<span class="hl sng">'s</span>
<span class="hl sng">, as in scala.TraversableOnce.The former operation is used</span>
<span class="hl sng">for</span>
<span class="hl sng"> merging values within a partition</span>
<span class="hl sng">, and the latter is used</span>
<span class="hl sng">for merging</span>
<span class="hl sng">values between partitions.To avoid memory allocation</span>
<span class="hl sng">, both of these</span>
<span class="hl sng">functions are allowed to modify and</span>
<span class="hl sng">return their first argument</span>
<span class="hl sng">instead of creating a new U.</span>
</code></pre>
<pre><code class="language-scala"><span class="hl kwa">val</span> kv <span class="hl opt">=</span> data<span class="hl opt">.</span><span class="hl kwd">map</span><span class="hl opt">(</span>_<span class="hl opt">.</span><span class="hl kwd">split</span><span class="hl opt">(</span><span class="hl sng">"="</span><span class="hl opt">)).</span><span class="hl kwd">map</span><span class="hl opt">(</span>v <span class="hl opt">=> (</span><span class="hl kwd">v</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">),</span> <span class="hl kwd">v</span><span class="hl opt">(</span><span class="hl num">1</span><span class="hl opt">))).</span><span class="hl kwd">cache</span><span class="hl opt">()</span>
<span class="hl kwa">val</span> initialCount <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl kwa">val</span> addToCounts <span class="hl opt">= (</span>n<span class="hl opt">:</span> Int<span class="hl opt">,</span> v<span class="hl opt">:</span> String<span class="hl opt">) =></span> n <span class="hl opt">+</span> <span class="hl num">1</span>
<span class="hl kwa">val</span> sumPartitionCounts <span class="hl opt">= (</span>p1<span class="hl opt">:</span> Int<span class="hl opt">,</span> p2<span class="hl opt">:</span> Int<span class="hl opt">) =></span> p1 <span class="hl opt">+</span> p2
<span class="hl kwa">val</span> countByKey <span class="hl opt">=</span> kv<span class="hl opt">.</span><span class="hl kwd">aggregateByKey</span><span class="hl opt">(</span>initialCount<span class="hl opt">)(</span>addToCounts<span class="hl opt">,</span> sumPartitionCounts<span class="hl opt">)</span>
<span class="hl opt">+---+---+</span>
<span class="hl opt">|</span> _1 <span class="hl opt">|</span> _2 <span class="hl opt">|</span>
<span class="hl opt">+---+---+</span>
<span class="hl opt">|</span> bar <span class="hl opt">|</span>
<span class="hl num">3</span> <span class="hl opt">|</span>
<span class="hl opt">|</span> foo <span class="hl opt">|</span>
<span class="hl num">5</span> <span class="hl opt">|</span>
<span class="hl opt">+---+---+</span>
</code></pre>
<h2>Catalyst</h2>
<ul>
<li><a href="https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html">catalyst optimizer explained</a></li>
<li><a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-Expression.html">sql expression doc</a></li>
</ul>
<h2>SparkSession</h2>
<p><a href="http://velvia.github.io/Spark-Concurrent-Fast-Queries/">spark concurrent fast queries</a></p>
<h2>Show session configs</h2>
<pre><code class="language-scala">spark<span class="hl opt">.</span><span class="hl kwd">sql</span><span class="hl opt">(</span><span class="hl sng">"set"</span><span class="hl opt">).</span><span class="hl kwd">show</span><span class="hl opt">()</span>
</code></pre>
<h2>Partition a table on disk</h2>
<p>This generates test data:</p>
<pre><code class="language-python"><span class="hl slc"># pip3 install fastparquet pandas numpy</span>
<span class="hl kwa">def</span> <span class="hl kwd">generate_data</span><span class="hl opt">(</span>n<span class="hl opt">,</span> seed<span class="hl opt">):</span>
<span class="hl kwa">import</span> pandas <span class="hl kwa">as</span> pd
<span class="hl kwa">import</span> numpy <span class="hl kwa">as</span> np
np<span class="hl opt">.</span>random<span class="hl opt">.</span><span class="hl kwd">seed</span><span class="hl opt">(</span>seed<span class="hl opt">)</span>
np_array <span class="hl opt">=</span> np<span class="hl opt">.</span>random<span class="hl opt">.</span><span class="hl kwd">random_sample</span><span class="hl opt">(</span>n<span class="hl opt">)</span> <span class="hl slc"># generates an array of M random values</span>
df <span class="hl opt">=</span> pd<span class="hl opt">.</span><span class="hl kwd">DataFrame</span><span class="hl opt">(</span>np_array<span class="hl opt">,</span> columns<span class="hl opt">=[</span><span class="hl sng">"x"</span><span class="hl opt">])</span>
df<span class="hl opt">[</span><span class="hl sng">"seed"</span><span class="hl opt">] =</span> seed
df<span class="hl opt">.</span><span class="hl kwd">reset_index</span><span class="hl opt">().</span><span class="hl kwd">to_parquet</span><span class="hl opt">(</span>f<span class="hl sng">"/tmp/bucket/part-{str(seed).zfill(5)}.parquet"</span><span class="hl opt">)</span>
df <span class="hl opt">=</span> spark<span class="hl opt">.</span>read<span class="hl opt">.</span><span class="hl kwd">parquet</span><span class="hl opt">(</span><span class="hl sng">"/tmp/bucket"</span><span class="hl opt">).</span><span class="hl kwd">withColumn</span><span class="hl opt">(</span><span class="hl sng">"part"</span><span class="hl opt">,</span> <span class="hl kwd">expr</span><span class="hl opt">(</span><span class="hl sng">"case when x > 0.5 then 'bob' else 'jim' end"</span><span class="hl opt">))</span>
<span class="hl slc"># this creates multiple json per partitions</span>
df<span class="hl opt">.</span><span class="hl kwd">repartition</span><span class="hl opt">(</span><span class="hl num">1000</span><span class="hl opt">,</span> <span class="hl sng">"part"</span><span class="hl opt">).</span>write<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"json"</span><span class="hl opt">).</span><span class="hl kwd">mode</span><span class="hl opt">(</span><span class="hl sng">"overwrite"</span><span class="hl opt">).</span><span class="hl kwd">partitionBy</span><span class="hl opt">(</span><span class="hl sng">"part"</span><span class="hl opt">).</span><span class="hl kwd">save</span><span class="hl opt">(</span><span class="hl sng">"/tmp/df"</span><span class="hl opt">)</span>
<span class="hl slc"># this creates one json per partition</span>
df<span class="hl opt">.</span><span class="hl kwd">repartition</span><span class="hl opt">(</span><span class="hl num">1000</span><span class="hl opt">,</span> <span class="hl sng">"part"</span><span class="hl opt">).</span>write<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"json"</span><span class="hl opt">).</span><span class="hl kwd">mode</span><span class="hl opt">(</span><span class="hl sng">"overwrite"</span><span class="hl opt">).</span><span class="hl kwd">partitionBy</span><span class="hl opt">(</span><span class="hl sng">"part"</span><span class="hl opt">).</span><span class="hl kwd">save</span><span class="hl opt">(</span><span class="hl sng">"/tmp/df"</span><span class="hl opt">)</span>
</code></pre>
<h2>Spark Plan</h2>
<p>The spark plans can be accessed from the <code>QueryExection</code> object.</p>
<p><a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-QueryExecution.html">see</a></p>
<h2>Generate data</h2>
<pre><code> val df = spark.range(0, 2000).selectExpr(
"id as _1",
"cast(id % 10 as byte) as _2",
"cast(id % 10 as short) as _3",
"cast(id % 10 as int) as _4",
"cast(id % 10 as float) as _5",
"cast(id % 10 as double) as _6",
"cast(id % 10 as decimal(20,0)) as _7",
"cast(id % 2 as boolean) as _8",
"cast(cast(1618161925000 + (id % 10) * 1000 * 60 * 60 * 24 as timestamp) as date) as _9",
"cast(1618161925000 + (id % 10) as timestamp) as _10"
)
</code></pre>
<h2>Estimate dataframe size in bytes</h2>
<pre><code>import org.apache.spark.util.SizeEstimator
SizeEstimator.estimate(b)
</code></pre>
<h2>Cache</h2>
<p>Columnar cache:</p>
<pre><code>spark.catalog.cacheTable("tableName") or dataFrame.cache()
spark.catalog.uncacheTable("tableName") or dataFrame.unpersist()
</code></pre>
<h2>Transform uuid as long</h2>
<p>There is no collision :</p>
<pre><code class="language-scala"><span class="hl kwa">def</span> <span class="hl kwd">stringToUUID</span><span class="hl opt">(</span>uuid<span class="hl opt">:</span> String<span class="hl opt">):</span> Long <span class="hl opt">= {</span>
<span class="hl kwa">return</span> java<span class="hl opt">.</span>util<span class="hl opt">.</span>UUID<span class="hl opt">.</span><span class="hl kwd">fromString</span><span class="hl opt">(</span>uuid<span class="hl opt">).</span><span class="hl kwd">getMostSignificantBits</span><span class="hl opt">() &</span> java<span class="hl opt">.</span>lang<span class="hl opt">.</span>Long<span class="hl opt">.</span>MAX_VALUE
<span class="hl opt">}</span>
<span class="hl kwa">val</span> stringToUUIDUdf <span class="hl opt">=</span> <span class="hl kwd">udf</span><span class="hl opt">((</span>uuid<span class="hl opt">:</span> String<span class="hl opt">) =></span> <span class="hl kwd">stringToUUID</span><span class="hl opt">(</span>uuid<span class="hl opt">))</span>
d<span class="hl opt">.</span><span class="hl kwd">withColumn</span><span class="hl opt">(</span><span class="hl sng">"user_id_long"</span><span class="hl opt">,</span> <span class="hl kwd">stringToUUIDUdf</span><span class="hl opt">(</span><span class="hl kwd">expr</span><span class="hl opt">(</span><span class="hl sng">"user_id"</span><span class="hl opt">))).</span>show
</code></pre>
<h2>ordored uuid</h2>
<ul>
<li><a href="https://muhamadhhassan.me/using-unique-identifiers-as-primary-key-in-mysql">ULID</a></li>
<li><a href="https://www.percona.com/blog/store-uuid-optimized-way/">UUID4</a></li>
</ul>
<h2>Read partitionned table</h2>
<pre><code class="language-scala">spark<span class="hl opt">.</span>read
<span class="hl opt">.</span><span class="hl kwd">option</span><span class="hl opt">(</span><span class="hl sng">"basePath"</span><span class="hl opt">,</span> <span class="hl sng">"/tmp/my_table"</span><span class="hl opt">)</span> <span class="hl slc">// will get field date</span>
<span class="hl opt">.</span><span class="hl kwd">option</span><span class="hl opt">(</span><span class="hl sng">"spark.sql.sources.partitionColumnTypeInference.enabled"</span><span class="hl opt">,</span> <span class="hl sng">"false"</span><span class="hl opt">)</span> <span class="hl slc">// won't infer date as date type</span>
<span class="hl opt">.</span><span class="hl kwd">format</span><span class="hl opt">(</span><span class="hl sng">"parquet"</span><span class="hl opt">)</span>
<span class="hl opt">.</span><span class="hl kwd">read</span><span class="hl opt">(</span><span class="hl sng">"/tmp/my_table/date=2022-01-01/*.parquet"</span><span class="hl opt">)</span>
</code></pre>
<h2>tuning</h2>
<ul>
<li>Increase shuffle block size to improve compression efficiently</li>
<li><a href="https://mageswaran1989.medium.com/spark-optimizations-for-advanced-users-spark-cheat-sheet-d74464618c20">several advices, including skew, shuffle and config</a></li>
<li><a href="https://trongkhoanguyen.com/spark/understand-the-shuffle-component-in-spark-core/">shuffle blog</a></li>
<li><a href="https://blog.scottlogic.com/2018/03/22/apache-spark-performance.html">performance</a></li>
<li><a href="http://hydronitrogen.com/apache-spark-shuffles-explained-in-depth.html">shuffle</a></li>
<li><a href="https://javarevisited.blogspot.com/2011/04/garbage-collection-in-java.html">GC</a></li>
<li><a href="https://github.com/jaceklaskowski/mastering-apache-spark-book/blob/master/spark-rpc-netty.adoc#settings">RPC</a></li>
</ul>
<h2>videos</h2>
<ul>
<li><a href="https://www.youtube.com/watch?v=7ooZ4S7Ay6Y&ab_channel=SparkSummit">spark core</a></li>
<li><a href="https://www.waitingforcode.com/apache-spark-sql/spark-sql-checkpoints/read">Checkpointing</a></li>
<li><a href="https://spark.apache.org/docs/3.3.1/monitoring.html#viewing-after-the-fact">Spark ui doc</a></li>
<li><a href="https://www.youtube.com/watch?v=te-GqfVFx8k&ab_channel=DevoxxFR">spark performances R. Luta</a></li>
<li><a href="https://www.youtube.com/watch?v=r-0SlaYj_Bk&ab_channel=DevoxxFR">spark ui video R. Luta</a></li>
<li><a href="https://www.databricks.com/blog/2023/04/18/spark-connect-available-apache-spark.html">Spark driver api</a></li>
</ul>
<h2>Spark datasources</h2>
<h3>Datasource v1</h3>
<ul>
<li><a href="https://www.slideshare.net/databricks/yin-huai-20150325meetupwithdemos">slides of the main dev</a></li>
</ul>
<ol>
<li><code>BaseRelation</code>: abstraction of a dataframe loaded from a datasource, provides the schema of the data</li>
<li><code>RelationProvider</code>: handle user's options and create a baseRelation</li>
<li><code>TableScan</code>: read the data, and construct the rows</li>
</ol>
<h3>Datasource v2</h3>
<ul>
<li><a href="https://blog.madhukaraphatak.com/categories/datasource-v2-series/">Blog spark2</a></li>
<li><a href="https://blog.madhukaraphatak.com/categories/datasource-v2-spark-three/">Spark3</a></li>
<li><a href="https://docs.google.com/document/d/1DDXCTCrup4bKWByTalkXWgavcPdvur8a4eEu8x1BzPM/edit#">Ref doc</a></li>
<li><a href="https://spot.io/blog/apache-spark-3-2-release-main-features-and-whats-new-for-spark-on-kubernetes/">Spark 3 release</a></li>
<li><a href="https://github.com/apache/hudi/blob/master/rfc/rfc-38/rfc-38.md">hudi dsv2 migration</a></li>
</ul>
<h2>udf java</h2>
<pre><code class="language-java">sqlContext<span class="hl opt">.</span><span class="hl kwd">udf</span><span class="hl opt">()</span>
<span class="hl opt">.</span><span class="hl kwd">register</span><span class="hl opt">(</span><span class="hl sng">"strlen"</span><span class="hl opt">,</span>
<span class="hl opt">(</span>String s<span class="hl opt">)-></span>s<span class="hl opt">.</span><span class="hl kwd">length</span><span class="hl opt">(),</span>DataTypes<span class="hl opt">.</span>StringType<span class="hl opt">);</span>
</code></pre>
<h2>Memory</h2>
<ul>
<li>From spark cookbook, Rishi Yadav</li>
<li>Heap is used for objects</li>
<li>60% is used for caching rdd= storage memory</li>
<li>40% for spark computation buffer =execution memory</li>
<li>execution can evict storage up to 50%. This is configured by <code>spark.memory.StorageFraction</code></li>
</ul>
<h3>GC</h3>
<ul>
<li>Use G1 for spark (vs cms and gc )</li>
<li>for streaming prefer cms</li>
<li><a href="https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html">GC spark benchmark</a></li>
<li><code>spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps</code></li>
</ul>
<h3>kryo</h3>
<pre><code class="language-scala">scala <span class="hl opt">></span> sc<span class="hl opt">.</span>getConf<span class="hl opt">.</span><span class="hl kwd">registerKryoClasses</span><span class="hl opt">(</span><span class="hl kwd">Array</span><span class="hl opt">(</span>classOf<span class="hl opt">[</span>com<span class="hl opt">.</span>
infoobjects<span class="hl opt">.</span>CustomClass1<span class="hl opt">],</span> classOf<span class="hl opt">[</span>com<span class="hl opt">.</span>infoobjects<span class="hl opt">.</span>CustomClass2<span class="hl opt">])</span>
</code></pre>
<pre><code>.set(“spark.kryoserializer.buffer.max”, “128m”)
.set(“spark.kryoserializer.buffer”, “64m”)
</code></pre>
<h2>Vocabulary</h2>
<ul>
<li>Spark produces one <code>job</code> per <code>action</code></li>
<li>a <code>job</code> contains one <code>stage</code> plus one other per <code>wide transformation</code></li>
<li>a <code>stage</code> contains several tasks, based on parallelism</li>
</ul>
<h2>web ui</h2>
<ul>
<li><code>http://web-ui/metrics/json</code> produces all the <code>metrics.properties</code> helps to monitor metrics
![[Screenshot_20230501-225402_NewPipe.png]]</li>
</ul>
<p>![[Screenshot_20230501-231933_NewPipe.png]]</p>
<p>![[Screenshot_20230501-222826_NewPipe.png]]</p>
<h2>optimization</h2>
<ul>
<li><code>spark.unsafe.offHeap</code> enable</li>
<li><code>spark.sql.codegen</code></li>
<li><code>spark.sql.inMemoryColumnarStorage.compressed</code></li>
<li><code>spark.cleaner.ttl</code> in seconds, to clean up any metadata (stages generated, tasks generated, and so on)</li>
<li><code>spark.default.parallelism</code> use 3 tasks per CPU core overall</li>
<li>spark.shuffle.manager=tungsten-sort (from spark in action)</li>
<li><a href="https://aws.amazon.com/blogs/big-data/best-practices-for-successfully-managing-memory-for-apache-spark-applications-on-amazon-emr/">bunch of optims here</a></li>
</ul>
<h2>Dates</h2>
<ul>
<li><a href="https://www.databricks.com/blog/2020/07/22/a-comprehensive-look-at-dates-and-timestamps-in-apache-spark-3-0.html">handling dates in spark 3</a></li>
</ul>
<h2>Skew joins</h2>
<ul>
<li><a href="https://github.com/apache/datafu/blob/1.6.0/datafu-spark/src/main/scala/datafu/spark/SparkDFUtils.scala#L290">datafu source code</a></li>
</ul>
<h2>Joins explained</h2>
<ul>
<li><a href="https://www.bigdatainrealworld.com/how-does-shuffle-sort-merge-join-work-in-spark/">shuffle merge sort join</a></li>
<li><a href="https://www.bigdatainrealworld.com/how-does-shuffle-hash-join-work-in-spark/">shuffle hash join</a></li>
<li><a href="https://www.bigdatainrealworld.com/how-does-broadcast-hash-join-work-in-spark/">broadcast hash join</a></li>
<li><a href="https://www.bigdatainrealworld.com/how-does-broadcast-nested-loop-join-work-in-spark/">broadcast nested loop</a></li>
<li><a href="https://github.com/apache/spark/pull/39571">bloom filter join hint</a></li>
<li><a href="https://github.com/apache/spark/pull/35789">bloom filter join</a></li>
<li><a href="https://docs.google.com/document/d/1foTkDSM91VxKgkEcBMsuAvEjNybjja-uHk-r3vtXWFE/mobilebasic">storage partition join</a>
To use shuffle hash join <code>set spark.sql.join.preferSortMergeJoin=false</code> is not enougth
use <code>df1.join(df2.hint("shuffle_hash"), ....)</code></li>
<li><a href="https://docs.google.com/document/d/1ndwynp1RPxaQ0dxykCWD-KW4MYxFWv-ojJzOMAei1S8/mobilebasic">Runtime filter</a></li>
<li><a href="https://github.com/apache/spark/pull/35789">row level runtime filter</a> <a href="https://docs.google.com/document/d/16IEuyLeQlubQkH8YuVuXWKo2-grVIoDJqQpHZrE7q04/edit#heading=h.4v65wq7vzy4q">also</a></li>
</ul>
<h2>Spark infer partition type</h2>
<p>see <code>spark.sql.sources.partitionColumnTypeInference.enabled</code></p>
<h2>Logging</h2>
<pre><code>add -Dlog4j.configuration=log4j.xml to spark.executor.extraJavaOptions
so the executors pick it up.
</code></pre>
<h2>Bucketing</h2>
<p>Spark bucketing in hive metastore:</p>
<pre><code>spark.sql.sources.schema.bucketCol.0 | tbl_id
spark.sql.sources.provider | parquet
numFiles | 2
spark.sql.sources.schema.numBuckets | 4
spark.sql.create.version | 3.2.3
spark.sql.sources.schema | {"type":"struct","fields":[{"name":"tbl_id","type":"long","nullable":true,"metadata":{}},{"name":"kafka_datetime","type":"timestamp","nullable":true,"metadata":{}},{"name":"event_timesta
:"timestamp","nullable":true,"metadata":{}},{"name":"version","type":"string","nullable":true,"metadata":{}},{"name":"event_date","type":"string","nullable":true,"metadata":{}},{"name":"event_hour","type":"string","nullable":true
":{}}]}
totalSize | 3824
spark.sql.sources.schema.sortCol.0 | event_timestamp
spark.sql.sources.schema.numBucketCols | 1
spark.sql.sources.schema.numSortCols | 1
transient_lastDdlTime | 1683402911
</code></pre>
<ul>
<li><a href="https://towardsdatascience.com/best-practices-for-bucketing-in-spark-sql-ea9f23f7dd53">spark bucketing article</a></li>
<li><a href="https://medium.com/@deepa.account/comparison-between-spark-and-hive-bucketing-827ec41fe58d">hive / spark bucketing</a></li>
<li><a href="https://trino.io/blog/2019/05/29/improved-hive-bucketing.html">trino support</a></li>
<li><a href="https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html#engine-versions-reference-0003-improvements-and-new-features">athena v3 support</a></li>
<li><a href="https://github.com/apache/hudi/pull/7834">hudi bucket index support bulk insert</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/HUDI/RFC+-+29%3A+Hash+Index">hudi hash index rfc</a></li>
<li><a href="https://github.com/apache/hudi/pull/3173">hudi hash index pr</a></li>
<li><a href="https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md">Hudi consistent hash index</a></li>
</ul>
<h2>Spark parquet</h2>
<ul>
<li><a href="https://db-blog.web.cern.ch/blog/luca-canali/2017-09-performance-analysis-cpu-intensive-workload-apache-spark">spark parquet is cpu bound</a></li>
</ul>
<h2>Alluxio</h2>
<ul>
<li><a href="https://www.uber.com/en-FR/blog/speed-up-presto-with-alluxio-local-cache/">Uber blog</a></li>
<li></li>
</ul>
<h2>Spark parquet inspection</h2>
<pre><code class="language-bash">pip3 <span class="hl kwc">install</span> ipython
parquet<span class="hl kwb">-tools</span> inspect
</code></pre>
<h2>streaming</h2>
<ul>
<li><a href="https://www.waitingforcode.com/apache-spark-structured-streaming/dynamic-resource-allocation-structured-streaming/read">Dynamic allocation</a></li>
</ul>
<h2>spark listener analyser</h2>
<p><a href="https://www.uber.com/en-FR/blog/spark-analysers-catching-anti-patterns-in-spark-apps/">Spark uber optim</a></p>
<h2>graph</h2>
<ul>
<li><a href="https://neo4j.com/blog/cypher-for-apache-spark/">Cypher</a></li>
</ul>
<h2>Metastore configuration</h2>
<ul>
<li><a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-hive-metastore.html">according to this doc, both jdbc and thrift can be used to connect spark to HMS</a></li>
<li><a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-external.html">using only database for metastore</a></li>
</ul>
<h2>spark s3 commiter</h2>
<ul>
<li>the default commiter writes first to <code>_temporary</code> then move files sequentially from the driver to the target folder</li>
<li>the <code>_magic</code> commiter <code>spark.hadoop.fs.s3a.bucket.all.committer.magic.enabled</code>, use a magic protocol to directly write
within the target by using a <code>__magic</code> folder</li>
<li>you must set <code>s3a</code> as the prefix</li>
</ul>
<h2>stage barrier</h2>
<ul>
<li><a href="https://youtu.be/daXEp4HmS-E?t=1785">Explanation youtube</a></li>
<li><a href="http://beginnershadoop.com/2020/04/02/barrier-execution-mode-in-spark/">more on barrier</a></li>
<li><a href="https://docs.google.com/document/d/1GvcYR6ZFto3dOnjfLjZMtTezX0W5VYN9w1l4-tQXaZk/edit">design document</a></li>
<li><a href="https://luminousmen.com/post/spark-tips-partition-tuning">use localCheckpoint</a></li>
<li><a href="https://www.waitingforcode.com/apache-spark-sql/spark-sql-checkpoints/read">Checkpoints blog</a></li>
</ul>
<h2>shuffle</h2>
<ul>
<li><a href="https://medium.com/swlh/revealing-apache-spark-shuffling-magic-b2c304306142">Shuffle blog</a></li>
</ul>
<h2>native function</h2>
<ul>
<li><a href="https://neapowers.com/apache-spark/native-functions-catalyst-expressions/">Blog on how to</a></li>
</ul>
<h2>Options</h2>
<p><a href="https://bigdataenthusiast.medium.com/spark-configurations-precedence-2014535ea5be">Options precedence</a>:</p>
<blockquote>
<p>Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file.</p>
</blockquote>
<ul>
<li>Number rows per files: <code>spark.sql.files.maxRecordsPerFile</code></li>
</ul>
<h2>kylin spark backend</h2>
<ul>
<li><a href="https://apachecon.com/acasia2021/sessions/1133.html">Apache con kylin 4</a></li>
<li><a href="https://apachecon.com/acasia2021/sessions/1131.html">kyuubi</a></li>
</ul>
<h2>connectors</h2>
<ul>
<li><a href="https://github.com/awslabs/emr-dynamodb-connector">Dynamodb</a></li>
</ul>
<h2>hive</h2>
<ul>
<li><a href="https://stackoverflow.com/a/56482010/3865083">Creation like managed/external</a></li>
</ul>
<h2>Parquet integration</h2>
<ul>
<li><a href="https://cloudsqale.com/2021/03/19/spark-reading-2022-why-the-number-of-tasks-can-be-much-larger-than-the-number-of-row-groups/">number tasks / row groups</a></li>
</ul>
<h2>Spark case sensitive</h2>
<ul>
<li>
<p><a href="https://github.com/apache/spark/blob/branch-3.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L830C50-L830C91">spark.sql.hive.caseSensitiveInferenceMode</a></p>
</li>
<li>
<p><a href="https://blog.allegro.tech/2015/08/spark-kafka-integration.html">Task not serialisable</a>and <a href="https://medium.com/@falguerasaleix/spark-infamous-task-not-serializable-error-8474ff1c8132">also here</a></p>
</li>
<li>
<p><a href="https://mkuthan.github.io/blog/2016/01/29/spark-kafka-integration2/">Kafka spark integration</a></p>
</li>
</ul>
<h2>kubernetes</h2>
<p>-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</p>
<p>spark.executor.memoryOverhead</p>
<p>spark.memory.offHeap.enabled": "true"</p>
<pre><code>"spark.kubernetes.memoryOverheadFactor": "0.2"
</code></pre>
<p>About xms on Tarn ans k8s:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/781#issuecomment-1250345153</p>
<h2>Profiling</h2>
<ul>
<li>
<p><a href="https://github.com/uber-common/jvm-profiler?uclick_id=26becb41-8e48-40d9-9829-4d9dbc9e4b89">Uber profiler</a></p>
</li>
<li>
<p>spark.sparkContext.getMemoryStatus().inUse()</p>
</li>
</ul>
<h1>If you are planning to run Spark for a long
time on a cluster, you may wish to enable spark.cleaner.ttl. By default, Spark
does not clean up any metadata (stages generated, tasks generated, and so on); set this
to a non-zero value in seconds to clean up the metadata after that length of time.</h1>
<ul>
<li></li>
<li><a href="https://michaelheil.medium.com/understanding-common-performance-issues-in-apache-spark-deep-dive-data-spill-7cdba81e697e">blog about memory management</a></li>
<li><a href="https://michaelheil.medium.com/understanding-common-performance-issues-in-apache-spark-deep-dive-data-spill-7cdba81e697e">general spark optim</a></li>
<li><a href="https://michaelheil.medium.com/understanding-common-performance-issues-in-apache-spark-deep-dive-data-skew-e962909f3d07">data skew</a></li>
<li><a href="https://medium.com/@DalvikRohan/apache-spark-memory-allocation-deep-dive-fbe93dd8f69d">deep dive memory</a></li>
</ul>
<ol>
<li>Overhead memory -> spark.executor.memoryOverhead</li>
<li>Heap Memory -> spark.executor.memory</li>
<li>Off Heap Memory -> spark.memory.offHeap.size</li>
<li>Pyspark Memory -> spark.executor.pyspark.memory</li>
</ol>
<pre><code>Off-heap memory is a great way to reduce GC pauses because it's not in the GC's scope. However, it brings an overhead of
serialization and deserialization. The latter in its turn makes that the off-heap data can be sometimes put onto heap
memory and hence be exposed to GC. Also, the new data format brought by Project Tungsten (array of bytes) helps to
reduce the GC overhead. These 2 reasons make that the use of off-heap memory in Apache Spark applications should be
carefully planned and, especially, tested.
</code></pre>
<h3>file path</h3>
<p>One can access file path from <code>_metadata</code> field or the dedicated function. <a href="https://stackoverflow.com/questions/39868263/spark-load-data-and-add-filename-as-dataframe-column#39872830">See</a>
<a href="https://stackoverflow.com/questions/39868263/spark-load-data-and-add-filename-as-dataframe-column#39872830">See</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Spark — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2024-01-14 18:33</time> </div>https://blog.parisni.com/wiki/keycloakKeycloak2023-04-23T09:59:00+00:00<div id="generated-toc"> </div><hr> <p>#security
#matrix</p>
<h1>Keycloak</h1>
<ul>
<li>OpenID Connect or SAML 2.0 IdPs</li>
<li>LDAP and Active Directory</li>
<li>OpenID Connect, OAuth 2.0 and SAML 2.0</li>
<li><a href="https://www.keycloak.org/getting-started/getting-started-docker">docker install</a></li>
</ul>
<h2>matrix support</h2>
<ul>
<li><a href="https://areweoidcyet.com/">matrix dont support openidconnect</a></li>
<li><a href="https://matrix-org.github.io/synapse/latest/usage/configuration/user_authentication/single_sign_on/index.html">here says it supports both</a></li>
<li><a href="https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#sso">synapse sso configuration</a></li>
</ul>
<h2>cryptpad support</h2>
<ul>
<li><a href="https://github.com/xwiki-labs/cryptpad/pull/749">Old SSO PR</a></li>
<li><a href="https://cryptpad.fr/kanban/#/2/kanban/view/PLM0C3tFWvYhd+EPzXrbT+NxB76Z5DtZhAA5W5hG9wo/">Sso is in their roadmap</a></li>
</ul>
<h2>jupyter support</h2>
<ul>
<li><a href="https://jupyterhub.readthedocs.io/en/stable/tutorial/getting-started/authenticators-users-basics.html#use-oauthenticator-to-support-oauth-with-popular-service-providers">existing providers</a></li>
<li><a href="https://oauthenticator.readthedocs.io/en/latest/reference/api/gen/oauthenticator.generic.html">Generic Oauth2</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Keycloak — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-23 09:59</time> </div>https://blog.parisni.com/wiki/RocksdbRocksdb2023-04-21T22:21:00+00:00<div id="generated-toc"> </div><hr> <h1>Rocksdb</h1>
<h2>Blogs</h2>
<p><a href="https://artem.krylysov.com/blog/2023/04/19/how-rocksdb-works/">used at datadog</a></p>
<ul>
<li>Lsmtree</li>
<li>forked feom leveldb</li>
<li>bloom filter optional</li>
</ul>
<p>#database</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Rocksdb — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-21 22:21</time> </div>https://blog.parisni.com/wiki/evolutionEvolution2022-02-19T18:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<span class="w3-tag w3-red tag">email</span>
<span class="w3-tag w3-red tag">linux</span>
<h1>Evolution</h1>
<p>Install it with flatpak. See <a href="../ubuntu/#flatpak">ubuntu flatpak</a>. So far, this
is the most reliable way and it has uptodate installed packages</p>
<h2>Installation</h2>
<p>The below procedure adds emacs key-binding and dark mode for evolution installed with flatpak.</p>
<pre><code class="language-bash">flatpak <span class="hl kwc">install</span> flathub org.gnome.Evolution
flatpak <span class="hl kwc">install</span> org.gtk.Gtk3theme.Yaru<span class="hl kwb">-dark</span>
<span class="hl kwc">cp</span> .config<span class="hl opt">/</span>gtk<span class="hl kwb">-3</span>.0<span class="hl opt">/</span>settings.ini <span class="hl opt">/</span>home<span class="hl opt">/</span>nparis<span class="hl opt">/</span>.var<span class="hl opt">/</span>app<span class="hl opt">/</span>org.gnome.Evolution<span class="hl opt">/</span>config<span class="hl opt">/</span>gtk<span class="hl kwb">-3</span>.0<span class="hl opt">/</span>
<span class="hl kwc">cp</span> .config<span class="hl opt">/</span>gtk<span class="hl kwb">-3</span>.0<span class="hl opt">/</span>gtk.css <span class="hl opt">/</span>home<span class="hl opt">/</span>nparis<span class="hl opt">/</span>.var<span class="hl opt">/</span>app<span class="hl opt">/</span>org.gnome.Evolution<span class="hl opt">/</span>config<span class="hl opt">/</span>gtk<span class="hl kwb">-3</span>.0<span class="hl opt">/</span>
</code></pre>
<pre><code class="language-bash"><span class="hl kwc">cat</span> .config<span class="hl opt">/</span>gtk<span class="hl kwb">-3</span>.0<span class="hl opt">/</span>settings.ini
<span class="hl opt">[</span>Settings<span class="hl opt">]</span>
gtk<span class="hl kwb">-application-prefer-dark-theme</span><span class="hl opt">=</span>true
gtk<span class="hl kwb">-key-theme-name</span> <span class="hl opt">=</span> Emacs
gtk<span class="hl kwb">-icon-theme-name</span> <span class="hl opt">=</span> <span class="hl sng">"Yaru-dark"</span>
gtk<span class="hl kwb">-can-change-accels</span> <span class="hl opt">=</span> <span class="hl num">1</span>
</code></pre>
<p>You can edit the gtk emacs key bindings here (added alt-backspace)</p>
<pre><code class="language-bash"><span class="hl kwc">cat</span> .config<span class="hl opt">/</span>gtk<span class="hl kwb">-3</span>.0<span class="hl opt">/</span>gtk.css
@binding<span class="hl kwb">-set</span> gtk<span class="hl kwb">-emacs-text-entry</span>
<span class="hl opt">{</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>b"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>logical<span class="hl kwb">-positions</span><span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<shift><ctrl>b"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>logical<span class="hl kwb">-positions</span><span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>f"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>logical<span class="hl kwb">-positions</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<shift><ctrl>f"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>logical<span class="hl kwb">-positions</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<alt>b"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>words<span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<shift><alt>b"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>words<span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<alt>f"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>words<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<shift><alt>f"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>words<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>a"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>paragraph<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<shift><ctrl>a"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>paragraph<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>e"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>paragraph<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<shift><ctrl>e"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>paragraph<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>w"</span> <span class="hl opt">{</span> <span class="hl sng">"cut-clipboard"</span> <span class="hl opt">() };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>y"</span> <span class="hl opt">{</span> <span class="hl sng">"paste-clipboard"</span> <span class="hl opt">() };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>d"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>chars<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<alt>d"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>word<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>k"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>paragraph<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<alt>backslash"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>whitespace<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<alt>space"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>whitespace<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">)</span>
<span class="hl sng">"insert-at-cursor"</span> <span class="hl opt">(</span><span class="hl sng">" "</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<alt>KP_Space"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>whitespace<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">)</span>
<span class="hl sng">"insert-at-cursor"</span> <span class="hl opt">(</span><span class="hl sng">" "</span><span class="hl opt">) };</span>
<span class="hl opt">/*</span>
<span class="hl opt">*</span> Some non<span class="hl kwb">-Emacs</span> keybindings people are attached to
<span class="hl opt">*/</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>u"</span> <span class="hl opt">{</span> <span class="hl sng">"move-cursor"</span> <span class="hl opt">(</span>paragraph<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">,</span> <span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>paragraph<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>h"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>chars<span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<ctrl>w"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>word<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">) };</span>
<span class="hl kwb">bind</span> <span class="hl sng">"<alt>BackSpace"</span> <span class="hl opt">{</span> <span class="hl sng">"delete-from-cursor"</span> <span class="hl opt">(</span>word<span class="hl kwb">-ends</span><span class="hl opt">,</span> <span class="hl kwb">-1</span><span class="hl opt">) };</span>
<span class="hl opt">}</span>
</code></pre>
<p>Uncomment the accels entries and override the Evolution key-map. This shall be
done when Evolution is not running, otherwise the editions will be lost after
restart:</p>
<pre><code class="language-bash"><span class="hl kwc">cat</span> ~<span class="hl opt">/</span>.var<span class="hl opt">/</span>app<span class="hl opt">/</span>org.gnome.Evolution<span class="hl opt">/</span>config<span class="hl opt">/</span>evolution<span class="hl opt">/</span>accels
<span class="hl opt">;</span> evolution.bin GtkAccelMap rc<span class="hl kwb">-file</span> <span class="hl opt">-*-</span> scheme <span class="hl opt">-*-</span>
<span class="hl opt">;</span> this <span class="hl kwc">file</span> is an automated accelerator map dump
<span class="hl opt">(</span>gtk_accel_path <span class="hl sng">"<Actions>/mail/mail-next"</span> <span class="hl sng">"j"</span><span class="hl opt">)</span>
<span class="hl opt">(</span>gtk_accel_path <span class="hl sng">"<Actions>/mail/mail-print"</span> <span class="hl sng">""</span><span class="hl opt">)</span>
<span class="hl opt">(</span>gtk_accel_path <span class="hl sng">"<Actions>/mail/mail-previous"</span> <span class="hl sng">"k"</span><span class="hl opt">)</span>
</code></pre>
<h2>Configuration</h2>
<ul>
<li>Preference > Mail Preference > Plain Text Mode > Show Plain Text If Present</li>
<li>View > Preview > Vertical View</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Evolution — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-02-19 18:09</time> </div>https://blog.parisni.com/wiki/wireguardWireguard2022-03-20T18:00:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">security</span>
<span class="w3-tag w3-red tag">vpn</span>
<span class="w3-tag w3-red tag">dns</span></p>
<h1>Wireguard</h1>
<p>A VPN can be usefull in several contexts:</p>
<ul>
<li>secure access to internet from un-trusted networks</li>
<li>secure access to private resources (ssh, local website)</li>
</ul>
<h2>Setup wireguard</h2>
<ol>
<li>install wireguard on both server / client</li>
<li>choose a private ip for each</li>
<li>configure both server / client (create pub/priv keys and bellow configuration)</li>
<li>run <code>wg-quick up wg0</code> on both</li>
</ol>
<p>Client config:</p>
<pre><code>cat /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <the client private key>
Address = <a client private ip>/32
DNS = <a local DNS server ip>
[Peer]
PublicKey = <the server public key>
Endpoint = <the server public ip>:<the server wireguard port>
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25
</code></pre>
<p>Server config:</p>
<pre><code>cat /etc/wireguard/wg0.conf
[Interface]
PostUp = sysctl -w net.ipv4.ip_forward=1; iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o <network interface> -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o <network interface> -j MASQUERADE
PrivateKey = <the server private key>
Address = <a server private ip>/32
ListenPort = <the server wireguard port>
[Peer]
PublicKey = <the client public key>
AllowedIPs = <a client private ip>/32 # the ip address in the VPN network of the client you just created
</code></pre>
<h2>Protect ssh with wireguard</h2>
<ol>
<li>configure sshd on the server with the bellow configuration</li>
<li>limit access to ssh port from the wireguard server private ip</li>
<li>disable internet access to the ssh port</li>
</ol>
<p>sshd config:</p>
<pre><code>cat /etc/ssh/sshd_config
ListenAddress <a server private ip>
Port 7184
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding yes
</code></pre>
<p>ufw config:</p>
<pre><code>cat /etc/ufw/applications.d/sshd
[SSHD]
title=SSHD
description=SSHD
ports=7184/tcp
</code></pre>
<pre><code>ufw allow from <a client private ip>/32 to any app "SSHD"
</code></pre>
<h2>Setup a local DNS</h2>
<p>Install and Setup <code>unbound</code></p>
<pre><code class="language-bash"><span class="hl kwc">cat</span> <span class="hl opt">/</span>etc<span class="hl opt">/</span>unbound<span class="hl opt">/</span>unbound.conf
server<span class="hl opt">:</span>
chroot<span class="hl opt">:</span> <span class="hl sng">""</span>
num<span class="hl kwb">-threads</span><span class="hl opt">:</span> <span class="hl num">1</span>
verbosity<span class="hl opt">:</span> <span class="hl num">1</span>
root<span class="hl kwb">-hints</span><span class="hl opt">:</span> <span class="hl sng">"/etc/unbound/root.hints"</span>
trust<span class="hl kwb">-anchor-file</span><span class="hl opt">:</span> <span class="hl sng">"/etc/unbound/trusted-key.key"</span>
interface<span class="hl opt">:</span> <span class="hl num">0.0.0.0</span>
max<span class="hl kwb">-udp-size</span><span class="hl opt">:</span> <span class="hl num">3072</span>
access<span class="hl kwb">-control</span><span class="hl opt">:</span> <span class="hl num">0.0.0.0</span><span class="hl opt">/</span><span class="hl num">0</span> refuse
access<span class="hl kwb">-control</span><span class="hl opt">:</span> <span class="hl num">127.0.0.1</span> allow
access<span class="hl kwb">-control</span><span class="hl opt">: <</span>private ip<span class="hl opt">>/</span><span class="hl num">24</span> allow
private<span class="hl kwb">-address</span><span class="hl opt">: <</span>private ip<span class="hl opt">>/</span><span class="hl num">24</span>
hide<span class="hl kwb">-identity</span><span class="hl opt">:</span> <span class="hl kwc">yes</span>
hide<span class="hl kwb">-version</span><span class="hl opt">:</span> <span class="hl kwc">yes</span>
harden<span class="hl kwb">-glue</span><span class="hl opt">:</span> <span class="hl kwc">yes</span>
harden<span class="hl kwb">-dnssec-stripped</span><span class="hl opt">:</span> <span class="hl kwc">yes</span>
unwanted<span class="hl kwb">-reply-threshold</span><span class="hl opt">:</span> <span class="hl num">10000000</span>
val<span class="hl kwb">-log-level</span><span class="hl opt">:</span> <span class="hl num">1</span>
cache<span class="hl kwb">-min-ttl</span><span class="hl opt">:</span> <span class="hl num">1800</span>
cache<span class="hl kwb">-max-ttl</span><span class="hl opt">:</span> <span class="hl num">14400</span>
prefetch<span class="hl opt">:</span> <span class="hl kwc">yes</span>
prefetch<span class="hl kwb">-key</span><span class="hl opt">:</span> <span class="hl kwc">yes</span>
</code></pre>
<p>Configure wireguard client with DNS (see above)</p>
<p>Check for dnsleak:</p>
<ul>
<li><a href="https://dnsleak.com/">dnsleak</a></li>
</ul>
<p>Check for DNSSEC:</p>
<ul>
<li><code>unbound-host -C /etc/unbound/unbound.conf -v sigok.verteiltesysteme.net</code></li>
<li><code>unbound-host -C /etc/unbound/unbound.conf -v sigfail.verteiltesysteme.net</code></li>
</ul>
<p>Use resolv.conf:</p>
<pre><code>cat /etc/resolvconf.conf
name_servers=<the vpn server private ip>
resolv_conf_options="trust-ad"
</code></pre>
<p>Update resolv.conf:</p>
<pre><code>resolvconf -u
</code></pre>
<h2>Protect local website</h2>
<p>On the nginx side:</p>
<pre><code class="language-bash"><span class="hl slc"># replace</span>
listen <span class="hl num">443</span> ssl<span class="hl opt">;</span>
<span class="hl slc"># with</span>
listen <span class="hl opt"><</span>a private vpn ip<span class="hl opt">>:</span><span class="hl num">443</span> ssl<span class="hl opt">;</span>
</code></pre>
<p>Also for letsencrypt to use the acme-challenge (otherwise it won't succeed):</p>
<pre><code>server {
server_name *.domain.org;
listen [::]:80;
listen 80;
location /.well-known/acme-challenge/ {
root /var/www;
try_files $uri =404;
}
access_log off;
return 301 https://$host$request_uri;
}
</code></pre>
<p>On the OVH dns side (or whatever):</p>
<p>On the unbound side:</p>
<pre><code> local-data: "the.website.com IN A <a private vpn ip>"
</code></pre>
<p>Now nginx only provide those website to people using the VPN and the DNS redirects to it.</p>
<h2>Generate a qrcode for your android wireguard client</h2>
<ol>
<li>generate a priv/pub pair key on the server side</li>
<li>create a client conf on the server side</li>
<li>add the Peer on the server file</li>
<li>reload the wireguard service <code>systemctl reload wg-quick@wg0.service</code></li>
<li>generate a qrcode <code>qrencode -t ansiutf8 -o /tmp/<user>.png -t png -r /etc/wireguard/clients/<user>.conf</code></li>
</ol>
<h2>Make sshd daemon work at reboot</h2>
<p>The default systemd sshd requires network to start. When sshd listen to
wireguard private ip, then add those in the systemd unit file:</p>
<pre><code>After=network.target wg-quick@wg0.service
Requires=sys-devices-virtual-net-wg0.device
</code></pre>
<h1>References</h1>
<ul>
<li><a href="https://www.scaleway.com/en/docs/tutorials/install-wireguard/">scaleway tutorial</a></li>
<li><a href="https://www.procustodibus.com/blog/2020/12/wireguard-terminology/">terminology</a></li>
<li><a href="https://jamesledger.net/securing-ssh-with-a-jump-host/">ssh jump host</a></li>
<li><a href="https://www.linuxquestions.org/questions/linux-security-4/openvpn-ssh-listenaddress-4175607641-print/">mailing list</a></li>
<li><a href="https://security.stackexchange.com/questions/242586/logging-into-remote-server-via-ssh-using-vpn-wireguard-is-it-foolproof">protecting ssh with vpn</a></li>
<li><a href="https://dnsleak.com/">dnsleak</a></li>
<li><a href="https://wiki.archlinux.org/title/Unbound">unbound archlinux</a></li>
<li><a href="https://www.asrivas.me/blog/configuring-unbound-as-a-local-dns-server/">unbound local</a></li>
<li><a href="https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup/">vpn setup</a></li>
<li><a href="https://serversideup.net/generating-wireguard-qr-codes-for-fast-mobile-deployments/">generate qrcode</a></li>
<li><a href="https://unix.stackexchange.com/a/624988/276549">sshd after wireguard</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Wireguard — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-03-20 18:00</time> </div>https://blog.parisni.com/wiki/dnsDNS2022-01-01T14:45:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<span class="w3-tag w3-red tag">network</span>
<h1>DNS</h1>
<h2>Unbound</h2>
<h3>Add dns filter</h3>
<p><a href="https://github.com/notracking/hosts-blocklists/issues/308#issuecomment-728269408">Notracking</a> maintain a tracker list.</p>
<p>modify unbound.conf:</p>
<pre><code> control-enable: yes
control-use-cert: "no"
</code></pre>
<p>Add a systemctl timer:</p>
<p>/etc/systemd/system/notracking.service</p>
<pre><code>[Unit]
Description=No tracking
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/etc/unbound/update_blacklist.sh
#ExecStop=/bin/kill -9 ${MAINPID}
WorkingDirectory=/usr/local/etc/unbound/
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=read-only
ProtectSystem=strict
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @memlock @module \
@mount @obsolete @privileged @reboot @resources @setuid \
@swap @raw-io
ReadOnlyPaths=/
ReadWritePaths=/usr/local/etc/unbound/
PrivateDevices=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
[Install]
WantedBy=multi-user.target
</code></pre>
<p>/etc/systemd/system/notracking.timer</p>
<pre><code>[Unit]
Description=Run notrackking daily
[Timer]
OnCalendar=*-*-* 7:55:00
Persistent=true
[Install]
WantedBy=timers.target
</code></pre>
<pre><code class="language-bash">systemctl <span class="hl kwb">enable</span> notracking.timer
systemctl start notracking.timer
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=DNS — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-01-01 14:45</time> </div>https://blog.parisni.com/wiki/readingReading2020-05-11T23:33:00+00:00<div id="generated-toc"> </div><hr> <div id="refman"> <div id="refman-sidebar"> <div id="generated-toc"> </div> </div> <div id="refman-main">
<p><span class="w3-tag w3-red tag">books</span>
<span class="w3-tag w3-red tag">method</span></p>
<h1>Reading</h1>
<p>From Bernard Stiegler, applied for 5 years in prison:</p>
<ol>
<li>wake up: poesie (stefane malarmé)</li>
<li>phylosophy until lunch</li>
<li>synthesis of morning reading</li>
<li>reading of synthesis of previous day</li>
<li>reading of marcel proust</li>
</ol>
</div></div><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Reading — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-05-11 23:33</time> </div>https://blog.parisni.com/wiki/tidalTidal2020-07-15T21:48:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-orange tag">music</span>
<span class="w3-tag w3-orange tag">haskell</span>
<span class="w3-tag w3-orange tag">emacs</span></p>
<h1>Tidal</h1>
<h2>Installation</h2>
<h3>installation de supercollider</h3>
<pre><code class="language-bash">nix<span class="hl kwb">-env -i</span> supercollider
</code></pre>
<p>Il est alors possible de le lancer via:</p>
<pre><code class="language-bash">$<span class="hl opt">></span> scide
</code></pre>
<h3>installation de superdirt</h3>
<pre><code>Quarks.checkForUpdates({Quarks.install("SuperDirt", "v1.1.1"); thisProcess.recompile()})
SuperDirt.start
# produit cette erreur...
# Booting server 'localhost' on address 127.0.0.1:57110.
# could not initialize audio.
# Server 'localhost' exited with exit code 0.
</code></pre>
<h3>installation de tidal emacs</h3>
<pre><code class="language-bash">nix<span class="hl kwb">-env -iA</span> nixos.haskellPackages.tidal
</code></pre>
<p>Ajouter dans init.el:</p>
<pre><code class="language-lisp"><span class="hl slc">;; tidal</span>
<span class="hl opt">(</span><span class="hl kwa">add-to-list</span> <span class="hl opt">'</span>load-path <span class="hl sng">"~/.emacs.d/plugins/tidal/"</span><span class="hl opt">)</span>
<span class="hl opt">(</span><span class="hl kwa">require</span> <span class="hl opt">'</span>tidal<span class="hl opt">)</span>
<span class="hl opt">(</span><span class="hl kwa">require</span> <span class="hl opt">'</span>haskell-mode<span class="hl opt">)</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Tidal — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-07-15 21:48</time> </div>https://blog.parisni.com/wiki/javaJava2023-06-01T23:18:00+00:00<div id="generated-toc"> </div><hr> <p>#programming</p>
<h1>Java</h1>
<h2>Heap dump</h2>
<h3>Generate a Heap dump</h3>
<h4>On demand</h4>
<p>The jcmd tool produce a binary heap dump</p>
<pre><code class="language-bash">jcmd <span class="hl opt"><</span>pid<span class="hl opt">></span> GC.heap_dump <span class="hl opt"><</span><span class="hl kwc">file</span><span class="hl kwb">-path</span><span class="hl opt">></span>
</code></pre>
<h4>On crash</h4>
<p>The below option does not add overhead, and using it in production is a good idea.</p>
<pre><code class="language-bash">java <span class="hl kwb">-XX</span><span class="hl opt">:+</span>HeapDumpOnOutOfMemoryError <span class="hl kwb">-XX</span><span class="hl opt">:</span>HeapDumpPath<span class="hl opt">=<</span><span class="hl kwc">file</span><span class="hl kwb">-or-dir-path</span><span class="hl opt">></span>
</code></pre>
<h3>Analyse a thread dump</h3>
<p>Eclipse Memory Annalyser allows to analyse the resulting dump files. <a href="http://memoryanalyzer.blogspot.com/2008/05/automated-heap-dump-analysis-finding.html">this tutorial</a> explains well how to use the tool.</p>
<h2>Reading</h2>
<ul>
<li>Building Microservices: Designing Fine-Grained Systems, Second Edition (O’Reilly Media, 2021)</li>
<li>Microservices Patterns by Chris Richardson (Manning, 2018)</li>
</ul>
<h2>Running tests with maven and set log level</h2>
<pre><code>MAVEN_OPTS='-Dorg.slf4j.simpleLogger.defaultLogLevel=error' mvn test
</code></pre>
<h2>Set maven version</h2>
<pre><code>mvn versions:set -DgroupId=* -DartifactId=* -DnewVersion=0.13.0 -DgenerateBackupPoms=false
</code></pre>
<h2>java concepts</h2>
<ul>
<li>A Java virtual machine (JVM) is an abstract machine. It is a specification that provides a runtime environment in which Java bytecode can be executed.</li>
<li>The Java Runtime Environment (JRE) is used to provide a runtime environment. It is the implementation of the JVM that physically exists. It contains a set of libraries and other files that the JVM uses at runtime. Oracle bought Sun Microsystems in 2010; since then, new versions and patches have been actively provided. Other companies, such as IBM, provide their own implementations of the JVM.</li>
<li>The Java Development Kit (JDK) contains the JRE, documentation, and Java tools. This is what Java developers install on their machines. A smart editor like IntelliJ IDEA or Eclipse will require you to provide the location of the JDK so classes and documentation can be loaded and used during development.</li>
</ul>
<h2>jmx</h2>
<ul>
<li>Java Management Extensions (JMX) framework</li>
<li><a href="https://www.jmdoudoux.fr/java/dej/chap-jmx.htm">jmx tutorial</a></li>
<li>visualVM is a jdk tool to explore jmx metrics of the application running in the respective jvm</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Java — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-06-01 23:18</time> </div>https://blog.parisni.com/wiki/archlinuxArchlinux2020-03-08T02:08:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span></p>
<h1>Archlinux</h1>
<h2>Aur package</h2>
<p><a href="https://linuxhint.com/aur_arch_linux/">source</a></p>
<ul>
<li>look for a package on the <a href="https://aur.archlinux.org/">aur</a></li>
<li>clone it <code>git clone https://aur.archlinux.org/<pkgName>.git</code></li>
<li>build it <code>makepkg</code></li>
<li>install it <code>makepkg -sri</code></li>
<li>optional: delete it with pacman</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Archlinux — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-03-08 02:08</time> </div>https://blog.parisni.com/wiki/systemdSystemd2023-07-30T22:54:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span></p>
<h1>Systemd</h1>
<h2>Systemd-run</h2>
<p>Run with the current user environment, with cgroup quota :</p>
<pre><code class="language-bash">systemd<span class="hl kwb">-run --user --scope</span> sh <span class="hl kwb">-c</span> <span class="hl sng">'echo hello ; sleep 10 ; echo</span> <span class="hl ipl">$(pwd)</span><span class="hl sng">'</span>
systemd<span class="hl kwb">-run --user -pCPUQuota</span><span class="hl opt">=</span><span class="hl num">200</span><span class="hl opt">%</span> <span class="hl kwb">--scope</span> sh <span class="hl kwb">-c</span> <span class="hl sng">'echo hello ; sleep 10 ; echo</span> <span class="hl ipl">$(pwd)</span><span class="hl sng">'</span>
</code></pre>
<h2>Journalctl</h2>
<h3>Filter</h3>
<p>It centralizes all the systemd logs. You can filter by many criteria. It has a
handy command completion:</p>
<pre><code class="language-bash">journalctl _COMM<span class="hl opt">=</span>prometheus <span class="hl kwb">--since</span> <span class="hl sng">'2021-11-01 11:00'</span> <span class="hl kwb">--until</span> <span class="hl sng">'2021-11-01 12:00'</span>
</code></pre>
<h3>Relative time filter</h3>
<pre><code class="language-bash">journalctl _COMM<span class="hl opt">=</span>prometheus <span class="hl kwb">--since -1h</span>
</code></pre>
<h3>Follow</h3>
<p>This allows to see incoming logs:</p>
<pre><code class="language-bash">journalctl _COMM<span class="hl opt">=</span>sshd <span class="hl kwb">-f</span>
</code></pre>
<h3>Kernel</h3>
<pre><code class="language-bash">journalctl <span class="hl kwb">-k</span>
</code></pre>
<h3>Boot</h3>
<pre><code class="language-bash">journalctl <span class="hl kwb">-b</span>
</code></pre>
<h3>Priority</h3>
<pre><code class="language-text">0: Emergency. The system is unusable.
1: Alert. A condition has been flagged that should be corrected immediately.
2: Critical. This covers crashes, coredumps, and significant failures in primary applications.
3: Error. An error has been reported, but it is not considered severe.
4: Warning. Brings a condition to your attention that, if ignored, may become an error.
5: Notice. Used to report events that are unusual, but not errors.
6: Information. Regular operational messages. These do not require action.
7: Debug. Messages put into applications to make it easier for them to debug them.
</code></pre>
<h2>Edit systemd files</h2>
<p>The system files are located in <code>/usr/lib/systemd/system</code> and are overwriten when the system get updated.</p>
<p>There is two options to make your own unit:</p>
<ol>
<li>create a systemd file in <code>/etc/systemd/system</code>: it will overwrite the system's one</li>
<li>run <code>systemctl edit <service-name></code>: it will add content into
<code>/etc/systemd/system/<service>.service.d/override.conf</code>. Note that it does not
need daemon-reload and can only append config rows.</li>
</ol>
<h2>Cleanup journal</h2>
<pre><code class="language-sh">journalctl <span class="hl kwb">--disk-usage</span>
journalctl <span class="hl kwb">--vacuum-size</span><span class="hl opt">=</span><span class="hl num">2</span>G
</code></pre>
<h2>Get logs from service</h2>
<pre><code class="language-sh">journalctl <span class="hl kwb">-u</span> matrix<span class="hl kwb">-synapse</span>.service <span class="hl kwb">-f -n</span> <span class="hl num">1000</span>
</code></pre>
<h2>Redirect docker to journal</h2>
<pre><code class="language-yaml"> <span class="hl kwa">logging:</span>
<span class="hl kwa">driver:</span> journald
<span class="hl kwa">options:</span>
<span class="hl kwa">tag:</span> my<span class="hl opt">-</span>docker<span class="hl opt">-</span>app
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Systemd — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-07-30 22:54</time> </div>https://blog.parisni.com/wiki/kubernetesKubernetes2021-11-23T18:06:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">kubernetes</span>
<span class="w3-tag w3-red tag">docker</span></p>
<h1>Kubernetes</h1>
<h2>K3s</h2>
<h3>Install</h3>
<pre><code class="language-bash">sudo apt <span class="hl kwb">-y</span> <span class="hl kwc">install</span> open<span class="hl kwb">-iscsi</span>
curl <span class="hl kwb">-sfL</span> https<span class="hl opt">://</span>get.k3s.io <span class="hl opt">|</span> sh <span class="hl opt">-</span>
sudo <span class="hl kwc">cp</span> <span class="hl opt">/</span>etc<span class="hl opt">/</span>rancher<span class="hl opt">/</span>k<span class="hl num">3</span>s<span class="hl opt">/</span>k3s.yaml .kube<span class="hl opt">/</span>config
sudo kubectl apply <span class="hl kwb">-f</span> https<span class="hl opt">://</span>raw.githubusercontent.com<span class="hl opt">/</span>longhorn<span class="hl opt">/</span>longhorn<span class="hl opt">/</span>v1.1<span class="hl num">.2</span><span class="hl opt">/</span>deploy<span class="hl opt">/</span>longhorn.yaml
</code></pre>
<h3>Stop/Start</h3>
<pre><code class="language-bash"><span class="hl opt">/</span>usr<span class="hl opt">/</span>local<span class="hl opt">/</span>bin<span class="hl opt">/</span>k3s<span class="hl kwb">-killall</span>.sh
systemctl start k3s
</code></pre>
<h3>Run together with other remote</h3>
<pre><code class="language-bash"><span class="hl kwb">export</span> KUBECONFIG<span class="hl opt">=</span>~<span class="hl opt">/</span>.kube<span class="hl opt">/</span>config<span class="hl kwb">-k3s</span>
<span class="hl slc"># use tools</span>
kubectl ...
helm ...
k9s ...
</code></pre>
<h2>Helm</h2>
<h3>Pass variables</h3>
<pre><code class="language-bash">helm <span class="hl kwc">install</span> airflow apache<span class="hl kwb">-airflow</span><span class="hl opt">/</span>airflow <span class="hl kwb">--namespace</span> airflow <span class="hl kwb">--values</span> custom<span class="hl kwb">-values</span>.yml
</code></pre>
<h2>Kubectl</h2>
<h3>Volumes</h3>
<pre><code class="language-bash"><span class="hl slc"># List the pvc</span>
kubectl get pvc <span class="hl kwb">-n</span> airflow
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
logs<span class="hl kwb">-airflow-worker-0</span> Bound pvc<span class="hl kwb">-4e9fdfef-7a16-4299-b68a-89ee92a12589</span> <span class="hl num">100</span>Gi RWO <span class="hl kwb">local-path</span> <span class="hl num">8</span>m15s
data<span class="hl kwb">-airflow-postgresql-0</span> Bound pvc<span class="hl kwb">-94c585fd-451f-4728-a790-f6b61a2f2b6e</span> <span class="hl num">8</span>Gi RWO <span class="hl kwb">local-path</span> <span class="hl num">8</span>m15s
redis<span class="hl kwb">-db-airflow-redis-0</span> Bound pvc<span class="hl kwb">-95d2ca98-0fe3-424b-a939-6ffd23edacc7</span> <span class="hl num">1</span>Gi RWO <span class="hl kwb">local-path</span> <span class="hl num">8</span>m15s
<span class="hl slc"># Delete one pvc</span>
kubectl delete pvc logs<span class="hl kwb">-airflow-worker-0 -n</span> airflow
<span class="hl slc"># Delete ALL the pvc</span>
kubectl delete pvc <span class="hl kwb">-n</span> airflow <span class="hl kwb">--all</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Kubernetes — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-23 18:06</time> </div>https://blog.parisni.com/wiki/scalaScala2023-05-02T09:48:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">development</span></p>
<h1>Scala</h1>
<h2>metals LSP</h2>
<p>Yet it is fully compatible with scala 2.12.10 version. The 2.11
version only support limited set of features.</p>
<p>The setup is well explained in the metals documentation. There is both
<code>metals-emacs</code> and something to putt within the <code>ini.el</code> file.</p>
<p>This can take advantage of <code>magit</code> library and integrates well with it.</p>
<h2>Logging</h2>
<p><code>scala-logging</code> is a great tool. It's a wrapper of slf4j, which
handles both log4j and logback. In order to work with logback you have
to add those dependencies<code>logback-classic</code> and <code>log4j-over-slf4j</code>.</p>
<h3>logback</h3>
<p>In order to configure logback, put a logback.xml in the
<code>src/main/resources</code> folder. For tests, the
<code>src/test/resources/logback.xml</code> file will overwrite the main
behavior.</p>
<p>The below configuration allows to specify different levels for libraries:</p>
<pre><code class="language-xml"><span class="hl kwa"><configuration></span>
<span class="hl kwa"><appender</span> <span class="hl kwb">name</span>=<span class="hl sng">"CONSOLE"</span> <span class="hl kwb">class</span>=<span class="hl sng">"ch.qos.logback.core.ConsoleAppender"</span><span class="hl kwa">></span>
<span class="hl kwa"><encoder></span>
<span class="hl kwa"><pattern></span>%date{ISO8601} %highlight([%-<span class="hl num">5l</span>evel]): %cyan(%logger{<span class="hl num">15</span>}) %msg%n<span class="hl kwa"></pattern></span>
<span class="hl kwa"></encoder></span>
<span class="hl kwa"></appender></span>
<span class="hl kwa"><root</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span><span class="hl kwa">></span>
<span class="hl kwa"><appender-ref</span> <span class="hl kwb">ref</span>=<span class="hl sng">"CONSOLE"</span><span class="hl kwa">/></span>
<span class="hl kwa"></root></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.apache.hadoop"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.apache.spark"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.apache.hadoop.hive"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.apache.spark.repl.SparkILoop$SparkILoopInterpreter"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"apache.spark.repl.SparkIMain$exprTyper"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.eclipse.jetty.util.component.AbstractLifeCycle"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.eclipse.jetty"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"io.frama.parisni.spark.postgres"</span> <span class="hl kwb">level</span>=<span class="hl sng">"INFO"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.apache.spark.repl.SparkILoop$SparkILoopInterpreter"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"org.apache.spark.repl.SparkIMain$exprType"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"><logger</span> <span class="hl kwb">name</span>=<span class="hl sng">"com.opentable.db.postgres.embedded.EmbeddedPostgres"</span> <span class="hl kwb">level</span>=<span class="hl sng">"WARN"</span> <span class="hl kwb">includeLocation</span>=<span class="hl sng">"true"</span><span class="hl kwa">/></span>
<span class="hl kwa"></configuration></span>
</code></pre>
<h3>log4j</h3>
<p>It is unmaintained since 2015. However apache spark is still based on
it. The <a href="https://beuss.developpez.com/tutoriels/java/jakarta/log4j/">excellent
introduction</a>
explains how inheritance works and how library logging can have
dedicated logging level than the application itself.</p>
<h2>Read a string file</h2>
<pre><code>import scala.io.Source
Source.fromFile("example.txt", "UTF-8").mkstring
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Scala — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-05-02 09:48</time> </div>https://blog.parisni.com/wiki/browser-webWeb Browser2023-12-21T22:23:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">web</span></p>
<h1>Web Browser</h1>
<h2>Ungoogled-Chromium</h2>
<p>It is not possible to install extensions thought the chrome store. You
need to install them manually:</p>
<p><code>https://clients2.google.com/service/update2/crx?response=redirect&acceptformat=crx2,crx3&prodversion=[VERSION]&x=id%3D[EXTENSION_ID]%26installsource%3Dondemand%26uc</code></p>
<ul>
<li>VERSION: for example 84.0</li>
<li>EXTENSION_ID: the hash can be found on the extension url</li>
</ul>
<p>It has several cons:</p>
<ul>
<li>cannot change default shortkeys (like any chromium based browser but vivaldi)</li>
<li>cannot install extensions easily</li>
</ul>
<h2>Firefox</h2>
<p>It has several cons:</p>
<ul>
<li>~~cannot change default shortkeys~~</li>
<li>~~might be~~ is possible <a href="https://superuser.com/a/1747680">with autoconfig</a></li>
</ul>
<h3>Extensions</h3>
<ul>
<li>vimium-c</li>
<li>dark-reader</li>
<li>browserpass</li>
<li>auto refresh page</li>
<li>tabs are windows</li>
<li>ublock origin</li>
<li>consent-o-matic</li>
<li>privacy badger</li>
</ul>
<h3>Remove key bindings</h3>
<p>My FF keys config:</p>
<ul>
<li>
<p><a href="https://www.reddit.com/r/firefox/comments/tb6o42/tip_you_can_disable_the_alt_key_menu_bar_keyboard/#:~:text=The%20direct%20about%3Aconfig%20filter,and%20toggle%20it%20to%20False%2e">disable alt for menu</a></p>
</li>
<li>
<p><code>ui.key.menuAccessKeyFocuses=false</code></p>
</li>
<li>
<p><code>browser.gesture.swipe.left=</code></p>
</li>
<li>
<p><code>browser.gesture.swipe.right=</code></p>
</li>
<li>
<p><a href="https://superuser.com/a/1747680/375607">tips from here, also explained how to find command keys</a></p>
</li>
<li>
<p>in order to enable the console toolbox:</p>
</li>
</ul>
<pre><code>devtools.chrome.enabled = true
devtools.debugger.remote-enabled = true
</code></pre>
<p><code>/usr/lib/firefox/browser/defaults/preferences/syspref.js</code></p>
<pre><code>// This file can be used to configure global preferences for Firefox
// Example: Homepage
//pref("browser.startup.homepage", "http://www.weebls-stuff.com/wab/");
pref("general.config.filename", "config.js");
pref("general.config.obscure_value", 0);
pref("general.config.sandbox_enabled", false);
</code></pre>
<p><code>/usr/lib/firefox/config.js</code></p>
<pre><code>// IMPORTANT DE GARDER CETTE LIGNE
try {
let { classes: Cc, interfaces: Ci, manager: Cm } = Components;
const Services = globalThis.Services;
const {SessionStore} = Components.utils.import('resource:///modules/sessionstore/SessionStore.jsm');
function ConfigJS() { Services.obs.addObserver(this, 'chrome-document-global-created', false); }
ConfigJS.prototype = {
observe: function (aSubject) { aSubject.addEventListener('DOMContentLoaded', this, {once: true}); },
handleEvent: function (aEvent) {
let document = aEvent.originalTarget; let window = document.defaultView; let location = window.location;
if (/^(chrome:(?!\/\/(global\/content\/commonDialog|browser\/content\/webext-panels)\.x?html)|about:(?!blank))/i.test(location.href)) {
if (window._gBrowser) {
// Place your keyboard shortcut changes here
// ...
// ...
//'focusURLBar2',
//'goBackKb',
//'goForwardKb',
const keys = [
'addBookmarkAsKb', 'bookmarkAllTabsKb', 'focusURLBar', 'goBackKb2', 'goForwardKb2', 'goHome', 'key_aboutProcesses', 'key_close', 'key_closeWindow', 'key_gotoHistory', 'key_newNavigator', 'key_newNavigatorTab', 'key_openAddons', 'key_openDownloads', 'key_privatebrowsing', 'key_quitApplication', 'key_sanitize', 'key_savePage', 'key_screenshot', 'key_search', 'key_search2', 'key_selectAll', 'key_selectLastTab', 'key_selectTab1', 'key_selectTab2', 'key_selectTab3', 'key_selectTab4', 'key_selectTab5', 'key_selectTab6', 'key_selectTab7', 'key_selectTab8', 'key_showAllTabs', 'key_switchTextDirection', 'key_toggleMute', 'key_togglePictureInPicture', 'key_toggleReaderMode', 'key_undoCloseTab', 'key_undoCloseWindow', 'key_viewInfo', 'key_viewSource', 'key_wrCaptureCmd', 'key_wrToggleCaptureSequenceCmd', 'manBookmarkKb', 'openFileKb', 'printKb', 'showAllHistoryKb', 'viewBookmarksSidebarKb', 'viewBookmarksToolbarKb',
];
for (key of keys) {
window.document.getElementById(key).remove();
}
}
}
}
};
if (!Services.appinfo.inSafeMode) { new ConfigJS(); }
} catch(ex) {};
</code></pre>
<h3>Remove tabs</h3>
<ul>
<li><a href="https://superuser.com/a/1424494/375607">by using userChrome.css</a></li>
<li><a href="https://www.userchrome.org/help-with-userchrome-css.html">help about userchrome</a></li>
</ul>
<pre><code class="language-shell">vim /home/nparis/.mozilla/firefox-esr/vkj2hxae.default-esr115/chrome/userChrome.css
/* hides the native tabs */
#TabsToolbar {
visibility: collapse;
}
</code></pre>
<h3>Disable share screen popup</h3>
<blockquote>
<p>in about:config set privacy.webrtc.legacyGlobalIndicator to false and privacy.webrtc.hideGlobalIndicator to true</p>
</blockquote>
<ul>
<li><a href="https://www.reddit.com/r/firefox/comments/k20ddt/how_to_disable_the_pop_up_showing_mic_and_cam/">hide it in the about:config</a></li>
</ul>
<h3>Pre-install extensions</h3>
<ul>
<li><a href="https://support.mozilla.org/en-US/kb/deploying-firefox-with-extensions">use firefox enterprise</a></li>
<li><a href="https://packages.debian.org/buster/firefox-esr">install firefox enterprise</a></li>
</ul>
<h2>Vivaldi</h2>
<p>It has several cons:</p>
<ul>
<li>proprietary software</li>
<li>slow</li>
</ul>
<h2>Qutebrowser</h2>
<p>https://qutebrowser.org/doc/install.html#tox</p>
<p>pros:</p>
<ul>
<li>fairly good adblocker</li>
<li>readline shortkeys in both command / input</li>
<li>minimalist</li>
</ul>
<p>cons:</p>
<ul>
<li>does not discover all pleroma links</li>
</ul>
<h3>Search with “startpage”</h3>
<p>This provides a GET search in clear</p>
<pre><code class="language-python">c<span class="hl opt">.</span>url<span class="hl opt">.</span>searchengines <span class="hl opt">= {</span><span class="hl sng">'DEFAULT'</span><span class="hl opt">:</span> <span class="hl sng">'https://www.startpage.com/sp/search?query={}&prfh=disable_open_in_new_windowEEE0N1Ndisable_video_family_filterEEE1N1Nenable_post_methodEEE0N1Nenable_proxy_safety_suggestEEE1N1Nenable_stay_controlEEE0N1Ngeo_mapEEE0N1Nlang_homepageEEEs%2Fblak%2Ffr%2FN1NlanguageEEEenglishN1Nlanguage_uiEEEfrancaisN1Nnum_of_resultsEEE10N1Nother_iaEEE1N1Nsearch_results_regionEEEallN1NsuggestionsEEE0N1Nwikipedia_iaEEE1N1Nwt_unitEEEcelsius&language=english&t=blak&lui=francais&cat=web&sc=wb8Rggx5esKu20&abp=-1'</span><span class="hl opt">}</span>
</code></pre>
<h3>Disable HTS on chromium</h3>
<ul>
<li>https://www.thesslstore.com/blog/clear-hsts-settings-chrome-firefox/</li>
</ul>
<h2>Vimium-c (firefox & chromium)</h2>
<p>Together with gtk-emacs binding only those are necessary:</p>
<pre><code># delete backward character
mapkey <c-b:i> <f3rmc>
map <f3rmc> editText run="move,backward,character,exec,move,"
# delete backward character
mapkey <c-k:i> <f3rfor>
map <f3rfor> editText run="extend,forward,lineboundary,exec,delete,"
# delete backward character
mapkey <c-u:i> <f3rback>
map <f3rback> editText run="extend,backward,lineboundary,exec,delete,"
</code></pre>
<p>Add mapkey :</p>
<pre><code>mapkey <a-d:i> <f3rmf>
map <f3rmf> editText run="extend,forward,word,exec,delete,"
# delete forward character
mapkey <c-d:i> <f3rmcf>
map <f3rmcf> editText run="extend,forward,character,exec,delete,"
# delete backward word
mapkey <a-backspace:i> <f3rm>
map <f3rm> editText run="extend,backward,word,exec,delete,"
# delete backward character
mapkey <c-h:i> <f3rmc>
map <f3rmc> editText run="extend,backward,character,exec,delete,"
mapkey <c-e:i> <f3end>
map <f3end> editText run="move,forward,lineboundary,exec,move,"
mapkey <a-f:i> <f3wend>
map <f3wend> editText run="move,forward,word,exec,move,"
mapkey <a-b:i> <f3wbig>
map <f3wbig> editText run="move,backward,word,exec,move,"
# not used because mapped to select text. use home instead
# mapkey <c-a> <f3bg>
# map <f3bg> editText run="move,backward,lineboundary,exec,move,"
mapkey <c-k:i> <f3rmline>
map <f3rmline> editText run="extend,forward,lineboundary,exec,delete,"
mapkey <c-n:i> <s-down>
mapkey <c-p:i> <s-up>
</code></pre>
<h2>Chromium</h2>
<h3>Install chromium</h3>
<p>Install <a href="https://askubuntu.com/a/1206153">without snap on ubuntu 20.04</a> to make
browser-pass (password-store) working.</p>
<ul>
<li>install the extension</li>
<li>also install the <code>sudo apt install webext-browserpass</code></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Web Browser — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-21 22:23</time> </div>https://blog.parisni.com/wiki/blogBlog2023-04-21T22:28:00+00:00<div id="generated-toc"> </div><hr> <h1>Blog</h1>
<h2>Content</h2>
<ul>
<li>books</li>
<li>work notes encrypted</li>
<li>private notes</li>
<li>public wiki</li>
<li>blog files</li>
</ul>
<h2>Multi-devices</h2>
<ul>
<li>All this in stores in git</li>
<li>sync between différent devices</li>
<li>use gitcrypt https://github.com/AGWA/git-crypt</li>
</ul>
<h2>Layout</h2>
<ul>
<li>/blog/
<ul>
<li>/Books/</li>
<li>/site/wiki/</li>
<li>/site/games/</li>
<li>/site/blog/</li>
<li>/site/pictures/</li>
<li>/site/reading/</li>
<li>/site/film/</li>
<li>/private/</li>
<li>/encrypted/</li>
</ul>
</li>
</ul>
<p>#knowlege</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Blog — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-21 22:28</time> </div>https://blog.parisni.com/wiki/qutebrowserQutebrowser2021-08-19T16:06:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">data</span></p>
<h1>Qutebrowser</h1>
<h2>Use password-store</h2>
<ul>
<li><a href="https://wiki.archlinux.org/title/Qutebrowser">documentation</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Qutebrowser — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-19 16:06</time> </div>https://blog.parisni.com/wiki/passwordPassword2021-12-28T23:03:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">security</span></p>
<h1>Password</h1>
<p>So far, password-store is the best option, for cli and browsers.</p>
<h2>Password-sore</h2>
<p>It's a bash tool, based on git, gpg, xclip and other linux tools. By combining them,
it inherits the best feature :</p>
<ul>
<li><code>+</code>: the cli is very efficient (path navigatation, auto-complete, clipboard expiration)</li>
<li><code>+</code>: decentralized (each device has the whole)</li>
<li><code>+</code>: can store almost anything (password, texts, list...)</li>
<li><code>+</code>: integrates with browsers, android ...</li>
<li><code>-</code>: not great for team and password sharing (share a private key)</li>
</ul>
<h2>Vaultwarden</h2>
<ul>
<li><code>+</code>: great for team and password sharing</li>
<li><code>+</code>: share media (images, video...)</li>
<li><code>+</code>: integrates with browsers, android ...</li>
<li><code>-</code>: centralized architecture (a database/API design), trouble if it gets down</li>
<li><code>-</code>: has a bad cli (no auto-complete, no clipboard integration)</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Password — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-28 23:03</time> </div>https://blog.parisni.com/wiki/mariadbMariadb2021-10-23T15:06:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">database</span></p>
<h1>Mariadb</h1>
<h2>Réaliser un dump simple</h2>
<pre><code class="language-bash">mysqldump database <span class="hl opt">> /</span>tmp<span class="hl opt">/</span>backup<span class="hl kwb">-file</span>.sql
</code></pre>
<h2>Charger un dump simple</h2>
<pre><code class="language-bash">mysql database <span class="hl opt">< /</span>tmp<span class="hl opt">/</span>backup<span class="hl kwb">-file</span>.sql
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mariadb — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-10-23 15:06</time> </div>https://blog.parisni.com/wiki/mysqlMySQL2022-02-07T22:43:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">database</span></p>
<h1>MySQL</h1>
<h2>Connect from cli</h2>
<pre><code class="language-bash">mysql <span class="hl kwb">-u</span> <span class="hl opt"><</span>user<span class="hl opt">></span> <span class="hl kwb">-P</span> <span class="hl num">3306</span> <span class="hl kwb">-p</span>
</code></pre>
<h2>Dump the database</h2>
<pre><code class="language-bash">mysqldump <span class="hl kwb">-u</span> <span class="hl opt"><</span>user<span class="hl opt">></span> <span class="hl kwb">-P</span> <span class="hl num">3306</span> <span class="hl kwb">--all-databases -p</span> <span class="hl opt">> </</span>path<span class="hl opt">/</span>to<span class="hl opt">/</span><span class="hl kwc">file</span><span class="hl opt">></span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=MySQL — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-02-07 22:43</time> </div>https://blog.parisni.com/wiki/git-cryptGit-Crypt2023-04-21T22:51:00+00:00<div id="generated-toc"> </div><hr> <p>#versioning
#security</p>
<h1>Git-Crypt</h1>
<p>First of all: install <a href="https://www.agwa.name/projects/git-crypt/">git-crypt</a></p>
<h2>Initialize an encrypted repository</h2>
<p><a href="https://github.com/AGWA/git-crypt#using-git-crypt">see documentation</a></p>
<h2>Work on a encrypted repository</h2>
<p>Either <strong>import a gpg key</strong> ...:</p>
<pre><code class="language-bash">git clone <span class="hl opt"><</span>the repos<span class="hl opt">></span>
<span class="hl kwb">cd</span> <span class="hl opt"><</span>the repos<span class="hl opt">></span>
git unlock
</code></pre>
<p>...or use the <strong>symetric key</strong> (see below):</p>
<pre><code class="language-bash">git clone <span class="hl opt"><</span>the repos<span class="hl opt">></span>
<span class="hl kwb">cd</span> <span class="hl opt"><</span>the repos<span class="hl opt">></span>
git unlock <span class="hl opt">/</span>path<span class="hl opt">/</span>to<span class="hl opt">/</span>key
</code></pre>
<h2>Export a gpg key</h2>
<pre><code class="language-bash">gpg <span class="hl kwb">--export-secret-keys</span> YOUR_ID_HERE <span class="hl opt">></span> private.key
</code></pre>
<h2>Import a gpg key</h2>
<pre><code class="language-bash">gpg <span class="hl kwb">--import</span> private.key
gpg <span class="hl kwb">--edit-key</span> YOUR_ID_HERE
<span class="hl slc"># type “trust”</span>
<span class="hl slc"># type “5”</span>
<span class="hl slc"># type “save”</span>
</code></pre>
<h2>Export a symetric key</h2>
<pre><code class="language-bash">git<span class="hl kwb">-crypt export-key</span> <span class="hl opt">/</span>path<span class="hl opt">/</span>to<span class="hl opt">/</span>key
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Git-Crypt — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-21 22:51</time> </div>https://blog.parisni.com/wiki/mu4eMu4e2021-01-17T15:57:00+00:00<div id="generated-toc"> </div><hr> <div id="refman"> <div id="refman-sidebar"> <div id="generated-toc"> </div> </div> <div id="refman-main">
<p><span class="w3-tag w3-red tag">email</span>
<span class="w3-tag w3-red tag">emacs</span></p>
<h1>Mu4e</h1>
<p>Important de mettre en place la fonction "pinentry emacs" pour que le
mot de passe soit demandé (dans le cas d´emacs-terminal mode)</p>
<h2>Crypt / Decrypt</h2>
<p>Installer les paquets:</p>
<ul>
<li>pinentry</li>
</ul>
<p>Editer le fichier <code>.gnupg/gpg-agent.conf</code> pour ajouter :</p>
<pre><code>pinentry-program /usr/bin/pinentry-emacs
default-cache-ttl 9999
max-cache-ttl 9999
allow-emacs-pinentry
</code></pre>
<p>Redemarrer <code>gpg-agent</code>:</p>
<p>IMPORTANT: cela ne fonctionne que si une instance d´emacs est en cours (pas avec emacsclient)</p>
<pre><code class="language-bash">pkill gpg<span class="hl kwb">-agent</span>
</code></pre>
<p>Attention, toute utilisation de gpg ulterieure demandera dans emacs et
non dans le tty !</p>
<h2>Sidebar</h2>
<p>Le paquet <code>emacs-overview</code> permet d´afficher la liste des sous-maildir
de manière identique à mutt. Il faut penser à raffraichir les comptes
(par exemple: <cc> <cu>).</cu></cc></p>
<p>Cela fonctionne bien à condition d´avoir ajouté une seconde windows
dans laquelle afficher le contenu des maildir.</p>
<h2>Draft</h2>
<h3>Ajourner un email</h3>
<p><c-c> <c-d> envoie l'email dans les drafts.</c-d></c-c></p>
<h3>Reprendre un email draft</h3>
<p><c e=""> permet de ré-éditer un email</c></p>
<h3>Gestion des listes</h3>
<p>The 1.3.x mu4e version provide an answer list feature.</p>
<h2>Marks</h2>
<pre><code class="language-txt">mark for/as | keybinding | description
-------------+-------------+------------------------------
'something' | *, <insert> | mark now, decide later
delete | D, <delete> | delete
flag | + | mark as 'flagged' ('starred')
move | m | move to some maildir
read | ! | mark as read
refile | r | mark for refiling
trash | d | move to the trash folder
untrash | = | remove 'trash' flag
unflag | - | remove 'flagged' mark
unmark | u | remove mark at point
unmark all | U | remove all marks
unread | ? | marks as unread
action | a | apply some action
</code></pre>
<p>Then use <code>x</code> to apply any mark</p>
<h2>Next / Previous unread message</h2>
<ul>
<li><code>[ [</code>: next</li>
<li><code>] ]</code>: previous</li>
</ul>
</div></div><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mu4e — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-01-17 15:57</time> </div>https://blog.parisni.com/wiki/i3I32023-06-30T01:27:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span></p>
<h1>I3</h1>
<h2>D-menu</h2>
<p>The</p>
<p>By default, desktop application are located in <code>/usr/share/applications/</code> or <code>.local/share/applications/</code>:</p>
<pre><code class="language-ini"><span class="hl kwa">[Desktop Entry]</span>
<span class="hl kwb">Version</span><span class="hl opt">=</span><span class="hl num">1.0</span>
<span class="hl kwb">Name</span><span class="hl opt">=</span>Qutebrowser
<span class="hl kwb">GenericName</span><span class="hl opt">=</span>Web Browser
<span class="hl kwb">Comment</span><span class="hl opt">=</span>Access the Internet
<span class="hl kwb">Exec</span><span class="hl opt">=</span>qutebrowser <span class="hl opt">%</span>U
<span class="hl kwb">Terminal</span><span class="hl opt">=</span>false
<span class="hl kwb">X-MultipleArgs</span><span class="hl opt">=</span>false
<span class="hl kwb">Type</span><span class="hl opt">=</span>Application
<span class="hl kwb">Icon</span><span class="hl opt">=</span>qutebrowser
<span class="hl kwb">Categories</span><span class="hl opt">=</span>Network;WebBrowser;
<span class="hl kwb">MimeType</span><span class="hl opt">=</span>text<span class="hl opt">/</span>html;text<span class="hl opt">/</span>xml;application<span class="hl opt">/</span>xhtml_xml;x<span class="hl opt">-</span>scheme<span class="hl opt">-</span>handler<span class="hl opt">/</span>http;x<span class="hl opt">-</span>scheme<span class="hl opt">-</span>handler<span class="hl opt">/</span>https;
<span class="hl kwb">StartupNotify</span><span class="hl opt">=</span>true
<span class="hl kwb">X-AppInstall-Package</span><span class="hl opt">=</span>qutebrowser
</code></pre>
<p>Also, the icon can be placed into <code>.local/share/icons/</code></p>
<h2>Lock screen</h2>
<p><a href="https://github.com/google/xsecurelock">xsecurelock</a> is from far, the best screen locker out there.</p>
<p>and use lock.sh wrapper:</p>
<pre><code>#!/bin/sh
XSECURELOCK_SHOW_DATETIME=1 XSECURELOCK_DATETIME_FORMAT="%A, %Y-%m-%d %H:%M" XSECURELOCK_PASSWORD_PROMPT=cursor xsecurelock
</code></pre>
<p>Also, in order to lock before suspend:</p>
<p><code>systemctl enable xsecurelock@parisni</code>
cat /etc/systemd/system/xsecurelock@.service</p>
<pre><code>[Unit]
Description=Lock X session using xsecurelock
Before=sleep.target
[Service]
Type=simple
User=%i
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/%i/.Xauthority
Environment=XSECURELOCK_SHOW_DATETIME=1
Environment=XSECURELOCK_DATETIME_FORMAT="%%A, %%Y-%%m-%%d %%H:%%M"
ExecStart=/usr/local/bin/xsecurelock
[Install]
WantedBy=suspend.target
</code></pre>
<h2>Modes</h2>
<p><code>Alt_gr</code> is a good candidate for the <code>mode</code> key because it is accessible to
the right thumb wich is probably the sronger finger we have.</p>
<p>To be able to set it, one way is to switch both <code>Alt_R</code> and <code>Alt_gr</code> and set
<code>$mod = Mod1</code>.</p>
<p>An other good things is to get the <code>caps_lock</code> key do <code>Escape</code>, which is
essential in vim/emacs key bindings.</p>
<p>All those changes can be set in the <code>/etc/default/keyboard</code> file:</p>
<pre><code class="language-bash">XKBMODEL<span class="hl opt">=</span><span class="hl sng">"pc105"</span>
XKBLAYOUT<span class="hl opt">=</span><span class="hl sng">"us"</span>
XKBVARIANT<span class="hl opt">=</span><span class="hl sng">"altgr-intl"</span>
XKBOPTIONS<span class="hl opt">=</span><span class="hl sng">"caps:swapescape,lv3:lalt_switch,lv3:ralt_alt"</span>
BACKSPACE<span class="hl opt">=</span><span class="hl sng">"guess"</span>
</code></pre>
<p>And the <code>~/.config/i3/config</code>:</p>
<pre><code class="language-bash">exec_always <span class="hl kwb">--no-startup-id</span> <span class="hl sng">'~/bin/xmap'</span>
<span class="hl kwb">set</span> <span class="hl kwd">$mod</span> Mod1
font pango<span class="hl opt">:</span>monospace <span class="hl num">8</span>
floating_modifier <span class="hl kwd">$mod</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>n <span class="hl kwb">exec</span> i3<span class="hl kwb">-sensible-terminal</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>p <span class="hl kwb">kill</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>d <span class="hl kwb">exec --no-startup-id</span> i3<span class="hl kwb">-dmenu-desktop</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>j focus left
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>k focus right
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>Left focus left
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>Right focus right
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>Shift<span class="hl opt">+</span>k move right
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>Shift<span class="hl opt">+</span>j move left
bindsym Control<span class="hl opt">+</span><span class="hl num">1</span> workspace <span class="hl num">1</span>
bindsym Control<span class="hl opt">+</span><span class="hl num">2</span> workspace <span class="hl num">2</span>
bindsym Control<span class="hl opt">+</span><span class="hl num">3</span> workspace <span class="hl num">3</span>
bindsym Control<span class="hl opt">+</span><span class="hl num">4</span> workspace <span class="hl num">4</span>
bindsym Control<span class="hl opt">+</span>Shift<span class="hl opt">+</span><span class="hl num">1</span> move container to workspace <span class="hl num">1</span>
bindsym Control<span class="hl opt">+</span>Shift<span class="hl opt">+</span><span class="hl num">2</span> move container to workspace <span class="hl num">2</span>
bindsym Control<span class="hl opt">+</span>Shift<span class="hl opt">+</span><span class="hl num">3</span> move container to workspace <span class="hl num">3</span>
bindsym Control<span class="hl opt">+</span>Shift<span class="hl opt">+</span><span class="hl num">4</span> move container to workspace <span class="hl num">4</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>e <span class="hl kwb">exec</span> <span class="hl sng">"i3-nagbar -t warning -m 'You pressed the exit shortcut. Do you really want to exit i3? This will end your X session.' -b 'Yes, exit i3' 'i3-msg exit'"</span>
<span class="hl slc"># search by title on existing windows</span>
<span class="hl slc"># see https://github.com/micahcc/i3-wm-scripts</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>e <span class="hl kwb">exec</span> python ~<span class="hl opt">/</span>bin<span class="hl opt">/</span>nextfind
mode <span class="hl sng">"resize"</span> <span class="hl opt">{</span>
<span class="hl slc"># These bindings trigger as soon as you enter the resize mode</span>
<span class="hl slc"># Pressing left will shrink the window’s width.</span>
<span class="hl slc"># Pressing right will grow the window’s width.</span>
<span class="hl slc"># Pressing up will shrink the window’s height.</span>
<span class="hl slc"># Pressing down will grow the window’s height.</span>
bindsym j resize shrink width <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
bindsym k resize grow height <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
bindsym l resize shrink height <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
bindsym m resize grow width <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
<span class="hl slc"># same bindings, but for the arrow keys</span>
bindsym Left resize shrink width <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
bindsym Down resize grow height <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
bindsym Up resize shrink height <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
bindsym Right resize grow width <span class="hl num">10</span> px or <span class="hl num">10</span> ppt
<span class="hl slc"># back to normal: Enter or Escape</span>
bindsym Return mode <span class="hl sng">"default"</span>
bindsym Escape mode <span class="hl sng">"default"</span>
<span class="hl opt">}</span>
bar <span class="hl opt">{</span>
status_command i3status
tray_output primary
<span class="hl opt">}</span>
<span class="hl kwb">exec --no-startup-id</span> nm<span class="hl kwb">-applet</span>
bindsym XF86AudioRaiseVolume <span class="hl kwb">exec</span> <span class="hl sng">"amixer -q sset Master,0 3+ unmute ; amixer -c 1 -q sset Headset,0 3+ unmute"</span>
bindsym XF86AudioLowerVolume <span class="hl kwb">exec</span> <span class="hl sng">"amixer -q sset Master,0 3- unmute ; amixer -c 1 -q sset Headset,0 3- unmute"</span>
bindsym XF86AudioMute <span class="hl kwb">exec</span> <span class="hl sng">"amixer -D pulse set Master 1+ toggle ; amixer -c 1 -D pulse set Headset 1+ toggle"</span>
bindsym XF86MonBrightnessUp <span class="hl kwb">exec --no-startup-id</span> bash <span class="hl kwb">-c</span> <span class="hl sng">'b=</span><span class="hl ipl">$(xrandr --verbose | grep -i brightness | head -n 1 | cut -d" " -f2)</span> <span class="hl sng">&& b2=</span><span class="hl ipl">$(echo -e "import math\nf = lambda x: (1 - 1/(1+math.exp(4*x-2)</span><span class="hl sng">))</span><span class="hl esc">\n</span><span class="hl sng">print(</span><span class="hl esc">\\\"</span><span class="hl sng">%.2f</span><span class="hl esc">\\\"</span> <span class="hl sng">% f(0.1+</span><span class="hl ipl">$b</span><span class="hl sng">))" | python3) && xrandr --output eDP-1 --brightness</span> <span class="hl ipl">$b2</span><span class="hl sng">'</span>
bindsym XF86MonBrightnessDown <span class="hl kwb">exec --no-startup-id</span> bash <span class="hl kwb">-c</span> <span class="hl sng">'b=</span><span class="hl ipl">$(xrandr --verbose | grep -i brightness | head -n 1 | cut -d" " -f2)</span> <span class="hl sng">&& b2=</span><span class="hl ipl">$(echo -e "import math\nf = lambda x: (1 - 1/(1+math.exp(4*x-2)</span><span class="hl sng">))</span><span class="hl esc">\n</span><span class="hl sng">print(</span><span class="hl esc">\\\"</span><span class="hl sng">%.2f</span><span class="hl esc">\\\"</span> <span class="hl sng">% f(-0.1+</span><span class="hl ipl">$b</span><span class="hl sng">))" | python3) && xrandr --output eDP-1 --brightness</span> <span class="hl ipl">$b2</span><span class="hl sng">'</span>
<span class="hl kwb">exec --no-startup-id</span> xrandr <span class="hl kwb">--output</span> eDP<span class="hl kwb">-1 --mode</span> <span class="hl num">1280</span>x720 <span class="hl kwb">--rate</span> <span class="hl num">60</span> <span class="hl kwb">--scale</span> <span class="hl num">1</span>x1 <span class="hl kwb">--dpi</span> <span class="hl num">96</span> <span class="hl kwb">--primary</span>
workspace_layout tabbed
popup_during_fullscreen smart
for_window <span class="hl opt">[</span>window_role<span class="hl opt">=</span><span class="hl sng">"pop-up"</span><span class="hl opt">]</span> floating <span class="hl kwb">enable</span>
for_window <span class="hl opt">[</span>window_role<span class="hl opt">=</span><span class="hl sng">"task_dialog"</span><span class="hl opt">]</span> floating <span class="hl kwb">enable</span>
for_window <span class="hl opt">[</span>title<span class="hl opt">=</span><span class="hl sng">"Preferences$"</span><span class="hl opt">]</span> floating <span class="hl kwb">enable</span>
for_window <span class="hl opt">[</span>class<span class="hl opt">=</span><span class="hl sng">"(?i)mplayer"</span><span class="hl opt">]</span> floating <span class="hl kwb">enable</span>
<span class="hl kwb">set</span> <span class="hl kwd">$Locker</span> lock.sh
<span class="hl kwb">set</span> <span class="hl kwd">$mode_system</span> System <span class="hl opt">(</span>l<span class="hl opt">)</span> lock<span class="hl opt">, (</span>e<span class="hl opt">)</span> <span class="hl kwb">logout</span><span class="hl opt">, (</span>s<span class="hl opt">)</span> <span class="hl kwb">suspend</span><span class="hl opt">, (</span>h<span class="hl opt">)</span> hibernate<span class="hl opt">, (</span>r<span class="hl opt">)</span> reboot<span class="hl opt">, (</span>Shift<span class="hl opt">+</span>s<span class="hl opt">)</span> shutdown
mode <span class="hl sng">"</span><span class="hl ipl">$mode_system</span><span class="hl sng">"</span> <span class="hl opt">{</span>
bindsym l <span class="hl kwb">exec --no-startup-id</span> <span class="hl kwd">$Locker</span><span class="hl opt">,</span> mode <span class="hl sng">"default"</span>
bindsym e <span class="hl kwb">exec --no-startup-id</span> i3<span class="hl kwb">-msg exit</span><span class="hl opt">,</span> mode <span class="hl sng">"default"</span>
bindsym s <span class="hl kwb">exec --no-startup-id</span> <span class="hl kwd">$Locker</span> <span class="hl opt">&&</span> systemctl <span class="hl kwb">suspend</span><span class="hl opt">,</span> mode <span class="hl sng">"default"</span>
bindsym h <span class="hl kwb">exec</span> <span class="hl sng">"sudo pm-hibernate"</span><span class="hl opt">,</span> mode <span class="hl sng">"default"</span>
bindsym r <span class="hl kwb">exec</span> <span class="hl sng">"sudo reboot"</span><span class="hl opt">,</span> mode <span class="hl sng">"default"</span>
bindsym Shift<span class="hl opt">+</span>s <span class="hl kwb">exec</span> <span class="hl sng">"shutdown now"</span><span class="hl opt">,</span> mode <span class="hl sng">"default"</span>
<span class="hl slc"># back to normal: Enter or Escape</span>
bindsym Return mode <span class="hl sng">"default"</span>
bindsym Escape mode <span class="hl sng">"default"</span>
<span class="hl opt">}</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>Escape mode <span class="hl sng">"</span><span class="hl ipl">$mode_system</span><span class="hl sng">"</span>
bindsym Print <span class="hl kwb">exec</span> scrot <span class="hl sng">'Screenshot-nparis_%Y-%m-%d_%H:%M:%S.png'</span> <span class="hl kwb">-e</span> <span class="hl sng">'mv</span> <span class="hl ipl">$f</span> <span class="hl sng">/tmp/ && gimp /tmp/</span><span class="hl ipl">$f</span><span class="hl sng">'</span>
bindsym <span class="hl kwb">--release</span> <span class="hl kwd">$mod</span><span class="hl opt">+</span>Print <span class="hl kwb">exec</span> scrot <span class="hl kwb">-s -e</span> <span class="hl sng">'mv</span> <span class="hl ipl">$f</span> <span class="hl sng">/tmp/ && gimp /tmp/</span><span class="hl ipl">$f</span><span class="hl sng">'</span>
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>i <span class="hl kwb">exec</span> diodon
bindsym <span class="hl kwd">$mod</span><span class="hl opt">+</span>x mode <span class="hl sng">"</span><span class="hl ipl">$mode_display</span><span class="hl sng">"</span>
<span class="hl kwb">exec --no-startup-id</span> pasystray <span class="hl kwb">--include-monitors -a</span>
for_window <span class="hl opt">[</span>class<span class="hl opt">=</span><span class="hl sng">"jetbrains-idea-ce"</span><span class="hl opt">]</span> floating disable<span class="hl opt">;</span>
workspace_auto_back_and_forth <span class="hl kwc">yes</span>
focus_follows_mouse no
</code></pre>
<h2>Playing with windows</h2>
<pre><code class="language-bash">xprop
<span class="hl slc"># then click on the application</span>
WM_CLASS<span class="hl opt">(</span>STRING<span class="hl opt">) =</span> <span class="hl sng">"instance"</span><span class="hl opt">,</span> <span class="hl sng">"class"</span>
</code></pre>
<h2>Tree</h2>
<pre><code>i3-save-tree --workspace 1 > ~/.i3/workspace-1.json
i3-msg "workspace 1; append_layout ~/.i3/workspace-1.json"
</code></pre>
<h2>Use external screen</h2>
<pre><code>xrandr_screen () {
xrandr --output eDP-1 --off --noprimary && xrandr --output $(xrandr|grep " connected"|tail -n 1| cut -f 1 -d " ") --mode 1920x1080 --rate 75 --scale 1x1 --dpi 96 --primary
}
xrandr_laptop () {
xrandr --output eDP-1 --mode 1360x768 --rate 60 --scale 1x1 --dpi 96 --primary && xrandr --output $(xrandr|grep " connected"|tail -n 1| cut -f 1 -d " ") --off
}
</code></pre>
<h2>Configure lightm screen at startup</h2>
<pre><code>$ cat /etc/lightdm/lightdm.conf.d/nparis.conf
[SeatDefaults]
autologin-guest=false
display-setup-script=/usr/local/bin/xrandr-lightdm
#autologin-user=nparis
#autologin-user-timeout=0
#autologin-session=lightdm-autologin
</code></pre>
<pre><code>$ cat /usr/local/bin/xrandr-lightdm
#!/usr/bin/env bash
xrandr --output eDP-1 --mode 1440x810 --rate 75 --scale 1x1 --primary
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=I3 — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-06-30 01:27</time> </div>https://blog.parisni.com/wiki/parquetParquet2023-07-27T22:43:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">versionning</span></p>
<h1>Parquet</h1>
<h2>Compression</h2>
<p>Get parquet file compression:</p>
<pre><code class="language-python"><span class="hl kwa">import</span> pyarrow <span class="hl kwa">as</span> pa
<span class="hl kwa">import</span> pyarrow<span class="hl opt">.</span>parquet <span class="hl kwa">as</span> pq
parquet_file <span class="hl opt">=</span> pq<span class="hl opt">.</span><span class="hl kwd">ParquetFile</span><span class="hl opt">((</span><span class="hl sng">"/tmp/enc/part-00000-e9f69621-ef56-42b4-bf1b-997e8f2b088e-c000.snappy.parquet"</span><span class="hl opt">))</span>
parquet_file<span class="hl opt">.</span>metadata<span class="hl opt">.</span><span class="hl kwd">row_group</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">).</span><span class="hl kwd">column</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">)</span>
<span class="hl opt"><</span>pyarrow<span class="hl num">._</span>parquet<span class="hl opt">.</span>ColumnChunkMetaData <span class="hl kwb">object</span> at <span class="hl num">0x7fa724db64f0</span><span class="hl opt">></span>
file_offset<span class="hl opt">:</span> <span class="hl num">4</span>
file_path<span class="hl opt">:</span>
physical_type<span class="hl opt">:</span> INT32
num_values<span class="hl opt">:</span> <span class="hl num">1</span>
path_in_schema<span class="hl opt">:</span> foo
is_stats_set<span class="hl opt">:</span> <span class="hl kwa">True</span>
statistics<span class="hl opt">:</span>
<span class="hl opt"><</span>pyarrow<span class="hl num">._</span>parquet<span class="hl opt">.</span>Statistics <span class="hl kwb">object</span> at <span class="hl num">0x7fa725374180</span><span class="hl opt">></span>
has_min_max<span class="hl opt">:</span> <span class="hl kwa">True</span>
<span class="hl kwb">min</span><span class="hl opt">:</span> <span class="hl num">1</span>
<span class="hl kwb">max</span><span class="hl opt">:</span> <span class="hl num">1</span>
null_count<span class="hl opt">:</span> <span class="hl num">0</span>
distinct_count<span class="hl opt">:</span> <span class="hl num">0</span>
num_values<span class="hl opt">:</span> <span class="hl num">1</span>
physical_type<span class="hl opt">:</span> INT32
logical_type<span class="hl opt">:</span> <span class="hl kwa">None</span>
<span class="hl kwd">converted_type</span> <span class="hl opt">(</span>legacy<span class="hl opt">):</span> NONE
compression<span class="hl opt">:</span> SNAPPY
encodings<span class="hl opt">: (</span><span class="hl sng">'BIT_PACKED'</span><span class="hl opt">,</span> <span class="hl sng">'PLAIN'</span><span class="hl opt">)</span>
has_dictionary_page<span class="hl opt">:</span> <span class="hl kwa">False</span>
dictionary_page_offset<span class="hl opt">:</span> <span class="hl kwa">None</span>
data_page_offset<span class="hl opt">:</span> <span class="hl num">4</span>
total_compressed_size<span class="hl opt">:</span> <span class="hl num">29</span>
total_uncompressed_size<span class="hl opt">:</span> <span class="hl num">27</span>
</code></pre>
<h2>Parquet-cli</h2>
<h3>Basic install</h3>
<ul>
<li><a href="https://github.com/apache/parquet-mr/blob/2cf7951ca686e5b602661079a05acd1b27ac95db/parquet-cli/README.md?plain=1#L34">installation</a></li>
</ul>
<p>in short:</p>
<ol>
<li>git clone parquet-mr</li>
<li>in parquet-cli, mvn install -DskipTests</li>
<li>mvn dependency:copy-dependencies</li>
<li>then: <code>alias parquet-cli="java -cp 'target/parquet-cli-1.12.2.jar:target/dependency/*' org.apache.parquet.cli.Main"</code></li>
</ol>
<p>eg:</p>
<pre><code>parquet-cli meta file.parquet|grep column
</code></pre>
<h3>Docker installation</h3>
<ul>
<li><a href="https://raw.githubusercontent.com/takamoto/docker-parquet-cli/main/Dockerfile">Dockerfile</a></li>
</ul>
<pre><code>docker build run parquet-cli:latest
docker run -v /tmp/:/data parquet-cli:latest pages /data/file.parquet
</code></pre>
<h2>Parquet row groups</h2>
<ul>
<li><a href="https://docs.dremio.com/software/data-formats/parquet-files/#recommended-configuration">dremio recommends</a> one row group per parquet file, and uses 256MB row groups</li>
<li><a href="https://parquet.apache.org/docs/file-format/configurations/">parquet itself</a> recommends 512MB row groups</li>
</ul>
<h2>Parquet encoding</h2>
<ul>
<li><a href="https://dataninjago.com/2021/12/07/databricks-deep-dive-3-parquet-encoding/">deep dive</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Parquet — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-07-27 22:43</time> </div>https://blog.parisni.com/wiki/ldapLdap2023-07-21T16:04:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">security</span></p>
<h1>Ldap</h1>
<h2>Start a ldap server</h2>
<pre><code class="language-bash">docker run <span class="hl kwb">-p</span> <span class="hl num">389</span><span class="hl opt">:</span><span class="hl num">389</span> <span class="hl kwb">-p</span> <span class="hl num">636</span><span class="hl opt">:</span><span class="hl num">636</span> <span class="hl kwb">--volume</span> <span class="hl opt">/</span>data<span class="hl opt">/</span>slapd<span class="hl opt">/</span>database<span class="hl opt">:/</span>var<span class="hl opt">/</span>lib<span class="hl opt">/</span>ldap <span class="hl kwb">--name</span> my<span class="hl kwb">-openldap-container --detach</span> osixia<span class="hl opt">/</span>openldap<span class="hl opt">:</span>latest
</code></pre>
<h2>Search into a ldap server</h2>
<pre><code class="language-bash">docker <span class="hl kwb">exec</span> my<span class="hl kwb">-openldap-container</span> ldapsearch <span class="hl kwb">-x -H</span> ldap<span class="hl opt">://</span><span class="hl num">0.0.0.0</span><span class="hl opt">:</span><span class="hl num">389</span> <span class="hl kwb">-b</span> <span class="hl kwc">dc</span><span class="hl opt">=</span>example<span class="hl opt">,</span><span class="hl kwc">dc</span><span class="hl opt">=</span>org <span class="hl kwb">-D</span> <span class="hl sng">"cn=admin,dc=example,dc=org"</span> <span class="hl kwb">-w</span> admin
</code></pre>
<h2>Add a user</h2>
<pre><code class="language-bash">docker <span class="hl kwb">exec</span> my<span class="hl kwb">-openldap-container</span> ldapadd <span class="hl kwb">-x -D</span> <span class="hl sng">"cn=admin,dc=example,dc=org"</span> <span class="hl kwb">-w</span> admin <span class="hl kwb">-f</span> <span class="hl opt">/</span>container<span class="hl opt">/</span>service<span class="hl opt">/</span>slapd<span class="hl opt">/</span>assets<span class="hl opt">/</span>test<span class="hl opt">/</span>new<span class="hl kwb">-user</span>.ldif <span class="hl kwb">-H</span> ldap<span class="hl opt">://</span><span class="hl num">0.0.0.0</span><span class="hl opt">:</span><span class="hl num">389</span>
</code></pre>
<h2>Delete a user</h2>
<pre><code class="language-bash">docker <span class="hl kwb">exec</span> openldap ldapdelete <span class="hl kwb">-x -D</span> <span class="hl sng">"cn=admin,dc=example,dc=org"</span> <span class="hl kwb">-w</span> admin <span class="hl sng">"uid=nico,dc=example,dc=org"</span> <span class="hl kwb">-H</span> ldap<span class="hl opt">://</span><span class="hl num">0.0.0.0</span><span class="hl opt">:</span><span class="hl num">389</span>
</code></pre>
<h2>Generate a password</h2>
<p>The salt is contained in the result, so multiplle call returns multiple results.</p>
<pre><code class="language-bash"><span class="hl slc"># into the docker container</span>
slappasswd <span class="hl kwb">-h</span> <span class="hl opt">{</span>SSHA<span class="hl opt">}</span> <span class="hl kwb">-s</span> abcd123
</code></pre>
<h2>Great tools</h2>
<ul>
<li><a href="https://github.com/leenooks/phpLDAPadmin">phpLDAPadmin has a docker image</a></li>
<li><a href="https://github.com/npenkov/ldap-passwd-webui">ldap password webui</a></li>
</ul>
<pre><code class="language-yaml"> <span class="hl kwa">webui:</span>
<span class="hl kwa">image: npenkov/docker-ldap-passwd-webui:</span>latest
<span class="hl kwa">ports:</span>
<span class="hl opt">-</span> <span class="hl sng">"8081:8080"</span>
<span class="hl kwa">environment:</span>
<span class="hl kwa">LPW_TITLE:</span> <span class="hl sng">"Change your global password for example.org"</span>
<span class="hl kwa">LPW_HOST:</span> <span class="hl sng">"openldap"</span>
<span class="hl kwa">LPW_PORT:</span> <span class="hl sng">"389"</span>
<span class="hl kwa">LPW_ENCRYPTED:</span> <span class="hl sng">"false"</span>
<span class="hl kwa">LPW_START_TLS:</span> <span class="hl sng">"false"</span>
<span class="hl kwa">LPW_SSL_SKIP_VERIFY:</span> <span class="hl sng">"true"</span>
<span class="hl kwa">LPW_USER_DN:</span> <span class="hl sng">"uid=%s,dc=example,dc=org"</span>
<span class="hl kwa">LPW_USER_BASE:</span> <span class="hl sng">"dc=example,dc=org"</span>
<span class="hl kwa">LPW_PATTERN:</span> <span class="hl sng">'.{8,}'</span>
<span class="hl kwa">LPW_PATTERN_INFO:</span> <span class="hl sng">"Password must be at least 8 characters long."</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Ldap — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-07-21 16:04</time> </div>https://blog.parisni.com/wiki/browser-mobileMobile Browser2021-12-28T23:03:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">web</span></p>
<h1>Mobile Browser</h1>
<p>So far, firefox is the best option on mobile.</p>
<h2>Firefox</h2>
<p>So far, the best.</p>
<ul>
<li><code>+</code>: bottom navbar</li>
<li><code>+</code>: handle vpn tls with no error</li>
<li><code>+</code>: handle well password autofill</li>
<li><code>-</code>: homepage does not offer a simple page</li>
<li><code>-</code>: no built-in dark-reader (but plugins do)</li>
<li><code>-</code>: telemetry (but can be deactivated)</li>
</ul>
<h2>Kiwi Browser</h2>
<ul>
<li><code>+</code>: bottom navbar</li>
<li><code>+</code>: handle well password autofill</li>
<li><code>+</code>: built-in dark-reader</li>
<li><code>-</code>: trouble with vpn and tls</li>
<li><code>-</code>: trouble with some websites</li>
</ul>
<h2>Brave</h2>
<p>so much functionality and noise.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mobile Browser — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-28 23:03</time> </div>https://blog.parisni.com/wiki/airflowAirflow2023-08-21T12:23:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">data</span></p>
<h1>Airflow</h1>
<h2>Trigger dags from external dag</h2>
<p>This involves 3 dags:</p>
<ol>
<li>the external dag to follow</li>
<li>the external dag to run</li>
<li>the trigger dag</li>
</ol>
<p>the 2 formers are classical dags. both 1 and 3 MUST have the same
<strong>schedule_interval</strong>, while the 2 should be <strong>@once</strong>. It is usefull
to be able to run it manually.</p>
<p>trigger dag:</p>
<pre><code class="language-python"><span class="hl kwa">from</span> airflow<span class="hl opt">.</span>sensors<span class="hl opt">.</span>external_task_sensor <span class="hl kwa">import</span> ExternalTaskSensor
<span class="hl kwa">from</span> airflow<span class="hl opt">.</span>operators<span class="hl opt">.</span>dagrun_operator <span class="hl kwa">import</span> TriggerDagRunOperator
default_args <span class="hl opt">= {</span>
<span class="hl sng">'owner'</span><span class="hl opt">:</span> <span class="hl sng">'airflow'</span><span class="hl opt">,</span>
<span class="hl sng">'depends_on_past'</span><span class="hl opt">:</span> <span class="hl kwa">False</span><span class="hl opt">,</span>
<span class="hl sng">'start_date'</span><span class="hl opt">:</span> airflow<span class="hl opt">.</span>utils<span class="hl opt">.</span>dates<span class="hl opt">.</span><span class="hl kwd">days_ago</span><span class="hl opt">(</span><span class="hl num">7</span><span class="hl opt">),</span>
<span class="hl sng">'email_on_failure'</span><span class="hl opt">:</span> <span class="hl kwa">True</span><span class="hl opt">,</span>
<span class="hl sng">'email'</span><span class="hl opt">: [</span>email<span class="hl opt">]</span>
<span class="hl opt">}</span>
dag <span class="hl opt">=</span> <span class="hl kwd">DAG</span><span class="hl opt">(</span>dagName<span class="hl opt">,</span> default_args<span class="hl opt">=</span>default_args<span class="hl opt">,</span> schedule_interval<span class="hl opt">=</span><span class="hl sng">'0 21 * * *'</span><span class="hl opt">)</span>
<span class="hl kwa">with</span> dag<span class="hl opt">:</span>
wait_for_parent <span class="hl opt">=</span> <span class="hl kwd">ExternalTaskSensor</span><span class="hl opt">(</span>
task_id<span class="hl opt">=</span><span class="hl sng">'wait_for_parent'</span><span class="hl opt">,</span>
external_dag_id<span class="hl opt">=</span><span class="hl sng">'the-external-dag-id'</span><span class="hl opt">,</span>
external_task_id<span class="hl opt">=</span><span class="hl sng">'the-external-dag-id.the-external-task-id'</span>
<span class="hl opt">)</span>
run_task <span class="hl opt">=</span> <span class="hl kwd">TriggerDagRunOperator</span><span class="hl opt">(</span>
task_id<span class="hl opt">=</span><span class="hl sng">"run-task"</span><span class="hl opt">,</span>
trigger_dag_id<span class="hl opt">=</span><span class="hl sng">"the-second-external-dag-id-to-run"</span><span class="hl opt">,</span>
execution_date<span class="hl opt">=</span>datetime<span class="hl opt">.</span><span class="hl kwd">now</span><span class="hl opt">()</span>
<span class="hl opt">)</span>
wait_for_parent <span class="hl opt">>></span> run_task
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Airflow — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-08-21 12:23</time> </div>https://blog.parisni.com/wiki/scalewayScaleway2021-10-10T21:19:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">cloud</span></p>
<h1>Scaleway</h1>
<h2>Bucket</h2>
<p>Storage is free below 75GO</p>
<h3>Hosting Website</h3>
<ol>
<li>create a bucket <code>blog.parisni.com</code> (matching the hostname)</li>
<li>turn it to public & enable website feature</li>
<li>upload the static content with awscli (<code>aws s3 cp <local/path> s3://blog.parisni.com --recursive</code>)</li>
<li>redirect your domain to the bucket</li>
</ol>
<p>NOTE: to date, the website cannot benefit from SSL.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Scaleway — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-10-10 21:19</time> </div>https://blog.parisni.com/wiki/bluetoothBluetooth on linux2020-11-15T17:40:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">network</span></p>
<h1>Bluetooth on linux</h1>
<h2>Blueman</h2>
<h3>Setup</h3>
<p><a href="https://www.maketecheasier.com/setup-bluetooth-in-linux/">setup bluetooth on ubuntu</a></p>
<p>Then run <code>blueman-manager</code> and trust the device.</p>
<h3>Connect a samsung galaxy</h3>
<ol>
<li>on the bluetooth tray icon, select <em>make discoverable</em></li>
<li>on the phone bluetooth app, select <em>scan</em></li>
<li>on the phone gallery, select <em>share</em> and then <em>bluetooth</em></li>
</ol>
<h2>bluetoohctl</h2>
<h3>peer a device</h3>
<p><a href="https://www.reddit.com/r/i3wm/comments/9wqt4k/connect_to_bluetooth_devices_on_i3/">source</a></p>
<pre><code class="language-bash">bluetoothctl
$ power on
$ agent on
$ scan on
$ pair <span class="hl opt"><</span>mac address<span class="hl opt">></span>
$ connect <span class="hl opt"><</span>mac address<span class="hl opt">></span>
</code></pre>
<h3>send file to device</h3>
<p><a href="https://techiesanswer.com/ubuntu-command-line/unix-bluetoothctl-and-bluetooth-sendto-tools-to-send-file/">source</a></p>
<pre><code class="language-bash">bluetooth<span class="hl kwb">-sendfile</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Bluetooth on linux — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-11-15 17:40</time> </div>https://blog.parisni.com/wiki/acid-formatAcid Tables2023-12-05T13:02:00+00:00<div id="generated-toc"> </div><hr> <p>#spark</p>
<h1>Acid Tables</h1>
<h2>Comparison</h2>
<ul>
<li><a href="https://www.dremio.com/blog/comparison-of-data-lake-table-formats-apache-iceberg-apache-hudi-and-delta-lake/">Dremio</a></li>
</ul>
<h2>apache iceberg</h2>
<ul>
<li><a href="https://github.com/microsoft/hyperspace/pull/358">Indexing hyperspace</a></li>
<li><a href="https://youtu.be/jCXpFagJsbo">foko conf</a></li>
<li><a href="https://developer.ibm.com/articles/the-why-and-how-of-partitioning-in-apache-iceberg/">hidden partition</a></li>
<li><a href="https://www.dremio.com/blog/deep-dive-into-configuring-your-apache-iceberg-catalog-with-apache-spark/">Config spark</a></li>
<li><a href="https://www.dremio.com/blog/puffins-and-icebergs-additional-stats-for-apache-iceberg-tables/">puffin indexes</a></li>
<li><a href="https://www.dremio.com/blog/compaction-in-apache-iceberg-fine-tuning-your-iceberg-tables-data-files/">compaction</a></li>
<li><a href="https://www.dremio.com/blog/apache-iceberg-and-the-right-to-be-forgotten/">Cow and mor delete</a></li>
<li><a href="https://iceberg.apache.org/spec/#sorting">stats index spec</a></li>
<li></li>
</ul>
<h2>Delta-lake</h2>
<h3>Features</h3>
<ul>
<li><a href="https://docs.google.com/document/d/1lv35ZPfioopBbzQ7zT82LOev7qV7x4YNLkMr2-L5E_M/edit#heading=h.dd2cc57oc5wk">vectorization</a></li>
<li><a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#deletion-vectors">Delta spec</a></li>
<li><a href="https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-delta-live-tables-dlt">gdpr</a></li>
<li><a href="https://docs.databricks.com/delta/tune-file-size.html#optimized-writes">tuning</a></li>
<li><a href="https://microsoft.github.io/hyperspace/docs/ug-supported-data-formats/#delta-lake">indexing hyperacale</a></li>
<li><a href="https://docs.delta.io/latest/delta-change-data-feed.html">change data feed</a></li>
<li><a href="https://docs.delta.io/latest/optimizations-oss.html#multi-part-checkpointing">metadata files are in parquet</a></li>
<li><a href="https://docs.delta.io/latest/concurrency-control.html">occ</a></li>
<li><a href="https://yousry.medium.com/delta-lake-universal-format-a-first-look-9dfa28b68b72">universal reads iceberg hudi</a></li>
<li><a href="https://blog.ippon.fr/2022/02/07/delta-lake-la-taille-compte/">cow optim ippon</a></li>
<li><a href="https://docs.databricks.com/delta/tune-file-size.html#set-a-target-file-size">autotune size parquet</a></li>
<li><a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md">delta lake spec</a></li>
<li><a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#per-file-statistics">stats spec</a></li>
<li><a href="https://www.dremio.com/blog/the-why-and-how-of-using-apache-iceberg-on-databricks/">about uniforme ans iceberg</a></li>
</ul>
<h3>limitations</h3>
<ul>
<li><a href="https://medium.com/@ritik20023/delta-lake-upserting-without-primary-key-f4a931576b0">cannot upsert w/o pk</a></li>
<li><a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#change-data-files">CDC duplicates the data in a folder</a></li>
</ul>
<h2>Hudi</h2>
<h3>Blogs</h3>
<ul>
<li><a href="https://medium.com/walmartglobaltech/lakehouse-at-fortune-1-scale-480bcb10391b">walmart</a></li>
<li><a href="https://www.uber.com/en-FR/blog/ubers-lakehouse-architecture/">uber</a> <a href="https://www.youtube.com/watch?v=2yN8Cga4ODA">incremental video</a></li>
<li><a href="https://aws.amazon.com/blogs/big-data/how-zoom-implemented-streaming-log-ingestion-and-efficient-gdpr-deletes-using-apache-hudi-on-amazon-emr/">zoom</a></li>
<li><a href="https://hudi.apache.org/tech-specs/#metadata">spec</a></li>
<li></li>
</ul>
<h3>tips</h3>
<ul>
<li><a href="https://hudi.apache.org/docs/configurations/#hoodiebulkinsertuserdefinedpartitionerclass">Custom partitionner</a></li>
<li><a href="https://github.com/apache/hudi/blob/2c2abaf14bd0638051c641aaff529114756d3a9f/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RowSpatialCurveSortPartitioner.java">Bulk insert zorder partitionner</a></li>
<li><a href="https://www.onehouse.ai/blog/top-3-things-you-can-do-to-get-fast-upsert-performance-in-apache-hudi">onehouse indexes for upserts</a></li>
<li><a href="https://medium.com/@simpsons/can-you-concurrently-write-data-to-apache-hudi-w-o-any-lock-provider-51ea55bf2dd6">avoid OCC</a></li>
<li><a href="https://dzone.com/articles/delta-hudi-and-iceberg-the-data-lakehouse-trifecta">bench</a></li>
<li><a href="https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison">wow bench</a></li>
<li><a href="https://aws.amazon.com/fr/blogs/big-data/choosing-an-open-table-format-for-your-transactional-data-lake-on-aws/">aws bench</a></li>
<li><a href="https://airbyte.com/blog/data-lake-lakehouse-guide-powered-by-table-formats-delta-lake-iceberg-hudi">yet another bench</a></li>
<li><a href="https://medium.com/@simpsons/speed-up-your-write-latencies-using-bucket-index-in-apache-hudi-2f7c297493dc">bucket</a></li>
<li><a href="https://www.onehouse.ai/blog/getting-started-manage-your-hudi-tables-with-the-admin-hudi-cli-tool">hudi cli</a></li>
<li><a href="https://www.waitingforcode.com/apache-hudi/acid-file-formats-writing-apache-hudi/read">hudi writers explained</a></li>
<li><a href="https://www.waitingforcode.com/apache-hudi/table-file-formats-reading-path-apache-hudi/read">hudi reader explained</a></li>
<li><a href="https://github.com/apache/hudi/pull/8758">record index PR</a></li>
<li><code>hudi.metadata-listing-enabled=TRUE</code> <a href="https://docs.aws.amazon.com/athena/latest/ug/querying-hudi.html">see athena official doc</a>
or <a href="https://apache-hudi.slack.com/archives/C4D716NPQ/p1684745312652149?thread_ts=1684739715.822219&cid=C4D716NPQ">or hudi slack</a></li>
<li><code>hive.hudi-metadata-enabled</code> <a href="https://github.com/prestodb/presto/commit/ef1fd25c582631513ccdd097e0a654cda44ec3dc">for presto</a></li>
<li><a href="https://medium.com/@simpsons/apache-hudi-compaction-6e6383790234">compaction</a></li>
<li><a href="https://medium.com/@simpsons/cleaner-and-archival-in-apache-hudi-9e15b08b2933">clean and archive</a></li>
<li><a href="https://hudi.apache.org/blog/2021/02/13/hudi-key-generators/">keygwnerator</a></li>
<li><a href="https://medium.com/@simpsons/bulk-insert-sort-modes-with-apache-hudi-c781e77841bc">bulk insert sort</a></li>
<li><a href="https://m.youtube.com/watch?v=sgfMdeD-yk4">early conflict detection</a></li>
<li><a href="https://medium.com/@simpsons/timeline-server-in-apache-hudi-b5be25f85e47?source=rss-24018a271c9b------2">timeline server</a></li>
<li><a href="https://hudi.apache.org/docs/disaster_recovery">savepoint</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets">record based index</a></li>
<li><a href="https://github.com/apache/hudi/issues/3737">one can move from COW into MOR table</a>
and <a href="https://github.com/apache/hudi/issues/2689">back mor to cow after compaction</a>
and <a href="https://hudi.apache.org/docs/faq/#how-to-convert-an-existing-cow-table-to-mor">FAQ</a></li>
<li><a href="https://medium.com/@simpsons/different-table-types-in-apache-hudi-datalake-apachehudi-cow-mor-f508c474cb8c">base files, slices, logs explained</a></li>
<li><a href="https://medium.com/@simpsons/monitoring-table-stats-22684eb70ee1">table stats tool</a></li>
<li><a href="https://programmer.ink/think/understanding-its-core-concepts-from-hudi-persistence-files.html">hudi file layout</a></li>
<li><a href="https://medium.com/@simpsons/hoodie-timeline-foundational-pillar-for-acid-transactions-be871399cbae">timeline deep dive</a></li>
<li><a href="https://www.uber.com/en-FR/blog/fast-copy-on-write-within-apache-parquet/">fast copy on write</a></li>
<li><a href="https://github.com/apache/hudi/blob/f1afb1bf04abdc94a26d61dc302f36ec2bbeb15b/rfc/rfc-68/rfc-68.md">fast copy on write hudi rfc</a></li>
<li><a href="https://github.com/apache/hudi/issues/506#issuecomment-440462970">insert step by step</a></li>
<li><a href="https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/ColumnStatsIndexSupport.scala#L53">column stats index</a></li>
<li><code>hoodie.combine.before.delete</code> when already deduplicated, avoids distinct count during deletion</li>
<li><a href="https://www.onehouse.ai/blog/the-road-to-an-open-and-interoperable-lakehouse">one table vs uniform</a></li>
<li><a href="https://medium.com/@simpsons/the-memo-we-missed-on-a-table-format-war-326397754e5">table format war</a></li>
<li><a href="https://medium.com/apache-hudi-blogs/employing-the-right-indexes-for-fast-updates-deletes-in-apache-hudi-814d863635f6">how to choose index</a></li>
<li><a href="https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md#future-plan">Disruptor helps both insert/bulk only</a></li>
<li><a href="https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=147427331#content/view/147427331">read path also MDT spec</a></li>
<li><a href="https://github.com/apache/hudi/blob/master/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieRealtimeRecordReader.java#L39">Mor reader code</a></li>
<li><a href="https://github.com/apache/hudi/pull/9235">spark new reader rfc</a></li>
<li><code>hoodie.datasource.hive_sync.filter_pushdown_enabled</code> when compare HMS and new added partition, this push down
partitions (otz will fetch all)</li>
</ul>
<h3>GDPR handling</h3>
<h4>Case of huge append only tables:</h4>
<p>One can use MOR tables, and bulk insert data, with global sort for read performance. This would lead to adding base
files only.
As for updates/deletes, it could be done by adding logs files on a regular basis. The table compaction could be done on
a monthly basis, which would reduce drastically the file amplification.
As for the <a href="https://hudi.apache.org/docs/configurations/#compactiontriggerstrategy">compaction strategy</a>, they could
trigger an inline compaction with large resources from time to time. Also they be carefully with the occ to avoid
killing the long running job. Likely they can limit the compact ion
scope <a href="https://hudi.apache.org/docs/configurations/#hoodiecompactionlogfilenumthreshold">either by number of logs per base file</a>
or
on <a href="https://hudi.apache.org/docs/configurations/#hoodiecompactiondaybasedtargetpartitions">day based partition strategy</a>;
there
is <a href="https://github.com/apache/hudi/tree/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/strategy">lot of strategies</a></p>
<p>An other approach would be to leverage the uber fast cow feature.</p>
<p>Questions remains:</p>
<ul>
<li>would insert produce base files for new partitions ? Yes, no log file w/o base file</li>
<li>would reading base files only partition be optimised ?</li>
<li>would compaction need huge resources ? There is multiple strategies, to split the task</li>
<li>would changing table layout mor / cow possible ? <a href="https://github.com/apache/hudi/issues/2689">Yes see</a></li>
</ul>
<h4>case of normal tables</h4>
<p>Either method can be used.</p>
<h3>metadata table</h3>
<ul>
<li>can be used by athena by specifying <code>hudi.metadata-listing-enabled=TRUE</code></li>
<li>after reading the MDT that
log : <code>table.HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE)</code></li>
<li>may be read incrementally by subsequently system such starrocks</li>
<li><a href="https://github.com/apache/hudi/blob/b5f749ccbf14de0f2cdced0918719ccdc113f2d5/hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java#L283">record level index</a></li>
</ul>
<h3>Hudi cli</h3>
<p>You can rollback commits, but from the latest to the newest. Not a given commit.</p>
<pre><code>connect --path s3://foo/bar
help commit rollback
commit rollback --commit 20230605210008584 --rollbackUsingMarkers false
</code></pre>
<ul>
<li>from 0.13, there is a bundle jar to run the CLI</li>
</ul>
<h3>Z-order</h3>
<ul>
<li>bulk insert with z-order/hilbert curve directly</li>
<li></li>
</ul>
<pre><code> "hoodie.clustering.plan.strategy.sort.columns": "col1,col2,col3", "hoodie.layout.optimize.build.curve.sample.size": "3",
"hoodie.layout.optimize.curve.build.method": "sample",
"hoodie.layout.optimize.strategy": "hilbert",
"hoodie.bulkinsert.user.defined.partitioner.class": "org.apache.hudi.execution.bulkinsert.RowSpatialCurveSortPartitioner",
</code></pre>
<ul>
<li><a href="https://aws.amazon.com/blogs/database/z-order-indexing-for-multifaceted-queries-in-amazon-dynamodb-part-2/">dynamodb z-order</a></li>
</ul>
<h3>Bloom investigations</h3>
<p>In apache spark:</p>
<ul>
<li><a href="https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOutputWriter.scala#L31-L36">ParquetOutputWriter in spark</a>
asks for parquet:ParquetOutputFormat which get the bloom configs</li>
<li><a href="https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala#L414C7-L492">ParquetUtils in spark</a>
has PrepareWrite function, which propagate to ParquetOurputWriter</li>
<li><a href="https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetWrite.scala#L35-L41">ParquetWrite in spark</a>
has prepareWrite function, which propagate to ParquetUtils.prepareWrite</li>
<li><a href="https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetTable.scala#L46-L50">ParquetTable in spark</a>
uses ParquetWrite</li>
<li><a href="https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetDataSourceV2.scala#L32-L37">ParquetDatasourceV2 in spark</a>
uses ParquetTable in getTable (then for read and write)</li>
<li><a href="https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala#L24">not really related: ParquetWriter in spark</a></li>
</ul>
<p>In apache hudi:</p>
<p>For bulk_insert:</p>
<ul>
<li>
<p><a href="">parquet:ParquetWriter is used from public constructor</a></p>
</li>
<li>
<p>[HoodieBaseParquetWriter extends ParquetWriter]</p>
</li>
<li>
<p>[HoodieInternalRowParquetWriter extends HoodieBaseParquetWriter]</p>
</li>
<li>
<p>[HoodieInternalRowFileWriterFactory return a HoodieInternalRowParquetWriter]</p>
</li>
<li>
<p>[HoodieRowCreateHandle returns a HoodieInternalRowFileWriterFactory]</p>
</li>
<li>
<p><a href="https://github.com/apache/iceberg/pull/4938">Iceberg bloom</a></p>
</li>
</ul>
<p>For insert:</p>
<ul>
<li>[HoodieBaseParquetWriter extends ParquetWriter]</li>
<li>[HoodieAvroParquetWriter]</li>
</ul>
<h3>Comment Investigation</h3>
<ul>
<li>when reading a regular hive table, it uses the <code>HiveSessionCatalog</code> which likely grab comments</li>
<li>its a v2SessionCatalog</li>
</ul>
<h3>Sorting</h3>
<h4>Custom sort</h4>
<ul>
<li><code>hoodie.bulkinsert.user.defined.partitioner.class=RowCustomColumnsSortPartitioner</code></li>
<li><code>hoodie.bulkinsert.user.defined.partitioner.sort.columns=col1,col2,col3</code></li>
<li><code>hoodie.bulkinsert.shuffle.parallelism=20</code></li>
</ul>
<p>It does a sort (order by) on specified columns and then coalesce, based on shuffle parallelism.</p>
<h4>GlobalSortPartitionerWithRows</h4>
<ul>
<li><code>hoodie.bulkinsert.sort.mode=GLOBAL_SORT</code></li>
<li><code>hoodie.bulkinsert.shuffle.parallelism=20</code></li>
</ul>
<p>It does a sort (order by) on both <code>_hoodie_partition_path</code> and <code>_hoodie_record_key</code> columns and then coalesce, based on
shuffle parallelism</p>
<h4>PartitionPathRepartitionAndSortPartitionerWithRows</h4>
<ul>
<li><code>hoodie.bulkinsert.sort.mode=PARTITION_PATH_REPARTITION_AND_SORT</code></li>
<li><code>hoodie.bulkinsert.shuffle.parallelism=20</code></li>
</ul>
<p>It does a repartition based on <code>_hoodie_partition_path</code> and then sortWithinPartitions based again
on <code>_hoodie_partition_path</code>.</p>
<p>Weird to both repartition and sort, the docs explains:</p>
<blockquote>
<p>this sort mode does an additional step of sorting the records based on the partition path
within a single Spark partition, given that data for multiple physical partitions can be sent to the same Spark
partition and executor. If data is skewed (most records are intended for a handful of partition paths among all) then
this can cause an imbalance among Spark executors.</p>
</blockquote>
<h4>PartitionSortPartitionerWithRows</h4>
<ul>
<li><code>hoodie.bulkinsert.sort.mode=PARTITION_SORT</code></li>
<li><code>hoodie.bulkinsert.shuffle.parallelism=20</code></li>
</ul>
<p>Same as PartitionPathRepartitionAndSortPartitionerWithRows, but no repartition before sorting.</p>
<h4>z-order support</h4>
<ul>
<li><a href="https://github.com/apache/hudi/blob/2c2abaf14bd0638051c641aaff529114756d3a9f/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/SpatialCurveSortPartitionerBase.java">Hudi can bulk insert directly in z-order/hilbert curve</a></li>
</ul>
<h3>Sizing</h3>
<ul>
<li><a href="https://hudi.apache.org/docs/configurations/#hoodieparquetpagesize">page size</a> = 1_048_576</li>
<li><a href="https://hudi.apache.org/docs/configurations/#hoodieparquetblocksize">row group size</a> = 125_829_120</li>
<li><a href="https://hudi.apache.org/docs/configurations/#hoodieparquetmaxfilesize">parquet max size</a> = 125_829_120</li>
</ul>
<h3>Merge On read tables</h3>
<ul>
<li><strong>athena</strong> likely not optimized with MOR tables, since it scans the whole parquet files
<pre><code class="language-sql"><span class="hl kwa">select</span> <span class="hl opt">*</span> <span class="hl kwa">from</span> test_hudi_mor_ro <span class="hl kwa">where</span> part <span class="hl opt">=</span> <span class="hl sng">'C'</span> <span class="hl kwa">and</span> uuid<span class="hl opt">=</span><span class="hl num">11</span>
<span class="hl slc">-- Execution time: 802 ms, Data scanned: 0 B, Approximate cost: $0.00</span>
<span class="hl kwa">select</span> <span class="hl opt">*</span> <span class="hl kwa">from</span> test_hudi_mor_rt <span class="hl kwa">where</span> part <span class="hl opt">=</span> <span class="hl sng">'C'</span> <span class="hl kwa">and</span> uuid<span class="hl opt">=</span><span class="hl num">11</span>
<span class="hl slc">-- Execution time: 2569 ms, Data scanned: 425.39 KB, Approximate cost: $0.00</span>
</code></pre>
</li>
<li><strong>spark</strong> cannot read <code>_rt</code> tables by datasource
<pre><code>: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.PartitionedFile.<init>(Lorg/apache/spark/sql/catalyst/InternalRow;Ljava/lang/String;JJ[Ljava/lang/String;)V
at org.apache.hudi.BaseMergeOnReadSnapshotRelation.$anonfun$buildSplits$2(MergeOnReadSnapshotRelation.scala:237)
at scala.Option.map(Option.scala:230)
at org.apache.hudi.BaseMergeOnReadSnapshotRelation.$anonfun$buildSplits$1(MergeOnReadSnapshotRelation.scala:235)
at scala.collection.immutable.List.map(List.scala:293)
at org.apache.hudi.BaseMergeOnReadSnapshotRelation.buildSplits(MergeOnReadSnapshotRelation.scala:231)
at org.apache.hudi.BaseMergeOnReadSnapshotRelation.collectFileSplits(MergeOnReadSnapshotRelation.scala:223)
at org.apache.hudi.BaseMergeOnReadSnapshotRelation.collectFileSplits(MergeOnReadSnapshotRelation.scala:64)
at org.apache.hudi.HoodieBaseRelation.buildScan(HoodieBaseRelation.scala:353)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.$anonfun$apply$4(DataSourceStrategy.scala:360)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.$anonfun$pruneFilterProject$1(DataSourceStrategy.scala:394)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.pruneFilterProjectRaw(DataSourceStrategy.scala:473)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.pruneFilterProject(DataSourceStrategy.scala:393)
at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.apply(DataSourceStrategy.scala:360)
at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:63)
</code></pre>
</li>
</ul>
<h3>Stats columns</h3>
<ul>
<li><a href="https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/ColumnStatsIndexSupport.scala#L219">column stat support</a></li>
</ul>
<h3>plugins</h3>
<h4>trino</h4>
<ul>
<li><a href="https://youtu.be/IiDOmAEOXUM">trino connector</a></li>
<li><a href="https://github.com/trinodb/trino/tree/1461847bbd034dd2ffdd44903e2589b13fb947e6/plugin/trino-hudi/src/main/java/io/trino/plugin/hudi">Trino plugin code</a></li>
<li><a href="https://github.com/trinodb/trino/blob/1461847bbd034dd2ffdd44903e2589b13fb947e6/plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiPartitionManager.java#L93">MDT usage for partitions</a></li>
<li><a href="https://github.com/trinodb/trino/blob/1461847bbd034dd2ffdd44903e2589b13fb947e6/plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiFileSkippingManager.java#L176">MDT stats data skipping</a></li>
<li><a href="https://github.com/apache/hudi/blob/3923bcdd599be0752044809ba0af6df260586738/rfc/rfc-64/rfc-64.md">rfc for unified plugins</a></li>
<li><a href="https://trino.io/episodes/41.html">trino on hudi</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Acid Tables — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-05 13:02</time> </div>https://blog.parisni.com/wiki/diskDisk2022-11-06T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">os</span>
<span class="w3-tag w3-red tag">linux</span></p>
<h1>Disk</h1>
<h2>Encrypted raid 1</h2>
<h3>Create a raid with mdadm</h3>
<pre><code>fdisk /dev/sda
fdisk /dev/sdc
mdadm -E /dev/sd[a-c]
mdadm -E /dev/sda1
mdadm -E /dev/sdc1
mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 /dev/sdc1
cat /proc/mdstat
mkdir /mnt/2TO
mkfs.ext4 /dev/md0
mount /dev/md0 /mnt/2TO
</code></pre>
<h3>Encrypt with luks</h3>
<pre><code>mdadm --detail /dev/md0
shred --verbose --random-source=/dev/urandom --iterations=3 /dev/md0
cryptsetup -y -v luksFormat /dev/md0
cryptsetup --verbose --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random luksFormat /dev/md0
man cryptsetup
gparted
modprobe dm-crypt
cryptsetup --cipher=aes-xts-plain --verify-passphrase --key-size=512 luksFormat /dev/md0
cryptsetup luksOpen /dev/md0 cryptdisk
mkfs.ext4 /dev/mapper/cryptdisk
mount -t ext4 /dev/mapper/cryptdisk /mnt
mount -t ext4 /dev/mapper/cryptdisk /mnt/2TO
</code></pre>
<h3>Configure startup</h3>
<p>This allows to ask for password at startup.
Set the UUID (present in /dev/part-by-uuid/<uuid>) for the mapper raid1</uuid></p>
<pre><code>/etc/crypttab
</code></pre>
<h3>Open on an other computer</h3>
<pre><code>mdadm --assemble /dev/md0 /dev/sda1 /dev/sdc1
apt install cryptsetup
cryptsetup luksOpen /dev/md0 cryptdisk
mount /dev/mapper/cryptdisk /mnt/cryptdisk
</code></pre>
<h3>Adding a luks key</h3>
<p>In order to skip password at boot (security issue) one cat create a key and provide its path to crypttab</p>
<p>see details
https://linuxconfig.org/how-to-use-a-file-as-a-luks-device-key</p>
<pre><code>cryptsetup luksDump /dev/md0 # just summarize luks info
dd if=/dev/urandom of=/cryptdisk-key bs=512 count=8 # create a key
cryptsetup luksAddKey /dev/md0 /cryptdisk-key # adds a new password with the key
</code></pre>
<p>now you can specify /cryptdisk-key as the key</p>
<h3>Schdeuled resync</h3>
<p>Resync check the md array to fix and sync disks. On debian, this is done every first sunday night per month</p>
<pre><code>root@natus:/home/natus# cat /etc/cron.d/mdadm
# By default, run at 00:57 on every Sunday, but do nothing unless the day of
# the month is less than or equal to 7. Thus, only run on the first Sunday of
# each month. crontab(5) sucks, unfortunately, in this regard; therefore this
# hack (see #380425).
57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Disk — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-11-06 21:22</time> </div>https://blog.parisni.com/wiki/pythonPython2023-04-23T20:48:00+00:00<div id="generated-toc"> </div><hr> <p>#programming</p>
<h1>Python</h1>
<h2>Create a cli executable</h2>
<h3>Use click</h3>
<ul>
<li><a href="https://click.palletsprojects.com/en/8.0.x/setuptools/">use setuptools</a></li>
</ul>
<h2>Environments</h2>
<h3>virualenv</h3>
<p>it is limited to the python version present on the host linux
distribution. it handles the pip package manager.</p>
<h3>anaconda</h3>
<p>anaconda allows to get multiple version of python even if the host
linux distribution does not. it handles both conda and pip package
managers, but conda is the prefered way.</p>
<h3>nix</h3>
<p>nix proposes an alternate way of packaging python environments. It is
not compatible with pip.</p>
<h3>guix</h3>
<p>it has less packages than nix and has the same drawbacks: python
packages has to be packaged the guix way to be accessible.</p>
<h3>pyenv/direnv</h3>
<p>This activates the virtualenv when you go into the folder.</p>
<p>Installation:</p>
<pre><code>curl https://pyenv.run | bash
# add to your bashrc
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
</code></pre>
<pre><code>source ~/.bahsrc
curl -sfL https://direnv.net/install.sh | bash
eval "$(direnv hook bash)"
</code></pre>
<p>Then install the python version you want:</p>
<pre><code>source ~/.bahsrc
pyenv install 3.8.11
</code></pre>
<p>Then in any python project:</p>
<pre><code>echo "layout pyenv 3.8.11" > .envrc
direnv allow
</code></pre>
<h2>Data</h2>
<h3>Pandas</h3>
<h3>Koalas</h3>
<h3>Map/Reduce</h3>
<h3>Dask</h3>
<p>Dask peut cohabiter avec spark sur un cluster hadoop via
<a href="https://docs.dask.org/en/latest/">dask-yarn</a>. Il peut lire/ecrire sur
hdfs des fichiers parquet/orc.</p>
<p>Le fonctionnement est largement inspiré de spark: dask s'appuie sur la
résolution d'un DAG lazy, avec possibilité de caching des étapes.</p>
<p>La finalité de dask est de transformer des big-data en small-data pour
pouvoir les traiter avec pandas.</p>
<h2>API</h2>
<h3>Flask</h3>
<h2>Dashboard</h2>
<h3>Dash</h3>
<h2>Database</h2>
<h3>ZODB</h3>
<p><a href="https://github.com/zopefoundation/ZODB">ZODB</a> uses “object traversal”
to retrieve information from the persistance layer. It supports ACID
transaction and is ideal for complex datamodel.</p>
<h2>Compilation</h2>
<h3>numba</h3>
<p><a href="https://github.com/numba/numba">numba</a></p>
<h3>cython</h3>
<h2>One liners</h2>
<p>Read file content:</p>
<pre><code class="language-python"><span class="hl slc">## Reading file contents into a list: one-liner</span>
file_lines <span class="hl opt">= [</span>line<span class="hl opt">.</span><span class="hl kwd">strip</span><span class="hl opt">()</span> <span class="hl kwa">for</span> line <span class="hl kwa">in</span> <span class="hl kwb">open</span><span class="hl opt">(</span>filename<span class="hl opt">)]</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Python — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-23 20:48</time> </div>https://blog.parisni.com/wiki/pleromaPleroma2020-06-20T21:42:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">federation</span>
<span class="w3-tag w3-red tag">libre</span></p>
<h1>Pleroma</h1>
<h2>Installation</h2>
<p>https://docs-develop.pleroma.social/backend/installation/arch_linux_en/</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Pleroma — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-20 21:42</time> </div>https://blog.parisni.com/wiki/duplicityDuplicity2022-11-06T00:38:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">backup</span></p>
<h1>Duplicity</h1>
<h2>Backup an encrypted directory on s3</h2>
<pre><code class="language-bash"><span class="hl slc">#!/usr/bin/env sh</span>
<span class="hl slc"># Your GPG key</span>
GPG_KEY<span class="hl opt">=<</span>the<span class="hl kwb">-gpg-key</span><span class="hl opt">></span>
SOURCE<span class="hl opt">=/</span>
<span class="hl slc"># The S3 destination followed by bucket name</span>
DEST<span class="hl opt">=</span><span class="hl sng">"s3://the-bucket"</span>
<span class="hl kwb">alias</span> duplicity<span class="hl opt">=</span><span class="hl sng">"docker run --rm --user root \</span>
<span class="hl sng"> -e AWS_ACCESS_KEY_ID=<id> \</span>
<span class="hl sng"> -e AWS_SECRET_ACCESS_KEY=<key> \</span>
<span class="hl sng"> -e PASSPHRASE=<gpg pass> \</span>
<span class="hl sng"> -v /root/.cache:/home/duplicity/.cache/duplicity \</span>
<span class="hl sng"> -v /root/.gnupg:/home/duplicity/.gnupg \</span>
<span class="hl sng"> -v /:/data:ro \</span>
<span class="hl sng"> wernight/duplicity \</span>
<span class="hl sng"> duplicity"</span>
duplicity \
<span class="hl kwb">--full-if-older-than</span> <span class="hl num">1</span>M \
<span class="hl kwb">--encrypt-key</span><span class="hl opt">=</span><span class="hl kwd">${GPG_KEY}</span> \
<span class="hl kwb">--sign-key</span><span class="hl opt">=</span><span class="hl kwd">${GPG_KEY}</span> \
<span class="hl kwb">--allow-source-mismatch</span> \
<span class="hl kwb">--s3-use-glacier</span> \
<span class="hl kwb">--s3-region-name</span><span class="hl opt">=<</span>s3<span class="hl kwb">-region</span><span class="hl opt">></span> \
<span class="hl kwb">--s3-endpoint-url</span><span class="hl opt">=<</span>s3<span class="hl kwb">-endpoint</span><span class="hl opt">>></span> \
<span class="hl kwb">--include</span><span class="hl opt">=/</span>opt \
<span class="hl kwb">--exclude</span><span class="hl opt">=/**</span> \
<span class="hl kwd">${SOURCE} ${DEST}</span>
duplicity remove<span class="hl kwb">-all-but-n-full</span> <span class="hl num">1</span> <span class="hl kwb">--force</span> <span class="hl kwd">${DEST}</span>
</code></pre>
<h2>Restore the backup</h2>
<p>This will restore the lastest state of the backup. Apparently, it is also
possible to play with time <code>--restore-time</code>.</p>
<pre><code class="language-bash">SOURCE<span class="hl opt">=</span><span class="hl sng">"s3://the-bucket"</span>
DEST<span class="hl opt">=/</span>mnt<span class="hl opt">/</span>backup<span class="hl opt">/</span>mailu
duplicity \
restore \
<span class="hl kwb">--encrypt-key</span><span class="hl opt">=</span><span class="hl kwd">${GPG_KEY}</span> \
<span class="hl kwb">--sign-key</span><span class="hl opt">=</span><span class="hl kwd">${GPG_KEY}</span> \
<span class="hl kwb">--s3-region-name</span><span class="hl opt">=<</span>s3<span class="hl kwb">-region</span><span class="hl opt">></span> \
<span class="hl kwb">--s3-endpoint-url</span><span class="hl opt">=<</span>s3<span class="hl kwb">-endpoint</span><span class="hl opt">>></span> \
<span class="hl kwd">${SOURCE} ${DEST}</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Duplicity — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-11-06 00:38</time> </div>https://blog.parisni.com/wiki/evinceEvince2021-10-17T12:01:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">pdf</span></p>
<h1>Evince</h1>
<h2>Save current layout</h2>
<p>When you click in the outline, this keeps the layout:</p>
<pre><code class="language-bash">gsettings <span class="hl kwb">set</span> org.gnome.Evince allow<span class="hl kwb">-links-change-zoom</span> false
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Evince — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-10-17 12:01</time> </div>https://blog.parisni.com/wiki/goaccessGoaccess2021-12-15T20:01:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">monitoring</span>
<span class="w3-tag w3-red tag">web</span></p>
<h1>Goaccess</h1>
<p>This tool is very light and provides a geo-location of ip. It is much lighter
than ELK, and cover the need of having a view on the hit on the web server.</p>
<h2>Install</h2>
<p>The real-time rendering works with a web-socket. The installation <a href="https://di-marco.net/blog/it/2020-01-02-real_time_goaccess_dashboard_with_nginx/">is very well
described
here</a>.</p>
<pre><code># wget https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
geoip-database /usr/local/etc/goaccess/dbip-city-lite-2021-12.mmdb
# Path where the persisted database files are stored on disk.
# The default value is the /tmp directory.
db-path /usr/local/etc/goaccess/
# Persist parsed data into disk.
persist true
# Load previously stored data from disk.
# Database files need to exist. See `persist`.
restore true
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Goaccess — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-15 20:01</time> </div>https://blog.parisni.com/wiki/gititGitit2020-03-08T02:08:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">wiki</span></p>
<h1>Gitit</h1>
<p>Gitit is a wiki based on git. Pages are rendered by <code>git cat</code> and
history with <code>git log</code>. This makes the wiki easy to maintain without
database. However this also makes the wiki not adapted to many users.</p>
<h2>Installation</h2>
<p>gitit is broken in nix. It is also broken in arch. It works when
compiling with <code>stack</code>:</p>
<pre><code class="language-bash">stack build <span class="hl kwb">--nix --nix-packages</span> zlib
stack <span class="hl kwc">install</span>
</code></pre>
<p>Then the <code>gitit</code> executable is available.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Gitit — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-03-08 02:08</time> </div>https://blog.parisni.com/wiki/iptablesIptables2022-03-06T23:45:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">firewall</span>
<span class="w3-tag w3-red tag">security</span></p>
<h1>Iptables</h1>
<h2>List every ipv4 rules</h2>
<pre><code class="language-bash">iptables <span class="hl kwb">-t</span> nat <span class="hl kwb">-L -n -v</span>
</code></pre>
<h2>Drop rules in specified table</h2>
<pre><code>iptables -t nat -F
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Iptables — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-03-06 23:45</time> </div>https://blog.parisni.com/wiki/fontFont2022-02-07T22:35:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">design</span></p>
<h1>Font</h1>
<h2>Luciole</h2>
<p><a href="https://www.luciole-vision.com">luciole-vision</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Font — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-02-07 22:35</time> </div>https://blog.parisni.com/wiki/AstronomyAstronomy2023-07-04T18:32:00+00:00<div id="generated-toc"> </div><hr> <h1>Astronomy</h1>
<ul>
<li><a href="http://www.astrosurf.com/delcroix/PicDuMidi_T1M/PicduMidi_T1M_Europlanet_workshop.htm">Pic du midi, photo 1M</a></li>
<li><a href="https://www.damianpeach.com/picdumidi2017.htm">pic du midi 2017</a></li>
<li><a href="https://www.europlanet-society.org/pic-net-ground-based-images-press-release/">pic du midi</a></li>
<li><a href="https://sdo.gsfc.nasa.gov/">solar eruption nasa</a></li>
<li><a href="https://webbtelescope.org/images">james webb</a></li>
<li><a href="https://www.nasa.gov/mission_pages/hubble/multimedia/index.html">Hubble images</a></li>
<li><a href="https://hubblesite.org/images/">Hubble site</a></li>
<li></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Astronomy — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-07-04 18:32</time> </div>https://blog.parisni.com/wiki/postgresPostgresql2024-01-14T18:33:00+00:00<div id="generated-toc"> </div><hr> <p>#database</p>
<h1>Postgresql</h1>
<h2>Changement de version</h2>
<p>Avec archlinux tout va très vite. Et lorsqu´ils décident de changer de
version majeure de postgres, après le reboot c´est le drame.</p>
<p>Comment gérer la crise?</p>
<h3>Simplement pg_upgrade</h3>
<pre><code class="language-bash"><span class="hl slc"># creer un noveau cluster postgres</span>
initdb <span class="hl kwb">-D</span> <span class="hl opt">/</span>mnt<span class="hl opt">/</span>cryptdisk<span class="hl opt">/</span>Postgres<span class="hl opt">/</span>data <span class="hl kwb">--locale</span><span class="hl opt">=</span>en_US.UTF<span class="hl kwb">-8</span>
<span class="hl slc">## installer les paquets postgresql-old-upgrade</span>
pacman.<span class="hl kwc">install</span> postgresql<span class="hl kwb">-old-update</span>
<span class="hl slc">## realiser la migration</span>
pg_upgrade <span class="hl kwb">-b</span> <span class="hl opt">/</span>opt<span class="hl opt">/</span>pgsql<span class="hl kwb">-11</span><span class="hl opt">/</span>bin <span class="hl kwb">-B</span> <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin <span class="hl kwb">-d</span> <span class="hl opt">/</span>opt<span class="hl opt">/</span>postgres<span class="hl opt">/</span>data <span class="hl kwb">-D</span> <span class="hl opt">/</span>mnt<span class="hl opt">/</span>cryptdisk<span class="hl opt">/</span>Postgres<span class="hl opt">/</span>data
<span class="hl slc">## lancer le vacuum</span>
.<span class="hl opt">/</span>analyze_new_cluster.sh
<span class="hl slc">## supprimer les anciennes données</span>
.<span class="hl opt">/</span>delete_old_cluster.sh
</code></pre>
<p>Il faut aussi penser à modifier les scripts systemd:</p>
<pre><code class="language-bash">...
<span class="hl slc"># en particulier la variable PGROOT</span>
Environment<span class="hl opt">=</span>PGROOT<span class="hl opt">=/</span>mnt<span class="hl opt">/</span>cryptdisk<span class="hl opt">/</span>Postgres
...
</code></pre>
<p>Désormais, la dernière version de postgres est utilisée.</p>
<h2>Calculer les statistiques</h2>
<pre><code class="language-sql"><span class="hl kwa">VACUUM ANALYZE</span> <span class="hl opt"><</span><span class="hl kwa">table</span><span class="hl opt">>;</span>
</code></pre>
<h2>Récupérer les éléments de statistiques</h2>
<pre><code class="language-sql"><span class="hl kwa">SELECT</span> schemaname<span class="hl opt">,</span> relname<span class="hl opt">,</span> last_analyze
<span class="hl kwa">FROM</span> pg_stat_all_tables
<span class="hl kwa">WHERE</span> relname <span class="hl opt">=</span> <span class="hl sng">'city'</span><span class="hl opt">;</span>
</code></pre>
<h2>Configurer le cluster</h2>
<p><a href="https://pgtune.leopard.in.ua/#/">This website is helpful</a></p>
<h2>Install postgis</h2>
<p><a href="https://postgis.net/install/">source</a>
:</p>
<pre><code class="language-bash">yum <span class="hl kwc">install</span> postgis30_12.x86_64
yum <span class="hl kwc">install</span> postgis30_12<span class="hl kwb">-utils</span>.x86_64
</code></pre>
<pre><code class="language-sql"><span class="hl slc">-- Enable PostGIS (as of 3.0 contains just geometry/geography)</span>
<span class="hl kwa">CREATE</span> EXTENSION postgis<span class="hl opt">;</span>
<span class="hl slc">-- enable raster support (for 3+)</span>
<span class="hl kwa">CREATE</span> EXTENSION postgis_raster<span class="hl opt">;</span>
<span class="hl slc">-- Enable Topology</span>
<span class="hl kwa">CREATE</span> EXTENSION postgis_topology<span class="hl opt">;</span>
<span class="hl slc">-- Enable PostGIS Advanced 3D</span>
<span class="hl slc">-- and other geoprocessing algorithms</span>
<span class="hl slc">-- sfcgal not available with all distributions</span>
<span class="hl kwa">CREATE</span> EXTENSION postgis_sfcgal<span class="hl opt">;</span>
<span class="hl slc">-- fuzzy matching needed for Tiger</span>
<span class="hl kwa">CREATE</span> EXTENSION fuzzystrmatch<span class="hl opt">;</span>
<span class="hl slc">-- rule based standardizer</span>
<span class="hl kwa">CREATE</span> EXTENSION address_standardizer<span class="hl opt">;</span>
<span class="hl slc">-- example rule data set</span>
<span class="hl kwa">CREATE</span> EXTENSION address_standardizer_data_us<span class="hl opt">;</span>
<span class="hl slc">-- Enable US Tiger Geocoder</span>
<span class="hl kwa">CREATE</span> EXTENSION postgis_tiger_geocoder<span class="hl opt">;</span>
</code></pre>
<h2>Passer des variables à un script via psql</h2>
<pre><code class="language-bash">psql <span class="hl kwb">-v</span> var1<span class="hl opt">=</span>foo <span class="hl kwb">-v</span> var2<span class="hl opt">=</span>bar <span class="hl kwb">-f</span> <span class="hl kwc">file</span>.sql
</code></pre>
<p>On peut à la fois passer des variables simples et des variables inclues dans des chaines de caractères.</p>
<pre><code class="language-sql"><span class="hl kwa">select</span> <span class="hl opt">*</span>
<span class="hl kwa">from</span> <span class="hl kwc">:var1</span>
<span class="hl kwa">where</span> <span class="hl opt">:</span><span class="hl sng">'var2'</span>
</code></pre>
<h2>Secure Dump script</h2>
<pre><code class="language-bash"><span class="hl slc">#! /bin/sh</span>
<span class="hl kwb">set -e</span>
f<span class="hl opt">=/</span>srv<span class="hl opt">/</span>otherfs<span class="hl opt">/</span>thebackup
<span class="hl kwc">rm</span> <span class="hl kwb">-f</span> <span class="hl sng">"</span><span class="hl ipl">$f</span><span class="hl sng">.new"</span> <span class="hl slc"># Remove a previous, failed backup, if any</span>
pg_dump <span class="hl kwb">-Fc -d</span> ourdatabase <span class="hl opt">></span><span class="hl sng">"</span><span class="hl ipl">$f</span><span class="hl sng">.new"</span>
pg_restore <span class="hl kwb">-l</span> <span class="hl sng">"</span><span class="hl ipl">$f</span><span class="hl sng">.new"</span> <span class="hl opt">>/</span>dev<span class="hl opt">/</span>null
<span class="hl kwc">mv</span> <span class="hl sng">"</span><span class="hl ipl">$f</span><span class="hl sng">.new"</span> <span class="hl sng">"</span><span class="hl ipl">$f</span><span class="hl sng">"</span>
<span class="hl kwb">exit</span> <span class="hl num">0</span> <span class="hl slc"># In case the previous line is a conditional like "if" or "&&" or "||".</span>
</code></pre>
<h2>Réaliser un dump simple</h2>
<pre><code class="language-bash">pg_dump database <span class="hl opt">> /</span>tmp<span class="hl opt">/</span>db.sql
</code></pre>
<h2>Charger un dump simple</h2>
<pre><code class="language-bash">psql roundcube <span class="hl opt">< /</span>tmp<span class="hl opt">/</span>db.sql
</code></pre>
<h2>Réaliser un dump optimisé</h2>
<p>ce script réalise un dump binaire, avec 4 threads:</p>
<pre><code class="language-bash"><span class="hl slc">#!/bin/bash</span>
<span class="hl kwb">set -e</span>
TMZ<span class="hl opt">=</span><span class="hl kwd">$(date +"%Y%m%d-%H%M%S")</span>
PG_VERSION<span class="hl opt">=</span><span class="hl num">12.1</span>
PG_HOST<span class="hl opt">=<</span>the <span class="hl kwc">host</span><span class="hl opt">></span>
DATABASE<span class="hl opt">=<</span>the database<span class="hl opt">></span>
PG_USER<span class="hl opt">=<</span>the user<span class="hl opt">></span>
THREAD<span class="hl opt">=</span><span class="hl num">4</span>
PG_FOLDER<span class="hl opt">=/</span>usr<span class="hl opt">/</span>pgsql<span class="hl opt">-</span><span class="hl kwd">$PG_VERSION</span><span class="hl opt">/</span>bin
DUMP_FOLDER<span class="hl opt">=/</span>dump<span class="hl opt">/</span><span class="hl kwd">$DATABASE</span><span class="hl opt">/</span>
<span class="hl kwd">$PG_FOLDER</span><span class="hl opt">/</span>pg_dump <span class="hl kwb">-h</span> <span class="hl kwd">$PG_HOST</span> <span class="hl kwb">-p</span> <span class="hl num">5432</span> <span class="hl kwb">-U</span> <span class="hl kwd">$PG_USER</span> <span class="hl kwb">-w -Fd</span> <span class="hl kwd">$DATABASE</span> <span class="hl kwb">-n</span> <span class="hl opt"><</span>schema1<span class="hl opt">></span> <span class="hl kwb">-n</span> <span class="hl opt"><</span>schema2<span class="hl opt">></span> <span class="hl kwb">-j</span><span class="hl kwd">$THREAD</span> <span class="hl kwb">-f</span> <span class="hl kwd">$DUMP_FOLDER</span><span class="hl opt">/</span>
<span class="hl kwb">echo</span> <span class="hl sng">"create database</span> <span class="hl ipl">$DATABASE</span><span class="hl sng">"</span> <span class="hl opt">></span> <span class="hl kwd">$DUMP_FOLDER</span><span class="hl opt">/</span>create_database.sql
<span class="hl kwb">echo</span> <span class="hl sng">"\c</span> <span class="hl ipl">$DATABASE</span><span class="hl sng">"</span> <span class="hl opt">>></span> <span class="hl kwd">$DUMP_FOLDER</span><span class="hl opt">/</span>create_database.sql
<span class="hl kwd">$PG_FOLDER</span><span class="hl opt">/</span>pg_dumpall <span class="hl kwb">-h</span> <span class="hl kwd">$PG_HOST</span> <span class="hl kwb">-p</span> <span class="hl num">5432</span> <span class="hl kwb">-U</span> <span class="hl kwd">$PG_USER</span> <span class="hl kwb">-w --globals-only</span> <span class="hl opt">>></span> <span class="hl kwd">$DUMP_FOLDER</span><span class="hl opt">/</span>create_database.sql
</code></pre>
<h2>Charger un dump optimisé</h2>
<pre><code class="language-bash"><span class="hl slc">#!/bin/bash</span>
<span class="hl kwb">set -e</span>
TMZ<span class="hl opt">=</span><span class="hl kwd">$(date +"%Y%m%d-%H%M%S")</span>
PG_VERSION<span class="hl opt">=</span><span class="hl num">12</span>
PG_HOST<span class="hl opt">=<</span>the <span class="hl kwc">host</span><span class="hl opt">></span>
DATABASE<span class="hl opt">=<</span>the db<span class="hl opt">></span>
USER<span class="hl opt">=<</span>the user<span class="hl opt">></span>
THREAD<span class="hl opt">=</span><span class="hl num">4</span>
PG_FOLDER<span class="hl opt">=/</span>usr<span class="hl opt">/</span>pgsql<span class="hl opt">-</span><span class="hl kwd">$PG_VERSION</span><span class="hl opt">/</span>bin
DUMP_FOLDER<span class="hl opt">=/</span>tmp<span class="hl opt">/</span><span class="hl kwd">$DATABASE</span><span class="hl opt">/</span>
<span class="hl kwc">cat</span> <span class="hl kwd">$DUMP_FOLDER</span><span class="hl opt">/</span>create_database.sql <span class="hl opt">></span> <span class="hl kwd">$PG_FOLDER</span><span class="hl opt">/</span>psql <span class="hl kwb">-U</span> <span class="hl kwd">$USER</span> <span class="hl kwb">-W -h</span> <span class="hl kwd">$PG_HOST</span> <span class="hl kwb">-d</span> postgres
<span class="hl kwd">$PG_FOLDER</span><span class="hl opt">/</span>pg_restore <span class="hl kwb">-Fd</span> <span class="hl kwd">$DUMP_FOLDER</span> <span class="hl kwb">-n</span> <span class="hl opt"><</span>schema1<span class="hl opt">></span> <span class="hl kwb">-n</span> <span class="hl opt"><</span>schema2<span class="hl opt">></span> <span class="hl kwb">-j</span><span class="hl kwd">$THREAD</span> <span class="hl kwb">-U</span> <span class="hl kwd">$USER</span> <span class="hl kwb">-h</span> <span class="hl kwd">$PG_HOST</span> <span class="hl kwb">-d</span> <span class="hl kwd">$DATABASE</span>
</code></pre>
<h2>Table corrompue</h2>
<p><code>psycopg2.errors.DataCorrupted: invalid page in block 2 of relation base/1687445/1688449</code></p>
<ol>
<li><code>SELECT oid, relname as damaged_table FROM pg_class WHERE oid=1688449;</code></li>
<li><code>SET zero_damaged_pages = on;</code></li>
<li><code>VACUUM FULL damaged_table;</code></li>
<li><code>REINDEX TABLE damaged_table;</code></li>
</ol>
<p><a href="https://stackoverflow.com/questions/5220344/postgresql-invalid-page-header-in-block">from</a></p>
<h2>psqlrc</h2>
<pre><code class="language-bash">\<span class="hl kwb">set</span> QUIET ON
<span class="hl esc">\t</span>iming on
\pset pager <span class="hl num">0</span>
\x auto
<span class="hl kwb">--</span> Put a space <span class="hl kwa">in</span> front of queries you don<span class="hl sng">'t want to save</span>
<span class="hl sng">\set HISTCONTROL ignorespace</span>
<span class="hl sng">\set HISTFILE ~/.psql_history- :DBNAME</span>
<span class="hl sng">\set HISTSIZE -1</span>
<span class="hl sng">\pset null '</span>NULL<span class="hl sng">'</span>
<span class="hl sng">\pset border 2</span>
<span class="hl sng"></span>
<span class="hl sng">\set QUIET OFF</span>
<span class="hl sng">\encoding unicode</span>
<span class="hl sng"></span>
<span class="hl sng">\echo '</span><span class="hl esc">\n</span>Current Host Server Date Time <span class="hl opt">:</span> <span class="hl sng">'`date` '</span><span class="hl esc">\n</span><span class="hl sng">'</span>
<span class="hl sng"></span>
<span class="hl sng">\echo '</span>Administrative queries<span class="hl opt">:</span><span class="hl esc">\n</span><span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>settings<span class="hl esc">\t</span><span class="hl kwb">--</span> Server Settings<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>conninfo<span class="hl esc">\t</span><span class="hl kwb">--</span> Server connections<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>activity<span class="hl esc">\t</span><span class="hl kwb">--</span> Server activity<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>locks<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Lock info<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>waits<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Waiting queires<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>dbsize<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Database Size<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>tablesize<span class="hl esc">\t</span><span class="hl kwb">--</span> Tables Size<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>uselesscol<span class="hl esc">\t</span><span class="hl kwb">--</span> Useless columns<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>uptime<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Server uptime<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>menu<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Help Menu<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t\\</span>h<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Help with SQL commands<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t\\</span>?<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Help with psql commands<span class="hl esc">\n</span><span class="hl sng">'</span>
<span class="hl sng"></span>
<span class="hl sng">\echo '</span>Development queries<span class="hl opt">:</span><span class="hl esc">\n</span><span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span>sp<span class="hl esc">\t\t</span><span class="hl kwb">--</span> Current Search Path<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span><span class="hl kwc">clear</span><span class="hl esc">\t\t</span><span class="hl kwb">--</span> Clear screen<span class="hl sng">'</span>
<span class="hl sng">\echo '</span><span class="hl esc">\t\t\t</span><span class="hl opt">:</span><span class="hl kwc">ll</span><span class="hl esc">\t\t</span><span class="hl kwb">--</span> List<span class="hl esc">\n</span><span class="hl sng">'</span>
<span class="hl sng"></span>
<span class="hl sng">-- Administration queries</span>
<span class="hl sng"></span>
<span class="hl sng">\set menu '</span><span class="hl esc">\\</span>i ~<span class="hl opt">/</span>.psqlrc<span class="hl sng">'</span>
<span class="hl sng"></span>
<span class="hl sng">\set settings '</span><span class="hl kwa">select</span> name<span class="hl opt">,</span> setting<span class="hl opt">,</span>unit<span class="hl opt">,</span>context from pg_settings<span class="hl opt">;</span><span class="hl sng">'</span>
<span class="hl sng"></span>
<span class="hl sng">\set conninfo '</span><span class="hl kwa">select</span> usename<span class="hl opt">,</span> count<span class="hl opt">(*)</span> from pg_stat_activity group by usename<span class="hl opt">;</span><span class="hl sng">'</span>
<span class="hl sng"></span>
<span class="hl sng">\set activity '</span><span class="hl kwa">select</span> datname<span class="hl opt">,</span> pid<span class="hl opt">,</span> usename<span class="hl opt">,</span> client_addr<span class="hl opt">,</span> substring<span class="hl opt">(</span>query<span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">50</span><span class="hl opt">)</span> as query<span class="hl opt">, (</span> <span class="hl kwa">select</span> array_agg<span class="hl opt">(</span>unnest<span class="hl opt">)</span> from <span class="hl opt">(</span><span class="hl kwa">select</span> distinct unnest<span class="hl opt">(</span>array_remove<span class="hl opt">(</span>array_remove<span class="hl opt">(</span>array_remove<span class="hl opt">(</span>array_remove<span class="hl opt">(</span>array_remove<span class="hl opt">(</span>array_remove<span class="hl opt">(</span>array_remove<span class="hl opt">(</span>regexp_matches<span class="hl opt">(</span>lower<span class="hl opt">(</span>query<span class="hl opt">),</span> <span class="hl sng">''</span>\<span class="hl opt">(</span>from<span class="hl opt">|</span>into<span class="hl opt">|</span><span class="hl kwc">join</span><span class="hl opt">|</span>update<span class="hl opt">|</span>delete\<span class="hl opt">) +([</span>A<span class="hl kwb">-z_</span><span class="hl esc">\\</span>.<span class="hl sng">"]+)'',''g''), NULL), ''from''), ''into''), ''update''), ''delete''),''set''), ''join'')) as unnest ) as b) as tables, wait_event, wait_event_type, state_change from pg_stat_activity where state =</span> <span class="hl esc">\'</span><span class="hl sng">active</span><span class="hl esc">\'</span> <span class="hl sng">and query !~</span> <span class="hl esc">\'</span><span class="hl sng">pg_stat_activity</span><span class="hl esc">\'</span><span class="hl sng">;'</span>
<span class="hl sng"></span>
<span class="hl sng">\set activityQuery 'select pid, usename, query from pg_stat_activity where state =</span> <span class="hl esc">\'</span><span class="hl sng">active</span><span class="hl esc">\'</span> <span class="hl sng">and query !~</span> <span class="hl esc">\'</span><span class="hl sng">pg_stat_activity</span><span class="hl esc">\'</span><span class="hl sng">;'</span>
<span class="hl sng"></span>
<span class="hl sng">\set waits 'SELECT pg_stat_activity.pid, pg_stat_activity.query, pg_stat_activity.waiting, now() - pg_stat_activity.query_start AS</span> <span class="hl esc">\"</span><span class="hl sng">totaltime</span><span class="hl esc">\"</span><span class="hl sng">, pg_stat_activity.backend_start FROM pg_stat_activity WHERE pg_stat_activity.query !~</span> <span class="hl esc">\'</span><span class="hl sng">%IDLE%</span><span class="hl esc">\'</span><span class="hl sng">::text AND pg_stat_activity.waiting = true;'</span>
<span class="hl sng"></span>
<span class="hl sng">\set dbsize 'SELECT d.datname as Name, pg_catalog.pg_get_userbyid(d.datdba) as Owner, CASE WHEN pg_catalog.has_database_privilege(d.datname,</span> <span class="hl esc">\'</span><span class="hl sng">CONNECT</span><span class="hl esc">\'</span><span class="hl sng">) THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname)) ELSE</span> <span class="hl esc">\'</span><span class="hl sng">No Access</span><span class="hl esc">\'</span> <span class="hl sng">END as Size FROM pg_catalog.pg_database d order by CASE WHEN pg_catalog.has_database_privilege(d.datname,</span> <span class="hl esc">\'</span><span class="hl sng">CONNECT</span><span class="hl esc">\'</span><span class="hl sng">) THEN pg_catalog.pg_database_size(d.datname) ELSE NULL END desc LIMIT 20;'</span>
<span class="hl sng"></span>
<span class="hl sng">\set tablesize ' SELECT relname AS "</span>relation<span class="hl sng">", pg_size_pretty(pg_total_relation_size(C.oid)) AS "</span>total_size<span class="hl sng">", pg_size_pretty(pg_indexes_size(C.oid)) "</span>index_size<span class="hl sng">", reltuples::bigint AS approximate_row_count FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN (</span><span class="hl esc">\'</span><span class="hl sng">pg_catalog</span><span class="hl esc">\'</span><span class="hl sng">,</span> <span class="hl esc">\'</span><span class="hl sng">information_schema</span><span class="hl esc">\'</span><span class="hl sng">) AND C.relkind <></span> <span class="hl esc">\'</span><span class="hl sng">i</span><span class="hl esc">\'</span> <span class="hl sng">AND nspname !~</span> <span class="hl esc">\'</span><span class="hl sng">^pg_toast</span><span class="hl esc">\'</span> <span class="hl sng">ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 50; '</span>
<span class="hl sng"></span>
<span class="hl sng">\set uselesscol 'SELECT nspname, relname, attname, typname, (stanullfrac*100)::int AS null_percent, case when stadistinct &gt;= 0 then stadistinct else abs(stadistinct)*reltuples end AS</span> <span class="hl esc">\"</span><span class="hl sng">distinct</span><span class="hl esc">\"</span><span class="hl sng">, case 1 when stakind1 then stavalues1 when stakind2 then stavalues2 end AS</span> <span class="hl esc">\"</span><span class="hl sng">values</span><span class="hl esc">\"</span> <span class="hl sng">FROM pg_class c JOIN pg_namespace ns ON (ns.oid=relnamespace) JOIN pg_attribute ON (c.oid=attrelid) JOIN pg_type t ON (t.oid=atttypid) JOIN pg_statistic ON (c.oid=starelid AND staattnum=attnum) WHERE nspname NOT LIKE E</span><span class="hl esc">\'</span><span class="hl sng">pg</span><span class="hl esc">\\\\</span><span class="hl sng">_%</span><span class="hl esc">\'</span> <span class="hl sng">AND nspname !=</span> <span class="hl esc">\'</span><span class="hl sng">information_schema</span><span class="hl esc">\'</span> <span class="hl sng">AND relkind=</span><span class="hl esc">\'</span><span class="hl sng">r</span><span class="hl esc">\'</span> <span class="hl sng">AND NOT attisdropped AND attstattarget != 0 AND reltuples &gt;= 100 AND stadistinct BETWEEN 0 AND 1 ORDER BY nspname, relname, attname;'</span>
<span class="hl sng"></span>
<span class="hl sng">\set tmpfiles 'SELECT temp_files AS "</span>Temporary files<span class="hl sng">" , pg_size_pretty(temp_bytes) AS "</span>Size of temporary files<span class="hl sng">" FROM pg_stat_database db order by 2 desc;'</span>
<span class="hl sng"></span>
<span class="hl sng">\set uptime 'select now() - pg_postmaster_start_time() AS uptime;'</span>
<span class="hl sng"></span>
<span class="hl sng">\set locks ' SELECT blocked_locks.pid AS blocked_pid, blocked_activity.usename AS blocked_user, blocking_locks.pid AS blocking_pid, blocking_activity.usename AS blocking_user, substring(blocked_activity.query,1,50) AS blocked_statement, substring(blocking_activity.query,1,50) AS current_statement_in_blocking_process FROM pg_catalog.pg_locks blocked_locks JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.pid != blocked_locks.pid JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid; '</span>
<span class="hl sng"></span>
<span class="hl sng">\set cleaning 'select relname,last_vacuum, last_autovacuum, last_analyze, last_autoanalyze from pg_stat_user_tables order by SUBSTRING(relname FROM ''([0-9]+)'')::BIGINT ASC;'</span>
<span class="hl sng"></span>
<span class="hl sng">-- Development queries:</span>
<span class="hl sng"></span>
<span class="hl sng">\set sp 'SHOW search_path;'</span>
<span class="hl sng">\set clear '</span><span class="hl esc">\\</span><span class="hl sng">! clear;'</span>
<span class="hl sng">\set ll '</span><span class="hl esc">\\</span><span class="hl sng">! ls -lrt;'</span>
</code></pre>
<h2>show enum values</h2>
<pre><code>\dT+
</code></pre>
<h2>Show cache hit ratio</h2>
<p>Value ~= 100% means shared memory is sufficient.</p>
<pre><code class="language-sql"><span class="hl kwa">SELECT</span>
datname<span class="hl opt">,</span> <span class="hl num">100</span> <span class="hl opt">*</span> blks_hit <span class="hl opt">/ (</span>blks_hit <span class="hl opt">+</span> blks_read<span class="hl opt">)</span> <span class="hl kwa">as</span> cache_hit_ratio
<span class="hl kwa">FROM</span> pg_stat_database <span class="hl kwa">WHERE</span> <span class="hl opt">(</span>blks_hit <span class="hl opt">+</span> blks_read<span class="hl opt">) ></span> <span class="hl num">0</span><span class="hl opt">;</span>
<span class="hl opt">+</span><span class="hl slc">-----------+-----------------+</span>
| datname | cache_hit_ratio |
<span class="hl opt">+</span><span class="hl slc">-----------+-----------------+</span>
| <span class="hl kwa">NULL</span> | <span class="hl num">99</span> |
| codimd | <span class="hl num">99</span> |
| synapse | <span class="hl num">99</span> |
| wallabag | <span class="hl num">99</span> |
| postgres | <span class="hl num">99</span> |
| pleroma | <span class="hl num">99</span> |
| miniflux | <span class="hl num">99</span> |
| funkwhale | <span class="hl num">99</span> |
| peertube | <span class="hl num">99</span> |
| bitwarden | <span class="hl num">99</span> |
| mailu | <span class="hl num">99</span> |
| roundcube | <span class="hl num">99</span> |
<span class="hl opt">+</span><span class="hl slc">-----------+-----------------+</span>
</code></pre>
<h2>Read only user</h2>
<pre><code class="language-sql"><span class="hl slc">-- create</span>
<span class="hl kwa">CREATE ROLE</span> the_user <span class="hl kwa">WITH LOGIN PASSWORD</span> <span class="hl sng">'Test1234'</span>
<span class="hl kwa">NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE</span> NOREPLICATION <span class="hl kwa">VALID UNTIL</span> <span class="hl sng">'infinity'</span><span class="hl opt">;</span>
\<span class="hl kwa">connect</span> the_database<span class="hl opt">;</span> <span class="hl slc">-- essentiel de se connecter dans la db sinon marche pas</span>
<span class="hl kwa">GRANT CONNECT ON DATABASE</span> the_database <span class="hl kwa">TO</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">GRANT USAGE ON SCHEMA public TO</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">GRANT SELECT ON ALL TABLES IN SCHEMA public TO</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">REVOKE CREATE ON SCHEMA public FROM PUBLIC</span><span class="hl opt">;</span>
<span class="hl kwa">ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">ALTER USER</span> the_user <span class="hl kwa">SET</span> idle_in_transaction_session_timeout <span class="hl kwa">TO</span> <span class="hl num">60000</span><span class="hl opt">;</span> <span class="hl slc">-- 1 minute</span>
<span class="hl slc">-- to drop the user</span>
<span class="hl kwa">REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">REVOKE ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public FROM</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">REVOKE ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public FROM</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">DROP OWNED BY</span> the_user<span class="hl opt">;</span>
<span class="hl kwa">DROP role</span> the_user<span class="hl opt">;</span>
</code></pre>
<h2>blog</h2>
<ul>
<li><a href="https://ottertune.com/blog/the-part-of-postgresql-we-hate-the-most/">MVCC</a></li>
</ul>
<h2>Query/query plan obfuscation</h2>
<ul>
<li><a href="https://explain.depesz.com/">depesz</a></li>
</ul>
<h2>Order by alphanumeric</h2>
<pre><code>ORDER BY SUBSTRING(col_name FROM ''([0-9]+)'')::BIGINT ASC
</code></pre>
<h2>you should always log postgres</h2>
<ul>
<li><a href="https://www.endpointdev.com/blog/2012/06/logstatement-postgres-all-full-logging/">reason</a></li>
</ul>
<h2>Show create table</h2>
<pre><code class="language-sql"><span class="hl kwa">CREATE OR REPLACE FUNCTION public</span>.<span class="hl kwd">generate_create_table_statement</span><span class="hl opt">(</span>p_table_name <span class="hl kwb">character</span> <span class="hl kwa">varying</span><span class="hl opt">)</span>
<span class="hl kwa">RETURNS SETOF</span> <span class="hl kwb">text</span> <span class="hl kwa">AS</span>
$BODY$
<span class="hl kwa">DECLARE</span>
v_table_ddl <span class="hl kwb">text</span><span class="hl opt">;</span>
column_record record<span class="hl opt">;</span>
table_rec record<span class="hl opt">;</span>
constraint_rec record<span class="hl opt">;</span>
firstrec <span class="hl kwb">boolean</span><span class="hl opt">;</span>
<span class="hl kwa">BEGIN</span>
<span class="hl kwa">FOR</span> table_rec <span class="hl kwa">IN</span>
<span class="hl kwa">SELECT</span> c.relname <span class="hl kwa">FROM</span> pg_catalog.pg_class c
<span class="hl kwa">LEFT JOIN</span> pg_catalog.pg_namespace n <span class="hl kwa">ON</span> n.oid <span class="hl opt">=</span> c.relnamespace
<span class="hl kwa">WHERE</span> relkind <span class="hl opt">=</span> <span class="hl sng">'r'</span>
<span class="hl kwa">AND</span> relname~ <span class="hl opt">(</span><span class="hl sng">'^('</span>||p_table_name||<span class="hl sng">')$'</span><span class="hl opt">)</span>
<span class="hl kwa">AND</span> n.nspname <span class="hl opt"><></span> <span class="hl sng">'pg_catalog'</span>
<span class="hl kwa">AND</span> n.nspname <span class="hl opt"><></span> <span class="hl sng">'information_schema'</span>
<span class="hl kwa">AND</span> n.nspname <span class="hl opt">!</span>~ <span class="hl sng">'^pg_toast'</span>
<span class="hl kwa">AND</span> pg_catalog.<span class="hl kwd">pg_table_is_visible</span><span class="hl opt">(</span>c.oid<span class="hl opt">)</span>
<span class="hl kwa">ORDER BY</span> c.relname
LOOP
<span class="hl kwa">FOR</span> column_record <span class="hl kwa">IN</span>
<span class="hl kwa">SELECT</span>
b.nspname <span class="hl kwa">as schema_name</span><span class="hl opt">,</span>
b.relname <span class="hl kwa">as table_name</span><span class="hl opt">,</span>
a.attname <span class="hl kwa">as column_name</span><span class="hl opt">,</span>
pg_catalog.<span class="hl kwd">format_type</span><span class="hl opt">(</span>a.atttypid<span class="hl opt">,</span> a.atttypmod<span class="hl opt">)</span> <span class="hl kwa">as</span> column_type<span class="hl opt">,</span>
<span class="hl kwa">CASE WHEN</span>
<span class="hl opt">(</span><span class="hl kwa">SELECT substring</span><span class="hl opt">(</span>pg_catalog.<span class="hl kwd">pg_get_expr</span><span class="hl opt">(</span>d.adbin<span class="hl opt">,</span> d.adrelid<span class="hl opt">)</span> <span class="hl kwa">for</span> <span class="hl num">128</span><span class="hl opt">)</span>
<span class="hl kwa">FROM</span> pg_catalog.pg_attrdef d
<span class="hl kwa">WHERE</span> d.adrelid <span class="hl opt">=</span> a.attrelid <span class="hl kwa">AND</span> d.adnum <span class="hl opt">=</span> a.attnum <span class="hl kwa">AND</span> a.atthasdef<span class="hl opt">)</span> <span class="hl kwa">IS NOT NULL THEN</span>
<span class="hl sng">'DEFAULT '</span>|| <span class="hl opt">(</span><span class="hl kwa">SELECT substring</span><span class="hl opt">(</span>pg_catalog.<span class="hl kwd">pg_get_expr</span><span class="hl opt">(</span>d.adbin<span class="hl opt">,</span> d.adrelid<span class="hl opt">)</span> <span class="hl kwa">for</span> <span class="hl num">128</span><span class="hl opt">)</span>
<span class="hl kwa">FROM</span> pg_catalog.pg_attrdef d
<span class="hl kwa">WHERE</span> d.adrelid <span class="hl opt">=</span> a.attrelid <span class="hl kwa">AND</span> d.adnum <span class="hl opt">=</span> a.attnum <span class="hl kwa">AND</span> a.atthasdef<span class="hl opt">)</span>
<span class="hl kwa">ELSE</span>
<span class="hl sng">''</span>
<span class="hl kwa">END as</span> column_default_value<span class="hl opt">,</span>
<span class="hl kwa">CASE WHEN</span> a.attnotnull <span class="hl opt">=</span> <span class="hl kwa">true THEN</span>
<span class="hl sng">'NOT NULL'</span>
<span class="hl kwa">ELSE</span>
<span class="hl sng">'NULL'</span>
<span class="hl kwa">END as</span> column_not_null<span class="hl opt">,</span>
a.attnum <span class="hl kwa">as</span> attnum<span class="hl opt">,</span>
e.max_attnum <span class="hl kwa">as</span> max_attnum
<span class="hl kwa">FROM</span>
pg_catalog.pg_attribute a
<span class="hl kwa">INNER JOIN</span>
<span class="hl opt">(</span><span class="hl kwa">SELECT</span> c.oid<span class="hl opt">,</span>
n.nspname<span class="hl opt">,</span>
c.relname
<span class="hl kwa">FROM</span> pg_catalog.pg_class c
<span class="hl kwa">LEFT JOIN</span> pg_catalog.pg_namespace n <span class="hl kwa">ON</span> n.oid <span class="hl opt">=</span> c.relnamespace
<span class="hl kwa">WHERE</span> c.relname <span class="hl opt">=</span> table_rec.relname
<span class="hl kwa">AND</span> pg_catalog.<span class="hl kwd">pg_table_is_visible</span><span class="hl opt">(</span>c.oid<span class="hl opt">)</span>
<span class="hl kwa">ORDER BY</span> <span class="hl num">2</span><span class="hl opt">,</span> <span class="hl num">3</span><span class="hl opt">)</span> b
<span class="hl kwa">ON</span> a.attrelid <span class="hl opt">=</span> b.oid
<span class="hl kwa">INNER JOIN</span>
<span class="hl opt">(</span><span class="hl kwa">SELECT</span>
a.attrelid<span class="hl opt">,</span>
<span class="hl kwa">max</span><span class="hl opt">(</span>a.attnum<span class="hl opt">)</span> <span class="hl kwa">as</span> max_attnum
<span class="hl kwa">FROM</span> pg_catalog.pg_attribute a
<span class="hl kwa">WHERE</span> a.attnum <span class="hl opt">></span> <span class="hl num">0</span>
<span class="hl kwa">AND NOT</span> a.attisdropped
<span class="hl kwa">GROUP BY</span> a.attrelid<span class="hl opt">)</span> e
<span class="hl kwa">ON</span> a.attrelid<span class="hl opt">=</span>e.attrelid
<span class="hl kwa">WHERE</span> a.attnum <span class="hl opt">></span> <span class="hl num">0</span>
<span class="hl kwa">AND NOT</span> a.attisdropped
<span class="hl kwa">ORDER BY</span> a.attnum
LOOP
<span class="hl kwa">IF</span> column_record.attnum <span class="hl opt">=</span> <span class="hl num">1</span> <span class="hl kwa">THEN</span>
v_table_ddl<span class="hl opt">:=</span><span class="hl sng">'CREATE TABLE '</span>||column_record.<span class="hl kwa">schema_name</span>||<span class="hl sng">'.'</span>||column_record.<span class="hl kwa">table_name</span>||<span class="hl sng">' ('</span><span class="hl opt">;</span>
<span class="hl kwa">ELSE</span>
v_table_ddl<span class="hl opt">:=</span>v_table_ddl||<span class="hl sng">','</span><span class="hl opt">;</span>
<span class="hl kwa">END IF</span><span class="hl opt">;</span>
<span class="hl kwa">IF</span> column_record.attnum <span class="hl opt"><=</span> column_record.max_attnum <span class="hl kwa">THEN</span>
v_table_ddl<span class="hl opt">:=</span>v_table_ddl||<span class="hl kwd">chr</span><span class="hl opt">(</span><span class="hl num">10</span><span class="hl opt">)</span>||
<span class="hl sng">' '</span>||column_record.<span class="hl kwa">column_name</span>||<span class="hl sng">' '</span>||column_record.column_type||<span class="hl sng">' '</span>||column_record.column_default_value||<span class="hl sng">' '</span>||column_record.column_not_null<span class="hl opt">;</span>
<span class="hl kwa">END IF</span><span class="hl opt">;</span>
<span class="hl kwa">END</span> LOOP<span class="hl opt">;</span>
firstrec <span class="hl opt">:=</span> <span class="hl kwa">TRUE</span><span class="hl opt">;</span>
<span class="hl kwa">FOR</span> constraint_rec <span class="hl kwa">IN</span>
<span class="hl kwa">SELECT</span> conname<span class="hl opt">,</span> <span class="hl kwd">pg_get_constraintdef</span><span class="hl opt">(</span>c.oid<span class="hl opt">)</span> <span class="hl kwa">as</span> constrainddef
<span class="hl kwa">FROM</span> pg_constraint c
<span class="hl kwa">WHERE</span> conrelid<span class="hl opt">=(</span>
<span class="hl kwa">SELECT</span> attrelid <span class="hl kwa">FROM</span> pg_attribute
<span class="hl kwa">WHERE</span> attrelid <span class="hl opt">= (</span>
<span class="hl kwa">SELECT</span> oid <span class="hl kwa">FROM</span> pg_class <span class="hl kwa">WHERE</span> relname <span class="hl opt">=</span> table_rec.relname
<span class="hl opt">)</span> <span class="hl kwa">AND</span> attname<span class="hl opt">=</span><span class="hl sng">'tableoid'</span>
<span class="hl opt">)</span>
LOOP
v_table_ddl<span class="hl opt">:=</span>v_table_ddl||<span class="hl sng">','</span>||<span class="hl kwd">chr</span><span class="hl opt">(</span><span class="hl num">10</span><span class="hl opt">);</span>
v_table_ddl<span class="hl opt">:=</span>v_table_ddl||<span class="hl sng">'CONSTRAINT '</span>||constraint_rec.conname<span class="hl opt">;</span>
v_table_ddl<span class="hl opt">:=</span>v_table_ddl||<span class="hl kwd">chr</span><span class="hl opt">(</span><span class="hl num">10</span><span class="hl opt">)</span>||<span class="hl sng">' '</span>||constraint_rec.constrainddef<span class="hl opt">;</span>
firstrec <span class="hl opt">:=</span> <span class="hl kwa">FALSE</span><span class="hl opt">;</span>
<span class="hl kwa">END</span> LOOP<span class="hl opt">;</span>
v_table_ddl<span class="hl opt">:=</span>v_table_ddl||<span class="hl sng">');'</span><span class="hl opt">;</span>
<span class="hl kwa">RETURN NEXT</span> v_table_ddl<span class="hl opt">;</span>
<span class="hl kwa">END</span> LOOP<span class="hl opt">;</span>
<span class="hl kwa">END</span><span class="hl opt">;</span>
$BODY$
<span class="hl kwa">LANGUAGE</span> plpgsql <span class="hl kwa">VOLATILE</span>
<span class="hl kwa">COST</span> <span class="hl num">100</span><span class="hl opt">;</span>
<span class="hl kwa">select</span> <span class="hl kwd">generate_create_table_statement</span><span class="hl opt">(</span><span class="hl sng">'my_table'</span><span class="hl opt">)</span>
</code></pre>
<h2>Reorder columns aka tetris</h2>
<pre><code class="language-sql"><span class="hl kwa">SELECT</span> a.attname<span class="hl opt">,</span> t.typname<span class="hl opt">,</span> t.typalign<span class="hl opt">,</span> t.typlen
<span class="hl kwa">FROM</span> pg_class c
<span class="hl kwa">JOIN</span> pg_attribute a <span class="hl kwa">ON</span> <span class="hl opt">(</span>a.attrelid <span class="hl opt">=</span> c.oid<span class="hl opt">)</span>
<span class="hl kwa">JOIN</span> pg_type t <span class="hl kwa">ON</span> <span class="hl opt">(</span>t.oid <span class="hl opt">=</span> a.atttypid<span class="hl opt">)</span>
<span class="hl kwa">WHERE</span> c.relname <span class="hl opt">=</span> <span class="hl sng">'my_table'</span> <span class="hl kwa">AND</span> a.attnum <span class="hl opt">>=</span> <span class="hl num">0</span>
<span class="hl kwa">ORDER BY</span> t.typlen <span class="hl kwa">DESC</span>
</code></pre>
<p>https://medium.com/@rhlkmr089/in-house-data-lake-with-cdc-processing-hudi-docker-878cee483ca0</p>
<h2>cdc</h2>
<ul>
<li><a href="https://medium.com/@rhlkmr089/in-house-data-lake-with-cdc-processing-hudi-docker-878cee483ca0">Debezium , Kafka and hudi</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Postgresql — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2024-01-14 18:33</time> </div>https://blog.parisni.com/wiki/networkNetwork2020-05-23T20:41:00+00:00<div id="generated-toc"> </div><hr>
<p><span class="w3-tag w3-red tag">security</span></p>
<h1>Network</h1>
<h2>Test Port Open</h2>
<p>this will test if the remote has port open:</p>
<pre><code class="language-bash">nc <span class="hl kwb">-zv</span> <span class="hl opt"><</span><span class="hl kwc">host</span><span class="hl opt">> <</span>port<span class="hl opt">></span>
</code></pre>
<h2>Test Ports Open</h2>
<p>this will scan for open/closed ports on the remote :</p>
<pre><code class="language-bash">nmap <span class="hl kwb">-p</span> <span class="hl num">1</span><span class="hl kwb">-10000</span> <span class="hl opt"><</span><span class="hl kwc">host</span><span class="hl opt">></span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Network — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-05-23 20:41</time> </div>https://blog.parisni.com/wiki/ansibleAnsible2022-01-23T01:37:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">deployment</span></p>
<h1>Ansible</h1>
<h2>Encrypt</h2>
<pre><code>ansible-vault encrypt_string --name 'a_variable_name' --vault-password-file <path-to-password>
<paste>
ctrl+d
</code></pre>
<h2>Decrypt</h2>
<pre><code>ansible-vault decrypt --vault-password-file <path-to-password>
<paste>
$ANSIBLE_VAULT;1.1;AES256
65396366366339373335323261323466666239366438633535306432373061356664303733363738
3665303762346264373361393031393730393461303339660a633231663737626439643638333735
37643832376435313437336465636262646234326633353433636330346665646530393463646664
6331636234373632310a646265316539346434313735623432656564633939653263376339333531
3933
ctrl+d
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Ansible — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-01-23 01:37</time> </div>https://blog.parisni.com/wiki/davicalDavical2022-01-12T14:13:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">caldav</span></p>
<h1>Davical</h1>
<p>It is a caldav implementation based on php and postgresql.</p>
<h2>Install</h2>
<p>The <a href="https://github.com/heiderich/davical">docker image</a> works out of the box.</p>
<p>For example :</p>
<pre><code>docker run -d --name davical-test -p 8095:80 datze/davical_https
</code></pre>
<p>Then you get administration access with default password: admin/12345</p>
<p>You can use a nginx reverse proxy on top of it, with letsencrypt to get ssl.</p>
<h2>Administrative</h2>
<p>By visiting <code>http://<ip>:8095/</code> you get the admin interface.</p>
<p>Features:</p>
<ul>
<li>create users</li>
<li>create group</li>
<li>add user to group</li>
<li>create group ressources...</li>
</ul>
<h2>Clients</h2>
<p>Tested with :</p>
<ul>
<li>evolution (linux)</li>
<li>davx5 (android)</li>
</ul>
<h2>Etar / davx5</h2>
<ol>
<li>in Etar, Settings > Add CalDav calendar > Login with url and user name</li>
<li>set <code>https://your-cal-url/caldav.php/<user></code> and user/password</li>
<li>set a unique email identifier</li>
<li>this creates a davx5 entry. Return to Etar, and the calendar should appear in the list.</li>
</ol>
<h2>LDAP</h2>
<p>According to the
<a href="https://wiki.davical.org/index.php/Configuration/Authentication_Settings/LDAP">documentation</a>,
davical can manage users and group from a ldap endpoint.</p>
<p>In order to inactive a user, there is two options :</p>
<ol>
<li>delete the user in ldap</li>
<li>change its password from ldap</li>
</ol>
<p>Configure the ldap in davical.php.</p>
<p>If no TLS, imortant to comment (this is not a boolean)</p>
<pre><code class="language-php"><span class="hl kwc">$c-</span><span class="hl opt">></span>authenticate_hook<span class="hl opt">[</span><span class="hl sng">'call'</span><span class="hl opt">] =</span> <span class="hl sng">'LDAP_check'</span><span class="hl opt">;</span>
<span class="hl kwc">$c-</span><span class="hl opt">></span>authenticate_hook<span class="hl opt">[</span><span class="hl sng">'config'</span><span class="hl opt">] =</span> <span class="hl kwa">array</span><span class="hl opt">(</span>
<span class="hl sng">'host'</span> <span class="hl opt">=></span> <span class="hl sng">'openldap'</span><span class="hl opt">,</span> <span class="hl slc">//host name of your LDAP Server</span>
<span class="hl sng">'port'</span> <span class="hl opt">=></span> <span class="hl sng">'389'</span><span class="hl opt">,</span> <span class="hl slc">//port</span>
<span class="hl com">/* For the initial bind to be anonymous leave bindDN and passDN</span>
<span class="hl com"> commented out */</span>
<span class="hl slc">//DN to bind to this server enabling to perform request</span>
<span class="hl sng">'bindDN'</span><span class="hl opt">=></span> <span class="hl sng">'cn=admin,dc=example,dc=org'</span><span class="hl opt">,</span>
<span class="hl slc">//Password of the previous bindDN to bind to this server enabling to perform request</span>
<span class="hl sng">'passDN'</span><span class="hl opt">=></span> <span class="hl sng">'admin'</span><span class="hl opt">,</span>
<span class="hl sng">'protocolVersion'</span> <span class="hl opt">=></span> <span class="hl sng">'3'</span><span class="hl opt">,</span> <span class="hl slc">//Version of LDAP protocol to use</span>
<span class="hl sng">'baseDNUsers'</span><span class="hl opt">=></span> <span class="hl sng">'dc=example,dc=org'</span><span class="hl opt">,</span> <span class="hl slc">//where to look at valid user</span>
<span class="hl sng">'filterUsers'</span> <span class="hl opt">=></span> <span class="hl sng">'objectClass=inetOrgPerson'</span><span class="hl opt">,</span> <span class="hl slc">//filter which must validate a user according to RFC4515, i.e. surrounded by brackets</span>
<span class="hl sng">'baseDNGroups'</span> <span class="hl opt">=></span> <span class="hl sng">'ou=divisions,dc=tennaxia,dc=net'</span><span class="hl opt">,</span> <span class="hl slc">//where to look for groups</span>
<span class="hl sng">'filterGroups'</span> <span class="hl opt">=></span> <span class="hl sng">'objectClass=groupOfUniqueNames'</span><span class="hl opt">,</span> <span class="hl slc">//filter with same rules as filterUsers</span>
<span class="hl com">/** /!\ "username" should be set and "updated" must be set **/</span>
<span class="hl sng">'mapping_field'</span> <span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl sng">"username"</span> <span class="hl opt">=></span> <span class="hl sng">"uid"</span><span class="hl opt">,</span>
<span class="hl sng">"updated"</span> <span class="hl opt">=></span> <span class="hl sng">"modifyTimestamp"</span><span class="hl opt">,</span>
<span class="hl sng">"fullname"</span> <span class="hl opt">=></span> <span class="hl sng">"cn"</span> <span class="hl opt">,</span>
<span class="hl sng">"email"</span> <span class="hl opt">=></span><span class="hl sng">"mail"</span>
<span class="hl opt">),</span> <span class="hl slc">//used to create the user based on his ldap properties</span>
<span class="hl sng">'group_mapping_field'</span> <span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl sng">"username"</span> <span class="hl opt">=></span> <span class="hl sng">"cn"</span><span class="hl opt">,</span>
<span class="hl sng">"updated"</span> <span class="hl opt">=></span> <span class="hl sng">"modifyTimestamp"</span><span class="hl opt">,</span>
<span class="hl sng">"fullname"</span> <span class="hl opt">=></span> <span class="hl sng">"cn"</span> <span class="hl opt">,</span>
<span class="hl sng">"members"</span> <span class="hl opt">=></span><span class="hl sng">"memberUid"</span>
<span class="hl opt">),</span> <span class="hl slc">//used to create the group based on the ldap properties</span>
<span class="hl com">/** used to set default value for all users, will be overcharged by ldap if defined also in mapping_field **/</span>
<span class="hl sng">'default_value'</span> <span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl sng">"date_format_type"</span> <span class="hl opt">=></span> <span class="hl sng">"E"</span><span class="hl opt">,</span><span class="hl sng">"locale"</span> <span class="hl opt">=></span> <span class="hl sng">"fr_FR"</span><span class="hl opt">),</span>
<span class="hl com">/** foreach key set start and length in the string provided by ldap</span>
<span class="hl com"> example for openLDAP timestamp : 20070503162215Z **/</span>
<span class="hl sng">'format_updated'</span><span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl sng">'Y'</span> <span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">,</span><span class="hl num">4</span><span class="hl opt">),</span><span class="hl sng">'m'</span> <span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl num">4</span><span class="hl opt">,</span><span class="hl num">2</span><span class="hl opt">),</span><span class="hl sng">'d'</span><span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl num">6</span><span class="hl opt">,</span><span class="hl num">2</span><span class="hl opt">),</span><span class="hl sng">'H'</span> <span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl num">8</span><span class="hl opt">,</span><span class="hl num">2</span><span class="hl opt">),</span><span class="hl sng">'M'</span><span class="hl opt">=></span><span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl num">10</span><span class="hl opt">,</span><span class="hl num">2</span><span class="hl opt">),</span><span class="hl sng">'S'</span> <span class="hl opt">=></span> <span class="hl kwa">array</span><span class="hl opt">(</span><span class="hl num">12</span><span class="hl opt">,</span><span class="hl num">2</span><span class="hl opt">)),</span>
<span class="hl slc">// 'startTLS' => false, // Require that TLS is used for LDAP?</span>
<span class="hl slc">// If ldap_start_tls is not working, it is probably</span>
<span class="hl slc">// because php wants to validate the server's</span>
<span class="hl slc">// certificate. Try adding "TLS_REQCERT never" to the</span>
<span class="hl slc">// ldap configuration file that php uses (e.g. /etc/ldap.conf</span>
<span class="hl slc">// or /etc/ldap/ldap.conf). Of course, this lessens security!</span>
<span class="hl sng">'scope'</span> <span class="hl opt">=></span> <span class="hl sng">'subtree'</span><span class="hl opt">,</span> <span class="hl slc">// Search scope to use, defaults to subtree.</span>
<span class="hl slc">// Allowed values: base, onelevel, subtree.</span>
<span class="hl opt">);</span>
<span class="hl com">/* If there is some user you do not want to sync from LDAP, put their username in this list */</span>
<span class="hl kwc">$c-</span><span class="hl opt">></span>do_not_sync_from_ldap <span class="hl opt">=</span> <span class="hl kwa">array</span><span class="hl opt">(</span> <span class="hl sng">'admin'</span> <span class="hl opt">=></span> <span class="hl kwa">true</span> <span class="hl opt">);</span>
<span class="hl kwa">include</span><span class="hl opt">(</span><span class="hl sng">'drivers_ldap.php'</span><span class="hl opt">);</span>
</code></pre>
<h2>Shared collection</h2>
<p>This applies for any collection (card/calendar)</p>
<ol>
<li>create a group</li>
<li>add both users to group</li>
<li>create a collection in the group</li>
<li>grant access to that collection to users</li>
<li>bind to that collections ; /user1/boundcollection --> /group/shared-collection</li>
</ol>
<p>The binding will emulate the collection owner are users. Then Etar calendar
allows to create event for a shared calendar.</p>
<h2>Access</h2>
<p>By default, the user creation form grants anything to a Person. This leads the
latter to have full access. Don't forget to remove all of them.</p>
<h2>Tickets</h2>
<p>They are used to share a calendar with an external person/tool, avoiding to
register. This allows to integrate davical into an external tool such google calendar.</p>
<pre><code>https://<davical-domain>/public.php/<davical-user>/calendar/?ticket=<davical-ticket>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Davical — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-01-12 14:13</time> </div>https://blog.parisni.com/wiki/plikPlik2022-11-05T14:24:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">wetransfert</span>
<span class="w3-tag w3-red tag">files</span>
<span class="w3-tag w3-red tag">self-host</span></p>
<h1>Plik</h1>
<p>It has interessting features such: s3 support, server side encryption (s3
only), restricted access to users, powerfull multi-arch cli.</p>
<p>Equivalent tools:</p>
<ul>
<li><a href="https://framagit.org/fiat-tux/hat-softwares/lufi/-/blob/master/lufi.conf.template">lufi</a> <a href="https://github.com/Hamzelot/lufi-docker">docker-image</a></li>
<li><a href="https://gitlab.com/mojo42/Jirafeau">jirafeau</a></li>
<li><a href="https://github.com/dutchcoders/transfer.sh/">transfer.sh</a></li>
<li><a href="https://github.com/freedomofpress/securedrop">securedrop</a></li>
</ul>
<h2>Docker Compose</h2>
<p>Use postgres, restrict upload to created users</p>
<pre><code>MaxTTLStr = "0" # 0 : No limit
ProtectedByPassword = true # Allow users to protect the download with a password
NoWebInterface = false # Disable web user interface
Authentication = true # Enable authentication
NoAnonymousUploads = true # Prevent unauthenticated users to upload files
[MetadataBackendConfig]
Driver = "postgres"
ConnectionString = "postgres://<user>:<password>@<ip>:<port>/<db>"
</code></pre>
<h2>Server cli</h2>
<pre><code>./plikd user create --login <login> --name <username> --password <password> --email <email> --admin
</code></pre>
<h2>Server UI</h2>
<ul>
<li>generate a token in order the cli client can connect</li>
</ul>
<h2>Client</h2>
<pre><code>plik --update # init a .plikrc file
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Plik — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-11-05 14:24</time> </div>https://blog.parisni.com/wiki/sourcehutSourcehut2021-12-28T17:29:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">versionning</span></p>
<h1>Sourcehut</h1>
<h2>Start a sourcehut project</h2>
<p>Init the repository:</p>
<ol>
<li>add your rsa pub key in the profile</li>
<li>create a repository <code>test-git-mail</code></li>
<li>create a mailing list associated <code>test-git-mail</code> <a href="https://lists.sr.ht/~parisni/test-git-mail">accessible from here</a></li>
<li><code>git clone git@git.sr.ht:~parisni/test-git-mail</code></li>
<li>init the repos with something and <code>git push</code> an initial commit</li>
</ol>
<h2>Send/apply patches</h2>
<p>Send a patch :</p>
<ol>
<li>install and configure <code>git send-mail</code> <a href="https://git-send-email.io/">from the tutorial</a></li>
<li>clone the repository locally</li>
<li>configure the local sendmail <code>git config sendemail.to "~parisni/test-git-mail@lists.sr.ht"</code></li>
<li>add commit, and then <code>git send-email HEAD^</code> or <code>git send-email qefsfhjkh asdfhqe</code> for multiple commits</li>
</ol>
<p>Receive a patch :</p>
<ol>
<li><a href="https://drewdevault.com/2020/04/20/Configuring-aerc-for-git.html?hmsr=joyk.com">configure aerc</a></li>
<li>get email and apply patch with aerc keys-map</li>
<li>push commits</li>
</ol>
<p>Ask for motifications :</p>
<ol>
<li>answer to the email patch</li>
</ol>
<p>Send a modified patch :</p>
<ol>
<li>fix the errors</li>
<li><code>git commit --amend</code></li>
<li><code>git send-email --annotate -v2 HEAD^</code></li>
</ol>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Sourcehut — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-28 17:29</time> </div>https://blog.parisni.com/wiki/csvCsv2021-09-02T15:15:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">data</span></p>
<h1>Csv</h1>
<h2>Edit csv from command line</h2>
<p>Use <a href="https://github.com/andmarti1424/sc-im">sc-im</a></p>
<pre><code class="language-bash">sc<span class="hl kwb">-im</span> my.csv
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Csv — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-09-02 15:15</time> </div>https://blog.parisni.com/wiki/hbaseHbase2020-05-23T20:41:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">bigdata</span></p>
<h1>Hbase</h1>
<h2>Fixing currupted tables</h2>
<p>This will fix corrupted blocks on hbase table. Hbase has a x2 replication factor by default. Even in case the hdfs x3 replication factor has failed, hbase can regenerate.</p>
<pre><code class="language-bash"><span class="hl slc"># Diagnosis</span>
hbase hbck <span class="hl kwb">-details</span>
<span class="hl slc"># Fix</span>
hbase hbck <span class="hl kwb">-fix</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Hbase — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-05-23 20:41</time> </div>https://blog.parisni.com/wiki/icrIRC2020-09-04T16:07:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">social</span></p>
<h1>IRC</h1>
<h2>Weechat</h2>
<h3>Search for room</h3>
<pre><code>/search -re <the regex>
</code></pre>
<h3>Join a room</h3>
<pre><code>/join #foo
</code></pre>
<h3>Quit a room</h3>
<pre><code>/part
</code></pre>
<h3>Short Keys</h3>
<pre><code><ctrl+x>: change server
<ctrl+n>: move down room
<ctrl+p>: move up room
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=IRC — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-09-04 16:07</time> </div>https://blog.parisni.com/wiki/dvdDVD2022-06-19T18:53:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">media</span></p>
<h1>DVD</h1>
<h2>Image subtitle</h2>
<p>Some dvd have subtitles as an image.</p>
<p>You can burn the subs into the video. However the palette might have strange color. You can modify the palette colors as below:</p>
<pre><code>ffmpeg -probesize 100M -analyzeduration 120M -palette "ffffff,000000,ffffff,000000,ffffff,ffffff,ffffff,ffffff,ffffff,ffffff,ffffff,ffffff,ffffff,ffffff,ffffff,ffffff" -i input.mkv -filter_complex "[0:v][1:s]overlay" -acodec copy -vcodec libx264 output.mp4
</code></pre>
<p>source: https://en.m.wikibooks.org/wiki/FFMPEG_An_Intermediate_Guide/subtitle_options</p>
<h2>Best ripper</h2>
<p><a href="https://debian-facile.org/viewtopic.php?id=24453">Found here</a></p>
<pre><code>#!/bin/bash
## golgot200
## Testeur smolski
## 29/04/2020
START_TIME=$(date +%s)
PATH=$PATH:/sbin:/usr/sbin
## CONDITION : UN SEUL DVD-VIDÉO.
mounting_directory=$( mount | grep '/dev/sr' | grep -Po 'on\K.*(?=type)' | sed 's|\(.*\)/.*|\1|' | uniq )
COUNT_VIDEO_TS=$(find ${mounting_directory} -maxdepth 3 -name 'VIDEO_TS.IFO' 2> /dev/null | wc -l)
DEVICE=$(blkid /dev/sr* | awk '{print $1}' | sed -e 's/://g')
COUNT_DEVICE=$(blkid /dev/sr* | awk '{print $1}' | sed -e 's/://g' | wc -l)
if [[ "$COUNT_VIDEO_TS" -eq 0 ]]; then
echo -e "\e[3;4;32m Aucun DVD-VIDEO monté : Abandon\e\n[0m"
exit 0
elif [[ "$COUNT_VIDEO_TS" -gt 1 ]]; then
echo -e "\e[3;4;32m Il y a plus d'un DVD-VIDEO détecté : Abandon\e\n[0m"
exit 0
fi
if [[ "$COUNT_VIDEO_TS" -eq 1 ]] && [[ "$COUNT_DEVICE" -eq 1 ]]; then
echo -e "\e[3;4;32m Les conditions sont réunies : On continue\e\n[0m"
TITLE=$(lsdvd /dev/sr0 | head -1 | awk '{print $3}')
TITLE_COUNT=$(lsdvd "$DEVICE" | grep -c ^'Title: ')
fi
echo "Proceed..."
echo -e "DVD-Vidéo : \e[32m$TITLE\e[0m"
read -r -p "Ce nom convient-il ? [Y/n] " input
case $input in
[yY][eE][sS]|[yY])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[32mOui\e[0m"
echo ""
mkdir -p "$TITLE/IFOS"
mkdir -p "$TITLE/VOBSUBS"
;;
[nN][oO]|[nN])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[33mNon\e[0m"
read -r -p "Entrer un nouveau nom: " TITLE
if [[ -z "$TITLE" ]]; then
echo "Invalid input..."
echo "Quit"
exit 1
else
echo ""
mkdir -p "$TITLE/IFOS"
mkdir -p "$TITLE/VOBSUBS"
fi
;;
*)
echo "Invalid input..."
echo "Quit"
exit 1
;;
esac
default_limit=-1
echo -e "\e[3;4;32m La limite est fixée pour encoder tous les titres.\e[0m"
read -r -p "Encoder tous les titres ? [Y/n] " input
case $input in
[yY][eE][sS]|[yY])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[32mOui, on encode tous les titres.\e[0m"
echo "$default_limit" >> "$TITLE/time_limit.txt"
echo ""
;;
[nN][oO]|[nN])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[33mNon, je ne veux encoder que les titres d'une certaine durée.\e[0m"
read -r -p "Entrer une limite (ex:1800) en secondes: " time_limit
if [[ -z "$time_limit" ]]; then
echo "Invalid input..."
echo "Quit"
rm -r "$TITLE"
echo -e "Supression du dossier \e[32m$TITLE\e[0m"
exit 1
else
times=$((time_limit / 60))
echo -e "Seul les titres de \e[94m$times minutes\e[0m et plus seront pris en compte."
echo "$time_limit" >> "$TITLE/time_limit.txt"
echo ""
fi
;;
*)
echo "Invalid input..."
echo "Quit"
rm -r "$TITLE"
echo -e "Supression du dossier \e[32m$TITLE\e[0m"
exit 1
;;
esac
echo -e "\e[3;4;32m la langue française est privilégiée par défaut.\e[0m"
read -r -p "Privilégier cette option ? [Y/n] " input
case $input in
[yY][eE][sS]|[yY])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[32mOui, on garde cette option.\e\n[0m"
echo "1" > "$TITLE/choix_langues.txt"
;;
[nN][oO]|[nN])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[33mNon, on garde toutes les langues.\e[0m"
echo "0" > "$TITLE/choix_langues.txt"
;;
*)
echo "Invalid input..."
echo "Quit"
rm -r "$TITLE"
echo -e "Supression du dossier \e[32m$TITLE\e[0m"
exit 1
;;
esac
read -r -p "Conserver le bitrate d'origine ? [Y/n] " input
case $input in
[yY][eE][sS]|[yY])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[32mOui\e[0m"
for (( i=1; i<=TITLE_COUNT; i++ ))
do
BITRATE=$(mplayer -noconfig all -nocache -vo null -ao null -frames 0 -identify -dvd-device "$DEVICE" dvdnav://"$i" 2>/dev/null | \
grep "VIDEO:" | awk '{print$8}' | sed 's/\.[^ ]*/ /g')
echo "$BITRATE" >> "$TITLE/bitrate.txt"
done
;;
[nN][oO]|[nN])
echo -e "\e[3;4;32m Réponse:\n\e[0m \e[33mNon, on applique un bitrate de son choix\e[0m"
read -r -p "Appliquer un nouveau Bitrate (ex 1500): " newbitrate
echo -e "\e[3;4;33mChoix du bitrate:\n\e[0m \e[3;6;94m$newbitrate kbits/s\e[0m (pour tous les titres)"
if [[ -z "$newbitrate" ]]; then
echo -e " \e[3;6;94m Bitrate erroné\e[0m"
echo "Quit"
rm -r "$TITLE"
echo -e "Supression du dossier \e[32m$TITLE\e[0m"
exit 1
else
for (( i=1; i<=TITLE_COUNT; i++ ))
do
echo "$newbitrate" >> "$TITLE/bitrate.txt"
done
fi ;;
*)
echo "Invalid input..."
echo "Quit"
rm -r "$TITLE"
echo -e "Supression du dossier \e[32m$TITLE\e[0m"
exit 1
;;
esac
for (( i=1; i<=TITLE_COUNT; i++ ))
do
bv=$(sed -n "$i p" "$TITLE/bitrate.txt")
bitrate_k=$((bv/1))k
bufsize_k=$((bv/2))k
LENGTH=$(mplayer -noconfig all -nocache -vo null -ao null -frames 0 -identify -dvd-device "$DEVICE" dvdnav://"$i" 2>/dev/null | \
sed -E -n 's/^.*ID_LENGTH=([0-9]+).*$/\1/p')
#time_limit=$(sed q "$TITLE/time_limit.txt") ## utilité ? ##
[[ "$LENGTH" -le "$time_limit" ]] && continue
mplayer -noconfig all -nocache -dvd-device "$DEVICE" dvdnav://"$i" \
-dumpstream -dumpfile "$TITLE/$i-$TITLE.vob"
INTERLEAVED_FRAMES=$(ffmpeg -filter:v idet -frames:v 200 -an -f rawvideo \
-y /dev/null -i "$TITLE/$i-$TITLE.vob" 2>&1 | \
awk '/(Single|Multi) frame detection/ {sum += $8+$10}
END { print sum }')
if [[ "$INTERLEAVED_FRAMES" -gt "100" ]]; then
FILTERS='-filter:v yadif=0:-1:0,hue=b=0.0:s=0.7,hqdn3d=1.5:1.5:4:4,unsharp=luma_msize_x=5:luma_msize_y=5:luma_amount=0.5,noise=c0s=2:c1s=2:c2s=2:c0f=a+t'
else
FILTERS='-filter:v hue=b=0.0:s=0.7,hqdn3d=1.5:1.5:4:4,unsharp=luma_msize_x=5:luma_msize_y=5:luma_amount=0.5,noise=c0s=2:c1s=2:c2s=2:c0f=a+t'
fi
COUNT_AUDIO=$(ffprobe "$TITLE/$i-$TITLE.vob" 2>&1 | grep -c "Audio:")
FPS=$(mplayer -noconfig all -nocache -vo null -ao null -frames 0 -identify "$TITLE/$i-$TITLE.vob" 2>/dev/null | grep "VIDEO:" | awk '{print$6}')
RATIO=$(lsdvd -x "$DEVICE" -t "$i" | grep -m1 "Aspect ratio:" | awk '{print$11}' | sed -e 's/\//\:/g' | tr -d ',')
O_CHANNELS=$(ffprobe "$TITLE/$i-$TITLE.vob" 2>&1 | grep -c '(Audio: ac3, 0 channels)')
REMOVE_O_CHANNELS=$(ffprobe "$TITLE/$i-$TITLE.vob" 2>&1 | awk '/Stream/ && /Audio:/ && /kb\/s|\(LC\),/ { print $2 }' | \
cut -d "[" -f2 | cut -d "]" -f1 | awk '{ printf " -map i:"$0" " }')
ALLSHOW_TAG=$(mplayer -vo null -ao null -frames 0 -identify dvdnav://"$i" -dvd-device "$DEVICE" 2>/dev/null | \
awk '/audio stream:/ {printf " -metadata:s:a:"$3" language="$8" -metadata:s:a:"$3" title="$5$6""}')
MAP_ID_STREAMS=$(lsdvd -a "$DEVICE" -t "$i" 2> /dev/null | grep "Language:" | sed -e 's/c/1c/g' | awk '{ printf " -map i:"$21"" }')
MAP_ID_STREAMS_LPCM=$(lsdvd -a "$DEVICE" -t "$i" 2> /dev/null | grep "lpcm" | awk '{ printf " -map i:"$22"" }')
COUNT_LPCM=$(lsdvd -a "$DEVICE" -t "$i" |grep -c 'lpcm')
AIDS_LPCM=$(lsdvd -a "$DEVICE" -t "$i" | awk '/lpcm/ { print $22 }')
## VOB PRÉSENT.
## AUCUN AUDIO.
if [[ -e "$TITLE/$i-$TITLE.vob" ]] && [[ "$COUNT_AUDIO" -eq "0" ]]; then
echo -e "Le bitrate du titre n°\e[94m$i\e[0m sera donc de : \e[32m$bitrate_k\e[0m"
sleep 2
ffmpeg -hide_banner -analyzeduration 1000M -probesize 1000M -vsync 2 -r "$FPS" -i "$TITLE/$i-$TITLE.vob" \
-map 0:v -metadata title="$TITLE" \
-c:v libx264 -b:v "$bitrate_k" -maxrate "$bitrate_k" -bufsize "$bufsize_k" \
-loglevel repeat+verbose -movflags faststart \
-x264opts 'keyint=300:min-keyint=25:8x8dct:sliced-threads=0' \
-an -movflags +faststart -max_muxing_queue_size 9999 \
-deblock 1:1 -flags +loop -qcomp 0.60 -qblur 0.5 -coder 1 -me_range 6 -sc_threshold 42 -bf 10 -trellis 2 -mbtree 1 -qmin 0 -qmax 52 \
$FILTERS \
-preset veryfast -y "$TITLE/$i-$TITLE.mkv"
rm -f "$TITLE/$i-$TITLE.vob"
fi
## VOB PRÉSENT.
## AU MOINS UN AUDIO OU PLUS.
## SANS 0 CHANNEL.
## SANS LPCM.
if [[ -e "$TITLE/$i-$TITLE.vob" ]] && [[ "$COUNT_AUDIO" -ge "1" ]] && [[ "$O_CHANNELS" -eq "0" ]] && [[ "$COUNT_LPCM" -eq "0" ]]; then
echo -e "Le bitrate du titre n°\e[94m$i\e[0m sera donc de : \e[32m$bitrate_k\e[0m"
sleep 2
ffmpeg -hide_banner -analyzeduration 1000M -probesize 1000M -vsync 2 -r "$FPS" -i "$TITLE/$i-$TITLE.vob" \
-map 0:v -metadata title="$TITLE" \
-c:v libx264 -b:v "$bitrate_k" -maxrate "$bitrate_k" -bufsize "$bufsize_k" \
-loglevel repeat+verbose -movflags faststart \
-x264opts 'keyint=300:min-keyint=25:8x8dct:sliced-threads=0' \
${MAP_ID_STREAMS} -c:a copy ${ALLSHOW_TAG} -movflags +faststart -max_muxing_queue_size 9999 \
-deblock 1:1 -flags +loop -qcomp 0.60 -qblur 0.5 -coder 1 -me_range 6 -sc_threshold 42 -bf 10 -trellis 2 -mbtree 1 -qmin 0 -qmax 52 \
$FILTERS \
-preset veryfast -y "$TITLE/$i-$TITLE.mkv"
# ffmpeg -hwaccel auto -c:v mpeg2_cuvid -analyzeduration 1000M -probesize 1000M -i "$TITLE/$i-$TITLE.vob" \
# -map 0:v -metadata title="$TITLE" \
# -c:v h264_nvenc -preset medium -b:v $BITRATE_K -bufsize $BUFSIZE_K -profile:v high -bf 3 -b_ref_mode 0 -temporal-aq 1 -rc-lookahead 20 -vsync 0 \
# -loglevel repeat+verbose \
# -filter:v hue=b=0.0:s=0.7,hqdn3d=1.5:1.5:4:4,unsharp=luma_msize_x=5:luma_msize_y=5:luma_amount=0.5,noise=c0s=2:c1s=2:c2s=2:c0f=a+t \
# ${MAP_ID_STREAMS} -c:a copy $ALLSHOW_TAG -movflags +faststart \
# -y "$TITLE/$i-$TITLE.mkv"
rm -f "$TITLE/$i-$TITLE.vob"
fi
## VOB PRÉSENT.
## AU MOINS UN AUDIO OU PLUS.
## UN 0 CHANNEL OU PLUS.
## SANS LPCM.
if [[ -e "$TITLE/$i-$TITLE.vob" ]] && [[ "$COUNT_AUDIO" -ge "1" ]] && [[ "$O_CHANNELS" -ge "1" ]] && [[ "$COUNT_LPCM" -eq "0" ]]; then
echo -e "Le bitrate du titre n°\e[94m$i\e[0m sera donc de : \e[32m$bitrate_k\e[0m"
sleep 2
ffmpeg -hide_banner -analyzeduration 1000M -probesize 1000M -vsync 2 -r "$FPS" -i "$TITLE/$i-$TITLE.vob" \
-map 0:v -metadata title="$TITLE" \
-c:v libx264 -b:v "$bitrate_k" -maxrate "$bitrate_k" -bufsize "$bufsize_k" \
-loglevel repeat+verbose -movflags faststart \
-x264opts 'keyint=300:min-keyint=25:8x8dct:sliced-threads=0' \
${REMOVE_O_CHANNELS} -c:a copy ${ALLSHOW_TAG} -movflags +faststart -max_muxing_queue_size 9999 \
-deblock 1:1 -flags +loop -qcomp 0.60 -qblur 0.5 -coder 1 -me_range 6 -sc_threshold 42 -bf 10 -trellis 2 -mbtree 1 -qmin 0 -qmax 52 \
$FILTERS \
-preset veryfast -y "$TITLE/$i-$TITLE.mkv"
rm -f "$TITLE/$i-$TITLE.vob"
fi
## VOB PRÉSENT.
## AVEC OU SANS AUDIO(S).
## SANS 0 CHANNEL.
## AU MOINS UN LPCM OU PLUS.
if [[ -e "$TITLE/$i-$TITLE.vob" ]] && [[ "$COUNT_AUDIO" -ge "0" ]] && [[ "$O_CHANNELS" -eq "0" ]] && [[ "$COUNT_LPCM" -ge "1" ]]; then
echo -e "Le bitrate du titre n°\e[94m$i\e[0m sera donc de : \e[32m$bitrate_k\e[0m"
sleep 2
ffmpeg -hide_banner -analyzeduration 1000M -probesize 1000M -vsync 2 -r "$FPS" -i "$TITLE/$i-$TITLE.vob" \
-map 0:v -metadata title="$TITLE" \
-c:v libx264 -b:v "$bitrate_k" -maxrate "$bitrate_k" -bufsize "$bufsize_k" \
-loglevel repeat+verbose -movflags faststart \
-x264opts 'keyint=300:min-keyint=25:8x8dct:sliced-threads=0' \
${MAP_ID_STREAMS_LPCM} -acodec pcm_s16be ${ALLSHOW_TAG} -movflags +faststart -max_muxing_queue_size 9999 \
-deblock 1:1 -flags +loop -qcomp 0.60 -qblur 0.5 -coder 1 -me_range 6 -sc_threshold 42 -bf 10 -trellis 2 -mbtree 1 -qmin 0 -qmax 52 \
$FILTERS \
-preset veryfast -y "$TITLE/$i-$TITLE.mkv"
rm -f "$TITLE/$i-$TITLE.vob"
mkdir -p "$TITLE/WAVE"
for n in $AIDS_LPCM; do
mplayer -noconfig all -nocache -benchmark -vc null -vo null \
-ao pcm:fast:file="$TITLE/WAVE/[$i-($n)]-$TITLE.wav" \
-identify -dvd-device "$DEVICE" dvdnav://"$i" -ni -aid "$n"
done
fi
choix_langues=$(sed q "$TITLE/choix_langues.txt")
COUNT_AUDIO_FR=$(mplayer -noconfig all -nocache -vo null -ao null -frames 0 -identify "$TITLE/$i-$TITLE.mkv" 2>/dev/null |awk '/-alang fr,/ { print $9 }' | wc -w)
## AU MOINS UN AUDIO FRANÇAIS OU PLUS.
if [[ -e "$TITLE/$i-$TITLE.mkv" ]] && [[ "$COUNT_AUDIO_FR" -ge "1" ]] && [[ "$choix_langues" -eq "1" ]] ; then
mkvmerge -o "$TITLE/$i-$TITLE-[Language: Français].mkv" -a fre "$TITLE/$i-$TITLE.mkv"
rm -f "$TITLE/$i-$TITLE.mkv"
mv "$TITLE/$i-$TITLE-[Language: Français].mkv" "$TITLE/$i-$TITLE.mkv"
fi
## ON INDIQUE LA LANGUE DE L'AUDIO DANS LE RENOMAGE DU TITRE ET LE NOMBRE.
INFO=$(mplayer -vo null -ao null -frames 0 -identify "$TITLE/$i-$TITLE.mkv" 2>/dev/null | \
awk '/alang/ { x = x $9 ""}
END { sub(/,*$/, "", x); print x }')
COUNT_INFO=$(mplayer -vo null -ao null -frames 0 -identify "$TITLE/$i-$TITLE.mkv" 2>/dev/null | grep -c "alang")
COUNT=$(mplayer -vo null -ao null -frames 0 -identify "$TITLE/$i-$TITLE.mkv" 2>/dev/null | grep -c "aid")
## UNE OU PLUSIEURS INFOS SUR L'AUDIO ET UN OU PLUSIEURS AUDIOS.
## INTÉGRATION DES CHAPITRES.
if [[ -e "$TITLE/$i-$TITLE.mkv" ]] && [[ "$COUNT_INFO" -ge "1" ]] && [[ "$COUNT" -ge "1" ]] ; then
mv "$TITLE/$i-$TITLE.mkv" "$TITLE/$i-$TITLE-[Audio: $INFO].mkv"
dvdxchap -t "$i" "$DEVICE" > "$TITLE/$i-$TITLE-Chapters.txt"
mkvmerge "$TITLE/$i-$TITLE-[Audio: $INFO].mkv" --chapters "$TITLE/$i-$TITLE-Chapters.txt" -o "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters].mkv"
rm -f "$TITLE/$i-$TITLE-[Audio: $INFO].mkv"
rm -f "$TITLE/$i-$TITLE-Chapters.txt"
mediainfo "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters].mkv" >> "$TITLE/IFOS/$i-$TITLE-[Audio: $INFO + Chapters].log"
fi
SIDFR=$(lsdvd -s "$DEVICE" -t "$i" 2> /dev/null |awk '/Subtitle:/ && /Language: fr/ { sub(/,$/ ,"", $2);
if ($2 ~ /^[0-9]+$/) print (--$2) }')
COUNT_SID_FR=$(echo "$SIDFR" |wc -w)
## AU MOINS UN SUB FRANÇAIS OU PLUS (ON EXTRAIT TOUT ÇA).
if [[ "$COUNT_SID_FR" -ge "1" ]]; then
mkdir -p "$TITLE/VOBSUBS/$i-$TITLE"
for n in $SIDFR; do mencoder -dvd-device "$DEVICE" dvdnav://"$i" -nosound -ovc frameno -force-avi-aspect "$RATIO" -o /dev/null \
-ifo "/run/media/$USER/$TITLE/VIDEO_TS/VTS_01_0.IFO" -sid "$n" -vobsubout "$TITLE/$n-vobsubs-fr" -vobsuboutindex "$n"
## ON CORRIGE LES ID VIDES SI BESOIN.
sed -i 's/id:.*,/id: fr,/g' "$TITLE/$n-vobsubs-fr.idx"
taille_idx=$(stat -c%s "$TITLE/$n-vobsubs-fr.idx")
taille_sub=$(stat -c%s "$TITLE/$n-vobsubs-fr.sub")
if [[ "$taille_idx" -gt 1000 ]] && [[ "$taille_sub" -gt 0 ]] ; then
echo -e "\e[32m$n-vobsubs-fr.idx et $n-vobsubs-fr.sub sont bons.\e[0m"
else
## ON ISOLE LES VOBSUBS INVALIDES DANS LE DOSSIER VOBSUBS
echo -e "\e[32m$n-vobsubs-fr.idx ou $n-vobsubs-fr.sub incorrect ... déplacement des deux.\e[0m"
mv "$TITLE/$n-vobsubs-fr.idx" "$TITLE/$n-vobsubs-fr.sub" "$TITLE/VOBSUBS/$i-$TITLE"
fi
done
fi
## ON COMPTE CE QU'IL RESTE DE BON EN IDX/SUB.
COUNT_IDX_FR=$(find "$TITLE" -maxdepth 1 -name '*.idx' | wc -l)
COUNT_SUB_FR=$(find "$TITLE" -maxdepth 1 -name '*.sub' | wc -l)
## CONDITION : UN IDX/SUB OU PLUS.
if [[ -e "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters].mkv" ]] && [ "$COUNT_IDX_FR" -ge "1" ] && [ "$COUNT_SUB_FR" -ge "1" ]
then
mkvmerge "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters].mkv" --language "0:fre" "$TITLE"/*.idx -o "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters + Sub].mkv"
## ON DÉPLACE LE RESTANT DES VOBSUBS UNE FOIS MIXÉS AVEC LES INVALIDES.
mv "$TITLE"/*.idx "$TITLE"/*.sub "$TITLE/VOBSUBS/$i-$TITLE"
rm -f "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters].mkv"
INFO_SUB=$(mplayer -vo null -ao null -frames 0 -identify "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters + Sub].mkv" 2>/dev/null | \
awk '/slang/ && /fre/ { x = x $9 ","}
END { sub(/,*$/, "", x); print x }')
mv "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters + Sub].mkv" "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters + Sub: $INFO_SUB].mkv"
mediainfo "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters + Sub: $INFO_SUB].mkv" >> "$TITLE/IFOS/$i-$TITLE-[Audio: $INFO + Chapters + Sub: $INFO_SUB].log"
rm -f "$TITLE/IFOS/$i-$TITLE-[Audio: $INFO + Chapters].log"
fi
## AUCUN IDX ET SUB. ON TAG EN NOSUB
if [[ -e "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters].mkv" ]] && [ "$COUNT_IDX_FR" -eq "0" ] && [ "$COUNT_SUB_FR" -eq "0" ]
then
mv "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters].mkv" "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters NoSub].mkv"
mediainfo "$TITLE/$i-$TITLE-[Audio: $INFO + Chapters NoSub].mkv" >> "$TITLE/IFOS/$i-$TITLE-[Audio: $INFO + Chapters NoSub].log"
rm -f "$TITLE/IFOS/$i-$TITLE-[Audio: $INFO + Chapters].log"
fi
## PAS D'INFO ET UN SEUL AUDIO. ON TAG EN UND
if [[ -e "$TITLE/$i-$TITLE.mkv" ]] && [[ "$COUNT_INFO" -eq "0" ]] && [[ "$COUNT" -eq "1" ]] ; then
mv "$TITLE/$i-$TITLE.mkv" "$TITLE/$i-$TITLE-[Audio: Und].mkv"
mediainfo "$TITLE/$i-$TITLE-[Audio: Und].mkv" >> "$TITLE/IFOS/$i-$TITLE-[Audio: Und].log"
fi
## PAS D'AUDIO DONC PAS D'INFO. ON TAG EN NOSOUND
if [[ -e "$TITLE/$i-$TITLE.mkv" ]] && [[ "$COUNT_INFO" -eq "0" ]] && [[ "$COUNT" -eq "0" ]] ; then
mv "$TITLE/$i-$TITLE.mkv" "$TITLE/$i-$TITLE-[Audio: NoSound].mkv"
mediainfo "$TITLE/$i-$TITLE-[Audio: NoSound].mkv" >> "$TITLE/IFOS/$i-$TITLE-[Audio: NoSound].log"
fi
done
rm -f "$TITLE/bitrate.txt"
rm -f "$TITLE/time_limit.txt"
rm -f "$TITLE/choix_langues.txt"
echo -e "\e[32mJob terminé.\e[0m"
echo -e "\e[32mTemps total écoulé:\e[0m \e[3;6;94m$(date -ud "@$(($(date +%s) - START_TIME))" +%T)\e[0m (HH:MM:SS)"
</code></pre>
<h2>Ripper</h2>
<p>With <code>mencoder</code> (see <a href="https://martin.ankerl.com/2008/12/25/ripping-multilanguage-dvds-with-subtitles-using-only-mencoder/">more details here</a>):</p>
<p>As root:</p>
<pre><code>
$ ls -l /dev/{cd,dvd,scd,sr}*
lrwxrwxrwx 1 root root 3 20 juil. 20:31 /dev/cdrom -> sr0
brw-rw----+ 1 root optical 11, 0 20 juil. 20:31 /dev/sr0
$ sudo ln -s /dev/sr0 /dev/dvd
$ mencoder dvd://1 -ovc lavc -lavcopts vcodec=mpeg4:vhq:vbitrate="1200" -vf scale -zoom -xy 640 -oac mp3lame -lameopts br=128 -o /<path to avi file>.avi
</code></pre>
<h2>Transform VIDEO_TS into iso</h2>
<pre><code class="language-bash">HandBrakeCLI <span class="hl kwb">-i</span> VIDEO_TS <span class="hl kwb">--main-feature -o</span> <span class="hl opt">/</span>tmp<span class="hl opt">/</span>mynewvid.mkv
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=DVD — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-06-19 18:53</time> </div>https://blog.parisni.com/wiki/emacsEmacs2021-08-18T00:37:00+00:00<div id="generated-toc"> </div><hr> <div id="refman"> <div id="refman-sidebar"> <div id="generated-toc">
</div>
</div> <div id="refman-main">
<p><span class="w3-tag w3-red tag">editor</span></p>
<h1>Emacs</h1>
<h2>Comparaison de deux fichier</h2>
<p>Une manière simple de comparer deux fichiers est la commande
<code>compare-windows</code>. Il faut avoir ouvert deux fenetres avec chacune son
fichier. Par la suite, on peut accéder aux différences en re-lancant
la commande <code>compare-windows</code> à plusieurs reprises, jusqu´à ce que le
processus soit satisfaisant --- ou terminé.</p>
<h2>Gestion git avec Magit</h2>
<pre><code class="language-bash">c c <span class="hl opt">:</span> commit
P p <span class="hl opt">:</span> push sur la meme branche
F p <span class="hl opt">:</span> pull sur la meme branche
l <span class="hl kwb">--</span> <span class="hl opt"><</span>fichier<span class="hl opt">></span> l <span class="hl opt">:</span> afficher tous les commit d<span class="hl sng">'un même fichier</span>
<span class="hl sng">ll ri: rebase interactif</span>
</code></pre>
<h2>Completion des chemins</h2>
<p>helm offre une completion de <code>path</code>. A condition de précéder par un
guillemet. par la suite, la navigation se réalise via les touches
<code>readlines</code>, y compris <code>C-j</code></p>
<h2>Utiliser tramp pour editer des fichiers distants</h2>
<p>Editer des fichiers distants via emacs est interessant:</p>
<ul>
<li>utiliser ses configuration</li>
<li>bénéficier de helm pour la navigation d´arborescence</li>
<li>emacs n´est pas installé par défaut sur des serveurs distants</li>
</ul>
<pre><code class="language-lisp"><span class="hl opt"><</span>C-x<span class="hl opt">> <</span>C-f<span class="hl opt">></span> <span class="hl slc">;;puis</span>
<span class="hl opt">/</span>ssh<span class="hl opt">:<</span>user<span class="hl opt">@</span>host<span class="hl opt">>:<</span>remote-path<span class="hl opt">></span>
</code></pre>
<p>On peut aussi rebondir sur plusieurs remotes:</p>
<pre><code class="language-lisp"><span class="hl opt">/</span>ssh<span class="hl kwc">:user</span><span class="hl opt">@</span>remotehost|ssh<span class="hl kwc">:user2</span><span class="hl opt">@</span>remotehost2<span class="hl opt">:/</span>path
</code></pre>
<h2>Éditer le contenu des dossiers avec Dired</h2>
<p>Lancer le mode dired dans un dossier:</p>
<p>Les principales commandes sont <a href="https://stackoverflow.com/questions/5220344/postgresql-invalid-page-header-in-block">listées ici</a>.</p>
<pre><code class="language-lisp"><span class="hl opt"><</span>C-x d<span class="hl opt">>,</span> then choose the folder
<span class="hl opt"><</span>C-x C-q<span class="hl opt">></span> or i<span class="hl opt">,</span> turn editing mode on
</code></pre>
<p>Then you can :</p>
<ul>
<li>delete a file with <code>D</code></li>
<li>batch rename files <code>:s/foo/bar</code></li>
</ul>
<p><a href="https://stackoverflow.com/a/10661439">Apparently, "rsync" method is more robust than ssh</a></p>
<p>The bellow code allows to open multiple files:</p>
<pre><code class="language-lisp"><span class="hl opt">(</span><span class="hl kwa">eval-after-load</span> <span class="hl sng">"dired"</span>
<span class="hl opt">'(</span><span class="hl kwa">progn</span>
<span class="hl opt">(</span><span class="hl kwa">define-key</span> dired-mode-map <span class="hl sng">"F"</span> <span class="hl opt">'</span>my-dired-find-file<span class="hl opt">)</span>
<span class="hl opt">(</span><span class="hl kwa">defun</span> <span class="hl kwb">my-dired-find-file</span> <span class="hl opt">(&</span>optional arg<span class="hl opt">)</span>
<span class="hl sng">"Open each of the marked files, or the file under the point, or when prefix arg, the next N files "</span>
<span class="hl opt">(</span><span class="hl kwa">interactive</span> <span class="hl sng">"P"</span><span class="hl opt">)</span>
<span class="hl opt">(</span><span class="hl kwa">let</span><span class="hl opt">* ((</span><span class="hl kwa">fn-list</span> <span class="hl opt">(</span><span class="hl kwa">dired-get-marked-files</span> nil arg<span class="hl opt">)))</span>
<span class="hl opt">(</span><span class="hl kwa">mapc</span> <span class="hl opt">'</span>find-file fn-list<span class="hl opt">)))))</span>
</code></pre>
<h2>Gestion des layout windows</h2>
<p>La sauvegarde / restauration est perdue si la session emacs et terminée.</p>
<h3>Sauvegarder</h3>
<pre><code class="language-lisp"><span class="hl opt"><</span>C-x r w<span class="hl opt">> <</span>la lettre du layout<span class="hl opt">></span>
</code></pre>
<h3>Restaurer</h3>
<pre><code class="language-lisp"><span class="hl opt"><</span>C-x r j<span class="hl opt">> <</span>la lettre du layout<span class="hl opt">></span>
</code></pre>
<h2>Terminaux</h2>
<p><code>vterm</code> s´impose comme le terminal emacs le plus complet et robuste.</p>
<h2>Remote color</h2>
<p>Pour avoir des couleurs 24bit sur un remote terminal on peut appliquer
<a href="https://stackoverflow.com/a/50577683">cette méthode</a></p>
<h2>Packages</h2>
<h3>window-numbering</h3>
<p>This package allows to identify window with number and jump to any
with <code><M-[1-9]></code>. This is far better than the default <code><C-x o></code></p>
<h2>Spelling</h2>
<ul>
<li><code>M-x flyspell-mode</code>: souligne les mots du document</li>
<li><code>M-x ispell-change-dictionnary</code>: propose de choisir un autre dictionnaires</li>
</ul>
<pre><code class="language-bash"><span class="hl slc"># installation sous archlinux</span>
<span class="hl slc"># les paquets</span>
aspell<span class="hl kwb">-fr</span> aspell<span class="hl kwb">-en</span>
</code></pre>
<h2>Python</h2>
<p>A complete overview <a href="https://realpython.com/emacs-the-best-python-editor/">can be found here</a></p>
<h3>elpy</h3>
<p>Allows to run portion of code inside a python repl. Interesting command:</p>
<ul>
<li><code>elpy-test-pytest-runner</code>: run pytest on cursor or every tests</li>
<li><code>elpy-autopep8-fix-code</code>: fix code formatting accoring to <code>tox.ini</code> file an pep8 standard</li>
<li><code>jedi:goto-definition</code>: show the source of the python object under the cursor</li>
</ul>
<h3>Pyvenv</h3>
<p>On peut charger un environnement <strong>venv</strong> ou <strong>conda</strong> via <code>M^X pyvenv-activate</code> et spécifier:</p>
<ul>
<li>le chemin du virtualenv</li>
<li><code>anaconda3/envs/<non venv></code></li>
</ul>
<pre><code class="language-bash">M<span class="hl kwb">-x</span> pyvenv<span class="hl kwb">-activate</span>
<span class="hl slc"># then choose the venv root dir</span>
</code></pre>
<h3>jedi</h3>
<pre><code class="language-bash">jedi<span class="hl opt">:</span>setup
</code></pre>
<h3>realgud</h3>
<p><a href="https://github.com/realgud/realgud/">realgud</a> allows to debug python
and also place breakpoints thanks to the python <code>pdb</code> module</p>
<ul>
<li><code>F9</code>: add breakpoint</li>
<li><code>F10</code> : go to next</li>
<li><code>D</code>: remove all breakpoints</li>
</ul>
<h2>Treat underscore as word</h2>
<p>This will allow to treat underscore or dash, depending on the
programming context as words - only in case of pressing <code>#</code> or <code>*</code></p>
<pre><code class="language-lisp"><span class="hl opt">(</span><span class="hl kwa">with-eval-after-load</span> <span class="hl opt">'</span>evil
<span class="hl opt">(</span><span class="hl kwa">setq-default</span> evil-symbol-word-search t<span class="hl opt">))</span>
</code></pre>
<h2>Usage of frame</h2>
<p>Multiple frames can be created. Navigation between them is <code>C-x 5 o</code>. Each frame can have its own layout. That's great, for example to
have magit in an other frame.</p>
</div></div><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Emacs — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-18 00:37</time> </div>https://blog.parisni.com/wiki/mpdMpd2020-03-08T02:08:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">music</span></p>
<h1>Mpd</h1>
<h2>Install</h2>
<pre><code class="language-bash"><span class="hl slc"># the serveur</span>
pacman.<span class="hl kwc">install</span> mpd
<span class="hl slc">## the command line client</span>
pacman.<span class="hl kwc">install</span> mpc
</code></pre>
<p>Edit the <code>/etc/mpd.conf</code>, in particular add the musique folder.</p>
<pre><code class="language-bash"><span class="hl slc"># scan the database</span>
mpc update
</code></pre>
<h2>Use with command line</h2>
<pre><code class="language-bash">mpc search title <span class="hl opt"><</span>my<span class="hl kwb">-search</span><span class="hl opt">></span>
</code></pre>
<h2>Use with emacs</h2>
<pre><code class="language-bash">todo
</code></pre>
<h2>Audio Configuration</h2>
<p>https://mpd.fandom.com/wiki/PulseAudio
https://www.musicpd.org/doc/html/user.html#audio-output-format
https://opensource.com/article/17/1/linux-plays-sound</p>
<pre><code class="language-bash">voir dans nixos
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mpd — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-03-08 02:08</time> </div>https://blog.parisni.com/wiki/intellijIntelliJ2023-11-14T15:31:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">ide</span>
<span class="w3-tag w3-red tag">editor</span></p>
<h1>IntelliJ</h1>
<h2>Git Integration</h2>
<h3>Git reviews</h3>
<p>The github/gitlab reviews are painful / heterogeneous. The tool allows to get
fetch the PR and review the code easily.</p>
<h2>Vim Integration</h2>
<p><code>Vimidea</code> let you use vim within intellij. To my experience, it si incredibly
well integrated. Some features are also better integrated. (for examble,
readline shortcuts within the comand bar)</p>
<h2>Reformat SQL on selected rows</h2>
<pre><code>:!pg_format
</code></pre>
<h3>Vim plugins</h3>
<p>You can use <a href="https://github.com/JetBrains/ideavim/wiki/Emulated-plugins#highlightedyank">several vim plugin</a></p>
<p>~/.ideavimrc</p>
<pre><code>Plug 'machakann/vim-highlightedyank'
set highlightedyank
Plug 'tpope/vim-surround'
set surround
set hlsearch
</code></pre>
<h2>Maven integration</h2>
<ul>
<li>Refresh a project: <code>right click pom.xml > Maven > Generate sources and update folders</code></li>
</ul>
<h2>Tmux integration</h2>
<p>The below configuration allows to:</p>
<ol>
<li>use tmux as a terminal in idea</li>
<li>share copy/paste between tmux and idea</li>
<li>restore tmux session in case of idea exit/crash</li>
</ol>
<ul>
<li>create a custom bash script <code>/home/parisni/bin/tmux-idea</code>:
<pre><code class="language-shell">#!/usr/bin/env bash
# we assume a idea project is associated w/ a unique path
# replace dot w/ underscore since that's a special char for tmux
session_name="${PWD//./_}"
# resume session if exists
tmux attach -t "$session_name" || tmux new-session -s "$session_name"
</code></pre>
</li>
<li>In idea: Tool>Terminal>Shell> <code>/home/parisni/bin/tmux-idea</code></li>
<li>In tmux:
<pre><code class="language-shell"># in order to use xclip as a proxy for copy
# then idea works w/ tmux
bind -T copy-mode-vi y send-keys -X copy-pipe-and-cancel 'xclip -in -selection clipboard'
# Use v to trigger selection
bind-key -T copy-mode-vi v send-keys -X begin-selection
</code></pre>
</li>
</ul>
<h2>Python integration</h2>
<ol>
<li>install the python plugin</li>
<li>in Project Structure > Facets, Add facet and choose Python</li>
<li>in Modules choose the python module path, and in dependency, Add Jars or Directory</li>
<li>choose the source root</li>
<li>now come back to the Project file listing, and Mark Directory as source root again</li>
</ol>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=IntelliJ — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-11-14 15:31</time> </div>https://blog.parisni.com/wiki/imgImage2020-06-13T19:20:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">image</span></p>
<h1>Image</h1>
<h2>EXIF metadata</h2>
<h3>Manipulation</h3>
<p>Exiftool allows to play with image metadata:</p>
<ul>
<li>show: <code>exiv2 <the image.jpeg></code></li>
<li>delete: <code>exiv2 rm <the image.jpeg></code></li>
</ul>
<h3>Installation:</h3>
<ul>
<li>Ubuntu : <code>sudo apt-get install libimage-exiftool-perl</code></li>
<li>Archlinux:<code>sudo pacman -S perl-image-exiftool</code></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Image — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-13 19:20</time> </div>https://blog.parisni.com/wiki/solrSolr2020-06-27T17:30:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">database</span></p>
<h1>Solr</h1>
<h2>Gestion des dates</h2>
<p>Dans solr, les dates sont stoquées en UTC. Cela a des implications complexes pour le client:</p>
<ul>
<li>transformer les dates extraites de solrj dans la timezone du client</li>
<li>lorsque des filtres temporels sont appliqués il faut normaliser les dates en UTC</li>
</ul>
<h2>Gestion des node</h2>
<p>Pour changer de configuration (nombre de shard, port...) une instance, je n'ai pas d'autre option que de réinstaller solr.</p>
<h2>inner joins</h2>
<pre><code>fq={!join from=patient to=id fromIndex=list score=none v='type:400000'} AND name:bob
</code></pre>
<p>In order to join a sharded collection on solr-cloud, the joined
collection must be replicated on every nodes.</p>
<p>In terms of performances, adding <code>score=none</code> allow to make the search
search proportional to the joined number of rows. Without this, it
will be stable and only depend on the collection size.</p>
<p>Interestingly, if the join is used in <code>fq</code> then it is cached by
solr. Then any new query on the collection with the same inner join
will be faster.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Solr — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-27 17:30</time> </div>https://blog.parisni.com/wiki/sympaSimpa2020-03-08T02:08:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">email</span></p>
<h1>Simpa</h1>
<h2>Installation</h2>
<p>Plusieurs dockerfiles existent, mais l´installation semble
obligatoirement interactive, ce qui rend les choses plus compliquées.
Les depots yaourt fournissent un paquet <code>sympa</code>.</p>
<h3>Base de données</h3>
<p>Il faut penser à installer le package pacman <code>perl-dbd-pg</code> pour que la
connection à postgres se réalise.</p>
<h3>Sendmail</h3>
<p>Pour avoir un sendmail fonctionnel, il faut penser à démarrer le
service. Il est possible de tester son fonctionnement:</p>
<pre><code class="language-bash"><span class="hl kwb">echo</span> <span class="hl kwd">$HOSTNAME</span> <span class="hl opt">| /</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>sendmail <span class="hl kwb">-v</span> natus@riseup.net
natus@riseup.net... Connecting to <span class="hl opt">[</span><span class="hl num">127.0.0.1</span><span class="hl opt">]</span> via relay...
<span class="hl num">220</span> nc<span class="hl kwb">-ass-vip</span>.sdv.fr ESMTP Sendmail <span class="hl num">8.15.2</span><span class="hl opt">/</span><span class="hl num">8.15.2</span><span class="hl opt">;</span> Sat<span class="hl opt">,</span> <span class="hl num">15</span> Feb <span class="hl num">2020 19</span><span class="hl opt">:</span><span class="hl num">28</span><span class="hl opt">:</span><span class="hl num">10</span> <span class="hl opt">+</span><span class="hl num">0100</span>
<span class="hl opt">>>></span> EHLO nc<span class="hl kwb">-ass-vip</span>.sdv.fr
<span class="hl num">250</span><span class="hl kwb">-nc-ass-vip</span>.sdv.fr Hello localhost <span class="hl opt">[</span><span class="hl num">127.0.0.1</span><span class="hl opt">],</span> pleased to meet you
<span class="hl num">250</span><span class="hl kwb">-ENHANCEDSTATUSCODES</span>
<span class="hl num">250</span><span class="hl kwb">-PIPELINING</span>
<span class="hl num">250</span><span class="hl kwb">-EXPN</span>
<span class="hl num">250</span><span class="hl kwb">-VERB</span>
<span class="hl num">250</span><span class="hl kwb">-8BITMIME</span>
<span class="hl num">250</span><span class="hl kwb">-SIZE</span>
<span class="hl num">250</span><span class="hl kwb">-DSN</span>
<span class="hl num">250</span><span class="hl kwb">-ETRN</span>
<span class="hl num">250</span><span class="hl kwb">-AUTH</span> DIGEST<span class="hl kwb">-MD5</span> CRAM<span class="hl kwb">-MD5</span>
<span class="hl num">250</span><span class="hl kwb">-DELIVERBY</span>
<span class="hl num">250</span> HELP
<span class="hl opt">>>></span> VERB
<span class="hl num">250 2.0.0</span> Verbose mode
<span class="hl opt">>>></span> MAIL From<span class="hl opt">:<</span>natus@nc<span class="hl kwb">-ass-vip</span>.sdv.fr<span class="hl opt">></span> SIZE<span class="hl opt">=</span><span class="hl num">15</span> AUTH<span class="hl opt">=</span>natus@nc<span class="hl kwb">-ass-vip</span>.sdv.fr
<span class="hl num">250 2.1.0</span> <span class="hl opt"><</span>natus@nc<span class="hl kwb">-ass-vip</span>.sdv.fr<span class="hl opt">></span>... Sender ok
<span class="hl opt">>>></span> RCPT To<span class="hl opt">:<</span>natus@riseup.net<span class="hl opt">></span>
<span class="hl opt">>>></span> DATA
<span class="hl num">250 2.1.5</span> <span class="hl opt"><</span>natus@riseup.net<span class="hl opt">></span>... Recipient ok
<span class="hl num">354</span> Enter <span class="hl kwc">mail</span><span class="hl opt">,</span> end with <span class="hl sng">"."</span> on a line by itself
<span class="hl opt">>>></span> .
<span class="hl num">050</span> <span class="hl opt"><</span>natus@riseup.net<span class="hl opt">></span>... Connecting to mx1.riseup.net. via esmtp...
<span class="hl num">050 220</span> mx1.riseup.net ESMTP <span class="hl opt">(</span>spam is not appreciated<span class="hl opt">)</span>
<span class="hl num">050</span> <span class="hl opt">>>></span> EHLO nc<span class="hl kwb">-ass-vip</span>.sdv.fr
<span class="hl num">050 250</span><span class="hl kwb">-mx1</span>.riseup.net
<span class="hl num">050 250</span><span class="hl kwb">-PIPELINING</span>
<span class="hl num">050 250</span><span class="hl kwb">-SIZE</span> <span class="hl num">25600000</span>
<span class="hl num">050 250</span><span class="hl kwb">-ETRN</span>
<span class="hl num">050 250</span><span class="hl kwb">-STARTTLS</span>
<span class="hl num">050 250</span><span class="hl kwb">-ENHANCEDSTATUSCODES</span>
<span class="hl num">050 250</span><span class="hl kwb">-8BITMIME</span>
<span class="hl num">050 250</span> DSN
<span class="hl num">050</span> <span class="hl opt">>>></span> STARTTLS
<span class="hl num">050 220 2.0.0</span> Ready to start TLS
<span class="hl num">050</span> <span class="hl opt">>>></span> EHLO nc<span class="hl kwb">-ass-vip</span>.sdv.fr
<span class="hl num">050 250</span><span class="hl kwb">-mx1</span>.riseup.net
<span class="hl num">050 250</span><span class="hl kwb">-PIPELINING</span>
<span class="hl num">050 250</span><span class="hl kwb">-SIZE</span> <span class="hl num">25600000</span>
<span class="hl num">050 250</span><span class="hl kwb">-ETRN</span>
<span class="hl num">050 250</span><span class="hl kwb">-ENHANCEDSTATUSCODES</span>
<span class="hl num">050 250</span><span class="hl kwb">-8BITMIME</span>
<span class="hl num">050 250</span> DSN
<span class="hl num">050</span> <span class="hl opt">>>></span> MAIL From<span class="hl opt">:<</span>natus@nc<span class="hl kwb">-ass-vip</span>.sdv.fr<span class="hl opt">></span> SIZE<span class="hl opt">=</span><span class="hl num">303</span>
<span class="hl num">050 250 2.1.0</span> Ok
<span class="hl num">050</span> <span class="hl opt">>>></span> RCPT To<span class="hl opt">:<</span>natus@riseup.net<span class="hl opt">></span>
<span class="hl num">050</span> <span class="hl opt">>>></span> DATA
<span class="hl num">050 250 2.1.5</span> Ok
<span class="hl num">050 354</span> End data with <span class="hl opt"><</span>CR<span class="hl opt">><</span>LF<span class="hl opt">></span>.<span class="hl opt"><</span>CR<span class="hl opt">><</span>LF<span class="hl opt">></span>
<span class="hl num">050</span> <span class="hl opt">>>></span> .
<span class="hl num">050 250 2.0.0</span> Ok<span class="hl opt">:</span> queued as <span class="hl num">48</span>KdzJ0tYhzFd8d
<span class="hl num">050</span> <span class="hl opt"><</span>natus@riseup.net<span class="hl opt">></span>... Sent <span class="hl opt">(</span>Ok<span class="hl opt">:</span> queued as <span class="hl num">48</span>KdzJ0tYhzFd8d<span class="hl opt">)</span>
<span class="hl num">250 2.0.0 01</span>FISAiT005455 Message accepted <span class="hl kwa">for</span> delivery
natus@riseup.net... Sent <span class="hl opt">(</span><span class="hl num">01</span>FISAiT005455 Message accepted <span class="hl kwa">for</span> delivery<span class="hl opt">)</span>
Closing connection to <span class="hl opt">[</span><span class="hl num">127.0.0.1</span><span class="hl opt">]</span>
<span class="hl opt">>>></span> QUIT
<span class="hl num">221 2.0.0</span> nc<span class="hl kwb">-ass-vip</span>.sdv.fr closing connection
</code></pre>
<p>Cependant, il reste des configuration à réaliser spécifiques à
<code>sympa</code>. La documentation les décrit mais il faut bien comprendre la
notion d´aliases. <a href="https://sympa-community.github.io/manual/install/configure-http-server-spawnfcgi.html">voir la documentation</a></p>
<h3>Configuration de nginx</h3>
<p>À réaliser.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Simpa — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-03-08 02:08</time> </div>https://blog.parisni.com/wiki/msgEncrypted Messaging Apps2021-07-20T21:14:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">messaging</span>
<span class="w3-tag w3-red tag">security</span></p>
<h1>Encrypted Messaging Apps</h1>
<p>Only the FOSS applications are listed here. The closed-source cannot
be trusted for privacy.</p>
<h2>Matrix</h2>
<ul>
<li>federated, no need for central server</li>
<li>group conversation</li>
<li>IOS / Android / Desktop / Web</li>
</ul>
<h2>Session</h2>
<ul>
<li><a href="https://github.com/oxen-io">self hosting</a></li>
<li>signal fork</li>
<li>no phone number but an alphanumeric</li>
<li>handle async messaging</li>
<li>group conversation</li>
<li>NO voice call</li>
<li>IOS / Android / Desktop</li>
<li>Multiple device can be linked</li>
</ul>
<h2>Wire</h2>
<ul>
<li><a href="https://github.com/wireapp/wire-server">self hosting possible</a></li>
<li>cofunded by skype creator</li>
<li>designed to scale on kubernetes</li>
<li>group conversation</li>
<li>video</li>
<li>voice call</li>
</ul>
<h2>Deltachat</h2>
<ul>
<li>distributed, no need for central server</li>
<li>group conversation</li>
<li>based on email</li>
<li>encrytion with pgp</li>
<li>metadata leaks in the mail</li>
<li>IOS / Android / Desktop</li>
<li>Multiple device can be linked</li>
<li>message you send are not visible on each device</li>
</ul>
<h2>Jami</h2>
<ul>
<li>distributed, no need for central server</li>
<li>peer-to-peer</li>
<li>group conversation</li>
<li>NO async message</li>
<li>Multiple device can be linked</li>
<li>message you send are not visible on each device</li>
</ul>
<h2>Snikket</h2>
<ul>
<li>based on XMPP</li>
</ul>
<h2>Tox</h2>
<ul>
<li>distributed, no need for central server</li>
<li>peer-to-peer</li>
<li>NO async message</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Encrypted Messaging Apps — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-07-20 21:14</time> </div>https://blog.parisni.com/wiki/musicMusic2021-09-02T15:15:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">music</span></p>
<h1>Music</h1>
<h2>Extract music from mkv</h2>
<p><a href="https://www.hecticgeek.com/extract-audio-using-ffmpeg-ubuntu/">source</a></p>
<p>you can then convert the “mka” into flac, or ogg with vlc-media-player.</p>
<h2>Edit id3-tags</h2>
<p>Use <a href="https://github.com/KristoforMaynard/music-tag">music-tag</a></p>
<pre><code class="language-bash">python <span class="hl kwb">-m</span> music_tag <span class="hl kwb">--to-csv</span> tags.csv .<span class="hl opt">/</span>sample
<span class="hl slc"># Edit the csv and re-import it to apply the id3</span>
python <span class="hl kwb">-m</span> music_tag <span class="hl kwb">--from-csv</span> tags.csv
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Music — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-09-02 15:15</time> </div>https://blog.parisni.com/wiki/nexusNexus2020-07-07T10:17:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">java</span>
<span class="w3-tag w3-red tag">maven</span></p>
<h1>Nexus</h1>
<h2>Configuration</h2>
<h2>Maven configuration</h2>
<h2>Cleanup snapshots</h2>
<p>It is possible to schedule tasks. In particular there is a template to
delete snapshots, and only keep a defined number of them.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Nexus — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-07-07 10:17</time> </div>https://blog.parisni.com/wiki/umlUML2021-11-26T18:43:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">versionning</span></p>
<h1>UML</h1>
<h2>PlantUML</h2>
<p>This is FOSS java tool to built UML content from a DSL.</p>
<pre><code>@startuml
package "Some Group" {
HTTP - [First Component]
[Another Component]
}
node "Other Groups" {
FTP - [Second Component]
[First Component] --> FTP
}
cloud {
[Example 1]
}
database "MySql" {
folder "This is my folder" {
[Folder 3]
}
frame "Foo" {
[Frame 4]
}
}
[Another Component] --> [Example 1]
[Example 1] --> [Folder 3]
[Folder 3] --> [Frame 4]
@enduml
</code></pre>
<p><img src="/images/puml.webp" alt="Component UML"></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=UML — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-26 18:43</time> </div>https://blog.parisni.com/wiki/sqlSql Clients2020-03-26T08:05:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">sql</span>
<span class="w3-tag w3-red tag">emacs</span></p>
<h1>Sql Clients</h1>
<h2>Orgmode SQL</h2>
<p><a href="https://orgmode.org/worg/org-contrib/babel/languages.html">That post</a> introduced me the org-mode sql. I find it convenient to get the result within an org notebook. However still, I have not found equivalent to psql shortcuts such <code>\d</code> to list all tables, and so on.</p>
<pre><code class="language-lisp"><span class="hl slc">;; https://orgmode.org/worg/org-contrib/babel/languages.html</span>
<span class="hl slc">;; active Babel languages</span>
<span class="hl opt">(</span><span class="hl kwa">org-babel-do-load-languages</span>
<span class="hl opt">'</span>org-babel-load-languages
<span class="hl opt">'((</span><span class="hl kwa">sql</span> . t<span class="hl opt">)</span>
<span class="hl opt">(</span><span class="hl kwa">emacs-lisp</span> . nil<span class="hl opt">)))</span>
</code></pre>
<pre><code>
#+begin_src sql :engine postgresql :dbhost localhost :dbuser myuser:database thedb :dbport theport
select "savedAt", substring("content" ,1 , 30) b
from "Notes"
limit 3
#+end_src
#+RESULTS:
| savedAt | b |
|----------------------------+-------------------------------|
| 2019-11-05 09:58:43.363+01 | |
| 2019-11-02 14:46:27.037+01 | installé sur la db de natus > |
| 2019-11-02 15:26:33.183+01 | |
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Sql Clients — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-03-26 08:05</time> </div>https://blog.parisni.com/wiki/pdfpdf2023-12-23T16:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">scala</span>
<span class="w3-tag w3-red tag">web</span></p>
<h1>pdf</h1>
<h2>pdf viewer</h2>
<p>I have for long time being disapointed with <strong>evince</strong> pdf reader with
it's lack of capacity to zoom in and out. The same happened with
<strong>foxit reader</strong> and its lack of history management in its linux
version.</p>
<p>Finally I discovered <strong>mupdf</strong>. It has a vi key biding, has a great
zooming window, allows back in the history. The only missing feature
is beeing able to follow links with keyboard as <strong>luakit</strong> does.</p>
<h3>keybinding</h3>
<ul>
<li><em>i</em> : dark mode</li>
<li><em>/</em> : search</li>
<li><em>W</em> : fit to page</li>
<li><em>t</em> : go back to last jump</li>
</ul>
<h2>Concatenating pdf</h2>
<p>The command below is great to concat multiple pdf together. I had some
problem with pdftk. You can also add a pdf title, and chose the pagesize</p>
<pre><code class="language-bash">gs <span class="hl kwb">-dBATCH -dNOPAUSE -sPAPERSIZE</span><span class="hl opt">=</span>a4 <span class="hl kwb">-q -sDEVICE</span><span class="hl opt">=</span>pdfwrite <span class="hl kwb">-sOutputFile</span><span class="hl opt">=</span>merged.pdf mine1.pdf mine2.pdf <span class="hl kwb">-c</span> <span class="hl sng">"[ /Title (My New Title) /DOCINFO pdfmark"</span>
</code></pre>
<p>Also this command allows to compress a large pdf into something
smaller:</p>
<pre><code class="language-bash">gs <span class="hl kwb">-sDEVICE</span><span class="hl opt">=</span>pdfwrite <span class="hl kwb">-dCompatibilityLevel</span><span class="hl opt">=</span><span class="hl num">1.4</span> <span class="hl kwb">-dPDFSETTINGS</span><span class="hl opt">=/</span>default <span class="hl kwb">-dNOPAUSE -dQUIET -dBATCH -dDetectDuplicateImages -dCompressFonts</span><span class="hl opt">=</span>true <span class="hl kwb">-r150 -sOutputFile</span><span class="hl opt">=</span>output.pdf input.pdf
</code></pre>
<p>It is also possible to split pdf with pdfbox:</p>
<pre><code class="language-bash">java <span class="hl kwb">-jar</span> <span class="hl opt">/</span>app<span class="hl opt">/</span>edsr<span class="hl opt">/</span>script<span class="hl opt">/</span>java<span class="hl opt">/</span>pdfbox<span class="hl kwb">-app-2</span>.0<span class="hl num">.0</span><span class="hl kwb">-SNAPSHOT</span>.jar PDFSplit .<span class="hl opt">/</span>authorisation<span class="hl kwb">-inscription</span>.pdf
</code></pre>
<h2>pdf generation</h2>
<h3>from html</h3>
<p>weasyprint makes a brillant work to transform html page into pdf.</p>
<pre><code class="language-bash">pip3 <span class="hl kwc">install</span> weasyprint
weasyprint https<span class="hl opt">://</span>website<span class="hl opt">/</span>index.html index.pdf
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=pdf — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-23 16:22</time> </div>https://blog.parisni.com/wiki/crowdsecCrowdsec2021-12-05T22:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">security</span></p>
<h1>Crowdsec</h1>
<h2>Install</h2>
<ul>
<li>crowdsec is in the pacman repos</li>
<li>crowdsec-firewall is in the aur repos (yaourt)</li>
<li>it can use postgres as a backend</li>
<li>prometheus metrics can be vizualized with <a href="https://github.com/crowdsecurity/grafana-dashboards">graphana dashboards</a></li>
<li>also a metabase dashboard exist but can only be used with sqlite</li>
<li>a nginx-bouncer exists but only for ubuntu at the moment</li>
</ul>
<p>Create local-api tokens:</p>
<pre><code class="language-bash">cscli machines add <span class="hl kwb">-a</span>
cscli bouncers add firewall<span class="hl kwb">-bouncer</span>
</code></pre>
<p>Edit the config files :</p>
<pre><code>vim /etc/crowdsec/acquis.yaml
vim /etc/crowdsec/user.yaml
vim /etc/crowdsec/config.yaml
vim /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
vim /etc/prometheus/prometheus.yml
</code></pre>
<p>Show metrics:</p>
<pre><code class="language-bash">cscli metrics
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Crowdsec — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-05 22:09</time> </div>https://blog.parisni.com/wiki/teraformTerraform2020-08-05T23:04:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">infrastructure</span></p>
<h1>Terraform</h1>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Terraform — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-08-05 23:04</time> </div>https://blog.parisni.com/wiki/spring-cloud-streamSpring cloud stream2023-12-15T18:53:00+00:00<div id="generated-toc"> </div><hr> <p>#java
#spring</p>
<h1>Spring cloud stream</h1>
<h2>documentation</h2>
<ul>
<li><a href="https://docs.spring.io/spring-cloud-stream/reference/spring-cloud-stream/bindings.html#binding-and-binding-names">binding names</a></li>
<li><a href="https://docs.spring.io/spring-cloud-stream/reference/spring-cloud-stream/overview-error-handling.html#drop-failed-messages">stream error handling</a></li>
<li><a href="https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/spring-cloud-stream-binder-kafka.html#_error_handling">binder stream error handling</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Spring cloud stream — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-15 18:53</time> </div>https://blog.parisni.com/wiki/gimpGimp2021-11-07T19:15:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">image</span></p>
<h1>Gimp</h1>
<h2>Make some zone be transparent</h2>
<ol>
<li>Add gamma to the image</li>
<li>select by color</li>
<li>press delete</li>
</ol>
<h2>Underline text</h2>
<ol>
<li>Add a layer to the image (if not exit)</li>
<li>Choose pencil tool</li>
<li>Click</li>
<li>Press shift and click on the other extremity</li>
<li>You get your underline</li>
</ol>
<h2>Surround text</h2>
<ol>
<li>Choose selection (rectangle or ellipse)</li>
<li>Surround</li>
<li>Press Enter</li>
<li>Select > To path</li>
<li>Edit > Stroke path</li>
<li>You get your surround</li>
</ol>
<h2>Change image geometry</h2>
<ol>
<li>Choose the "perspective" tool</li>
<li>Click ane drag the image as you want</li>
<li>Apply by clicking "transform" in the perspective's popup</li>
</ol>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Gimp — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-07 19:15</time> </div>https://blog.parisni.com/wiki/kata-containerKata Containers2023-04-02T14:27:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span></p>
<h1>Kata Containers</h1>
<h2>What ?</h2>
<p>Kata containers, wrapp docker within ligh virtual machines. This leads to
provide security enhancement for docker applications.</p>
<h2>Install</h2>
<p>Docker config:</p>
<pre><code># edit /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --add-runtime kata-runtime=/usr/bin/kata-runtime -H fd:// --containerd=/run/containerd/containerd.sock
</code></pre>
<p>Containerd config:</p>
<pre><code class="language-bash"><span class="hl slc"># Add this above /etc/containerd/config.toml</span>
<span class="hl opt">[</span>plugins<span class="hl opt">]</span>
<span class="hl opt">[</span>plugins.<span class="hl sng">"io.containerd.grpc.v1.cri"</span><span class="hl opt">]</span>
<span class="hl opt">[</span>plugins.<span class="hl sng">"io.containerd.grpc.v1.cri"</span>.containerd<span class="hl opt">]</span>
default_runtime_name <span class="hl opt">=</span> <span class="hl sng">"kata"</span>
<span class="hl opt">[</span>plugins.<span class="hl sng">"io.containerd.grpc.v1.cri"</span>.containerd.runtimes<span class="hl opt">]</span>
<span class="hl opt">[</span>plugins.<span class="hl sng">"io.containerd.grpc.v1.cri"</span>.containerd.runtimes.kata<span class="hl opt">]</span>
runtime_type <span class="hl opt">=</span> <span class="hl sng">"io.containerd.kata.v2"</span>
</code></pre>
<p>Download the latest kata bin version:</p>
<pre><code class="language-bash"><span class="hl kwb">cd</span> <span class="hl opt">/</span>opt
wget https<span class="hl opt">://</span>github.com<span class="hl opt">/</span>kata<span class="hl kwb">-containers</span><span class="hl opt">/</span>kata<span class="hl kwb">-containers</span><span class="hl opt">/</span>releases<span class="hl opt">/</span>download<span class="hl opt">/</span><span class="hl num">3.0.2</span><span class="hl opt">/</span>kata<span class="hl kwb">-static-3</span>.0<span class="hl num">.2</span><span class="hl kwb">-x86_64</span>.<span class="hl kwc">tar</span>.xz
<span class="hl kwc">tar</span> <span class="hl kwb">-xvf</span> <span class="hl opt">/</span>opt<span class="hl opt">/</span>kata<span class="hl kwb">-static-3</span>.0<span class="hl num">.2</span><span class="hl kwb">-x86_64</span>.<span class="hl kwc">tar</span>.xz
<span class="hl slc"># those are needed</span>
<span class="hl kwc">ln</span> <span class="hl kwb">-s</span> <span class="hl opt">/</span>opt<span class="hl opt">/</span>kata<span class="hl opt">/</span>bin<span class="hl opt">/</span>kata<span class="hl kwb">-runtime</span> <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>kata<span class="hl kwb">-runtime</span>
<span class="hl kwc">ln</span> <span class="hl kwb">-s</span> <span class="hl opt">/</span>opt<span class="hl opt">/</span>kata<span class="hl opt">/</span>bin<span class="hl opt">/</span>kata<span class="hl kwb">-collect-data</span>.sh <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>kata<span class="hl kwb">-collect-data</span>.sh
</code></pre>
<p>Load kernel modules:</p>
<pre><code class="language-bash">modprobe vhost<span class="hl kwb">-vsock</span>
modprobe vhost<span class="hl kwb">-net</span>
modprobe vsock
</code></pre>
<p>Restart docker and containerd:</p>
<pre><code>systemctl restart containerd docker
</code></pre>
<p>Now you can start with kata runtime:</p>
<pre><code>docker run --rm -itd --runtime "io.containerd.kata.v2" --network none --name busybox2 busybox
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Kata Containers — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-02 14:27</time> </div>https://blog.parisni.com/wiki/videoVidéo2024-02-07T13:45:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">audio</span></p>
<h1>Vidéo</h1>
<h2>Yt-dlp</h2>
<pre><code># list subs
yt-dlp -F --list-sub https://www.arte.tv/fr/videos/075801-000-A/flee/
# download subs
--write-subs --sub-langs "fr" https://www.arte.tv/fr/videos/075801-000-A/flee/
</code></pre>
<p>Resulting vtt files can be merged simply by adding content from one into the other.</p>
<p>Peertube hanlde both vtt and srt.</p>
<h2>Mkv</h2>
<pre><code>ffmpeg -i my-film.mkv -map 0:s:0 ~/my-film.srt
</code></pre>
<blockquote>
<p>The first number is which input you select (0 is the first and only input in my command), second character is what type of stream you select (s is subtitles, a is audio, v is video), third is which stream you select out of those. So like this: -map input_file_index:stream_type_specifier:stream_index So the code above takes the Movie.mkv, then its subtitle streams, then the first subtitle stream. (counting starts with 0, because it's an index.)</p>
</blockquote>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Vidéo — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2024-02-07 13:45</time> </div>https://blog.parisni.com/wiki/darktableDarktable2023-02-03T16:08:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">image</span></p>
<h1>Darktable</h1>
<h2>darktable from graphic interface</h2>
<h3>Create a style</h3>
<p>One can create a pipeline of transformations and save them as
re-usable <strong>style</strong>.</p>
<h2>darktable from command line</h2>
<p><strong>darktable-cli</strong> is available together with <strong>darktable</strong> gui version.</p>
<h3>Apply a style to images</h3>
<p><a href="https://www.darktable.org/usermanual/en/overview_chapter.html#darktable_cli_commandline_parameters">cli documentation</a></p>
<p>this will apply "style name" and scale the image to 300px</p>
<pre><code class="language-bash">dakrtable<span class="hl kwb">-cli</span> <span class="hl opt"><</span>input <span class="hl kwc">file</span><span class="hl opt">> <</span>outpout <span class="hl kwc">file</span><span class="hl opt">></span> <span class="hl kwb">--style</span> <span class="hl opt"><</span>style name<span class="hl opt">></span> <span class="hl kwb">--width</span> <span class="hl num">300</span>
</code></pre>
<p>this can also apply to folders</p>
<pre><code class="language-bash">dakrtable<span class="hl kwb">-cli</span> <span class="hl opt"><</span>input folder<span class="hl opt">> <</span>outpout folder<span class="hl opt">>/<</span><span class="hl kwc">file</span> template name<span class="hl opt">></span>.<span class="hl opt"><</span>extension<span class="hl opt">></span> <span class="hl kwb">--style</span> <span class="hl opt"><</span>style name<span class="hl opt">></span> <span class="hl kwb">--width</span> <span class="hl num">300</span>
</code></pre>
<h2>darktable from lua</h2>
<p><a href="https://www.darktable.org/lua-api/">The lua api</a> offers the darktable features within a lua program.</p>
<h2>Refresh folder</h2>
<p><code>></code> import <code>></code> ad to library <code>></code> select only new pictures</p>
<h2>image processing in 3 modules</h2>
<p>The following basic adjustments are fundamental to scene-referred editing and will be required, to some extent, on the
majority of images. You can usually produce a good-looking image with these steps alone.
As you will be adjusting the tones and colors of the image, start by enabling color assessment mode (press Ctrl+B) and
perform the following edits on the zoomed-out image while in this mode.</p>
<ol>
<li>Set overall image brightness: First, set the overall (average) brightness of the image (the mid-gray point) by adjusting the exposure slider in the exposure module. This is a purely artistic setting and should be defined based on your intent – for example, for a high-key image you will set the average brightness to be lighter than for a low-key image. The color assessment mode provides you with two reference points to assist with this by surrounding the image with a white frame against a middle-gray background. At this point, don’t worry if the brightest parts of your image lose detail – this can be recovered in the next step. Note: The lens correction module can also affect the image brightness so you may want to consider enabling it before adjusting exposure.</li>
<li>Set white and black points: The next two steps use the filmic rgb module to define how the tones in your image will be mapped to the dynamic range of your display. Start by setting the white and black relative exposure sliders in the scene tab . These are purely technical settings, defining white and black relative to the mid-gray point you set in the previous step. If your image contains tones you want to treat as pure white or pure black you can use the color pickers beside the sliders to set these values (using the maximum and minimum brightness of the image). Otherwise set the values manually using the color assessment frames as a reference.</li>
<li>Adjust the contrast: Now move to the look tab in filmic rgb (for now we will skip the reconstruct tab ). Enable the look only view at the top of the module to see a representation of the filmic tone curve, which consists of a straight section in the middle (used to set the contrast of the mid-tones) and curved sections at the top and bottom (where the shadows and highlights are compressed to fit the dynamic range of the display). The contrast slider changes the slope of the straight section (the mid-tone image contrast), the latitude slider changes its length and the shadows/highlights balance slider changes its position. There is a lot of give-and-take involved here – if you want to increase the contrast of the mid-tones, you must sacrifice contrast in the shadows/ highlights and vice versa. The default settings of this module are tuned to work for the majority of images but you should experiment with these sliders to understand how they affect the image.</li>
<li>Color preservation: The tone mapping in the filmic rgb module attempts to redistribute the tones in your image without affecting color reproduction. While the default color preservation algorithm works for most images, you are encouraged to experiment by changing the preserve chrominance setting in the options tab if you do not like how the colors appear.</li>
<li>Saturation: Your image will probably not look very colorful at this point. You can adjust the global saturation of the image using the color balance rgb module. The “basic colorfulness” preset should provide you with generallyreasonable defaults, but you are encouraged to experiment further with these settings as required.</li>
</ol>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Darktable — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-02-03 16:08</time> </div>https://blog.parisni.com/wiki/Query-enginesQuery Engines2023-10-11T16:36:00+00:00<div id="generated-toc"> </div><hr> <h1>Query Engines</h1>
<h2>Starrocks</h2>
<h3>Ressources</h3>
<ul>
<li><a href="https://www.starrocks.io/blog/four-simple-ways-to-deploy-starrocks">deploying starrocks</a></li>
<li><a href="https://docs.celerdata.com/en-us/main/security/cloud_access_control/use_sso#enable-sso">okta integration</a></li>
<li><a href="https://github.com/StarRocks/starrocks/blob/3c66951310efd383c4e8629d39fa9af69391389a/fe/fe-core/src/main/java/com/starrocks/connector/hive/HiveStatisticsProvider.java">hive statistics provider</a></li>
<li><a href="https://docs.starrocks.io/en-us/latest/using_starrocks/query_cache">query cache</a></li>
<li><a href="https://docs.starrocks.io/en-us/3.0/deployment/deploy_shared_data">decoupled storage</a></li>
<li><a href="https://docs.starrocks.io/en-us/latest/data_source/Block_cache">Local cache</a></li>
<li><a href="https://docs.starrocks.io/en-us/3.0/administration/spill_to_disk">spill to disk</a></li>
<li><a href="https://docs.starrocks.io/en-us/3.0/administration/monitor_manage_big_queries">monitoring queries</a></li>
<li><a href="https://medium.com/@kangkaisen/mpp-pipeline-vs-grouped-execution-vs-stage-by-stage-f4616052474a">why starrocks outperforms spark</a></li>
<li><a href="https://doris.apache.org/docs/dev/lakehouse/multi-catalog/hive/#integrate-with-apache-ranger">ranger integration of doris</a></li>
<li><a href="https://www.starrocks.io/blog/starrocks-olap-workloads-with-starrocks-query-cache">Query cache</a></li>
</ul>
<h3>configs</h3>
<ul>
<li><code>set sql_dialect = 'trino'</code> <a href="https://docs.starrocks.io/en-us/3.0/reference/System_variable">see</a></li>
<li><code>set enable_pipeline_engine=true</code> for query cache</li>
<li><code>set enable_storage_cache=true</code> for decoupled storage local cache</li>
<li><code>SET enable_populate_block_cache=true</code> for block cache</li>
<li><code>SET enable_spill=true</code> also configure BE <code>spill_local_storage_dir</code></li>
</ul>
<h3>hudi</h3>
<h4>external catalog</h4>
<ul>
<li><code>aws.s3.enable_ssl</code></li>
</ul>
<h3>metadata sync</h3>
<ul>
<li><a href="https://docs.starrocks.io/en-us/latest/data_source/catalog/hudi_catalog#metadataupdateparams">Doc</a></li>
<li><a href="https://docs.starrocks.io/en-us/latest/data_source/catalog/hudi_catalog#appendix-understand-metadata-automatic-asynchronous-update">Async mexanism</a></li>
<li><a href="https://docs.starrocks.io/en-us/latest/data_source/catalog/hudi_catalog#automatic-incremental-update">incremental metadata sync</a></li>
</ul>
<h4>code</h4>
<ul>
<li><a href="https://github.com/StarRocks/starrocks/tree/branch-3.1/java-extensions/hudi-reader">Hudi reader</a></li>
<li><a href="https://github.com/StarRocks/starrocks/blob/b3e49e480e6c0ca8bc36264f5ca52f1c24074017/fe/fe-core/src/main/java/com/starrocks/catalog/HudiTable.java#L218">hudi table creates a thrift hudi table</a></li>
<li><a href="https://github.com/StarRocks/starrocks/blob/b3e49e480e6c0ca8bc36264f5ca52f1c24074017/fe/fe-core/src/main/java/com/starrocks/connector/hudi/HudiRemoteFileIO.java#L74">This returns the hudi files, based on timeline</a></li>
<li><a href="https://github.com/StarRocks/starrocks/blob/main/fe/fe-core/src/main/java/com/starrocks/connector/RemoteScanRangeLocations.java#L237">this works on the files</a></li>
<li><a href="https://github.com/StarRocks/starrocks/blob/main/fe/fe-core/src/main/java/com/starrocks/connector/RemoteScanRangeLocations.java#L168">there's likely option to use jni reader</a></li>
<li><a href="https://github.com/StarRocks/starrocks/blob/b3e49e480e6c0ca8bc36264f5ca52f1c24074017/fe/fe-core/src/main/java/com/starrocks/connector/hive/HiveStatisticsProvider.java">hudi fallbacks to hive statistics</a></li>
</ul>
<p>So the FE get file listing for partitions at runtime. Depending on the table type, it merges logs files or not.
It could also get the file listing from the hudi metada table.</p>
<h3>caching</h3>
<p>There is 3 mechanisms:</p>
<ol>
<li>Query cache: only used with native tables. It stores intermediate results, not the final ones</li>
<li>Storage cache: only used with native tables stored on cloud storage. It stores the new data also locally</li>
<li>Block cache: only used for external tables on cloud storage. It stores files locally, either disk or ram.</li>
</ol>
<h2>Athena</h2>
<p><a href="https://docs.aws.amazon.com/athena/latest/ug/performance-tuning-s3-throttling.html">From doc</a>:
Use ORC for complex types
Currently, when you query columns stored in Parquet that have complex data types (for example, array, map, or struct), Athena reads an entire row of data instead of selectively reading only the specified columns. This is a known issue in Athena. As a workaround, consider using ORC</p>
<ul>
<li><a href="https://docs.aws.amazon.com/athena/latest/ug/query-metrics-viewing.html">Query queues are automatic</a></li>
<li><a href="https://docs.aws.amazon.com/general/latest/gr/athena.html">quotas</a></li>
<li><a href="https://docs.aws.amazon.com/athena/latest/ug/capacity-management-requirements.html">how to choose dpu</a></li>
<li><a href="https://aws.amazon.com/blogs/big-data/reduce-cost-and-improve-query-performance-with-amazon-athena-query-result-reuse/">Query reuse=cache</a></li>
</ul>
<h3>jdbc</h3>
<p>Details in the simbra jdbc manual</p>
<pre><code>jdbc:awsathena://User=[AccessKey];Password=
[SecretKey];S3OutputLocation=[Output];[Property1]=[Value1];
[Property2]=[Value2];...
</code></pre>
<ul>
<li><code>enableResultReuseByAgeenableResultReuseByAge</code>: 0/1 This property specifies whether the connector reuses the query results for the same type of query.</li>
<li><code>maxResultReuseAgeInMinutesmaxResultReuseAgeInMinutes</code> : This property specifies whether the connector considers the age in minutes of previous query result for reuse. The range can be 0 to 10080 minutes.</li>
</ul>
<h2>dremio</h2>
<ul>
<li><a href="https://web.archive.org/web/20211022224127/https://www.dremio.com/dremio-vs-presto/">Bench vs prestos</a></li>
<li></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Query Engines — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-10-11 16:36</time> </div>https://blog.parisni.com/wiki/certsCerts2020-03-09T00:05:00+00:00<div id="generated-toc"> </div><hr>
<p><span class="w3-tag w3-red tag">web</span>
<span class="w3-tag w3-red tag">security</span></p>
<h1>Certs</h1>
<p>Let's encrypt provides free certificates. However, the configuration and maintenance is tedious.</p>
<h2>Certbot</h2>
<p><a href="https://certbot.eff.org/about/">Certbot</a> is a blazing fast way of
configuring https on your website, independantly of os, web server
software.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Certs — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-03-09 00:05</time> </div>https://blog.parisni.com/wiki/elkELK2021-12-06T23:18:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">monitoring</span></p>
<h1>ELK</h1>
<h2>Filebeat setup</h2>
<p>Modules can be enabled:</p>
<pre><code class="language-bash">filebeat modules <span class="hl kwb">enable</span> system nginx postgresql
</code></pre>
<p>So far, only direct access to elastic works with filebeat:</p>
<pre><code class="language-bash">filebeat <span class="hl kwb">--path</span>.home <span class="hl opt">/</span>usr<span class="hl opt">/</span>share<span class="hl opt">/</span>filebeat <span class="hl kwb">--path</span>.config <span class="hl opt">/</span>etc<span class="hl opt">/</span>filebeat <span class="hl kwb">--path</span>.data <span class="hl opt">/</span>var<span class="hl opt">/</span>lib<span class="hl opt">/</span>filebeat <span class="hl kwb">--path</span>.logs <span class="hl opt">/</span>var<span class="hl opt">/</span>log<span class="hl opt">/</span>filebeat setup <span class="hl kwb">-e</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=ELK — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-06 23:18</time> </div>https://blog.parisni.com/wiki/dslDomain Specific Language2020-08-02T13:21:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">coding</span></p>
<h1>Domain Specific Language</h1>
<p>There is many DSL implementation patterns:</p>
<ul>
<li>Internal DSL
<ul>
<li>smart API</li>
<li>fluent API</li>
<li>syntax tree manipulation</li>
<li>typed embedding</li>
<li>reflective metaprogramming</li>
<li>runtime metaprogramming</li>
<li>compile time metaprogramming</li>
</ul>
</li>
<li>External DSL
<ul>
<li>parser combinator</li>
<li>mixing DSL and embedded foreign code</li>
<li>xml to consumable resource</li>
</ul>
</li>
</ul>
<p><em>Internal DSL</em> are embedded the host programming language while
<em>Extenal DSL</em> are created from scratch with in <em>parse</em> / <em>process</em>
steps.</p>
<h2>Explicitly typed constraint in scala</h2>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Domain Specific Language — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-08-02 13:21</time> </div>https://blog.parisni.com/wiki/mountMount2020-11-15T17:51:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">filesystem</span></p>
<h1>Mount</h1>
<h2>mtp</h2>
<p>mtp is android protocol.</p>
<pre><code class="language-bash">$ <span class="hl kwb">pwd</span> <span class="hl slc"># prints "/run/user/1000/gvfs/mtp:host=%5Busb%3A003%2C096%5D/SanDisk SD card/Movies"</span>
$ rsync <span class="hl kwb">--verbose --progress --omit-dir-times --no-perms --recursive --inplace</span> ~<span class="hl opt">/</span>Videos<span class="hl opt">/</span> .<span class="hl opt">/</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mount — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-11-15 17:51</time> </div>https://blog.parisni.com/wiki/nvidiaNvidia2020-03-08T02:08:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">gpu</span></p>
<h1>Nvidia</h1>
<h2>Désactiver la carte nvidia</h2>
<pre><code class="language-bash">sudo <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>prime<span class="hl kwb">-select</span> nvidia
sudo <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>prime<span class="hl kwb">-select</span> intel
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Nvidia — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-03-08 02:08</time> </div>https://blog.parisni.com/wiki/TerraformTerraform2023-10-19T15:39:00+00:00<div id="generated-toc"> </div><hr> <h1>Terraform</h1>
<h2>secrets</h2>
<blockquote>
<p>any variable foo can be replaced by environmeny by naming it TF_VAR_foo</p>
</blockquote>
<blockquote>
<p>SECRETS ARE ALWAYS STORED IN TERRAFORM STATE</p>
</blockquote>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Terraform — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-10-19 15:39</time> </div>https://blog.parisni.com/wiki/bashBash2023-08-12T00:11:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span></p>
<h1>Bash</h1>
<h2>remove files with chinese character AND spaces</h2>
<ul>
<li>PCRE regexp with <code>-P</code></li>
<li>quote spaces with xargs + pattern</li>
</ul>
<pre><code class="language-bash"><span class="hl slc">#./[机器学习]Introduction to Machine Learning.pdf</span>
<span class="hl slc">#./算法竞赛入门经典(第2版) (算法艺术与信息学竞赛).pdf</span>
<span class="hl kwc">find</span> . <span class="hl opt">|</span> <span class="hl kwc">grep</span> <span class="hl kwb">-P</span> <span class="hl sng">'[\p{Han}]'</span><span class="hl opt">|</span><span class="hl kwc">xargs</span> <span class="hl kwb">-I</span> <span class="hl sng">'{}'</span> <span class="hl kwc">rm</span> <span class="hl sng">"{}"</span>
</code></pre>
<h2>remove space in filename</h2>
<p><code>prename</code> is a perl rewrite of rename. On archlinux its called
<code>perl-rename</code> and needs to be installed. Its prebuilt on debian:</p>
<pre><code class="language-bash"><span class="hl kwc">find</span> . <span class="hl kwb">-name</span> <span class="hl sng">'*.mkv'</span> <span class="hl kwb">-exec</span> prename <span class="hl sng">'s/ +/_/g'</span> <span class="hl opt">{} +</span>
</code></pre>
<h2>Create empty files at minimum depth 4</h2>
<pre><code class="language-bash"><span class="hl kwc">find</span> . <span class="hl kwb">-mindepth</span> <span class="hl num">4</span> <span class="hl kwb">-type</span> d <span class="hl kwb">-exec</span> <span class="hl kwc">touch</span> <span class="hl opt">{}/</span>state.json \<span class="hl opt">;</span>
</code></pre>
<h2>create a securized user</h2>
<pre><code class="language-bash">useradd <span class="hl kwb">-m -G</span> users <span class="hl kwb">-s</span> <span class="hl opt">/</span>bin<span class="hl opt">/</span>bash <span class="hl opt"><</span>the user<span class="hl opt">></span>
</code></pre>
<h2>run command for a securized user</h2>
<pre><code class="language-bash">sudo <span class="hl kwb">-Hu</span> <span class="hl opt"><</span>the user<span class="hl opt">> <</span>the <span class="hl kwb">command</span><span class="hl opt">></span>
</code></pre>
<h2>Git Prompt</h2>
<p>Shows details about git project when in the folder</p>
<pre><code>git clone https://github.com/magicmonty/bash-git-prompt.git ~/.bash-git-prompt --depth=1
# add to bashrc
if [ -f "$HOME/.bash-git-prompt/gitprompt.sh" ]; then
GIT_PROMPT_ONLY_IN_REPO=1
source $HOME/.bash-git-prompt/gitprompt.sh
fi
</code></pre>
<h2>Improved ls</h2>
<pre><code>alias ll="exa -l --git --all --octal-permissions --sort modified --time-style long-iso --group --color=auto --color-scale"
</code></pre>
<h2>Edit a file without vi</h2>
<pre><code>cat - > /path/to/file
# paste the content
# then ctrl+D
</code></pre>
<h2>Create a tunnel</h2>
<pre><code class="language-bash"><span class="hl kwc">ssh</span> <span class="hl kwb">-fNTML</span> <span class="hl num">9432</span><span class="hl opt">:</span>localhost<span class="hl opt">:</span><span class="hl num">5432</span> sshusername@you<span class="hl kwb">-server</span>.com
Then<span class="hl opt">,</span> just launch psql<span class="hl opt">,</span> connecting to port <span class="hl num">9432</span> <span class="hl kwc">at</span> localhost<span class="hl opt">:</span>
psql <span class="hl kwb">-h</span> localhost <span class="hl kwb">-p</span> <span class="hl num">9432</span> <span class="hl kwb">-U</span> <span class="hl opt"><</span>username<span class="hl opt">> <</span>dbname<span class="hl opt">></span>
</code></pre>
<h2>Get file magic number</h2>
<pre><code class="language-bash"><span class="hl num">18</span><span class="hl opt">:</span><span class="hl num">42</span> $ <span class="hl kwc">file</span> <span class="hl kwb">-i</span> assets<span class="hl opt">/</span>images<span class="hl opt">/</span>puml.<span class="hl opt">*</span>
assets<span class="hl opt">/</span>images<span class="hl opt">/</span>puml.png<span class="hl opt">:</span> image<span class="hl opt">/</span>png<span class="hl opt">;</span> charset<span class="hl opt">=</span>binary
assets<span class="hl opt">/</span>images<span class="hl opt">/</span>puml.webp<span class="hl opt">:</span> image<span class="hl opt">/</span>webp<span class="hl opt">;</span> charset<span class="hl opt">=</span>binary
</code></pre>
<h2>Get last created folders in current path</h2>
<pre><code class="language-bash"><span class="hl kwc">find</span> . <span class="hl kwb">-ctime -1</span><span class="hl opt">|</span>cut <span class="hl kwb">-d</span><span class="hl sng">'/'</span> <span class="hl kwb">-f2</span><span class="hl opt">|</span><span class="hl kwc">sort</span> <span class="hl kwb">-u</span>
</code></pre>
<h2>Tar</h2>
<h3>Tar from find</h3>
<pre><code class="language-bash"><span class="hl kwc">tar</span> czf <span class="hl opt"><</span><span class="hl kwc">file</span><span class="hl opt">></span>.tgz <span class="hl sng">`find . -name '*.json' -maxdepth 1`</span>
</code></pre>
<h3>Tar from find</h3>
<pre><code class="language-bash"><span class="hl kwc">tar</span> czf <span class="hl opt"><</span><span class="hl kwc">file</span><span class="hl opt">></span>.tgz <span class="hl opt"><</span>folder<span class="hl opt">></span> <span class="hl kwb">--exclude</span><span class="hl opt">=<</span>path<span class="hl opt">/</span>to<span class="hl opt">/*></span>
</code></pre>
<h2>Find exec multi args command</h2>
<ol>
<li>write a script in the path</li>
<li>make it executable</li>
<li>find . -exec ./script {} ;</li>
</ol>
<h2>Make sure files created in folder inherit group ownership</h2>
<pre><code class="language-bash"><span class="hl kwc">chmod</span> g<span class="hl opt">+</span>s <span class="hl opt"><</span>path<span class="hl opt">/</span>to<span class="hl opt">/</span>dif<span class="hl opt">></span>
</code></pre>
<h2>Rsync in place of scp</h2>
<p>Rsync handle files with spaces in it. Just add
<code>s</code>=<code>--protect-args</code> and quote the string</p>
<pre><code>rsync -avs user@host:"path with spaces" /tmp/
</code></pre>
<h2>Update apt keys</h2>
<pre><code>for PUBKEY in $(apt-get update 2>&1 | grep NO_PUBKEY | awk '{print $NF}')
do
wget -q "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x${PUBKEY}" -O - | sed -n '/BEGIN/,/END/p' | apt-key add - 2>/dev/null
done
</code></pre>
<h2>Restore deleted file</h2>
<pre><code>grep -a -C 2000 "a-string-present-in-the-file" /dev/sda2 | tee recovered_data
</code></pre>
<p><a href="https://studios.ptilouk.net/superflu-riteurnz/blog/2023-02-09_recovery.html">source</a></p>
<h2>Kill a background task</h2>
<pre><code class="language-sh">ctrl<span class="hl opt">+</span>z <span class="hl slc"># put current task in background</span>
<span class="hl kwb">jobs -p</span><span class="hl opt">|</span><span class="hl kwc">xargs</span> <span class="hl kwb">kill</span>
</code></pre>
<h2>Config ssh for git</h2>
<p>To avoid beeing able to connect with ssh</p>
<pre><code class="language-bash"><span class="hl slc"># create a blind user</span>
sudo adduser <span class="hl kwb">--disabled-password --shell</span> <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>git<span class="hl kwb">-shell --gecos</span> <span class="hl sng">"User"</span> git
<span class="hl slc"># allow local user to access the folder</span>
sudo usermod <span class="hl kwb">-a -G</span> git natus
<span class="hl slc"># init a git repo accessible by remote</span>
git init <span class="hl kwb">--bare</span>
<span class="hl slc"># and in the authorized key, prefix with</span>
no<span class="hl kwb">-port-forwarding</span><span class="hl opt">,</span>no<span class="hl kwb">-X11-forwarding</span><span class="hl opt">,</span>no<span class="hl kwb">-agent-forwarding</span><span class="hl opt">,</span>no<span class="hl kwb">-pty</span> <span class="hl kwc">ssh</span><span class="hl kwb">-rsa</span> <span class="hl opt"><</span><span class="hl kwa">then</span><span class="hl kwb">-rsa-public-key</span><span class="hl opt">></span>
</code></pre>
<p>also secure ssh:</p>
<pre><code class="language-bash"><span class="hl kwa">if</span> <span class="hl opt">[</span> <span class="hl kwb">-z</span> <span class="hl sng">"</span><span class="hl ipl">$(command grep "^ssh-users" '/etc/group')</span><span class="hl sng">"</span> <span class="hl opt">];</span> <span class="hl kwa">then</span>
<span class="hl kwb">command</span> addgroup <span class="hl kwb">--system</span> <span class="hl sng">"ssh-users"</span>
<span class="hl kwa">fi</span>
sudo usermod <span class="hl kwb">-a -G</span> <span class="hl kwc">ssh</span><span class="hl kwb">-users</span> natus
<span class="hl kwb">command echo</span> <span class="hl sng">'</span>
<span class="hl sng"># Limit access to users of group ssh-users</span>
<span class="hl sng">AllowGroups ssh-users'</span> <span class="hl opt">>> /</span>etc<span class="hl opt">/</span>ssh<span class="hl opt">/</span>sshd_config
</code></pre>
<h2>Benchmark command line</h2>
<pre><code>hyperfine <command>
</code></pre>
<h2>Know if using x11 or wayland</h2>
<pre><code>loginctl show-session $(loginctl | grep $(whoami) | awk '{print $1}') -p Type
</code></pre>
<h2>Get last modified files in any folder</h2>
<pre><code class="language-bash"><span class="hl slc"># this excludes hidden files</span>
<span class="hl kwc">find</span> <span class="hl opt">*</span> <span class="hl kwb">-type</span> f <span class="hl kwb">-printf</span> <span class="hl sng">"%T+ %p</span><span class="hl esc">\n</span><span class="hl sng">"</span> <span class="hl opt">|</span> <span class="hl kwc">sort</span>
</code></pre>
<h2>Start a xsession on a server</h2>
<h3>from startx</h3>
<p>As root, this will start a session for a given user:</p>
<pre><code># systemd-run --property PAMName=login \
--property User=<user-name> \
--property StandardInput=tty \
--property TTYPath=/dev/tty8 \
sh -c 'chvt 8 && startx /usr/bin/xterm -- :1'
</code></pre>
<p>Then the user is able to start programs with X support:</p>
<pre><code>DISPLAY=:1 /usr/bin/chromium "https://blog.parisni.com"
</code></pre>
<h3>from vnc server</h3>
<p><a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-debian-11">Follow vnc install</a>, in particular:</p>
<ul>
<li>start the vnc server (it will start a display, let's say :2)</li>
<li>create the systemd unit to enable it at startup</li>
<li>open firewall to access locally to the vnc server</li>
<li>Now you can also access thought remina with vnc plugin connection</li>
<li>also cron can run display apps</li>
</ul>
<pre><code>DISPLAY=:2 /usr/bin/chromium "https://blog.parisni.com"
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Bash — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-08-12 00:11</time> </div>https://blog.parisni.com/wiki/dockerDocker2023-12-11T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">containers</span></p>
<h1>Docker</h1>
<h2>Default ip addresses</h2>
<p>Set default docker address:</p>
<pre><code>{
"log-driver": "journald",
"bip": "172.26.0.1/16",
"default-address-pools": [
{
"base": "172.254.0.0/16",
"size": 24
}
]
}
</code></pre>
<h2>Show ip of all running containers</h2>
<pre><code>docker inspect $(docker ps -q ) --format='{{ printf "%-50s" .Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}'
</code></pre>
<h2>check security</h2>
<ul>
<li>Use <code>grype</code></li>
</ul>
<h2>Multiarch build</h2>
<pre><code class="language-shell">docker buildx create --name multiarch --driver docker-container --use
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Docker — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-11 21:09</time> </div>https://blog.parisni.com/wiki/torrentTransmission2021-08-14T13:38:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">torrent</span>
<span class="w3-tag w3-red tag">p2p</span></p>
<h1>Transmission</h1>
<h2>Magnet link</h2>
<ol>
<li>get the info-hash of the file</li>
<li>transform it into magnet link at <a href="http://romanr.info/magnet.html">info-hash to magnet link</a></li>
<li>run <code>transmission-cli -w <destination/folder> "<magnet link content>"</code></li>
</ol>
<h2>Active website</h2>
<ul>
<li><a href="https://www2.yggtorrent.si/">torrent411</a> accessible from tor-browser</li>
<li><a href="https://thepiratebay.org">the pirate bay</a> accessible from tor-browser</li>
</ul>
<h2>Personal Web interface</h2>
<p><a href="https://github.com/linuxserver/docker-transmission">see the docker application</a></p>
<h2>Blocklist</h2>
<ul>
<li><a href="https://www.iblocklist.com/list?list=fr">up-to-date blacklist</a></li>
<li>
<pre><code># edit settings.json
"blocklist-enabled": true,
"blocklist-url": "http://list.iblocklist.com/?list=fr&fileformat=p2p&archiveformat=gz",
transmission-remote host:port -n admin:password --blocklist-update
</code></pre>
</li>
<li>there is no need for updating since docker-transmission takes care of it <a href="https://github.com/linuxserver/docker-transmission/issues/14">see the issue</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Transmission — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-14 13:38</time> </div>https://blog.parisni.com/wiki/vimVim2023-12-12T22:43:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span>
<span class="w3-tag w3-red tag">editor</span></p>
<h1>Vim</h1>
<h2>Install neovim</h2>
<pre><code class="language-bash"><span class="hl slc"># this to be run at each release</span>
wget <span class="hl kwb">--quiet</span> https<span class="hl opt">://</span>github.com<span class="hl opt">/</span>neovim<span class="hl opt">/</span>neovim<span class="hl opt">/</span>releases<span class="hl opt">/</span>download<span class="hl opt">/</span>stable<span class="hl opt">/</span>nvim.appimage <span class="hl kwb">--output-document</span> nvim
<span class="hl kwc">chmod</span> <span class="hl opt">+</span>x nvim
sudo <span class="hl kwc">mv</span> nvim <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>
sudo <span class="hl kwc">ln</span> <span class="hl kwb">-s</span> <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/{</span>nvim<span class="hl opt">,</span>vim<span class="hl opt">}</span>
sudo update<span class="hl kwb">-alternatives --install</span> <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>editor editor <span class="hl opt">/</span>usr<span class="hl opt">/</span>bin<span class="hl opt">/</span>vim <span class="hl num">1</span>
sudo update<span class="hl kwb">-alternatives --config</span> editor
</code></pre>
<p>Add those into .bashrc:</p>
<pre><code>export EDITOR=/usr/bin/vim
export VISUAL=/usr/bin/vim
</code></pre>
<h2>Install plugins</h2>
<pre><code class="language-bash">curl <span class="hl kwb">-fLo</span> ~<span class="hl opt">/</span>.local<span class="hl opt">/</span>share<span class="hl opt">/</span>nvim<span class="hl opt">/</span>site<span class="hl opt">/</span>autoload<span class="hl opt">/</span>plug.vim <span class="hl kwb">--create-dirs</span> https<span class="hl opt">://</span>raw.githubusercontent.com<span class="hl opt">/</span>junegunn<span class="hl opt">/</span>vim<span class="hl kwb">-plug</span><span class="hl opt">/</span>master<span class="hl opt">/</span>plug.vim
</code></pre>
<pre><code class="language-bash"><span class="hl kwc">cat</span> ~<span class="hl opt">/</span>.config<span class="hl opt">/</span>nvim<span class="hl opt">/</span>init.vim
<span class="hl kwb">set</span> encoding<span class="hl opt">=</span>utf<span class="hl kwb">-8</span>
colo murphy
<span class="hl kwb">set</span> hlsearch
<span class="hl kwb">set</span> nonu
call plug<span class="hl slc">#begin()</span>
Plug <span class="hl sng">'https://github.com/ryvnf/readline.vim.git'</span>
Plug <span class="hl sng">'https://github.com/tpope/vim-surround.git'</span>
call plug<span class="hl slc">#end()</span>
<span class="hl sng">" in order to keep terminal cursor after vim quit</span>
<span class="hl sng">set guicursor=n-v-c:block,i-ci-ve:ver25,r-cr:hor20,o:hor50</span>
<span class="hl sng">\,a:blinkwait700-blinkoff400-blinkon250-Cursor/lCursor</span>
<span class="hl sng">\,sm:block-blinkwait175-blinkoff150-blinkon175</span>
</code></pre>
<p>Plugin install/update:</p>
<pre><code>:PlugInstall
:UpdateRemotePlugins
</code></pre>
<h2>Regexp</h2>
<h3>Gready / non Gready</h3>
<p>In vim, <code>.\{-}</code> is equivalent to <code>.*?</code> for non gready operators:</p>
<pre><code>:%s/jim.\{-}//g
</code></pre>
<h2>Delete blank lines</h2>
<pre><code>:g/^$/d
</code></pre>
<h2>No format on paste</h2>
<pre><code>:set paste
# Note: it breaks vim-readline
</code></pre>
<h2>No numbering</h2>
<pre><code>:set nonu
</code></pre>
<h2>Vim Surround</h2>
<p>Add surround:</p>
<ol>
<li>Visual a word, part of text</li>
<li><code>S"</code> -> "surround"</li>
<li><code>S(</code> -> (surround)</li>
</ol>
<p>Replace surround:</p>
<ol>
<li>On a "word" press <code>cs"'</code> to change to 'word'</li>
<li>On a "word" press <code>cs")</code> to change to (word)</li>
</ol>
<p>Delete surround:</p>
<ol>
<li>On a "word" press <code>ds"</code> to change to word</li>
</ol>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Vim — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-12 22:43</time> </div>https://blog.parisni.com/wiki/audioAudio2022-05-07T23:16:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">audio</span></p>
<h1>Audio</h1>
<h2>Télécharger une emission sur France Culture</h2>
<p>Testé avec chromium:</p>
<ol>
<li>écouter l'émission</li>
<li>clic droit sur le bandeau de lecture</li>
<li>inspecter l'élément</li>
<li>rechercher dans le source “mp3”</li>
<li>récupérer le lien pour télécharger</li>
</ol>
<h1>Transformer du webm en ogg</h1>
<pre><code class="language-bash">ffmpeg <span class="hl kwb">-i</span> <span class="hl kwc">file</span>.webm <span class="hl kwb">-vn -ab</span> <span class="hl num">128</span>k <span class="hl kwb">-ar</span> <span class="hl num">44100</span> <span class="hl kwb">-y</span> <span class="hl kwc">file</span>.ogg
</code></pre>
<h1>Show id3 tags</h1>
<pre><code class="language-bash">ffprobe <span class="hl kwb">-hide_banner -show_entries</span> format_tags<span class="hl opt">=</span>artist<span class="hl opt">,</span>title <span class="hl num">02</span>\ Snake\ Hips.mp3 <span class="hl opt">|</span> <span class="hl kwc">sed</span> <span class="hl sng">'s/\[[^]]*\]//g'</span> <span class="hl opt">|</span> cut <span class="hl kwb">-d</span> <span class="hl sng">"="</span> <span class="hl kwb">-f</span> <span class="hl num">2</span>
<span class="hl opt">[</span>mp3 @ <span class="hl num">0x5640e4f4e320</span><span class="hl opt">]</span> Estimating duration from bitrate<span class="hl opt">,</span> this may be inaccurate
Input <span class="hl slc">#0, mp3, from '02 Snake Hips.mp3':</span>
Metadata<span class="hl opt">:</span>
title <span class="hl opt">:</span> Snake Hips
album <span class="hl opt">:</span> Far Out Son Of Lung And The Ra
comment <span class="hl opt">:</span>
publisher <span class="hl opt">:</span> Astralwerks
track <span class="hl opt">:</span> <span class="hl num">2</span>
compilation <span class="hl opt">:</span> <span class="hl num">1</span>
album_artist <span class="hl opt">:</span> The Future Sound of London
genre <span class="hl opt">:</span> Electronica
composer <span class="hl opt">:</span> The Future Sound of London
artist <span class="hl opt">:</span> The Future Sound Of London
<span class="hl kwc">date</span> <span class="hl opt">:</span> <span class="hl num">1994</span>
Duration<span class="hl opt">:</span> <span class="hl num">00</span><span class="hl opt">:</span><span class="hl num">08</span><span class="hl opt">:</span><span class="hl num">33.82</span><span class="hl opt">,</span> start<span class="hl opt">:</span> <span class="hl num">0.000000</span><span class="hl opt">,</span> bitrate<span class="hl opt">:</span> <span class="hl num">320</span> kb<span class="hl opt">/</span>s
Stream <span class="hl slc">#0:0: Audio: mp3, 44100 Hz, stereo, s16p, 320 kb/s</span>
Snake Hips
The Future Sound Of London
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Audio — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-05-07 23:16</time> </div>https://blog.parisni.com/wiki/Sso.mdSso2023-07-05T00:52:00+00:00<div id="generated-toc"> </div><hr> <h1>Sso</h1>
<h2>opened connect</h2>
<ul>
<li><a href="https://openid.net/developers/how-connect-works/">Documentation</a></li>
</ul>
<h2>authelia</h2>
<ul>
<li><a href="https://www.authelia.com/overview/prologue/architecture/">works as a proxy w/ a session cookie</a></li>
<li><a href="https://www.authelia.com/configuration/first-factor/ldap/">Ldap for first factor provider</a></li>
<li><a href="https://www.authelia.com/integration/deployment/docker/#standalone-example">demo example</a></li>
<li><a href="https://www.authelia.com/integration/openid-connect/introduction/">support openid Beta</a></li>
</ul>
<h2>keycloak</h2>
<ul>
<li><a href="https://www.keycloak.org/getting-started/getting-started-docker">Demo example</a></li>
<li><a href="https://www.keycloak.org/">OpenID Connect, OAuth 2.0 and SAML 2.0</a></li>
<li><a href="https://www.keycloak.org/docs/latest/server_admin/index.html#_ldap">ldap</a></li>
<li></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Sso — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-07-05 00:52</time> </div>https://blog.parisni.com/wiki/springSpring2023-06-15T22:59:00+00:00<div id="generated-toc"> </div><hr> <p>#java</p>
<h1>Spring</h1>
<h2>spring equivalent</h2>
<ul>
<li>JSR-330 : standard specification</li>
</ul>
<p>Good spring resources to begin with:</p>
<ul>
<li>Spring Start Here by Laurențiu Spilcă (Manning, 2021)</li>
<li>Spring in Action , sixth edition, by Craig Walls (Manning, 2022).</li>
<li>Spring Boot: Up & Running by Mark Heckler (O’Reilly, 2021)</li>
</ul>
<h2>spring initializer</h2>
<ul>
<li><a href="https://start.spring.io">Create application link</a></li>
</ul>
<h2>spring and containers</h2>
<ul>
<li>integration of Spring Boot with Cloud Native Buildpacks ( https://buildpacks.io ),</li>
</ul>
<h2>spring data</h2>
<p>Support of nosql databases</p>
<h2>spring boot</h2>
<p>An set of opinioniated spring starter applications such web or other</p>
<h2>spring batch</h2>
<p>For batche jobs</p>
<h2>spring cloud</h2>
<ul>
<li>https://docs.spring.io/spring-cloud-stream/docs/4.0.3/reference/html/spring-cloud-stream.html#spring-cloud-stream-reference</li>
<li>it supports the <a href="https://docs.confluent.io/platform/current/streams/developer-guide/processor-api.html">processor api</a></li>
</ul>
<h2>spring annotations</h2>
<p>These annotations are called stereotype because they are part of a package named org.springframework.stereotype. This package groups together all annotations used to define beans. These annotations are also relevant to the role of a bean.</p>
<ul>
<li>@ComponentScan defines the packages that Spring should scan for annotations for bean definitions.</li>
<li>@ImportResource</li>
<li>@Component : @Service is a specialization of @Component, which indicates that the annotated class is providing a business service to other layers within the application.</li>
<li>@Service is used to define a service bean, which is a more complex functional bean that provides services that other beans may require</li>
<li>@Repository is used to define a bean that is used to retrieve/save data from/to a database, etc.</li>
<li>@Autowired to tell the Spring IoC container to look for a bean of that type and use it as an argument when calling that method. The stereotype annotations can have as a parameter the name of the resulting bean.</li>
<li>@Qualifier : done by name. This is done using the @Qualifier annotation, together with the @Autowired annotation, and</li>
<li>@Ressource</li>
<li>@Value</li>
<li>@Scope("prototype") used to define the bean scope, when the desired scope is other than singleton.</li>
<li>@Lazy annotation is used at the class level to declare beans that will be instantiated the first time they are accessed providing the name of the bean being injected as an argument for it.</li>
<li>@Configuration. The @Configuration annotation is used to inform Spring that this is a Java-based configuration file.</li>
<li>@Bean(initMethod = "init") will contain methods annotated with @Bean definitions that represent the bean declarations. The @Bean annotation is used to declare a Spring bean and the DI requirements</li>
<li>@DependsOn This annotation tells Spring that a certain bean depends on some other beans, so Spring will make sure that those beans are instantiated first.</li>
<li>@Profile (-Dspring.profiles.active="highschool")</li>
</ul>
<p>For testing purpose:</p>
<ul>
<li>@RunWith(SpringJUnit4ClassRunner.class)</li>
<li>@ContextConfiguration(classes={KindergartenConfig.class, HighschoolConfig.class})</li>
<li>@ActiveProfiles("kindergarten")</li>
</ul>
<h2>dependency injection</h2>
<ul>
<li>Constructor injection</li>
<li>method injection</li>
<li>field injection</li>
</ul>
<h2>spel</h2>
<ul>
<li>Spring Expression Language (SpEL)</li>
</ul>
<h2>spring docker</h2>
<ul>
<li>better to layer the jar content to save space, network and also image build time</li>
<li><code>java -Djarmode=layertools -jar catalog-service.jar extract</code></li>
<li><a href="http://mng.bz/M0xB">Cloud Native Buildpacks with Emily Casey</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Spring — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-06-15 22:59</time> </div>https://blog.parisni.com/wiki/ubuntuUbuntu2024-02-10T20:01:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">os</span>
<span class="w3-tag w3-red tag">linux</span></p>
<h1>Ubuntu</h1>
<h2>Apt</h2>
<h3>List installed packages</h3>
<pre><code>dpkg -l
</code></pre>
<h3>List installed files</h3>
<pre><code>dpkg -L <package-name>
</code></pre>
<h3>Search a package</h3>
<pre><code>apt-cache search <pattern>
</code></pre>
<h2>Login</h2>
<h3>Prevent suspend when laptop closed</h3>
<pre><code class="language-bash"><span class="hl kwc">cat</span> <span class="hl opt">/</span>etc<span class="hl opt">/</span>systemd<span class="hl opt">/</span>logind.conf
<span class="hl opt">[</span>Login<span class="hl opt">]</span>
HandleLidSwitch<span class="hl opt">=</span>ignore
</code></pre>
<h2>Installer</h2>
<h3>Snap</h3>
<p>It is well integrated with ubuntu. Sometimes flatpak is more up-to-date but
snap has the advantage to provide a command in the path.</p>
<p>Also, snap does not access to <code>/tmp</code> folder, which can be problematic.</p>
<h3>Flatpak</h3>
<p>Flatpak is as good as snap, with a different software coverage, but it does not
provide a command in the path: instead <code>flatpak run <tool></code>.</p>
<p>Also the sofware are installed by package name <code>flatpak install flathub org.gimp.GIMP</code></p>
<h2>Dark Mode</h2>
<h3>Gtk natives</h3>
<pre><code class="language-bash"><span class="hl kwc">cat</span> .gtkrc<span class="hl kwb">-2</span>.0
gtk<span class="hl kwb">-application-prefer-dark-theme</span><span class="hl opt">=</span>true
gtk<span class="hl kwb">-key-theme-name</span> <span class="hl opt">=</span> <span class="hl sng">"Emacs"</span>
gtk<span class="hl kwb">-icon-theme-name</span> <span class="hl opt">=</span> <span class="hl sng">"Yaru-dark"</span>
</code></pre>
<pre><code class="language-bash"><span class="hl num">18</span><span class="hl opt">:</span><span class="hl num">46</span> $ <span class="hl kwc">cat</span> .config<span class="hl opt">/</span>gtk<span class="hl kwb">-3</span>.0<span class="hl opt">/</span>settings.ini
<span class="hl opt">[</span>Settings<span class="hl opt">]</span>
gtk<span class="hl kwb">-application-prefer-dark-theme</span><span class="hl opt">=</span>true
gtk<span class="hl kwb">-key-theme-name</span> <span class="hl opt">=</span> emacs
gtk<span class="hl kwb">-icon-theme-name</span> <span class="hl opt">=</span> <span class="hl sng">"Yaru-dark"</span>
gtk<span class="hl kwb">-can-change-accels</span> <span class="hl opt">=</span> <span class="hl num">1</span>
</code></pre>
<h3>Gtk flatpak</h3>
<p><a href="https://itsfoss.com/flatpak-app-apply-theme/">see flatpak overrides</a></p>
<pre><code class="language-bash">sudo apt <span class="hl kwc">install</span> ubuntu<span class="hl kwb">-session</span> yaru<span class="hl kwb">-theme-gnome-shell</span> yaru<span class="hl kwb">-theme-gtk</span> yaru<span class="hl kwb">-theme-icon</span> yaru<span class="hl kwb">-theme-sound</span>
<span class="hl kwc">ln</span> <span class="hl kwb">-s</span> <span class="hl opt">/</span>usr<span class="hl opt">/</span>share<span class="hl opt">/</span>themes ~<span class="hl opt">/</span>.themes
sudo flatpak override <span class="hl kwb">--filesystem</span><span class="hl opt">=</span><span class="hl kwd">$HOME</span><span class="hl opt">/</span>.themes
sudo flatpak override <span class="hl kwb">--env</span><span class="hl opt">=</span>GTK_THEME<span class="hl opt">=</span>Yaru<span class="hl kwb">-dark</span>
</code></pre>
<h2>Limit resource to program</h2>
<pre><code class="language-bash"><span class="hl slc"># 10000 / 200000 = 0.05 == 5% CPU usage cap.</span>
<span class="hl kwb">echo</span> <span class="hl num">10000</span> <span class="hl opt">> /</span>sys<span class="hl opt">/</span>fs<span class="hl opt">/</span>cgroup<span class="hl opt">/</span>cpu<span class="hl opt">/</span>cpulimited<span class="hl opt">/</span>cpu.cfs_quota_us
<span class="hl kwb">echo</span> <span class="hl num">200000</span> <span class="hl opt">> /</span>sys<span class="hl opt">/</span>fs<span class="hl opt">/</span>cgroup<span class="hl opt">/</span>cpu<span class="hl opt">/</span>cpulimited<span class="hl opt">/</span>cpu.cfs_period_us
<span class="hl slc"># bash script</span>
sudo cgexec <span class="hl kwb">-g</span> cpu<span class="hl opt">:</span>cpulimited .<span class="hl opt">/</span>your_program_or_script.sh
<span class="hl slc"># existing program</span>
cgclassify <span class="hl kwb">-g</span> cpu<span class="hl opt">:</span>cpulimited <span class="hl opt"><</span>pidNum<span class="hl opt">></span>
</code></pre>
<p><a href="https://revx0r.com/using-linux-cgroups-to-cap-cpu-usage/">see cgroup</a></p>
<h2>Ignore poweroff button press</h2>
<ol>
<li>Open the file <code>/etc/systemd/logind.conf</code>:</li>
</ol>
<pre><code class="language-bash">sudo vim <span class="hl opt">/</span>etc<span class="hl opt">/</span>systemd<span class="hl opt">/</span>logind.conf
</code></pre>
<ol start="2">
<li>Find the line that starts with <code>HandlePowerKey=</code> and change its value from <code>poweroff</code> to <code>ignore</code>. If the line is commented out, uncomment it and change its value.</li>
</ol>
<pre><code class="language-bash">HandlePowerKey<span class="hl opt">=</span>ignore
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Ubuntu — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2024-02-10 20:01</time> </div>https://blog.parisni.com/wiki/playPlay Framework2020-06-28T23:51:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">scala</span>
<span class="w3-tag w3-red tag">web</span></p>
<h1>Play Framework</h1>
<h2>Json parsing</h2>
<p>play-json allows to create complex tree by nesting case classes.</p>
<pre><code class="language-scala"><span class="hl kwa">import</span> play<span class="hl opt">.</span>api<span class="hl opt">.</span>libs<span class="hl opt">.</span>json<span class="hl opt">.</span>Reads<span class="hl opt">.</span>_
<span class="hl kwa">import</span> play<span class="hl opt">.</span>api<span class="hl opt">.</span>libs<span class="hl opt">.</span>json<span class="hl opt">.</span>_
<span class="hl kwa">import</span> play<span class="hl opt">.</span>api<span class="hl opt">.</span>libs<span class="hl opt">.</span>functional<span class="hl opt">.</span>syntax<span class="hl opt">.</span>_
<span class="hl kwa">case class</span> <span class="hl kwd">Overall</span><span class="hl opt">(</span>overall<span class="hl opt">:</span> Double<span class="hl opt">)</span>
<span class="hl kwa">case class</span> <span class="hl kwd">Data</span><span class="hl opt">(</span>association_score<span class="hl opt">:</span> Overall<span class="hl opt">)</span>
<span class="hl kwa">case class</span> <span class="hl kwd">RootJson</span><span class="hl opt">(</span>from<span class="hl opt">:</span> Int<span class="hl opt">,</span> total<span class="hl opt">:</span> Int<span class="hl opt">,</span> data<span class="hl opt">:</span> Seq<span class="hl opt">[</span>Data<span class="hl opt">])</span>
<span class="hl slc">// this for simple element case class</span>
<span class="hl kwa">implicit val</span> overallReads<span class="hl opt">:</span> Reads<span class="hl opt">[</span>Overall<span class="hl opt">] =</span>
<span class="hl opt">(</span>JsPath \ <span class="hl sng">"overall"</span><span class="hl opt">).</span>read<span class="hl opt">[</span>Double<span class="hl opt">].</span><span class="hl kwd">map</span><span class="hl opt">(</span>Overall<span class="hl opt">)</span>
<span class="hl kwa">implicit val</span> dataReads<span class="hl opt">:</span> Reads<span class="hl opt">[</span>Data<span class="hl opt">] =</span>
<span class="hl opt">(</span>JsPath \ <span class="hl sng">"association_score"</span><span class="hl opt">).</span>read<span class="hl opt">[</span>Overall<span class="hl opt">].</span><span class="hl kwd">map</span><span class="hl opt">(</span>Data<span class="hl opt">)</span>
<span class="hl slc">// this for multiple element case class</span>
<span class="hl kwa">implicit val</span> rootJsonReads<span class="hl opt">:</span> Reads<span class="hl opt">[</span>RootJson<span class="hl opt">] = (</span>
<span class="hl opt">(</span>JsPath \ <span class="hl sng">"from"</span><span class="hl opt">).</span>read<span class="hl opt">[</span>Int<span class="hl opt">]</span> and
<span class="hl opt">(</span>JsPath \ <span class="hl sng">"total"</span><span class="hl opt">).</span>read<span class="hl opt">[</span>Int<span class="hl opt">]</span> and
<span class="hl opt">(</span>JsPath \ <span class="hl sng">"data"</span><span class="hl opt">).</span>read<span class="hl opt">[</span>Seq<span class="hl opt">[</span>Data<span class="hl opt">]]</span>
<span class="hl opt">)(</span>RootJson<span class="hl opt">.</span>apply _<span class="hl opt">)</span>
</code></pre>
<p>then this json can be parsed:</p>
<pre><code class="language-scala"> <span class="hl kwa">val</span> json<span class="hl opt">:</span> String <span class="hl opt">=</span> <span class="hl sng">"""</span>
<span class="hl sng"> {</span>
<span class="hl sng"> "total" : 1000,</span>
<span class="hl sng"> "from" : 10,</span>
<span class="hl sng"> "data" : [ {</span>
<span class="hl sng"> "association_score" : {</span>
<span class="hl sng"> "overall" : 40.0</span>
<span class="hl sng"> }</span>
<span class="hl sng"> } ]</span>
<span class="hl sng"> }</span>
<span class="hl sng"> """</span>
Json<span class="hl opt">.</span><span class="hl kwd">parse</span><span class="hl opt">(</span>json<span class="hl opt">).</span>validate<span class="hl opt">[</span>RootJson<span class="hl opt">]</span> <span class="hl kwa">match</span> <span class="hl opt">{</span>
<span class="hl kwa">case</span> <span class="hl kwd">JsSuccess</span><span class="hl opt">(</span>rootJson<span class="hl opt">,</span> _<span class="hl opt">) => {</span>
<span class="hl kwa">val</span> _<span class="hl opt">:</span> RootJson <span class="hl opt">=</span> rootJson
rootJson
<span class="hl opt">}</span>
<span class="hl kwa">case</span> e<span class="hl opt">:</span> JsError <span class="hl opt">=> {</span>
<span class="hl slc">// error handling flow</span>
<span class="hl kwa">throw new</span> Exception
<span class="hl opt">}</span>
<span class="hl opt">}</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Play Framework — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-28 23:51</time> </div>https://blog.parisni.com/wiki/asahi-linuxAsahi linux2023-12-10T00:11:00+00:00<div id="generated-toc"> </div><hr> <p>#linux
#macos</p>
<h1>Asahi linux</h1>
<h2>Installation</h2>
<h2>Screen problems</h2>
<ul>
<li>asahi linux <a href="https://social.treehouse.systems/@marcan/110354541574112092">does not really support Xorg</a></li>
<li>wayland has some drawbacks such as :
<ul>
<li>no support for sharing screen</li>
<li>problem to identify pop-up / notification in the <code>swaymsg -t get_tree</code> -> use mako</li>
</ul>
</li>
</ul>
<h2>Keyboard troubles</h2>
<ul>
<li><a href="https://michal.kosmulski.org/computing/articles/custom-keyboard-layouts-xkb.html">xkb setup</a></li>
<li>apparently there is <a href="https://foosel.net/til/how-to-remap-keys-under-linux-and-wayland/">also a keyd tool for that</a></li>
<li>f-keys needs fn. however it is possible to change that behavior. <a href="https://askubuntu.com/a/7553">see details</a>
<pre><code class="language-shell"># this works but disapears at reboot
echo 2 | sudo tee /sys/module/hid_apple/parameters/fnmode
# This needs to rebuild the boot
echo options hid_apple fnmode=2 | sudo tee -a /etc/modprobe.d/hid_apple.conf
sudo update-initramfs -u -k all
sudo reboot # optional
</code></pre>
</li>
</ul>
<h2>Browser troubles</h2>
<ul>
<li>
<p>snap chromium does not work with slack</p>
</li>
<li>
<p>snap chromium only share tabs during meet</p>
</li>
<li>
<p>there is no arm64 chromium without snap on ubuntu (mint only support amd64)</p>
</li>
<li>
<p>snap chromium does not support password store</p>
</li>
<li>
<p>~~firefox does not share tabs during meet~~</p>
</li>
<li>
<p>~~firefox 120 seg fault with arm64 (snap and deb)~~ fixed with 120.0.1</p>
</li>
<li>
<p>firefox-esr (115) deb works fine</p>
</li>
<li>
<p>firefox shortkey can be removed (see wiki web browser)</p>
</li>
<li>
<p>firefox supports natively wayland</p>
</li>
<li>
<p>vivaldi allows to share tabs</p>
</li>
<li>
<p>vivaldi provides deb arm64</p>
</li>
<li>
<p>vivaldi supports slack</p>
</li>
<li>
<p>vivaldi supports password store</p>
</li>
<li>
<p>vivaldi natively supports wayland via <a href="https://forum.vivaldi.net/topic/24429/wayland-support">configs</a> but I got tabs freeze and focus lost from tabs so I don't use it</p>
</li>
<li>
<p>vivaldi lags i don't know why</p>
</li>
</ul>
<h2>Intellij troubles</h2>
<ul>
<li>
<p>there is no <code>insert</code> key, then copy via <a href="https://stackoverflow.com/a/27928635">vim can be enabled</a></p>
</li>
<li>
<p>idea works with xwayland, but support for <a href="https://youtrack.jetbrains.com/issue/CWM-7461">wayland is coming</a></p>
</li>
<li>
<p>markdown fails with error <a href="https://youtrack.jetbrains.com/issue/JBR-4721/JCEF-fails-with-cannot-allocate-memory-in-static-TLS-block">but there is a workaround</a></p>
</li>
<li>
<p>idea does not honour xkb_variant nor xmodmap</p>
</li>
<li>
<p>however it honours xkb_options</p>
</li>
</ul>
<p>so I did:</p>
<ul>
<li>default layout : <code>ctrl_l - alt_l - super_l - space - super_r - alt_r</code></li>
<li><code>caps:escape,ctrl:swap_lwin_lctl,lv3:ralt_alt,lv3:lalt_switch,altwin:alt_super_win</code></li>
<li>resulting layout : <code>iso_lv3_shift - ctrl_l - space - alt_r - alt_r</code></li>
<li>used keys : <code>iso_lv3_shift - ctrl_l - space - alt_r</code></li>
</ul>
<p>Thats a good compromise with my other pc which:</p>
<ul>
<li>
<p>default layout: <code>ctrl_l - fn - super_l - alt_l - space - iso_lv3_shift - menu - ctrl_r</code></p>
</li>
<li>
<p><code>caps:swapescape,ctrl:swap_lalt_lctl,lv3:ralt_alt,lv3:lwin_switch,ctrl:menu_rctrl</code></p>
</li>
<li>
<p>resulting layout <code>super_l - fn - iso_lv3_shift - ctrl_l - space - alt_r - menu</code></p>
</li>
<li>
<p>used keys <code>iso_lv3_shift - ctrl_l - space - alt_r</code></p>
</li>
<li>
<p>a long standing bug with tmux <a href="https://youtrack.jetbrains.com/issue/IDEA-229458">can be fixed</a> by seting <code>:set status-position top</code></p>
</li>
</ul>
<h2>Trouble with gtk</h2>
<ul>
<li>gtk configs are not honoured</li>
<li>the only way to apply dark theme was
to <a href="https://www.reddit.com/r/Ubuntu/comments/17hiiop/after_update_to_2310_all_gnome_apps_default_to/">use this workaround</a>
by <code>ADW_DEBUG_COLOR_SCHEME=prefer-dark to /etc/environment</code></li>
<li>emacs key binding or gtk.css does not work either</li>
<li>apparently related <a href="https://www.reddit.com/r/Fedora/comments/17z64ys/comment/k9xnx7t/?utm_source=share&utm_medium=web2x&context=3">to recent gnome</a></li>
</ul>
<h3>gnome-text-editor</h3>
<pre><code class="language-shell">Gtk-WARNING **: 20:28:39.175: Theme parser error: gtk.css:1:1-13: Unknown @ rule
Gtk-WARNING **: 20:28:39.175: Theme parser error: gtk.css:43:3-19: No property named "gtk-key-bindings"
Adwaita-WARNING **: 20:12:48.637: Using GtkSettings:gtk-application-prefer-dark-theme with libadwaita is unsupported. Please use AdwStyleManager:color-scheme instead.
</code></pre>
<ul>
<li>it uses gtk4 (no way to set key binding ?) -> remove <code>.config/gtk-4.0/gtk.css</code></li>
<li>dark theme works by setting <code>ADW_DEBUG_COLOR_SCHEME</code></li>
</ul>
<h3>file-roller, evince, firefox ...</h3>
<pre><code>Using GtkSettings:gtk-application-prefer-dark-theme together with HdyStyleManager is unsupported. Please use HdyStyleManager:color-scheme instead
</code></pre>
<ul>
<li>apparently gtk3+</li>
<li><code>gsettings set org.gnome.desktop.interface color-scheme default</code> fixes the dark theme</li>
<li>copy <code>/usr/share/themes/Emacs/gtk-3.0/gtk-keys.css</code> into <code>.config/gtk-3.0/gtk.css</code> then emacs key binding works</li>
</ul>
<h2>Sound</h2>
<ul>
<li>only jack sound works</li>
<li>install <code>pavucontrol</code> to allow configuring both head/mic sound</li>
<li>install <code>pactl</code> to allow up/down on head sound from keyboard</li>
</ul>
<h2>Screen sharing</h2>
<ul>
<li>worked with gnome but not in sway initially (both wayland)</li>
<li>installed <code>sudo apt install xdg-desktop-portal-wlr</code> (not sure useful?)</li>
<li><a href="https://github.com/emersion/xdg-desktop-portal-wlr/wiki/%22It-doesn%27t-work%22-Troubleshooting-Checklist">got some working advices here</a></li>
<li>exec the below in sway:</li>
</ul>
<pre><code class="language-shell"># cat bin/screen-sharing
dbus-update-activation-environment --systemd WAYLAND_DISPLAY XDG_CURRENT_DESKTOP=sway
systemctl --user stop pipewire pipewire-pulse.socket xdg-desktop-portal xdg-desktop-portal-wlr wireplumber pipewire.socket
systemctl --user start pipewire pipewire-pulse.socket xdg-desktop-portal xdg-desktop-portal-wlr wireplumber pipewire.socket
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Asahi linux — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-10 00:11</time> </div>https://blog.parisni.com/wiki/graphvizGraphviz2021-11-25T15:41:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">versionning</span></p>
<h1>Graphviz</h1>
<h2>Subgraph</h2>
<p>One can use <code>subgraph</code> toadd group delimited with rectangles.</p>
<p><img src="/images/graphviz.webp" alt="The result"></p>
<pre><code class="language-dot"><span class="hl kwa">strict digraph</span> {
<span class="hl kwb">rankdir</span>=<span class="hl sng">"TB"</span>
scheduler_1 [<span class="hl kwb">shape</span>=<span class="hl sng">"record"</span> <span class="hl kwb">style</span>=<span class="hl sng">"filled"</span> <span class="hl kwb">fillcolor</span>=<span class="hl sng">"#3366ff"</span><span class="hl opt">,</span> <span class="hl kwb">label</span>=<span class="hl sng">"{ scheduler | LocalExecutor }"</span> ]
dags_mount_1 [<span class="hl kwb">shape</span>=<span class="hl sng">"record"</span> <span class="hl kwb">style</span>=<span class="hl sng">"filled"</span> <span class="hl kwb">fillcolor</span>=<span class="hl sng">"#ffff66"</span><span class="hl opt">,</span> <span class="hl kwb">label</span>=<span class="hl sng">"{ Dags | docker mount }"</span>]
postgres [<span class="hl kwb">shape</span>=<span class="hl sng">"record"</span> <span class="hl kwb">style</span>=<span class="hl sng">"filled"</span> <span class="hl kwb">fillcolor</span>=<span class="hl sng">"green"</span><span class="hl opt">,</span> <span class="hl kwb">label</span>=<span class="hl sng">"{Database | Postgres}"</span>]
webserver_1 [<span class="hl kwb">shape</span>=<span class="hl sng">"record"</span> <span class="hl kwb">style</span>=<span class="hl sng">"filled"</span> <span class="hl kwb">fillcolor</span>=<span class="hl sng">"#ff00ff"</span><span class="hl opt">,</span> <span class="hl kwb">label</span>=<span class="hl sng">"{ webserver }"</span> ]
<span class="hl kwa">subgraph</span> cluster_1 {
<span class="hl kwb">label</span>=<span class="hl sng">"Server"</span><span class="hl opt">;</span>
dags_mount_1
<span class="hl kwa">subgraph</span> cluster_2 {
<span class="hl kwb">label</span>=<span class="hl sng">"Docker"</span><span class="hl opt">;</span>
<span class="hl kwb">style</span>=<span class="hl kwc">filled</span><span class="hl opt">;</span>
<span class="hl kwb">color</span>=lightgrey<span class="hl opt">;</span>
webserver_1
scheduler_1
}
}
postgres
postgres <span class="hl opt">-></span> scheduler_1 [<span class="hl kwb">dir</span>=<span class="hl kwc">none</span>]
webserver_1 <span class="hl opt">-></span> postgres [<span class="hl kwb">dir</span>=<span class="hl kwc">none</span>]
scheduler_1 <span class="hl opt">-></span> dags_mount_1 [<span class="hl kwb">dir</span>=<span class="hl kwc">none</span>]
}
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Graphviz — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-25 15:41</time> </div>https://blog.parisni.com/wiki/cdCD2021-10-24T14:51:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">media</span></p>
<h1>CD</h1>
<h2>Ripper</h2>
<p>ABCDE let you grab/edit the id3 tags within the <code>$EDITOR</code>.</p>
<pre><code>abcde -o mp3
abcde -1 -o flac # whole cd in a single file
</code></pre>
<p>It can be configured to not bother on broken cd:</p>
<pre><code class="language-bash"><span class="hl slc"># ~/.abcde.conf</span>
CDPARANOIAOPTS<span class="hl opt">=</span><span class="hl sng">"--disable-paranoia"</span>
</code></pre>
<h2>Mount a mp3 cdrom</h2>
<pre><code>lsblk # to know the cdrom devince
mount /dev/<devince-id> /mnt/cdrom
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=CD — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-10-24 14:51</time> </div>https://blog.parisni.com/wiki/sweethome3dSweet Home 3D2022-01-23T12:12:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">3D</span>
<span class="w3-tag w3-red tag">2D</span></p>
<h1>Sweet Home 3D</h1>
<h2>Elevate floor</h2>
<ol>
<li>create a wall</li>
<li>change its dimension: height, thickness</li>
</ol>
<p>Then the wall is turned into elevated floor</p>
<h2>Make your own symbols</h2>
<p>This script generates a s3sh archive to be imported as a furniture library. Just add your images as png.</p>
<pre><code>#!/bin/bash
ff=PluginFurnitureCatalog.properties
echo "# PluginFurnitureCatalog.properties ( "$(date +"%d-%m-%Y") ")" > $ff
echo "# Copyright (c) 2007-2017 Emmanuel PUYBARET / eTeks <info@eteks.com>." >> $ff
echo -en '\n\n' >> $ff
echo "# Created by script Create2DSymbols.sh ( "$(date +"%d-%m-%Y") ")" >> $ff
echo "# To update, add new symbols in png-format in plan folder and run ./CreateSymbols.sh" >> $ff
echo "# Requires ImageMagick and zip" >> $ff
echo "# Attention:" >> $ff
echo "# The script resizes (to 256x256, but keeping proportions) the original images and copy the result" >> $ff
echo "# in catalog folder (if ImageMagick is installed)." >> $ff
echo "# The resulting library is zipped and moved (overwriting files with the same name) to ~/.eteks/sweethome3d/furniture/" >> $ff
echo "# Upon restart of SweetHome3D, the library is available. Alternatively, import it from ~/.eteks/sweethome3d/furniture/" >> $ff
echo "# The file name without file type suffix will be used as the symbol name: " >> $ff
echo "# The filename should have the synthax: SymbolName.Information.png" >> $ff
echo "# Alphabetical order following the file names." >> $ff
echo -en '\n\n' >> $ff
echo "description= Symbols Catalog for use in 2D plan" >> $ff
echo "version=1.0" >> $ff
echo "license=Free Art License 1.3" >> $ff
echo "provider=www.sweethome3d.com" >> $ff
echo -en '\n\n' >> $ff
cnt=0
#cn="catalog/"
#rm ListOfFile.txt
OIFS=$IFS
IFS=$'\n'
#cd catalog/
for nn in `ls plan | sort -V`
do
{
cp plan/$nn catalog/
cn=catalog/$nn
pn=plan/$nn
echo $nn
{ width=`convert "$pn" -format '%w' info:`
depth=`convert "$pn" -format '%h' info:`
convert -trim "$cn" "$cn"
convert "$cn" -resize 256x256 -background transparent -gravity center -extent 256x256 "$cn"
} || {
echo "Cannot find convert, a member of the ImageMagick suite. The images are not cropped and resized to 256x256."
echo -en '\n\n' >> $ff
echo "convert, a member of the ImageMagick suite, was unavailable when this file was created." >> $ff
echo "The images are thus not cropped and resized to 256x256." >> $ff
echo -en '\n\n' >> $ff
}
cnt=`expr $cnt + 1`
fname=${pn%.*} # remove everything after the last dot (e.q. file suffix but leave everything else)
fname=${fname##*/} #remove path -> filename
name=${fname%.*} #symbol name
echo id#$cnt=2DSymbol#$name >> $ff
echo name#$cnt=$name >> $ff
echo information#$cnt= >> $ff
echo category#$cnt="2DSymbols" >> $ff
echo icon#$cnt=/$cn >> $ff
echo planIcon#$cnt=/$pn >> $ff
echo model#$cnt=/invisibleCube.obj >> $ff
echo width#$cnt=$width >> $ff
echo depth#$cnt=$depth >> $ff
echo height#$cnt=10.0 >> $ff
echo elevation#$cnt=50.0 >> $ff
echo movable#$cnt=true >> $ff
echo doorOrWindow#$cnt=false >> $ff
echo creator#$cnt=dorin >> $ff
echo -en '\n\n' >> $ff
}
done
IFS=$OIFS
mylib=${PWD##*/}.sh3f
zip -r $mylib .
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Sweet Home 3D — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-01-23 12:12</time> </div>https://blog.parisni.com/wiki/KafkaKafka2023-08-25T20:34:00+00:00<div id="generated-toc"> </div><hr> <h1>Kafka</h1>
<h2>Kafka streaming</h2>
<ul>
<li><a href="https://kafka.apache.org/documentation/streams/developer-guide/">Documentation</a></li>
<li><a href="https://docs.confluent.io/platform/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config">State store aka ktable</a> can be tuned</li>
<li><a href="https://artem.krylysov.com/blog/2023/04/19/how-rocksdb-works/">how rocksdb works</a></li>
</ul>
<h3>About rocksdb</h3>
<ul>
<li>forked as CockroachDB / yugabyte</li>
<li>forked from leveldb</li>
<li>maintened by meta</li>
<li>written in c++, biding in java, rust, go</li>
<li>embeddable database</li>
<li>key value pair, more exacly bytes array pairs</li>
<li>ge/put/merge/delete also data iterator for scan</li>
<li>LSM-Tree
<ul>
<li>memtable to get the incoming data until 64MO limit reach</li>
<li>WAL (Write Ahead Logs) to avoid data lost during crash</li>
<li>SST (Static Sorted Table) can be compressed (snappy, zstd, gzip...)</li>
<li>Offset Index map (to allow binary search in compressed file)</li>
<li>optional Bloom filter : to make lookup keys don't exist faster</li>
</ul>
</li>
<li>space/read amplification: each flush to disk adds files to be merged</li>
<li>compaction creates a new level</li>
<li>compaction can cascade accros levels</li>
<li>k-way merge strategy (to merge multiple level of files)</li>
<li>reads traverse the whole level (memtables -> all level 0 -> target levels N -> bloom/index + read the block)</li>
<li>merge operation (=read-modified-write): thread safe read+put+DELETE</li>
</ul>
<h2>Kafka connect</h2>
<h3>secrets</h3>
<p>There is a mechanism to provide/implement secrets in kafka-connect:</p>
<ul>
<li><a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations">theory</a></li>
<li><a href="https://rmoff.net/2019/05/24/putting-kafka-connect-passwords-in-a-separate-file-/-externalising-secrets/">practice</a></li>
</ul>
<p>We could easily provide a EnvConfigProvider which look into linux env variables for values.</p>
<h2>Kafka auth</h2>
<ul>
<li><a href="https://developer.ibm.com/tutorials/kafka-authn-authz/">how to secure a kafka cluster</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Kafka — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-08-25 20:34</time> </div>https://blog.parisni.com/wiki/etesyncEteSync2022-02-06T12:01:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">caldav</span>
<span class="w3-tag w3-red tag">security</span></p>
<h1>EteSync</h1>
<h2>etebase</h2>
<p>This is a e2e backend written in python. A postgres database stores the django roles, permissions and groups informations.</p>
<p>The users data is stored encrypted in a folder tree. Data depends on the client application : card, tasks, calendar, notes.</p>
<p>Users can change their passwords, however they cannot loose them or never access again to their data.</p>
<p>There is no support for group to share the collections. A workaround consists of creating a dedicated user which can share its collections with a set of users.</p>
<h2>Admin UI</h2>
<p>Etebase has it's own admin ui, <code><etebase-domain>/admin</code> which allows to create
users and provide them a password. Later the user can change its password in
the etesync UI.</p>
<h2>WebUi</h2>
<p>It decrypts the data provided by the server and allows a set of feature such :</p>
<ul>
<li>show/edit calendar</li>
<li>show/edit tasks</li>
<li>show/edit cards</li>
<li>those are added to collections</li>
<li>collections can be shared with user either rw or ro</li>
</ul>
<h2>etesync-android</h2>
<p>You install it on your device and fill your credentials. It decrypts locally and interfaces with the android tools (calendar, contacts, tasks...)</p>
<h2>etesync-dav</h2>
<p>It aims at interface users computer thunderbirds or outlook. It is a binary based on radicale and decrypts the server data locally.</p>
<p>I tested it with vdirsyncer with success.</p>
<h2>etesync-notes</h2>
<p>Both a mobile and web app allows editing markdown notes.</p>
<h2>Clients</h2>
<h3>Android</h3>
<p>All right</p>
<h3>Iphone</h3>
<p>Etesync allows to share the collection with icloud. Some troubleshoot :</p>
<ul>
<li>contact pictures are likely lost in the sync process</li>
<li>the etesync "My tasks" collection makes the sync process fail, so remove it.</li>
</ul>
<h2>Resources</h2>
<ul>
<li><a href="https://notes.etesync.com">the notes webui</a></li>
<li><a href="https://pim.etesync.com">the caldav webui</a></li>
<li><a href="https://github.com/victor-rds/docker-etebase">the etebase docker</a></li>
<li><a href="https://github.com/etesync/etesync-web/pull/189/files">the caldav webui docker</a></li>
<li><a href="https://github.com/etesync/etesync-notes/tree/master/docker">the notes webui docker</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=EteSync — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-02-06 12:01</time> </div>https://blog.parisni.com/wiki/gitGit2024-01-14T18:33:00+00:00<div id="generated-toc"> </div><hr> <p>#versionning
#selfhosting</p>
<h1>Git</h1>
<h2>Branches</h2>
<h3>Récupérer un fichier de puis une autre branche <a href="https://stackoverflow.com/a/21718540">source</a></h3>
<pre><code class="language-bash"><span class="hl slc"># cherche depuis la branche distante server le fichier bookmark</span>
git checkout origin<span class="hl opt">/</span>server <span class="hl kwb">--</span> .w<span class="hl num">3</span>m<span class="hl opt">/</span>bookmark.html
</code></pre>
<h3>Revenir a un commit</h3>
<p>Si j'ai fait fausse route et que j'ai poussé les modif en ligne, je
peux revenir en arriere et faire de ce commit la version actuelle.</p>
<pre><code class="language-bash"><span class="hl slc"># retourner au commit f359</span>
git revert <span class="hl kwb">--no-commit</span> f359b96a1e1e680cf76ea06094e112b92b936b5e..HEAD
git commit
</code></pre>
<h3>Explorer les données d'un commit</h3>
<pre><code class="language-bash"><span class="hl slc"># retourner au commit f359</span>
git checkout f359b96a1e1e680cf76ea06094e112b92b936b5e
<span class="hl slc"># puis par exemple pour retourner dans la branche master</span>
git checkout master
</code></pre>
<h3>cesser de tracker un dossier</h3>
<p>Ajouter au .gitignore le dossier. Puis lancer la commande:</p>
<pre><code class="language-bash">git <span class="hl kwc">rm</span> <span class="hl kwb">-r --cached</span> <span class="hl opt"><</span>le<span class="hl kwb">-dossier</span><span class="hl opt">></span>
</code></pre>
<p>cela ne supprimera pas le dossier</p>
<h3>Annuler les modifs d'un fichier</h3>
<pre><code class="language-bash">git checkout <span class="hl kwb">--</span> fichier<span class="hl kwb">-a-annuler-les-modif</span>
</code></pre>
<h3>Revenir à un certain commit pour un certain fichier</h3>
<pre><code class="language-bash">git checkout <span class="hl opt"><</span>commit id<span class="hl opt">></span> <span class="hl kwb">--</span> fichier<span class="hl kwb">-a-annuler-les-modif</span>
</code></pre>
<h3>Rebase la branche master</h3>
<pre><code class="language-bash">git checkout master
git fetch <span class="hl kwb">--all</span> <span class="hl slc"># cf https://www.reddit.com/r/git/comments/8j70vv/should_i_fetch_before_pull/</span>
git pull
git checkout branche<span class="hl kwb">-de-travail</span>
git rebase master <span class="hl slc"># evite d'avoir les commit de master par dessus son history; ils seront mis avant les modifs de la branche</span>
</code></pre>
<h3>show history of file</h3>
<pre><code class="language-bash">git log <span class="hl kwb">-p</span> filename
<span class="hl slc"># OR</span>
git show <span class="hl kwb">-2</span> filename <span class="hl slc"># means number of history commit</span>
</code></pre>
<h3>git rename branch</h3>
<p>this will rename the branch</p>
<pre><code class="language-bash">git branch <span class="hl kwb">-m</span> gitlab<span class="hl opt">/</span>dev gitlab<span class="hl kwb">-dev</span>
</code></pre>
<h3>git push remote</h3>
<pre><code class="language-bash"><span class="hl slc"># this allows to push on an other remote, on the associated branch</span>
git push <span class="hl opt"><</span>other<span class="hl kwb">-remote</span><span class="hl opt">> <</span>branch<span class="hl kwb">-to-push</span><span class="hl opt">></span>
<span class="hl slc"># this allows to push on a specific remote branch git push <other-remote> <branch-to-push>:<other-branch-to-push-on></span>
</code></pre>
<h3>Supprimer des fichier dans un depot</h3>
<pre><code class="language-bash"><span class="hl slc"># permet de supprimer le fichier spark-meta/target/spark-meta-0.0.1.SNAPSHOT-shaded.jar</span>
<span class="hl slc"># de tout l´historique</span>
git filter<span class="hl kwb">-branch --force --index-filter</span> <span class="hl sng">'git rm --cached --ignore-unmatch spark-meta/target/spark-meta-0.0.1-SNAPSHOT-shaded.jar'</span> <span class="hl kwb">--prune-empty --tag-name-filter</span> <span class="hl kwc">cat</span> <span class="hl kwb">-- --all</span>
</code></pre>
<h3>Renommer un vieux commit</h3>
<p>En utilisant <em>magit</em>:</p>
<ol>
<li>désactiver les éventuels git-hooks (dans .git/hooks/pre-commit)</li>
<li><em>rebase</em> (<code>r</code>)</li>
<li>choisir <em>--preserve-merge</em> (<code>-p</code>)</li>
<li>choisir <em>rename commit</em> ( <code>w</code> )</li>
<li>choisir <code>C-c C-c</code> et renommer le commit</li>
<li>éventuellement, gérer les conflits, puis <em>continuer le rebase</em></li>
</ol>
<h3>Rebase et priorité sur des conflits</h3>
<p>Par defaut, le rebase choisit les fichiers dans la branche
<strong>yours</strong>. Il va écraser des éventuelles modifications locales. On
peut changer ce comportement par defaut:</p>
<pre><code>git rebase -X ours <branch>
</code></pre>
<h3>Récupérer une branche remote en local</h3>
<pre><code class="language-bash">git checkout <span class="hl kwb">--track</span> <span class="hl opt"><</span>remote<span class="hl opt">>/<</span>branch<span class="hl opt">></span>
</code></pre>
<h3>Remove submodule</h3>
<pre><code class="language-bash">submodule<span class="hl opt">=</span><span class="hl sng">"folder/submodule"</span>
git <span class="hl kwc">rm</span> <span class="hl sng">"</span><span class="hl ipl">$submodule</span><span class="hl sng">"</span>
<span class="hl kwc">rm</span> <span class="hl kwb">-rf</span> <span class="hl sng">".git/modules/</span><span class="hl ipl">$submodule</span><span class="hl sng">"</span>
git config <span class="hl kwb">-f</span> <span class="hl sng">".git/config"</span> <span class="hl kwb">--remove-section</span> <span class="hl sng">"submodule.</span><span class="hl ipl">$submodule</span><span class="hl sng">"</span> <span class="hl num">2</span><span class="hl opt">> /</span>dev<span class="hl opt">/</span>null
<span class="hl slc"># Commit the change</span>
git commit <span class="hl kwb">-m</span> <span class="hl sng">"Remove submodule</span> <span class="hl ipl">$submodule</span><span class="hl sng">"</span>
<span class="hl slc"># Restart the state</span>
git submodule deinit <span class="hl kwb">-f</span> .
git submodule update <span class="hl kwb">--init</span>
</code></pre>
<h3>Undo local commit</h3>
<pre><code>git reset HEAD~1 # where 1 is the number of commit to resume
</code></pre>
<h3>Undo several local commit and remove local files</h3>
<pre><code>git reset --hard <commit-id>
</code></pre>
<h2>bash prompt</h2>
<ol>
<li>
<p>Add https://raw.githubusercontent.com/git/git/master/contrib/completion/git-prompt.sh to ~/.git-prompt.sh</p>
</li>
<li>
<p>add this to .bashrc</p>
</li>
</ol>
<pre><code class="language-bash"><span class="hl kwa">if</span> <span class="hl opt">[</span> <span class="hl kwb">-f</span> ~<span class="hl opt">/</span>.git<span class="hl kwb">-prompt</span>.sh <span class="hl opt">];</span> <span class="hl kwa">then</span>
. ~<span class="hl opt">/</span>.git<span class="hl kwb">-prompt</span>.sh
GIT_PS1_SHOWDIRTYSTATE<span class="hl opt">=</span>true
GIT_PS1_SHOWCOLORHINTS<span class="hl opt">=</span>true
GIT_PS1_UNTRACKEDFILES<span class="hl opt">=</span>true
PS1<span class="hl opt">=</span><span class="hl sng">'</span><span class="hl ipl">${debian_chroot:+($debian_chroot)}</span><span class="hl sng">\[</span><span class="hl esc">\033</span><span class="hl sng">[01;32m\]\u@\h\[</span><span class="hl esc">\033</span><span class="hl sng">[00m\]:\[</span><span class="hl esc">\033</span><span class="hl sng">[01;34m\]\W\[</span><span class="hl esc">\033</span><span class="hl sng">[00m\]</span><span class="hl ipl">$(__git_ps1 " (%s)</span><span class="hl sng">")\$ '</span>
<span class="hl kwa">fi</span>
</code></pre>
<h2>Show difference between two tags</h2>
<pre><code>git diff tag1 tag2
# for only one file
git diff tag1 tag2 -- some/file/name
</code></pre>
<h2>Delete all branch but master</h2>
<pre><code>git branch | grep -v "master" | xargs git branch -D
</code></pre>
<h2>Delete a tag</h2>
<pre><code class="language-bash"><span class="hl slc"># local</span>
git tag <span class="hl kwb">--delete</span> <span class="hl num">1.6.3</span>
<span class="hl slc"># remote</span>
git push <span class="hl kwb">--delete</span> origin <span class="hl opt"><</span>tag<span class="hl opt">></span>
</code></pre>
<h2>Restore a file or path from a commit</h2>
<pre><code class="language-bash">git restore <span class="hl kwb">--source</span><span class="hl opt">=<</span>commit<span class="hl kwb">-id</span><span class="hl opt">></span> path<span class="hl opt">/</span>to<span class="hl opt">/</span>thing
</code></pre>
<h2>Blame omit commits</h2>
<pre><code class="language-bash"><span class="hl slc"># to get the full sha1</span>
git blame <span class="hl kwb">--line-porcelain</span> path<span class="hl opt">/</span>to<span class="hl opt">/</span><span class="hl kwc">file</span>
<span class="hl slc"># then omit it by copy/paste sha1 into file</span>
git blame <span class="hl kwb">--ignore-revs-file</span><span class="hl opt">=</span>.git<span class="hl kwb">-blame-ignore-revs</span> path<span class="hl opt">/</span>to<span class="hl opt">/</span><span class="hl kwc">file</span>
</code></pre>
<h2>Restore a stash file</h2>
<pre><code>git diff stash^! -- <filename> | git apply
</code></pre>
<h2>git server</h2>
<ul>
<li><a href="https://wiki.archlinux.org/title/Cgit">cgit</a></li>
<li><a href="https://www.howtoforge.com/tutorial/ubuntu-git-server-installation/">http git server</a></li>
<li><a href="https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server">Git ssh server</a></li>
</ul>
<h2>Configs</h2>
<h3>Don't consider file mode (chmod)</h3>
<pre><code>git config core.fileMode false
</code></pre>
<h2>Undo all changes including new files</h2>
<pre><code class="language-bash">git clean <span class="hl kwb">-fd</span>
</code></pre>
<h2>diff de fichiers</h2>
<ul>
<li><a href="https://jbuget.fr/posts/techniques-pour-fusionner-deux-versions-d-un-projet-shopify-grace-a-git/">Reference</a></li>
</ul>
<p>Below won't need a git folder</p>
<pre><code># eq to Linux diff
git diff --no-index --color-words <pathA> <pathB>
# eq to git status
git diff --no-index --name-status <pathA> <pathB>
</code></pre>
<h2>avoid merge conflicts</h2>
<ul>
<li><a href="https://jbuget.fr/posts/comment-eviter-les-conflits-dans-git/">Rules</a></li>
</ul>
<h2>resources</h2>
<ul>
<li><a href="https://doc.norang.ca/git-howto.html">progit</a></li>
<li></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Git — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2024-01-14 18:33</time> </div>https://blog.parisni.com/wiki/matrixMatrix2024-03-27T12:04:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">conversation</span></p>
<h1>Matrix</h1>
<h2>Install your own server</h2>
<p><a href="https://wiki.archlinux.org/index.php/Matrix">matrix server</a></p>
<p><a href="https://matrix.org/docs/guides/#installing-synapse">documentation</a></p>
<p><a href="https://www.natrius.eu/dokuwiki/doku.php?id=digital:server:matrixsynapse">ubuntu detail</a></p>
<h2>Reset a user password</h2>
<p><a href="https://paritoshbh.me/blog/reset-user-password-synapse-matrix-homeserver">this can be done in the database</a></p>
<h2>Connect to IRC libera channel</h2>
<p>To connect to <code>#channel</code> just join <code>#channel:libera.chat</code></p>
<h2>Use the admin api</h2>
<p>One has to be admin <code>update users set admin=1 where name = '....';</code></p>
<ul>
<li><a href="https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html">the api spec</a></li>
<li><a href="https://webapps.stackexchange.com/a/138497">The token can be found there</a></li>
</ul>
<h2>Ban a user</h2>
<pre><code>curl -X POST -H "Authorization: Bearer <MY_TOKEN>" -H "Content-Type: application/json" -d '{"erase": false}' "https://<matrix-server>/_synapse/admin/v1/deactivate/@<user>:<matrix-server>"
</code></pre>
<h2>Upgrade a room</h2>
<p>This will make the current room read-only, and create a new room <a href="https://github.com/matrix-org/matrix-doc/blob/main/proposals/1501-room-version-upgrades.md">see
rfc</a>.</p>
<p>Simply type this as a message (you will be asked to invite every participant) :</p>
<pre><code>/upgraderoom 7
</code></pre>
<p>Run an api request :</p>
<pre><code class="language-bash">curl <span class="hl kwb">-H</span> <span class="hl sng">'Authorization: Bearer <token-access>'</span> <span class="hl kwb">-H</span> <span class="hl sng">"Content-Type: application/json"</span> <span class="hl kwb">-X</span> POST https<span class="hl opt">://</span>matrix.interhop.org<span class="hl opt">/</span>_matrix<span class="hl opt">/</span>client<span class="hl opt">/</span>r<span class="hl num">0</span><span class="hl opt">/</span>rooms<span class="hl opt">/<</span>room<span class="hl kwb">-id-url-encoded</span><span class="hl opt">>/</span>upgrade <span class="hl kwb">-d</span> <span class="hl sng">'{"new_version": "6"}'</span>
</code></pre>
<ul>
<li>token-access: In element > All settings > Help about > Access Token (bottom page)</li>
<li>room-id: In element > Room Info</li>
</ul>
<h2>resources</h2>
<ul>
<li><a href="https://www.freie-messenger.de/en/systemvergleich/xmpp-matrix/">Comparison xmpp</a></li>
</ul>
<h2>SSO</h2>
<ul>
<li><a href="https://github.com/matrix-org/synapse/blob/develop/docs/sso_mapping_providers.md">User can create account from SSO providers</a></li>
<li><a href="https://matrix-org.github.io/synapse/latest/openid.html#keycloak">it supports openid connect through keycloak</a></li>
<li><a href="https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/b225d47ddb5918238cea11510be830bfba675439/docs/configuring-playbook-synapse.md?plain=1#L53-L79">ansible docker openid support</a></li>
<li><a href="https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/b225d47ddb5918238cea11510be830bfba675439/roles/custom/matrix-nginx-proxy/defaults/main.yml#L370">also there is likely something about proxy for openid</a></li>
</ul>
<h2>performances</h2>
<ul>
<li><a href="https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md">Sync v3</a></li>
<li><a href="https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-sliding-sync-proxy.md">ansible docker installé docs</a></li>
<li><a href="https://github.com/matrix-org/sliding-sync">implem</a></li>
<li><a href="https://gitlab.com/famedly/conduit">conduit rust</a></li>
<li><a href="https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-conduit.md">Ansible docker conduit</a></li>
<li><a href="https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-dendrite.md">ansible docker dendrite</a></li>
<li><a href="https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456">ansible docker workers</a></li>
<li><a href="https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-synapse.md#load-balancing-with-workers">ansible docker workers doc</a></li>
</ul>
<p>I guess keeping synapse using workers and sync v3 is a better choice than using beta homeservers with leas features and commiters.</p>
<h2>creating a bot</h2>
<ul>
<li><a href="https://simple-matrix-bot-lib.readthedocs.io/en/latest/manual.html">Simplebot</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Matrix — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2024-03-27 12:04</time> </div>https://blog.parisni.com/wiki/mavenMaven2023-06-23T20:16:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">java</span>
<span class="w3-tag w3-red tag">scala</span></p>
<h1>Maven</h1>
<h2>Download artefacts in dedicated repository</h2>
<p>Some software (such spark) need to have jars in local. Maven is a nice
way to synchronize nexus and a given repository.</p>
<p>This maven plugin allow to export the dependencies into the
<code>outputDirectory</code>. Moreover it can strip the version.</p>
<pre><code class="language-xml"> <span class="hl kwa"><profile></span>
<span class="hl kwa"><id></span>default<span class="hl kwa"></id></span>
<span class="hl kwa"><activation></span>
<span class="hl kwa"><activeByDefault></span>true<span class="hl kwa"></activeByDefault></span>
<span class="hl kwa"></activation></span>
<span class="hl kwa"><build></span>
<span class="hl kwa"><pluginManagement></span>
<span class="hl kwa"><plugins></span>
<span class="hl kwa"><plugin></span>
<span class="hl kwa"><groupId></span>org.apache.maven.plugins<span class="hl kwa"></groupId></span>
<span class="hl kwa"><artifactId></span>maven-dependency-plugin<span class="hl kwa"></artifactId></span>
<span class="hl kwa"><version></span><span class="hl num">3.1.2</span><span class="hl kwa"></version></span>
<span class="hl kwa"><executions></span>
<span class="hl kwa"><execution></span>
<span class="hl kwa"><id></span>get-package<span class="hl kwa"></id></span>
<span class="hl kwa"><phase></span>package<span class="hl kwa"></phase></span>
<span class="hl kwa"><goals></span>
<span class="hl kwa"><goal></span>copy-dependencies<span class="hl kwa"></goal></span>
<span class="hl kwa"></goals></span>
<span class="hl kwa"><configuration></span>
<span class="hl kwa"><outputDirectory></span>${project.build.directory}/dependencies/<span class="hl kwa"></outputDirectory></span>
<span class="hl kwa"><stripVersion></span>true<span class="hl kwa"></stripVersion></span>
<span class="hl kwa"></configuration></span>
<span class="hl kwa"></execution></span>
<span class="hl kwa"></executions></span>
<span class="hl kwa"></plugin></span>
<span class="hl kwa"></plugins></span>
<span class="hl kwa"></pluginManagement></span>
<span class="hl kwa"></build></span>
<span class="hl kwa"></profile></span>
</code></pre>
<p>It is then possible to specify the lastest version of a given
dependency and produce a striped version accessible from a given
folder.</p>
<pre><code class="language-xml"> <span class="hl kwa"><properties></span>
<span class="hl kwa"><spark-etl</span>.version<span class="hl kwa">></span>[<span class="hl num">1.0.10</span>,)<span class="hl kwa"></spark-etl</span>.version<span class="hl kwa">></span>
<span class="hl kwa"><omop-spark</span>.version<span class="hl kwa">></span>[<span class="hl num">1.0.1</span>,)<span class="hl kwa"></omop-spark</span>.version<span class="hl kwa">></span>
<span class="hl kwa"></properties></span>
<span class="hl kwa"><dependencies></span>
<span class="hl kwa"><dependency></span>
<span class="hl kwa"><groupId></span>io.frama.parisni<span class="hl kwa"></groupId></span>
<span class="hl kwa"><artifactId></span>spark-csv<span class="hl kwa"></artifactId></span>
<span class="hl kwa"><version></span>${spark-etl.version}<span class="hl kwa"></version></span>
<span class="hl kwa"><classifier></span>shaded<span class="hl kwa"></classifier></span>
<span class="hl kwa"><exclusions></span>
<span class="hl kwa"><exclusion></span>
<span class="hl kwa"><groupId></span>*<span class="hl kwa"></groupId></span>
<span class="hl kwa"><artifactId></span>*<span class="hl kwa"></artifactId></span>
<span class="hl kwa"></exclusion></span>
<span class="hl kwa"></exclusions></span>
<span class="hl kwa"></dependency></span>
<span class="hl kwa"></dependencies></span>
</code></pre>
<p>The dependency can be produced by using the below command:</p>
<pre><code class="language-bash">mvn <span class="hl kwb">-P</span> <span class="hl sng">"default,snapshot"</span> package \
<span class="hl kwb">-Dmaven</span>.wagon.http.ssl.insecure<span class="hl opt">=</span>true \ <span class="hl slc"># those are used because of ssl problems on my side...</span>
<span class="hl kwb">-Dmaven</span>.wagon.http.ssl.allowall<span class="hl opt">=</span>true \
<span class="hl kwb">-Dmaven</span>.wagon.http.ssl.ignore.validity.dates<span class="hl opt">=</span>true
</code></pre>
<h2>Configure jdks</h2>
<ul>
<li>in <code>~/.m2/toolchains.xml</code></li>
</ul>
<pre><code><toolchains xmlns="http://maven.apache.org/TOOLCHAINS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/TOOLCHAINS/1.1.0 http://maven.apache.org/xsd/toolchains-1.1.0.xsd">
<!-- JDK toolchains -->
<toolchain>
<type>jdk</type>
<provides>
<version>1.8</version>
<vendor>sun</vendor>
</provides>
<configuration>
<jdkHome>/opt/java8</jdkHome>
</configuration>
</toolchain>
<toolchain>
<type>jdk</type>
<provides>
<version>1.7</version>
<vendor>sun</vendor>
</provides>
<configuration>
<jdkHome>/opt/java7</jdkHome>
</configuration>
</toolchain>
</toolchains>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Maven — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-06-23 20:16</time> </div>https://blog.parisni.com/wiki/olapOlap Cubes2023-04-21T22:32:00+00:00<div id="generated-toc"> </div><hr> <h1>Olap Cubes</h1>
<h2>Courses</h2>
<ul>
<li><a href="https://docplayer.fr/957563-Entrepots-de-donnees-systemes-olap-rolap-molap-et-olap-5-1-introduction-aux-systemes.html">online cource about olap cubes</a></li>
</ul>
<p>#big-data</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Olap Cubes — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-21 22:32</time> </div>https://blog.parisni.com/wiki/logitechLogitech Media Server2021-03-14T22:08:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span>
<span class="w3-tag w3-red tag">music</span></p>
<h1>Logitech Media Server</h1>
<h2>Installation</h2>
<p><a href="https://github.com/Logitech/slimserver-platforms/tree/public/8.0/Docker">Logitech provide a docker file</a></p>
<p>The docker compose way is the best:</p>
<pre><code>version: '3'
services:
lms:
container_name: lms
image: lmscommunity/logitechmediaserver
volumes:
- /<somewhere>:/config:rw
- /<somewhere>:/music:ro
- /<somewhere>:/playlist:ro
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
ports:
- 9000:9000/tcp
- 9090:9090/tcp
- 3483:3483/tcp
- 3483:3483/udp
environment:
- PUID=1000
- PGID=1000
restart: always
</code></pre>
<p>Then with systemd it is easy to make it start at system startup.</p>
<pre><code>[Unit]
Description=Logitech Media Server Daemon
Requires=docker.service
After=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/<path to dockerfile>/
ExecStart=/usr/bin/env docker-compose up -d
ExecStop=/usr/bin/env docker-compose down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
</code></pre>
<pre><code>systemctl enable logitech
systemctl start logitech
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
93b29a299406 lmscommunity/logitechmediaserver "start-container" 44 minutes ago Up 42 minutes 0.0.0.0:3483->3483/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:9090->9090/tcp, 0.0.0.0:3483->3483/udp lms
</code></pre>
<h2>Security</h2>
<p>It is possible to add <a href="http://squeezeplayer.de/2011/05/3g-part-3-secure-your-server/">a basic
auth</a>. It
is however a weak protection.</p>
<h2>Firewall</h2>
<p>This will restrict access to LMS ports to local network only.</p>
<pre><code class="language-bash">ufw delete from <span class="hl num">192.168.1.0</span><span class="hl opt">/</span><span class="hl num">16</span> to any app <span class="hl sng">"LMS Full"</span>
</code></pre>
<pre><code class="language-bash"><span class="hl kwc">cat</span> <span class="hl opt">/</span>etc<span class="hl opt">/</span>ufw<span class="hl opt">/</span>applications.d<span class="hl opt">/</span>lms<span class="hl kwb">-webserver</span>
<span class="hl opt">[</span>LMS Full<span class="hl opt">]</span>
title<span class="hl opt">=</span>Logitech Media Player
description<span class="hl opt">=</span>Logitech Media Player
ports<span class="hl opt">=</span><span class="hl num">9000</span><span class="hl opt">,</span><span class="hl num">9090</span><span class="hl opt">,</span><span class="hl num">3483</span><span class="hl opt">/</span>tcp
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Logitech Media Server — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-03-14 22:08</time> </div>https://blog.parisni.com/wiki/ideaGenerate javadoc2020-06-28T14:21:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">editor</span></p>
<h1>Generate javadoc</h1>
<p>You can use the action <code>Fix doc comment</code>. It doesn't have a default
shortcut, but you can assign the Alt+Shift+J shortcut to it in the
Keymap.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Generate javadoc — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-28 14:21</time> </div>https://blog.parisni.com/wiki/pubkeyPubkeys2021-05-01T21:34:00+00:00<div id="generated-toc"> </div><hr> <div id="refman"> <div id="refman-sidebar"> <div id="generated-toc">
</div>
</div> <div id="refman-main">
<p><span class="w3-tag w3-red tag">security</span>
<span class="w3-tag w3-red tag">linux</span></p>
<h1>Pubkeys</h1>
<h2>converting for ssh</h2>
<pre><code>---- BEGIN SSH2 PUBLIC KEY ----
Comment: "rsa-key-20200324"
AAAAB3NzaC1yc2EAAAAQEApWmENmDe+CcHplywBJQAAAiBYbZ8nmSeQA3taleMPR
P8VMt7mdUWhN5d6lAOw9UR/bfqrx1XkkWLVL57OWCjbxrMyWMKZBL2ZAGn8Z+cDO
qlldSEdBStGVUQ4gW2XLQE9gXKfWOVapxwbckcIZuNwwJrQpe+wWwp2L8zGRGO+C
sgzh9FGC1/XMlLkIV3A9uRWnL6qD4aMCDkk+TIGJS5kP9UB0GEHvEU2vhFCoI8u+
xaAvAHd49fzUok1Q//H/Dxp1XMKsIyIw1fGOQqvcmx0iymX9C5gGHEBEw+3EBE0I
tDZxPKRwD9aKu9zRvwkeYGSnXDgMiTCffmV4VcVcAN104n8D0Q==
---- END SSH2 PUBLIC KEY ----
</code></pre>
<p>This will flatten the key:</p>
<pre><code>
ssh-keygen -f theSsh2PubKey -i > key.pub
</code></pre>
<h2>Change passphrase</h2>
<pre><code>ssh-keygen -p -f ~/.ssh/id_rsa
# if pass = empty then no pass is needed
</code></pre>
</div></div><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Pubkeys — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-05-01 21:34</time> </div>https://blog.parisni.com/wiki/vagrantVagrant2022-01-20T10:12:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">virtulization</span>
<span class="w3-tag w3-red tag">security</span></p>
<h1>Vagrant</h1>
<pre><code>vagrant up
vagrant provision
vagrant ssh
vagrant destroy
</code></pre>
<h2>Libvirt</h2>
<p>Kvm is much more efficient than virtual box. It is meant for production worload
with a small resources footprint on the host.</p>
<p>kvm has the <code>virsh</code> command line which has many features.</p>
<p>So far, it has no built-in way to deploy from the command line as vagrant does.</p>
<p>The vagrant-libvirt plugin provides the best of two worlds: a great way to
deploy kvm and a great way to manage kvms.</p>
<h2>References</h2>
<ul>
<li><a href="https://www.vagrantup.com/docs/synced-folders/basic_usage">vagrant documentatin</a></li>
<li><a href="https://github.com/vagrant-libvirt/vagrant-libvirt">vagrant libvirt github</a></li>
<li><a href="https://ostechnix.com/how-to-use-vagrant-with-libvirt-kvm-provider/">use vagrant together with kvm</a></li>
<li><a href="https://computingforgeeks.com/using-vagrant-with-libvirt-on-linux/">use vagrant together with kvm bis</a></li>
<li><a href="https://ostechnix.com/how-to-find-vagrant-machine-ip-address-from-host/">vagrant ip on host</a></li>
<li><a href="https://phoenixnap.com/kb/ubuntu-install-kvm">install kvm ubuntu</a></li>
<li><a href="https://bobcares.com/blog/virsh-command-to-delete-a-vm/">kvm delete vm</a></li>
<li><a href="https://linuxconfig.org/how-to-create-and-manage-kvm-virtual-machines-from-cli">kvm from cli</a></li>
<li><a href="https://www.cyberciti.biz/faq/how-to-create-create-snapshot-in-linux-kvm-vmdomain/">kvm snapshot tutorial</a></li>
<li><a href="https://app.vagrantup.com/generic/boxes/alpine38">alpine vagrant</a></li>
<li><a href="https://medium.com/@gamunu/use-vagrant-with-libvirt-unsupported-boxes-12e719d71e8e">convert boxes to libvirt</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Vagrant — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-01-20 10:12</time> </div>https://blog.parisni.com/wiki/cardavContacts2022-02-06T12:01:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">Contacts</span></p>
<h1>Contacts</h1>
<h2>Radicale</h2>
<p>This is a cardav server to selfhost contacts and tasks. It comes bundled with
mailu.</p>
<h2>Vdirsyncer</h2>
<p>The tool allows to sync a remote cardav to local. This can then be used by
tools such khard, which in turn can be used in aerc.</p>
<h3>Configure</h3>
<pre><code>[general]
status_path = "~/.vdirsyncer/status/"
[pair my_contacts]
a = "my_contacts_local"
b = "my_contacts_remote"
collections = ["from a", "from b"]
[storage my_contacts_local]
type = "filesystem"
path = "~/.contacts/"
fileext = ".vcf"
[storage my_contacts_remote]
type = "carddav"
# We can simplify this URL here as well. In theory it shouldn't matter.
url = "https://<domain>/webdav/<username>"
username = "<username>"
password = "<password>"
</code></pre>
<h3>Sync local and remote</h3>
<pre><code class="language-bash">vdirsyncer discover my_contacts
vdirsyncer sync
</code></pre>
<h2>Khard</h2>
<p>Khard is a cli manager for contacts. It integrates well with aerc mailing client.</p>
<h3>Add a new contact</h3>
<pre><code class="language-bash">khard new <span class="hl kwb">--open-editor</span>
<span class="hl slc"># then add at leat organisation and formatted name within $EDITOR</span>
</code></pre>
<h2>Aerc configuration</h2>
<p><em>~/.config/aerc/aerc.conf</em></p>
<pre><code># Specifies the command to be used to tab-complete email addresses. Any
# occurrence of "%s" in the address-book-cmd will be replaced with what the
# user has typed so far.
address-book-cmd=khard email --parsable --search-in-source-files --remove-first-line %s
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Contacts — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-02-06 12:01</time> </div>https://blog.parisni.com/wiki/luttesNon Violence2023-04-30T14:42:00+00:00<div id="generated-toc"> </div><hr> <h1>Non Violence</h1>
<ul>
<li><a href="https://reporterre.net/Quelle-place-pour-le-black-bloc-dans-les-luttes-ecolos">Non Violence I</a></li>
<li><a href="https://reporterre.net/La-non-violence-une-resistance-molle-qui-ne-provoque-pas-de-changement-profond">Non Violence II</a></li>
<li><a href="https://www.infolibertaire.net/peter-gelderloos-lechec-de-la-non-violence/">Peter Gelderloos</a></li>
<li><a href="https://en.m.wikipedia.org/wiki/Bhagat_Singh">Bhagat Singh</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Non Violence — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-30 14:42</time> </div>https://blog.parisni.com/wiki/FlinkFlink2023-07-14T00:55:00+00:00<div id="generated-toc"> </div><hr> <h1>Flink</h1>
<ul>
<li><a href="https://youtu.be/04pwWlxXL-U">Tuning jvm</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Flink — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-07-14 00:55</time> </div>https://blog.parisni.com/wiki/androidAndroid2023-10-29T10:44:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p>#phone</p>
<h1>Android</h1>
<h2>termux</h2>
<h3>Access to shared folder</h3>
<pre><code class="language-bash">termux<span class="hl kwb">-setup-storage</span>
</code></pre>
<p><a href="https://android.stackexchange.com/a/185949">from stack</a></p>
<h2>pdf/epub reader</h2>
<h3>EbookDroid</h3>
<p>It makes a fantastic job... without trackers or adds</p>
<h2>unlock</h2>
<ul>
<li><a href="https://www.fairphone.com/en/bootloader-unlocking-code-for-fairphone-3/">Doc</a></li>
<li><a href="https://iode.tech/fr/installation/#1611161852971-b5073e27-d67b">iodé</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Android — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-10-29 10:44</time> </div>https://blog.parisni.com/wiki/markdownMarkdown2022-01-04T00:10:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">formatting</span>
<span class="w3-tag w3-red tag">pdf</span></p>
<h1>Markdown</h1>
<h2>Internal link with section</h2>
<h3>Relative links</h3>
<pre><code class="language-markdown">![The link description](file.md#the-section-to-replace-space-with-minus-and-lower-case)
</code></pre>
<h3>Absolute links</h3>
<pre><code class="language-markdown">![The link description](/path/to/the/file.md#the-section-to-replace-space-with-minus-and-lower-case)
</code></pre>
<h3>Clickable images</h3>
<p>This allows to make the image clickable (and allow zooming):</p>
<pre><code>[![Alt description](image-link.png)](image-link.png)
</code></pre>
<h2>Generate papers pdf from markdown</h2>
<p><a href="https://gist.github.com/maxogden/97190db73ac19fc6c1d9beee1a6e4fc8">By using pandoc-citeproc</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Markdown — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-01-04 00:10</time> </div>https://blog.parisni.com/wiki/ufwUFW2023-04-17T22:31:00+00:00<div id="generated-toc"> </div><hr> <h1>UFW</h1>
<h2>Local applications</h2>
<pre><code>ufw default deny incoming
ufw default allow outgoing
ufw allow SSH
ufw limit SSH
ufw allow "WWW Full"
ufw allow from 192.168.202.0/24 to any app "POSTGRESQL"
ufw enable
# show active:
ufw status numbered
Status: active
To Action From
-- ------ ----
[ 1] 22 ALLOW IN Anywhere
[ 2] SSH ALLOW IN Anywhere
[ 3] WWW Full ALLOW IN Anywhere
[ 4] POSTGRESQL ALLOW IN 192.168.202.0/24
</code></pre>
<h2>Docker applications</h2>
<p>By default, a docker container adds rules in iptables to expose, bypassing UFW <a href="https://github.com/chaifeng/ufw-docker#solving-ufw-and-docker-issues">see</a>.</p>
<pre><code># append to /etc/ufw/after.rules
# BEGIN UFW AND DOCKER
*filter
:ufw-user-forward - [0:0]
:ufw-docker-logging-deny - [0:0]
:DOCKER-USER - [0:0]
-A DOCKER-USER -j ufw-user-forward
-A DOCKER-USER -j RETURN -s 10.0.0.0/8
-A DOCKER-USER -j RETURN -s 172.16.0.0/12
-A DOCKER-USER -j RETURN -s 192.168.0.0/16
-A DOCKER-USER -p udp -m udp --sport 53 --dport 1024:65535 -j RETURN
-A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 192.168.0.0/16
-A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 10.0.0.0/8
-A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 172.16.0.0/12
-A DOCKER-USER -j ufw-docker-logging-deny -p udp -m udp --dport 0:32767 -d 192.168.0.0/16
-A DOCKER-USER -j ufw-docker-logging-deny -p udp -m udp --dport 0:32767 -d 10.0.0.0/8
-A DOCKER-USER -j ufw-docker-logging-deny -p udp -m udp --dport 0:32767 -d 172.16.0.0/12
-A DOCKER-USER -j RETURN
-A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
-A ufw-docker-logging-deny -j DROP
COMMIT
# END UFW AND DOCKER
</code></pre>
<p>Now explicitly allow ports (use container port, not host)</p>
<p>For example, a mail service running on docker</p>
<pre><code>ufw route allow proto tcp from any to any port 25
ufw route allow proto tcp from any to any port 587
ufw route allow proto tcp from any to any port 993
</code></pre>
<p>#linux #security</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=UFW — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-17 22:31</time> </div>https://blog.parisni.com/wiki/hadoopHadoop2023-04-23T20:51:00+00:00<div id="generated-toc"> </div><hr> <p>#big-data</p>
<h1>Hadoop</h1>
<ul>
<li><a href="https://blog.naveenpn.com/memory-consumption-of-hadoop-namenode">The namenode consumption</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Hadoop — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-23 20:51</time> </div>https://blog.parisni.com/wiki/nfsNFS2023-06-03T14:34:00+00:00<div id="generated-toc"> </div><hr> <p>#linux
#disk</p>
<h1>NFS</h1>
<h2>Setup a nfs on debian</h2>
<h3>Server side</h3>
<pre><code class="language-sh">sudo apt <span class="hl kwc">install</span> nfs<span class="hl kwb">-kernel-server</span>
sudo systemctl restart nfs<span class="hl kwb">-kernel-server</span>
</code></pre>
<pre><code>cat /etc/exports
/var/nfs/general nfs_client_ip(rw,insecure,all_squash,sync,no_subtree_check)
sudo mkdir /var/nfs/general -p
sudo chown nobody:nogroup /var/nfs/general
</code></pre>
<h3>Client side</h3>
<pre><code>sudo apt install nfs-common
</code></pre>
<pre><code>cat /etc/fstab
nfs_server_ip:/var/nfs/general /media/nfs/general nfs auto,nofail,user,noatime,nolock,intr,tcp,actimeo=1800 0 0
sudo mkdir -p /media/nfs/general
mount -a
</code></pre>
<ul>
<li>user: allows non root user to mount/umount</li>
</ul>
<h2>Nfs optimizations</h2>
<ul>
<li><a href="https://cromwell-intl.com/open-source/performance-tuning/nfs.html">performance tuning</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=NFS — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-06-03 14:34</time> </div>https://blog.parisni.com/wiki/gameGame2022-05-07T23:16:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">game</span></p>
<h1>Game</h1>
<h2>Neogeo</h2>
<ul>
<li><a href="https://www.winkawaks.org/roms/neogeo/">roms download site</a></li>
<li><a href="https://www.planetemu.net/machine/mame">rom download site</a></li>
</ul>
<p>Download <a href="https://ia601507.us.archive.org/view_archive.php?archive=/25/items/mame-0.231-merged/neogeo.7z">the neogeo bios
files</a>
and zip them and put into <code>/usr/local/share/game/mame/roms/</code></p>
<p><a href="https://docs.mamedev.org/usingmame/defaultkeys.html">Key binding</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Game — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-05-07 23:16</time> </div>https://blog.parisni.com/wiki/keyboardKeyboard2023-11-05T16:02:00+00:00<div id="generated-toc"> </div><hr> <p>#device</p>
<h1>Keyboard</h1>
<h2>Layout us variant intl</h2>
<p><a href="https://dry.sailingissues.com/us-international-keyboard-layout.html">all the keys</a></p>
<ul>
<li>é : <code>altgr+e</code></li>
<li>á : <code>altgr+a</code></li>
<li>« : <code>altgr+[</code></li>
<li>» : <code>altgr+]</code></li>
<li>œ : <code>altgr+k</code></li>
<li>æ : <code>altgr+z</code></li>
<li>' : <code>altgr+'</code></li>
<li>ç : <code>altgr+,</code></li>
</ul>
<h2>Accents</h2>
<ul>
<li>à : <code>altgr + ~ + a</code></li>
<li>â : <code>altgr + 6 + a</code></li>
</ul>
<h2>Change the layout</h2>
<p>see <a href="https://man.openbsd.org/xkeyboard-config#Key_to_choose_3rd_level">see xkeyboard config for reference</a></p>
<pre><code class="language-bash">$ <span class="hl kwc">cat</span> <span class="hl opt">/</span>etc<span class="hl opt">/</span>default<span class="hl opt">/</span>keyboard
XKBMODEL<span class="hl opt">=</span><span class="hl sng">"pc105"</span>
XKBLAYOUT<span class="hl opt">=</span><span class="hl sng">"us"</span>
XKBVARIANT<span class="hl opt">=</span><span class="hl sng">"altgr-intl"</span>
<span class="hl slc"># the goal is to :</span>
<span class="hl slc"># 1. swap alt together</span>
<span class="hl slc"># 2. swap window and ctrl_left</span>
<span class="hl slc"># 3. swap menu and ctrl_right</span>
<span class="hl slc"># 4. swap escape and caps-lock</span>
<span class="hl slc"># third level is usually AltGr</span>
<span class="hl slc"># so seting lv3:lalt_switch means now third level is left alt</span>
<span class="hl slc"># also lv3:ralt_alt means now altGr is simple alt</span>
<span class="hl slc"># now ctrl:swap_lwin_lctl means swap left control and left win</span>
<span class="hl slc"># also ctrl:menu_rctrl means swap menu and right control</span>
XKBOPTIONS<span class="hl opt">=</span><span class="hl sng">"caps:swapescape,ctrl:menu_rctrl,ctrl:swap_lwin_lctl,lv3:lalt_switch,lv3:ralt_alt"</span>
BACKSPACE<span class="hl opt">=</span><span class="hl sng">"guess"</span>
</code></pre>
<p>then <code>systemctl restart keyboard-setup</code></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Keyboard — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-11-05 16:02</time> </div>https://blog.parisni.com/wiki/gradleGradle2023-11-29T21:33:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">build</span></p>
<h1>Gradle</h1>
<h2>Tests specific unit</h2>
<pre><code>gradle test --tests "package.class.method"
# also
gradle :submodule:test --tests "package.class.method"
# also
./gradlew :lakehouse:test -Dmaven.javadoc.skip=true --console=plain --tests "package.class.method"
</code></pre>
<h2>Show dependencies</h2>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Gradle — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-11-29 21:33</time> </div>https://blog.parisni.com/wiki/rssRSS2023-09-11T14:42:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">security</span>
<span class="w3-tag w3-red tag">news</span></p>
<h1>RSS</h1>
<h2>Subscribe to software releases</h2>
<h3>Gitlab</h3>
<pre><code class="language-html">https://<span class="hl kwa"><theproject></span>/-/tags?format=atom
</code></pre>
<h3>Microsoft Github</h3>
<pre><code class="language-html">https://<span class="hl kwa"><theproject></span>/releases.atom
</code></pre>
<h3>Medium</h3>
<pre><code class="language-html">https://medium.com/feed/@username
https://medium.com/feed/tag/[tag-name]
</code></pre>
<h3>Youtube</h3>
<ol>
<li>go to the channel page</li>
<li>paste the below js script into the console (F12)</li>
</ol>
<pre><code class="language-javascript"><span class="hl kwa">for</span> <span class="hl opt">(</span><span class="hl kwa">var</span> arrScripts <span class="hl opt">=</span> document<span class="hl opt">.</span><span class="hl kwd">getElementsByTagName</span><span class="hl opt">(</span><span class="hl sng">'script'</span><span class="hl opt">),</span> i <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">;</span> i <span class="hl opt"><</span> arrScripts<span class="hl opt">.</span>length<span class="hl opt">;</span> i<span class="hl opt">++) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span>arrScripts<span class="hl opt">[</span>i<span class="hl opt">].</span>textContent<span class="hl opt">.</span><span class="hl kwd">indexOf</span><span class="hl opt">(</span><span class="hl sng">'externalId'</span><span class="hl opt">) != -</span><span class="hl num">1</span><span class="hl opt">) {</span>
<span class="hl kwa">var</span> channelId <span class="hl opt">=</span> arrScripts<span class="hl opt">[</span>i<span class="hl opt">].</span>textContent<span class="hl opt">.</span><span class="hl kwd">match</span><span class="hl opt">(</span><span class="hl kwc">/\"externalId\"\s*\:\s*\"(.*?)\"/</span><span class="hl opt">)[</span><span class="hl num">1</span><span class="hl opt">];</span>
<span class="hl kwa">var</span> channelRss <span class="hl opt">=</span> <span class="hl sng">'https://www.youtube.com/feeds/videos.xml?channel_id='</span> <span class="hl opt">+</span> channelId<span class="hl opt">;</span>
<span class="hl kwa">var</span> channelTitle <span class="hl opt">=</span> document<span class="hl opt">.</span>title<span class="hl opt">.</span><span class="hl kwd">match</span><span class="hl opt">(</span><span class="hl kwc">/\(?\d*\)?\s?(.*?)\s\-\sYouTube/</span><span class="hl opt">)[</span><span class="hl num">1</span><span class="hl opt">];</span>
console<span class="hl opt">.</span><span class="hl kwd">log</span><span class="hl opt">(</span><span class="hl sng">'The rss feed of the channel</span> <span class="hl esc">\'</span><span class="hl sng">'</span> <span class="hl opt">+</span> channelTitle <span class="hl opt">+</span> <span class="hl sng">'</span><span class="hl esc">\'</span> <span class="hl sng">is:</span><span class="hl esc">\n</span><span class="hl sng">'</span> <span class="hl opt">+</span> channelRss<span class="hl opt">);</span>
<span class="hl kwa">break</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=RSS — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-09-11 14:42</time> </div>https://blog.parisni.com/wiki/makefileMakefile2021-11-07T10:25:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">automation</span></p>
<h1>Makefile</h1>
<h2>Pass args to a command</h2>
<p>The <em>filter-out</em> command removes the former arg from the latter args.</p>
<p><code>makefile</code> example:</p>
<pre><code class="language-bash"><span class="hl slc"># allows passing args to make file</span>
<span class="hl slc"># see https://stackoverflow.com/a/47008498/3865083</span>
<span class="hl slc"># use below command to get the args:</span>
<span class="hl slc"># $(filter-out $@,$(MAKECMDGOALS))</span>
<span class="hl opt">%:</span>
@<span class="hl opt">:</span>
action<span class="hl opt">:</span>
<span class="hl kwb">echo</span> <span class="hl sng">"</span><span class="hl ipl">$(filter-out $@,$(MAKECMDGOALS)</span><span class="hl sng">)"</span>
</code></pre>
<p>Then those will behave the same :</p>
<pre><code class="language-bash"><span class="hl kwc">make</span> action foo bar baz
<span class="hl slc"># OR</span>
<span class="hl kwc">make</span> foo bar action
</code></pre>
<h2>Default goal</h2>
<p>Add this:</p>
<pre><code class="language-bash">.DEFAULT_GOAL <span class="hl opt">:=</span> all
</code></pre>
<h2>Phony</h2>
<p>By default, make will skip tasks when the current folder contains action names.
<em>Phony</em> deactivate this feature.</p>
<pre><code class="language-bash">.PHONY<span class="hl opt">:</span> all
</code></pre>
<h2>Instanciate variables</h2>
<p>This way:</p>
<pre><code class="language-bash">var <span class="hl opt">:=</span> content
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Makefile — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-07 10:25</time> </div>https://blog.parisni.com/wiki/observabilityObservability2023-07-31T21:29:00+00:00<div id="generated-toc"> </div><hr> <p>#web
#database</p>
<h1>Observability</h1>
<h2>Grafana</h2>
<p>Grafana provides:</p>
<ul>
<li>exploring (online queries)</li>
<li>alerting (as a replacement of alertmanager)</li>
<li>dashboarding</li>
</ul>
<p>It integrates smoothly with loki (logging) and prometheus (metrics).</p>
<h2>Loki</h2>
<p>Loki stores logs either on local or object storage.</p>
<p>It handles various log sources.</p>
<p>I found the most proficient way is to use <code>journal</code> (systemd logs) in json
format. It is possible to redirect lot of tools into systemd. For eg:</p>
<ul>
<li>docker</li>
<li>postgres</li>
<li>nginx</li>
</ul>
<h2>Prometheus</h2>
<p>Prometheus can get metrics from system by pulling informations or by
making them push informations to a gateway. The grafana interface can
be plugged on the prometheus interface.</p>
<p>Overall config:</p>
<pre><code>scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
- job_name: 'matrix'
metrics_path: "/_synapse/metrics"
static_configs:
- targets: ['localhost:9101']
</code></pre>
<h3>Installation</h3>
<p>On archlinux, install <code>prometheus</code> and <code>grafana</code> packages. Both have a
systemd service, on ports <code>9090</code> and <code>3000</code>. The config files are
<code>/etc/prometheus</code> and <code>/etc/grafana.ini</code>.</p>
<h3>Use timescaledb</h3>
<p>Timescaledb can replace the default database backend. The promscale
extension improves the performances.</p>
<h3>Linux Metrics</h3>
<p>On archlinux, install <code>prometheus-node-exporter</code> package and activate
the service. It will provide a metric http endpoint on port
<code>9100</code>. Then configure prometheus to fetch the metrics. You can <a href="https://grafana.com/grafana/dashboards/1860">import the grafana dashboard</a>.</p>
<h3>Matrix</h3>
<p>The grafana configuration json <a href="https://github.com/matrix-org/synapse/blob/master/contrib/grafana/synapse.json">can be found
here</a>.
Configure synapse <code>homeserver.yaml</code>:</p>
<pre><code class="language-yaml"><span class="hl kwa">listeners:</span>
<span class="hl opt">-</span> <span class="hl kwa">port:</span> <span class="hl num">9101</span>
<span class="hl kwa">type:</span> metrics
<span class="hl kwa">bind_addresses:</span> <span class="hl opt">[</span><span class="hl sng">'0.0.0.0'</span><span class="hl opt">]</span>
<span class="hl kwa">enable_metrics:</span> true
</code></pre>
<h3>Jitsi</h3>
<ul>
<li><a href="https://blog.zwindler.fr/2020/06/08/superviser-votre-instance-jitsi-avec-prometheus-et-grafana/">This blog explains how to</a></li>
<li><a href="https://github.com/jitsi/jitsi-videobridge/blob/master/doc/statistics.md">The jitsi documentation</a></li>
<li><a href="https://grafana.com/grafana/dashboards/11925">The dashboard</a></li>
</ul>
<h3>Pleroma</h3>
<ul>
<li><a href="https://coffee-and-dreams.uk/tutorials/2019/11/06/monitoring-pleroma.html">This blog post</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Observability — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-07-31 21:29</time> </div>https://blog.parisni.com/wiki/gpgGpg2021-10-23T14:00:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">security</span></p>
<h1>Gpg</h1>
<h2>Expired key</h2>
<p>It is good practice to make key expire. In order to renew an expired key:</p>
<pre><code class="language-bash">gpg <span class="hl kwb">--edit-key</span> <span class="hl opt"><</span>the key ID<span class="hl opt">></span>
<span class="hl slc"># then choose tu subkey</span>
key <span class="hl num">1</span> <span class="hl slc"># for the second subkey</span>
expire
<span class="hl num">2</span>y <span class="hl slc"># for 2years</span>
save
</code></pre>
<h2>Export a private key</h2>
<pre><code class="language-bash">gpg <span class="hl kwb">--output</span> <span class="hl opt">/</span>tmp<span class="hl opt">/</span>thekey.secret<span class="hl kwb">-key --armor --export-secret-key</span> <span class="hl opt"><</span>the key id<span class="hl opt">></span>
</code></pre>
<h2>Import a private key</h2>
<pre><code class="language-bash">gpg <span class="hl kwb">--import</span> thekey.secret<span class="hl kwb">-key</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Gpg — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-10-23 14:00</time> </div>https://blog.parisni.com/wiki/libreofficeLibreoffice2021-12-27T13:03:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">editor</span></p>
<h1>Libreoffice</h1>
<h2>Install with snap</h2>
<pre><code>sudo snap install libreoffice
</code></pre>
<h2>Change language with snap</h2>
<ol>
<li><a href="https://askubuntu.com/a/1330980/333321">follow this procedure</a></li>
<li><a href="https://wiki.documentfoundation.org/Documentation/HowTo/install_extension">install extension</a></li>
<li><a href="https://ask.libreoffice.org/t/how-do-i-change-the-default-language/26772/2">change the language</a></li>
</ol>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Libreoffice — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-27 13:03</time> </div>https://blog.parisni.com/wiki/swaySway2023-12-10T19:52:00+00:00<div id="generated-toc"> </div><hr> <p>#linux</p>
<h1>Sway</h1>
<h2>Menu</h2>
<ul>
<li>~~rofi, change the theme <code>rofi-theme-selector</code>~~</li>
<li>wofi + <a href="https://github.com/tobiaspc/wofi-scripts">python script for windows</a></li>
</ul>
<h2>Bar</h2>
<ul>
<li>waybar</li>
<li><a href="https://github.com/Alexays/Waybar/wiki/Module:-Disk">modules documentation</a></li>
<li>needed: <code>sudo apt-get install -y fonts-font-awesome</code></li>
</ul>
<h2>Notifications</h2>
<ul>
<li>mako</li>
</ul>
<h2>Copy/paste</h2>
<ul>
<li>copyQ</li>
</ul>
<h2>Screens</h2>
<ul>
<li><code>wlr-randr</code> instead of <code>xrandr</code></li>
</ul>
<h2>Keyboards and mouse</h2>
<p>Get the device list:</p>
<pre><code class="language-shell">swaymsg -t get_inputs
</code></pre>
<p>Then configure each one:</p>
<pre><code>input "6127:24704:Lite-On_Tech_Lenovo_USB_Travel_Keyboard_with_Ultra_Nav" {
xkb_layout "us"
xkb_variant altgr-intl
xkb_options caps:escape,ctrl:swap_lalt_lctl,lv3:ralt_alt,lv3:switch
}
input "6127:24704:Lite-On_Tech_Lenovo_USB_Travel_Keyboard_with_Ultra_Nav_Mouse" {
dwt enabled
natural_scroll disabled
middle_emulation enabled
scroll_button 274
scroll_method on_button_down
}
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Sway — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-10 19:52</time> </div>https://blog.parisni.com/wiki/mailMail2023-04-30T17:47:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<hr>
<p><span class="w3-tag w3-red tag">linux</span>
<span class="w3-tag w3-red tag">mail</span></p>
<h1>Mail</h1>
<h2>Setup smtp linux mail</h2>
<ul>
<li><a href="https://moritzvd.com/email-with-smtp-debian-ubuntu/">follow this guide</a></li>
<li>except configure msmtp thought <code>/root/.msmtprc</code> instead of <code>/etc/msmtprc</code></li>
</ul>
<h2>mailu</h2>
<ul>
<li><a href="https://mailu.io/master/webadministration.html#id1">Configure rspamd</a></li>
<li><a href="https://rspamd.com/doc/modules/greylisting.html">configure greylisting</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mail — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-30 17:47</time> </div>https://blog.parisni.com/blog/concourseConcourse2023-09-11T14:42:00+00:00<div id="generated-toc"> </div><hr> <h1>Concourse</h1>
<h2>Commands</h2>
<pre><code class="language-bash"><span class="hl slc"># create/update a pipeline</span>
fly <span class="hl kwb">-t</span> tutorial <span class="hl kwb">set-pipeline -c</span> pipeline.yml <span class="hl kwb">-p</span> hello<span class="hl kwb">-world</span>
<span class="hl slc"># trigger a job</span>
fly <span class="hl kwb">-t</span> tutorial trigger<span class="hl kwb">-job -j</span> hello<span class="hl kwb">-world</span><span class="hl opt">/</span>job<span class="hl kwb">-hello-world</span>
<span class="hl slc"># ask for pipeline builds</span>
fly <span class="hl kwb">-t</span> tutorial builds
id name status start end duration team created by
<span class="hl num">60</span> hello<span class="hl kwb">-world</span><span class="hl opt">/</span>job<span class="hl kwb">-hello-world</span><span class="hl opt">/</span><span class="hl num">4</span> succeeded <span class="hl num">2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">16</span><span class="hl opt">:</span><span class="hl num">34</span><span class="hl opt">:</span><span class="hl num">11</span><span class="hl opt">+</span><span class="hl num">0200 2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">16</span><span class="hl opt">:</span><span class="hl num">34</span><span class="hl opt">:</span><span class="hl num">14</span><span class="hl opt">+</span><span class="hl num">0200 3</span>s main admin
<span class="hl num">59</span> hello<span class="hl kwb">-world</span><span class="hl opt">/</span>job<span class="hl kwb">-hello-world</span><span class="hl opt">/</span><span class="hl num">3</span> succeeded <span class="hl num">2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">16</span><span class="hl opt">:</span><span class="hl num">26</span><span class="hl opt">:</span><span class="hl num">01</span><span class="hl opt">+</span><span class="hl num">0200 2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">16</span><span class="hl opt">:</span><span class="hl num">26</span><span class="hl opt">:</span><span class="hl num">04</span><span class="hl opt">+</span><span class="hl num">0200 3</span>s main admin
<span class="hl num">15</span> hello<span class="hl kwb">-world</span><span class="hl opt">/</span>job<span class="hl kwb">-hello-world</span><span class="hl opt">/</span><span class="hl num">2</span> failed <span class="hl num">2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">15</span><span class="hl opt">:</span><span class="hl num">34</span><span class="hl opt">:</span><span class="hl num">40</span><span class="hl opt">+</span><span class="hl num">0200 2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">15</span><span class="hl opt">:</span><span class="hl num">34</span><span class="hl opt">:</span><span class="hl num">43</span><span class="hl opt">+</span><span class="hl num">0200 3</span>s main admin
<span class="hl num">13</span> hello<span class="hl kwb">-world</span><span class="hl opt">/</span>job<span class="hl kwb">-hello-world</span><span class="hl opt">/</span><span class="hl num">1</span> succeeded <span class="hl num">2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">14</span><span class="hl opt">:</span><span class="hl num">50</span><span class="hl opt">:</span><span class="hl num">49</span><span class="hl opt">+</span><span class="hl num">0200 2023</span><span class="hl kwb">-08-28</span>@<span class="hl num">14</span><span class="hl opt">:</span><span class="hl num">50</span><span class="hl opt">:</span><span class="hl num">52</span><span class="hl opt">+</span><span class="hl num">0200 3</span>s main admin
<span class="hl slc"># watch a pipeline / job / build-number</span>
fly <span class="hl kwb">-t</span> tutorial watch <span class="hl kwb">-j</span> hello<span class="hl kwb">-world</span><span class="hl opt">/</span>job<span class="hl kwb">-hello-world --build</span> <span class="hl num">4</span>
<span class="hl slc"># destroy pipelines</span>
fly <span class="hl kwb">-t</span> tutorial destroy<span class="hl kwb">-pipeline -p</span> hello<span class="hl kwb">-world</span>
</code></pre>
<h2>Parameter> Values in params sections in turn become environment variables within the task:</h2>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Concourse — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-09-11 14:42</time> </div>https://blog.parisni.com/people/varlam-chalamovVarlam Chalamov2023-04-25T16:36:00+00:00<div id="generated-toc"> </div><hr> <h1>Varlam Chalamov</h1>
<ul>
<li><a href="https://fr.m.wikipedia.org/wiki/Varlam_Chalamov">Wikipedia</a></li>
<li><a href="https://fr.m.wikipedia.org/wiki/R%C3%A9cits_de_la_Kolyma">récits de kolyma</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Varlam Chalamov — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-04-25 16:36</time> </div>https://blog.parisni.com/gallery/churchThe Church2021-12-29T17:15:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 65%; /* IE10 */
flex: 65%;
max-width: 65%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 55%;
flex: 55%;
max-width: 55%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>The Church</h1>
<p></p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/church/church-space.webp">
<img src="/images/gallery/church/church-space.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/church/church-pique.webp">
<img src="/images/gallery/church/church-pique.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/church/church-round.webp">
<img src="/images/gallery/church/church-round.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/church/church-echo.webp">
<img src="/images/gallery/church/church-echo.webp"></a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=The Church — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-29 17:15</time> </div>https://blog.parisni.com/gallery/poulpoPoulpo2021-12-29T18:47:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 50%; /* IE10 */
flex: 50%;
max-width: 50%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 50%;
flex: 50%;
max-width: 50%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Poulpo</h1>
<p></p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/poulpo/IMG_20211121_130255.webp">
<img src="/images/gallery/poulpo/IMG_20211121_130255.webp" alt="Poulpe rue des Amendiers" style="width=100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/poulpo/IMG_20211121_160902.webp">
<img src="/images/gallery/poulpo/IMG_20211121_160902.webp" alt="Poulpe metro ?" style="width=100%">
</a>
<a href="/images/gallery/poulpo/IMG_20211121_160746.webp">
<img src="/images/gallery/poulpo/IMG_20211121_160746.webp" alt="Poulpe metro ?" style="width=100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/poulpo/IMG_20211120_162156.webp">
<img src="/images/gallery/poulpo/IMG_20211120_162156.webp" alt="Poulpe rue de Tolbiac" style="width=100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/poulpo/IMG_20211120_151722.webp">
<img src="/images/gallery/poulpo/IMG_20211120_151722.webp" alt="Poulpe boulevard de Charonne" style="width=100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/poulpo/poulpo-volet.webp">
<img src="/images/gallery/poulpo/poulpo-volet.webp" alt="Poulpe boulevard de l'Hopital" style="width=100%">
</a>
</div>
<p>Eva, paris 3íème:</p>
<div class="column">
<a href="/images/gallery/poulpo/20211216_170150.webp">
<img src="/images/gallery/poulpo/20211216_170150.webp" alt="Paris 3ième, Eva" style="width=100%">
</a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Poulpo — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-29 18:47</time> </div>https://blog.parisni.com/gallery/the-doorsThe Doors2021-12-29T18:47:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 65%; /* IE10 */
flex: 65%;
max-width: 65%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 55%;
flex: 55%;
max-width: 55%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>The Doors</h1>
<p></p>
</div>
<blockquote>
<p>If the doors of perception were cleansed every thing would appear to man as it is, Infinite.</p>
</blockquote>
<div class="row">
<div class="column">
<a href="/images/gallery/doors/doors-squouat.webp">
<img src="/images/gallery/doors/doors-squouat.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/doors-closed.webp">
<img src="/images/gallery/doors/doors-closed.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/doors-1984.webp">
<img src="/images/gallery/doors/doors-1984.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/doors-desk.webp">
<img src="/images/gallery/doors/doors-desk.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/doors-green.webp">
<img src="/images/gallery/doors/doors-green.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/doors-lion.webp">
<img src="/images/gallery/doors/doors-lion.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/doors-monkeys.webp">
<img src="/images/gallery/doors/doors-monkeys.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/doors-poste.webp">
<img src="/images/gallery/doors/doors-poste.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/door-ogre.webp">
<img src="/images/gallery/doors/door-ogre.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/door-crab.webp">
<img src="/images/gallery/doors/door-crab.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/doors/door-stone.webp">
<img src="/images/gallery/doors/door-stone.webp"></a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=The Doors — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-29 18:47</time> </div>https://blog.parisni.com/gallery/ministryMinistries2021-12-12T19:17:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 65%; /* IE10 */
flex: 65%;
max-width: 65%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 55%;
flex: 55%;
max-width: 55%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Ministries</h1>
<p></p>
</div>
<p><strong>WAR</strong> IS <strong>PEACE</strong></p>
<p><strong>FREEDOM</strong> IS <strong>SLAVERY</strong></p>
<p><strong>IGNORANCE</strong> IS <strong>STRENGTH</strong></p>
<div class="row">
<div class="column">
<a href="/images/gallery/ministry/ministry-power.webp">
<img src="/images/gallery/ministry/ministry-power.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/ministry/ministry-war.webp">
<img src="/images/gallery/ministry/ministry-war.webp"></a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Ministries — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-12 19:17</time> </div>https://blog.parisni.com/gallery/streetStreet2021-12-29T17:25:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 65%; /* IE10 */
flex: 65%;
max-width: 65%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 55%;
flex: 55%;
max-width: 55%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Street</h1>
<p></p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/street/cassius-x.webp">
<img src="/images/gallery/street/cassius-x.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/street/street-gpg.webp">
<img src="/images/gallery/street/street-gpg.webp"></a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Street — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-29 17:25</time> </div>https://blog.parisni.com/gallery/raised-stonesRaised Stones2021-11-26T22:32:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 50%; /* IE10 */
flex: 50%;
max-width: 50%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 50%;
flex: 50%;
max-width: 50%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Raised Stones</h1>
<p></p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/raised-stones/IMG_20211112_145855.webp">
<img src="/images/gallery/raised-stones/IMG_20211112_145855.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/raised-stones/IMG_20210411_144804.webp">
<img src="/images/gallery/raised-stones/IMG_20210411_144804.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/raised-stones/IMG_20211112_144739.webp">
<img src="/images/gallery/raised-stones/IMG_20211112_144739.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/raised-stones/IMG_20211112_145107.webp">
<img src="/images/gallery/raised-stones/IMG_20211112_145107.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/raised-stones/IMG_20211112_145253.webp">
<img src="/images/gallery/raised-stones/IMG_20211112_145253.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/raised-stones/IMG_20211112_151920.webp">
<img src="/images/gallery/raised-stones/IMG_20211112_151920.webp">
</a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Raised Stones — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-26 22:32</time> </div>https://blog.parisni.com/gallery/silexSilex2021-11-26T22:32:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 50%; /* IE10 */
flex: 50%;
max-width: 50%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 50%;
flex: 50%;
max-width: 50%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Silex</h1>
<p>A first draft</p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/silex/20200412_115346.webp">
<img src="/images/gallery/silex/20200412_115346.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_115525.webp">
<img src="/images/gallery/silex/20200412_115525.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_115750.webp">
<img src="/images/gallery/silex/20200412_115750.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_115915.webp">
<img src="/images/gallery/silex/20200412_115915.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_115942.webp">
<img src="/images/gallery/silex/20200412_115942.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_120019.webp">
<img src="/images/gallery/silex/20200412_120019.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_120406.webp">
<img src="/images/gallery/silex/20200412_120406.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_120524.webp">
<img src="/images/gallery/silex/20200412_120524.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_120538.webp">
<img src="/images/gallery/silex/20200412_120538.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_120633.webp">
<img src="/images/gallery/silex/20200412_120633.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_121057.webp">
<img src="/images/gallery/silex/20200412_121057.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_121602.webp">
<img src="/images/gallery/silex/20200412_121602.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_121716.webp">
<img src="/images/gallery/silex/20200412_121716.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_121935.webp">
<img src="/images/gallery/silex/20200412_121935.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_122423.webp">
<img src="/images/gallery/silex/20200412_122423.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/silex/20200412_122658.webp">
<img src="/images/gallery/silex/20200412_122658.webp">
</a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Silex — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-26 22:32</time> </div>https://blog.parisni.com/gallery/flowerFlowers2021-11-26T22:32:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 50%; /* IE10 */
flex: 50%;
max-width: 50%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 50%;
flex: 50%;
max-width: 50%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Flowers</h1>
<p></p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/flower/medium-20200809_150030.webp">
<img src="/images/gallery/flower/medium-20200809_150030.webp" style="width:100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/flower/medium-20200809_150126.webp">
<img src="/images/gallery/flower/medium-20200809_150126.webp" style="width:100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/flower/medium-20200809_150215.webp">
<img src="/images/gallery/flower/medium-20200809_150215.webp" style="width:100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/flower/medium-20200809_150030.webp">
<img src="/images/gallery/flower/medium-20200809_150030.webp" style="width:100%">
</a>
</div>
<div class="column">
<a href="/images/gallery/flower/medium-20200809_150156.webp">
<img src="/images/gallery/flower/medium-20200809_150156.webp" style="width:100%">
</a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Flowers — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-26 22:32</time> </div>https://blog.parisni.com/gallery/vincennesVincennes2021-11-26T22:32:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 50%; /* IE10 */
flex: 50%;
max-width: 50%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 50%;
flex: 50%;
max-width: 50%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Vincennes</h1>
<p></p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150518.webp">
<img src="/images/gallery/vincennes/medium-20200713_150518.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150537.webp">
<img src="/images/gallery/vincennes/medium-20200713_150537.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150537.webp">
<img src="/images/gallery/vincennes/medium-20200713_150605.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150605.webp">
<img src="/images/gallery/vincennes/medium-20200713_150605.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150636.webp">
<img src="/images/gallery/vincennes/medium-20200713_150636.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150643.webp">
<img src="/images/gallery/vincennes/medium-20200713_150643.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150707.webp">
<img src="/images/gallery/vincennes/medium-20200713_150707.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150817.webp">
<img src="/images/gallery/vincennes/medium-20200713_150817.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150831.webp">
<img src="/images/gallery/vincennes/medium-20200713_150831.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150853.webp">
<img src="/images/gallery/vincennes/medium-20200713_150853.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_150908.webp">
<img src="/images/gallery/vincennes/medium-20200713_150908.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151232.webp">
<img src="/images/gallery/vincennes/medium-20200713_151232.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151245.webp">
<img src="/images/gallery/vincennes/medium-20200713_151245.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151302.webp">
<img src="/images/gallery/vincennes/medium-20200713_151302.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151456.webp">
<img src="/images/gallery/vincennes/medium-20200713_151456.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151514.webp">
<img src="/images/gallery/vincennes/medium-20200713_151514.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151528.webp">
<img src="/images/gallery/vincennes/medium-20200713_151528.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151541.webp">
<img src="/images/gallery/vincennes/medium-20200713_151541.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151607.webp">
<img src="/images/gallery/vincennes/medium-20200713_151607.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151622.webp">
<img src="/images/gallery/vincennes/medium-20200713_151622.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151647.webp">
<img src="/images/gallery/vincennes/medium-20200713_151647.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151655.webp">
<img src="/images/gallery/vincennes/medium-20200713_151655.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151718.webp">
<img src="/images/gallery/vincennes/medium-20200713_151718.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151725.webp">
<img src="/images/gallery/vincennes/medium-20200713_151725.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151738.webp">
<img src="/images/gallery/vincennes/medium-20200713_151738.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151745.webp">
<img src="/images/gallery/vincennes/medium-20200713_151745.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151816.webp">
<img src="/images/gallery/vincennes/medium-20200713_151816.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151920.webp">
<img src="/images/gallery/vincennes/medium-20200713_151920.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_151953.webp">
<img src="/images/gallery/vincennes/medium-20200713_151953.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_152017.webp">
<img src="/images/gallery/vincennes/medium-20200713_152017.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_152036.webp">
<img src="/images/gallery/vincennes/medium-20200713_152036.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_152116.webp">
<img src="/images/gallery/vincennes/medium-20200713_152116.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_152130.webp">
<img src="/images/gallery/vincennes/medium-20200713_152130.webp">
</a>
</div>
<div class="column">
<a href="/images/gallery/vincennes/medium-20200713_152147.webp">
<img src="/images/gallery/vincennes/medium-20200713_152147.webp">
</a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Vincennes — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-26 22:32</time> </div>https://blog.parisni.com/gallery/towersTowers2021-12-29T17:15:00+00:00<div id="generated-toc"> </div><hr> <style>
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family: Arial;
}
.header {
text-align: center;
padding: 32px;
}
.row {
display: -ms-flexbox; /* IE10 */
display: flex;
-ms-flex-wrap: wrap; /* IE10 */
flex-wrap: wrap;
padding: 0 4px;
}
/* Create four equal columns that sits next to each other */
.column {
-ms-flex: 65%; /* IE10 */
flex: 65%;
max-width: 65%;
padding: 0 4px;
}
.column img {
margin-top: 8px;
vertical-align: middle;
width: 100%;
}
/* Responsive layout - makes a two column-layout instead of four columns */
@media screen and (max-width: 800px) {
.column {
-ms-flex: 55%;
flex: 55%;
max-width: 55%;
}
}
/* Responsive layout - makes the two columns stack on top of each other instead of next to each other */
@media screen and (max-width: 600px) {
.column {
-ms-flex: 100%;
flex: 100%;
max-width: 100%;
}
}
</style>
<body>
<div class="header">
<h1>Towers</h1>
<p></p>
</div>
<div class="row">
<div class="column">
<a href="/images/gallery/towers/tower-parnasse.webp">
<img src="/images/gallery/towers/tower-parnasse.webp"></a>
</div>
<div class="column">
<a href="/images/gallery/towers/tower-bastille.webp">
<img src="/images/gallery/towers/tower-bastille.webp"></a>
</div>
</div>
</body><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Towers — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-29 17:15</time> </div>https://blog.parisni.com/blog/hive-reflexionHive Reflexions2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <p><span>Date:<time id="post-date">2018-05-10</time></span></p>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="hive-reflexions">Hive Reflexions</h1>
<p>Hive has several advantages over other Datawarehouse solutions.</p>
<ol type="1">
<li>it is java based. This allows it to take advantage of UDF. The best
and the moste complicate way to write them as generic UDFs. The other
important aspect is to make them <span class="title-ref">blessed
UDF</span> by mean they are global and accessible to any user. Generic
UDF can produce any hive type from integer to complexe map
structure.</li>
<li>it can read/write into hbase and take advantage of a key-value
store.</li>
<li>it is ACID compliant, and provide both insert/update. However, this
new feature is not well integrated with other tools such apache spark or
facebook presto.</li>
<li>it supports multiple level of performance tunning such
partitionning, bucketting, bloom filters, predicate pushdown.</li>
</ol>
<p>Still hive has several disavantages:</p>
<ol type="1">
<li>lack or support for sequences. The workaround right now is to either
use a UUID - this has several drawbacks such performances/storage issue.
A second approach is to use the <span class="title-ref">sequence_nb +
(row_number() over())</span> function where <span class="title-ref">sequence_nb</span> is maintained into a table - the
count of the inserted rows need to be added to the sequence. The last
solution is a long, but it cannot be used concurrently. The solution
cannot be used in production.</li>
</ol>
<h1 id="hive-configuration-tips">hive configuration tips</h1>
<p>Hive can have OOM. Then this parameter might help - <span class="title-ref">set hive.exec.orc.memory.pool = 1.0</span></p>
<p>Predicate Push Down is better when INSERT STMT use a SORT BY
"orc.bloom.filter.columns"="col1,col2"</p>
<p>It is also possible to update the bloom filter by running: <span class="title-ref">ANALYZE TABLE Table1 COMPUTE STATISTICS FOR
COLUMNS;</span></p>
<ul>
<li><a href="https://medium.com/@anishekagarwal/apache-hive-beeline-progress-bar-fd57c29da65d">show
progress bar in beeline</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Hive Reflexions — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/postgresql-reflexionPostgreSQL Reflexions2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-05-13</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="postgresql-reflexions">PostgreSQL Reflexions</h1>
<p>After 4 years of postgresql management, I d'like to relate my
experience.</p>
<p>Postgreql is incredible:</p>
<ul>
<li>its community is wonderful, the mailing list is very active and
structured</li>
<li>the recent years have been very productive for postgres, with many
analytics improvements and perspectives</li>
<li>it has one of the best support for transactions, and for example
deal with DDL in transaction</li>
<li>it's stable, we never had any crash in 4 years, or need to reboot
the server</li>
<li>it manage bot structured and unstructured data</li>
<li>free text indexes and features are very advanced</li>
</ul>
<p>However, sadly postgresql has some drawnbacks:</p>
<ul>
<li>it does not deal great with large volumne when comes analytics</li>
<li>because it may be used for all batch ingesting, batch exporting,
OLTP, dumps, analytics, the overall performances might be affected</li>
<li>it does not scale horizontaly</li>
<li>it has no fail over, or this lead to complex setup</li>
</ul>
<p>Complementing postgresql with hadoop:</p>
<p>For this reason, it has been decided to externalize some
responsibilities such batch exporting and analytics to an other
technology that scale horizontally: hadoop.</p>
<p>The good point is synchronizing data from postgresql to hive daily is
easy thanks to incremental sqoop import on hdfs, based on indexed
timestamps in postgreql. This allows pushing the data from postgresql to
hive ready to analyse. The two tools interface well where hive does not
handle sequence, or triggers, and postgresql does not manage huge data
historisation, and columnar aggregation.</p>
<dl>
<dt>While postgreqsl does not handle well aggregations on bilions rows,
it's children greenplum is supposed to.</dt>
<dd>
<p>Let's envisage a greenplum post soon.</p>
</dd>
</dl>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=PostgreSQL Reflexions — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/hive-mergeHive Merge2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-05-13</time></span>
<p>Estimated Time:<span id="reading-time">2 minutes</span></p>
<h1 id="hive-merge">Hive Merge</h1>
<p>Hive introduced ACID statement: INSERT, UPDATE, DELETE & MERGE.
It turns-out HIVE 1.2.x is not optimised enough to support MERGE
efficiently. An in depth documentation can be found <a href="https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Limitations">there</a></p>
<p>The code below :</p>
<ul>
<li>is idempotent in someway. Suppose for some reason, the <span class="title-ref">merge insert</span> statement fails. Then running
again all statements won't be an issue since no update at all will
append again. Same case if the <span class="title-ref">update</span>
statement fails.</li>
<li>Simple version: makes the assumption that any data both present in
the <span class="title-ref">staging</span> table and the <span class="title-ref">target</span> table needs to be updated.</li>
<li>SCD1 and SCD2 have slighly difference that are mentionned if
existing.</li>
<li>SCD2 has been tested (upsert version) on a large target table ( 1.3
Bilion rows * 100 columns) and a source table of 100 Milion rows. As a
result, the total runtime was les than 3 hours. (50% resource on a 5
computer hadoop cluster).</li>
</ul>
<h1 id="staging-step">0) Staging step</h1>
<p>Some data was put on hdfs in the <stg_table> to be merged with
a target table. This might be done by scoop incrementally. Also, this
might be the whole table. All the steps below will work the same in both
case.</p>
<h1 id="preparation-step">1) Preparation step</h1>
<p>The 3 table above are necessary for any of SCD1 or SCD2. They are
respectively the <span class="title-ref">transformation</span>, <span class="title-ref">insert</span>, <span class="title-ref">update</span>
candidate tables.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- create transformation table</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="kw">CREATE</span> <span class="kw">TEMPORARY</span> <span class="kw">TABLE</span> <span class="op"><</span>trf_table<span class="op">></span> <span class="kw">LIKE</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="co">-- create insert table</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="kw">CREATE</span> <span class="kw">TEMPORARY</span> <span class="kw">TABLE</span> <span class="op"><</span>ins_table<span class="op">></span> <span class="kw">LIKE</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a><span class="co">-- create update table</span></span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a><span class="kw">CREATE</span> <span class="kw">TEMPORARY</span> <span class="kw">TABLE</span> <span class="op"><</span>upd_table<span class="op">></span> <span class="kw">LIKE</span> <span class="op"><</span>target_table<span class="op">></span></span></code></pre></div>
<h1 id="transformation-step">2) Transformation step</h1>
<p>This step loads the trf table (<span class="title-ref">transformed</span>) from the stg table (<span class="title-ref">staging</span>). It adds the hash column, and is the
place to add other transformations/columns if needed.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- example without transformations</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="op"><</span>trf_table<span class="op">></span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span> , <span class="kw">HASH</span>(<span class="op">*</span>) <span class="kw">as</span> <span class="kw">hash</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>stg_table<span class="op">></span>;</span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="co">-- example with transformations</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="co">-- t1 and t2 are new columns added</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="op"><</span>trf_table<span class="op">></span></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span> , <span class="kw">HASH</span>(stg.<span class="op">*</span>) <span class="kw">as</span> <span class="kw">hash</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> (<span class="kw">SELECT</span> <span class="op">*</span> , t1(), t2()</span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> <span class="op"><</span>stg_table<span class="op">></span>) <span class="kw">as</span> stg;</span></code></pre></div>
<h1 id="dispatching-step">3) Dispatching step</h1>
<p>Simplified version:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- variation when you already know that rows are all different</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>trf_table<span class="op">></span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="op"><</span>upd_table<span class="op">></span> <span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">WHERE</span> <span class="op"><</span>maj_date<span class="op">></span> <span class="kw">IS</span> <span class="kw">NOT</span> <span class="kw">NULL</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="op"><</span>ins_table<span class="op">></span> <span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">WHERE</span> <span class="op"><</span>maj_date<span class="op">></span> <span class="kw">IS</span> <span class="kw">NULL</span></span></code></pre></div>
<p>Complete version:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- feed both update & insert table</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> (<span class="kw">SELECT</span> <span class="op">*</span></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> <span class="op"><</span>trf_table<span class="op">></span> trf</span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> <span class="op"><</span>target_table<span class="op">></span> targ</span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a> <span class="kw">ON</span> (</span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a> targ.<span class="op"><</span>end_date<span class="op">></span> <span class="kw">IS</span> <span class="kw">NULL</span></span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> trf.<span class="op"><</span>primary_key<span class="op">></span> <span class="op">=</span> targ.<span class="op"><</span>primary_key<span class="op">></span></span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a> )</span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb4-10"><a href="#cb4-10" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> trf.<span class="kw">hash</span> <span class="op"><></span> targ.<span class="kw">hash</span>) <span class="kw">as</span> updateable</span>
<span id="cb4-11"><a href="#cb4-11" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="kw">TABLE</span> <span class="op"><</span>upd_table<span class="op">></span> <span class="kw">SELECT</span> <span class="op">*</span> ;</span>
<span id="cb4-12"><a href="#cb4-12" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-13"><a href="#cb4-13" aria-hidden="true" tabindex="-1"></a><span class="co">-- feed the insert table with new rows</span></span>
<span id="cb4-14"><a href="#cb4-14" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="op"><</span>ins_table<span class="op">></span></span>
<span id="cb4-15"><a href="#cb4-15" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span></span>
<span id="cb4-16"><a href="#cb4-16" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>trf_table<span class="op">></span></span>
<span id="cb4-17"><a href="#cb4-17" aria-hidden="true" tabindex="-1"></a><span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb4-18"><a href="#cb4-18" aria-hidden="true" tabindex="-1"></a><span class="kw">AND</span> <span class="op"><</span>primary_key<span class="op">></span> <span class="kw">NOT</span> <span class="kw">IN</span> (</span>
<span id="cb4-19"><a href="#cb4-19" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="op"><</span>primary_key<span class="op">></span></span>
<span id="cb4-20"><a href="#cb4-20" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb4-21"><a href="#cb4-21" aria-hidden="true" tabindex="-1"></a>);</span></code></pre></div>
<h1 id="upsert-version">4.1) Upsert version</h1>
<p>SCD 1:</p>
<p>Notice the delete is used in place of an update. The reason is an
update cannot join an other table and thus apply updates where
needed.</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- first delete update rows</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="kw">DELETE</span> <span class="kw">FROM</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="kw">AND</span> <span class="op"><</span>primary_key<span class="op">></span> <span class="kw">IN</span> (</span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="op"><</span>primary_key<span class="op">></span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> <span class="op"><</span>upd_table<span class="op">></span>);</span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a><span class="co">-- second, insert all new rows</span></span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span></span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>ins_table<span class="op">></span></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a><span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span></span>
<span id="cb5-14"><a href="#cb5-14" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>upd_table<span class="op">></span>;</span></code></pre></div>
<p>SCD 2:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- first update old rows</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="kw">UPDATE</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a><span class="kw">SET</span> <span class="op"><</span>end_date<span class="op">></span> <span class="op">=</span> <span class="fu">current_date</span>()</span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a><span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a><span class="kw">AND</span> <span class="op"><</span>end_date<span class="op">></span> <span class="kw">IS</span> <span class="kw">NULL</span></span>
<span id="cb6-6"><a href="#cb6-6" aria-hidden="true" tabindex="-1"></a><span class="kw">AND</span> <span class="op"><</span>primary_key<span class="op">></span> <span class="kw">IN</span> (</span>
<span id="cb6-7"><a href="#cb6-7" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="op"><</span>primary_key<span class="op">></span></span>
<span id="cb6-8"><a href="#cb6-8" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> <span class="op"><</span>upd_table<span class="op">></span>);</span>
<span id="cb6-9"><a href="#cb6-9" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb6-10"><a href="#cb6-10" aria-hidden="true" tabindex="-1"></a><span class="co">-- second, insert all new rows</span></span>
<span id="cb6-11"><a href="#cb6-11" aria-hidden="true" tabindex="-1"></a><span class="kw">INSERT</span> <span class="kw">INTO</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb6-12"><a href="#cb6-12" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span></span>
<span id="cb6-13"><a href="#cb6-13" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>ins_table<span class="op">></span></span>
<span id="cb6-14"><a href="#cb6-14" aria-hidden="true" tabindex="-1"></a><span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb6-15"><a href="#cb6-15" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span></span>
<span id="cb6-16"><a href="#cb6-16" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>upd_table<span class="op">></span>;</span></code></pre></div>
<h1 id="merge-version">4.2) Merge version</h1>
<p>SCD 1:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- first update old rows</span></span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a><span class="kw">MERGE</span> <span class="kw">INTO</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb7-3"><a href="#cb7-3" aria-hidden="true" tabindex="-1"></a><span class="kw">USING</span></span>
<span id="cb7-4"><a href="#cb7-4" aria-hidden="true" tabindex="-1"></a>(</span>
<span id="cb7-5"><a href="#cb7-5" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="kw">null</span> <span class="kw">AS</span> join_key, upd.<span class="op">*</span> <span class="kw">FROM</span> <span class="op"><</span>upd_table<span class="op">></span></span>
<span id="cb7-6"><a href="#cb7-6" aria-hidden="true" tabindex="-1"></a><span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb7-7"><a href="#cb7-7" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op"><</span>primary_key<span class="op">></span> <span class="kw">AS</span> join_key, ins.<span class="op">*</span> <span class="kw">FROM</span> <span class="op"><</span>ins_table<span class="op">></span></span>
<span id="cb7-8"><a href="#cb7-8" aria-hidden="true" tabindex="-1"></a>) <span class="kw">AS</span> sub</span>
<span id="cb7-9"><a href="#cb7-9" aria-hidden="true" tabindex="-1"></a><span class="kw">ON</span> (sub.join_key <span class="op">=</span> <span class="op"><</span>target_table<span class="op">></span>.join_key</span>
<span id="cb7-10"><a href="#cb7-10" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> <span class="op"><</span>target_table<span class="op">></span>.<span class="op"><</span>end_date<span class="op">></span> <span class="kw">IS</span> <span class="kw">NULL</span>)</span>
<span id="cb7-11"><a href="#cb7-11" aria-hidden="true" tabindex="-1"></a><span class="cf">WHEN</span> MATCHED</span>
<span id="cb7-12"><a href="#cb7-12" aria-hidden="true" tabindex="-1"></a> <span class="cf">THEN</span> <span class="kw">UPDATE</span> <span class="kw">SET</span> col1<span class="op">=</span>sub.col1, <span class="op">..</span>. , <span class="kw">hash</span><span class="op">=</span>sub.<span class="kw">hash</span></span>
<span id="cb7-13"><a href="#cb7-13" aria-hidden="true" tabindex="-1"></a><span class="cf">WHEN</span> <span class="kw">NOT</span> MATCHED</span>
<span id="cb7-14"><a href="#cb7-14" aria-hidden="true" tabindex="-1"></a> <span class="cf">THEN</span> <span class="kw">INSERT</span> <span class="kw">VALUES</span> (sub.col1, <span class="op">..</span>. , sub.<span class="kw">hash</span>) ;</span></code></pre></div>
<p>SCD 2:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- first update old rows</span></span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a><span class="kw">MERGE</span> <span class="kw">INTO</span> <span class="op"><</span>target_table<span class="op">></span></span>
<span id="cb8-3"><a href="#cb8-3" aria-hidden="true" tabindex="-1"></a><span class="kw">USING</span></span>
<span id="cb8-4"><a href="#cb8-4" aria-hidden="true" tabindex="-1"></a>(</span>
<span id="cb8-5"><a href="#cb8-5" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="kw">null</span> <span class="kw">AS</span> join_key, upd.<span class="op">*</span> <span class="kw">FROM</span> <span class="op"><</span>upd_table<span class="op">></span></span>
<span id="cb8-6"><a href="#cb8-6" aria-hidden="true" tabindex="-1"></a><span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb8-7"><a href="#cb8-7" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="kw">null</span> <span class="kw">AS</span> join_key, ins.<span class="op">*</span> <span class="kw">FROM</span> <span class="op"><</span>ins_table<span class="op">></span></span>
<span id="cb8-8"><a href="#cb8-8" aria-hidden="true" tabindex="-1"></a><span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb8-9"><a href="#cb8-9" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op"><</span>primary_key<span class="op">></span> <span class="kw">AS</span> join_key, ins.<span class="op">*</span> <span class="kw">FROM</span> <span class="op"><</span>upd_table<span class="op">></span></span>
<span id="cb8-10"><a href="#cb8-10" aria-hidden="true" tabindex="-1"></a>) <span class="kw">AS</span> sub</span>
<span id="cb8-11"><a href="#cb8-11" aria-hidden="true" tabindex="-1"></a><span class="kw">ON</span> (sub.join_key <span class="op">=</span> <span class="op"><</span>target_table<span class="op">></span>.join_key</span>
<span id="cb8-12"><a href="#cb8-12" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> <span class="op"><</span>target_table<span class="op">></span>.<span class="op"><</span>end_date<span class="op">></span> <span class="kw">IS</span> <span class="kw">NULL</span>)</span>
<span id="cb8-13"><a href="#cb8-13" aria-hidden="true" tabindex="-1"></a><span class="cf">WHEN</span> MATCHED</span>
<span id="cb8-14"><a href="#cb8-14" aria-hidden="true" tabindex="-1"></a> <span class="cf">THEN</span> <span class="kw">UPDATE</span> <span class="kw">SET</span> <span class="op"><</span>end_date<span class="op">></span> <span class="op">=</span> <span class="fu">current_date</span>()</span>
<span id="cb8-15"><a href="#cb8-15" aria-hidden="true" tabindex="-1"></a><span class="cf">WHEN</span> <span class="kw">NOT</span> MATCHED</span>
<span id="cb8-16"><a href="#cb8-16" aria-hidden="true" tabindex="-1"></a> <span class="cf">THEN</span> <span class="kw">INSERT</span> <span class="kw">VALUES</span> (sub.col1, <span class="op">..</span>. , sub.<span class="kw">hash</span>) ;</span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Hive Merge — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/spark-reflexionSpark Reflexions2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-05-13</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="spark-reflexions">Spark Reflexions</h1>
<p>Spark is a Swiss Army Knife of a hadoop stack. It makes glue between
tools and is a de-facto interface for them.</p>
<p>Spark has a lot of advantages:</p>
<ul>
<li>it's a wonderfull connector: if you look at a way to connect to a
tool, spark already has a connector for you. Special mention to csv:
spark is also able to read multilined csv with quote, empty lines and
special character in it.</li>
<li>it can manipulate very huge dataset: in case the dataset does not
feat in memory, the spark partitionning feature allows spliting the
tasks in working pipelines</li>
<li>its parallel feature can take advantage of a hadoop cluster, but is
a great solution for a laptop too</li>
<li>the scala spark-shell is really handy for development. It's
auto-completion features are superior to any shell I've dealt with
before</li>
<li>it has SQL syntax, SPARKQL compliance, and also machine learning
distributed implementation.</li>
<li>it can be programmed in scala, python, R, java and julia.</li>
</ul>
<p>Spark as some disadvantages:</p>
<ul>
<li>time of writing, spark does not support hive ACID tables. The
workaround is to do a major compaction of the table before reading it
with spark. This means it cannot yet be integrated into a hive ETL
process based on ACID tables.</li>
<li>spark programming needs a lot of skills. It is then not a good
choice for intermediate enterprise in comparison to hive programming.
Again, hive ETL looks a better fit for a maintainability reason.</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Spark Reflexions — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/hbase-reflexionHbase Reflexions2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-05-13</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="hbase-reflexions">Hbase Reflexions</h1>
<p>hbase looks a powerfull tool in complement of hive. While hive suits
well for ETL and data historisation, it cannot offer sub-second access
to thousand of concurrent users.</p>
<p>Hbase comes with a powerful compagnion aka Phoenix that provides many
features, while keeping the hbase features intact. Phoenix provides a
jdbc driver to hbase. This distinction adds many improvements:</p>
<ol type="1">
<li>it offers sql, joins, secondary indexes, transactions, sequences on
top of hbase.</li>
<li>it simplifies hive, spark, solr, and general programming access to
hbase data.</li>
</ol>
<h2 id="jdbc-overview">JDBC overview</h2>
<p>jdbc access can be done with <a href="https://community.hortonworks.com/questions/47138/phoenix-query-server-connection-url-example.html">kerberos</a>
on the form</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode java"><code class="sourceCode java"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="st">"jdbc:phoenix:thin:url=<scheme>://<server-hostname>:<port>;authentication=SPNEGO;principal=my_user;keytab=/home/my_user/my_user.keytab"</span></span></code></pre></div>
<p>Phoenix connection thought JDBC needs : - a jdbc jar file - the
hbase-site.conf - a kerberos ticket</p>
<p>Phoenix does not provide a way for connection pooling. The costly
part of the connection is the hbase link. However it is cached within
the region servers and a new phoenix jdbc connection can be rebuild for
every query since it is a lightweight object.</p>
<h2 id="hive-integration">Hive integration</h2>
<p>Phoenix tables can be mounted into hive thanks to a <a href="https://phoenix.apache.org/hive_storage_handler.html">recent
plugin</a>. In comparaison to hbase plugin, this allows fast join to
hive table (to be tested). While this plugin needs phoenix 4.8.0+ HDP
ships with phoenix 4.7.0. However HDP Phoenix is a fork of Phoenix, and
it integrates this feature.</p>
<p>Because phoenix allows to mount a hbase table, hive a phoenix or a
hbase table, there is many ways to make hive querying hbase. Moreover
hive can create a regular table stored as a hbase or phoenix table wich
is slightly different. The best way is to mount phoenix as a hive
external table. First advantage is <em>phoenix</em> allows salted tables
by mean there is no need to manually handle the key to be well
distributed. Second, is loading directly into <em>phoenix</em> allows it
to manage its secondary indexes and also some other tools such
<em>solr</em>.</p>
<p>Finally, hive can access an external <em>phoenix</em> table and allow
both upsert and select. The plugin also provides a way to delete/update
from hive, however I haven't been able to reproduce yet since
transactional tables are only compatible with orc based tables.</p>
<h2 id="spark-integration">Spark integration</h2>
<p>Spark can read and write from phoenix <a href="https://stackoverflow.com/questions/40329968/apache-spark-ways-to-read-and-write-from-apache-phoenix-in-java">here
on SOF</a> or <a href="https://phoenix.apache.org/phoenix_spark.html">here on
official</a>. The good point is it can be done from scala or python.</p>
<h2 id="solr-integration">Solr integration</h2>
<p>Solr integration with hbase is a good idea: hbase contains the truth
and solr allows fast lookup to discover it. However, setting up solr on
top of hbase is a pain. Phoenix simplifies drastically that : <a href="https://nicholasmaillard.wordpress.com/2014/12/27/phoenix-to-solr-in-20-minutes/">here</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Hbase Reflexions — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/prestodb-reflexionPrestoDB reflexions2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-05-13</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="prestodb-reflexions">PrestoDB reflexions</h1>
<p>Prestodb is a cool distributed engine over hadoop. It can be compared
to hive LLAP, impala or apache DRILL for fast in-memory OLAP analysis
and data discovery.</p>
<p>The main disadvantages it has right now are:</p>
<ul>
<li>it is not ACID compliant and far from being: this means tables in
presto shall be created from scratch for analytics. Because we are
speaking about big data, this implies copying</li>
<li>it does not and never will support hive views: this would have been
an opportunity to create pseudo-acid workflows</li>
<li>it does not integrate with hbase</li>
<li>it does not integrate with YARN, or at least the attempt looks
discontinued on github.</li>
</ul>
<p>The main advantage it has over hive LLAP are:</p>
<ul>
<li>many more build-in functions</li>
<li>has been proven to scale well with tons of users</li>
<li>integrates with RDBMS such postgres or NOSQL stores such
mongodb</li>
</ul>
<p>Sor far, the best feet for our internal arhitecture is to go with
hive llap in replacement to prestodb. This will brings us the ability to
better integrate with hbase/phoenix/solr from one part, and also give us
the ability to query ACID version of the DWH without data
deduplication.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=PrestoDB reflexions — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/yarn-reflexionYarn Reflexions2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-05-16</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="yarn-reflexions">Yarn Reflexions</h1>
<p>Here are some Apache Yarn consideratoins.</p>
<h2 id="yarn-is-so-great">Yarn is so great</h2>
<dl>
<dt>Queue modifications</dt>
<dd>
<p>Some application needs extra cores/memory and need to be boosted
somehow? Yarns allows on the fly adjustement of queues. In our case, we
have nightly ETL and online users the day. It's then easy to dispatch
the resources where needed, and without turning off running
application.</p>
</dd>
<dt>GPU management</dt>
<dd>
<p>Hadoop 3 will provide a new dimension in yarn responsibility. Yarn
will be able to allocate GPU resources for users. This will be of
incredible help for machine learning programs and users to share and
optimize access to them.</p>
</dd>
</dl>
<h2 id="yarn-has-some-drawbacks">Yarn has some drawbacks</h2>
<dl>
<dt>Applications logs</dt>
<dd>
<p>They are stored locally. Same case for temporary files on the local
filesystem such map results. And there is no trivial way to store point
to hdfs. This is an essential point when configuring the cluster,
because the disk allocated for storing yarn user files (hive, spark....)
is a real bottleneck. Depending on the cluster workload, keep in mind
having a minimum of 2TO for this stuff, per machine.</p>
</dd>
<dt>Yarn superuser</dt>
<dd>
<p>There is now way to kill a process if you are not the owner of the
process. This makes things a bit more complex for an admin to supervise
such platform.</p>
</dd>
</dl>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Yarn Reflexions — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/termiteBest terminal Emulator2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-05-27</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="best-terminal-emulator">Best terminal Emulator</h1>
<p>Termite is way better than my previous terminals. It's sensibly based
on vim paradigm. It's able to look ahead in the history, copy from it
and back to insert mode.</p>
<p>An other great aspect is it notifies when something happens in it,
such end of process, new mail an so on.</p>
<h2 id="keybinding">keybinding</h2>
<ul>
<li><em>ctrl+space</em> : insert mode</li>
<li><em>escape</em> : normal mode</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Best terminal Emulator — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/jupyter-notenbooks-and-condaJupyter Notebooks and Conda2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-06-02</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="jupyter-notebooks-and-conda">Jupyter Notebooks and Conda</h1>
<p>After creating a environment with conda and installing packages, the
jupyter notebook environment may not see them. The method below adds a
new kernel based on the later environment and allow to benefit from
conda:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">conda</span> activate my-env</span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="ex">conda</span> create <span class="at">-n</span> my-env pip setuptools ipykernel python=3.5</span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="ex">python</span> <span class="at">-m</span> ipykernel install <span class="at">--user</span> <span class="at">--name</span> my-env <span class="at">--display-name</span> <span class="st">"Python (my-env)"</span></span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Jupyter Notebooks and Conda — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/maven-localManaging local jars with
maven2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-06-02</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="managing-local-jars-with-maven">Managing local jars with
maven</h1>
<p>Maven has for me a non user friendly documentation, with 80% of xml
content. For long time I have been stuck in how easily manage both
dependency from maven central and from local jars.</p>
<p>I finally got something quite general and functional. Here is how to
make this working fine:</p>
<p>The pom.xml file:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode xml"><code class="sourceCode xml"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="co"><!-- Add the manifest --></span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><<span class="kw">plugin</span>></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> <<span class="kw">groupId</span>>org.apache.maven.plugins</<span class="kw">groupId</span>></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a> <<span class="kw">artifactId</span>>maven-jar-plugin</<span class="kw">artifactId</span>></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a> <<span class="kw">version</span>>2.4</<span class="kw">version</span>></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a> <<span class="kw">configuration</span>></span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a> <<span class="kw">archive</span>></span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a> <<span class="kw">manifest</span>></span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a> <<span class="kw">mainClass</span>>your.class.Main</<span class="kw">mainClass</span>></span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a> </<span class="kw">manifest</span>></span>
<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a> </<span class="kw">archive</span>></span>
<span id="cb1-12"><a href="#cb1-12" aria-hidden="true" tabindex="-1"></a> </<span class="kw">configuration</span>></span>
<span id="cb1-13"><a href="#cb1-13" aria-hidden="true" tabindex="-1"></a></<span class="kw">plugin</span>></span></code></pre></div>
<div class="sourceCode" id="cb2"><pre class="sourceCode xml"><code class="sourceCode xml"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"><!-- Make a standalone build --></span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><<span class="kw">plugin</span>></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a> <<span class="kw">groupId</span>>org.apache.maven.plugins</<span class="kw">groupId</span>></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <<span class="kw">artifactId</span>>maven-shade-plugin</<span class="kw">artifactId</span>></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> <<span class="kw">version</span>>2.2</<span class="kw">version</span>></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> <<span class="kw">executions</span>></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a> <<span class="kw">execution</span>></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a> <<span class="kw">phase</span>>package</<span class="kw">phase</span>></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a> <<span class="kw">goals</span>></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a> <<span class="kw">goal</span>>shade</<span class="kw">goal</span>></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a> </<span class="kw">goals</span>></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a> <<span class="kw">configuration</span>></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a> <<span class="kw">outputFile</span>></span>
<span id="cb2-14"><a href="#cb2-14" aria-hidden="true" tabindex="-1"></a> ${project.build.directory}/${artifactId}-${version}-standalone.jar</span>
<span id="cb2-15"><a href="#cb2-15" aria-hidden="true" tabindex="-1"></a> </<span class="kw">outputFile</span>></span>
<span id="cb2-16"><a href="#cb2-16" aria-hidden="true" tabindex="-1"></a> </<span class="kw">configuration</span>></span>
<span id="cb2-17"><a href="#cb2-17" aria-hidden="true" tabindex="-1"></a> </<span class="kw">execution</span>></span>
<span id="cb2-18"><a href="#cb2-18" aria-hidden="true" tabindex="-1"></a> </<span class="kw">executions</span>></span>
<span id="cb2-19"><a href="#cb2-19" aria-hidden="true" tabindex="-1"></a></<span class="kw">plugin</span>></span></code></pre></div>
<div class="sourceCode" id="cb3"><pre class="sourceCode xml"><code class="sourceCode xml"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="co"><!-- import your local jar--></span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><<span class="kw">dependency</span>></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> <<span class="kw">groupId</span>>org.hsqldb</<span class="kw">groupId</span>></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> <<span class="kw">artifactId</span>>jdbcDriver</<span class="kw">artifactId</span>></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> <<span class="kw">version</span>>2.4.1</<span class="kw">version</span>></span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a></<span class="kw">dependency</span>></span></code></pre></div>
<div class="sourceCode" id="cb4"><pre class="sourceCode xml"><code class="sourceCode xml"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="co"><!-- create a local respository --></span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><<span class="kw">repositories</span>></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a> <<span class="kw">repository</span>></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a> <<span class="kw">id</span>>my-local-repo</<span class="kw">id</span>></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a> <<span class="kw">url</span>>file://${basedir}/repos</<span class="kw">url</span>></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a> </<span class="kw">repository</span>></span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a></<span class="kw">repositories</span>></span></code></pre></div>
<p>By the way one can add dependency jars by such command line. This
will populate the <em>repos</em> folder:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="ex">mvn</span> org.apache.maven.plugins:maven-install-plugin:2.3.1:install-file</span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="ex">-Dfile=lib/hsqldb.jar</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="ex">-DgroupId=</span><span class="st">'org.hsqldb'</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="ex">-DartifactId=</span><span class="st">'jdbcDriver'</span></span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a><span class="ex">-DgeneratePom=true</span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a><span class="ex">-Dpackaging=jar</span></span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a><span class="ex">-Dversion=2.4.1</span></span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a><span class="ex">-DlocalRepositoryPath=repos</span></span></code></pre></div>
<p>As a result, the compilation will result as a standalone jar,
containing both local and central dependencies.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Managing local jars with
maven — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/loading-postgres-from-sqoopLoading Postgres from Sqoop2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-06-02</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="loading-postgres-from-sqoop">Loading Postgres from Sqoop</h1>
<p>PostgreSQL and Hadoop are complementary technologies. PostgreSQL
outperforms Hadoop to retrieve small amount of information from multiple
tables. Hadoop outperforms PostgreSQL to transform multiple tables into
simplfied and aggregated tables. So, how to load PostgreSQL efficiently
from Hadoop ?</p>
<h2 id="sqoop-as-a-bridge">Sqoop as a bridge</h2>
<p>Hadoop stores data on its distributed filesystem: HDFS. At the end of
the day, the resulting procecced data is stored into a hdfs table and
may need to be loaded into a PosgreSQL table.</p>
<p>Copy is a efficient a robust PostgreSQL bulkloading function allowing
to load among from others CSV files. It is way faster than a bunch of
parallel prepared insert statements.</p>
<p>ORC is a efficient and compressed Hadoop data format. It is both
faster to build and to read compared to CSV files.</p>
<p>As the bridge for Hadoop and relational databases, Sqoop manages all
ORC, CSV, COPY and INSERT methodologies. When dealing with PostgreSQL,
there is no perfect solution.</p>
<h2 id="sqoop-best-approach">Sqoop best approach</h2>
<p>There is two different scenario to load PostgreSQL.</p>
<dl>
<dt>structured columns</dt>
<dd>
<p>Sqoop does not allow to use COPY together with ORC tables but only
with CSV. It splits them and run multiple COPY statements from each part
in parallel from HDFS. The CSV format needs to be defined in the sqoop
parameters.</p>
</dd>
<dt>unstructured columns</dt>
<dd>
<p>Text information may contain new lines. In this situation, Sqoop is
stuck when it splits the CSV into multiple parts. Yet it is not possible
to use the sqoop direct mode, and the parallel inserts is the only way
to succeed. By chance, it is not necessar to transform the result as
CSV, and it is finally possible to exploit ORC tables as-is.</p>
</dd>
</dl>
<p>Writing this post makes me thinking about a potential workaround to
load textual CSV from COPY by using only one mapper in this situation.
Since COPY is highly optimized it is possible that it won't degrade the
performances while making the overall process totally robust.</p>
<p>EDIT:</p>
<p>Indeed, using only one mapper allows to handle multiline CSV. Since
COPY is very fast and has been optimized to import dataset into
postgreSQL, in the general case, the method can be generalized to any
kind of CSV to work in any situation. That's the ultimate way to load
PostgreSQL from Hadoop:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">sqoop</span> export<span class="dt">\</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>--connect <span class="st">"jdbc:postgresql://<postgres_host>/<postgres_db>"</span><span class="dt">\</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a>--username <span class="op"><</span>postgres_user<span class="op">></span><span class="dt">\</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a>--password-file file:///home/<span class="va">$USER</span>/.password<span class="dt">\</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a>--export-dir /apps/hive/warehouse/<span class="op"><</span>my_db<span class="op">></span>/<span class="op"><</span>my_csv_table<span class="op">></span><span class="dt">\</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a>--table <span class="op"><</span>my_postgres_table<span class="op">></span><span class="dt">\</span></span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a>--columns <span class="st">"id, text_column"</span><span class="dt">\</span></span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a>-m 1<span class="dt">\</span></span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a>--direct<span class="dt">\</span></span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a>--lines-terminated-by <span class="st">'\n'</span><span class="dt">\</span></span>
<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a>--fields-terminated-by <span class="st">','</span><span class="dt">\</span></span>
<span id="cb1-12"><a href="#cb1-12" aria-hidden="true" tabindex="-1"></a>--input-null-string <span class="st">"</span><span class="dt">\\\\</span><span class="st">N"</span><span class="dt">\</span></span>
<span id="cb1-13"><a href="#cb1-13" aria-hidden="true" tabindex="-1"></a>--input-null-non-string <span class="st">"</span><span class="dt">\\\\</span><span class="st">N"</span><span class="dt">\</span></span>
<span id="cb1-14"><a href="#cb1-14" aria-hidden="true" tabindex="-1"></a>-- <span class="at">--schema</span> <span class="op"><</span>my_schema<span class="op">></span></span></code></pre></div>
<p>In order to prepare the table ready to be loaded by sqoop, it can be
transformed with a hive sql query:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="kw">CREATE</span> <span class="kw">TABLE</span> <span class="op"><</span>my_db<span class="op">></span>.<span class="op"><</span>my_csv_table<span class="op">></span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a>STORED <span class="kw">AS</span> textfile</span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="kw">ROW</span> FORMAT DELIMITED</span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a>FIELDS TERMINATED <span class="kw">BY</span> <span class="st">','</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a>LINES TERMINATED <span class="kw">BY</span> <span class="st">'</span><span class="ch">\n</span><span class="st">'</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="kw">AS</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> <span class="op">*</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> <span class="op"><</span>my_hive_table<span class="op">></span></span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Loading Postgres from Sqoop — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/eclipse-tipsEclipse Tips2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-06-03</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="eclipse-tips">Eclipse Tips</h1>
<dl>
<dt>darkest theme</dt>
<dd>
<p>simply go throught install menu from this <a href="https://www.eclipse.org/community/eclipse_newsletter/2017/february/article5.php">link</a></p>
</dd>
</dl>
<dl>
<dt>emacs mode</dt>
<dd>
<p>select it as a key binding in the preferences. The good point is
emacs shortcuts are also used in most linux terminals. The bad point is
when managing <em>emacs</em> and <em>vi</em> is a kind of challenge in
term of reflexes. The next step would be managing both <em>azerty</em>
and <em>qwerty</em> keyboards.</p>
</dd>
</dl>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Eclipse Tips — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/nvidia-18.04Nvidia & ubuntu 18.042021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-08-19</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="nvidia-ubuntu-18.04">Nvidia & ubuntu 18.04</h1>
<p>My new gigabyte aero 14 does ship a nvidia GTX1050. I had several
problem to make it working correctly.</p>
<h2 id="installation-from-scratch">Installation from scratch</h2>
<p>When installing ubuntu from my usb key it simply froze after setup. I
had to enter in maintenance mode, log as root and the resume to get a
VGA screen. After that I was able to install the nvidia drivers by
folowing the steps <a href="https://www.linuxbabe.com/ubuntu/install-nvidia-driver-ubuntu-18-04">the
folowing steps</a></p>
<h2 id="turning-in-nvidia-mode">Turning in nvidia mode</h2>
<p>The command <em>sudo prime-select nvidia</em> followed by a reboot
makes use of the nvidia chipset. As a result, the powertop tool show a
use of 15W.</p>
<h2 id="turning-intel-mode">Turning intel mode</h2>
<p>The command <em>sudo prime-select intel</em> followed by a reboot
makes use of the intel chipset. However a bug introduced with 18.04
makes both nvidia and intel running and it uses 20W and worst did not
allow to shutdown/reboot/suspend/hibernate. I ended up by using that <a href="https://www.linuxbabe.com/ubuntu/install-nvidia-driver-ubuntu-18-04">solution</a>
which reduce to 7W. Note that I had to make sure the grub was
<em>nouveau.modeset=0 nouveau.runpm=0</em> in the
<em>/etc/default/grub</em> file. After installed the <em>tlp</em> tool I
ended up using 5W.</p>
<p>Note that I tried <a href="https://bugs.launchpad.net/ubuntu/+source/nvidia-prime/+bug/1765363">that
other solution</a> but this resulted in system freeze at reboot.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Nvidia & ubuntu 18.04 — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/mutt-aesEncrypt & Sign Mails2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-08-25</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="encrypt-sign-mails">Encrypt & Sign Mails</h1>
<p>Cacert.org offers validated certificates with a network of trusted
people. That's fun !</p>
<h2 id="get-a-certificate-eg-cacert.org">Get a certificate (eg:
CAcert.org)</h2>
<ul>
<li>Client Certificate > new</li>
<li>add it to Firefox</li>
</ul>
<p>The key pair can then be extracted in p12 format from Firefox and it
is possible to transform it to rsa format:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">openssl</span> pkcs12 <span class="at">-in</span> ~/Downloads/cacert.p12 <span class="at">-out</span> /home/<span class="va">$USER</span>/.ssh/id_rsa_cacert</span></code></pre></div>
<h2 id="installing-neomutt-from-source-on-ubuntu-18.04">Installing
neomutt from source on Ubuntu 18.04</h2>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> apt install libncursesw5-dev</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> apt install libidn11-dev</span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="fu">git</span> clone https://github.com/neomutt/neomutt</span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="bu">cd</span> neomutt</span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="ex">./configure</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="fu">make</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> make install</span></code></pre></div>
<h2 id="loading-the-key-into-mutt">Loading the key into mutt</h2>
<p>It is necessary to setup and use smime_key command to manage keys. It
will generate a KeyID for the certificate (something like xxxxxx.0)</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">cp</span> /home/<span class="va">$USER</span>/git/neomutt/contrib/smime.rc ~/.smime.rc</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="ex">/usr/libexec/neomutt/smime_keys</span> init</span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="fu">mkdir</span> <span class="at">-p</span> ~/.smime/certificates</span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="fu">touch</span> ~/.smime/certificates/.index</span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="fu">mkdir</span> <span class="at">-p</span> ~/.smime/keys</span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a><span class="fu">touch</span> ~/.smime/keys/.index</span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a><span class="ex">/usr/libexec/neomutt/smime_keys</span> add_p12 ~/Download/cacert.p12</span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a><span class="ex">/usr/libexec/neomutt/smime_keys</span> refresh</span></code></pre></div>
<ul>
<li>Add to <span class="title-ref">source /home/$USER/.smime.rc</span>
to .muttrc</li>
<li>Setup the KeyID in the <em>.smime.rc</em> file</li>
</ul>
<h2 id="send-a-signed-mail">Send a signed mail</h2>
<p>After having written the mail in neomutt type <span class="title-ref">shift-S</span> and choose the <span class="title-ref">s</span> to sign the mail. It will choose the KeyID by
default, and ask for the key password if exists.</p>
<h2 id="loading-a-public-key">Loading a public key</h2>
<p>Right now, I don't know</p>
<h2 id="encrypt-a-mail">Encrypt a mail</h2>
<p>Choose <span class="title-ref">e</span> for encryption, and then
<span class="title-ref">enter</span> to list the existing public
keys.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Encrypt & Sign Mails — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/mutt-sidebarMutt & Sidebar2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-08-26</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="mutt-sidebar">Mutt & Sidebar</h1>
<p>When receiving many mails a day, having a single mail pool makes easy
to loose important emails. Using mailboxes (ie: sub folders) with rules
becomes a need. The good point is all the mailboxes are regularly
updated to alert when incoming email are received. The few steps are
generating a mailbox tree, mooving mails manually, mooving them
automatically, and finally mooving the history in batch.</p>
<h2 id="generating-a-mailbox-tree">Generating a mailbox tree</h2>
<p>Mailboxes consists of folder{cur, new, old} where <em>folder</em>
represents its name. The easy way to build them is from the mutt
interface, by pressing <span class="title-ref">s</span> on an email and
choosing the new mailboxe name. The root of the mailboxe is defined as
the <span class="title-ref">folder</span> variable.</p>
<h2 id="mooving-mails-manually">Mooving mails manually</h2>
<p>It is possible to redirect an email into a mailboxe by pressing <span class="title-ref">s</span> and then <span class="title-ref">=myMailBoxeChoice</span>. In case you want to put it
back to the general spool just type <span class="title-ref">=</span>
with an empty mailboxe.</p>
<h2 id="mooving-mails-automatically">Mooving mails automatically</h2>
<p>I have been able to write rules based on email features such <span class="title-ref">from</span> with the <a href="https://userpages.umbc.edu/~ian/procmail.html#tutorial">helpful
tutorial</a>. In my setup, <em>getmail</em> fetches the mails and
invokes <em>procmail</em> to redirect the mail somewhere. The idea is to
parse the email address with a regex and depending on the result
redirect it in the right <span class="title-ref">new</span> mailboxe
folder.</p>
<h2 id="mooving-mails-history">Mooving mails history</h2>
<p>I haven't yet a solution for that.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mutt & Sidebar — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/mutt-notmuchMutt Search on Steroids2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-08-30</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="mutt-search-on-steroids">Mutt Search on Steroids</h1>
<p>Mutt has a powerful feature to dig into the history: the
<strong>limit</strong> feature. By pressing <span class="title-ref">l</span> it is then possible to write a query to
filter the mails. It is comparable to the <strong>WHERE</strong> part of
a SQL statement and allow combining <strong>AND</strong> operators. In
particular the <strong>~b</strong> predicate allow to search within the
email's body and the <span class="title-ref">~X</span> lets filter
attachments.</p>
<p>However that feature acts as a <strong>grep</strong> command and
parse all the mails one by one. Searching in multiple dozen of thousand
emails can last dozen of minutes. Here comes the
<strong>notmuch</strong> program, that improves greatly the limit
feature with instant search and some other goodies (and also drawbacks).
The secret of this speed is it scan emails and produces a full text
index to be used later.</p>
<p>The installation is straightforward on ubuntu with <span class="title-ref">sudo apt install notmuch</span>. The configuration is
also piece of cake: <span class="title-ref">notmuch config</span> will
setup the maildir folder. Later the <span class="title-ref">notmuch
new</span> command allows to the mails and can be run again when new
mails arrive. There is also a way to automatize the indexation by setup
system notifications <a href="https://github.com/noah/notmuch-new-inotify">right there</a>.</p>
<p>The integration in mutt to replace the <em>limit</em> feature only
need the two rows below in the <em>.muttrc</em> configuration file.
Pressing <span class="title-ref">L</span> will propose to write a query
to filter the mails based on the index <a href="http://log.or.cz/?p=228">more information here</a>. To get the
full details on how to search within notmuch: <span class="title-ref">man notmuch-search-index</span> describes the syntax
of the search.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="co"># 'L' performs a notmuch query, showing only the results</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="ex">macro</span> index L <span class="st">"<enter-command>unset wait_key<enter><shell-escape>read -p 'notmuch query: ' x; echo </span><span class="dt">\$</span><span class="st">x >~/.cache/mutt_terms<enter><limit>~i </span><span class="dt">\"\`</span><span class="st">notmuch search --output=messages </span><span class="dt">\$</span><span class="st">(cat ~/.cache/mutt_terms) | head -n 600 | perl -le '@a=<>;chomp@a;s/\^id:// for@a;s/</span><span class="dt">\\</span><span class="st">+/</span><span class="dt">\\\\</span><span class="st">+/g for@a;s/=/</span><span class="dt">\\\\</span><span class="st">=/g for@a;$,=</span><span class="dt">\"</span><span class="st">|</span><span class="dt">\"</span><span class="st">;print@a'</span><span class="dt">\`\"</span><span class="st"><enter>"</span> <span class="st">"show only messages matching a notmuch pattern"</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="co"># 'a' shows all messages again (supersedes default <alias> binding)</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="ex">macro</span> index a <span class="st">"<limit>all\n"</span> <span class="st">"show all messages (undo limit)"</span></span></code></pre></div>
<p>There is some caveats with notmuch however. Its search within
attachment is not as powerfull as the built-in mutt. In particular, the
later allows to specify the number of attachments. Last but not least,
notmuch does only provide a stemmer for english. That's might be
possible to extend it with an existing implementation of the french
snowball parser.</p>
<h2 id="edit">EDIT:</h2>
<p><strong>mu</strong> is an alternative to <strong>notmuch</strong> .
It covers the features described above, and has a better syntax.</p>
<p>Here is the full documentation there <a href="http://manpages.ubuntu.com/manpages/bionic/man1/mu-find.1.html">mu
documentation</a></p>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Install</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> apt install maildir-utils</span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="co"># Index Maildir</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="ex">mu</span> index <span class="at">-m</span> my/maildir</span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="co"># setup in crontab or use inotify service</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="co"># to add to .muttrc</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a><span class="co"># Mode 1.</span></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a><span class="co">#</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a><span class="co"># this allows to search into all mailbowes at once</span></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a><span class="co"># however, it is a readonly mailbox because off symlinks</span></span>
<span id="cb2-14"><a href="#cb2-14" aria-hidden="true" tabindex="-1"></a><span class="ex">macro</span> index <span class="op"><</span>F8<span class="op">></span> <span class="st">"<shell-escape>mu find --clearlinks --format=links --linksdir=~/Maildir/search "</span> <span class="st">"mu find"</span></span>
<span id="cb2-15"><a href="#cb2-15" aria-hidden="true" tabindex="-1"></a><span class="ex">macro</span> index <span class="op"><</span>F9<span class="op">></span> <span class="st">"<change-folder-readonly>~/Maildir/search"</span> <span class="st">"mu find results"</span></span>
<span id="cb2-16"><a href="#cb2-16" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-17"><a href="#cb2-17" aria-hidden="true" tabindex="-1"></a><span class="co"># Mode 2.</span></span>
<span id="cb2-18"><a href="#cb2-18" aria-hidden="true" tabindex="-1"></a><span class="co">#</span></span>
<span id="cb2-19"><a href="#cb2-19" aria-hidden="true" tabindex="-1"></a><span class="co"># this allows to show the 600 first emails</span></span>
<span id="cb2-20"><a href="#cb2-20" aria-hidden="true" tabindex="-1"></a><span class="co"># it is</span></span>
<span id="cb2-21"><a href="#cb2-21" aria-hidden="true" tabindex="-1"></a><span class="ex">macro</span> index L <span class="st">"<enter-command>unset wait_key<enter><shell-escape>read -p 'mu query: ' x; echo </span><span class="dt">\$</span><span class="st">x >~/.cache/mutt_terms<enter><limit>~i </span><span class="dt">\"\`</span><span class="st">mu find --fields i --quiet 2> /dev/null </span><span class="dt">\$</span><span class="st">(cat ~/.cache/mutt_terms) | head -n 600 | perl -le '@a=<>;chomp@a;s/\^id:// for@a;s/</span><span class="dt">\\</span><span class="st">+/</span><span class="dt">\\\\</span><span class="st">+/g for@a;s/=/</span><span class="dt">\\\\</span><span class="st">=/g for@a;$,=</span><span class="dt">\"</span><span class="st">|</span><span class="dt">\"</span><span class="st">;print@a'</span><span class="dt">\`\"</span><span class="st"><enter>"</span> <span class="st">"show only messages matching a mu pattern"</span></span>
<span id="cb2-22"><a href="#cb2-22" aria-hidden="true" tabindex="-1"></a><span class="co"># 'a' shows all messages again (supersedes default <alias> binding)</span></span>
<span id="cb2-23"><a href="#cb2-23" aria-hidden="true" tabindex="-1"></a><span class="ex">macro</span> index a <span class="st">"<limit>all\n"</span> <span class="st">"show all messages"</span></span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mutt Search on Steroids — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/mutt-vimspellSpelling, mutt & vim2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-09-03</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="spelling-mutt-vim">Spelling, mutt & vim</h1>
<p>Having a correct spelling is important in society. This is true
specially when come time to write emails.</p>
<p>Here is a trick to activate vim spelling while editing email in
<strong>mutt</strong>. Add the following line into your .muttrc
configuration file:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="bu">set</span> editor=<span class="st">"vim -c 'set textwidth=72' -c':set spell' -c':setlocal spell spelllang=en_us'"</span></span></code></pre></div>
<p>This allows to check English spelling correctly.</p>
<p>Enjoy !</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Spelling, mutt & vim — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/email-best-practicesEmail Best Practices2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-09-03</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="email-best-practices">Email Best Practices</h1>
<p>I will post here all best practices about writing emails. A lot of
useful information can be found <a href="https://www.ietf.org/rfc/rfc3676.txt">there</a></p>
<h2 id="writing-email">Writing email</h2>
<ol>
<li><p>Choose an email object</p>
<p>For example this pattern is great: <strong>[Action] Topic</strong>.
<strong>Action</strong> to help your recipient to understand what you
want from him. <strong>topic</strong> is a small synthesis of the topic
in which the action is supposed to happen. This pattern has been advised
by Michael Kahn.</p></li>
<li><p>Keep the body narrow</p>
<p>The email should not size more than 72 character width. This improve
readability specially on small screen. In vim, the width can be setup,
and the shorkey <strong>g+q</strong> on a visual selection makes things
easy.</p></li>
</ol>
<h2 id="reply-to-email">Reply to email</h2>
<ol>
<li><p>Reply below your recipient</p>
<p>Bottom Posting (BP) is answering bellow the previous email as
opposite to Top Posting (TP) aka gmail way of doing. BP has many
advantages over TP. BP allows to read the emails in the natural order ;
specially for people who discover the thread. <a href="http://www.idallen.com/topposting.html">more information here</a>
Keep an empty row between the previous answer and yours for
readability.</p></li>
<li><p>Remove information</p>
<p>Remove the recipient signature, out of context content. In
particular, do not keep the recipient email within the email body for
security reasons.</p></li>
<li><p>Append your signature</p>
<p>A signature is generally on the bottom of the email, preceded by a
row containing: <em>--</em></p></li>
<li><p>Sign the email</p>
<p>Sign the email with your AES private key as a proof of your
identity.</p></li>
<li><p>Mask your mailing system</p>
<p>Email can store the <strong>User-agent</strong> field. This can be
found in the header of the email and may share precious information such
your operating system and its version. eg: <strong>User-Agent:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
Thunderbird/52.3.0</strong></p></li>
</ol>
<h2 id="mailing-list">Mailing List</h2>
<ol>
<li><p>Email the mailing list <strong>only</strong></p>
<p>Remove the previous writer. In any cases, he will receive the email
since he subscribed to the list.</p></li>
<li><p>Use a dedicated alias</p>
<p>Mailing list are usually public and this you will probably end up
with spam. A good trick is to use an alias combined with a rule such:
"message directed to the alias goes into the trash". Riseup.net provides
alias email.</p></li>
</ol>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Email Best Practices — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/hive-llap-resourceHive Llap2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-09-07</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="hive-llap">Hive Llap</h1>
<h2 id="resource-consumption">Resource consumption</h2>
<p>Hive llap is quite complicated to tune.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">hive.llap.daemon.task.scheduler.enable.preemption=true</span></span></code></pre></div>
<p><a href="https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html">a
great article</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Hive Llap — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/archlinux-pg11Postgresql 11 on Archlinux2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-09-27</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="postgresql-11-on-archlinux">Postgresql 11 on Archlinux</h1>
<p>I am fairly new to archlinux. Postgresql 11 is fairly new too. Here
are the steps to install that version.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="co"># get the lattest postgres in the repos</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="co"># avoid checking for sha fingerprint, because not the same version</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="ex">yaourt</span> <span class="at">--m-arg</span> <span class="st">"--skipchecksums --skippgpcheck"</span> <span class="at">-Sb</span> postgresql</span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="co"># 1. edit the PKGBUILD file, change to 11.0 version</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="co"># 2. comment the check() part</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a><span class="co"># 3. after building and installing, change to 11 the PGMAJORVERSION within:</span></span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> vim /usr/bin/postgresql-check-db-dir</span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a><span class="co"># make the link for psql command working</span></span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> ln <span class="at">-s</span> /usr/lib/libreadline.so.7 /usr/lib/libreadline.so.6</span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a><span class="co"># install that old version to make posgres command work</span></span>
<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a><span class="ex">yaourt</span> <span class="at">-S</span> icu54</span></code></pre></div>
<p>That's all. By the way, I do not have any idea right now on how to
upgrade minor versions in the future, but I will employ myself to update
this post.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Postgresql 11 on Archlinux — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/deploying-with-githup-pagesDeploying Nikola with Githup
Pages2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-11-11</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="deploying-nikola-with-githup-pages">Deploying Nikola with Githup
Pages</h1>
<h2 id="the-simple-way">The simple way</h2>
<p>Its about using the <em>docs</em> github pages method.</p>
<ol>
<li>create a new github repository</li>
<li>create a <em>/docs/</em> folder with the website in it</li>
<li>enable github pages and specify the <em>docs</em> folder as html
source</li>
</ol>
<h2 id="why-the-simple-way">Why the simple way</h2>
<p>The alternate way is to use a <em>gh-pages</em> branch to store the
website. The main problem is the method leads to highly complex
deployment method with <a href="http://www.damian.oquanta.info/posts/one-line-deployment-of-your-site-to-gh-pages.html">git
subtree</a> or <a href="https://www.asmeurer.com/blog/posts/moving-to-github-pages-with-nikola/">other
way</a></p>
<p>A second problem is this method make either duplicate website source
code, or makes building the website for each post.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Deploying Nikola with Githup
Pages — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/a-decent-python-configurationA decent python
configuration2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-11-11</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="a-decent-python-configuration">A decent python
configuration</h1>
<h2 id="use-anaconda">Use Anaconda</h2>
<p>Anaconda allows to have multiple python versions. This is much easier
than using <em>virtualenv</em></p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">conda</span> env list <span class="co"># lists all envs</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="ex">conda</span> env create py3.5 pip setuptools python=3.5 <span class="co"># create a python env</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="ex">conda</span> activate py3.5 <span class="co"># activate the latter</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="ex">conda</span> deactivate <span class="co"># deactivate it</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="ex">conda</span> env remove <span class="at">-n</span> py3.5 <span class="co"># remove it completely</span></span></code></pre></div>
<h2 id="use-pip-together-with-anaconda">Use pip together with
anaconda</h2>
<p>The package will be installed within the environement. pip has a much
broader list of packages than the anaconda channel.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="ex">pip</span> install <span class="op"><</span>my-package<span class="op">></span> # install a package</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="ex">pip</span> install <span class="at">--index-url</span> https://pypi.org/simple/ <span class="op"><</span>my-package<span class="op">></span> # specify the repository</span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="ex">pip</span> install . <span class="co"># a local package</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="ex">pip</span> remove <span class="op"><</span>my-package<span class="op">></span> # remove it</span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=A decent python
configuration — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/pyspark-considerationsExtracting raw data from
hdfs2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-11-22</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="extracting-raw-data-from-hdfs">Extracting raw data from
hdfs</h1>
<p>I have tested multiple ways to get data from hdfs. There is two
situations:</p>
<h2 id="the-data-fits-in-a-local-node-ram-memory">The data fits in a
local node RAM memory:</h2>
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="co"># will produce a local csv</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>sql(<span class="st">"select * from my_table where true"</span>)</span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> .toPandas()</span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a> .to_csv(<span class="st">"myLocalFile.csv"</span>, encoding<span class="op">=</span><span class="st">"utf8"</span>)</span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="co"># otherwise spark shell keeps running</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a>exit(<span class="dv">0</span>)</span></code></pre></div>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># run the python script:</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="va">PYTHONSTARTUP</span><span class="op">=</span>my/python/path/prog.py <span class="ex">pyspark</span> <span class="at">--master</span> yarn [...]</span></code></pre></div>
<h2 id="the-dataset-does-not-fit-into-ram-memory">The dataset does not
fit into RAM memory:</h2>
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="co"># will produce some csv on hdfs</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a>sql(<span class="st">"select * from my_table where true"</span>)</span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> .write()</span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> .<span class="bu">format</span>(<span class="st">"csv"</span>)</span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> .save(<span class="st">"/output/dir/on/hdfs/"</span>)</span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a><span class="co"># otherwise spark shell keeps running</span></span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a>exit(<span class="dv">0</span>)</span></code></pre></div>
<div class="sourceCode" id="cb4"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="co"># run the python script:</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="va">PYTHONPATH</span><span class="op">=</span>my/python/path/prog.py <span class="ex">pyspark</span> <span class="at">--master</span> yarn [...]</span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="co"># merge the files on the local filesystem</span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="ex">hadoop</span> fs <span class="at">-getmerge</span> /output/dir/on/hdfs/ /desired/local/output/file.csv</span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Extracting raw data from
hdfs — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/rendering-patients-in-solrRendering Patients in SolR
(part-1)2021-11-07T15:49:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-12-02</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="rendering-patients-in-solr-part-1">Rendering Patients in SolR
(part-1)</h1>
<p>Patient medical records contain a broad variety of data such numeric,
coding, time-series or medical reports. It can be translated as a json
document with a nested structure.</p>
<p>SolR allows to handle highly nested objects to be queried
efficiently. It provides pagination, and also faceting features wich are
very powerful to provide a patient centered search engine.</p>
<p>Patients in electronic health records:</p>
<p><img src="/images/patient_nested.webp" class="align-center" style="width:200.0%" alt="Patient Nested"></p>
<p>Consider the two patients below:</p>
<div class="sourceCode" id="cb1" data-code="json
"><pre class="sourceCode json"><code class="sourceCode json"></code></pre></div>
<p>It yet possible to ask relevant questions to SolR:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Count and give me one patient having during the same visit:</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="co"># - one XX1 code</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="co"># - one report containing "popeye"</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="ex">curl</span> http://localhost:8983/solr/patient/query <span class="at">-d</span> <span class="st">'</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="st">q=*:*</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="st">&fq={!parent which="content_type_s:visite"}</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="st"> ccam_ss:XX1</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a><span class="st">&fq={!parent which="content_type_s:visite"}</span></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="st"> {!complexphrase inOrder=true}</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a><span class="st"> doc_text_t:"popeye marche"~3</span></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a><span class="st"> AND doc_type_ss:APHP.CRH-H</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a><span class="st">&fl=id</span></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a><span class="st">&rows=1'</span></span></code></pre></div>
<p>Using blockJoin queries has some limitations: - it is not possible to
highlight matching text from a parent or a child document - it has some
impact on performances</p>
<p>The particular model used in this post has some limitations: - it is
not possible to highlight matching text at the patient level - it is not
possible to play with a sequence of events, such of encounter - queries
are complicated</p>
<p>Let's see a other approach in a next post.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Rendering Patients in SolR
(part-1) — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-07 15:49</time> </div>https://blog.parisni.com/blog/rendering-patients-in-solr-part-2Rendering Patients in SolR
(part-2)2021-11-07T15:49:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-12-04</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="rendering-patients-in-solr-part-2">Rendering Patients in SolR
(part-2)</h1>
<p>A patient search engine needs to meet several criteria: #. blazing
fast #. handle both structured and unstructured data #. for free text
retrieval: include NLP pipelines results #. define complex sequence of
events, on patient, and/or encounter level #. highlight results</p>
<p>The <a href="https://parisni.github.io/weblog/posts/rendering-patients-in-solr">previous
post</a> was a first attempt to modelize a patient in SolR. While it
offered solution for the three first crieteria, it was ineficient to
provide event based reasonning and also highlighting results. The below
method tries to answer more elegantly the question.</p>
<p>The idea is to modelize the patient as a list of nested encounters
with the last encounter beeing the first and vice-versa. Each encounter
object being a flat and denormalized document. The design allows to
reason sequence of events and also to highlight the part of the texts
that match the query.</p>
<p><img src="/images/encounter_nested.webp" class="align-center" style="width:200.0%" alt="Encounter Nested"></p>
<p>The denormalization has several advantages in SolR such speading up
the queries and also simplifying them. By combining multivalued texts
fields and path based text values the design pattern offers power and
simplicity. An other aspect of denormalization is to repeat the patient
demographic information in each encounter, to enable filtering on this
level, and also faceting on patients details.</p>
<p>Having nested encounters allows reasoning on sequence of event by
using blockJoin. Indeed this offers the opportunity to add query filter
based on childs. The idea is to add a level in each encounter by
counting the number of days from the last encounter. It is yet possible
to ask for: A patient having a breast cancer and whoom had a chirurgical
mamal implant within the 2 years.</p>
<p>One drawback of this design is it does not bring back patients but
encounters. One solution is to use faceting capabilities to count the
number of unique patients within the cohort.</p>
<p>EDIT: The method has been disapointing:</p>
<p>#. a parent object shall have a different nature than childs. It is
then not possible make a be a parent at any level of the chain. #. only
a parent object can be highlighted. While it is possible to reason on
childs content, it is not possible to highlight them</p>
<p>Let's invent a better method</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Rendering Patients in SolR
(part-2) — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-07 15:49</time> </div>https://blog.parisni.com/blog/rendering-patients-in-solr-part-3Rendering Patients in SolR
(part 3)2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-12-13</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="rendering-patients-in-solr-part-3">Rendering Patients in SolR
(part 3)</h1>
<p>The two previous attempts have failed to bring enough power and
flexibility in searching within patient history.</p>
<p>The current approach allows:</p>
<ol>
<li><em>highlighting</em> all the relevant information at the encounter
level</li>
<li>mixing <em>logical operators</em> (OR,AND,NOT) between
encounters</li>
<li>mixing <em>temporal order</em> between encounters</li>
</ol>
<p>First the process of building cohorts changes to:</p>
<ol>
<li>define the constraints on one encounter</li>
<li>eventually add patient level constraints</li>
<li>look for those matching encounters</li>
<li>loop 1. to 3. to define other encounters</li>
<li>add logical links between encounters/groups</li>
<li>add temporal links between encounters/groups</li>
</ol>
<p>A constraint on a encounter might be:</p>
<ul>
<li>a date</li>
<li>a delay</li>
<li>a number</li>
<li>a code</li>
<li>a code and a number</li>
<li>a code and a date</li>
<li>a complex natural langage query</li>
<li>a sequence of events</li>
<li>a combination of all of them (AND,OR,NOT)</li>
</ul>
<p>Constrained encounters can be then linked together. For example:</p>
<ul>
<li>encounterA: (37°C) => temp:37</li>
<li>encounterB: (36°C) => temp:36</li>
<li>encounterC: (35°C) => temp:35</li>
<li><dl>
<dt>encounterA AND encounterB (where encounterA = encounterB is
possible)</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37)&fq={!parent
which="type:encounter"}(temp:36)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>encounterA OR encounterB</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37) OR (temp:36)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>encounterA BEFORE encounterB</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37 AND
old.temp:36)&fq=(temp:36)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>(encounterA AND encounterB) BEFORE encounterC</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37)&fq={!parent
which="type:encounter"}(temp:36)&fq={!parent
which="type:encounter"}(temp:37 AND old.temp:35)&fq={!parent
which="type:encounter"}(temp:36 AND old.temp:35)&fq={!parent
which="type:encounter"}(temp:35)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>encounterA AND encounterB (where encounterA != encounterB)</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37 AND old.temp:36) OR
(temp:36 AND old.temp:37)&fq={!parent
which="type:encounter"}(temp:36)&fq={!parent
which="type:encounter"}(temp:37)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>(encounterA AND encounterB) BEFORE encounterC (where encounterA !=
encounterB)</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37 AND old.temp:36) OR
(temp:36 AND old.temp:37)&fq={!parent
which="type:encounter"}(temp:36)&fq={!parent
which="type:encounter"}(temp:37)&fq={!parent
which="type:encounter"}(temp:37 AND old.temp:35)&fq={!parent
which="type:encounter"}(temp:36 AND old.temp:35)&fq={!parent
which="type:encounter"}(temp:35)</li>
</ul>
</dd>
</dl></li>
</ul>
<h2 id="sacrifices">Sacrifices</h2>
<p>To enable the temporal/disjoint feature the documents are repeated in
a factorial way. This may lead to terrible situations. Only the textual
data should be stored. Moreover in each document, the historical
information should only be indexed and not stored. Even by doing this
the dimention of the index might be unreasonnable. An other problem is
the complexity to produce such documents: the serialization to json
might be a nightmare.</p>
<h2 id="less-ambitions">Less Ambitions</h2>
<p>By not enabling temporal/disjoint, what append's?</p>
<ul>
<li>encounterA: (37°C) => temp:37</li>
<li>encounterB: (36°C) => temp:36</li>
<li>encounterC: (35°C) => temp:35</li>
<li><dl>
<dt>encounterA AND encounterB (where encounterA = encounterB is
possible)</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37)&fq={!parent
which="type:encounter"}(temp:36)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>encounterA OR encounterB</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37) OR (temp:36)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>(encounterA OR encounterB) AND encounterC</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37) OR
(temp:36)&fq={!parent which="type:encounter"}(temp:35)</li>
</ul>
</dd>
</dl></li>
<li><dl>
<dt>(encounterA OR encounterB) NOT encounterC</dt>
<dd>
<ul>
<li>fq={!parent which="type:encounter"}(temp:37) OR
(temp:36)&fq=-_<a href="">query</a>:"{!parent
which="type:encounter"}(temp:35)"</li>
</ul>
</dd>
</dl></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Rendering Patients in SolR
(part 3) — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/multi-word-synonyms-in-solrMultiword synomyms in SolR2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-12-14</time></span>
<p>Estimated Time:<span id="reading-time">3 minutes</span></p>
<h1 id="multiword-synomyms-in-solr">Multiword synomyms in SolR</h1>
<p>Medical domain has multiple ways to tell about the same thing. It
becomes handy to use and maintain muliword synonyms thesauri.</p>
<p>SolR recently released a built-in solution to handle multi-word
synonyms and I have tested it successfully:</p>
<ul>
<li>synonyms are computed at query time <strong>only</strong>.</li>
<li>using multi-word synonyms are only compatible with
simple/lucene/eDismax query parsers right now.
<strong>complexphrase</strong> is broken with such dictionary.</li>
<li>multi-word synonyms are supposed to be used <strong>before
stemming</strong></li>
<li>adding new synonyms needs to delete the collection and reindex it.
(to be confirmed)</li>
</ul>
<p>Some questions remains: 1) How many synonyms can solr handle and
still perform well ? 2) How to make synonyms addition/modifications
without rebooting ? 3) Which is the better query parser together with
multi-word synonyms ?</p>
<h2 id="the-simple-query-parser">The simple query parser</h2>
<p>Given this field: <em>text:"say me bye or say hello world"</em> And
the given configuration:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode xml"><code class="sourceCode xml"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><<span class="kw">example</span>></span></code></pre></div>
<ul>
<li>one word <strong>text:(hello)</strong> matches <strong>"say me bye
or say <em>hello</em> world"</strong></li>
<li>one regex <strong>text:(/hel.</strong>/)** matches <strong>"say me
bye or say <em>hello</em> world"</strong></li>
<li>not one word <strong>text:(hello)</strong> matches <strong>"say me
bye or say <em>hello</em> world"</strong></li>
<li>one tailing wildcard <strong>text:(</strong>o)** matches
<strong>"say me bye or say <em>hello</em>
world"</strong></li>
<li>one fuzzy word <strong>text:(word~)</strong> matches <strong>"say me
bye or say hello <em>world</em>"</strong></li>
<li>two words <strong>text:(hello AND bye)</strong> matches <strong>"say
me <em>bye</em> or say <em>hello</em>
world"</strong></li>
<li>two words exact <strong>text:("hello world")</strong> matches
<strong>"say me bye or say <em>hello</em>
<em>world</em>"</strong></li>
<li>two words with distance <strong>text:("say world"~2)</strong>
matches <strong>"say me bye or <em>say</em> hello
<em>world</em>"</strong></li>
<li>one word AND two words with distance <strong>text:(bye AND "hello
world"~2)</strong> matches <strong>"say me <em>bye</em> or
say <em>hello</em> <em>world</em>"</strong></li>
<li>two words with distance AND two words with distance
<strong>text:("say bye"~2 AND "say world"~2)</strong> matches
<strong>"<em>say</em> me <em>bye</em> or
<em>say</em> hello <em>world</em>"</strong></li>
<li>one multi-word synonym: <strong>text:("see you soon"~2 AND "hello
world"~2)</strong> matches <strong>"say me <em>bye</em> or
say <em>hello</em> <em>world</em>"</strong></li>
</ul>
<p>Note: The simple query parser has some limations compared to the
complexphrase parser:</p>
<ul>
<li>it only keeps order for exact phrases</li>
<li>fuzzy, joker are not allowed for phrases</li>
</ul>
<p>However it has some advantages:</p>
<ul>
<li>it allows to define multiple groups of words</li>
<li>it allows regex on one word</li>
</ul>
<h2 id="how-to-handle-negation">How to handle negation</h2>
<p>SolR allows to look for <em>the absence</em> of a word or phrase, but
not for negations of them. One solution I found is to exploit the
multi-word synonyms, by linking all the various ways to express negation
to the <em>neg</em> word. Then it is possible to ask for the absence of
a negation.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="ex">never</span> had,never had any, has no, not, no anymore =<span class="op">></span> neg</span></code></pre></div>
<p>This can be used in this way:</p>
<ul>
<li><em>"text": "I never had any trouble"</em></li>
<li>searching for people with trouble: <em>text:((trouble) AND NOT ("neg
trouble"))</em></li>
</ul>
<h1 id="generalisation-of-negation">Generalisation of negation</h1>
<p>For a given <em>simple full-text</em> low level filter such a word or
a phrase:</p>
<ul>
<li><em>simple</em> => <em>(simple AND NOT "neg simple")</em></li>
<li><em>"very simple"</em> => <em>("very simple" AND NOT "neg very
simple)</em></li>
<li><em>less simple</em> => <em>( (less AND NOT "neg less") OR
(simple AND NOT "neg simple) )</em></li>
<li><em>less AND simple</em> => <em>( (less AND NOT "neg less") AND
(simple AND NOT "neg simple) )</em></li>
</ul>
<h1 id="simplifying-interface-for-user">Simplifying interface for
user</h1>
<p>The more user friendly interface in my mind would allo people to
write group of words and articulate them with logical operator such
AND/OR/NOT. You don't want them to manage phrase and word too. So when a
user look for:</p>
<ul>
<li><em>see you soon</em> => <em>"see you soon"</em></li>
</ul>
<p>They might check for ordered and compacted word, but by default we
the tolerate some distance:</p>
<ul>
<li><em>see you soon</em> => <em>"see you soon"</em> => <em>"see
you soon"~4</em></li>
</ul>
<p>In order to remove negated pattern, we apply the above method:</p>
<ul>
<li><em>see you soon</em> => <em>"see you soon"</em> => <em>"see
you soon"~4</em> => <em>"see you soon"~4 AND NOT "neg see you
soon"~4</em></li>
</ul>
<p>People might add other groups and articulate them together:</p>
<ul>
<li><em>("see you soon"~4 AND NOT "neg see you soon"~4) NOT ("an other
example"~4 AND NOT "neg an other example"~4)</em></li>
</ul>
<p>The above example would match a text containing <em>see you soon</em>
and not containing <em>an other example</em>. By the way any multi-word
synonym would be translated. This also means the user not able to use
jocker or fuzzy search. However, the jocker can be replaced with the
<em>stemming</em> process, which has the advantage of not breaking the
performances in case of very narrow joker query and also being
transparent for the end user. There is also a possible replacement for
fuzzy feature transparent to user.</p>
<h2 id="fuzzy-search-with-synonyms">Fuzzy search with synonyms</h2>
<p>Word Embedding offers the opportunity to freely produce typo
synonyms. I have tested succesfully a quite large list of them (~200k
entries) and the performances where not impacted. So most common error
are transparently integrated to end users.</p>
<h2 id="dated-delay-and-other-structured-informations">Dated, Delay and
other structured informations</h2>
<p>The simple query parser let add some structured informations within
the free-text to be queried. For example the below enriched text
allows:</p>
<ul>
<li><em>"text":"delay0002 dateCreat20181214 my example
free-text"</em></li>
<li><em>text:([delay0001 TO delay0003] AND [dateCreat201812 TO NOW] AND
"example")</em> this makes possible to filter based on dates.</li>
</ul>
<p>It is also possible to add many coded informations within the text to
be queried as structured data with full-text structured capabilities.
This will be very useful to get NLP pipelines results.</p>
<h2 id="dealing-with-multivalued-fields-mv">Dealing with multivalued
fields (MV)</h2>
<p>While the MVs look temptating for storing multiple occurence of the
same concept, they are not a good choice when dealing with full text
queries. Indeed the multiple values are not considered as independant
values but as a whole with some defined distance between values.</p>
<p>So how to modelize multiple occurences of the same concept within
SolR ? One alternate solution is to use multiple single fields for
textual datatype. For example it the encounter has two physician notes
both with a adverse event section, the resulting encounter document will
have two fields "pn.adverse1" and "pn.averse2". This implies that when
the user asks for something present into this section, the resulting
query should be modified accordingly</p>
<ul>
<li><em>"pn.adverse1":"hello world"</em></li>
<li><em>"pn.adverse2":"my example"</em></li>
<li><em>"pn.adverse:(hello AND example)"</em> will be replaced by:</li>
<li><em>"pn.adverse1:(hello AND example) OR pn.adverse2:(hello AND
example)"</em></li>
</ul>
<p>This makes also possible to mix full text queries with structured
queries based on the same field. Also there is some drawbacks. The
replacement mecanism makes mandatory to know in advance the maximum
number of fields of every documents. An other drawback is when the user
wants to look into every section of the same document. One solution is
to copyField</p>
<h2 id="conclusion">Conclusion</h2>
<p>The final method offers:</p>
<ul>
<li>multi-word synonyms</li>
<li>monoword synonyms</li>
<li>monoword typos (fuzzy match)</li>
<li>stemming</li>
<li>negation</li>
<li>AND/OR/NOT operators</li>
</ul>
<p>Still some aspects are missing:</p>
<ul>
<li>no weighted queries (but not needed for my use case)</li>
<li>no multivalued textual fields (this parser does not make difference
between them)</li>
</ul>
<p>This lets envisage a simple but powerful interface. Let's now see how
to transform a medical relational database and populate SolR.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Multiword synomyms in SolR — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/mixing-spark-and-solrMixing Spark and SolR2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-12-17</time></span>
<p>Estimated Time:<span id="reading-time">2 minutes</span></p>
<h1 id="mixing-spark-and-solr">Mixing Spark and SolR</h1>
<p>The spark-solr library makes it possible to interact with SolR-Cloud
- it is not compatible with SolR standalone. It allows both
<em>reading</em> and <em>writing</em> within SolR.</p>
<h2 id="reading-from-solr">Reading from SolR</h2>
<p>The library implements the <em>export</em> features which is fast to
retrieve DocValues enabled columns.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode scala"><code class="sourceCode scala"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="co">// configure the access</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> options <span class="op">=</span> <span class="ex">Map</span><span class="op">(</span> <span class="st">"collection"</span> <span class="op">-></span> <span class="st">"gettingstarted"</span><span class="op">,</span> <span class="st">"zkhost"</span> <span class="op">-></span> <span class="st">"localhost:9983"</span><span class="op">)</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="co">// use export to boost the retrieval</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="co">// choose fields and the query</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a>spark<span class="op">.</span>read<span class="op">.</span><span class="fu">format</span><span class="op">(</span><span class="st">"solr"</span><span class="op">).</span><span class="fu">options</span><span class="op">(</span>options<span class="op">).</span><span class="fu">option</span><span class="op">(</span><span class="st">"request_handler"</span><span class="op">,</span> <span class="st">"/export"</span><span class="op">).</span><span class="fu">option</span><span class="op">(</span><span class="st">"fields"</span><span class="op">,</span><span class="st">"id"</span><span class="op">).</span><span class="fu">option</span><span class="op">(</span><span class="st">"query"</span><span class="op">,</span><span class="st">"id:/.*.pdf/"</span><span class="op">).</span>load<span class="op">.</span>count</span></code></pre></div>
<h2 id="writing-from-spark">Writing from Spark</h2>
<div class="sourceCode" id="cb2"><pre class="sourceCode scala"><code class="sourceCode scala"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>csvDF<span class="op">.</span>write<span class="op">.</span><span class="fu">format</span><span class="op">(</span><span class="st">"solr"</span><span class="op">).</span><span class="fu">options</span><span class="op">(</span>options<span class="op">).</span><span class="fu">mode</span><span class="op">(</span>org<span class="op">.</span>apache<span class="op">.</span>spark<span class="op">.</span>sql<span class="op">.</span>SaveMode<span class="op">.</span>Overwrite<span class="op">).</span>save</span></code></pre></div>
<h2 id="creating-json-from-spark">Creating JSON from Spark</h2>
<p>Many useful sql functions to build json from spark exist. So far, I
have been using <em>to_json</em>, <em>named_stuct</em>, <em>map</em>,
<em>map_from_entries</em> and <em>concat_map</em> with some success.</p>
<p><strong>Generate a json object with zero to many items</strong>, from
zero to many rows:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="kw">with</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a>tmp <span class="kw">as</span> (<span class="kw">select</span> <span class="dv">1</span> <span class="kw">id</span>,<span class="st">'sec1'</span> sec,<span class="st">'a'</span> cont</span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> <span class="kw">union</span> <span class="kw">all</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> <span class="kw">select</span> <span class="dv">1</span>,<span class="st">'sec2'</span>, <span class="st">'b'</span></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> <span class="kw">union</span> <span class="kw">all</span></span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">select</span> <span class="dv">2</span>, <span class="st">'sec2'</span>,<span class="st">'c'</span>)</span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a><span class="kw">select</span> <span class="kw">id</span>, to_json(map_from_entries(collect_list(struct(sec, cont)))) <span class="kw">as</span> t</span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a><span class="kw">from</span> tmp</span>
<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a><span class="kw">group</span> <span class="kw">by</span> <span class="kw">id</span>;</span>
<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a><span class="co">-- +---+-----------------------+</span></span>
<span id="cb3-12"><a href="#cb3-12" aria-hidden="true" tabindex="-1"></a><span class="co">-- |id |t |</span></span>
<span id="cb3-13"><a href="#cb3-13" aria-hidden="true" tabindex="-1"></a><span class="co">-- +---+-----------------------+</span></span>
<span id="cb3-14"><a href="#cb3-14" aria-hidden="true" tabindex="-1"></a><span class="co">-- |1 |{"sec1":"a","sec2":"b"}|</span></span>
<span id="cb3-15"><a href="#cb3-15" aria-hidden="true" tabindex="-1"></a><span class="co">-- |2 |{"sec2":"c"} |</span></span>
<span id="cb3-16"><a href="#cb3-16" aria-hidden="true" tabindex="-1"></a><span class="co">-- +---+-----------------------+</span></span></code></pre></div>
<p><strong>Add an element within the resulting object</strong>, use
<em>concat_map</em> to merge them. By using the coalesce function to
replace null with empty map, this avoids the collateral effect of nulls
that would result into null in any case (null + x = null)</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="kw">with</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a>tmp <span class="kw">as</span> (<span class="kw">select</span> <span class="dv">1</span> <span class="kw">id</span>,<span class="st">'sec1'</span> sec,<span class="st">'a'</span> cont</span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a> <span class="kw">union</span> <span class="kw">all</span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a> <span class="kw">select</span> <span class="dv">1</span>,<span class="st">'sec2'</span>, <span class="st">'b'</span></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a> <span class="kw">union</span> <span class="kw">all</span></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">select</span> <span class="dv">2</span>, <span class="st">'sec2'</span>,<span class="st">'c'</span>)</span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a><span class="kw">select</span> <span class="kw">id</span>, to_json(</span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a> concat_map(</span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a> <span class="fu">coalesce</span>(map(<span class="ot">"id"</span>,<span class="kw">id</span>),map()),</span>
<span id="cb4-10"><a href="#cb4-10" aria-hidden="true" tabindex="-1"></a> <span class="fu">coalesce</span>(map_from_entries(</span>
<span id="cb4-11"><a href="#cb4-11" aria-hidden="true" tabindex="-1"></a> collect_list(</span>
<span id="cb4-12"><a href="#cb4-12" aria-hidden="true" tabindex="-1"></a> struct(sec, cont))),map()))) <span class="kw">as</span> t</span>
<span id="cb4-13"><a href="#cb4-13" aria-hidden="true" tabindex="-1"></a><span class="kw">from</span> tmp</span>
<span id="cb4-14"><a href="#cb4-14" aria-hidden="true" tabindex="-1"></a><span class="kw">group</span> <span class="kw">by</span> <span class="kw">id</span></span>
<span id="cb4-15"><a href="#cb4-15" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-16"><a href="#cb4-16" aria-hidden="true" tabindex="-1"></a><span class="co">-- +---+---------------------------------+</span></span>
<span id="cb4-17"><a href="#cb4-17" aria-hidden="true" tabindex="-1"></a><span class="co">-- |id |t |</span></span>
<span id="cb4-18"><a href="#cb4-18" aria-hidden="true" tabindex="-1"></a><span class="co">-- +---+---------------------------------+</span></span>
<span id="cb4-19"><a href="#cb4-19" aria-hidden="true" tabindex="-1"></a><span class="co">-- |1 |{"id":"1","sec1":"a","sec2":"b"}|</span></span>
<span id="cb4-20"><a href="#cb4-20" aria-hidden="true" tabindex="-1"></a><span class="co">-- |2 |{"id":"1","sec2":"c"} |</span></span>
<span id="cb4-21"><a href="#cb4-21" aria-hidden="true" tabindex="-1"></a><span class="co">-- +---+---------------------------------+</span></span></code></pre></div>
<p><strong>Create nested objects</strong> and place them into one parent
object. The use of a named_struct allows to nest objects within a first
parent object:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="kw">with</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a>secs <span class="kw">as</span> (<span class="kw">select</span> <span class="dv">1</span> <span class="kw">as</span> p_id, <span class="st">'sec1'</span> <span class="kw">as</span> sec, <span class="st">'hello'</span> <span class="kw">as</span> cont</span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a> <span class="kw">union</span> <span class="kw">all</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a> <span class="kw">select</span> <span class="dv">1</span> <span class="kw">as</span> p_id, <span class="st">'sec2'</span> <span class="kw">as</span> sec, <span class="st">'world'</span> cont),</span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a>tmp <span class="kw">as</span> (<span class="kw">select</span> p_id, map_from_entries(collect_list( struct(sec, cont) ) ) <span class="kw">as</span> t</span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">from</span> secs</span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a> <span class="kw">group</span> <span class="kw">by</span> p_id)</span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a><span class="kw">select</span> to_json( named_struct(<span class="st">'id'</span>, p_id, <span class="st">'_childDocuments_'</span>, collect_list(t))) <span class="kw">as</span> t</span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a><span class="kw">from</span> tmp</span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a><span class="kw">group</span> <span class="kw">by</span> p_id</span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a><span class="co">-- +-------------------------------------------------------------+</span></span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a><span class="co">-- |t |</span></span>
<span id="cb5-14"><a href="#cb5-14" aria-hidden="true" tabindex="-1"></a><span class="co">-- +-------------------------------------------------------------+</span></span>
<span id="cb5-15"><a href="#cb5-15" aria-hidden="true" tabindex="-1"></a><span class="co">-- |{"id":1,"_childDocuments_":[{"sec1":"hello","sec2":"world"}]}|</span></span>
<span id="cb5-16"><a href="#cb5-16" aria-hidden="true" tabindex="-1"></a><span class="co">-- +-------------------------------------------------------------+</span></span></code></pre></div>
<h2 id="good-practice-for-building-json-from-sql-within-spark">Good
practice for building json from SQL within spark</h2>
<p>Since it is possible to create json elements and concatenate them
into objects with joins, it improves readibility to create each kind of
element step by step and at the end concatenate them into larger
objects. This should not degrade the performance thanks to the spark
lazy evaluation design.</p>
<p>As a result, it is possible to begin with encounter information one
by one. Sections, then laboratories, medications and so on. Then it is
possible to build the encounter by combining all this stuff. Afterwards
comes the patient related informations. At the end, it is possible
include the encounters as nested childrens within the patients.</p>
<p>By using this design it will be quite easy to enrich any element, or
moove some into other, repeat, rename, remove them. It will be also easy
to debug. The overall prototype would look like the below steps:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="co">-- encounter labs step</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="co">-- encounter diags step</span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a><span class="co">-- encounter sections step</span></span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a><span class="co">-- patient step and nesting encounters into them</span></span>
<span id="cb6-6"><a href="#cb6-6" aria-hidden="true" tabindex="-1"></a><span class="co">-- good point is: named_struct automatically removes empty fields</span></span>
<span id="cb6-7"><a href="#cb6-7" aria-hidden="true" tabindex="-1"></a><span class="kw">with</span></span>
<span id="cb6-8"><a href="#cb6-8" aria-hidden="true" tabindex="-1"></a>secs <span class="kw">as</span> (<span class="kw">select</span> <span class="dv">1</span> <span class="kw">as</span> p_id, <span class="st">'sec1'</span> <span class="kw">as</span> sec, <span class="st">'hello'</span> <span class="kw">as</span> cont</span>
<span id="cb6-9"><a href="#cb6-9" aria-hidden="true" tabindex="-1"></a> <span class="kw">union</span> <span class="kw">all</span></span>
<span id="cb6-10"><a href="#cb6-10" aria-hidden="true" tabindex="-1"></a> <span class="kw">select</span> <span class="dv">1</span> <span class="kw">as</span> p_id, <span class="st">'sec2'</span> <span class="kw">as</span> sec, <span class="st">'world'</span> cont),</span>
<span id="cb6-11"><a href="#cb6-11" aria-hidden="true" tabindex="-1"></a>tmp <span class="kw">as</span> (<span class="kw">select</span> p_id, map_from_entries(collect_list( struct(sec, cont) ) ) <span class="kw">as</span> t</span>
<span id="cb6-12"><a href="#cb6-12" aria-hidden="true" tabindex="-1"></a> <span class="kw">from</span> secs</span>
<span id="cb6-13"><a href="#cb6-13" aria-hidden="true" tabindex="-1"></a> <span class="kw">group</span> <span class="kw">by</span> p_id)</span>
<span id="cb6-14"><a href="#cb6-14" aria-hidden="true" tabindex="-1"></a><span class="kw">select</span> to_json( named_struct(</span>
<span id="cb6-15"><a href="#cb6-15" aria-hidden="true" tabindex="-1"></a> <span class="st">'id'</span>, patient_id,</span>
<span id="cb6-16"><a href="#cb6-16" aria-hidden="true" tabindex="-1"></a> <span class="st">'birthDate'</span>, birthDate,</span>
<span id="cb6-17"><a href="#cb6-17" aria-hidden="true" tabindex="-1"></a> <span class="st">'_childDocuments_'</span>, collect_list(t)</span>
<span id="cb6-18"><a href="#cb6-18" aria-hidden="true" tabindex="-1"></a> )</span>
<span id="cb6-19"><a href="#cb6-19" aria-hidden="true" tabindex="-1"></a> ) <span class="kw">as</span> t</span>
<span id="cb6-20"><a href="#cb6-20" aria-hidden="true" tabindex="-1"></a><span class="kw">from</span> patients</span>
<span id="cb6-21"><a href="#cb6-21" aria-hidden="true" tabindex="-1"></a><span class="kw">left</span> <span class="kw">join</span> encounters <span class="kw">using</span> (patient_id)</span>
<span id="cb6-22"><a href="#cb6-22" aria-hidden="true" tabindex="-1"></a><span class="kw">group</span> <span class="kw">by</span> patient_id</span></code></pre></div>
<h2 id="limitations-of-spark-solr-lib">Limitations of spark-solr
lib</h2>
<ul>
<li>it does not support solr standalone (only solr-cloud)</li>
<li>it does not handle multi-valued for childDocuemnt yet (fixed with a
patch)</li>
<li>it does not handle two levels of childDocuments yet</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Mixing Spark and SolR — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/working-with-the-french-meshWhat's the MESH2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2018-12-19</time></span>
<p>Estimated Time:<span id="reading-time">5 minutes</span></p>
<h1 id="whats-the-mesh">What's the MESH</h1>
<h1 id="whats-the-french-mesh">What's the French MESH</h1>
<p>This is basically a fat XML without documentation(?)</p>
<h1 id="what-is-the-data-structure">What is the data structure</h1>
<p>The databricks spark-xml lib makes easy to transform an xml file into
a sparks dataframe.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode scala"><code class="sourceCode scala"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="kw">import</span> com<span class="op">.</span>databricks<span class="op">.</span>spark<span class="op">.</span>xml<span class="op">.</span>_</span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a>spark<span class="op">.</span>read<span class="op">.</span><span class="fu">format</span><span class="op">(</span><span class="st">"com.databricks.spark.xml"</span><span class="op">)</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="op">.</span><span class="fu">option</span><span class="op">(</span><span class="st">"rowTag"</span><span class="op">,</span> <span class="st">"DescriptorRecord"</span><span class="op">)</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="op">.</span><span class="fu">load</span><span class="op">(</span><span class="st">"fredesc2017.xml"</span><span class="op">)</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a><span class="op">.</span>persist</span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a><span class="op">.</span><span class="fu">registerTempTable</span><span class="op">(</span><span class="st">"b"</span><span class="op">).</span>printSchema</span></code></pre></div>
<p>The resulting dataframe schema is quite complicated.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode scala"><code class="sourceCode scala"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>#root</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> AllowableQualifiersList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> AllowableQualifier<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> Abbreviation<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierReferredTo<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> <span class="ex">Annotation</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> ConceptList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Concept<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-14"><a href="#cb2-14" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> CASN1Name<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-15"><a href="#cb2-15" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ConceptName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-16"><a href="#cb2-16" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-17"><a href="#cb2-17" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ConceptRelationList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-18"><a href="#cb2-18" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ConceptRelation<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-19"><a href="#cb2-19" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-20"><a href="#cb2-20" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> Concept1UI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-21"><a href="#cb2-21" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> Concept2UI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-22"><a href="#cb2-22" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> _RelationName<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-23"><a href="#cb2-23" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ConceptUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-24"><a href="#cb2-24" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> RegistryNumber<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-25"><a href="#cb2-25" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> RelatedRegistryNumberList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-26"><a href="#cb2-26" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> RelatedRegistryNumber<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-27"><a href="#cb2-27" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-28"><a href="#cb2-28" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ScopeNote<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-29"><a href="#cb2-29" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> TermList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-30"><a href="#cb2-30" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> Term<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-31"><a href="#cb2-31" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-32"><a href="#cb2-32" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DateCreated<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-33"><a href="#cb2-33" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> Day<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-34"><a href="#cb2-34" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> Month<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-35"><a href="#cb2-35" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> Year<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-36"><a href="#cb2-36" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> EntryVersion<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-37"><a href="#cb2-37" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> SortVersion<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-38"><a href="#cb2-38" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-39"><a href="#cb2-39" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> TermNote<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-40"><a href="#cb2-40" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> TermUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-41"><a href="#cb2-41" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ThesaurusIDlist<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-42"><a href="#cb2-42" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ThesaurusID<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-43"><a href="#cb2-43" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-44"><a href="#cb2-44" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> _ConceptPreferredTermYN<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-45"><a href="#cb2-45" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> _IsPermutedTermYN<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-46"><a href="#cb2-46" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> _LexicalTag<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-47"><a href="#cb2-47" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> _RecordPreferredTermYN<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-48"><a href="#cb2-48" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> TranslatorsScopeNote<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-49"><a href="#cb2-49" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> _PreferredConceptYN<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-50"><a href="#cb2-50" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> ConsiderAlso<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-51"><a href="#cb2-51" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> DateCreated<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-52"><a href="#cb2-52" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Day<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-53"><a href="#cb2-53" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Month<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-54"><a href="#cb2-54" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Year<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-55"><a href="#cb2-55" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> DateEstablished<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-56"><a href="#cb2-56" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Day<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-57"><a href="#cb2-57" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Month<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-58"><a href="#cb2-58" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Year<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-59"><a href="#cb2-59" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> DateRevised<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-60"><a href="#cb2-60" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Day<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-61"><a href="#cb2-61" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Month<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-62"><a href="#cb2-62" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> Year<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-63"><a href="#cb2-63" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> DescriptorName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-64"><a href="#cb2-64" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-65"><a href="#cb2-65" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> DescriptorUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-66"><a href="#cb2-66" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> EntryCombinationList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-67"><a href="#cb2-67" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> EntryCombination<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-68"><a href="#cb2-68" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-69"><a href="#cb2-69" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ECIN<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-70"><a href="#cb2-70" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorReferredTo<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-71"><a href="#cb2-71" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-72"><a href="#cb2-72" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-73"><a href="#cb2-73" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-74"><a href="#cb2-74" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierReferredTo<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-75"><a href="#cb2-75" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-76"><a href="#cb2-76" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-77"><a href="#cb2-77" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-78"><a href="#cb2-78" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> ECOUT<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-79"><a href="#cb2-79" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorReferredTo<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-80"><a href="#cb2-80" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-81"><a href="#cb2-81" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-82"><a href="#cb2-82" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-83"><a href="#cb2-83" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierReferredTo<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-84"><a href="#cb2-84" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-85"><a href="#cb2-85" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-86"><a href="#cb2-86" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> QualifierUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-87"><a href="#cb2-87" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> HistoryNote<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-88"><a href="#cb2-88" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> NLMClassificationNumber<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-89"><a href="#cb2-89" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> OnlineNote<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-90"><a href="#cb2-90" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> PharmacologicalActionList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-91"><a href="#cb2-91" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> PharmacologicalAction<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-92"><a href="#cb2-92" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-93"><a href="#cb2-93" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorReferredTo<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-94"><a href="#cb2-94" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-95"><a href="#cb2-95" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-96"><a href="#cb2-96" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-97"><a href="#cb2-97" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> PreviousIndexingList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-98"><a href="#cb2-98" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> PreviousIndexing<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-99"><a href="#cb2-99" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-100"><a href="#cb2-100" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> PublicMeSHNote<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-101"><a href="#cb2-101" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> SeeRelatedList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-102"><a href="#cb2-102" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> SeeRelatedDescriptor<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-103"><a href="#cb2-103" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-104"><a href="#cb2-104" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorReferredTo<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-105"><a href="#cb2-105" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorName<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-106"><a href="#cb2-106" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> <span class="ex">String</span><span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-107"><a href="#cb2-107" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> DescriptorUI<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-108"><a href="#cb2-108" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> TreeNumberList<span class="op">:</span> <span class="fu">struct</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-109"><a href="#cb2-109" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|--</span> TreeNumber<span class="op">:</span> <span class="fu">array</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-110"><a href="#cb2-110" aria-hidden="true" tabindex="-1"></a># <span class="op">|</span> <span class="op">|</span> <span class="op">|--</span> element<span class="op">:</span> <span class="fu">string</span> <span class="op">(</span>containsNull <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span>
<span id="cb2-111"><a href="#cb2-111" aria-hidden="true" tabindex="-1"></a># <span class="op">|--</span> _DescriptorClass<span class="op">:</span> <span class="dt">long</span> <span class="op">(</span>nullable <span class="op">=</span> <span class="kw">true</span><span class="op">)</span></span></code></pre></div>
<p>The folowing code extracts the first element of the term list. There
is 54 one of them in the 2017 mesh version. It is possible to get all of
them by union.</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode java"><code class="sourceCode java"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">sql</span><span class="op">(</span><span class="st">"""with</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a>tmp <span class="fu">as</span> <span class="op">(</span>select</span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> treenumberlist<span class="op">.</span><span class="fu">TreeNumber</span> as tree<span class="op">,</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> descriptorui as descriptor<span class="op">,</span></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> <span class="fu">explode</span><span class="op">(</span>conceptlist<span class="op">.</span><span class="fu">concept</span><span class="op">.</span><span class="fu">termlist</span><span class="op">.</span><span class="fu">term</span><span class="op">[</span><span class="dv">0</span><span class="op">])</span> as t</span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a> from b<span class="op">)</span></span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a>select tree<span class="op">,</span></span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a> descriptor<span class="op">,</span></span>
<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a> t<span class="op">.</span><span class="fu">string</span><span class="op">,</span></span>
<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a> t<span class="op">.</span><span class="fu">termui</span><span class="op">,</span></span>
<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a> t<span class="op">.</span><span class="fu">_ConceptPreferredTermYN</span><span class="op">,</span></span>
<span id="cb3-12"><a href="#cb3-12" aria-hidden="true" tabindex="-1"></a> <span class="fu">explode</span><span class="op">(</span>t<span class="op">.</span><span class="fu">thesaurusidlist</span><span class="op">.</span><span class="fu">thesaurusid</span><span class="op">)</span> as dict</span>
<span id="cb3-13"><a href="#cb3-13" aria-hidden="true" tabindex="-1"></a>from tmp<span class="st">""")</span></span>
<span id="cb3-14"><a href="#cb3-14" aria-hidden="true" tabindex="-1"></a><span class="op">.</span><span class="fu">show</span><span class="op">(</span><span class="dv">30</span><span class="op">,</span><span class="kw">true</span><span class="op">)</span></span>
<span id="cb3-15"><a href="#cb3-15" aria-hidden="true" tabindex="-1"></a><span class="co">// +--------------------+----------+--------------------+----------+-----------------------+----------------+</span></span>
<span id="cb3-16"><a href="#cb3-16" aria-hidden="true" tabindex="-1"></a><span class="co">// | tree|descriptor| string| termui|_ConceptPreferredTermYN| dict|</span></span>
<span id="cb3-17"><a href="#cb3-17" aria-hidden="true" tabindex="-1"></a><span class="co">// +--------------------+----------+--------------------+----------+-----------------------+----------------+</span></span>
<span id="cb3-18"><a href="#cb3-18" aria-hidden="true" tabindex="-1"></a><span class="co">// |[D03.633.100.221....| D000001| Calcimycin| T000002| Y| FDA SRS (2014)|</span></span>
<span id="cb3-19"><a href="#cb3-19" aria-hidden="true" tabindex="-1"></a><span class="co">// |[D03.633.100.221....| D000001| Calcimycin| T000002| Y| NLM (1975)|</span></span>
<span id="cb3-20"><a href="#cb3-20" aria-hidden="true" tabindex="-1"></a><span class="co">// |[D02.705.400.625....| D000002| Temefos| T000008| Y| FDA SRS (2014)|</span></span>
<span id="cb3-21"><a href="#cb3-21" aria-hidden="true" tabindex="-1"></a><span class="co">// |[D02.705.400.625....| D000002| Temefos| T000008| Y| INN (19XX)|</span></span>
<span id="cb3-22"><a href="#cb3-22" aria-hidden="true" tabindex="-1"></a><span class="co">// |[D02.705.400.625....| D000002| Temefos| T000008| Y| USAN (1974)|</span></span>
<span id="cb3-23"><a href="#cb3-23" aria-hidden="true" tabindex="-1"></a><span class="co">// |[D02.705.400.625....| D000002| Temephos| T000007| N| NLM (1996)|</span></span>
<span id="cb3-24"><a href="#cb3-24" aria-hidden="true" tabindex="-1"></a><span class="co">// |[J01.576.423.200....| D000003| Abattoirs| T000009| Y| NLM (1966)|</span></span>
<span id="cb3-25"><a href="#cb3-25" aria-hidden="true" tabindex="-1"></a><span class="co">// |[J01.576.423.200....| D000003| Slaughter Houses|T000901742| N| NLM (2017)|</span></span>
<span id="cb3-26"><a href="#cb3-26" aria-hidden="true" tabindex="-1"></a><span class="co">// |[J01.576.423.200....| D000003| Slaughter House|T000901743| N| NLM (2017)|</span></span>
<span id="cb3-27"><a href="#cb3-27" aria-hidden="true" tabindex="-1"></a><span class="co">// |[J01.576.423.200....| D000003| Slaughterhouses| T000010| N| UNK (19XX)|</span></span>
<span id="cb3-28"><a href="#cb3-28" aria-hidden="true" tabindex="-1"></a><span class="co">// |[L01.143.506.598....| D000004|Abbreviations as ...| T698652| Y| NLM (2008)|</span></span>
<span id="cb3-29"><a href="#cb3-29" aria-hidden="true" tabindex="-1"></a><span class="co">// | [A01.923.047]| D000005| Abdomen| T000012| Y| NLM (1966)|</span></span>
<span id="cb3-30"><a href="#cb3-30" aria-hidden="true" tabindex="-1"></a><span class="co">// |[C10.597.617.044....| D000006| Abdomen, Acute| T000013| Y| NLM (1966)|</span></span>
<span id="cb3-31"><a href="#cb3-31" aria-hidden="true" tabindex="-1"></a><span class="co">// |[C10.597.617.044....| D000006| Abdomen chirurgical|fre0138059| N|French thesaurus|</span></span>
<span id="cb3-32"><a href="#cb3-32" aria-hidden="true" tabindex="-1"></a><span class="co">// |[C10.597.617.044....| D000006| Abdomen urgent|fre0138060| N|French thesaurus|</span></span>
<span id="cb3-33"><a href="#cb3-33" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007| Abdominal Injuries| T000015| Y| NLM (1966)|</span></span>
<span id="cb3-34"><a href="#cb3-34" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007| Injuries, Abdominal| T000014| N| UNK (19XX)|</span></span>
<span id="cb3-35"><a href="#cb3-35" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007|Traumatismes de l...|fre0037777| Y|French thesaurus|</span></span>
<span id="cb3-36"><a href="#cb3-36" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007|Blessures abdomin...|fre0072220| N|French thesaurus|</span></span>
<span id="cb3-37"><a href="#cb3-37" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007|Blessures de l'ab...|fre0103940| N|French thesaurus|</span></span>
<span id="cb3-38"><a href="#cb3-38" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007| Lésions abdominales|fre0113338| N|French thesaurus|</span></span>
<span id="cb3-39"><a href="#cb3-39" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007|Lésions de l'abdomen|fre0113337| N|French thesaurus|</span></span>
<span id="cb3-40"><a href="#cb3-40" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007|Lésions traumatiq...|fre0103942| N|French thesaurus|</span></span>
<span id="cb3-41"><a href="#cb3-41" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007|Lésions traumatiq...|fre0103941| N|French thesaurus|</span></span>
<span id="cb3-42"><a href="#cb3-42" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C26.017]| D000007|Traumatismes abdo...|fre0047606| N|French thesaurus|</span></span>
<span id="cb3-43"><a href="#cb3-43" aria-hidden="true" tabindex="-1"></a><span class="co">// | [C04.588.033]| D000008| Abdominal Neoplasms| T000016| Y| NLM (1966)|</span></span>
<span id="cb3-44"><a href="#cb3-44" aria-hidden="true" tabindex="-1"></a><span class="co">// | [A02.633.567.050]| D000009| Abdominal Muscles| T000018| Y| NLM (1966)|</span></span>
<span id="cb3-45"><a href="#cb3-45" aria-hidden="true" tabindex="-1"></a><span class="co">// | [A02.633.567.050]| D000009| Muscle abdominal|fre0042987| N|French thesaurus|</span></span>
<span id="cb3-46"><a href="#cb3-46" aria-hidden="true" tabindex="-1"></a><span class="co">// | [A02.633.567.050]| D000009|Muscles de l'abdomen|fre0042988| N|French thesaurus|</span></span>
<span id="cb3-47"><a href="#cb3-47" aria-hidden="true" tabindex="-1"></a><span class="co">// |[A08.800.800.120....| D000010| Abducens Nerve| T000019| Y| NLM (1966)|</span></span>
<span id="cb3-48"><a href="#cb3-48" aria-hidden="true" tabindex="-1"></a><span class="co">// +--------------------+----------+--------------------+----------+-----------------------+----------------+</span></span>
<span id="cb3-49"><a href="#cb3-49" aria-hidden="true" tabindex="-1"></a><span class="co">// only showing top 30 rows</span></span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=What's the MESH — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/solr-joins-thought-sparkSolr joins thought Spark2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-01-13</time></span>
<p>Estimated Time:<span id="reading-time">2 minutes</span></p>
<h1 id="solr-joins-thought-spark">Solr joins thought Spark</h1>
<h2 id="context">Context</h2>
<p>SolR is great to search within complexes documents. However it is
weak to make complexes joins between documents and lack for expressive
languages. Combining both solr and spark let envisage having best of two
worlds.</p>
<h3 id="dataset">Dataset</h3>
<table style="width:33%;">
<colgroup>
<col style="width: 18%">
<col style="width: 15%">
</colgroup>
<tbody>
<tr class="odd">
<td>patient_id</td>
<td>visit_id</td>
</tr>
<tr class="even">
<td>1</td>
<td>1</td>
</tr>
<tr class="odd">
<td>1</td>
<td>2</td>
</tr>
<tr class="even">
<td>2</td>
<td>1</td>
</tr>
<tr class="odd">
<td>2</td>
<td>2</td>
</tr>
</tbody>
</table>
<h3 id="r1">R1</h3>
<table style="width:33%;">
<colgroup>
<col style="width: 18%">
<col style="width: 15%">
</colgroup>
<tbody>
<tr class="odd">
<td>patient_id</td>
<td>visit_id</td>
</tr>
<tr class="even">
<td>1</td>
<td>2</td>
</tr>
<tr class="odd">
<td>2</td>
<td>1</td>
</tr>
<tr class="even">
<td>2</td>
<td>2</td>
</tr>
</tbody>
</table>
<h3 id="r2">R2</h3>
<table style="width:33%;">
<colgroup>
<col style="width: 18%">
<col style="width: 15%">
</colgroup>
<tbody>
<tr class="odd">
<td>patient_id</td>
<td>visit_id</td>
</tr>
<tr class="even">
<td>1</td>
<td>2</td>
</tr>
<tr class="odd">
<td>2</td>
<td>2</td>
</tr>
</tbody>
</table>
<h2 id="use-case-q1-and-q2">Use case: Q1 AND Q2</h2>
<p>Let's say you have defined two queries Q1 and Q2 which have
respectively R1 and R2. Now you d'like to know which patients have both
kind of visits.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> R1.patient_id, R1.visit_id</span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> R1</span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="kw">JOIN</span> R2 <span class="kw">USING</span> (patient_id)</span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="kw">WHERE</span> <span class="kw">AND</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a>R2.visit_id <span class="op">!=</span> R1.visit_id</span></code></pre></div>
<h2 id="use-case-q1-or-q2">Use Case: Q1 OR Q2</h2>
<div class="sourceCode" id="cb2"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> patient_id, visit_id <span class="kw">FROM</span> R1)</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="kw">UNION</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> patient_id, visit_id <span class="kw">FROM</span> R2)</span></code></pre></div>
<h2 id="use-case-q1-and-not-q2">Use Case: Q1 AND NOT Q2</h2>
<div class="sourceCode" id="cb3"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> patient_id, visit_id <span class="kw">FROM</span> R1)</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="kw">MINUS</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> patient_id, visit_id <span class="kw">FROM</span> R2)</span></code></pre></div>
<h2 id="use-case-q1-then-q2">Use Case: Q1 THEN Q2</h2>
<p>Say we also have date along with data.</p>
<p>Say you want all Patients who had Q2 >1> Q1. You need to have
the dates.</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="kw">SELECT</span> R1.patient_id, R1.visit_id, R1.<span class="dt">date</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="kw">FROM</span> R1</span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="kw">JOIN</span> R2 <span class="kw">USING</span> (patient_id)</span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a><span class="kw">AND</span> R2.visit_id <span class="op">!=</span> R1.visit_id</span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a><span class="kw">AND</span> R2.<span class="dt">date</span> <span class="op">></span> R1.<span class="dt">date</span> <span class="op">+</span> <span class="dv">1</span></span></code></pre></div>
<h2 id="operations">Operations</h2>
<p>Given this formula: <strong>(Q1 AND Q2) <2< Q3</strong></p>
<div class="sourceCode" id="cb5"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="kw">WITH</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a>Q1 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">3</span> <span class="kw">as</span> <span class="dt">date</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">1</span>,<span class="dv">2</span></span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">4</span>),</span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a>Q2 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">3</span> <span class="kw">as</span> <span class="dt">date</span></span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">4</span>),</span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a>Q3 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">5</span> <span class="kw">as</span> <span class="dt">date</span></span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">6</span>),</span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a>T1 <span class="kw">as</span> (<span class="kw">SELECT</span> Q1.patient_id</span>
<span id="cb5-14"><a href="#cb5-14" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q1</span>
<span id="cb5-15"><a href="#cb5-15" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q2 <span class="kw">USING</span>(patient_id)</span>
<span id="cb5-16"><a href="#cb5-16" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb5-17"><a href="#cb5-17" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q2.visit_id <span class="op">!=</span> Q1.visit_id),</span>
<span id="cb5-18"><a href="#cb5-18" aria-hidden="true" tabindex="-1"></a>T2 <span class="kw">as</span> (<span class="kw">SELECT</span> Q1.patient_id</span>
<span id="cb5-19"><a href="#cb5-19" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q1</span>
<span id="cb5-20"><a href="#cb5-20" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q3 <span class="kw">USING</span>(patient_id)</span>
<span id="cb5-21"><a href="#cb5-21" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb5-22"><a href="#cb5-22" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.visit_id <span class="op">!=</span> Q1.visit_id</span>
<span id="cb5-23"><a href="#cb5-23" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.<span class="dt">date</span> <span class="op">></span> Q1.<span class="dt">date</span> <span class="op">+</span> <span class="dv">2</span>),</span>
<span id="cb5-24"><a href="#cb5-24" aria-hidden="true" tabindex="-1"></a>T3 <span class="kw">as</span> (<span class="kw">SELECT</span> Q2.patient_id</span>
<span id="cb5-25"><a href="#cb5-25" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q2</span>
<span id="cb5-26"><a href="#cb5-26" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q3 <span class="kw">USING</span>(patient_id)</span>
<span id="cb5-27"><a href="#cb5-27" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb5-28"><a href="#cb5-28" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.visit_id <span class="op">!=</span> Q2.visit_id</span>
<span id="cb5-29"><a href="#cb5-29" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.<span class="dt">date</span> <span class="op">></span> Q2.<span class="dt">date</span> <span class="op">+</span> <span class="dv">2</span>)</span>
<span id="cb5-30"><a href="#cb5-30" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T1)</span>
<span id="cb5-31"><a href="#cb5-31" aria-hidden="true" tabindex="-1"></a><span class="kw">INTERSECT</span></span>
<span id="cb5-32"><a href="#cb5-32" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T2)</span>
<span id="cb5-33"><a href="#cb5-33" aria-hidden="true" tabindex="-1"></a><span class="kw">INTERSECT</span></span>
<span id="cb5-34"><a href="#cb5-34" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T2)</span></code></pre></div>
<p>Given this formula: <strong>(Q1 OR Q2) AND Q3</strong></p>
<div class="sourceCode" id="cb6"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="kw">WITH</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a>Q1 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">3</span> <span class="kw">as</span> <span class="dt">date</span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">1</span>,<span class="dv">2</span></span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb6-6"><a href="#cb6-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">4</span>),</span>
<span id="cb6-7"><a href="#cb6-7" aria-hidden="true" tabindex="-1"></a>Q2 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">3</span> <span class="kw">as</span> <span class="dt">date</span></span>
<span id="cb6-8"><a href="#cb6-8" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb6-9"><a href="#cb6-9" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">4</span>),</span>
<span id="cb6-10"><a href="#cb6-10" aria-hidden="true" tabindex="-1"></a>Q3 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">5</span> <span class="kw">as</span> <span class="dt">date</span></span>
<span id="cb6-11"><a href="#cb6-11" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span> <span class="kw">ALL</span></span>
<span id="cb6-12"><a href="#cb6-12" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> <span class="dv">2</span>,<span class="dv">2</span>,<span class="dv">6</span>),</span>
<span id="cb6-13"><a href="#cb6-13" aria-hidden="true" tabindex="-1"></a>T1 <span class="kw">as</span> (<span class="kw">SELECT</span> Q1.patient_id</span>
<span id="cb6-14"><a href="#cb6-14" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q1</span>
<span id="cb6-15"><a href="#cb6-15" aria-hidden="true" tabindex="-1"></a> <span class="kw">UNION</span></span>
<span id="cb6-16"><a href="#cb6-16" aria-hidden="true" tabindex="-1"></a> <span class="kw">SELECT</span> Q2.patient_id</span>
<span id="cb6-17"><a href="#cb6-17" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q2),</span>
<span id="cb6-18"><a href="#cb6-18" aria-hidden="true" tabindex="-1"></a>T2 <span class="kw">as</span> (<span class="kw">SELECT</span> Q3.patient_id</span>
<span id="cb6-19"><a href="#cb6-19" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q3)</span>
<span id="cb6-20"><a href="#cb6-20" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T1)</span>
<span id="cb6-21"><a href="#cb6-21" aria-hidden="true" tabindex="-1"></a><span class="kw">INTERSECT</span></span>
<span id="cb6-22"><a href="#cb6-22" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T2)</span></code></pre></div>
<p>Given this formula: <strong>Q1 AND Q2 AND Q3</strong></p>
<div class="sourceCode" id="cb7"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="kw">WITH</span></span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a>Q1 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">1</span> <span class="kw">as</span> visit_id),</span>
<span id="cb7-3"><a href="#cb7-3" aria-hidden="true" tabindex="-1"></a>Q2 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id),</span>
<span id="cb7-4"><a href="#cb7-4" aria-hidden="true" tabindex="-1"></a>Q3 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">3</span> <span class="kw">as</span> visit_id),</span>
<span id="cb7-5"><a href="#cb7-5" aria-hidden="true" tabindex="-1"></a>T1 <span class="kw">as</span> (<span class="kw">SELECT</span> Q1.patient_id</span>
<span id="cb7-6"><a href="#cb7-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q1</span>
<span id="cb7-7"><a href="#cb7-7" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q2 <span class="kw">USING</span>(patient_id)</span>
<span id="cb7-8"><a href="#cb7-8" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb7-9"><a href="#cb7-9" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q2.visit_id <span class="op">!=</span> Q1.visit_id),</span>
<span id="cb7-10"><a href="#cb7-10" aria-hidden="true" tabindex="-1"></a>T2 <span class="kw">as</span> (<span class="kw">SELECT</span> Q2.patient_id</span>
<span id="cb7-11"><a href="#cb7-11" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q2</span>
<span id="cb7-12"><a href="#cb7-12" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q3 <span class="kw">USING</span>(patient_id)</span>
<span id="cb7-13"><a href="#cb7-13" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb7-14"><a href="#cb7-14" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.visit_id <span class="op">!=</span> Q2.visit_id),</span>
<span id="cb7-15"><a href="#cb7-15" aria-hidden="true" tabindex="-1"></a>T3 <span class="kw">as</span> (<span class="kw">SELECT</span> Q1.patient_id</span>
<span id="cb7-16"><a href="#cb7-16" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q1</span>
<span id="cb7-17"><a href="#cb7-17" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q3 <span class="kw">USING</span>(patient_id)</span>
<span id="cb7-18"><a href="#cb7-18" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb7-19"><a href="#cb7-19" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.visit_id <span class="op">!=</span> Q1.visit_id)</span>
<span id="cb7-20"><a href="#cb7-20" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T1)</span>
<span id="cb7-21"><a href="#cb7-21" aria-hidden="true" tabindex="-1"></a><span class="kw">INTERSECT</span></span>
<span id="cb7-22"><a href="#cb7-22" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T2)</span>
<span id="cb7-23"><a href="#cb7-23" aria-hidden="true" tabindex="-1"></a><span class="kw">INTERSECT</span></span>
<span id="cb7-24"><a href="#cb7-24" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T3)</span></code></pre></div>
<p>Given this formula: <strong>Q1 <1< Q2 <1< Q3</strong></p>
<div class="sourceCode" id="cb8"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="kw">WITH</span></span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a>Q1 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">1</span> <span class="kw">as</span> visit_id, <span class="dv">3</span> <span class="kw">as</span> <span class="dt">date</span>),</span>
<span id="cb8-3"><a href="#cb8-3" aria-hidden="true" tabindex="-1"></a>Q2 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">4</span> <span class="kw">as</span> <span class="dt">date</span>),</span>
<span id="cb8-4"><a href="#cb8-4" aria-hidden="true" tabindex="-1"></a>Q3 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">3</span> <span class="kw">as</span> visit_id, <span class="dv">5</span> <span class="kw">as</span> <span class="dt">date</span>),</span>
<span id="cb8-5"><a href="#cb8-5" aria-hidden="true" tabindex="-1"></a>T1 <span class="kw">as</span> (<span class="kw">SELECT</span> Q1.patient_id</span>
<span id="cb8-6"><a href="#cb8-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q1</span>
<span id="cb8-7"><a href="#cb8-7" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q2 <span class="kw">USING</span>(patient_id)</span>
<span id="cb8-8"><a href="#cb8-8" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb8-9"><a href="#cb8-9" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q2.visit_id <span class="op">!=</span> Q1.visit_id</span>
<span id="cb8-10"><a href="#cb8-10" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q2.<span class="dt">date</span> <span class="op">>=</span> Q1.<span class="dt">date</span> <span class="op">+</span> <span class="dv">1</span>),</span>
<span id="cb8-11"><a href="#cb8-11" aria-hidden="true" tabindex="-1"></a>T2 <span class="kw">as</span> (<span class="kw">SELECT</span> Q2.patient_id</span>
<span id="cb8-12"><a href="#cb8-12" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q2</span>
<span id="cb8-13"><a href="#cb8-13" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q3 <span class="kw">USING</span>(patient_id)</span>
<span id="cb8-14"><a href="#cb8-14" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb8-15"><a href="#cb8-15" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.visit_id <span class="op">!=</span> Q2.visit_id</span>
<span id="cb8-16"><a href="#cb8-16" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q3.<span class="dt">date</span> <span class="op">>=</span> Q2.<span class="dt">date</span> <span class="op">+</span> <span class="dv">1</span>)</span>
<span id="cb8-17"><a href="#cb8-17" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T1)</span>
<span id="cb8-18"><a href="#cb8-18" aria-hidden="true" tabindex="-1"></a><span class="kw">INTERSECT</span></span>
<span id="cb8-19"><a href="#cb8-19" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T2)</span></code></pre></div>
<p>Given this formula: <strong>Q1 <1< Q2 AND NOT Q3</strong></p>
<div class="sourceCode" id="cb9"><pre class="sourceCode sql"><code class="sourceCode sql"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="kw">WITH</span></span>
<span id="cb9-2"><a href="#cb9-2" aria-hidden="true" tabindex="-1"></a>Q1 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">1</span> <span class="kw">as</span> visit_id, <span class="dv">3</span> <span class="kw">as</span> <span class="dt">date</span>),</span>
<span id="cb9-3"><a href="#cb9-3" aria-hidden="true" tabindex="-1"></a>Q2 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">2</span> <span class="kw">as</span> visit_id, <span class="dv">4</span> <span class="kw">as</span> <span class="dt">date</span>),</span>
<span id="cb9-4"><a href="#cb9-4" aria-hidden="true" tabindex="-1"></a>Q3 <span class="kw">as</span> (<span class="kw">SELECT</span> <span class="dv">1</span> <span class="kw">as</span> patient_id, <span class="dv">3</span> <span class="kw">as</span> visit_id, <span class="dv">5</span> <span class="kw">as</span> <span class="dt">date</span>),</span>
<span id="cb9-5"><a href="#cb9-5" aria-hidden="true" tabindex="-1"></a>T1 <span class="kw">as</span> (<span class="kw">SELECT</span> Q1.patient_id</span>
<span id="cb9-6"><a href="#cb9-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">FROM</span> Q1</span>
<span id="cb9-7"><a href="#cb9-7" aria-hidden="true" tabindex="-1"></a> <span class="kw">JOIN</span> Q2 <span class="kw">USING</span>(patient_id)</span>
<span id="cb9-8"><a href="#cb9-8" aria-hidden="true" tabindex="-1"></a> <span class="kw">WHERE</span> <span class="kw">TRUE</span></span>
<span id="cb9-9"><a href="#cb9-9" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q2.visit_id <span class="op">!=</span> Q1.visit_id</span>
<span id="cb9-10"><a href="#cb9-10" aria-hidden="true" tabindex="-1"></a> <span class="kw">AND</span> Q2.<span class="dt">date</span> <span class="op">>=</span> Q1.<span class="dt">date</span> <span class="op">+</span> <span class="dv">1</span>)</span>
<span id="cb9-11"><a href="#cb9-11" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> T1)</span>
<span id="cb9-12"><a href="#cb9-12" aria-hidden="true" tabindex="-1"></a><span class="kw">MINUS</span></span>
<span id="cb9-13"><a href="#cb9-13" aria-hidden="true" tabindex="-1"></a>(<span class="kw">SELECT</span> <span class="op">*</span> <span class="kw">FROM</span> Q3)</span></code></pre></div>
<h2 id="isolating-matching-records">Isolating matching records</h2>
<p>For each Qn it would be interesting to get the matching visit. This
allows to run the solr queries afterwards.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Solr joins thought Spark — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/earth-sana-in-corpore-sanoEarth sana in corpore sano2021-11-07T15:49:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-03-31</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="earth-sana-in-corpore-sano">Earth sana in corpore sano</h1>
<h2 id="society-produces-more-and-more-energy">Society produces more and
more Energy</h2>
<p>Human have gradualy been less and less physical: hunter/trapper,
farmer, workers and novadays digitalized works. In the meantime, it
turns out food has been more and more easy to get. We saturate of
energy.</p>
<p>While sport is a fairly ealthy activity, the lost of energy is a
pity. Worst, going to the gym room does not collect that energy but in
addition needs energy to make those sport machine active. This man-power
could be use for other purposes.</p>
<p><img src="/images/gym.webp" class="align-center" style="width:75.0%" alt="Energy"></p>
<h2 id="society-produces-more-and-more-waste">Society produces more and
more Waste</h2>
<p>Waste appeared in the begining of 20th century. Before that, most of
human production were reused: from animal bones and skin to human feces.
However novodays, the waste is far from being a profitable industry.</p>
<p>We all are aware that industry produces way too much plastic
packaging. While most people pay attention to put thoses wastes into
trash, some do not. The reasons of this lack of care of packaging is
unknown and might be related to education, mood or loss of attention.
However what is clearly identified is the reason why industry uses such
amount of packaging: this ease customers experience.</p>
<p><img src="/images/coca.webp" class="align-center" style="width:75.0%" alt="Packaging"></p>
<h2 id="eco-sport-best-of-two-worlds">Eco-Sport: Best of Two Worlds</h2>
<p>Let's say every jogger would gather every waste he finds in a plastic
bag. This would make it's practice way more fun, and also it's energy
would be used efficiently. In addition let's say people would see that
jogger gather those things, I bet a few proportion (children in
particular) would adopt this practice. Worst case, they would not be so
prone to leave rubbish liying anymore !</p>
<p><img src="/images/eco-sport.webp" class="align-center" style="width:75.0%" alt="Echo Sport"></p>
<p>There is so many healthy sports to practice in town, forest, sea,
river and so on and each might be turned in eco-sport! I have right now
no idea what eco-friendly gesture a boxer might have...</p>
<p><img src="/images/eco-boxe.webp" class="align-center" style="width:75.0%" alt="Echo Boxe"></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Earth sana in corpore sano — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-07 15:49</time> </div>https://blog.parisni.com/blog/going-barefootGoing Barefoot2021-08-20T15:30:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-04-07</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="going-barefoot">Going Barefoot</h1>
<h2 id="every-body-has-unbalanced-legs">Every body has unbalanced
legs</h2>
<p>Humans are not symetric. As a result the legs are not exactly the
same size. For some the size diffence leads to pathologies particularly
for sportsmen. Thurty percent of running people have the iliotibial band
syndrome which is a consequence of such unbalenced body. As a chronic
disease it is complicated to heal and sometimes changing shoes or
environment is sufficient to relapse and a podiastrist treatment is
often necessary.</p>
<h2 id="podiastrists-advice-orthopaedic-insoles">Podiastrists advice
orthopaedic insoles</h2>
<p>Most of them will sell to you their insoles in order to rebalance
your posture properly. Insoles have multiple drawbacks: - they costs a
lot, are not reimbursed, and need to be replaced every year - they imply
a dependency to your podiastrist - your body will adapt to the insole,
and it is discuradged to walk without</p>
<h2 id="the-minimal-shoes-awareness">The minimal shoes awareness</h2>
<p>Being overly dependant of a piece of cardboard was innapropriate to
me. Since almost every body is asymetric and almost 30% of runner are
using insoles, how did humans were hunting in the 200 million past years
without our highly skilled podiastrists, and even without shoes ? Am I
an not supposed to travel by myself ? The minimal shoes were an elegant
solution to all that doubt.</p>
<h2 id="walking-with-minimal-shoes">Walking with minimal shoes</h2>
<p>I bought a pair of minimal shoes, and tried to walk as usual, but
lighter on my feets. Sadly this has not improved a lot my condition an
my knee pain. After a while I realized this was not the right way of
doing. When walking barefoot I was naturally walking with forefoot
before. I have been walking this way for about 2 weeks, and my knee is
far better. Also I can feel my muscle adapting: I am much more using my
calves, and my foot toes and so on. This muscle reinforcement will be
needed when I will be able to run this way soon.</p>
<h2 id="running-with-minimal-shoes">Running with minimal shoes</h2>
<p>While walking forefoot is far from obvious, running with minimal
shoes does not allow other attitude. I have tested multiple series of
few distance and I felt the tendons of my feets not ready to support
more distance right now. Also the Achylles tendons is very requested
that way.</p>
<h2 id="conclusion-after-two-month">Conclusion after two month</h2>
<p>My iliac band syndrome looks far better. Even after walking hours, I
do not feel pain by night in the knee. I fell my feets being more and
more robusts, and my body balance more stable. I am thinking this
transition would be useful for people with back of hip pain too.</p>
<p>Better than having orthopeadic insoles, walking barefoot is possible
and advised. The physiologic changes are huge and the transition must be
slow.</p>
<p>There is no possible concession with using minimal shoes: there is no
return. It's over with regular shoes.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Going Barefoot — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-20 15:30</time> </div>https://blog.parisni.com/blog/importance-of-teachingImportance of Teaching2021-11-07T15:49:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-07-07</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="importance-of-teaching">Importance of Teaching</h1>
<p>Today knowledge <strong>exists</strong>. Search-engine and
recomendation-algorithms makes knowledge <strong>accessible</strong>. We
live in a knowledge <strong>opulence</strong>. Does this mean we make a
good use of the knowledge ?</p>
<h1 id="opulence-is-counter-productive">Opulence is counter
productive</h1>
<p><img src="/images/fish.webp" alt="image"></p>
<p>For multiple reason among reducing consumption from predatory, fish
use <strong>schooling</strong>: they regroup and swim in the same
direction. An interpretation is that the predator is less able to make
choice in this context of opulence.</p>
<p>Paradoxally, living in a megalopole usually increase lonely and
people decrease their activity. Opulence of offer limit the usage.</p>
<p>I am pretty sure the opulence of information on the web acts in the
same manner. The potential is huge, but the usage is low.</p>
<h1 id="faster-but-less-emotion">Faster, but less emotion</h1>
<p>Before internet, knowledge was physically shared among people or
books. The knowledge came together with an emotional context. Some
memory method use emotion to ehnance performances.</p>
<p>Internet knowledge does not provide such emotion together with the
information. Therefore, the memory processus cannot be enhanced
naturally. The is a need to reinforce the internet knowledges.</p>
<h1 id="teaching">Teaching</h1>
<p>There is multiple ways of teaching: explaining to others, writing a
blog-post, self explaining by mind. In every cases, teaching need a
synthesis step, and structure the informations to improve understanding
and finally ease the learning.</p>
<p>Teaching is a powerful way of consolidating the knowledges. It helps
to raise error or superficial understanding. It has interesting
advantages: the teaching materials can be used to re-activate
knowledges, and also reduce the cognitive load.</p>
<h1 id="conclusion">Conclusion</h1>
<p>In the days of knowledge opulence, there is a numeric hygiene to fix
the informations and take advantage of it.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Importance of Teaching — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-07 15:49</time> </div>https://blog.parisni.com/blog/benefits-of-functional-linux-distroBenefits of functional
linux distro2020-06-06T22:05:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-08-17</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="benefits-of-functional-linux-distro">Benefits of functional
linux distro</h1>
<p>Operating System is a tool to build and create other tools. OS is
secondary and the perfect one should be less present as possible. Less
time fighting with installation, less time with configuration, less time
with frustration.</p>
<p>My first contact with linux has been ubuntu, and I am still using it
at work: using a mainstream tool, is a security and the promize to find
out solution fast on forum, blogs. When comes new version of ubuntu LTS,
this is a pain to migrate, and always a lost of energy so it is usual to
be frustrated and keeping and old OS with old software dependency.</p>
<p>Then I gave a try to rolling-release distribution: Archlinux. The
good point is you never late and get the latest version on anything.
However I felt that OS was taking more and more importance in my daily
practice ; any package upgrade was a critical action. I finally become
very quiet with upgrade, and paradoxally late again with software
release.</p>
<p>By chance I recently discovered functional programming in general,
and functional OS in particular. Nixos and Guix are both exploiting the
idea of immutability to build a very robust OS design. This allows to
rollback any bad situation very easily. The cost of this power is to
learn how to master the beast: haskell or scheme programming language
respectively. An I d'say that's an opportunity to learn new languages of
interest.</p>
<p>While guix is more appealing for its full-GNU support I choosed
nixos. The reason is the packet manager called nix can be installed and
used on any distribution. Therefore I can take my time to learn haskell
and keep ubuntu as a security for daily primary objective: having a
transparent and robust OS.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Benefits of functional
linux distro — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-06 22:05</time> </div>https://blog.parisni.com/blog/easy-neovim-with-nixEasy neovim with nix2020-06-06T22:05:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-08-22</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="easy-neovim-with-nix">Easy neovim with nix</h1>
<p>In this post, I will show how to maintain a neovim configuration with
the <strong>nix</strong> package manager. Nix can be installed on any
linux distribution with a single sudo bash script.</p>
<p>One of the good point is it ease the way to install and configure
softwares: below is an example on how to install neomutt and several vim
packages. In case a package does not yet exists or is not up-to-date,
the nixpkgs github repository is the place to make pull requests.</p>
<p>In case the merge is accepted, the nix unstable channel will get the
modification few hours later, and any body is able to install the
package.</p>
<p>Nix is well documented and the community looks extraordinary active.
However the syntax and what happens behind the scene is for now a source
of curiosity for me: I have not yet switch to <strong>nixos</strong>,
and only be using the nix manager on few servers.</p>
<p>Still some packages are bogus, but things looks moving fast and I
such active community makes the invest confident.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode haskell"><code class="sourceCode haskell"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="pp"># .config/nixpkgs/config.nix</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>{</span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> packageOverrides <span class="ot">=</span> pkgs<span class="op">:</span> with pkgs; rec {</span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a> myNeovim <span class="ot">=</span> neovim<span class="op">.</span>override {</span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a> vimAlias <span class="ot">=</span> true;</span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a> configure <span class="ot">=</span> {</span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a> customRC <span class="ot">=</span> ''</span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a> colo murphy</span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a> set number</span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a> set encoding<span class="ot">=</span>utf<span class="op">-</span><span class="dv">8</span></span>
<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a> au <span class="dt">FileType</span> python setlocal formatprg<span class="ot">=</span>autopep8\ <span class="op">-</span></span>
<span id="cb1-12"><a href="#cb1-12" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb1-13"><a href="#cb1-13" aria-hidden="true" tabindex="-1"></a> function<span class="op">!</span> <span class="dt">DoPrettyXML</span>()</span>
<span id="cb1-14"><a href="#cb1-14" aria-hidden="true" tabindex="-1"></a> <span class="kw">let</span> l<span class="op">:</span>origft <span class="ot">=</span> <span class="op">&</span>ft</span>
<span id="cb1-15"><a href="#cb1-15" aria-hidden="true" tabindex="-1"></a> set ft<span class="ot">=</span></span>
<span id="cb1-16"><a href="#cb1-16" aria-hidden="true" tabindex="-1"></a> 1s<span class="op">/<?</span>xml <span class="op">.*?>//</span>e</span>
<span id="cb1-17"><a href="#cb1-17" aria-hidden="true" tabindex="-1"></a> 0put <span class="ot">=</span>'<span class="op"><</span><span class="dt">PrettyXML</span><span class="op">></span>'</span>
<span id="cb1-18"><a href="#cb1-18" aria-hidden="true" tabindex="-1"></a> <span class="op">$</span>put <span class="ot">=</span>'<span class="op"></</span><span class="dt">PrettyXML</span><span class="op">></span>'</span>
<span id="cb1-19"><a href="#cb1-19" aria-hidden="true" tabindex="-1"></a> silent <span class="op">%!</span>xmllint <span class="co">--format -</span></span>
<span id="cb1-20"><a href="#cb1-20" aria-hidden="true" tabindex="-1"></a> 2d</span>
<span id="cb1-21"><a href="#cb1-21" aria-hidden="true" tabindex="-1"></a> <span class="op">$</span>d</span>
<span id="cb1-22"><a href="#cb1-22" aria-hidden="true" tabindex="-1"></a> <span class="dv">1</span></span>
<span id="cb1-23"><a href="#cb1-23" aria-hidden="true" tabindex="-1"></a> exe <span class="st">"set ft="</span> <span class="op">.</span> l<span class="op">:</span>origft</span>
<span id="cb1-24"><a href="#cb1-24" aria-hidden="true" tabindex="-1"></a> endfunction</span>
<span id="cb1-25"><a href="#cb1-25" aria-hidden="true" tabindex="-1"></a> command<span class="op">!</span> <span class="dt">PrettyXML</span> call <span class="dt">DoPrettyXML</span>()</span>
<span id="cb1-26"><a href="#cb1-26" aria-hidden="true" tabindex="-1"></a> '';</span>
<span id="cb1-27"><a href="#cb1-27" aria-hidden="true" tabindex="-1"></a> packages<span class="op">.</span>myVimPackage <span class="ot">=</span> with pkgs<span class="op">.</span>vimPlugins; {</span>
<span id="cb1-28"><a href="#cb1-28" aria-hidden="true" tabindex="-1"></a> start <span class="ot">=</span> [ </span>
<span id="cb1-29"><a href="#cb1-29" aria-hidden="true" tabindex="-1"></a> airline</span>
<span id="cb1-30"><a href="#cb1-30" aria-hidden="true" tabindex="-1"></a> vim<span class="op">-</span>nix</span>
<span id="cb1-31"><a href="#cb1-31" aria-hidden="true" tabindex="-1"></a> molokai</span>
<span id="cb1-32"><a href="#cb1-32" aria-hidden="true" tabindex="-1"></a> tagbar</span>
<span id="cb1-33"><a href="#cb1-33" aria-hidden="true" tabindex="-1"></a> easymotion</span>
<span id="cb1-34"><a href="#cb1-34" aria-hidden="true" tabindex="-1"></a> nerdtree</span>
<span id="cb1-35"><a href="#cb1-35" aria-hidden="true" tabindex="-1"></a> readline<span class="op">-</span>vim</span>
<span id="cb1-36"><a href="#cb1-36" aria-hidden="true" tabindex="-1"></a> ];</span>
<span id="cb1-37"><a href="#cb1-37" aria-hidden="true" tabindex="-1"></a> opt <span class="ot">=</span> [ ];</span>
<span id="cb1-38"><a href="#cb1-38" aria-hidden="true" tabindex="-1"></a> }; </span>
<span id="cb1-39"><a href="#cb1-39" aria-hidden="true" tabindex="-1"></a> };</span>
<span id="cb1-40"><a href="#cb1-40" aria-hidden="true" tabindex="-1"></a> };</span>
<span id="cb1-41"><a href="#cb1-41" aria-hidden="true" tabindex="-1"></a> myPackages <span class="ot">=</span> pkgs<span class="op">.</span>buildEnv {</span>
<span id="cb1-42"><a href="#cb1-42" aria-hidden="true" tabindex="-1"></a> name <span class="ot">=</span> <span class="st">"myPackages"</span>;</span>
<span id="cb1-43"><a href="#cb1-43" aria-hidden="true" tabindex="-1"></a> paths <span class="ot">=</span> [</span>
<span id="cb1-44"><a href="#cb1-44" aria-hidden="true" tabindex="-1"></a> myNeovim</span>
<span id="cb1-45"><a href="#cb1-45" aria-hidden="true" tabindex="-1"></a> vlc</span>
<span id="cb1-46"><a href="#cb1-46" aria-hidden="true" tabindex="-1"></a> ];</span>
<span id="cb1-47"><a href="#cb1-47" aria-hidden="true" tabindex="-1"></a> };</span>
<span id="cb1-48"><a href="#cb1-48" aria-hidden="true" tabindex="-1"></a> };</span>
<span id="cb1-49"><a href="#cb1-49" aria-hidden="true" tabindex="-1"></a>}</span></code></pre></div>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># it is possible to refresh the channels </span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="ex">nix-channel</span> <span class="at">--update</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="co"># it is possible to install an apply both vimrc/packages from this command:</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="ex">nix-env</span> <span class="at">-i</span> myPackages</span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="co"># it is possible to remove this by doing</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="ex">nix-env</span> <span class="at">-e</span> myPackages</span></code></pre></div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Easy neovim with nix — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-06 22:05</time> </div>https://blog.parisni.com/blog/spark-interactions-with-postgresSpark interactions with
Postgres2020-06-06T22:05:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-08-24</time></span>
<p>Estimated Time:<span id="reading-time">less than a minute</span></p>
<h1 id="spark-interactions-with-postgres">Spark interactions with
Postgres</h1>
<p>Both spark and sqoop allow nativelly to ingest/load relational
databases such postgresql. After being disapointed by both, I have
finally developped an optimized library to better bridge spark and
postgres: reading from postgres, writing to postgres from spark.</p>
<h1 id="the-spark-postgres-library">The spark-postgres library</h1>
<h1 id="comparison-with-sqoop">Comparison with Sqoop</h1>
<h1 id="automation-with-yaml">Automation with yaml</h1>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Spark interactions with
Postgres — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-06 22:05</time> </div>https://blog.parisni.com/blog/fhir-search-engine-resource-operatorsFHIR Search Engine -
Resource Operators2020-06-06T22:05:00+00:00<div id="generated-toc"> </div><hr> <span>Date:<time id="post-date">2019-10-12</time></span>
<p>Estimated Time:<span id="reading-time">2 minutes</span></p>
<h1 id="fhir-search-engine---resource-operators">FHIR Search Engine -
Resource Operators</h1>
<h1 id="technical-consideration">Technical Consideration</h1>
<ul>
<li>solr de-normalization faster selection query performances and
expressivity</li>
<li>solr de-normalization slower update perforamces and add
complexity</li>
<li>solr-join can be used in solr-cloud - the joined collection SHALL be
replicated over every shard.</li>
<li>solr-join performances are linear among the size of the joined
collection (local parameter: score=none).</li>
<li>solr-join query time is up to 1 sec on a 1M sized collection.</li>
<li>solr-join only implements inner joins without access to the joined
collection fields</li>
<li>spark-solr provides fast access to solr-cloud fields with
docValues=true or type string, long, timestamp</li>
<li>spark-solr produces a dataframe from a solr-cloud collection thought
the /export streaming expression handler</li>
<li>spark-solr supports any query filters to subset the collection</li>
<li>spark-solr offers a large set of joins, intersect, union,
aggregation and windowing</li>
<li>fhir _query search, allows top down search - Observation from
patient with identifier</li>
<li>fhir _has allows bottom-up search - Patient having Observation with
coding</li>
<li>fhir _graphql does not allow temporal operators (?)</li>
</ul>
<h1 id="technical-conclusion">Technical Conclusion</h1>
<ul>
<li>solr-join are not compatible with large tables</li>
<li>top-down-search can be implemented with solr de-normalized /
solr-join</li>
<li>top-down-search prefers solr-denormalized since the joined
collection are huge (patient/encounter)</li>
<li>the de-normalization can cover patient/encounter details - spark can
fetch them from the patient/encounter delta files (including the
_lastUpdated)</li>
<li>the _has covers <span class="title-ref">and</span> operator but
lacks <span class="title-ref">or</span>, <span class="title-ref">exept</span>, <span class="title-ref">after</span>,
<span class="title-ref">before</span>, <span class="title-ref">not</span>, <span class="title-ref">count</span> and
operator scope <span class="title-ref">(</span> and <span class="title-ref">)</span></li>
<li>the _has could be implemented by creating a dedicated collection
with all resources in it.</li>
<li>the _has could be implemented with spark-solr with <span class="title-ref">and</span> operators only (patients semi-left
joins)</li>
<li>adding a long <span class="title-ref">patient</span> field to the
fhir patient resource will be usefull</li>
<li>faceting, and unique faceting fields and denormalized fields
(extension in the bundle)</li>
<li><span class="title-ref">[Base]/Condition?patient.identifier=xxx&encounter.lenght=10</span></li>
</ul>
<h1 id="implementation">Implementation</h1>
<p>It is possible from <span class="title-ref">spring-data-solr</span>
to build the solr query. This query can be passed to <span class="title-ref">spark-solr</span>.</p>
<p>fhir-syntax -> solr-syntax</p>
<ul>
<li>(Condition?active=false) AFTER(24, recorded-date, created-date)
(Composition?status=canceled&_text=patient&type:below=letter)</li>
<li>ConditionAphp(q=*:<em>&active=true) AFTER(encounter, 24,
recorded-date, created-date)
CompositionAphp(q=</em>:*&status=canceled)</li>
<li>ConditionAphp(q=*:<em>&active=true) AND(encounter)
CompositionAphp(q=</em>:*&status=canceled)</li>
<li>AFTER(AND(Condition(query), Composition(query), encounter),
Medication(query), patient, "start-date", "end-date")</li>
</ul>
<p>AND(Dataframe, Dataframe, Column) OR(Dataframe, Dataframe, Column)
EXCEPT(Dataframe, Dataframe) AFTER(Dataframe, Dataframe, column, column,
column) OCCURRENCE(Dataframe, number) NOT_IN(Dataframe, Dataframe)</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode java"><code class="sourceCode java"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="kw">import</span> <span class="im">org</span><span class="op">.</span><span class="im">springframework</span><span class="op">.</span><span class="im">data</span><span class="op">.</span><span class="im">solr</span><span class="op">.</span><span class="im">core</span><span class="op">.</span><span class="im">DefaultQueryParser</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>DefaultQueryParser b <span class="op">=</span> <span class="kw">new</span> <span class="fu">DefaultQueryParser</span><span class="op">(</span><span class="kw">new</span> <span class="fu">SimpleSolrMappingContext</span><span class="op">());</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="bu">String</span> z <span class="op">=</span> b<span class="op">.</span><span class="fu">createQueryStringFromNode</span><span class="op">(</span>search<span class="op">.</span><span class="fu">getCriteria</span><span class="op">(),</span> <span class="kw">null</span><span class="op">);</span></span></code></pre></div>
<p>Things go like this:</p>
<p>fhir-client: fhir-syntax-operators -> fhir-backend:
fhir-syntax-operator => solr-syntax-operator -> spark-backend:
parser,joins, results -> create a patient list within solr</p>
<h1 id="group-versus-list">Group versus List</h1>
<ul>
<li>it is possible to filter resources by Group thanks to reverse
chaining</li>
<li><span class="title-ref">[Base]/Patient?_has:Group:member:_id=xxxx</span></li>
<li><span class="title-ref">[Base]/Encounter?patient._has:Group:member:_id=xxxx</span></li>
<li><span class="title-ref">[Base]/Observation?patient._has:Group:member:_id=xxxx</span></li>
<li><span class="title-ref">[Base]/Composition?patient._has:Group:member:_id=xxxx</span></li>
<li>it is possible to filter resources by List thanks to _list
parameter:</li>
<li><span class="title-ref">[Base]/Patient?_list=xxxx</span> where _list
is a list of patients</li>
<li><span class="title-ref">[Base]/Encounter?_list=xxxx</span> where
_list is a list of encounters</li>
</ul>
<h1 id="results-informations">Results Informations</h1>
<ul>
<li>the resulting json should contain:
<ul>
<li>the results on the whole</li>
<li>the results on the practitioner ward</li>
<li>the faceting results</li>
</ul></li>
</ul>
<p>Most relevant FHIR ressources have two, three or four of those
elements:</p>
<ol>
<li>id</li>
<li>patient</li>
<li>encounter</li>
<li>date</li>
</ol>
<h1 id="patient-level">Patient Level</h1>
<p>Every filtered ressource being linked to a <strong>patient</strong>
it is possible to operate on them all:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode scala"><code class="sourceCode scala"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r0 <span class="op">=</span> <span class="op">(</span><span class="dv">1</span><span class="op">::</span><span class="dv">2</span><span class="op">::</span><span class="dv">3</span><span class="op">::</span><span class="dv">6</span><span class="op">::</span><span class="dv">7</span><span class="op">::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">)</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r1 <span class="op">=</span> <span class="op">(</span><span class="dv">1</span><span class="op">::</span><span class="dv">2</span><span class="op">::</span><span class="dv">3</span><span class="op">::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">)</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r2 <span class="op">=</span> <span class="op">(</span><span class="dv">2</span><span class="op">::</span><span class="dv">3</span><span class="op">::</span><span class="dv">6</span><span class="op">::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">)</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r3 <span class="op">=</span> <span class="op">(</span><span class="dv">1</span><span class="op">::</span><span class="dv">3</span><span class="op">::</span><span class="dv">7</span><span class="op">::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">)</span></span></code></pre></div>
<ul>
<li>`r1 AND r2 AND NOT r3`:
r1.intersect(r1).join(r3,"patient"::Nil,"leftanti").dropDuplicates.show</li>
<li>`r1 OR r2 OR NOT r3`:
r1.union(r2).union(r0.join(r3,"patient"::Nil,"leftanti")).dropDuplicates.show</li>
<li>`(r1 OR r2) AND r3`:
r3.intersect(r1.union(r2)).dropDuplicates.show</li>
</ul>
<h1 id="encounter-level">Encounter Level</h1>
<div class="sourceCode" id="cb3"><pre class="sourceCode scala"><code class="sourceCode scala"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r1 <span class="op">=</span> <span class="op">((</span><span class="dv">1</span><span class="op">,</span><span class="dv">10</span><span class="op">)::(</span><span class="dv">1</span><span class="op">,</span><span class="dv">11</span><span class="op">)::(</span><span class="dv">3</span><span class="op">,</span><span class="dv">12</span><span class="op">)::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">,</span> <span class="st">"encounter"</span><span class="op">)</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r2 <span class="op">=</span> <span class="op">((</span><span class="dv">1</span><span class="op">,</span><span class="dv">11</span><span class="op">)::(</span><span class="dv">3</span><span class="op">,</span><span class="dv">12</span><span class="op">)::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">,</span> <span class="st">"encounter"</span><span class="op">)</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r3 <span class="op">=</span> <span class="op">(</span><span class="dv">1</span><span class="op">::</span><span class="dv">3</span><span class="op">::</span><span class="dv">7</span><span class="op">::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">)</span></span></code></pre></div>
<ul>
<li>`r1 SAME_ENC r2`: r1.join(r2, "encounter"::Nil,
"leftsemi").select("patient")</li>
<li>`r1 SAME_ENC NOT r2`: r1.join(r2, "encounter"::Nil,
"leftanti").select("patient")</li>
<li>`r1 SAME_ENC r2 OR_PAT r3`: r1.join(r2, "encounter"::Nil,
"leftsemi").select("patient").union(r3)</li>
</ul>
<h1 id="date-level">Date Level</h1>
<div class="sourceCode" id="cb4"><pre class="sourceCode scala"><code class="sourceCode scala"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r1 <span class="op">=</span> <span class="op">((</span><span class="dv">1</span><span class="op">,</span><span class="kw">new</span> <span class="ex">Date</span><span class="op">(</span>1L<span class="op">))::(</span><span class="dv">1</span><span class="op">,</span><span class="kw">new</span> <span class="ex">Date</span><span class="op">(</span>2L<span class="op">))::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">,</span> <span class="st">"date"</span><span class="op">)</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="kw">val</span> r2 <span class="op">=</span> <span class="op">((</span><span class="dv">1</span><span class="op">,</span><span class="kw">new</span> <span class="ex">Date</span><span class="op">(</span>1L<span class="op">))::(</span><span class="dv">3</span><span class="op">,</span><span class="kw">new</span> <span class="ex">Date</span><span class="op">(</span>2L<span class="op">))::</span>Nil<span class="op">).</span><span class="fu">toDF</span><span class="op">(</span><span class="st">"patient"</span><span class="op">,</span> <span class="st">"date"</span><span class="op">)</span></span></code></pre></div>
<ul>
<li>`r1 AFTER r2`: r1.join(r2, "encounter"::Nil,
"leftsemi").select("patient")</li>
<li>`r1 AFTER_24h r2`: r1.as("r1").join(r2.as("r2"), "patient"::Nil,
"left").filter(col("r1.date").>=(expr("r2.date + INTERVAL 24
HOURS")))</li>
<li>`r1 NOT AFTER_24h r2`: r1.as("r1").join(r2.as("r2"), "patient"::Nil,
"full").filter(col("r1.date").isNull ||
!col("r1.date").>=(col("r2.date")))</li>
</ul>
<h1 id="all-together">All Together</h1>
<h1 id="number-of-occurrence">Number Of Occurrence</h1>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=FHIR Search Engine -
Resource Operators — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-06 22:05</time> </div>https://blog.parisni.com/blog/postgres-sd1-and-sc2-strategy-comparisonPostgres SCD1 and SCD2 strategy comparison2020-06-06T22:05:00+00:00<div id="generated-toc"> </div><hr>
<p><span>Published on <time id="post-date">2019-11-17</time></span>
<span id="reading-time" style="visibility:hidden;">5 minutes</span></p>
<h1>Postgres SCD1 and SCD2 strategy comparison</h1>
<p>In this experimentation, we will demonstrate that using one CTE which
differentiate both update and insert improve significantly the merge processus for both SCD1 and SCD2</p>
<h1>Experiment tables</h1>
<pre><code class="language-sql"><span class="hl kwa">drop table if exists</span> target<span class="hl opt">;</span>
<span class="hl kwa">drop table if exists</span> tmp<span class="hl opt">;</span>
<span class="hl kwa">create table</span> <span class="hl kwd">target</span><span class="hl opt">(</span><span class="hl kwa">id</span> <span class="hl kwb">int</span><span class="hl opt">,</span> cd <span class="hl kwb">text</span><span class="hl opt">,</span> hash <span class="hl kwb">int</span><span class="hl opt">,</span> end_datetime <span class="hl kwa">timestamp</span><span class="hl opt">);</span>
<span class="hl kwa">create table</span> <span class="hl kwd">tmp</span><span class="hl opt">(</span><span class="hl kwa">id</span> <span class="hl kwb">int</span><span class="hl opt">,</span> cd <span class="hl kwb">text</span><span class="hl opt">,</span> hash <span class="hl kwb">int</span><span class="hl opt">);</span>
<span class="hl slc">-- TARGET</span>
<span class="hl kwa">insert into</span> <span class="hl kwd">target</span><span class="hl opt">(</span><span class="hl kwa">id</span><span class="hl opt">,</span>cd<span class="hl opt">,</span>hash<span class="hl opt">)</span> <span class="hl kwa">SELECT</span> g.<span class="hl opt">*</span> <span class="hl kwa">as id</span><span class="hl opt">,</span> <span class="hl sng">'jim'</span> <span class="hl kwa">as</span> cd<span class="hl opt">,</span> g.<span class="hl opt">*</span> <span class="hl kwa">as</span> hash <span class="hl kwa">FROM</span> <span class="hl kwd">generate_series</span><span class="hl opt">(</span><span class="hl num">1</span><span class="hl opt">,</span><span class="hl num">10000000</span><span class="hl opt">)</span> g<span class="hl opt">;</span>
<span class="hl slc">-- TMP</span>
<span class="hl kwa">insert into</span> <span class="hl kwd">tmp</span><span class="hl opt">(</span><span class="hl kwa">id</span><span class="hl opt">,</span>cd<span class="hl opt">,</span>hash<span class="hl opt">)</span> <span class="hl kwa">SELECT</span> g.<span class="hl opt">*</span> <span class="hl kwa">as id</span><span class="hl opt">,</span> <span class="hl sng">'jim'</span> <span class="hl kwa">as</span> cd<span class="hl opt">,</span> g.<span class="hl opt">*</span> <span class="hl kwa">as</span> hash <span class="hl kwa">FROM</span> <span class="hl kwd">generate_series</span><span class="hl opt">(</span><span class="hl num">4000000</span><span class="hl opt">,</span><span class="hl num">5000000</span><span class="hl opt">)</span> g<span class="hl opt">;</span>
<span class="hl kwa">insert into</span> <span class="hl kwd">tmp</span><span class="hl opt">(</span><span class="hl kwa">id</span><span class="hl opt">,</span>cd<span class="hl opt">,</span>hash<span class="hl opt">)</span> <span class="hl kwa">SELECT</span> g.<span class="hl opt">*</span> <span class="hl kwa">as id</span><span class="hl opt">,</span> <span class="hl sng">'john'</span> <span class="hl kwa">as</span> cd<span class="hl opt">,</span> g.<span class="hl opt">*</span> <span class="hl kwa">as</span> hash <span class="hl kwa">FROM</span> <span class="hl kwd">generate_series</span><span class="hl opt">(</span><span class="hl num">4000000</span><span class="hl opt">,</span><span class="hl num">5000000</span><span class="hl opt">)</span> g<span class="hl opt">;</span>
<span class="hl kwa">insert into</span> <span class="hl kwd">tmp</span><span class="hl opt">(</span><span class="hl kwa">id</span><span class="hl opt">,</span>cd<span class="hl opt">,</span>hash<span class="hl opt">)</span> <span class="hl kwa">SELECT</span> g.<span class="hl opt">*</span> <span class="hl kwa">as id</span><span class="hl opt">,</span> <span class="hl sng">'john'</span> <span class="hl kwa">as</span> cd<span class="hl opt">,</span> g.<span class="hl opt">*</span> <span class="hl kwa">as</span> hash <span class="hl kwa">FROM</span> <span class="hl kwd">generate_series</span><span class="hl opt">(</span><span class="hl num">14000000</span><span class="hl opt">,</span><span class="hl num">15000000</span><span class="hl opt">)</span> g<span class="hl opt">;</span>
</code></pre>
<h1>Atomic implementation scd1</h1>
<pre><code class="language-sql"><span class="hl kwa">BEGIN</span><span class="hl opt">;</span>
<span class="hl kwa">LOCK TABLE</span> target <span class="hl kwa">IN EXCLUSIVE MODE</span><span class="hl opt">;</span>
<span class="hl kwa">explain analyse</span>
<span class="hl kwa">with</span> fj <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">select</span>
tmp.<span class="hl opt">*</span>
<span class="hl opt">,</span> <span class="hl kwa">case when</span> targ.<span class="hl kwa">id is null then false else true end as</span> upd
<span class="hl kwa">from</span> target targ
<span class="hl kwa">full outer join</span> tmp <span class="hl kwa">on</span> <span class="hl opt">(</span>targ.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">where</span> targ.<span class="hl kwa">id is null or</span> targ.hash <span class="hl opt">!=</span> tmp.hash
<span class="hl opt">),</span>
upd <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">update</span> target targ <span class="hl kwa">set id</span><span class="hl opt">=</span>fj.<span class="hl kwa">id</span><span class="hl opt">,</span> cd<span class="hl opt">=</span>fj.cd<span class="hl opt">,</span> hash<span class="hl opt">=</span>fj.hash
<span class="hl kwa">from</span> fj
<span class="hl kwa">where</span> fj.<span class="hl kwa">id</span> <span class="hl opt">=</span> targ.<span class="hl kwa">id</span>
<span class="hl kwa">and</span> fj.upd <span class="hl kwa">is true</span><span class="hl opt">)</span>
<span class="hl kwa">insert into</span> target <span class="hl kwa">select id</span><span class="hl opt">,</span> cd<span class="hl opt">,</span> hash <span class="hl kwa">from</span> fj <span class="hl kwa">where</span> fj.upd <span class="hl kwa">is false</span><span class="hl opt">;</span>
<span class="hl kwa">ROLLBACK</span><span class="hl opt">;</span>
<span class="hl slc">-- QUERY PLAN </span>
<span class="hl slc">-- ----------------------------------------------------------------------------------------------------------------------------------------------</span>
<span class="hl slc">-- Insert on target (cost=1139239.61..1364236.44 rows=49999 width=48) (actual time=6858.761..6858.761 rows=0 loops=1)</span>
<span class="hl slc">-- CTE fj</span>
<span class="hl slc">-- -> Hash Full Join (cost=318113.85..515190.96 rows=9999859 width=44) (actual time=3340.633..5629.141 rows=1000001 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp.id = targ.id)</span>
<span class="hl slc">-- Filter: ((targ.id IS NULL) OR (targ.hash <> tmp.hash))</span>
<span class="hl slc">-- Rows Removed by Filter: 11000001</span>
<span class="hl slc">-- -> Seq Scan on tmp (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.007..227.686 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=154053.60..154053.60 rows=9999860 width=8) (actual time=2919.111..2919.111 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 131072 Batches: 256 Memory Usage: 2553kB</span>
<span class="hl slc">-- -> Seq Scan on target targ (cost=0.00..154053.60 rows=9999860 width=8) (actual time=0.042..1421.153 rows=10000000 loops=1)</span>
<span class="hl slc">-- CTE upd</span>
<span class="hl slc">-- -> Update on target targ_1 (cost=337644.85..624048.65 rows=49999 width=118) (actual time=109.975..109.975 rows=0 loops=1)</span>
<span class="hl slc">-- -> Hash Join (cost=337644.85..624048.65 rows=49999 width=118) (actual time=109.974..109.974 rows=0 loops=1)</span>
<span class="hl slc">-- Hash Cond: (fj_1.id = targ_1.id)</span>
<span class="hl slc">-- -> CTE Scan on fj fj_1 (cost=0.00..224996.83 rows=49999 width=104) (actual time=109.973..109.973 rows=0 loops=1)</span>
<span class="hl slc">-- Filter: (type = 'update'::text)</span>
<span class="hl slc">-- Rows Removed by Filter: 1000001</span>
<span class="hl slc">-- -> Hash (cost=154053.60..154053.60 rows=9999860 width=18) (never executed)</span>
<span class="hl slc">-- -> Seq Scan on target targ_1 (cost=0.00..154053.60 rows=9999860 width=18) (never executed)</span>
<span class="hl slc">-- -> CTE Scan on fj (cost=0.00..224996.83 rows=49999 width=48) (actual time=3340.639..6016.017 rows=1000001 loops=1)</span>
<span class="hl slc">-- Filter: (type = 'insert'::text)</span>
<span class="hl slc">-- Planning time: 0.518 ms</span>
<span class="hl slc">-- Execution time: 6974.737 ms</span>
</code></pre>
<pre><code class="language-sql"><span class="hl kwa">BEGIN</span><span class="hl opt">;</span>
<span class="hl kwa">explain analyse</span>
<span class="hl kwa">with</span> u <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">UPDATE</span> target
<span class="hl kwa">SET id</span><span class="hl opt">=</span>tmp.<span class="hl kwa">id</span><span class="hl opt">,</span> cd<span class="hl opt">=</span>tmp.cd<span class="hl opt">,</span> hash<span class="hl opt">=</span>tmp.hash
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">WHERE</span> tmp.<span class="hl kwa">id</span> <span class="hl opt">=</span> target.<span class="hl kwa">id</span>
<span class="hl opt">)</span>
<span class="hl kwa">INSERT INTO</span> target
<span class="hl kwa">SELECT</span> tmp.<span class="hl opt">*</span>
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">LEFT OUTER JOIN</span> target <span class="hl kwa">ON</span> <span class="hl opt">(</span>target.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">WHERE</span> target.<span class="hl kwa">id IS NULL</span><span class="hl opt">;</span>
<span class="hl kwa">ROLLBACK</span><span class="hl opt">;</span>
<span class="hl slc">-- QUERY PLAN </span>
<span class="hl slc">-- ----------------------------------------------------------------------------------------------------------------------------------------------------------</span>
<span class="hl slc">-- Insert on target (cost=945906.04..1113139.12 rows=1 width=20) (actual time=5206.659..5206.659 rows=0 loops=1)</span>
<span class="hl slc">-- CTE u</span>
<span class="hl slc">-- -> Update on target target_2 (cost=371406.47..595983.57 rows=3000003 width=32) (actual time=8586.434..8586.434 rows=0 loops=1)</span>
<span class="hl slc">-- -> Hash Join (cost=371406.47..595983.57 rows=3000003 width=32) (actual time=2899.041..5459.650 rows=2000002 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp_1.id = target_2.id)</span>
<span class="hl slc">-- -> Seq Scan on tmp tmp_1 (cost=0.00..46217.03 rows=3000003 width=18) (actual time=0.024..403.061 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=169457.54..169457.54 rows=10999754 width=18) (actual time=2896.588..2896.588 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 65536 Batches: 256 Memory Usage: 2195kB</span>
<span class="hl slc">-- -> Seq Scan on target target_2 (cost=0.00..169457.54 rows=10999754 width=18) (actual time=0.012..1319.621 rows=10000000 loops=1)</span>
<span class="hl slc">-- -> Hash Anti Join (cost=349922.47..517155.54 rows=1 width=20) (actual time=3003.185..4367.119 rows=1000001 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp.id = target_1.id)</span>
<span class="hl slc">-- -> Seq Scan on tmp (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.027..220.187 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=169457.54..169457.54 rows=10999754 width=4) (actual time=2591.285..2591.285 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 131072 Batches: 256 Memory Usage: 2400kB</span>
<span class="hl slc">-- -> Seq Scan on target target_1 (cost=0.00..169457.54 rows=10999754 width=4) (actual time=0.035..1162.839 rows=10000000 loops=1)</span>
<span class="hl slc">-- Planning time: 0.206 ms</span>
<span class="hl slc">-- Execution time: 13793.154 ms</span>
</code></pre>
<pre><code class="language-sql"><span class="hl kwa">BEGIN</span><span class="hl opt">;</span>
<span class="hl kwa">explain analyse</span>
<span class="hl kwa">with</span> u <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">UPDATE</span> target
<span class="hl kwa">SET id</span><span class="hl opt">=</span>tmp.<span class="hl kwa">id</span><span class="hl opt">,</span> cd<span class="hl opt">=</span>tmp.cd<span class="hl opt">,</span> hash<span class="hl opt">=</span>tmp.hash
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">WHERE</span> tmp.<span class="hl kwa">id</span> <span class="hl opt">=</span> target.<span class="hl kwa">id</span>
<span class="hl kwa">returning</span> target.<span class="hl kwa">id</span>
<span class="hl opt">),</span>
i <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">select</span> tmp.<span class="hl opt">*</span>
<span class="hl kwa">from</span> tmp
<span class="hl kwa">left join</span> u <span class="hl kwa">using</span> <span class="hl opt">(</span><span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">where</span> u.<span class="hl kwa">id is null</span>
<span class="hl opt">)</span>
<span class="hl kwa">INSERT INTO</span> target
<span class="hl kwa">SELECT</span> i.<span class="hl opt">*</span>
<span class="hl kwa">FROM</span> i
<span class="hl kwa">LEFT OUTER JOIN</span> target <span class="hl kwa">ON</span> <span class="hl opt">(</span>target.<span class="hl kwa">id</span> <span class="hl opt">=</span> i.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">WHERE</span> target.<span class="hl kwa">id IS NULL</span><span class="hl opt">;</span>
<span class="hl kwa">ROLLBACK</span><span class="hl opt">;</span>
<span class="hl slc">-- QUERY PLAN </span>
<span class="hl slc">-- ----------------------------------------------------------------------------------------------------------------------------------------------------------</span>
<span class="hl slc">-- Insert on target (cost=2111532.64..2235532.95 rows=750001 width=48) (actual time=17158.676..17158.676 rows=0 loops=1)</span>
<span class="hl slc">-- CTE u</span>
<span class="hl slc">-- -> Update on target target_2 (cost=438935.84..682731.95 rows=3000003 width=32) (actual time=2858.907..8387.049 rows=1000001 loops=1)</span>
<span class="hl slc">-- -> Hash Join (cost=438935.84..682731.95 rows=3000003 width=32) (actual time=2858.892..5487.211 rows=2000002 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp.id = target_2.id)</span>
<span class="hl slc">-- -> Seq Scan on tmp (cost=0.00..46217.03 rows=3000003 width=18) (actual time=0.026..376.514 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=200268.26..200268.26 rows=12999726 width=18) (actual time=2856.471..2856.471 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 65536 Batches: 256 Memory Usage: 2195kB</span>
<span class="hl slc">-- -> Seq Scan on target target_2 (cost=0.00..200268.26 rows=12999726 width=18) (actual time=0.014..1352.460 rows=10000000 loops=1)</span>
<span class="hl slc">-- CTE i</span>
<span class="hl slc">-- -> Merge Anti Join (cost=936289.71..1015254.85 rows=1500002 width=12) (actual time=10701.594..10942.156 rows=1000001 loops=1)</span>
<span class="hl slc">-- Merge Cond: (tmp_1.id = u.id)</span>
<span class="hl slc">-- -> Sort (cost=471508.34..479008.35 rows=3000003 width=12) (actual time=1143.032..1468.744 rows=3000003 loops=1)</span>
<span class="hl slc">-- Sort Key: tmp_1.id</span>
<span class="hl slc">-- Sort Method: external merge Disk: 72464kB</span>
<span class="hl slc">-- -> Seq Scan on tmp tmp_1 (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.075..266.799 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Sort (cost=464781.37..472281.38 rows=3000003 width=4) (actual time=8993.435..9113.745 rows=1000001 loops=1)</span>
<span class="hl slc">-- Sort Key: u.id</span>
<span class="hl slc">-- Sort Method: external merge Disk: 13800kB</span>
<span class="hl slc">-- -> CTE Scan on u (cost=0.00..60000.06 rows=3000003 width=4) (actual time=2858.909..8670.080 rows=1000001 loops=1)</span>
<span class="hl slc">-- -> Hash Anti Join (cost=413545.84..537546.15 rows=750001 width=48) (actual time=13088.538..15120.417 rows=1000001 loops=1)</span>
<span class="hl slc">-- Hash Cond: (i.id = target_1.id)</span>
<span class="hl slc">-- -> CTE Scan on i (cost=0.00..30000.04 rows=1500002 width=40) (actual time=10701.598..11209.117 rows=1000001 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=200268.26..200268.26 rows=12999726 width=4) (actual time=2386.139..2386.139 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 131072 Batches: 256 Memory Usage: 2400kB</span>
<span class="hl slc">-- -> Seq Scan on target target_1 (cost=0.00..200268.26 rows=12999726 width=4) (actual time=0.025..1091.998 rows=10000000 loops=1)</span>
<span class="hl slc">-- Planning time: 0.474 ms</span>
<span class="hl slc">-- Execution time: 17204.783 ms</span>
</code></pre>
<pre><code class="language-sql"><span class="hl kwa">BEGIN</span><span class="hl opt">;</span>
<span class="hl kwa">explain ANALYSE with</span> u <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">INSERT INTO</span> target
<span class="hl kwa">SELECT</span> tmp.<span class="hl opt">*</span>
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">LEFT OUTER JOIN</span> target <span class="hl kwa">ON</span> <span class="hl opt">(</span>target.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">WHERE</span> target.<span class="hl kwa">id IS NULL</span>
<span class="hl opt">)</span>
<span class="hl kwa">UPDATE</span> target
<span class="hl kwa">SET id</span><span class="hl opt">=</span>tmp.<span class="hl kwa">id</span><span class="hl opt">,</span> cd<span class="hl opt">=</span>tmp.cd<span class="hl opt">,</span> hash<span class="hl opt">=</span>tmp.hash
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">WHERE</span> tmp.<span class="hl kwa">id</span> <span class="hl opt">=</span> target.<span class="hl kwa">id</span><span class="hl opt">;</span>
<span class="hl kwa">ROLLBACK</span><span class="hl opt">;</span>
<span class="hl slc">-- QUERY PLAN </span>
<span class="hl slc">-- ---------------------------------------------------------------------------------------------------------------------------------------------------------</span>
<span class="hl slc">-- Update on target (cost=1181490.50..1444504.63 rows=3000003 width=32) (actual time=8281.352..8281.352 rows=0 loops=1)</span>
<span class="hl slc">-- CTE u</span>
<span class="hl slc">-- -> Insert on target target_1 (cost=477168.21..675026.30 rows=1 width=20) (actual time=5430.576..5430.576 rows=0 loops=1)</span>
<span class="hl slc">-- -> Hash Anti Join (cost=477168.21..675026.30 rows=1 width=20) (actual time=2954.507..4443.851 rows=1000001 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp_1.id = target_2.id)</span>
<span class="hl slc">-- -> Seq Scan on tmp tmp_1 (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.019..256.940 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=231078.98..231078.98 rows=14999698 width=4) (actual time=2520.272..2520.272 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 131072 Batches: 256 Memory Usage: 2400kB</span>
<span class="hl slc">-- -> Seq Scan on target target_2 (cost=0.00..231078.98 rows=14999698 width=4) (actual time=0.014..1175.381 rows=10000000 loops=1)</span>
<span class="hl slc">-- -> Hash Join (cost=506464.21..769478.33 rows=3000003 width=32) (actual time=2964.008..5611.788 rows=2000002 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp.id = target.id)</span>
<span class="hl slc">-- -> Seq Scan on tmp (cost=0.00..46217.03 rows=3000003 width=18) (actual time=0.071..429.298 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=231078.98..231078.98 rows=14999698 width=18) (actual time=2961.366..2961.366 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 65536 Batches: 256 Memory Usage: 2195kB</span>
<span class="hl slc">-- -> Seq Scan on target (cost=0.00..231078.98 rows=14999698 width=18) (actual time=0.034..1409.287 rows=10000000 loops=1)</span>
<span class="hl slc">-- Planning time: 0.424 ms</span>
<span class="hl slc">-- Execution time: 13712.030 ms</span>
</code></pre>
<pre><code class="language-sql"><span class="hl esc">\t</span>iming
<span class="hl kwa">BEGIN</span><span class="hl opt">;</span>
<span class="hl kwa">UPDATE</span> target
<span class="hl kwa">SET id</span><span class="hl opt">=</span>tmp.<span class="hl kwa">id</span><span class="hl opt">,</span> cd<span class="hl opt">=</span>tmp.cd<span class="hl opt">,</span> hash<span class="hl opt">=</span>tmp.hash
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">WHERE</span> tmp.<span class="hl kwa">id</span> <span class="hl opt">=</span> target.<span class="hl kwa">id</span><span class="hl opt">;</span>
<span class="hl kwa">INSERT INTO</span> target
<span class="hl kwa">SELECT</span> tmp.<span class="hl opt">*</span>
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">LEFT OUTER JOIN</span> target <span class="hl kwa">ON</span> <span class="hl opt">(</span>target.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">WHERE</span> target.<span class="hl kwa">id IS NULL</span><span class="hl opt">;</span>
<span class="hl kwa">ROLLBACK</span><span class="hl opt">;</span>
<span class="hl slc">-- Time: 5329,939 ms (00:05,330)</span>
<span class="hl slc">-- Time: 12554,331 ms (00:12,554)</span>
</code></pre>
<h1>Atomic implementation scd2</h1>
<pre><code class="language-sql"><span class="hl kwa">begin</span><span class="hl opt">;</span>
<span class="hl kwa">explain analyse</span>
<span class="hl kwa">with</span> i <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">INSERT INTO</span> target
<span class="hl kwa">SELECT</span> tmp.<span class="hl opt">*</span>
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">LEFT OUTER JOIN</span> target <span class="hl kwa">ON</span> <span class="hl opt">(</span>target.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">WHERE</span> target.<span class="hl kwa">id IS NULL</span>
<span class="hl opt">),</span>
u <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">INSERT INTO</span> target
<span class="hl kwa">SELECT</span> tmp.<span class="hl opt">*</span>
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">JOIN</span> target <span class="hl kwa">ON</span> <span class="hl opt">(</span>target.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">WHERE</span> target.hash <span class="hl kwa">is distinct from</span> tmp.hash
<span class="hl kwa">returning</span> target.<span class="hl kwa">id</span>
<span class="hl opt">)</span>
<span class="hl kwa">UPDATE</span> target
<span class="hl kwa">SET</span> end_datetime <span class="hl opt">=</span> <span class="hl kwd">now</span><span class="hl opt">()</span>
<span class="hl kwa">FROM</span> u
<span class="hl kwa">WHERE</span> u.<span class="hl kwa">id</span> <span class="hl opt">=</span> target.<span class="hl kwa">id</span><span class="hl opt">;</span>
<span class="hl kwa">rollback</span><span class="hl opt">;</span>
<span class="hl slc">-- QUERY PLAN </span>
<span class="hl slc">-- --------------------------------------------------------------------------------------------------------------------------------------------------------</span>
<span class="hl slc">-- Update on target (cost=1560116.39..1800721.53 rows=3000003 width=54) (actual time=6156.396..6156.396 rows=0 loops=1)</span>
<span class="hl slc">-- CTE i</span>
<span class="hl slc">-- -> Insert on target target_1 (cost=400062.40..552387.47 rows=1 width=20) (actual time=8926.725..8926.725 rows=0 loops=1)</span>
<span class="hl slc">-- -> Hash Anti Join (cost=400062.40..552387.47 rows=1 width=20) (actual time=3602.728..5597.313 rows=1000001 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp.id = target_2.id)</span>
<span class="hl slc">-- -> Seq Scan on tmp (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.450..300.704 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=250713.29..250713.29 rows=9103129 width=4) (actual time=3092.523..3092.523 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 131072 Batches: 128 Memory Usage: 3781kB</span>
<span class="hl slc">-- -> Seq Scan on target target_2 (cost=0.00..250713.29 rows=9103129 width=4) (actual time=0.010..1519.702 rows=10000000 loops=1)</span>
<span class="hl slc">-- CTE u</span>
<span class="hl slc">-- -> Insert on target target_3 (cost=400062.40..589887.51 rows=3000003 width=20) (actual time=6156.392..6156.393 rows=0 loops=1)</span>
<span class="hl slc">-- -> Hash Join (cost=400062.40..589887.51 rows=3000003 width=20) (actual time=6156.391..6156.391 rows=0 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp_1.id = target_4.id)</span>
<span class="hl slc">-- Join Filter: (target_4.hash IS DISTINCT FROM tmp_1.hash)</span>
<span class="hl slc">-- Rows Removed by Join Filter: 2000002</span>
<span class="hl slc">-- -> Seq Scan on tmp tmp_1 (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.013..334.453 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=250713.29..250713.29 rows=9103129 width=8) (actual time=3359.346..3359.346 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 131072 (originally 131072) Batches: 256 (originally 128) Memory Usage: 4087kB</span>
<span class="hl slc">-- -> Seq Scan on target target_4 (cost=0.00..250713.29 rows=9103129 width=8) (actual time=0.014..1677.709 rows=10000000 loops=1)</span>
<span class="hl slc">-- -> Hash Join (cost=417841.40..658446.55 rows=3000003 width=54) (actual time=6156.394..6156.395 rows=0 loops=1)</span>
<span class="hl slc">-- Hash Cond: (u.id = target.id)</span>
<span class="hl slc">-- -> CTE Scan on u (cost=0.00..60000.06 rows=3000003 width=32) (actual time=6156.394..6156.394 rows=0 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=250713.29..250713.29 rows=9103129 width=18) (never executed)</span>
<span class="hl slc">-- -> Seq Scan on target (cost=0.00..250713.29 rows=9103129 width=18) (never executed)</span>
<span class="hl slc">-- Planning time: 0.220 ms</span>
<span class="hl slc">-- Execution time: 15083.304 ms</span>
</code></pre>
<pre><code class="language-sql"><span class="hl kwa">begin</span><span class="hl opt">;</span>
<span class="hl kwa">explain analyze</span>
<span class="hl kwa">with</span> tmp <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">select</span>
tmp.<span class="hl opt">*</span>
<span class="hl opt">,</span> <span class="hl kwa">case when</span> targ.<span class="hl kwa">id is null then false else true end as</span> upd
<span class="hl kwa">from</span> target targ
<span class="hl kwa">full outer join</span> tmp <span class="hl kwa">on</span> <span class="hl opt">(</span>targ.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span><span class="hl opt">)</span>
<span class="hl kwa">where</span> targ.<span class="hl kwa">id is null or</span> targ.hash <span class="hl opt">!=</span> tmp.hash
<span class="hl opt">),</span>
u <span class="hl kwa">as</span> <span class="hl opt">(</span>
<span class="hl kwa">UPDATE</span> target
<span class="hl kwa">SET</span> end_datetime <span class="hl opt">=</span> <span class="hl kwd">now</span><span class="hl opt">()</span>
<span class="hl kwa">FROM</span> tmp
<span class="hl kwa">where</span>
target.<span class="hl kwa">id</span> <span class="hl opt">=</span> tmp.<span class="hl kwa">id</span>
<span class="hl kwa">AND</span> tmp.upd <span class="hl kwa">is true</span>
<span class="hl opt">)</span>
<span class="hl kwa">INSERT INTO</span> <span class="hl kwd">target</span> <span class="hl opt">(</span><span class="hl kwa">id</span><span class="hl opt">,</span> cd<span class="hl opt">,</span> hash<span class="hl opt">)</span>
<span class="hl kwa">SELECT id</span><span class="hl opt">,</span> cd<span class="hl opt">,</span> hash
<span class="hl kwa">FROM</span> tmp<span class="hl opt">;</span>
<span class="hl kwa">rollback</span><span class="hl opt">;</span>
<span class="hl slc">-- QUERY PLAN </span>
<span class="hl slc">-- ----------------------------------------------------------------------------------------------------------------------------------------------</span>
<span class="hl slc">-- Insert on target (cost=1166655991509.07..1166656191396.43 rows=9994368 width=48) (actual time=8666.204..8666.204 rows=0 loops=1)</span>
<span class="hl slc">-- CTE tmp</span>
<span class="hl slc">-- -> Hash Full Join (cost=438165.30..635221.42 rows=9994368 width=13) (actual time=4531.084..7278.771 rows=1000001 loops=1)</span>
<span class="hl slc">-- Hash Cond: (tmp_1.id = targ.id)</span>
<span class="hl slc">-- Filter: ((targ.id IS NULL) OR (targ.hash <> tmp_1.hash))</span>
<span class="hl slc">-- Rows Removed by Filter: 11000001</span>
<span class="hl slc">-- -> Seq Scan on tmp tmp_1 (cost=0.00..46217.03 rows=3000003 width=12) (actual time=0.140..307.256 rows=3000003 loops=1)</span>
<span class="hl slc">-- -> Hash (cost=274194.69..274194.69 rows=9994369 width=8) (actual time=3974.719..3974.719 rows=10000000 loops=1)</span>
<span class="hl slc">-- Buckets: 131072 Batches: 256 Memory Usage: 2553kB</span>
<span class="hl slc">-- -> Seq Scan on target targ (cost=0.00..274194.69 rows=9994369 width=8) (actual time=0.059..2009.929 rows=10000000 loops=1)</span>
<span class="hl slc">-- CTE u</span>
<span class="hl slc">-- -> Update on target target_1 (cost=0.00..1166655356287.65 rows=49943700856896 width=50) (actual time=98.574..98.574 rows=0 loops=1)</span>
<span class="hl slc">-- -> Nested Loop (cost=0.00..1166655356287.65 rows=49943700856896 width=50) (actual time=98.573..98.573 rows=0 loops=1)</span>
<span class="hl slc">-- -> CTE Scan on tmp tmp_2 (cost=0.00..199887.36 rows=4997184 width=24) (actual time=98.572..98.572 rows=0 loops=1)</span>
<span class="hl slc">-- Filter: (upd IS TRUE)</span>
<span class="hl slc">-- Rows Removed by Filter: 1000001</span>
<span class="hl slc">-- -> Materialize (cost=0.00..382727.54 rows=9994369 width=18) (never executed)</span>
<span class="hl slc">-- -> Seq Scan on target target_1 (cost=0.00..274194.69 rows=9994369 width=18) (never executed)</span>
<span class="hl slc">-- -> CTE Scan on tmp (cost=0.00..199887.36 rows=9994368 width=48) (actual time=4531.092..7663.395 rows=1000001 loops=1)</span>
<span class="hl slc">-- Planning time: 0.137 ms</span>
<span class="hl slc">-- Execution time: 8769.648 ms</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Postgres SCD1 and SCD2 strategy comparison — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-06 22:05</time> </div>https://blog.parisni.com/blog/first-contact-with-spark-jobserverFirst contact with spark-jobserver2020-06-20T19:02:00+00:00<div id="generated-toc"> </div><hr>
<div id="generated-toc"> </div>
<hr>
<p><span>Published on <time id="post-date">2019-11-24</time></span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>First contact with spark-jobserver</h1>
<p>This describe how to test locally a spark-jobserver instance. <a href="https://github.com/spark-jobserver/">This derive from the readme</a>. First impressions are great.</p>
<h2>Server side</h2>
<pre><code class="language-bash">git clone git@github.com<span class="hl opt">:</span>spark<span class="hl kwb">-jobserver</span><span class="hl opt">/</span>spark<span class="hl kwb">-jobserver</span>.git
</code></pre>
<h2>Activate Basic Auth</h2>
<pre><code class="language-yaml"><span class="hl slc"># add this at the end config/local.conf</span>
shiro <span class="hl opt">{</span>
authentication <span class="hl opt">=</span> on
<span class="hl slc"># absolute path to shiro config file, including file name</span>
config.path <span class="hl opt">=</span> <span class="hl sng">"config/shiro.ini.basic"</span>
<span class="hl opt">}</span>
<span class="hl slc"># specify the number of query per slot (default 8)</span>
spark <span class="hl opt">{</span>
jobserver.max<span class="hl opt">-</span>jobs<span class="hl opt">-</span>per<span class="hl opt">-</span>context <span class="hl opt">=</span> <span class="hl num">2</span>
<span class="hl opt">}</span>
</code></pre>
<p><a href="https://github.com/spark-jobserver/spark-jobserver/blob/2eaa041070e5a4a2f97b01f8903c0f9207b0d452/job-server/src/test/scala/spark/jobserver/JavaJobSpec.scala">More detail in the source code</a></p>
<h2>Increase the jar size</h2>
<pre><code class="language-yaml"><span class="hl slc"># in spark section</span>
short<span class="hl opt">-</span>timeout <span class="hl opt">=</span> <span class="hl num">60</span> s
<span class="hl slc"># in root section</span>
spray.can.server <span class="hl opt">{</span>
parsing.max<span class="hl opt">-</span>content<span class="hl opt">-</span>length <span class="hl opt">=</span> 300m
idle<span class="hl opt">-</span>timeout <span class="hl opt">=</span> 400s
request<span class="hl opt">-</span>timeout <span class="hl opt">=</span> 300s
<span class="hl opt">}</span>
</code></pre>
<h2>Start the server</h2>
<pre><code class="language-bash">sbt
<span class="hl slc"># Run this</span>
job<span class="hl kwb">-server-extras</span><span class="hl opt">/</span>reStart config<span class="hl opt">/</span><span class="hl kwb">local</span>.conf <span class="hl kwb">--- -Xmx8g</span>
</code></pre>
<h2>Client</h2>
<pre><code class="language-bash"><span class="hl slc"># compile the jar</span>
sbt job<span class="hl kwb">-server-tests</span><span class="hl opt">/</span>package
<span class="hl slc"># add the jar and give it the "test-binary" name</span>
curl <span class="hl kwb">--basic --user</span> <span class="hl sng">'user:pwd'</span> X POST localhost<span class="hl opt">:</span><span class="hl num">8090</span><span class="hl opt">/</span>binaries<span class="hl opt">/</span><span class="hl kwb">test-binary -H</span> <span class="hl sng">"Content-Type: application/java-archive"</span> <span class="hl kwb">--data-binary</span> @job<span class="hl kwb">-server-tests</span><span class="hl opt">/</span>target<span class="hl opt">/</span>scala<span class="hl kwb">-2</span>.11<span class="hl opt">/</span>job<span class="hl kwb">-server-tests_2</span>.11<span class="hl kwb">-0</span>.9<span class="hl num">.1</span><span class="hl kwb">-SNAPSHOT</span>.jar
<span class="hl slc"># pass the basic auth</span>
curl localhost<span class="hl opt">:</span><span class="hl num">8090</span><span class="hl opt">/</span>contexts <span class="hl kwb">--basic --user</span> <span class="hl sng">'user:pwd'</span> <span class="hl kwb">-d</span> <span class="hl sng">""</span> <span class="hl sng">"localhost:8090/contexts/test-context?num-cpu-cores=4&memory-per-node=512m"</span>
<span class="hl slc"># get the contexts</span>
curl <span class="hl kwb">-k --basic --user</span> <span class="hl sng">'user:pw'</span> https<span class="hl opt">://</span>localhost<span class="hl opt">:</span><span class="hl num">8090</span><span class="hl opt">/</span>contexts
<span class="hl slc"># get the jobs</span>
curl <span class="hl kwb">-k --basic --user</span> <span class="hl sng">'user:pw'</span> https<span class="hl opt">://</span>localhost<span class="hl opt">:</span><span class="hl num">8090</span><span class="hl opt">/</span><span class="hl kwb">jobs</span>
<span class="hl slc"># run a job based on the test-binary file</span>
<span class="hl slc"># also make it syncronized and increase the timeout</span>
curl <span class="hl kwb">--basic --user</span> <span class="hl sng">'user:pwd'</span> <span class="hl kwb">-d</span> <span class="hl sng">"input.string = a b c a b see"</span> <span class="hl sng">"localhost:8090/jobs?appName=test-binary&classPath=spark.jobserver.WordCountExample&context=test-context&sync=true&timeout=100000"</span>
</code></pre>
<h2>Async Client</h2>
<pre><code class="language-json">curl <span class="hl opt">--</span>basic <span class="hl opt">--</span>user user<span class="hl opt">:</span>pw' <span class="hl opt">-</span>d <span class="hl sng">"input.string = a b c a b see"</span> <span class="hl sng">"localhost:8090/jobs?appName=omop-spark-job&classPath=io.frama.parisni.spark.omop.job.WordCountExample&context=test-context&sync=false&timeout=100000"</span>
<span class="hl kwa">{</span>
<span class="hl kwc">"duration"</span><span class="hl opt">:</span> <span class="hl sng">"Job not done yet"</span><span class="hl opt">,</span>
<span class="hl kwc">"classPath"</span><span class="hl opt">:</span> <span class="hl sng">"io.frama.parisni.spark.omop.job.WordCountExample"</span><span class="hl opt">,</span>
<span class="hl kwc">"startTime"</span><span class="hl opt">:</span> <span class="hl sng">"2020-06-12T16:43:46.980+02:00"</span><span class="hl opt">,</span>
<span class="hl kwc">"context"</span><span class="hl opt">:</span> <span class="hl sng">"test-context"</span><span class="hl opt">,</span>
<span class="hl kwc">"status"</span><span class="hl opt">:</span> <span class="hl sng">"STARTED"</span><span class="hl opt">,</span>
<span class="hl kwc">"jobId"</span><span class="hl opt">:</span> <span class="hl sng">"d492de45-dc1e-4d08-996c-19b47fdb986f"</span><span class="hl opt">,</span>
<span class="hl kwc">"contextId"</span><span class="hl opt">:</span> <span class="hl sng">"47607883-d90c-456e-8884-d160f4c41480"</span>
<span class="hl kwa">}</span>
curl <span class="hl opt">-</span>X GET <span class="hl opt">--</span>basic <span class="hl opt">--</span>user 'user<span class="hl opt">:</span>pw' <span class="hl sng">"localhost:8090/jobs/d492de45-dc1e-4d08-996c-19b47fdb986f"</span>
<span class="hl kwa">{</span>
<span class="hl kwc">"duration"</span><span class="hl opt">:</span> <span class="hl sng">"0.157 secs"</span><span class="hl opt">,</span>
<span class="hl kwc">"classPath"</span><span class="hl opt">:</span> <span class="hl sng">"io.frama.parisni.spark.omop.job.WordCountExample"</span><span class="hl opt">,</span>
<span class="hl kwc">"startTime"</span><span class="hl opt">:</span> <span class="hl sng">"2020-06-12T16:43:46.980+02:00"</span><span class="hl opt">,</span>
<span class="hl kwc">"context"</span><span class="hl opt">:</span> <span class="hl sng">"test-context"</span><span class="hl opt">,</span>
<span class="hl kwc">"result"</span><span class="hl opt">:</span> <span class="hl kwa">{</span>
<span class="hl kwc">"b"</span><span class="hl opt">:</span> <span class="hl num">2</span><span class="hl opt">,</span>
<span class="hl kwc">"a"</span><span class="hl opt">:</span> <span class="hl num">2</span><span class="hl opt">,</span>
<span class="hl kwc">"see"</span><span class="hl opt">:</span> <span class="hl num">1</span><span class="hl opt">,</span>
<span class="hl kwc">"c"</span><span class="hl opt">:</span> <span class="hl num">1</span>
<span class="hl kwa">}</span><span class="hl opt">,</span>
<span class="hl kwc">"status"</span><span class="hl opt">:</span> <span class="hl sng">"FINISHED"</span><span class="hl opt">,</span>
<span class="hl kwc">"jobId"</span><span class="hl opt">:</span> <span class="hl sng">"d492de45-dc1e-4d08-996c-19b47fdb986f"</span><span class="hl opt">,</span>
<span class="hl kwc">"contextId"</span><span class="hl opt">:</span> <span class="hl sng">"47607883-d90c-456e-8884-d160f4c41480"</span>
<span class="hl kwa">}</span>
</code></pre>
<h2>Enabling sparkContext with hive support</h2>
<p>By extending the <code>SparkSessionJob</code> class and using the below code, you get the hive support:</p>
<pre><code class="language-bash">context<span class="hl kwb">-create</span><span class="hl opt">:</span>
curl <span class="hl kwb">-X</span> POST <span class="hl kwb">-k --basic --user</span> <span class="hl sng">'</span><span class="hl ipl">${USER}</span><span class="hl sng">:</span><span class="hl ipl">${PWD}</span><span class="hl sng">'</span> <span class="hl sng">"</span><span class="hl ipl">${HOST}</span><span class="hl sng">:</span><span class="hl ipl">${PORT}</span><span class="hl sng">/contexts/test-context?num-cpu-cores=4&memory-per-node=512m&context-factory=spark.jobserver.context.SessionContextFactory"</span>
</code></pre>
<p>Complex result (use array/maps as object returned):</p>
<pre><code class="language-json"><span class="hl kwa">{</span>
<span class="hl kwc">"jobId"</span><span class="hl opt">:</span> <span class="hl sng">"4e85111f-98aa-4736-8153-11f937d4fd2f"</span><span class="hl opt">,</span>
<span class="hl kwc">"result"</span><span class="hl opt">:</span> <span class="hl kwb">[</span><span class="hl kwa">{</span>
<span class="hl kwc">"list"</span><span class="hl opt">:</span> <span class="hl num">1</span><span class="hl opt">,</span>
<span class="hl kwc">"count"</span><span class="hl opt">:</span> <span class="hl sng">"800"</span><span class="hl opt">,</span>
<span class="hl kwc">"resourceType"</span><span class="hl opt">:</span> <span class="hl sng">"Patient"</span>
<span class="hl kwa">}</span><span class="hl opt">,</span> <span class="hl kwa">{</span>
<span class="hl kwc">"list"</span><span class="hl opt">:</span> <span class="hl num">2</span><span class="hl opt">,</span>
<span class="hl kwc">"count"</span><span class="hl opt">:</span> <span class="hl sng">"100"</span><span class="hl opt">,</span>
<span class="hl kwc">"resourceType"</span><span class="hl opt">:</span> <span class="hl sng">"Condition"</span>
<span class="hl kwa">}</span><span class="hl kwb">]</span>
<span class="hl kwa">}</span>
</code></pre>
<h2>Passing config parameters</h2>
<p>One way is to store them into the <code>env.conf</code> file, in the <code>passthough</code>
section. They are then accessible from the
<code>runtime.contextConfig</code>variable within the <code>runJob</code> method.</p>
<p>They can be accessed from code thought <code>val ZK = runtime.contextConfig.getString("passthrough.the.variable")</code></p>
<h2>Passing extra options</h2>
<p>This can be done with the <code>env.sh</code> file. In this example, we configure
the basic auth to solr cloud.</p>
<pre><code class="language-bash">MANAGER_EXTRA_SPARK_CONFS<span class="hl opt">=</span><span class="hl sng">"spark.executor.extraJavaOptions=-Dsolr.httpclient.config=<path-to>/auth_qua.txt -Dhdp.version=2.6.5.0-292|spark.driver.extraJavaOptions=-Dhdp.version=2.6.5.0-292,-Dsolr.httpclient.config=<path-to>/auth_qua.txt,spark.yarn.submit.waitAppCompletion=false|spark.files=</span><span class="hl ipl">$appdir</span><span class="hl sng">/log4jcluster.properties,</span><span class="hl ipl">$conffile</span><span class="hl sng">"</span>
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=First contact with spark-jobserver — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-06-20 19:02</time> </div>https://blog.parisni.com/blog/book-sectionIntroducing the "Readings" section2020-08-04T14:43:00+00:00<div id="generated-toc"> </div><hr> <div id="refman">
<div id="refman-sidebar">
<div id="generated-toc"> </div>
</div>
<div id="refman-main">
<p><span>Published on <time id="post-date">2020-03-08</time></span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Introducing the "Readings" section</h1>
<h3>Principle</h3>
<p>Books are meant to be <strong>read</strong>. Also books should be <strong>reviewed and
criticized</strong>. I am introducing a new book section with several goals
in mind:</p>
<ul>
<li>read more books</li>
<li>read carefuly</li>
<li>keep notes, and summary</li>
</ul>
<h3>Content</h3>
<p>Books, comics, novel, essay, philosophy and also research papers will be covered.
In term of content, I can imagine:</p>
<ul>
<li>reference</li>
<li>citation</li>
<li>synthesis</li>
<li>form analysis</li>
<li>and also cover picture</li>
</ul>
<p>I wonder I it's legal to share citations, and pictures of comics. This
is something I will check out.</p>
<h3>Form</h3>
<p>I guess I could distinguish several categories, and provide <strong>views per
categories</strong>.</p>
<h4>Entry preview</h4>
<p>Each entry preview would have:</p>
<ul>
<li>title</li>
<li>summary</li>
<li>thumbnail</li>
<li>stars</li>
</ul>
<h4>Entry content</h4>
<p>Depending on the category, each book entry might have:</p>
<ul>
<li>summary</li>
<li>screenshots</li>
<li>citations</li>
</ul>
<h3>Next Steps</h3>
<p>I will need to explore existing things on the web. Still the content
and reading can start with this goal. I can think of:</p>
<ul>
<li><a href="https://qblog.aaronsw.com/page/7">raw meat by aaronsw</a></li>
<li><a href="https://pinboard.in/u:aaronsw/">pinboard</a></li>
<li><a href="http://www.aaronsw.com/weblog/">raw thought</a></li>
<li><a href="https://codepen.io/pun-isher/pen/qjZaXN">timeline</a></li>
</ul>
</div></div><div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Introducing the " readings"="" section="" —="" software="" lobotomy"="">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2020-08-04 14:43</time> </div>https://blog.parisni.com/reading/midnight-expressMidnight Express2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-03-09</time></span>
<span id="book-year" style="visibility:hidden;">1977</span>
<span id="book-author" style="visibility:hidden;">William Hayes</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★</span>
<span id="stars" style="visibility:hidden;">★</span></span></p>
<h1>Midnight Express</h1>
<p id="post-excerpt">
The famous movie Midnight Express was originally a book inspired by the real life of his author.
</p><p>One of the significant moment happens in the psychiatric
hospital. After advancing further into a kind of cave, the narrator is
witness of a scene worthy H.P. Lovecraft, or Ernesto Sabato - "Héros
et Tombes". A procession of insane prisonners are walking slowly all
day long around a large pillar of stone. All of them are turning in
the same flow direction, but the narrator which does the inverse.</p>
<p></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Midnight Express — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/huckleberry-finnAdventure of Huckleberry Finn2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-03-09</time></span>
<span id="book-year" style="visibility:hidden;">1884</span>
<span id="book-author" style="visibility:hidden;">Mark Twain</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Adventure of Huckleberry Finn</h1>
<pre><code><p id="post-excerpt">Adventures of Huckleberry Finn (or, in more recent editions, The Adventures of Huckleberry Finn) is a novel by Mark Twain, first published in the United Kingdom in December 1884 and in the United States in February 1885.</p>
</code></pre>
<h2>Summary</h2>
<p>Commonly named among the Great American Novels, the work is among the
first in major American literature to be written throughout in
vernacular English, characterized by local color regionalism. It is
told in the first person by Huckleberry "Huck" Finn, the narrator of
two other Twain novels (Tom Sawyer Abroad and Tom Sawyer, Detective)
and a friend of Tom Sawyer. It is a direct sequel to The Adventures of
Tom Sawyer.</p>
<p>The book is noted for its colorful description of people and places
along the Mississippi River. Set in a Southern antebellum society that
had ceased to exist over 20 years before the work was published,
Adventures of Huckleberry Finn is an often scathing satire on
entrenched attitudes, particularly racism.</p>
<h3>Cover</h3>
<p>second level</p>
<h3>Derived productions</h3>
<p>second level</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Adventure of Huckleberry Finn — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/speerAu coeur du troisième reich2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-04-16</time></span>
<span id="book-year" style="visibility:hidden;">1970</span>
<span id="book-author" style="visibility:hidden;">Albert Speer</span>
<span id="reading-time" style="visibility:hidden;">3 minutes
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Au coeur du troisième reich</h1>
<blockquote>
<p>Il m´est arrivé exactement la même chose qu´aux grands du parti: eux
gâchaient leur vie de famille par une vie d´apparat, rigidifiée dans
les attitudes de l´etiquette officielle; moi, au contraire, en
devenant l´esclave de mon travail.</p>
</blockquote>
<h2>Architecte du Reich</h2>
<p>Albert Speer centre son récit sur Hitler et en fait le détail
minutieux. On y découvre un homme profondément calculateur et
manipulateur - un arnaqueur en somme. Perfide, procrastinateur et
médiocre à bien des égards, c´est à travers leur passion commune pour
l´architecture - motivée par sa mégalomanie - que les deux hommes se
rejoignent. Génie de l´organisation, Speer est sous le charme de
l´exceptionnel tribun qui exploite ses compétences et son ambition
naive.</p>
<p>À travers ce récit, on découvre l´envers du décors des grandes étapes
de la descente aux enfers de l´Allemagne, réarmement, annexion de
l´Autriche, pacte germano-soviétique, invasion de la Pologne, drole de
guerre... Sur chacun de ces tournants, Hitler semble en pleine
possession de la situation quoi que cela alimente sa mégalomanie qui
lui coutera défaites après defaite son ambition d´un empire
millérnaire qu´il revait vraisemblablement depuis des décennies.</p>
<h2>Ministre de l´armement</h2>
<p>Dans la seconde partie de l´ouvrage, Speer décrit son rôle de ministre
de l´armement et l´approche mise en oeuvre pour industrialier et
donner un second souffle au reich. Les passages suivants issus du
chapitre "improvisation organisee" reflètent les grandes lignes de
l´approche:</p>
<blockquote>
<p>[Walter Rathenau] avait découvert que si les entreprises
échangeaient leurs connaissances techniques, si on opérait une
division du travail entre les usines, et si on normalisait et
standardiait la fabrication, la production pouvait faire un bond
spectaculaire.</p>
</blockquote>
<blockquote>
<p>J´étais consterné à l´idée que les méthodes bureaucratiques puissent
s´íntroduire dans ma propre création. Je ne cessais d´exhorter mes
collaborateurs à ne pas établir de documents, à régler les questions
verbalement ou par téléphone, directement et sans formalités, bref à
éviter à tout prix de constituer des ¨dossiers¨ pour reprendre le
jargon administratif.</p>
</blockquote>
<blockquote>
<p>Les méthodes que j´appliquais, et qui étaient celles d'une gestion
économique démocratique, fure également un facteur décisif. Le
principe consistait à faire confiance jusqu´au bout aux industriels
reponsables. Cette méthode récompensait leur esprit d´initiative,
éveillait en eux le sens des responsabilités, stimulait leur esprit
de décision [...]</p>
</blockquote>
<blockquote>
<p>Au fond, j´exploitais cette attitude, fréquente chez les
techniciens, qui consiste à se consacrer à son travail sans se poser
de questions. Le rôle du technicien étant apparement dégagé de tout
aspect moral, il n´y eut pendant longtemps de leur part aucune
réflexion sur la valeur de leur propre activité.[...] le technicien
n´était plus en mesure d´apercevoir les conséquences de son activité
anonyme.</p>
</blockquote>
<h2>Commandant des armées</h2>
<p>Speer met en parallèle la progressive implication d´Hitler dans les
affaires des généraux et l´accumulation des défaites. Si les premières
victoires éclair ont été aquises avec une approche non conventionnelle
et audacieuse, cette chance du débutant ne se reproduit pas. Hitler,
qui maitrise tous les détails des chars, avions et canons n´a pas la
vision d´ensemble et la connaissance du terrain en particulier dans
les airs et par grand froid. Sa propention à s´entourer de non
spécialistes qui le confortent indéfectiblement dans ses choix
entraineront les catastrophes de Stalingrad où des centaines de
miliers de morts auraient pû être évités.</p>
<p>C´est ce qui motivera la tentative d´assacinat d´Hitler par des
généraux.</p>
<h2>Rapport avec les cadres du Reich</h2>
<p>Borman est après Hitler le sujet principal du récit de Speer. Dans
ses maneuvres continuelles pour manipuler le Fuhrer, Borman n´a de
cesse de saper la confiance de ce dernier envers ses ennemis. En
tenant le rôle de secrétaire du parti ce la lui permet de filtrer et
d´avoir connaissance de tous les dossier.</p>
<p>Gõring prend une grande place dans le récit. Si Speer reconnait qu´il
a eu par le passé une grande influence, il le présente à de multiples
reprises comme désinvesti, décridibilisé et en perdition. Par ses
actions en tant que commandant de l´armée de l´air, puis son attitude
de dénégation de la puissance russe puis allíée sur laqueĺle Hitler
s´est appuyé dans son entêtement, il a eu un rôle prépondérant dans la
chute de l´Allemagne.</p>
<p>Goebels est présenté parmi les cadre les plus lucides sur le sort de
l´Allemagne. Il fait aussi parti des rares cadre diplomés car Hitler
avait une propention à nommer des non-initiés à des postes de haute
responsabilité; par exemple Ribbentrop était marchand de Vin de
formation (ministre des affaires étrangères) tandis que Speer était
architecte (ministre de l´armement).</p>
<p>Himmler est peu cité. Speer a tenté de le ranger à ses côtés mais on
comprends que ce dernier en retour à tenté de l´assaciner en le
confiant à son médecin et ami: Karl Franz Gebhard. Himmler a fait
partie du complot destiné à se débarasser de Speer, avec Gõring et
Borman. Cette tentative de meurtre s´inscrit dans le contexte de la
mise en oeuvre des V2 au debut de 1944 ou Speer tentait d´ameliorer
les conditions de vies des prisoniers de camps. Himmler ayant depuis
1942 des vues sur le ministère de l´armement, on peut comprendre
l´intêret qu´il avait dans la disparition de Speer.</p>
<h2>La débacle</h2>
<p>Très tôt Speer prend conscience de la défaite du Reich, c´est à dire à
l´hiver de la campagne de Russie, en 42. Au risque d´être écarté du
pouvoir par Hitler, il ne se cachera jamais de ce constat et n´aura de
cesse de tenter de faire passer le message. Dans les derniers mois,
Speer va combattre la politique de la terre brulée imposée par
Hitler. Pour cette raison, ce dernier le rayera de son testament
politique en faveur de Saur.</p>
<p>On obtient des détails psychologiques précis sur la fin de Hitler et
de Eva Braun. Le premier est anéanti, désincarné, tandis que la
seconde se montre guillerette à l´approche de cette échéance. Il est
très clair qu´elle a choisit ce suicide.</p>
<h2>Sur le banc des accusés</h2>
<p>Speer sera en contact étroit avec les Généraux Américains avec qui ils
partagent en particulier sur l´efficacité des méthodes de
bombardements.</p>
<p>Rescapé du tribunal de Nuremberg, Speer niera toute sa vie, et dans ce
livre avoir été conscient du sort des Juifs dans les camps. Hors on
sait depuis qu´il a validé et signé les plans d´Auschwitz. Il ne
pouvait donc pas ignorer l´inconcevable. Le mot Juif apparait à une
seule reprise dans le texte de 800 pages.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Au coeur du troisième reich — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/alice-in-wonderlandAlice In Wonderland2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-05-24</time></span>
<span id="book-year" style="visibility:hidden;">1865</span>
<span id="book-author" style="visibility:hidden;">Lewis Carroll</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Alice In Wonderland</h1>
<p>This has been written in 1865. It is related to the absurd litterature
movement (kafka, buzzati, becket, camus...) born during the second
world war. It is also related to little Nemo in slumber land written
in 1905, inspired by sleep. Winsor Mc Cay was abusing from drugs, and
Alice In Wonderland might be related to Edgar Allan Poe and other
fantastic tales.</p>
<p>I particularly enjoyed the absurd dialogues where logic, physic and
mathematic are negated.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Alice In Wonderland — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/sous-letoile-dautomneUnder Høststjærnen2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-05-29</time></span>
<span id="book-year" style="visibility:hidden;">1906</span>
<span id="book-author" style="visibility:hidden;">Knut Hamsun</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Under Høststjærnen</h1>
<p>"Sous l'étoile d´automne" raconte les aventures d´un jeune norvégien
en quète de repos intellectuel. Tour à tour flaneur, bucheron,
puisatier il passe de ferme en ferme, de compagnons en compagnes. Le
narrateur tente de cacher sa réussite sociale mais se montre inventif,
curieux et audacieux.</p>
<p>Ses aventures s´apparentent aux tribulations de Casanova ou de Barry
Lyndon mais avec le soin de dépeindre une nature vivifiante.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Under Høststjærnen — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/a-descent-into-the-maelstromA Descent into the Maelström2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-05-31</time></span>
<span id="book-year" style="visibility:hidden;">1841</span>
<span id="book-author" style="visibility:hidden;">Edgar Allan Poe</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>A Descent into the Maelström</h1>
<p>Amoung short-stories of "Tales of the Grotesque and Arabesque" this
one and "MS. found into a bottle" both depict marvellous nature of
oceans.</p>
<p>Three brothers are caught in a violent storm which train them into a
whirlpool called Maelström. Time is arrested, elements get out of
control, and only the smartest brother save his life.</p>
<p>i felt a parallel with <a href="https://en.wikipedia.org/wiki/The_Three_Little_Pigs">"the tree little pigs"</a>, where the storm would
be the wolf and where each brother adopt a different
strategy. Interstingly in 1840, the first printed versions of the
fable existed in the America.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=A Descent into the Maelström — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/voyage-au-bout-de-la-nuitVoyage au bout de la nuit2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-06-07</time></span>
<span id="book-year" style="visibility:hidden;">1932</span>
<span id="book-author" style="visibility:hidden;">Louis-Ferdinant Céline</span>
<span id="reading-time" style="visibility:hidden;">4 minutes
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Voyage au bout de la nuit</h1>
<h2>Inapte à la guerre</h2>
<p>Céline décrit à travers son narrateur, Ferdinand Bardamu, la guerre
14-18 dans différentes dimensions. Il décrit davantage l´adversité
avec la hiérarchie militaire, et "à l´arrière" dans les hôpitaux
psychiatriques militaires ou au contact des civils. La société tout
entière met la pression sur les soldats pour aller au casse pipe. En
ne parlant pratiquement pas de la guerre et de l'adversaire à
proprement parlé cela en fait un pamphlet contre la guerre.</p>
<h2>Le Congo</h2>
<p>Une fois en encore, les compatriotes de Ferdinand en font un bouc
émissaire. Sans raison apparente, c´est toute le pavillon qui
l´embarque pour L'Afrique qui se lie contre lui.</p>
<p>Il s´engage immédiatement pour "tenir une factorie dans la brousse",
métier qu´il exercera bien peu et qui consiste a faire du commerce
avec les autochtones. Il déclare bien vite le paludisme et sera vendu
par ces derniers à un armateur.</p>
<h2>Les États-Unis</h2>
<h3>L´arrivée</h3>
<blockquote>
<p>Figurez vous qu´elle était debout leur ville, absolument droite. New York c´est une ville debout. On en avait déjà vu nous des villes bien sûr, et des belles encore, et des ports et des fameux même. Mais chez nous, n´est-ce pas, elles sont couchées les villes, au bord de la mer ou sur les fleuves, elles s´allongent sur le paysage, ells attendent le voyageur, tandis que celle-là l´Américaine, elle ne se pâmait pas, non, elle se tenait bien raide, là, pas baisante du tout, raide à faire peur.</p>
</blockquote>
<p>C´est en tant que galérien et par surprise que Ferdinand approche les
côtes de New York.</p>
<h3>Les usines ford</h3>
<p>Pour gagner sa croûte, F. se rend à Detroit dans les fameuses usine
Ford où "ils prennent tout le monde".</p>
<blockquote>
<p>Une fois rhabillés, nous fûmes répartis en files traînardes, par groupes hésitants en renfort vers ces endroits d´où nous arrivaient les fracas énormes de la mécanique. Tout tremblait dans l´immense édifice et soi-même des pieds aux oreilles possédés par le tremblement, il en venait des vitres et du plancher et de la ferraille, des secousses, vibré de haut en bas. On en devenait machine aussi soi-même à force et de toute sa viande encore tremblotante dans ce bruit de rage énorme qui vous prenait le dedans et le tour de la tête et plus bas vous agitant les tripes et remontait aux yeux par petits coups précipites, infinis, inlassables.</p>
</blockquote>
<h3>Le départ</h3>
<p>F. rencontre une prostituée qu´il séduit et avec laquelle il passe des
moments inoubliables. Cependant, il rencontre à nouveau Robinson, et
c´est à nouveau le moment pour lui de quitter les États-Unis pour
revenir en France.</p>
<blockquote>
<p>Bonne, admirable Molly, je veux si elle peut encore me lire, d´un endroit que je ne connais pas, qu´elle sache bien que je n´ai pas changé pour elle, que je l´aime encore et toujours, à m manière, qu´elle peut venir ici quand elle voudra partager mon pain et ma furtive destinée. Si elle n´est plus belle, eh bien tant pis ! Nous nous arrangerons ! J´ai gardé tant de beauté d´elle en moi, si vivace, si chaude que j´en ai bien pour tous les deux et pour au moins vingt ans encore, le temps d´en finir.</p>
</blockquote>
<h2>Retour en France</h2>
<h3>La médecine</h3>
<p>À son retour à Paris, F. reprend ses études et se fait médecin de
quartier. Il a toutes les peines du monde à se faire payer.</p>
<p>Il présente la sexualité comme un vecteur de violence qui frappe les
miséreux. La prostitution, les avortements, la masturbation: le
médecin de ville est le témoin et le confident de ces scènes. On voit
comment les familles tentent de l´instrumentaliser pour garder une
place dans la société.</p>
<blockquote>
<p>Quand ils l´avaient tellement battue qu´elle ne pouvait plus hurler, leur fille, elle criait encore un peu quand même à chaque fois qu´elle respirait, d´un petit coup. J´entendais l´homme alors qui disait a ce moment-lá: "Viens toi grande ! Vite ! Vient par lá !"Tout heureux. C´était a la mère qu´il parlait comme ça, et puis la porte d´à côté claquait derrière eux. Un jour, c´est elle qui lui a dit, je l´ai entendu : "Ah ! je t'aime Julien, tellement, que je te boufferais ta merde, même si tu faisais des étrons grands comme ça..."</p>
</blockquote>
<p>À nouveau, la rencontre fortuite avec Robinson, sonne la fin de la
récréation. Céline démontre comment la misère entraîne et contraint
les individus dans des péripéties. Eux ne font qu´accepter et tolérer
les méfaits que la vie met à leur disposition.</p>
<blockquote>
<p>J'avais l'habitude et même le goût de ces méticuleuses observations
intimes. Quand on s´arrête à la façon par exemple dont sont formés
et proférés les mots, elles ne résistent guère nos phrases au
désastre de leur décor baveux. C´est plus compliqué et plus pénible
que la défécation notre effort mécanique de la conversation. Cette
corolle de chair bouffie, la bouche, qui se convulse á siffler,
aspire et se démène, pousse toutes espèces de sons visqueux à
travers le barrage puant de la carie dentaire, quelle punition !
voilà pourtant ce qu´on nous adjure de transposer en idéal. C´est
difficile. Puisque nous sommes que des enclos de tripes tièdes et
mal pourries nous aurons toujours du mal avec le sentiment. Amoureux
ce n´est rien c´est tenir ensemble qui est difficile. L´ordure elle,
ne cherche ni à durer, ni à croire. Ici, sur ce point nous sommes
bien plus malheureux que la merde, cet enragement à persévérer dans
notre état constitue l´incroyable torture.</p>
</blockquote>
<h3>Toulouse</h3>
<p>Toujours guidé par Robinson, entraîné dans la nuit, F. débarque à
Toulouse comme en vacances. Robinson et la vielle qu´il a tenté
d´assassiner sont en affaire ensemble pour faire la visite d´un caveau
pour le compte de l´Église. Ce dernier est en outre en passe de se
marier et malgré des affaire florissantes, Robinson n´a de cesse de se
plaindre. Si F. n´a jamais fait état de son amitié pour Robinson, on
comprend à travers la déception qu´il décrit avec précision que
c´était le cas. Cette amitié se basait sur une forme de partage de la
nuit, partage du voyage et des aventures qu´il se rend compte ne plus
ou ne finalement jamais avoir vraiment partagé avec son compagnon.</p>
<blockquote>
<p>Ils en ont des pitié les gens, pour les invalides et les aveugles et
on peut dire qu´ils en ont de l´amour en réserve. Y en a
énormément. On peut pas dire le contraire. Seulement c´est
malheureux qu´ils demeurent si vaches avec tant d´amour en réserve
les gens. Ça ne sort pas, voilà tout. C´est pris en dedans, ça reste
en dedans, ça leur sert à rien. Ils en crèvent en dedans, d´amour.</p>
</blockquote>
<h3>La psychiatrie</h3>
<p>Comme toujours, F. est rattrapé par ses vieilles connaissances rentré
à Paris. Un confrêre lui trouve une place dans un asile psychiatrique
en banlieue.</p>
<blockquote>
<p>Je me tenais au bord dangereux des fous, à leur lisière pour ainsi
dire, à force d´être toujours aimable avec eux, ma nature.</p>
</blockquote>
<p>Il va en particulier apprendre l´anglais au directeur de
l´établissement et cela va transformer son destin. À la manière de Don
Quichotte qui à force de lecture d´histoire de chevalerie va parcourir
le monde, le directeur va tout plaquer pour voyager en Angleterre et y
disparaître. F. se voit alors en charge de l´asile, ce qui ne va pas
tarder à attirer les convoitises de ses compères de Toulouse.</p>
<blockquote>
<p>Dans le cas où nous étions, un homme, un costaud, m´aurait fait
peur, mais d´elle j´avais rien à craindre. Elle était moins forte
que moi, comme on dit. Depuis toujours l´envie me tenait de claquer
une tête ainsi possédée par la colère pour voir comment qu´elles
tournent les têtes en colère dans ces cas-là. Ça ou un beau chèque,
c´est ce qu´il faut pour voir d´un seul coup virer d´un bond toutes
les passions qui sont à louvoyer dans une tête. C´est beau comme une
belle manoeuvre à la voile sur une mer agitée. Toute la personne
s´incline dans une vent nouveau. Je voulais voir ça.</p>
</blockquote>
<h3>La fête foraine</h3>
<p>L´amour maladif et la dépression font mauvais ménage. Et ce n´est pas
une fête artificielle telle que les manèges qui peuvent rattraper ces
maladies.</p>
<p>Dans ce dernier épisode, Céline approfondie les sentiments des
personnages tous plus inadaptés à la vie en société. Cela fait l´effet
d´un vieux mur fait avec des pierres biscornues mal choisies qui se
disloque.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Voyage au bout de la nuit — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/les-bourreaux-de-staline-katyn-1940Les bourreaux de Staline: Katyn 19402021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-06-08</time></span>
<span id="book-year" style="visibility:hidden;">2020</span>
<span id="book-author" style="visibility:hidden;">Cédric Tourbe</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Les bourreaux de Staline: Katyn 1940</h1>
<blockquote>
<p>Il ne tire pas. Il fait son travail.</p>
</blockquote>
<p>Le documentaire revient sur un évènement qui a traversé la seconde
partie du 20ième siècle. L´avenement de Staline s´est construit sur
des pratiques de purges et de massacres depuis les années 1920. Par
exemple, on estime les grandes purges ont supprimé près de 30% de la
population. Ou alors en Ukraine, la famine forcée fera 4 milions de
morts, lors de la <a href="https://fr.wikipedia.org/wiki/Famines_sovi%C3%A9tiques_de_1931-1933">collectivisme de l´agrigulture</a> en Ukraine.</p>
<p>Quand 20 ans plus tard, les soviétiques annexent la pologne
conformément aux accords secrets du traité de non-aggression
germano-soviétique, c´est tout naturellement qu´ils vont exécuter à
bout portant 20.000 officiers polonais dans le plus grand secret, puis
en accusant l´Allemagne Nazie.</p>
<p>Pourtant, Churchill aura tous les éléments 3 ans plus tard faisant
accuser les Bolcheviques. Mais plutot que de froisser Staline à un
moment crucial de la guerre, il taira l´information.</p>
<h1>Un régime fondé sur le crime</h1>
<p>Ce documentaire ancre le récit dans l´histoire et ce relief contraste
la terreur mise en place à tous les échellons de parti
bolchévique. Celui qui tuera par le parti, périra par le parti.</p>
<div>
<iframe width="100%" height="415" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/9393c109-df7b-4b78-b6e5-8f88f4f7b11d" frameborder="0" allowfullscreen=""></iframe>
</div>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Les bourreaux de Staline: Katyn 1940 — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/dick_au-service-du-maitreAu service du Maître2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-06-20</time></span>
<span id="book-year" style="visibility:hidden;">1955</span>
<span id="book-author" style="visibility:hidden;">Philip K. Dick</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★</span>
<span id="stars" style="visibility:hidden;">★</span></span></p>
<h1>Au service du Maître</h1>
<h2>Immunity</h2>
<p>Dick dépeint une société sous l´emprise des télépathes (TP) qui
profitent de leur aptitude à lire dans les pensées pour asservir la
société.</p>
<p>Les bons citoyens n´ont rien à se reprocher et cet état de fait n´est
donc pas un problème. Les discidents sont quant à eux éliminés, et il
est facile pour les TP de remonter les réseaux de résistances avec les
informations lues dans les pensées.</p>
<p>Mais cette faculté va se retourner contre eux, car si aucune pensée ne
peut échapper aux télépathes, toute vérité n´est pas bonne à savoir.</p>
<h2>The chromium Fence</h2>
<p>Dick dépeint une société divisée politiquement entre deux mouvements
antagonistes: les puristes et les naturalistes. C´est sur l´hygiène
que ces clans s´opposent, les premiers prônant une humanité aseptisée
tandis que les seconds défendant une animalité assumée.</p>
<p>Le narrateur quant à lui n´entend pas choisir de camp et à ce titre
est persécuté aussi bien par l´un que par l´autre. Jusqu´à la mort il
défendra son libre arbitre et cette radicalité est décrite par ses
bourreau comme quelque chose d´effrayant</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Au service du Maître — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/ness-of-brodgardLe Ness de Brodgard2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-06-21</time></span>
<span id="book-year" style="visibility:hidden;">2020</span>
<span id="book-author" style="visibility:hidden;">Peter Eeckhout</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Le Ness de Brodgard</h1>
<p><a href="https://en.wikipedia.org/wiki/Ness_of_Brodgar">wikipedia</a></p>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/64b03a57-9183-4b94-a92d-29c2a758ea0c" frameborder="0" allowfullscreen=""></iframe>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Le Ness de Brodgard — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/trosky_ma-vieMa vie2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-07-04</time></span>
<span id="book-year" style="visibility:hidden;">1930</span>
<span id="book-author" style="visibility:hidden;">Léon Trotsky</span>
<span id="reading-time" style="visibility:hidden;">5 minutes
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Ma vie</h1>
<h2>Une enfance entre la campagne et la ville</h2>
<p>T. donne les détails de son enfance. On comprend qu´à l´époque
bénéficier d´une éducation est rare. Il faut que la famille ait les
moyens de les financer. L´appartenance ethnique de la famille a de
l´importance, puisque des quotas sont appliqués pour l´entré dans les
études - en particulier pour les juifs. Enfin, une selection sur les
capacités des élèves est opérée.</p>
<p>L´enfance de T. lui permet d´observer la société de la campagne et de
la ville. L´essentiel de cette enfance consiste à une élévation et une
ouverture intelectuelle, au contact de toujours davantage de maîtres
et mentors. Il lit et écrit très tôt et très fort, encouragé et
corrigé par les adultes mais aussi initié, à travers des veillées de
lectures, et la rédaction de vers.</p>
<p>Les émotions dont il fait état concernent exclusivement les rapports
dominants/dominés et très tôt il prend la défence des faibles et
s´oppose violemment aux puissants:</p>
<blockquote>
<p>[...] sympathie pour les opprimés, indignation contre les injustices</p>
</blockquote>
<h2>La politisation</h2>
<p>Vers 18 ans, T. commence son initiation à la politique. Les idées
socialistes finissent par le convaincre. C´est d´une part à travers la
lecture des ouvrages tels que <em>la logique</em> de <a href="https://fr.wikipedia.org/wiki/John_Stuart_Mill">John Stuart
Mill</a>, <em>Culture
primitive</em> de
<a href="https://en.wikipedia.org/wiki/Julius_Lippert_(historian)">Lippert</a>,
l´<em>Utilitarisme</em> de
<a href="https://fr.wikipedia.org/wiki/Jeremy_Bentham">Bentham</a>, l´<em>Histoire
de la révolution francaise</em> de
<a href="https://fr.wikipedia.org/wiki/Fran%C3%A7ois-Auguste_Mignet">Mignet</a>,
et des journaux et d´autre part par la fréquentation de membres
actifs et militants que T. va façonner ses idées. Plusieurs de ses
écrits de l´époque ne seront pas publiés, critiques littéraire, pièces
de théatre. À force de réunions et de remues-méninges et de
tractations auprès des ouvriers, les premiers résultats vont
arriver...en même temps que les premiers emprisonnements. Cette ardeur
au travail fut déclenchée par cette <em>métaphore du haricot</em> dans une
taverne:</p>
<blockquote>
<p>C´est très simple: je met un haricot sur la table, c´est le tsar; autour de lui, d´autres haricots: c´est les ministres, les généraux; ensuite, les nobles, les marchands; et ce tas de haricot, c´est le simple peuple. Et maintenant, je demande: ou est le tsar ? L´orateur montre le haricot du milieu: "Ou sont les ministres?" il montre ceux qui entourent le haricot du milieu.</p>
</blockquote>
<blockquote>
<p>C'est comme j'ai dit, reprend-il, et l'autre est d'accord. Mais attend...attends maintenant... il ferme tout à fait l´oeil gauche. Là, je mêle, de la main, tous les haricots ensembles... et bien, que je dis, ou est le tsar ? Ou sont les ministres ? comment s´y retrouver ? comment s´y retrouver... on ne les voit plus, c´est bien ça, que je dis, on ne les voit plus... il faut seulement mélanger tous les haricots.</p>
</blockquote>
<h2>La Révolution perpétuelle</h2>
<p>T. va construire ses idées révolutionnaires à travers la lecture,
l´écriture, les voyages dans toute l´Europe mais surtout la prison et
la déportation.</p>
<p>La première révolution à laquelle T. participe est celle de 1905. Elle
se solde par un échec et T. est condamné à nouveau à la déportation.
<a href="https://en.wikipedia.org/wiki/Soviet_(council)">soviet</a></p>
<p>T. vivra dans divers pays dont la Suède, l´Autriche, la Suisse et
l´Angleterre. Dans chacun d´eux, il mènera la révolution à travers la
rédaction d´articles dans des journaux aux tirages internationaux. Ces
journaux ont eu comme effet de mettre sur pied un réseau de
révolutionnaires gauchistes.</p>
<p>Outre la critique des écrivains et penseurs, T. brosse le portrait
implacable de nombreuses personnes qu´il a rencontrés. Parmi les
centaines de portrait, il s´attarde sur celui de <a href="https://en.wikipedia.org/wiki/Jean_Jaur%C3%A8s">Jean
Jaures</a> et d´<a href="https://en.wikipedia.org/wiki/August_Bebel">August
Bebel</a>, ses meilleurs
alliés français et allemands, qui vont disparaître avant la révolution
de 1917.</p>
<h2>L´exil aux USA</h2>
<p>Tout comme Bardamu dans <em>voyage au bout de la nuit</em>, T. va débarquer à New York !</p>
<blockquote>
<p>Un dimanche, le 13 janvier, nous arrivons devant New York. À trois heures du matin, réveil général. Nous sommes en place. Il fait sombre. Il fait froid. Du vent. De la pluie. Sur la berge, un humide amoncellement d´édifices. Le Nouveau Monde... Je me trouvais à New York., cité fabuleusement prosaïque, de l´automatisme capitaliste, où triomphent, dans les rues, la théorie esthétique du cubisme, et , dans les coeurs, la philosophie morale du dollar. New York m´en imposait parce qu´il exprime au mieux l´esprit moderne.</p>
</blockquote>
<p>En effet, après avoir été expulsé de France, puis d´Espagne sous
l´impulsion des services secrets Russes, T. est d´abord destiné à
l´exil à Cuba, puis finalement devant son refus catégorique aux
États-Unis.</p>
<h2>Révolution 1917</h2>
<p>Immédiatement après la nouvelle de la révolution en cours en Russie,
T. et sa famille embarquent. À l´ambassade, l´Angleterre accorde le
droit de passage aux réfugiés politiques. Cependant, arrivé au Canada,
T. et les autres sont incarcérés pendant un mois comme dangereux
socialistes. Mais le premier gouvernement révolutionnaire russe finit
par demander sa relaxe sous l´impulsion de Lénine.</p>
<h3>Installation à Pétrograd</h3>
<p>Pétrograd, alors sous le contrôle Allemand (anciennement St
Petersbourg) est le lieux de ralliement des bolchéviques. T. enchaîne
d´innombrables meetings en opposition au gouvernement révolutionnaire
à tendance libérale. C´est de Petrograd que la révolution Bolchévique
naîtra quelques mois plus tard.</p>
<h3>Guerre civile</h3>
<p>Après avoir repris le pouvoir aux mains de <a href="https://en.wikipedia.org/wiki/Alexander_Kerensky">Kerensky</a> la révolution s´est heurté à de nombreuses difficultés. En premier lieu la première guerre mondiale dans laquelle elle était engagée en particulier face à l´Allemagne et l´Autriche. T. était alors en charge des affaires étrangères et il a appliqué une stratégie de temporisation avec pour objectif de propager la révolution dans les pays de l´Est. Le <a href="https://fr.wikipedia.org/wiki/Trait%C3%A9_de_Brest-Litovsk">traité de Brest-Litovsk</a> fut une véritable tribune pour le communisme, ou était dénoncée la guerre des bourgeois sur le dos du prolétariat. Les bolchéviques refusèrent de signer la paix afin de ne pas ternir les idéaux révolutionnaires. Après l´echec des négociations et la reprise du front, T. démissionna et fut nommé à la tête de l´<a href="https://fr.wikipedia.org/wiki/Arm%C3%A9e_rouge">armée rouge</a>, où il fit preuve de poigne et motiva les hommes en les imprégnant de discours communistes. Enfin, la <a href="https://fr.wikipedia.org/wiki/Guerre_civile_russe">guerre civile russe</a>, largement entretenue par l´Angleterre, la France et la Finlande accapara T. qui à bord <a href="https://en.wikipedia.org/wiki/Trotsky%27s_train">de son train</a> pendant deux ans et demi parcouru les 14 fronts dans l´Ouest et au sud de la Russie face à <a href="https://fr.wikipedia.org/wiki/Arm%C3%A9es_blanches">l´armée blanche</a>.</p>
<p>T. insiste et donne tous les éléments pour prouver sa bonne entente avec Lénine.
Il démontre en disséquant chaque étapes de la mise sur pieds du gouvernement bolchévique que tous deux étaient sinon toujours d´accord sur les choix à faire, toujours dans une entente honnête et ouverte à la recherche de compromis toujours meilleurs.
À l´inverse, il décrit avec des détails croissants dans le récit les manipulations de Staline.
Il raconte à quel point Lénine avait perdu confiance et se méfiait de Staline jusqu´à donner ses dernières forces dans sa lutte pour l´évincer du pouvoir.
Par malchance, l´état de santé de Lénine l´empêchât d´aller au bout de son objectif comme la suite de l´histoire de l´URSS le prouve.
T. a été prévenu et était conscient des complots de Staline. Il a fait le choix de conserver son énergie pour la révolution; là où Staline mettait tous ses efforts à rassembler et mettre en place le maximum d´ennemis contre Trotsky.</p>
<h3>La Troïka</h3>
<p>Afin d´influencer le bureau lors des votes concernant la gestion du
parti, Staline va constituer une
<a href="https://en.wikipedia.org/wiki/List_of_leaders_of_the_Soviet_Union#List_of_troikas">troïka</a>
avec Kamenev et Zinoviev. T. les décrit tous deux comme infiniment
supérieurs à Staline politiquement, mais dénués de caractère et de
courage.</p>
<p>La maladie a eu une grande importance dans la prise de pouvoir
progressive et méthodique de Staline. Lénine est emporté par une
maladie artérielle qui le réduit au silence, tandis que Trotsky
souffre de fièvres qui le clouent au lit durant de très longs
mois. Ceci laisse toute la latitude à Staline et ses sbires de réduire
le pouvoir de T. d´abord timidement, puis sans vergogne. T. résume la
situation: les épigones ne pouvant pas s´attaquer directement à T.,
ils mettent en opposition ce dernier et le défunt Lénine.</p>
<p>Dans son titre "Dieu et Juif", Gainsbourg évouque la Troïka des purs
(Trotsky, Kamenev, Zinoviev), qui intervint dans un second temps
lorsque ces deux derniers ont pris conscience de la dangerosité de
Staline.</p>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/d9955baf-e4ca-4db7-bddd-1365edb1320d" frameborder="0" allowfullscreen=""></iframe>
<pre><code class="language-text">Le Capital tu as lu de l'Israélite
Karl Marx un beau bouquin
Et le trio bolchevik la troîka des purs eh bien
Tous trois de race sémite
Je te le prouverai tout à l'heure
Dieu est Juif
Juif et Dieu
Grigori Ievseîetch Apfelbaum dit Zinoviev
Lev Borissovitch Rosenfeld dit Kamenev
Lev Davidovitch Bronstein dit Trotsky
</code></pre>
<p>T. revient sur les raisons de l´échec révolutionnaire et de l´approche
qu´il était la meilleure de prendre. Il semble ne rien regretter:</p>
<blockquote>
<p>voilà pourquoi quand il s´agit de la lutte de grands principes, le
révolutionnaire ne peut avoir qu´une règle: "fais ce que dois,
advienne que pourra"</p>
</blockquote>
<h2>La déportation et l´exil</h2>
<p>Il était vraisemblablement délicat de se débarrasser de Trotsky en
raison du fort soutient ouvrier et de l´opposition en général. Le
<a href="https://fr.wikipedia.org/wiki/Gu%C3%A9p%C3%A9ou">guépéou</a> emmène de force T. et
sa famille dans un village à la frontière de la Chine. En l´espace de
6mois, il reçu et envoya 1500 courriers ou télégrammes. Le parti lui
intima l´ordre de cesser de commander l´opposition.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Ma vie — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/functional-programming-in-scalaFunctional programming in scala2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-07-13</time></span>
<span id="book-year" style="visibility:hidden;">2018</span>
<span id="book-author" style="visibility:hidden;">Martin Odersky</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Functional programming in scala</h1>
<p>The <a href="https://courseware.epfl.ch/courses/course-v1:EPFL+progfun1+2018_T1/courseware/116bff35543a4932a43dc6ac3602ce74/22f9667e539a47c0aea94b1411afb6d0/1?activate_block_id=block-v1%3AEPFL%2Bprogfun1%2B2018_T1%2Btype%40vertical%2Bblock%404bc0eecec50a4e69b06636fa81f64478">cheat sheet</a> recaps the course.</p>
<h2>List</h2>
<h3>More functions on lists</h3>
<p><a href="https://www.youtube.com/watch?v=Lww2PZwrH0Q">lecture 5.1</a></p>
<h3>Pairs and Tuples</h3>
<p><a href="https://www.youtube.com/watch?v=UULUoj2Itek">lectre 5.2</a></p>
<p>pairs allows pattern matching:</p>
<pre><code class="language-scala"><span class="hl opt">(</span>x<span class="hl opt">,</span> y<span class="hl opt">)</span> <span class="hl kwa">match</span> <span class="hl opt">{</span>
<span class="hl kwa">case</span> <span class="hl opt">(</span>Nil<span class="hl opt">,</span> y<span class="hl opt">) =></span> y
<span class="hl kwa">case</span> <span class="hl opt">(</span>x<span class="hl opt">,</span> Nil<span class="hl opt">) =></span> x
_ <span class="hl opt">=></span>
<span class="hl opt">}</span>
</code></pre>
<h3>Implicit parameters</h3>
<p><a href="https://www.youtube.com/watch?v=MFkW6zQQVIw">lecture 5.3</a></p>
<p>It's a good idea to specify <em>functions arguments</em> in the additional
parameters, because the compiler can infer the types of the functions
parameters from the regular arguments:</p>
<pre><code class="language-scala"><span class="hl slc">// no necessary to specify w and z types</span>
<span class="hl kwa">def</span> <span class="hl kwd">function</span><span class="hl opt">(</span>x<span class="hl opt">:</span>Int<span class="hl opt">,</span> y<span class="hl opt">:</span>Int<span class="hl opt">)(</span>ord<span class="hl opt">:</span> Ordering<span class="hl opt">):</span>Boolean <span class="hl opt">= {</span>
<span class="hl kwd">ord</span><span class="hl opt">(</span>x<span class="hl opt">,</span>y<span class="hl opt">) <</span> <span class="hl num">0</span>
<span class="hl opt">}</span>
</code></pre>
<p>Then it is possible to chose an implicit parameter because the
compiler is able to infer the kind of implicit function to look up.</p>
<pre><code class="language-scala"><span class="hl slc">// no necessary to specify w and z types</span>
<span class="hl kwa">def</span> <span class="hl kwd">function</span><span class="hl opt">(</span>x<span class="hl opt">:</span>Int<span class="hl opt">,</span> y<span class="hl opt">:</span>Int<span class="hl opt">)(</span><span class="hl kwa">implicit</span> ord<span class="hl opt">:</span> Ordering<span class="hl opt">):</span>Boolean <span class="hl opt">= {</span>
<span class="hl kwd">ord</span><span class="hl opt">(</span>x<span class="hl opt">,</span>y<span class="hl opt">) <</span> <span class="hl num">0</span>
<span class="hl opt">}</span>
</code></pre>
<h3>Higher order functions</h3>
<p><a href="https://www.youtube.com/watch?time_continue=1&v=McfmstwBYsE">lecture 5.4</a></p>
<p>A high order function takes a other function as argument.</p>
<h3>Reduction of lists</h3>
<p><a href="https://www.youtube.com/watch?time_continue=8&v=ZQvLqqYhJ2Y">lecture 5.5</a></p>
<pre><code class="language-scala"><span class="hl opt">(</span>x<span class="hl opt">,</span> y<span class="hl opt">)) =></span> x <span class="hl opt">*</span> y
<span class="hl slc">// can be rewritten</span>
_ <span class="hl opt">*</span> _
<span class="hl slc">// however</span>
<span class="hl opt">(</span>x<span class="hl opt">,</span> x<span class="hl opt">)) =></span> x <span class="hl opt">*</span> x
<span class="hl slc">// cannot because each _ represents a new parameter</span>
</code></pre>
<h3>Reasoning about concat</h3>
<p><a href="https://www.youtube.com/watch?v=J5v3N149f3g">lecture 5.6</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Functional programming in scala — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/movie/functional-program-design-in-scalaFunctional program design in scala2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-07-14</time></span>
<span id="book-year" style="visibility:hidden;">2018</span>
<span id="book-author" style="visibility:hidden;">Martin Odersky</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Functional program design in scala</h1>
<p>Thee <a href="https://courseware.epfl.ch/courses/course-v1:EPFL+progfun2+2018_T1/courseware/651f1b406e1d44a59c791a2c2e60e8a6/8bfe81c280ee467a8097589f70074e3d/1?activate_block_id=block-v1%3AEPFL%2Bprogfun2%2B2018_T1%2Btype%40vertical%2Bblock%40f5ed015f512645589d85970514876c5f">cheat sheet</a> recaps the course.</p>
<h1>For expression and monads</h1>
<p><a href="https://www.youtube.com/watch?v=7aINDO1LgHI">Monads</a></p>
<p>for expression can always be translated into high order functions
(map, flatmap, filters...)</p>
<p>A monad is a type which implement flatMap and a unit.</p>
<pre><code class="language-scala"><span class="hl kwa">def</span> flatMap<span class="hl opt">[</span>U<span class="hl opt">](</span>f<span class="hl opt">:</span> T <span class="hl opt">=></span> M<span class="hl opt">[</span>U<span class="hl opt">]):</span> M<span class="hl opt">[</span>U<span class="hl opt">]</span>
<span class="hl kwa">def</span> unit<span class="hl opt">[</span>T<span class="hl opt">](</span>x <span class="hl opt">:</span>T<span class="hl opt">):</span> M<span class="hl opt">[</span>T<span class="hl opt">]</span>
</code></pre>
<p>Moreover a monad must conform to 3 laws:</p>
<ul>
<li>associativity.</li>
</ul>
<pre><code class="language-scala">m flatMap f flatMap g <span class="hl opt">==</span> m <span class="hl kwd">flatMap</span> <span class="hl opt">(</span>x <span class="hl opt">=></span> <span class="hl kwd">f</span><span class="hl opt">(</span>x<span class="hl opt">)</span> flatMap g<span class="hl opt">)</span>
</code></pre>
<ul>
<li>left unit</li>
</ul>
<pre><code class="language-scala"><span class="hl kwd">unit</span><span class="hl opt">(</span>x<span class="hl opt">)</span> flatMap f <span class="hl opt">==</span> <span class="hl kwd">f</span><span class="hl opt">(</span>x<span class="hl opt">)</span>
</code></pre>
<ul>
<li>right unit</li>
</ul>
<pre><code class="language-scala">m flatMap unit <span class="hl opt">==</span> m
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Functional program design in scala — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/douglas-adams_le-guide-du-voyageur-galactiqueLe guide du voyageur galactique2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-07-15</time></span>
<span id="book-year" style="visibility:hidden;">1979</span>
<span id="book-author" style="visibility:hidden;">Douglas Adams</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★</span>
<span id="stars" style="visibility:hidden;">★★★★</span></span></p>
<h1>Le guide du voyageur galactique</h1>
<p>Des extraterrestres anéantissent totalement la terre ne laissant que
deux survivants, porteurs du fameux <em>guide du voyageur galactique</em>,
qui va les aider dans leurs aventures dans l´univers.</p>
<h2>Un drôle de vaisseau</h2>
<p>Contre toute attente, Arthur et Ford se retrouvent embarqués sur un
vaisseau propulsé par un générateur d´improbabilité infinies. Dans ce
dernier, les évènements les moins probables se réalisent.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Le guide du voyageur galactique — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/romeu-moura_panoptique-coach-agileLes coach agiles sont des jésuites2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-07-18</time></span>
<span id="book-year" style="visibility:hidden;">2020</span>
<span id="book-author" style="visibility:hidden;">Romeu Moura</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Les coach agiles sont des jésuites</h1>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/e081ec39-5758-4f98-a483-7445e573f487" frameborder="0" allowfullscreen=""></iframe>
<p>Via la <a href="https://fr.wikipedia.org/wiki/Syst%C3%A9mique">systémique</a>,
R. postule que les systèmes sont composés de trois éléments:</p>
<ul>
<li>entité</li>
<li>but</li>
<li>relation</li>
</ul>
<p>Il ajoute que les systèmes ont des <em>comportements émergents</em>, qui ne
sont pas souhaités, ni souhaitables par les entités.</p>
<p>Le but de tout système est de se conserver.</p>
<p>L´effet d´une entité peut être imperceptible mais du point de vue
systémique être très fort. Il peut être fort du point de vue
individuel mais imperceptible du point de vue systémique. L´exemple de
la Joconde et de la statue ou du racisme (des milions de petits effets
sont d´un grand effet).</p>
<p>Nuance entre Pouvoir et Contrôle. Le pouvoir s´appuie sur l´influence.</p>
<p>Quelques références:</p>
<ul>
<li><a href="https://fr.wikipedia.org/wiki/Utilitarisme">L´utilitarisme - Bentham</a></li>
<li><a href="https://fr.wikipedia.org/wiki/Surveiller_et_punir">Surveiller et punir - Foucault</a></li>
<li><a href="https://fr.wikipedia.org/wiki/Discours_de_la_servitude_volontaire">La servitude volontaire - La Boetie</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Les coach agiles sont des jésuites — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/movie/werner-herzog_fitzcarraldoFitzcarraldo2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-07-19</time></span>
<span id="book-year" style="visibility:hidden;">1977</span>
<span id="book-author" style="visibility:hidden;">Werner Herzog</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Fitzcarraldo</h1>
<p>En Amazonie, Fitzgerald a monté une entreprise de voie ferrée qui
périclite par manque de besoin et par la difficulté de la tâche. Par
la suite, il monte un entreprise de fabrique de glaçons par des
procédés chimiques. Il est associé à Mme Mollie. qui tient un bordel et par
ce biais lui permet de financer ses fantaisies.</p>
<p>Pour son troisième projet, il décide de démarrer une plantation
d´hévéas en plein coeur de l´amazonie dans un domaine sous contrôle
des indiens Jivaros - réputés très dangereux et qui réalisent des
réductions de têtes.</p>
<p>F. exploite le mythe des indiens Jivaros selon lequel un dieu blanc
les guidera vers un lieu dénué de souffrances. Il répond aux flèches
des indiens par l´opéra de Verdi. Les Jivaros sont impressionnés par la
jonque puis par la dynamite et la glace.</p>
<p>Afin d´atteindre le domaine d´exploitation des hévéas, il leur faut
atteindre un fleuve et franchir des rapides qui réduiraient en miettes
la jonque. F. décide de faire passer le navire par dessus la colline
"comme une vache passe par dessus la ferme".</p>
<p>Après avoir terrassé la colline à la dynamite, par un système à base
de nombreuses poulies et avec l´aide de centaines de Jivaros, ils
parviennent à faire passer la jonque sur l´autre bras du fleuve.</p>
<p>À la faveur de la pleine lune et de l´ivresse de la fête qui s´ensuit,
les Jivaros décident de sacrifier le "char sacré" pour apaiser les
esprits des indiens morts durant la manoeuvre. Ils se retrouvent
finalement à leur point de départ. Le bateau est revendu à son
propriétaire. Mais avant, ils ont l´occasion de jouer un opéra de
Verdi par Caruso dessus.</p>
<p>Deux ans plus tard, <a href="https://fr.wikipedia.org/wiki/Apocalypse_Now">Apocalypse
Now</a> sortait. On peut
dire que les grands moments de l´opéra de Wagner dans les hélicoptères
et la traversée du fleuve sur la frégate et l´attaque des indiens est
largement inspirée de ce chef d´oeuvre.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Fitzcarraldo — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/movie/naomie-klein_shock-doctrineLa stratégie du choc / Capitalisme du désastre2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-08-05</time></span>
<span id="book-year" style="visibility:hidden;">2015</span>
<span id="book-author" style="visibility:hidden;">Naomi Klein</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>La stratégie du choc / Capitalisme du désastre</h1>
<p>Le documentaire reprend l´histoire des 60 dernières années sous le
prisme de la stratégie du choc tel que pensé par Naomi Klein dans son
livre 2007 du même nom.</p>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/23e98fc1-f862-48d2-a0d9-1ed17a2d56fe" frameborder="0" allowfullscreen=""></iframe>
<h2>La torture par le choc</h2>
<p>En 1951 un programme de recherche sur la privation sensorielle financé
par l´armé a débuté. Ils se sont rendu compte que ces privations sont
plus dures à supporter que les douleurs physiques ou morales. Le
psychiatre <a href="https://en.wikipedia.org/wiki/Donald_Ewen_Cameron">Donald
Cameron</a> s´est
livré à des expériences sur les électrochocs, et martèlements
audio. Nombreuses expériences menées à cette époque sont reprises dans
le <a href="https://en.wikipedia.org/wiki/U.S._Army_and_CIA_interrogation_manuals#CIA_manuals">manuel d´interrogatoire de la
CIA</a></p>
<h2>La dérégulation financière par le choc</h2>
<p>Par ailleurs <a href="https://en.wikipedia.org/wiki/Milton_Friedman">Milton
Friedman</a> construisait
sa théorie comme quoi une thérapie de choc devait être appliquée à
l´économie pour faire accepter à la population les dérégulations.</p>
<h2>Le Chili</h2>
<p>Après des décennies de démocratie aux Chili, <a href="https://en.wikipedia.org/wiki/Salvador_Allende">Salvator
Allende</a> décida de
nationaliser de nombreuses société y compris celle de la téléphonie
dans laquelle <a href="https://en.wikipedia.org/wiki/Richard_nixon">Richard
Nixon</a> avait des
intérêts. La CIA mis le <a href="https://en.wikipedia.org/wiki/Augusto_Pinochet">Général
Pinochet</a> au pouvoir
pour appliquer la dérégulation économique qui se solda par une
inflation inédite dans un pays jusqu´à présent prospère.</p>
<h2>L´Argentine</h2>
<p><a href="https://en.wikipedia.org/wiki/Jorge_Rafael_Videla">Général Videla</a></p>
<h2>L´UK & les USA</h2>
<p>Dans les années 80 deux disciples de Friedman arrivent au pouvoir dans
deux pays anglo-saxons: <a href="https://en.wikipedia.org/wiki/Margaret_Thatcher">Margaret
Thatcher</a> et <a href="https://en.wikipedia.org/wiki/Ronald_Reagan">Ronald
Reagan</a>.</p>
<p>En Angleterre, Thatcher fera plier les syndicats des mineurs après un
ans de bras de fer. Puis pour augmenter sa cote de popularité en
dégringolade, elle engagera son pays dans une guerre à l´autre bout du
monde <a href="https://fr.wikipedia.org/wiki/Guerre_des_Malouines">aux
Malouines</a>. Des
mesures économiques suivirent ce plébiscite: privatisation de nombreux
secteurs de l´économie, gaz, électricité, aéronautique, téléphonie...
On parle de
<a href="https://en.wikipedia.org/wiki/Big_Bang_(financial_markets)">Big-Bang</a>
économique.</p>
<p>Aux USA, Reagan, était accompagné de disciples de Friedman tel que
<a href="https://en.wikipedia.org/wiki/Donald_Rumsfeld">Donald Rumsfeld</a>.</p>
<h2>L´URSS</h2>
<p>Boris Yeltsin,</p>
<h2>L´Irak</h2>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=La stratégie du choc / Capitalisme du désastre — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/damien-macdonald_notre-dame-de-parisNotre Dame de Paris2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-08-05</time></span>
<span id="book-year" style="visibility:hidden;">2020</span>
<span id="book-author" style="visibility:hidden;">Damien MacDonald</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<p><span id="ystars" style="visibility:hidden;">★★★★</span>
<span id="stars" style="visibility:hidden;">☆</span></p>
<h1>Notre Dame de Paris</h1>
<p>Ce roman graphique a la particularité d´être totalement fidèle au
texte de Hugo, ce qui permet d´en apprécier la puissance et la
profondeur.</p>
<p><img src="/images/gallery/notredame/medium_0001.webp" alt=""></p>
<p><img src="/images/gallery/notredame/medium_0002.webp" alt=""></p>
<p><img src="/images/gallery/notredame/medium_0003.webp" alt=""></p>
<p><img src="/images/gallery/notredame/medium_0005.webp" alt=""></p>
<p><img src="/images/gallery/notredame/medium_0006.webp" alt=""></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Notre Dame de Paris — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/gillo-pontecorvo_la-bataille-algerLa bataille d´Alger2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-08-09</time></span>
<span id="book-year" style="visibility:hidden;">1966</span>
<span id="book-author" style="visibility:hidden;">Gillo Pontecorvo</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>La bataille d´Alger</h1>
<p>Le film si longtemps interdit.</p>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/91c55289-9560-4c4c-b1c4-c695b6de66c6" frameborder="0" allowfullscreen=""></iframe>
<p>Une analyse du film ; de son élaboration à sa diffusion.</p>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/1315bf33-6ff7-4e36-8370-6de80da2b41a" frameborder="0" allowfullscreen=""></iframe>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=La bataille d´Alger — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/antonio-gramsci_lettres-de-prisonLettres de prison2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-08-15</time></span>
<span id="book-year" style="visibility:hidden;">1947</span>
<span id="book-author" style="visibility:hidden;">Antonio Gransci</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Lettres de prison</h1>
<h2>À Ustica</h2>
<iframe width="425" height="350" frameborder="0" scrolling="no" marginheight="0" marginwidth="0" src="https://www.openstreetmap.org/export/embed.html?bbox=7.800292968750001%2C35.862343734896506%2C15.798339843750002%2C41.43449030894922&layer=mapnik" style="border: 1px solid black"></iframe><br><small><a href="https://www.openstreetmap.org/#map=7/38.703/11.799">View Larger Map</a></small>
<p>Sur cette petite île au nord de la Sicile, Gramsci. se retrouve
emprisonné en compagnie d´autres détenus politique (anarchistes,
communistes, anciens députés). Ils sont mis à l´écart des détenus de
droits communs qui sont traités et dans une situation bien différente
(violence, usurier, alcoolisme).</p>
<p>Afin de ne pas "s´abrutir", G. demande à ses proches des livres, en
particulier de quoi apprendre l´Allemand et le Russe, des livres sur
l´économie, l´histoire et la géographie ainsi que de la littérature
tel que la Divine Comédie de Dante. De plus, des cours sont organisés
pour les détenus politiques auxquels se mêlent quelques villageois.</p>
<p>G. n´a pas tellement à se plaindre de sa situation dans un endroit si
paisible hormis qu´il dort trop peu pour différentes raisons y compris
sa propre nature de petit dormeur.</p>
<h1>À Milan</h1>
<p>Après trois mois sur l´île, G. est transféré (en 20 jours!) à la
prison de Milan. À nouveau, il est correctement traité: il peut écrire
deux lettres par semaines, recevoir et lire 8 livre et 7 journaux par
semaine. Il regrette de ne pas pouvoir écrire et prendre des notes ce
qui l´empêche de mener des travaux intellectuels rigoureux.</p>
<p>G. souffre de l´attente des lettres qu´il ne reçois jamais assez
souvent à son goût. Il a par ailleurs de grandes difficultés à écrire
à sa femme et des liens toujours plus forts avec sa belle soeur,
c.-à-d. la soeur de sa femme.</p>
<h1>À Turin</h1>
<p>Le transfert de G. prend 19 jours au cours desquelles il décrit une
terrible souffrance analogues au “feu de Saint-Antoine” l'empêchant de
dormir pendant 12 jours. Par la suite, il se retrouve dans une cellule
avec 4 prisonniers politiques malades de la tuberculose. Pour la
première fois, il s'adresse avec véhémence à Tania au sujet de des
initiatives qu'elle prend sans son accord voire avec son désaccord.</p>
<p>À partir de 1929, soit 2 ans(?) après son internement G. obtient
l'accord pour pouvoir écrire dans sa cellule. Il commence par rédiger
des traductions pour se refaire l'esprit.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Lettres de prison — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/origine-du-capitalismeL'origine du capitalisme2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-08-16</time></span>
<span id="book-year" style="visibility:hidden;">2014</span>
<span id="book-author" style="visibility:hidden;">Ilan Ziv</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>L'origine du capitalisme</h1>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/80242c69-5a40-459b-b078-c667210d90b8" frameborder="0" allowfullscreen=""></iframe>
<h2>La théorie de l´économie de marché</h2>
<p><a href="https://en.wikipedia.org/wiki/Adam_Smith">Adam Smith</a> est considéré
comme celui qui a théorisé et constaté le capitalisme en de 1776 avec
sa publication <a href="https://fr.wikipedia.org/wiki/Recherches_sur_la_nature_et_les_causes_de_la_richesse_des_nations">La richesse des
Nations</a>. Il
s´inspire entre autre de l´école de la physiocratie du médecin
<a href="https://fr.wikipedia.org/wiki/Fran%C3%A7ois_Quesnay">François
Quesnay</a> qui
cherche à décrire l´économie en mécanismes cycliques comme l´est la
fonction cardio-vasculaire. Ils décrivent donc les flux circulaires
entre producteurs et consommateurs. Pour Smith, l´origine de
l´économie provient de l´inclinaison de l´homme à faire du troc et du
commerce.</p>
<p>Cependant cette théorie est erronée, le mécanisme de "réciprocité" qui
consiste à mettre en commun les productions est davantage en place que
le troc dans les tribus primitives.</p>
<h2>La découverte de l´Amérique</h2>
<p>On peut voir les voyages des conquistador comme des entrepreneurs qui
étaient financés par des investisseurs. Les rendements peuvent être
considérables en cas de découvertes. À l´instar des startup actuelles,
beaucoup de ces entrepreneurs échouaient, mais il suffisait qu´un seul
conquiert quelque chose pour que le bénéfices rattrape les
investissements.</p>
<p>Des gens comme
<a href="https://fr.wikipedia.org/wiki/Hern%C3%A1n_Cort%C3%A9s">Cortes</a>
étaient poussés par leurs dettes pour tenter une prouesse et levaient
des financements de manière désespérés. Une spirale de conquêtes, de
profits, d´investissements et de découvertes s´est débridée.</p>
<h2>Un impact mondial</h2>
<p>Les mines d´argents d´Amérique du Sud sont l´occasion pour la Chine de
se tourner avec un grand bénéfice vers une monnaie en argent. Ceci eut
pour effet d´encourager les petits producteurs indépendants. En
revanche, l´effet fut inverse pour ceux d´Europe. En effet le système
des "enclosure" va créer des grandes fermes en fusionnant plusieurs
petites fermes de manière coercitive et mettre ces dépossédés sur la
paille. Avec le supports législatif pour criminaliser la pauvreté, les
gens ont été poussés à devenir de la main d´oeuvre prolétarienne.</p>
<h2>Le commerce triangulaire</h2>
<p>Les marchandises allaient d´Europe vers l´Afrique. Des esclaves
allaient entre l´Afrique et l´Amérique. Des denrées depuis l´Amérique
vers l´Europe. Adam Smith était témoins de ces procédés, mais il a
fermé les yeux dans ses théories. En 400 ans d´esclavage, le racisme
est né comme une conséquence, probablement comme manière de supporter
la maltraitance appliquée aux populations.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=L'origine du capitalisme — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/george-orwell_la-ferme-des-animauxLa ferme des animaux2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-08-16</time></span>
<span id="book-year" style="visibility:hidden;">1945</span>
<span id="book-author" style="visibility:hidden;">George Orwell</span>
<span id="reading-time" style="visibility:hidden;">2 minutes
<span id="ystars" style="visibility:hidden;">★★★★</span>
<span id="stars" style="visibility:hidden;">★</span></span></p>
<h1>La ferme des animaux</h1>
<blockquote>
<p>Les animaux sont tous égaux. Certains sont plus égaux que les autres.</p>
</blockquote>
<p>À l´approche de sa fin, un vieux verrat se remémore les paroles d´une
vieille chanson révolutionnaire, qui exhorte les animaux à retrouver
leur indépendance vis-à-vis de l´homme. Il réunit les animaux de la
ferme et après sa plaidoirie, tous entonnent le champs de la
révolution, jusqu´à ce que le fermier remettent de l´ordre dans ce
raffut.</p>
<p>Les animaux s´entendent pour résister au fermier et à la première
occasion vont le chasser de la ferme. Plus tard il reviendra avec des
miliciens pour reprendre le contrôle mais les animaux adopteront une
stratégie militaire pour l´emporter: la fameuse <strong>bataille de
l´Étable</strong>. À l´issue de quoi les deux cochons Napoléon et Boule de
Neige seront médaillés.</p>
<p>La ferme sera alors rebaptisée la ferme des animaux, et sept
commandements seront promulgués ainsi que des rites hebdomadaires
autour du crâne du vieux verrat, du champs révolutionnaire au son des
fusils. Il est décidé que les contacts avec l´homme et ses dérivés
(maison, lits, outils...) soient proscrits. L´organisation sociale de
la ferme s´appuie sur les qualités de chaque catégorie d´animaux: les
cochons, les plus intelligents réfléchissent et décident de la marche
à suivre; les autre se mettent au travail.</p>
<p>Les deux cochons en chef ne sont pas d´accord sur un certain nombre de
choix et les autres animaux départagent lors de débats. En particulier
Napoléon milite pour renforcer la ferme tandis que Boule de Neige
souhaite propager la révolution aux autres ferme. En outre, ce dernier
propose l´édification d´un moulin qui aurait pour bénéfice d´aider les
animaux et leur octroyer une semaine de 3 jours, avec bien sûr un
travail initial de mise en place important.</p>
<p>Napoléon, en passe de perdre le débat en appelle a ses chiens qu´il a
élevé et fait fuir Boule de Neige de la ferme. Dès lors, Napoléon est
le seul maître et il va progressivement faire basculer la ferme à sa
façon. En premier lieu il va petit à petit modifier les sept
commandements. Il va aussi peu à peu modifier l´histoire de la ferme,
en désignant Boule de Neige comme un ennemi, responsable de toutes les
infortunes. Enfin le travail des animaux deviendra toujours plus
difficile, avec une retraite qu´aucun d´atteindra à cause du
surmenage.</p>
<p>Finalement le commerce reprend avec les humains et les cochons
investirons la ferme et dormirons dans les lits Finalement le commerce
reprend avec les humains et les cochons investirons la ferme et
dormirons dans les lits. Il iront jusqu´à marcher à deux pattes et
fraterniser avec les hommes qui seront élogieux à l´égard de la ferme
et dans leur capacité à faire travailler les animaux supérieure aux
homme.</p>
<p>Publié en 1945, la fable d´Orwell est une métaphore de la révolution
communiste et de son devenir. Marx, Lénine/Trotsky et Staline sont
sous les traîts des cochons, tandis que le cheval de trait représente
la classe ouvrière, le corbeau le Clergé et l´âne l´anarchiste. Les
hommes et les fermes avoisinantes représentent les pays européens,
monarchies, républiques, fachisme. Orwell dépeint comment l´avidité et
la soif de pouvoir peuvent s´appuyer sur la force de travail, la
confiance, la sottise et la bienveillance pour mettre en place des
totalitarismes indéboulonables.</p>
<p>Orwell aura relativement tôt une vision claire de la révolution russe,
relativement à beaucoup comme J.P Sartre qui soutiendront Staline
jusqu´en 1956 au moment de la répression de l´<a href="https://fr.wikipedia.org/wiki/Insurrection_de_Budapest">insurection de
Budapest</a>. Cependant,
les mémoires de Trotsky publiées 15 ans plus tôt n´étaient pas
équivoques en ce qui concernait le stalinisme.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=La ferme des animaux — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/soljenitsyne_le-premier-cercleLe Premier Cercle2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-08-31</time></span>
<span id="book-year" style="visibility:hidden;">1957</span>
<span id="book-author" style="visibility:hidden;">Soljenitsyne</span>
<span id="reading-time" style="visibility:hidden;">2 minutes</span></p>
<h1>Le Premier Cercle</h1>
<p>Ce roman historique s´inscrit dans la fin du stalinisme. Plusieurs
histoires parallèles semblent converger peu à peu. Les péripéties des
protagonistes sont l´occasion pour Soljenitsyne de discuter ses thèses
critiques sur l´histoire ou de donner des détails avérés ou issus de
l´imaginaire collectif ou du sien.</p>
<h2>La charachka: le goulag des ingénieurs</h2>
<p>Dans cette catégorie de goulag on a droit à de la viande et du beurre,
ce qui en fait le moins pire de ce type d´endroit. Ceci-dit, on a le
droit à des visites seulement une fois par an et donc ce qui pèse le
plus sur les prisonniers: pas de femme. On y travaille en principe 12h
par jour et ne sont admis en majorité que les ingénieurs ou divers
savants pour avancer sur des projets d´ordre technologiques et
secrets.</p>
<h2>Les relations hiérarchiques</h2>
<p>Le roman décrit en détail les interactions entre des personnages ayant
des statuts dans la société très différents. Entre les ingénieurs et
les externes, entre les gardiens et les responsables du charachka,
entre ministres et Staline, entre son secrétaire personnel et ce
dernier. Dans la plupart des cas, la relation est faussée, sur des
bases de défiance et de crainte.</p>
<h2>Réflexions</h2>
<h3>Sur la religion</h3>
<p>La question du Christianisme est discuté par les «zeks»
(prisonniers). Est-il nécessaire de suivre la Bible au mot prêt ou
bien peut-on s´accorder des légèretés avec le dogme ?</p>
<h3>Sur l´anarchisme</h3>
<h3>Sur la linguistique</h3>
<h3>Sur la littérature</h3>
<p>Un certain nombre d'écrivains et d'œuvres sont citées dans l'ouvrage;
parfois avec des critiques qui engagent les protagonistes et leur
caractère.</p>
<blockquote>
<p><strong>Gorki</strong> pense fort justement, mais il est plutôt rébarbatif. <strong>Maiakovski</strong> de même mais il est un peu balourd. <strong>Salkykov-Chtchedrine</strong> est un ami du progrès, certes, mais l'ensemble de ses œuvres est à vous tuer raide d'ennui ; et puis <strong>Tourguéniev</strong>, bridé par son idéal d'aristocrate ; <strong>Gontcharov</strong>, qui se rattache à la naissance du capitalisme en Russie ; <strong>Léon Tolstoï</strong> et ses prises de positions en faveur de la paysannerie patriarcale (leur professeur leur avait déconseillé la lecture des romans de Tolstoï qui sont bien longs, et ne peuvent que troubler la clarté des lumineuses critiques qui leur ont été consacrées); ensuite c'était la revue générale d'absolus inconnus comme <strong>Stepniak-Kravtchinski</strong>, <strong>Dostoïevski</strong>, <strong>Soukhovo-Kobyline</strong>, dont il n'était même pas utile de retenir les titres. Pour cette file étirée au long des années, seul <strong>Pouchkine</strong> brillait comme un beau soleil.</p>
</blockquote>
<h3>La transgression</h3>
<p>L'escapade en train pour la nature est l'occasion pour deux
personnages clé d'enfreindre les règles et de se rapprocher dans cette
démarche. Cela fait irrésistiblement penser à <a href="https://en.wikipedia.org/wiki/Nineteen_Eighty-Four">1984
d'Orwell</a>,
d'autant que l'homme se met à boiter - détail qui ne semble fait que
pour affirmer ce clin d'œil.</p>
<h3>Sur la peinture</h3>
<h3>Sur le progrès</h3>
<blockquote>
<ul>
<li>En art, par exemple, il n'y a aucun progrès ! Et il ne peut y en avoir !</li>
<li>De fait ! De fait ! Et c'est saisissant ! s'extasia Nerjine. Il y a eu un Rembrandt au XVII siècle et on peut toujours esayer aujourd'hui de lui faire la pige ! Qu'était la technique au XVII siècle ? Aujourd'hui elle nous paraît à la mesure de sauvages. Quelles pouvaient être les innovations techniques dans les années 1870 ? Des amusettes pour enfants. Et c'est dans ces années là qu'on a écrit Anna Karénine. Que peut on me proposer de supérieur ?</li>
</ul>
</blockquote>
<h3>Sur l'amitié</h3>
<h3>Sur la condition féminine</h3>
<h3>Sur la hiérarchie sociale</h3>
<blockquote>
<p>Il faut remodeler la conscience de l'humanité de façcon que les hommes n'aient l'orgueil que du trit manuel et qu'ils rougissent d'être surveillants, ou directeurs, ou pontes d'un parti. Il faut faire en sorte que le titre de ministre soit aussi peu claironnant que celui d'égoutier. Le métier de ministre est indispensable lui aussi, mais infamant. Si une fille épouse un fonctionnaire de l'État, que cette souillure rejaillisse sur toute la famille ! Voilà mon socialisme à moi !</p>
</blockquote>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Le Premier Cercle — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/le-proces-eichmannLe procès d'Adolf Eichmann2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-05</time></span>
<span id="book-year" style="visibility:hidden;">1961</span>
<span id="book-author" style="visibility:hidden;">Michael Parzan</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Le procès d'Adolf Eichmann</h1>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/6e268acd-6666-4eca-8366-6e0c97802aff" frameborder="0" allowfullscreen=""></iframe>
<p>Ce procès qui s'est tenu en Israel en 1961 a été largement médiatisé
et retransmit à l'international sur des chaines de télévisions. De
manière interessante, la grande majorité du public, y compris
israelien a découvert les camps d'extermination; soit 15 ans après la
fin de la guerre. C'est dire combien une chappe de plomb a pesé sur
les mémoires. C'est aussi peut-être dû à la nature du média: Primo
Levi avait décrit les camps dès 1947 avec ”Si c'est un Homme”. De même
les procès d'Auschwitz s'étaient tenu en 1952 en Allemagne. Mais la
télévision a <em>visiblement</em> un pouvoir d'information largement supérieur
aux livres, journaux et bouche à oreille.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Le procès d'Adolf Eichmann — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/movie/fritz-bauer_un-procureur-contre-le-nazismeFritz Bauer, un procureur contre le nazisme2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-06</time></span>
<span id="book-year" style="visibility:hidden;">2020</span>
<span id="book-author" style="visibility:hidden;">arte</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Fritz Bauer, un procureur contre le nazisme</h1>
<iframe width="560" height="315" sandbox="allow-same-origin allow-scripts allow-popups" src="https://video.ploud.fr/videos/embed/7985bd4c-304d-460c-8621-e0547f9ef54e" frameborder="0" allowfullscreen=""></iframe>
<p>Le procureur général juif Fritz Bauer a mené les rares procès allemands
contre les nazi à l'après guerre. Il a eu de nombreuses difficulté et
son entreprise n'a pas été un succès à la mesure de l'enjeu.</p>
<p>Il lui aura fallu patienter 4 ans avant de pouvoir revenir en
Allemagne sous prétexte que les allemands n'étaient pas prêts à avoir
des cadres juifs sitôt la fin de la guerre.</p>
<p>La majorité de ses collègues procureurs avaient collaboré et soutenu
les injustices du régime nazi et il pouvait compter sur eux pour lui
mettre des battons dans les roues. C'est pour cette raison qu'il donna
Eichmann aux israéliens et que le procès ne se passa donc pas en
Allemagne, privant le pays d'un débat nécessaire. Il devra d'ailleurs
s'éteindre alors qu'il initiait un procès contre ses pairs.</p>
<p>Le procès d'Auschwitz qu'il mena en 1964 tout de même fut une tribune
et l'occasion pour de nombreux allemands de découvrir l'horreur des
camps, ainsi que la banalité des coupables. De manière intéressante,
aucun d'eux ne se repentirent ou avouèrent leur culpabilité malgré
leur participation active dans les camps. En règle générale, lors des
génocides, les auteurs ne se sentent pas responsables de leurs actes
et invoquent leur devoir auprès de la hiérarchie. On peut cependant
citer Albert Speer, ministre de l'armement de Hitler qui s'il n'a pas
admit avoir été au courant des massacres dans les camps (ce qui s'est
avéré être un mensonge) a tout de même reconnu l'horreur et son
repentir d'avoir participé indirectement à ces crimes.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Fritz Bauer, un procureur contre le nazisme — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/alberto-breccia_les-mythes-de-cthulhuLes Mythes de Cthulhu2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-06</time></span>
<span id="book-year" style="visibility:hidden;">1973</span>
<span id="book-author" style="visibility:hidden;">Alberto Breccia</span>
<span id="reading-time" style="visibility:hidden;">4 minutes
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Les Mythes de Cthulhu</h1>
<blockquote>
<p>« J'ai vu s'ouvrir l'univers des ténèbres où tournent les planeètes noires. Elles tournent dans son horreur inconnue : sans ordre, sans éclat et sans nom » Némésis
<img src="/images/gallery/cthulhu/1.webp" alt=""></p>
</blockquote>
<p>(texte de postface des éditions Rackham)
« Je me suis rapidement aperçu que les moyens traditionnels de la bande dessinée n'étaient pas suffisants pour représenter l'univers de Lovecraft, et j'ai commencé à expérimenter de nouvelles techniques, comme le monotype ou le collage. Ces monstre sans forme, comme ceux que j'ai créés dans <em>El Eternauta</em> sont ainsi faits parce que je ne voulais pas me limiter à donner au lecteur ma propre représentation de ces monstres ; je voulais qu'il y ajoute du sien : qu'il utilise cette base informe que je lui donnais, pour y greffer ses propres frayeurs, sa propre peur. [...] Au début, il s'agissait d'un défi : je voulais savoir si j'étais capable de dessiner ce que Lovecraft écrivait. J'ignore si j'ai réussi, mais je peux vous assurer que, pendant deux ou trois ans - je ne me souviens pas exactement combien de temps ce travail m'a pris-, j'ai vécu complètement immergé dans le monde de Lovecraft. [...] Et j'expérimentais de nouvelles techniques, tout en essayant de restituer le plus fidèlement possible ce que l'écrivain avait imaginé dans ses nouvelles. »</p>
<p>Par ces mots, tirés d'une interview de 1989, Breccia donne le sens ultime de son travail d'adaptation des <em>Mythes de Cthulhu</em>. Dans le même entretien, il ajoute que sa rencontre avec l'œuvre de l'écrivain de Providence a été totalement fortuite : la veille d'un long voyage en train de Madrid à Milan, il avait acheté une édition de poche de l'<em>Abomination de Dunwich</em> (ou, peut-être, du <em>Cauchemar d'Innsmouth</em>). La lecture de ces pages dans le train aurait fait naître dans l'esprit du dessinateur l'idée d'adapter Lovecraft en bande dessinée.</p>
<p><img src="/images/gallery/cthulhu/2.webp" alt="">
Dans les entretiens de Juan Sasturain (datant de l'automne 1987 et recueillis dans le volume <em>Breccia, el Viejo</em>), Alberto Breccia situe cet épisode en 1959, au cours de son tout premier voyage en Europe. Quatorze ans vont s'écouler avant que Breccia commence l'adaptation du premier <em>Mythe</em>. Au cours du même entretien, il ajoute : « ... À cette époque, j'avais rassemblé tous les textes des Mythes, je les avais bien étudiés et je sentais d'être en mesure de les adapter. Quoi qu'il en soit, j'ai dû réaliser plusieurs essais de la première adaptation, <em>Le cérémonial</em>. Ils n'ont rien donné et je les ai jetés. [...] Je ne me rappelle plus l'ordre exact, mais après <em>Le cérémonial</em> j'ai dessine <em>Le Cauchemar d'Insmouth</em> et <em>Le monstre sur le seuil</em>; c'est avec ces trois travaux qu'en 1973 j'ai pris la décision de partir [pour l'Europe]. [...] Des nouvelles perspectives s'ouvraient à moi ; je n'étais plus un salarié mais un professionnel qui pouvait consacrer le temps qu'il fallait à chaque travail. Je commençais à goûter au plaisir de dessiner d'une autre façon. Je ne sais pas, j'ai du mal à expliquer ce que je ressentais.»</p>
<p>Breccia, en réalité, ne faisait que renouer avec une fréquentation commencée encore plut tôt. Dès sa jeunesse, le dessinateur avait dû lire, avec les pages de Lord Dunsany ou de Lafcadio Hearn, les œuvres de Lovecraft dans l'édition argentine de la revue Weird Tales. Le temps et les lectures successives avaient probablement effacé le souvenir de l'écrivain ; peut-être seulement son nom, puisque les atmosphère oppressantes de Lovecraft ou de Dunsany n'avaient jamais cessé d'alimenter l'imaginaire du maître uruguayen. Pendant toute sa longue carrière, il ne cessera jamais d'en faire une de ses principales références. Au début des années 1970, Breccia explorait la brumeuse région qui se trouve entre littérature et bande dessinée. Ses motivations étaient en même temps d'ordre théorique (comment « mélanger » les genres tout en gardant les caractéristiques qui leur sont propres), formel (enrichir la « grammaire » de la bande dessinée de nouveaux artifices narratifs) et enfin plastiques.</p>
<p><img src="/images/gallery/cthulhu/3.webp" alt="">
Le dessinateur avait commencé, en collaboration avec Norberto Buescaglia, l'adaptation en bande dessinée de <em>Rapport sur les aveugles</em>, extrait du roman <em>Héros et tombes</em> d'Ernesto Sabato. Le travail de découpage était terminé, et une bonne partie de l'œuvre était au stade de crayonné. Mais ce travail - qui finalement verra le jour presque trente ans plus tard - n'avait pas plu à Sábato. L'écrivain ne souhaitait pas que l'on modifie, même de façon marginale, le texte de son œuvre. Breccia et Buscaglia, dans leur travail, avaient dû plier le texte aux nécessités de l'adaptation et, face aux énormes difficultés que supposait la contrainte imposée par Sábato, ils avaient finalement décidé d'abandonner le projet. Breccia s'était douc retourné vers Lovecraft, qu'il venait de redécouvrir ; après avoir adapté lui-même <em>Le Cérémonial</em>, il avait proposé à Buscaglia de poursuivre la collaboration commencée avec <em>Rapport sur les aveugles</em> et adapter les autres récits du cycle de Mythes.</p>
<p>Ce choix n'est pas surprenant si l'on pense aux similitudes qui existent entre les atmosphères de Lovecraft et celle du roman de Sábato ; Breccia voulait utiliser le dessin pour reproduire ces sensations d'angoisse que Lovecraft parvenait si admirablement à construire avec les mots. Le dessinateur voulait produire des images susceptibles de réveiller chez le lecteur les peurs qu'il garde enfouies en lui. Le point central de sa recherche devient donc l'élaboration d'éléments formels et d'un langage plastique qui alimentent une sensation d'horreur et suggèrent la présence inquiétante de mondes et de créatures invisibles mais sensibles, l'immanence d'un univers parallèle, origine et négation du nôtre.</p>
<p><img src="/images/gallery/cthulhu/4.webp" alt="">
L'adaptation de Breccia et Buscaglia se base essentiellement sur les images ; le texte, le plus souvent des citations du texte original, constitue le fil rouge de la narration, presque toujours menée en première personne et avec peu de dialogues. Dans les ébauches de ces histoires qui sont arrivées jusqu'à nous, Breccia se limite à un découpage sommaire : les personnages sont à peine esquissés, les volumes simplement mis en place, les détails pratiquement absents. Et aux endroits qui, dans l'œuvre finalisé, seront occupés par les plus extraordinaires inventions graphiques du dessinateur, on trouve souvent les mots : « voir texte. »</p>
<p>Ce qui saute aux yeux à la vue des originaux de ces pages, c'est une sorte d'instantanéité de la création, fruit de l'inspiration, de la maîtrise et du hasard, dans le but de plier la matière à son vouloir. Dans les puissants coups de pinceau qui donnent vie aux monstres lovecraftiens, dans les papiers collés si sèchement déchirés qui se transforment en paysages désolés, dans le souffle qui étale l'encre de Chine sur la feuille pour en faire des arbres squelettiques, ressort intacte la relation torturée et presque physique que Breccia entretenait avec le dessin. Comme pour accroître l'effet produit sur le lecteur par ces sortes de taches de Rorschach, Breccia choisit de représenter notre univers, qui s'oppose à celui de Yog-Sothoth, dans un style réaliste qui frôle souvent l'académisme.</p>
<p>Le but est atteint ; à mesure qu'il cherche à déchiffrer et à donner un sens à cette matière « d'un autre monde », le lecteur est pris au piège de ses propres angoisses et phobies. Comme dans les pages de Lovecraft, un sentiment de gène et d'horreur ne tarde pas à se matérialiser, inquiétant et insondable. Il n'y a pas de doute, le monde oppressant et halluciné de l'écrivain de Providence se reproduit, intacte, dans l'œuvre de Breccia. Plus de quarante ans après leur réalisation, les <em>Mythes de Cthulhu</em> sont un exemple ineffable des complexes relations entre littérature et bande dessinée.</p>
<p>Les récits qui constituent le cycle des Mythes de Cthulhu ont été réalisés en moins de deux ans, de 1973 à 1975 et ont été publiés pour la première fois dans la revue italienne Il mago à partir de novembre 1973. Le premier recueil en volume, publié par l'éditeur italien L'isola Travota en 1978, et sur lequel se sont basées toutes les éditions successives, ne comprenait pas _Celui qui chuchotait dans les ténèbres_qui a été publié pour la prenière fois en 1979 dans la revue argentine Il Pendulo. La photogravure de cette édition a été réalisée en partant des planches originales, à l'exception de <em>Celui qui hantait les ténèbres</em>, qui a été repris de l'édition italienne car la plupart de ses pages ont été perdues.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Les Mythes de Cthulhu — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/reading/homere_odysseeOdyssée2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-21</time></span>
<span id="book-year" style="visibility:hidden;">-800 av. J-C</span>
<span id="book-author" style="visibility:hidden;">Homère</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Odyssée</h1>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Odyssée — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/movie/werner-herzog_grizzly-manGrizzly Man2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-22</time></span>
<span id="book-year" style="visibility:hidden;">2005</span>
<span id="book-author" style="visibility:hidden;">Werner Herzog</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Grizzly Man</h1>
<p>Le documentaire reprend des séquences réalisée par <a href="https://en.wikipedia.org/wiki/Timothy_Treadwell">Thimothy
Treadwell</a> auprès des
Grizzly qu'il fréquenta pendant 13 étés. Acteur raté, il était devenu
une célébrité en réalisant ces films engagés pour la cause des ours et
et des animaux sauvages.</p>
<p>Hersog compare la démarche de Tim à celle de
<a href="https://en.wikipedia.org/wiki/Henry_David_Thoreau">Henry David
Thoreau</a> ou encore
de <a href="https://en.wikipedia.org/wiki/Chris_McCandless">Chris McCandless</a>
; un rejet de la société humaine.</p>
<p>Anectote amusante, l'ours qui dévora Thimothy portait le tatouage
n°141. Le bus dans lequel Chris termina ses jours portait le n°142.</p>
<p>Une cassette a enregistré les derniers moments de Tim et de sa
compagne. En ayant écouté une partie Herzog prétendra n'avoir jamais
entendu rien de plus effroyable.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Grizzly Man — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/people/philip-ulrichPhilip Ulrich2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-24</time></span>
<span id="people-birthyear" style="visibility:hidden;">1951</span>
<span id="people-deathyear" style="visibility:hidden;"></span>
<span id="people-profession" style="visibility:hidden;">Programmer</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Philip Ulrich</h1>
<ul>
<li><a href="https://philippeulrich.com/">Philip Ulrich</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Philip Ulrich — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/people/alvy-ray-smithAlvy Ray Smith2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-24</time></span>
<span id="people-birthyear" style="visibility:hidden;">1943</span>
<span id="people-deathyear" style="visibility:hidden;"></span>
<span id="people-profession" style="visibility:hidden;">Programmer</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Alvy Ray Smith</h1>
<ul>
<li><a href="http://alvyray.com/">Alvy Ray Smith</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Alvy Ray Smith — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/people/wendy-carlosWendy Carlos2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-24</time></span>
<span id="people-birthyear" style="visibility:hidden;">1939</span>
<span id="people-deathyear" style="visibility:hidden;"></span>
<span id="people-profession" style="visibility:hidden;">Musician</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Wendy Carlos</h1>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Wendy Carlos — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/movie/cedric-gruat_drole-de-guerreLa drôle de guerre2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-09-24</time></span>
<span id="book-year" style="visibility:hidden;">2018</span>
<span id="book-author" style="visibility:hidden;">Cédric Gruat</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>La drôle de guerre</h1>
<p>Le documentaire revient sur la “Drôle de Guerre” telle qu'elle fut
nommée assez tôt. Il tente le difficile exercice d'offir un regard
objectif en faisant abstraction de l'issue. Pour comprendre ce que
l'Histoire rend <em>incompréhensible</em>.</p>
<p>« Depuis pas mal de temps nous ne faisions que nous débattre dans un rêve. Nous révions la guerre et la paix. Jamais l'Histoire n'avait parut aussi loin de ceux qui la font. C'est cela le propre de notre éṕoque : d'avoir profondément désorganisé le réel. De nous avoir fait perdre notre confiance dans les choses et les êtres. Les machines s'en sont mêlées: la TSF, le cinéma, le téléphone... Toutes ces machines inventées pour nous soustraire au contact direct. C'est peut-être pour cette raison là qu'on est perdus dans l'évènement. Pour celle-là ou d'autres, d'ailleurs qu'est-ce que ça fait ? Est-ce que je vais me mettre moi aussi à inventer des esplications ? On y comprend rien, voilà tout. » <a href="https://fr.wikipedia.org/wiki/Georges_Hyvernaud">Georges Hyvernaud</a></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=La drôle de guerre — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/movie/marguerite-duras_india-songIndia Song2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-10-02</time></span>
<span id="book-year" style="visibility:hidden;">1975</span>
<span id="book-author" style="visibility:hidden;">Marguerite Duras</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>India Song</h1>
<p>Le vice consul des Indes <a href="https://fr.wikipedia.org/wiki/Michael_Lonsdale">Mickael
Lonsdale</a> succombe à
l'éblouissante <a href="https://fr.wikipedia.org/wiki/Delphine_Seyrig">Delphine
Seyrig</a>. Ce qui est
extraordinaire est que la réalité dépasse la fiction puisque
M. Lonsdale étaient réellement fou de D. Seyrig, amour non réciproque
qu'il n'oubliera jamais. Il restera seul toute sa vie.</p>
<p><a href="https://fr.wikipedia.org/wiki/Bruno_Nuytten">Bruno Nuytten</a> est le
directeur de la photographie. Il le fut pour “les valseuses” la
“la meilleure façon de marcher” ou encore “Manon des sources”.</p>
<p><img src="/images/india-song/green.webp" alt="Lahore c'est moi">
<img src="/images/india-song/breath.webp" alt="Lahore c'est moi">
<img src="/images/india-song/bike.webp" alt="Lahore c'est moi">
<img src="/images/india-song/smile.webp" alt="Lahore c'est moi">
<img src="/images/india-song/sea.webp" alt="Lahore c'est moi"></p>
<pre><code class="language-text">Il s'est produit un déchirement de l'air. Sa jupe contre les arbres. Elle m'a regardé:
- je ne savais pas que vous existiez. Calcutta est devenu pour moi une forme de l'espoir.
- j'aime Mickael Richardson. Je ne suis pas libre de cet amour.
- je le sais. je vous aime ainsi, dans l'amour de Mickael Richardson. Ça ne m'importe pas. Je parle faux. Vous entendez ma voix ? Elle leur fait peur.
- oui
- de qui est elle ? J'ai tiré sur moi à Lahore sans en mourir. Les autres me séparent de Lahore. Je ne m'en sépare pas. Lahore c'est moi. Vous comprenez aussi ?
- oui, ne criez pas.
- oui. Vous êtes avec moi devant Lahore. Je le sais. Vous êtes en moi. Je vous emmenerai en moi. Et vous tirerez avec moi sur les lépreux de Shalimar. Qu'y pouvez vous ? Je n'avais pas besoin de vous inviter à danser pour vous connaître. Et vous le savez.
- je le sais.
- il est tout à fait inutile qu'on aille plus loin, vous et moi. Nous n'avons rien à nous dire. Nous sommes les mêmes.
- je crois ce que vous venez de dire.
- les histoires d'amour vous les vivez avec d'autres. Nous n'avons pas besoin de ça. Je voulais connaître l'odeur de vos cheveux. C'est ce qui vous explique que j... après la réception, vous restez entre intimes. je voudrais rester avec vous une fois.
- vous n'avez aucune chance.
- il me chasseraient.
- oui, vous êtes quelqu'un qu'il leur faut oublier.
- comme Lahore?
- oui
- que vais-je devenir ?
- vous serez nommé loin de Calcutta.
- c'est ce que vous, vous désirez ?
- oui
- bon. et quand cela va t'il finir ?
- avec votre mort je crois
- quel est ce mal. le miens ?
- l'intelligence
- de vous ? je vais leur demander qu'ils me gardent ici ce soir.
- faites comme vous voudrez
- pour que quelque chose ait lieu entre vous et moi. un incident public. je ne sais que crier. et qu'ils le sachent au moins qu'on peut crier un amour. je le sais ils seront mal à l'aise. et puis ils recommenceront à parler. je sais même que vous ne direz à personne que vous étié d'accord.
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=India Song — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/reading/cyril-collard_les-nuits-fauvesLes nuits fauves2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-10-11</time></span>
<span id="book-year" style="visibility:hidden;">1989</span>
<span id="book-author" style="visibility:hidden;">Cyril Collard</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<p><span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></p>
<h1>Les nuits fauves</h1>
<p>Second roman de <a href="https://fr.wikipedia.org/wiki/Cyril_Collard">Cyril
Collard</a>, il fut adapté
en film et joué par l'auteur, primé aux César en 1993 quelques jours
après son décès du VIH. <a href="https://www.grignoux.be/dossiers/035/">Critique en ligne du
film</a></p>
<p>L'écriture est puissante, tranchante et foudroyante.</p>
<p>Le récit est ancré à Paris, avec des teintes du Maghreb. Dans un
contexte post guerre d'Algérie, de skinhead, toxicos, homosexuels,
sado-maso, travestis... entre les risques, l'innocence, les mensonges
et le poids de la naissance des tares héritées, les personnages se
croisent, se frôlent et se percutent.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Les nuits fauves — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/blog/postgresql-versus-greenplumPostgresql Versus Greenplum2021-11-07T11:53:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2020-12-04</time></span>
<span id="reading-time" style="visibility:hidden;">4 minutes</span></p>
<h1>Postgresql Versus Greenplum</h1>
<h2>Server configuration</h2>
<ul>
<li>a dataset of 30M/60M and 100M rows</li>
<li>16GO RAM</li>
<li>500GO SSD</li>
<li>8 core</li>
</ul>
<h2>Postgresql</h2>
<ul>
<li>
<p>version 13.1</p>
</li>
<li>
<p>madlib 1.17</p>
</li>
<li>
<p>configured with pgtune</p>
<pre><code>max_connections = 20
shared_buffers = 4GB
effective_cache_size = 12GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 500
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 26214kB
min_wal_size = 4GB
max_wal_size = 16GB
max_worker_processes = 8
max_parallel_workers_per_gather = 4
max_parallel_workers = 8
max_parallel_maintenance_workers = 4
</code></pre>
</li>
</ul>
<h2>Greenplum</h2>
<ul>
<li>version 6.12</li>
<li>madlib 1.17</li>
<li>4 segments</li>
<li>default configurations</li>
</ul>
<h2>Overall results</h2>
<div>
<img src="/images/pg_vs_gp2.webp" height="700">
</div>
<h2>conclusion</h2>
<p>From this preliminary results, Greeplum is superior to postgreSQL on a
standalone server, which is very surprining because it is supposed to
work on a distrinbuted cluster.</p>
<p>Greenplum is way faster than postgres in the context of datawarhousing
and managing large tables in every domains:
insert/update/delete/select. It provides a set of tools such gptext,
madlib, map-reduce, pl/python-R and PFX which looks relevant. Also it
provides 'queues' to makes multi-users collaborate better.</p>
<p>While postgreSQL is effective on a 30M rows table, it struggles to
handle larger tables.</p>
<h2>SQL</h2>
<h3>Postgresql</h3>
<pre><code class="language-sql"><span class="hl kwa">create table</span> <span class="hl kwd">labevents</span><span class="hl opt">(</span>
row_id <span class="hl kwb">integer</span>
<span class="hl opt">,</span> subject_id <span class="hl kwb">integer</span>
<span class="hl opt">,</span> hadm_id <span class="hl kwb">integer</span>
<span class="hl opt">,</span> itemid <span class="hl kwb">integer</span>
<span class="hl opt">,</span> charttime <span class="hl kwa">timestamp</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">)</span> <span class="hl kwa">without time zone</span>
<span class="hl opt">,</span> <span class="hl kwa">value</span> <span class="hl kwb">character</span> <span class="hl kwa">varying</span><span class="hl opt">(</span><span class="hl num">200</span><span class="hl opt">)</span>
<span class="hl opt">,</span> valuenum <span class="hl kwa">double precision</span>
<span class="hl opt">,</span> valueuom <span class="hl kwb">character</span> <span class="hl kwa">varying</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">)</span>
<span class="hl opt">,</span> <span class="hl kwa">flag</span> <span class="hl kwb">character</span> <span class="hl kwa">varying</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">)</span>
<span class="hl opt">,</span> mimic_id <span class="hl kwb">integer</span>
<span class="hl opt">) ;</span>
<span class="hl kwa">CREATE TABLE</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># \<span class="hl kwa">copy</span> labevents <span class="hl kwa">from</span> <span class="hl sng">'labevents.csv'</span> <span class="hl kwa">csv</span> <span class="hl opt">;</span>
<span class="hl kwa">COPY</span> <span class="hl num">27854055</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">353750.640</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">05</span><span class="hl kwc">:53</span>.751<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">18009889</span> |
<span class="hl num">9779594</span> | abnormal
<span class="hl num">64572</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">939.223</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select</span> c1.row_id<span class="hl opt">,</span> <span class="hl kwa">row_number</span><span class="hl opt">()</span> <span class="hl kwa">over</span><span class="hl opt">(</span><span class="hl kwa">partition by</span> c1.itemid <span class="hl kwa">order by</span> c1.row_id <span class="hl kwa">desc</span><span class="hl opt">)</span> rk <span class="hl kwa">from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">limit</span> <span class="hl num">10</span><span class="hl opt">;</span>
row_id | rk
<span class="hl slc">----------+----</span>
<span class="hl num">27906665</span> | <span class="hl num">1</span>
<span class="hl num">27906621</span> | <span class="hl num">2</span>
<span class="hl num">27906612</span> | <span class="hl num">3</span>
<span class="hl num">27906561</span> | <span class="hl num">4</span>
<span class="hl num">27906550</span> | <span class="hl num">5</span>
<span class="hl num">27906507</span> | <span class="hl num">6</span>
<span class="hl num">27906370</span> | <span class="hl num">7</span>
<span class="hl num">27906325</span> | <span class="hl num">8</span>
<span class="hl num">27906270</span> | <span class="hl num">9</span>
<span class="hl num">27906257</span> | <span class="hl num">10</span>
<span class="hl opt">(</span><span class="hl num">10</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">13714.732</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">00</span><span class="hl kwc">:13</span>.715<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">insert into</span> labevents <span class="hl kwa">select</span> <span class="hl opt">*</span> <span class="hl kwa">from</span> labevents<span class="hl opt">;</span>
<span class="hl kwa">INSERT</span> <span class="hl num">0 27854055</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">423631.155</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">07</span><span class="hl kwc">:03</span>.631<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">36019778</span> |
<span class="hl num">19559188</span> | abnormal
<span class="hl num">129144</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">5295.608</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">00</span><span class="hl kwc">:05</span>.296<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select</span> c1.row_id<span class="hl opt">,</span> <span class="hl kwa">row_number</span><span class="hl opt">()</span> <span class="hl kwa">over</span><span class="hl opt">(</span><span class="hl kwa">partition by</span> c1.itemid <span class="hl kwa">order by</span> c1.row_id <span class="hl kwa">desc</span><span class="hl opt">)</span> rk <span class="hl kwa">from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">limit</span> <span class="hl num">10</span><span class="hl opt">;</span>
row_id | rk
<span class="hl slc">----------+----</span>
<span class="hl num">27906665</span> | <span class="hl num">1</span>
<span class="hl num">27906665</span> | <span class="hl num">2</span>
<span class="hl num">27906621</span> | <span class="hl num">3</span>
<span class="hl num">27906621</span> | <span class="hl num">4</span>
<span class="hl num">27906612</span> | <span class="hl num">5</span>
<span class="hl num">27906612</span> | <span class="hl num">6</span>
<span class="hl num">27906561</span> | <span class="hl num">7</span>
<span class="hl num">27906561</span> | <span class="hl num">8</span>
<span class="hl num">27906550</span> | <span class="hl num">9</span>
<span class="hl num">27906550</span> | <span class="hl num">10</span>
<span class="hl opt">(</span><span class="hl num">10</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">126347.700</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">02</span><span class="hl kwc">:06</span>.348<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">join</span> labevents c2 <span class="hl kwa">on</span> c1.row_id <span class="hl opt">=</span> c2.row_id <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">72039556</span> |
<span class="hl num">39118376</span> | abnormal
<span class="hl num">258288</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">234570.663</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">03</span><span class="hl kwc">:54</span>.571<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">insert into</span> labevents <span class="hl kwa">select</span> <span class="hl opt">*</span> <span class="hl kwa">from</span> labevents<span class="hl opt">;</span>
<span class="hl kwa">INSERT</span> <span class="hl num">0 55708110</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">1201961.089</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">20</span><span class="hl kwc">:01</span>.961<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">72039556</span> |
<span class="hl num">39118376</span> | abnormal
<span class="hl num">258288</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">7277.858</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">00</span><span class="hl kwc">:07</span>.278<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select</span> c1.row_id<span class="hl opt">,</span> <span class="hl kwa">row_number</span><span class="hl opt">()</span> <span class="hl kwa">over</span><span class="hl opt">(</span><span class="hl kwa">partition by</span> c1.itemid <span class="hl kwa">order by</span> c1.row_id <span class="hl kwa">desc</span><span class="hl opt">)</span> rk <span class="hl kwa">from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">limit</span> <span class="hl num">10</span><span class="hl opt">;</span>
row_id | rk
<span class="hl slc">----------+----</span>
<span class="hl num">27906665</span> | <span class="hl num">1</span>
<span class="hl num">27906665</span> | <span class="hl num">2</span>
<span class="hl num">27906665</span> | <span class="hl num">3</span>
<span class="hl num">27906665</span> | <span class="hl num">4</span>
<span class="hl num">27906621</span> | <span class="hl num">5</span>
<span class="hl num">27906621</span> | <span class="hl num">6</span>
<span class="hl num">27906621</span> | <span class="hl num">7</span>
<span class="hl num">27906621</span> | <span class="hl num">8</span>
<span class="hl num">27906612</span> | <span class="hl num">9</span>
<span class="hl num">27906612</span> | <span class="hl num">10</span>
<span class="hl opt">(</span><span class="hl num">10</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">465615.775</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">07</span><span class="hl kwc">:45</span>.616<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">join</span> labevents c2 <span class="hl kwa">on</span> c1.row_id <span class="hl opt">=</span> c2.row_id <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">-----------+----------</span>
<span class="hl num">288158224</span> |
<span class="hl num">156473504</span> | abnormal
<span class="hl num">1033152</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">494615.953</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">08</span><span class="hl kwc">:14</span>.616<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">delete from</span> labevents <span class="hl kwa">where</span> row_id <span class="hl opt"><</span> <span class="hl num">5000000</span><span class="hl opt">;</span>
<span class="hl kwa">DELETE</span> <span class="hl num">19952316</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">157689.975</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">02</span><span class="hl kwc">:37</span>.690<span class="hl opt">)</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">update</span> labevents <span class="hl kwa">set flag</span> <span class="hl opt">=</span> <span class="hl sng">'new flag'</span> <span class="hl kwa">where</span> row_id <span class="hl opt"><</span> <span class="hl num">10000000</span><span class="hl opt">;</span>
<span class="hl kwa">UPDATE</span> <span class="hl num">19949040</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">810058.857</span> <span class="hl kwd">ms</span> <span class="hl opt">(</span><span class="hl num">13</span><span class="hl kwc">:30</span>.059<span class="hl opt">)</span>
</code></pre>
<pre><code>create unlogged table tbl_range (data1 text, data2 text, randomnumber double precision);
insert into tbl_range
( select
('{a,b,c,d,e,f}'::text[])[1 + floor(random() * 6)] as data1
,('{g,h,i,j,k,l}'::text[])[1 + floor(random() * 6)] as data2
,random() * 1000 as randomnumber
from generate_series(1,400000000));
SELECT madlib.pivot('tbl_range', 'pivout', 'data1', 'data2', 'randomnumber', 'sum,max');
</code></pre>
<h3>Greenplum</h3>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">create table</span> <span class="hl kwd">labevents</span><span class="hl opt">(</span>
row_id <span class="hl kwb">integer</span>
<span class="hl opt">,</span>subject_id <span class="hl kwb">integer</span>
<span class="hl opt">,</span>hadm_id <span class="hl kwb">integer</span>
<span class="hl opt">,</span>itemid <span class="hl kwb">integer</span>
<span class="hl opt">,</span>charttime <span class="hl kwa">timestamp</span><span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">)</span> <span class="hl kwa">without time zone</span>
<span class="hl opt">,</span><span class="hl kwa">value</span> <span class="hl kwb">character</span> <span class="hl kwa">varying</span><span class="hl opt">(</span><span class="hl num">200</span><span class="hl opt">)</span>
<span class="hl opt">,</span>valuenum <span class="hl kwa">double precision</span>
<span class="hl opt">,</span>valueuom <span class="hl kwb">character</span> <span class="hl kwa">varying</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">)</span>
<span class="hl opt">,</span><span class="hl kwa">flag</span> <span class="hl kwb">character</span> <span class="hl kwa">varying</span><span class="hl opt">(</span><span class="hl num">20</span><span class="hl opt">)</span>
<span class="hl opt">,</span>mimic_id <span class="hl kwb">integer</span>
<span class="hl opt">)</span> <span class="hl kwa">with</span> <span class="hl opt">(</span>appendonly<span class="hl opt">=</span><span class="hl kwa">true</span><span class="hl opt">,</span> orientation<span class="hl opt">=</span><span class="hl kwa">column</span><span class="hl opt">,</span> compresstype<span class="hl opt">=</span>ZSTD <span class="hl opt">);</span>
NOTICE<span class="hl opt">:</span> <span class="hl kwa">Table</span> doesn<span class="hl sng">'t have '</span>DISTRIBUTED <span class="hl kwa">BY</span><span class="hl sng">' clause -- Using column named '</span>row_id<span class="hl sng">' as the Greenplum Database data distribution key for this table.</span>
<span class="hl sng">HINT: The '</span>DISTRIBUTED <span class="hl kwa">BY</span><span class="hl sng">' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.</span>
<span class="hl sng">CREATE TABLE</span>
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># \<span class="hl kwa">copy</span> labevents <span class="hl kwa">from</span> <span class="hl sng">'/home/nparis/labevents.csv'</span> <span class="hl kwa">csv</span> <span class="hl opt">;</span>
<span class="hl kwa">COPY</span> <span class="hl num">27854055</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">80878</span><span class="hl opt">,</span><span class="hl num">637</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">18009889</span> |
<span class="hl num">9779594</span> | abnormal
<span class="hl num">64572</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">1164</span><span class="hl opt">,</span><span class="hl num">303</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select</span> c1.row_id<span class="hl opt">,</span> <span class="hl kwa">row_number</span><span class="hl opt">()</span> <span class="hl kwa">over</span><span class="hl opt">(</span><span class="hl kwa">partition by</span> c1.itemid <span class="hl kwa">order by</span> c1.row_id <span class="hl kwa">desc</span><span class="hl opt">)</span> rk <span class="hl kwa">from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">limit</span> <span class="hl num">10</span><span class="hl opt">;</span>
row_id | rk
<span class="hl slc">----------+----</span>
<span class="hl num">27907576</span> | <span class="hl num">1</span>
<span class="hl num">27907382</span> | <span class="hl num">2</span>
<span class="hl num">27901097</span> | <span class="hl num">3</span>
<span class="hl num">27901054</span> | <span class="hl num">4</span>
<span class="hl num">27900783</span> | <span class="hl num">5</span>
<span class="hl num">27897288</span> | <span class="hl num">6</span>
<span class="hl num">27896669</span> | <span class="hl num">7</span>
<span class="hl num">27894567</span> | <span class="hl num">8</span>
<span class="hl num">27892281</span> | <span class="hl num">9</span>
<span class="hl num">27884982</span> | <span class="hl num">10</span>
<span class="hl opt">(</span><span class="hl num">10</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">12569</span><span class="hl opt">,</span><span class="hl num">391</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">insert into</span> labevents <span class="hl kwa">select</span> <span class="hl opt">*</span> <span class="hl kwa">from</span> labevents<span class="hl opt">;</span>
<span class="hl kwa">INSERT</span> <span class="hl num">0 27854055</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">79361</span><span class="hl opt">,</span><span class="hl num">173</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">36019778</span> |
<span class="hl num">19559188</span> | abnormal
<span class="hl num">129144</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">2353</span><span class="hl opt">,</span><span class="hl num">157</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select</span> c1.row_id<span class="hl opt">,</span> <span class="hl kwa">row_number</span><span class="hl opt">()</span> <span class="hl kwa">over</span><span class="hl opt">(</span><span class="hl kwa">partition by</span> c1.itemid <span class="hl kwa">order by</span> c1.row_id <span class="hl kwa">desc</span><span class="hl opt">)</span> rk <span class="hl kwa">from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">limit</span> <span class="hl num">10</span><span class="hl opt">;</span>
row_id | rk
<span class="hl slc">----------+----</span>
<span class="hl num">27906665</span> | <span class="hl num">1</span>
<span class="hl num">27906665</span> | <span class="hl num">2</span>
<span class="hl num">27906621</span> | <span class="hl num">3</span>
<span class="hl num">27906621</span> | <span class="hl num">4</span>
<span class="hl num">27906612</span> | <span class="hl num">5</span>
<span class="hl num">27906612</span> | <span class="hl num">6</span>
<span class="hl num">27906561</span> | <span class="hl num">7</span>
<span class="hl num">27906561</span> | <span class="hl num">8</span>
<span class="hl num">27906550</span> | <span class="hl num">9</span>
<span class="hl num">27906550</span> | <span class="hl num">10</span>
<span class="hl opt">(</span><span class="hl num">10</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">25409</span><span class="hl opt">,</span><span class="hl num">642</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">join</span> labevents c2 <span class="hl kwa">on</span> c1.row_id <span class="hl opt">=</span> c2.row_id <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">72039556</span> |
<span class="hl num">39118376</span> | abnormal
<span class="hl num">258288</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">101654</span><span class="hl opt">,</span><span class="hl num">716</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">insert into</span> labevents <span class="hl kwa">select</span> <span class="hl opt">*</span> <span class="hl kwa">from</span> labevents<span class="hl opt">;</span>
<span class="hl kwa">INSERT</span> <span class="hl num">0 55708110</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">112203</span><span class="hl opt">,</span><span class="hl num">521</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">----------+----------</span>
<span class="hl num">72039556</span> |
<span class="hl num">39118376</span> | abnormal
<span class="hl num">258288</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">3941</span><span class="hl opt">,</span><span class="hl num">551</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select</span> c1.row_id<span class="hl opt">,</span> <span class="hl kwa">row_number</span><span class="hl opt">()</span> <span class="hl kwa">over</span><span class="hl opt">(</span><span class="hl kwa">partition by</span> c1.itemid <span class="hl kwa">order by</span> c1.row_id <span class="hl kwa">desc</span><span class="hl opt">)</span> rk <span class="hl kwa">from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">limit</span> <span class="hl num">10</span><span class="hl opt">;</span>
row_id | rk
<span class="hl slc">----------+----</span>
<span class="hl num">27906665</span> | <span class="hl num">1</span>
<span class="hl num">27906665</span> | <span class="hl num">2</span>
<span class="hl num">27906665</span> | <span class="hl num">3</span>
<span class="hl num">27906665</span> | <span class="hl num">4</span>
<span class="hl num">27906621</span> | <span class="hl num">5</span>
<span class="hl num">27906621</span> | <span class="hl num">6</span>
<span class="hl num">27906621</span> | <span class="hl num">7</span>
<span class="hl num">27906621</span> | <span class="hl num">8</span>
<span class="hl num">27906612</span> | <span class="hl num">9</span>
<span class="hl num">27906612</span> | <span class="hl num">10</span>
<span class="hl opt">(</span><span class="hl num">10</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">65279</span><span class="hl opt">,</span><span class="hl num">232</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">select count</span><span class="hl opt">(*)</span> c<span class="hl opt">,</span> c1.<span class="hl kwa">flag from</span> labevents <span class="hl kwa">as</span> c1 <span class="hl kwa">join</span> labevents c2 <span class="hl kwa">on</span> c1.row_id <span class="hl opt">=</span> c2.row_id <span class="hl kwa">group by</span> c1.<span class="hl kwa">flag order by</span> c <span class="hl kwa">desc limit</span> <span class="hl num">10</span> <span class="hl opt">;</span>
c | <span class="hl kwa">flag</span>
<span class="hl slc">-----------+----------</span>
<span class="hl num">288158224</span> |
<span class="hl num">156473504</span> | abnormal
<span class="hl num">1033152</span> | delta
<span class="hl opt">(</span><span class="hl num">3</span> <span class="hl kwa">rows</span><span class="hl opt">)</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">188816</span><span class="hl opt">,</span><span class="hl num">067</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">delete from</span> labevents <span class="hl kwa">where</span> row_id <span class="hl opt"><</span> <span class="hl num">5000000</span><span class="hl opt">;</span>
<span class="hl kwa">DELETE</span> <span class="hl num">19952316</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">8930</span><span class="hl opt">,</span><span class="hl num">371</span> ms
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">update</span> labevents <span class="hl kwa">set flag</span> <span class="hl opt">=</span> <span class="hl sng">'new flag'</span> <span class="hl kwa">where</span> row_id <span class="hl opt"><</span> <span class="hl num">10000000</span><span class="hl opt">;</span>
<span class="hl kwa">UPDATE</span> <span class="hl num">19949040</span>
<span class="hl kwa">Time</span><span class="hl opt">:</span> <span class="hl num">32977</span><span class="hl opt">,</span><span class="hl num">965</span> ms
</code></pre>
<pre><code class="language-sql">nparis@gnubuntu<span class="hl opt">:</span>~<span class="hl opt">/</span>bin$ <span class="hl opt">/</span>opt<span class="hl opt">/</span>greenplum<span class="hl opt">-</span><span class="hl kwa">db</span><span class="hl opt">-</span><span class="hl num">6</span><span class="hl opt">-</span><span class="hl num">6.12.1</span><span class="hl opt">/</span>bin<span class="hl opt">/</span>gpload <span class="hl opt">-</span>f gpload.yaml
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:16:38</span>|INFO|gpload <span class="hl kwa">session</span> started <span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:16:38</span>
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:16:38</span>|INFO|started gpfdist <span class="hl opt">-</span>p <span class="hl num">8081</span> <span class="hl opt">-</span>P <span class="hl num">8082</span> <span class="hl opt">-</span>f <span class="hl sng">"/home/nparis/labevents.csv"</span> <span class="hl opt">-</span>t <span class="hl num">30</span>
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:17:49</span>|INFO|running <span class="hl kwa">time</span><span class="hl opt">:</span> <span class="hl num">71.18</span> seconds
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:17:49</span>|INFO|<span class="hl kwa">rows</span> Inserted <span class="hl opt">=</span> <span class="hl num">27854055</span>
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:17:49</span>|INFO|<span class="hl kwa">rows</span> Updated <span class="hl opt">=</span> <span class="hl num">0</span>
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:17:49</span>|INFO|<span class="hl kwa">data</span> formatting errors <span class="hl opt">=</span> <span class="hl num">0</span>
<span class="hl num">2020</span><span class="hl opt">-</span><span class="hl num">12</span><span class="hl opt">-</span><span class="hl num">04 18</span><span class="hl kwc">:17:49</span>|INFO|gpload succeeded
</code></pre>
<pre><code class="language-sql">warehouse<span class="hl opt">=</span># <span class="hl kwa">create table</span> <span class="hl kwd">tbl_range</span> <span class="hl opt">(</span>data1 <span class="hl kwb">text</span><span class="hl opt">,</span> data2 <span class="hl kwb">text</span><span class="hl opt">,</span> randomnumber <span class="hl kwa">double precision</span><span class="hl opt">)</span> <span class="hl kwa">with</span> <span class="hl opt">(</span>appendonly<span class="hl opt">=</span><span class="hl kwa">true</span><span class="hl opt">,</span> orientation<span class="hl opt">=</span><span class="hl kwa">column</span><span class="hl opt">,</span> compresstype<span class="hl opt">=</span>ZLIB <span class="hl opt">);</span>
NOTICE<span class="hl opt">:</span> <span class="hl kwa">Table</span> doesn<span class="hl sng">'t have '</span>DISTRIBUTED <span class="hl kwa">BY</span><span class="hl sng">' clause -- Using column named '</span>data1<span class="hl sng">' as the Greenplum Database data distribution key for this table.</span>
<span class="hl sng">HINT: The '</span>DISTRIBUTED <span class="hl kwa">BY</span><span class="hl sng">' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.</span>
<span class="hl sng">CREATE TABLE</span>
<span class="hl sng">Time: 352,343 ms</span>
<span class="hl sng">warehouse=# insert into tbl_range</span>
<span class="hl sng">( select</span>
<span class="hl sng"> ('</span><span class="hl opt">{</span>a<span class="hl opt">,</span>b<span class="hl opt">,</span>c<span class="hl opt">,</span>d<span class="hl opt">,</span>e<span class="hl opt">,</span>f<span class="hl opt">}</span><span class="hl sng">'::text[])[1 + floor(random() * 6)] as data1</span>
<span class="hl sng"> ,('</span><span class="hl opt">{</span>g<span class="hl opt">,</span>h<span class="hl opt">,</span>i<span class="hl opt">,</span>j<span class="hl opt">,</span>k<span class="hl opt">,</span>l<span class="hl opt">}</span><span class="hl sng">'::text[])[1 + floor(random() * 6)] as data2</span>
<span class="hl sng"> ,random() * 1000 as randomnumber</span>
<span class="hl sng">from generate_series(1,100000000)) ;</span>
<span class="hl sng">INSERT 0 100000000</span>
<span class="hl sng">Time: 145408,190 ms</span>
<span class="hl sng">warehouse=# SELECT madlib.pivot('</span>tbl_range<span class="hl sng">', '</span>pivout<span class="hl sng">', '</span>data1<span class="hl sng">', '</span>data2<span class="hl sng">', '</span>randomnumber<span class="hl sng">', '</span><span class="hl kwa">sum</span><span class="hl opt">,</span><span class="hl kwa">max</span><span class="hl sng">');</span>
<span class="hl sng"> pivot</span>
<span class="hl sng">-------</span>
<span class="hl sng"></span>
<span class="hl sng">(1 row)</span>
<span class="hl sng"></span>
<span class="hl sng">Time: 67074,359 ms</span>
</code></pre>
<pre><code>create table tbl_range (data1 text, data2 text, randomnumber double precision) with (appendonly=true, orientation=column, compresstype=ZLIB );
insert into tbl_range
( select
('{a,b,c,d,e,f}'::text[])[1 + floor(random() * 6)] as data1
,('{g,h,i,j,k,l}'::text[])[1 + floor(random() * 6)] as data2
,random() * 1000 as randomnumber
from generate_series(1,400000000));
SELECT madlib.pivot('tbl_range', 'pivout', 'data1', 'data2', 'randomnumber', 'sum,max');
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Postgresql Versus Greenplum — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-11-07 11:53</time> </div>https://blog.parisni.com/blog/an-archeology-mobile-applicationAn archeology mobile application2021-03-28T22:16:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-03-28</time></span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>An archeology mobile application</h1>
<h2>There is a tool I need</h2>
<p>I regularly find ancient artefacts (mostly prehistoric silex). I would
find highly valuable to store the GPS coordinate of those, and get a
map.</p>
<p>I know this practice is subject to regulation. However, most of the
time, I found the stuff in ploughed fields or in the forest trails. My
guess is having the tool to store and share the information could help
the research by building an open-database.</p>
<p>Such a tool could be free software, accesible to anybody. Some feature
will allow collaboration (annotation) and also contact between
users. The dataset (including photos) will be released as open-data.</p>
<h2>Features</h2>
<ul>
<li>create an account</li>
<li>upload a geologalized photo</li>
<li>show artifacts on a map
<ul>
<li>filter by category</li>
<li>filter by user</li>
</ul>
</li>
<li>add metadata</li>
<li>suggest metadata</li>
<li>export open-data</li>
</ul>
<h2>Implemention</h2>
<ul>
<li>python API (fastapi)</li>
<li>postgresql database, postgis</li>
<li>object storage (store the images)</li>
<li>imagemagic tools</li>
<li>android client application</li>
<li>website client application</li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=An archeology mobile application — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-03-28 22:16</time> </div>https://blog.parisni.com/blog/self-hosted-mailSelf hosting emails in the 21th centory2021-06-13T16:43:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-06-13</time></span>
<span id="reading-time" style="visibility:hidden;">2 minutes</span></p>
<h1>Self hosting emails in the 21th centory</h1>
<h2>Why self hosting emails</h2>
<p>Mailing technology still active after decades of production. It's one
of the simpliest and universal way of sharing information. There is
out there a lot of very efficient solutions such gmail or
protonmail.</p>
<p>But mail is FUN with a very broad spectrum of technology to
master. For example, mailu is built over containers and targets
kubernetes, it has a flask api/cli, is maintained by a prolific
community using great automation tools on the continuous delivery and
also bots on matrix. It has a machine learning based spam filter.</p>
<p>Also mail is serious with personal informations, that shall be kept
secret. Hacking ofter takes place on the mailing area, and
understanding how this stuff works is a great skill.</p>
<h2>Available mailing selfhosted solutions</h2>
<p>Mail In A Box (aka MIAB) looks the most advanced solution.</p>
<p>Mailu is quite new and it naturally comes with less features. One
major distinction compared to MIAB is its configurability. Most of the
modules are optional or sereval choices availables.</p>
<h2>Ham & Spam</h2>
<p>The <code>smtp port 25</code> shall be opened. However most internet box close it
by defaul. It is often possible to open it from the box web ui.</p>
<p>The <code>reverse dns</code>is a huge obstacle to self host email. The domain
name associated with the home ip server has to be the same as the mail
domain name. Sadly most internet providers does not allow clients to
change the value of rDNS (which is trivial on the cloud).</p>
<h2>Using a relay</h2>
<p>Because sending trusted email from a home server is so tricky one can
use a mail provider (such riseup) to deliver the email. Also the
provider can route the email to your selfhosted server. In this case,
you still use your existing email, but nothing is stored on the
provider side but on your own instance.</p>
<p>The main advantage with this is you manage your storage yourself. Also
in term of privacy, your provider does not store your email
history. By the way the email only transit on his side.</p>
<h2>Email client</h2>
<p>I have been using in the past multiple cli email clients such <code>mutt</code>,
<code>neomutt</code> and <code>mu4e</code> together with local indexing tools such <code>notmuch</code>
or <code>mu</code>. Those tools are not designed to work with imap but with
pop. This means the emails are pulled locally and indexed.</p>
<p>Having a self-hosted email solution such <code>mailu</code> or <code>mail in a box</code>
let use the dovecot full text searching feature based on xapian or
solr. Together with the imap protocol, it leverages server search with
great performances. Moreover imap is far better than pop because all
your client will have the same view on the maildir, and there is no
need to store locally the emails which improves security.</p>
<p>In the terminal, <code>aerc</code> is a great cli email client focused on imap
and it supports server side FTS (<code>filter -a</code>).</p>
<p>On desktop, evolution also is a great software, and it provides imap
push in order to get real time notifications and decrease server load.</p>
<p>On mobile, <code>k9mail</code> is great, and provides draft email encryption,
autocrypt. Sadly the FTS is not yet supported. I would be glad to help
on this. I have also tested fairemail, but most interesting features
are limited to the paid version.</p>
<h2>Calendar, contacts and tasks</h2>
<p>Having our contacts and calendar together with the email is
important. Mailu provides a simple python package <code>radicale</code> which
brings this features. It does not come with a visual calendar, but can
export vcal and ics formats. Mobile applications such <code>etar calendar</code>,
<code>tasks</code>, can connect to radicale and synchronize content thought <code>davx5</code>.</p>
<p>Mail-in-a-box comes with next-cloud calendar which is a more complete
solution but too heavy for my taste.</p>
<p>Roundcube webmail is able to connect to radicale carddav feature
only. Sadly it does not provide any decent calendar plugin.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Self hosting emails in the 21th centory — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-06-13 16:43</time> </div>https://blog.parisni.com/blog/python-coding-envRelax with python2021-08-18T22:32:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-07-15</time></span>
<span id="reading-time" style="visibility:hidden;">2 minutes</span></p>
<h1>Relax with python</h1>
<p>Most programmer knows python and like it. I have been working with it
a lot recently, and there is few important caveat to really have good
time.</p>
<h2>Forget vim or emacs: Use a tool called an IDE</h2>
<p>I have much affection for both vim and emacs and being currently
writing this post with the latter. I tried my best to setup a full
fledged python with it and I won't do it again. I lost myself in few
tutorials and when in the end I got a result it was far from the
feature a python IDE can offer.</p>
<p>Intellij IDE has a python plugin which turn it into Pycharm. Unlike
vim and emacs, they parse the python code with an AST, which provide
some powerful feature:</p>
<ul>
<li>a powerful python debugger !</li>
<li>find where is used the given object in the project</li>
<li>batch rename object in the project</li>
<li>refactor piece of code</li>
</ul>
<p>It also come with feature we like:</p>
<ul>
<li>an authentic vim key binding, including search/replace, tab navigation</li>
<li>an integrated git manager (as good as magit for emacs)</li>
<li>search for files and search within files</li>
<li>a nice dark mode</li>
<li>pytest integration</li>
<li>last but not least, it manages most programming languages out of the box</li>
</ul>
<h2>Manage your python versions with pyenv</h2>
<p>Managing multiple versions of python is not trivial, and there is many
ways for this. You can go with the packaged versions of the OS (such
apt), but it won't have new versions and you will end up adding weird
source list and repository. I did use anaconda which is a huge
reptilian mess.</p>
<p>Pyenv is really the silver bullet for this. What it does is provide a
command line for installing (download and compile) every python
version that exist including minors and beta. You can then choose for
each project the one you want, without ANY friction. Pyenv is not a
replacement with virtualenv at all, they actually combine: you build
venv with a given python version provided by pyenv.</p>
<h2>Manage you project environment with direnv</h2>
<p>Python is so popular, you end up with many
virtualenv. Activating/deactivating them becomes really painful. And
you will forget to do so, and compromise your pip environment easily.</p>
<p>direnv automates the management of virtualenvs when you navigate in
the linux projects, it activate / deactivate them accordingly. You
never have to do this again.</p>
<p>It has a pyenv binding and it's super easy to specify the python
version you d'like in your venv. This produce a virtualenv folder
called .direnv at the root of your project. In order to make intellij
not considering this folder's python sources as part of the project,
you simply have to "exclude" it with a right click.</p>
<h2>Format your code with black</h2>
<p>Python syntax rules are great, but it is not perfect. There is still
freedom in the way every one format there code. Black takes care of
everything. Its main advantage is it formats the same source code the
same way, independently of how it is formatted at the beginning.</p>
<h2>Commit your code with pre-commit</h2>
<p>There is many small details that everyone wants: format python,
markdown, xml, yaml... All this can be added BEFORE committing to git
with a git-hook. pre-commit takes care of all, and let you add those
step within a descriptive configuration.</p>
<h2>Test your code with tox</h2>
<p>When you test the code you don't only check the source code acts
correctly, you also test its environment works. Tox encapsulates the
tests in order they will work on your computer but also on the other's
one. With tox, you avoid the "it works on my laptop" syndrome.</p>
<h2>Build your changelog with Towncrier</h2>
<p>Maintaining a changelog is important and there is several tools out
there. Towncrier simplifies this process, and with few efforts, you
get a clear and maintained changelog.md file.</p>
<h2>Publish your package on pypi with twine</h2>
<p>Publishing on pypi can be straightforward when you use the right
tools. Both setuptools and twine are fine for this. You also have to
get several files present. <a href="https://pgjones.dev/blog/packaging-without-setup-py-2020">This
blog</a> is
helpful for this.</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Relax with python — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-08-18 22:32</time> </div>https://blog.parisni.com/reading/herbert_duneDune2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-09-04</time></span>
<span id="book-year" style="visibility:hidden;">1965</span>
<span id="book-author" style="visibility:hidden;">Frank Herbert</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★</span>
<span id="stars" style="visibility:hidden;">★</span></span></p>
<h1>Dune</h1>
<h1>Le livre</h1>
<pre><code>Il devrait exister une science de la contrariété. Les gens ont besoin
d'épreuves difficiles et d'oppression pour developper leurs muscles psychiques.
Extraits de Les Dits de Muad'Dib, par la Princesse Irulan.
</code></pre>
<h1>Le film</h1>
<h2>Le jeux vidéo</h2>
<p>En 1992, sort le jeu Dune, suivit de près par Dune 2. En réalité le premier
opus est porté par une fine équipe française qui porte le projet à bout de bras
de manière clandestine. On peut encore y jouer en récupérant l'iso du cd-rom,
et en installant dosbox sous linux.</p>
<p>C'est une expérience immerssive, notamment grâce à <a href="https://www.youtube.com/watch?v=FjHon6yg-r8">Spice
Opera</a>, la musique de <a href="https://en.wikipedia.org/wiki/St%C3%A9phane_Picq">Stéphane
Picq</a> et <a href="https://fr.wikipedia.org/wiki/Philippe_Ulrich">Philippe
Ulrich</a>. Ce dernier est à la
fois musicien, programmeur informatique, graphiste, scenariste, producteur.
Outre le jeux <a href="https://en.wikipedia.org/wiki/Captain_Blood_(video_game)">L’Arche du Capitaine
Blood</a> On lui doit
notamment ce couplé, dans <a href="https://philippeulrich.com/author/philippeulrich/page/2/">Un délicieux
Carnage</a>:</p>
<p><strong>Mégatonne</strong></p>
<pre><code>Les sanglots longs
Des électrons
De l’atome
Percent mon cœur
D’une chaleur
Mégatonne
Tout suffocant
Et blême quand
Sonne l’heure H
Je me souviens
Des jours anciens Et je FLASHE…
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Dune — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/people/neal-stephensonNeal Stephenson2021-12-03T21:09:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-10-13</time></span>
<span id="people-birthyear" style="visibility:hidden;">1959</span>
<span id="people-deathyear" style="visibility:hidden;"></span>
<span id="people-profession" style="visibility:hidden;">SF Writer</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Neal Stephenson</h1>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Neal_Stephenson">Neal Stephenson</a></li>
</ul>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Neal Stephenson — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:09</time> </div>https://blog.parisni.com/movie/jodorowsky_la-montagne-sacreeLa Montagne Sacrée2021-12-09T22:13:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-10-24</time></span>
<span id="book-year" style="visibility:hidden;">1973</span>
<span id="book-author" style="visibility:hidden;">Alejandro Jodorowsky</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>La Montagne Sacrée</h1>
<p><img src="/images/la-montagne-sacree/monk.webp" alt="Monk">
<img src="/images/la-montagne-sacree/monkey-monk.webp" alt="Monkey-Monk">
<img src="/images/la-montagne-sacree/god.webp" alt="God">
<img src="/images/la-montagne-sacree/inca.webp" alt="Inca">
<img src="/images/la-montagne-sacree/rabit.webp" alt="Rabit">
<img src="/images/la-montagne-sacree/frog.webp" alt="Frog">
<img src="/images/la-montagne-sacree/fox.webp" alt="Fox">
<img src="/images/la-montagne-sacree/barby.webp" alt="Barby">
<img src="/images/la-montagne-sacree/fire.webp" alt="Fire"></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=La Montagne Sacrée — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-09 22:13</time> </div>https://blog.parisni.com/reading/martin-ernstsen_faimFaim2021-12-03T21:22:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-11-07</time></span>
<span id="book-year" style="visibility:hidden;">2020</span>
<span id="book-author" style="visibility:hidden;">Martin Ernststen</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Faim</h1>
<blockquote>
<p>Dans mon plaisir de ce beau matin, je me chantais une chansonnette, on
sentait une vraie émotion dans ma voix et je me bouleversai aux larmes.</p>
</blockquote>
<center>
<p><img src="/images/gallery/faim/IMG_20211107_130514.webp" alt=""></p>
<p><img src="/images/gallery/faim/IMG_20211107_131110.webp" alt=""></p>
<p><img src="/images/gallery/faim/IMG_20211107_130927.webp" alt=""></p>
<p><img src="/images/gallery/faim/IMG_20211107_130821.webp" alt=""></p>
<p><img src="/images/gallery/faim/IMG_20211107_130715.webp" alt=""></p>
</center>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Faim — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-03 21:22</time> </div>https://blog.parisni.com/blog/demenagementAujourd'hui: je déménage demain2021-12-01T23:43:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2021-11-25</time></span>
<span id="reading-time" style="visibility:hidden;">6 minutes</span></p>
<h1>Aujourd'hui: je déménage demain</h1>
<p>Je déménage dans 3 mois, disons sur la planete Mars. Dans ce billet, je biffe les sociétés
en question, appelons les:</p>
<ul>
<li>FRIC: Une société fournissant téléphonie et internet</li>
<li>CPAFER: Une société fournissant téléphonie et internet, mais aussi
l'infrastructure réseau</li>
<li>Abrasif: Une société revendant de la téléphonie et internet pour le compte de
FRIC et CPAFER</li>
<li>SUM: Une société fournissant électricité</li>
<li>OEUF-DES-HAIES: Une société fournissant électricité</li>
<li>CanalPlouf: Une société de télévision</li>
<li>Clitoriche: Une société fournissant téléphonie et internet, partenaire de CanalPlouf</li>
<li>Le service des Empotés: Une administration responsable de la récolte des
finances</li>
<li>PapierQ: Cette société m'a été conseillée par l'agence immobilière, et se
présente comme un organisme financé par des acteurs divers, pour simplifier
les aspects administratifs des déménagements, en particulier l'energie,
l'internet et la téléphonie. De plus, ils ont des contrats avec des offres
exclusives</li>
<li>STOP-DIRECT: Mon ancienne banque: j'ai dû en changer pour obtenir un prêt
immobilier ailleurs.</li>
</ul>
<p>Dans cette aventure, je cherche à me doter d'internet, du téléphone fixe, et de
l'electricité de manière optimale. En quête secondaire, je tente de changer de
banque sans faire de vagues.</p>
<h2>JOUR 1</h2>
<h3>Clitoriche</h3>
<p>Le site internet de la planete Mars donne des détails sur la mise en place de la
fibre dans la ville ou j'emmenage dans 3 mois. Je commence par signaler au
webmaster un lien mort qui m'empeche de confirmer par moi même l'eligibilté de
mon domicile. Je vois cependant le contact de Clitoriche, société mettant en
œuvre la fibre dans ma ville.</p>
<p>Je les contacte donc avec pour objectif de vérifier mon éligibilité. J'indique
habiter dans un immeuble au numéro 666. La personne me répond que le logement ne
contient qu'un seul habitant et que cela doit être une erreur dans leur
fichier, et qu'une demande de rectification doit être faite avant de souscrire.
Je me frotte les mains avec le sentiment de gagner du temps en évitant des
complications à m'y prendre à l'avance.</p>
<h2>JOUR 2</h2>
<h3>Clitoriche</h3>
<p>Entre temps, en feuilletant les papiers OEUF-DES-HAIES, je réalise que j'habite au 666B.
Pour éviter d'engager une procédure inutile, je rapelle Clitoriche pour leur
signifier mon erreur. Après avoir expliqué le contexte, la personne au bout du
fil me demande mon code de souscription que j'ai dû recevoir par mail. Je suis
formel: je n'ai pas reçu de mail. Pour retrouver mon dossier, je donne mon
adresse email. Il ne lui est pas possible de changer mon addresse pour le 666B.
La personne me conseille alors de rappeler, et de demander le service de
souscription pour résilier mon compte en fournissant mon code. Je lui fais
remarquer que je n'ai justement pas ce code. La personne me demande de
patienter. Après 10 minutes d'attente, la ligne est coupée.</p>
<p>Je rappelle donc Clitoriche avec une stratégie différente: je demande à
souscrire un nouveau contrat. Je vais donc au bout de la démarche en précisant
le 666B cette fois, et je reçois enfin l'email, avec le code. C'est un forfait
fibre à 33€/mois, sans télévision, avec une téléphonie payante à la minute pour
les appels sortants: ce n'est pas donné mais après tout, ce sont les seuls à
fournir de la fibre ?</p>
<p>Je ne réalise pas à ce moment là que je ne souhaitais pas prendre de contrat si
tôt mais simplement vérifier mon éligibilité à la fibre.</p>
<h3>PapierQ</h3>
<p>Avec un sentiment d'accomplissement, j'enchaine avec PapierQ.</p>
<p>Je souhaite gérer l'electricité avec eux. Très cordial, mon interlocuteur me
conseille de prendre un contrat à taux fixe au plus vite étant donné
l'augmentation du prix de l'energie dûe à la crise du gaz avec les Russes. Il
me fait l'éloge de SUM énergie, et sa filiale Verte. Je refrenne mon envie de
tempérer son engouement vis à vis de SUM, et nous passons à la simulation.
Après vérification, le verdict tombe: le forfait que SUM me propose est de
160€ par mois. Je tombe de si haut, que je ne ressens rien tant le choc est
violent et sec. Nous tombons d'accord pour considérer une autre option, à
savoir payer en fonction de ce qui est consommé. Cela est plus raisonnable, 9€
pour le forfait, 5€ pour l'assurance 24/24 panne électrique, et le
kilowat/heure utilisé. Plutôt enjoué par la tournure des évenements, je lui
demande néammoins si l'assurance est optionnelle, ce qu'il me confirme avec une
comme une gène dans la voix. Je ne me gène pas pour lui dire que j'aimerais
autant ne pas prendre cette assurance.</p>
<p>Avant de se séparer bons amis, il me fait remarquer qu'il peut aussi me faire
une comparaison qui n'engage à rien sur les offres internet. Je suis déjà
engagé avec Clitoriche, mais finalement, pourquoi ne pas bénéficier de leurs bons
conseils voire de leurs offres spéciales ?</p>
<p>Lui n'est pas habilité sur la partie internet et il me passe son collègue pour
lequel il a une estime et une confiance sans borne. Rapidement celui-ci
m'oriente vers FRIC. Après plusieurs confirmations de sa part: au 666B, je suis
bien éligible. Il m'annonce que le forfait disponible est a 20€/mois pendant
1an puis 44€ - mais que je pourrai changer - avec plus de 200 chaines de télé
et la téléphonie complète. Je lui fais part de mon désintéret pour la télé,
mais il n'a pas d'autre forfait à me proposer. Fort bien, cela me fait tout de
même gagner près de 14€ par mois par rapport à Clitoriche. C'est une belle
victoire, et je me réjouis d'annoncer mes prouesses à ma femme quand elle
va rentrer.</p>
<p>Après avoir reçu de moi des explications liriques mais confuses et quelques
recherches de son côté, mon épouse me fait remarquer qu'un forfait FRIC semble
plus adapté et surtout moins cher que ce que j'ai souscrit. Immédiatement,
j'envoie un mail mesuré à PapierQ pour leur indiquer qu'un forfait plus
interessant que ce qu'ils m'ont fait souscrire semble exister, et que je leur
serais gré de me basculer dessus.</p>
<h2>JOUR 3</h2>
<h3>Empotés</h3>
<p>Je reçois un email de mon ancienne banque STOP-DIRECT qui m'indique que mon
solde est désormais de 0€ et que tout à été envoyé à ma nouvelle banque. Je
suis étonné car le courrier que j'avais reçu quelques jours avant m'indiquait
que je devais leur envoyer un RIB et des justificatifs pour qu'ils procèdent à
ce transfert. Mais soit, ce qui m'inquiète davantage est ma taxe d'habitation,
qui devait être débitée sur mon ancien compte. Ce n'est pas par choix, j'avais
fait le changement de coordonnées banquaires sur le site du service des
empotés, mais apparement cela n'a pas été pris en compte. Je tente donc de me
connecter aus site de STOP-DIRECT, mais mon compte à été gelé. Au téléphone, le
conseiller m'indique qu'en effet, un débit du service des Empotés est en cours
mais que celui-ci sera refusé, et que cela n'est pas aussi grâve que si il ne
pouvait pas être honoré.</p>
<p>Sur le site du service des Empotés, je cherche par tous les diables la manière
de changer de coordonnées banquaires, et je finis par constater que la prise en
compte des nouveau RIB se fait au moins un mois après son changement. Je les
contacte pour 1. me plaindre du fait de ne pas avoir été prévenu au moment de
changer de RIB de la lenteur de ce processus et 2. leur demander comment
procéder pour que le prélèvement soit effectué sur ma nouvelle banque.</p>
<p>Joie: je reçois une réponse dans la journée ! Je me rends sur le site pour
consulter ma messagerie sécurisée. En quelques mots laconiques on me confirme
"qu'en effet le changement de RIB prend un certain temps". Je dois me
concentrer avec énergie pour prendre la sage décision de poliement reformuler
ma question pour cette fois obtenir la marche à suivre pour qu'ils soient
payés, et que je ne devienne pas interdit banquaire.</p>
<h3>PapierQ</h3>
<p>Je reçois comme réponse que trop tard, si je veux changer de forfait, je dois
appeler un numéros FRIC (pour la modique somme de 0.35€/min).</p>
<h2>JOUR 5</h2>
<h3>Clitoriche</h3>
<p>Comme prévu, je contacte Clitoriche avec cette fois, mon numéros de souscription.
Je reçois d'abord des excuses car leurs techniciens on indiqué un problème
sur mon dossier. J'en viens à expliquer mes problèmes à savoir que 1.
l'option téléphone n'est pas suffisante 2. j'ai vu un contrat 40% moins cher
chez eux et 3. je souhaite débuter mes mensualités dans 3 mois.</p>
<p>Concernant l'option téléphone, si je débourse 5€ supplémentaires je pourrai
téléphoner sur les fixes en illimité. Si je débourse 4€ de plus par mois, c'est
aussi vers les mobiles que pourrais appeler sans limite. Cela nous s'élève
désormais à près de 45€/mois.</p>
<p>Concernant le contrat moins cher, j'ai précisé que c'est sur "le réseau CPAFER"
que cette offre s'appuie. En entendant le mot clé CPAFER, mon interlocuteur ne
cache plus sa gène et m'indique que Clitoriche n'est pas CPAFER mais une filiale de
CanalPlouf. Je précise que l'offre est visible en première page sur leur site.
Après vérification, je dois noter le numéro de Clitoriche-CPAFER, une autre filiale.</p>
<p>Concernant la question du début de mes mensualités, on me répond qu'il faut
attendre que mon compte soit débloqué, je pourrai les rappeler pour
avoir des détails. Je fais remarquer que c'est une question générale mais je
réalise que cette considération abstraite n'est pas à la porté de mon
interlocuteur, et je le remercie pour ses réponses précises.</p>
<p>Le site que j'ai sous les yeux propose une simulation d'éligibilité à la fibre.
Je tente ma chance et je vois que je suis éligible à Bouygues, FRIC et une
vente privée, chacun avec un numéro de téléphone. Je décroche mon téléphone
pour appeler le numéro indiquant FRIC. La messagerie automatique m'indique que
je suis sur le service: Abrasif. Un conseiller me répond, je commence par lui
faire part de mon désarois: je pensais appeler FRIC. On me répond qu'Abrasif est
affiliée à FRIC. J'explique mon déménagement, ma prochaine addresse. La
personne vérifie mon éligibilité et le verdicte tombe: Je ne suis pas éligible
à la fibre, encore moins celle de FRIC.</p>
<h3>Le service des Empotés</h3>
<p>Je suis notifié par mail qu'une réponse du service des Empotés m'a été faite.
Une ligne laconique m'indique en substance que je peux leur envoyer un chèque.
Cette réponse déclenche une série de questions: quel chéquier (je ne l'ai pas
encore reçu), quelle addresse, quel ordre, quelle guarantie que mon nouveau RIB
ne va pas être prélevé aussi ? Je pense que cette conversation ne fait que
commencer.</p>
<h3>FRIC</h3>
<p>Après avoir reçu un mail et un sms de FRIC m'engageant à prendre rendez-vous
avec un technicien pour l'installation de ma ligne, je décide de consulter mon
espace client sur le site internet. À nouveau, je constate que le processus est
figé à la première étape : vérification de l'éligibilité.</p>
<p>J'envisage alors de téléphoner à FRIC au numéro présent dans le mail: le 4242.
Très méfiant lorsque je lis "Gratuit depuis une ligne FRIC", je vérifie sur
plusieurs sites, et je suis convaincu: ce numéro est au prix d'un appel local,
c'est à dire gratuit dans le cadre de mon forfait mobile. En revanche, je
risque d'attendre très longtemps.</p>
<p>La messagerie m'indique mon numéro de client (déduit de mon numéros de mobile).
Un conseiller me répond instantanément. Je lui indique que 1. je me suis trompé
d'offre 2. j'ai un doute sur mon éligibilité fibre et 3. j'aimerais débuter mes
mensualités dans 3 mois.</p>
<p>Concernant mon offre, étant donné que rien n'a été déclenché de leur côté, il
me conseille de me rétracter et pour m'éviter d'avoir à rédiger une lettre, me
propose de le faire de son côté: voilà, c'est fait.</p>
<p>Pour l'éligibilité à la fibre, il vérifie et me confirme que c'est le cas. Je
lui fais part de ma réserve étant donné que le réseau CPAFER n'est pas présent, il
m'explique qu'il s'agit d'un autre réseau: planete Mars fibre qui est associatif,
probablement mis en œuvre par la région. En outre, il m'indique que l'immeuble
d'à côté, le 666B ne l'est pas. Je réalise que le conseiller PapierQ m'a
inscrit par erreur au 666 avec FRIC, et je lui fais part qu'il est probable que
j'habite au 666B. Il me donne un détail sur le 666 qui compte 3 habitant: cela me
conforte dans l'idée que j'habite finalement au 666 et que:</p>
<ul>
<li>l'adresse sur le contrat d'electricité est éronné (ce n'est pas le 666B)</li>
<li>Clitoriche n'a pas le bon nombre d'habitant au 666</li>
</ul>
<p>À suivre...</p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Aujourd'hui: je déménage demain — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2021-12-01 23:43</time> </div>https://blog.parisni.com/people/didier-lestradeDidier Lestrade2022-01-01T14:45:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2022-01-01</time></span>
<span id="people-birthyear" style="visibility:hidden;">1958</span>
<span id="people-deathyear" style="visibility:hidden;"></span>
<span id="people-profession" style="visibility:hidden;">Activist</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Didier Lestrade</h1>
<p>Fondateur d'Act Up-Paris (inspiré de celle de New-York), il relate :</p>
<blockquote>
<p>on peut parler pendant une heure, mais si ça ne débouche pas sur une action, alors
c'est une discussion qui a servit a rien</p>
</blockquote>
<blockquote>
<p>on considérait que les gens devaient s'accrocher. si tu ne fous rien, tu ne m'intéresses pas. Si tu ne fais pas
un effort pour apprendre, pour lire ceci, pour écrire cela, pour participer t'es un poids mort. à un militant
qui arrivait, on demandait "qu'est-ce que tu veux faire ? ou que sais-tu faire ?"</p>
</blockquote>
<blockquote>
<p>je leur dis aux écolos, vous êtes trop gentils, il faut arriver maintenant à
une confrontation physique. Aller chez eux, terroriser leurs enfants avec des
casseroles, bloquer leurs téléphones, faire réellement pression. [...] on se
retrouve comme pour le sida face à des multinationales encore plus puissantes,
mais on connait toujours leur point faible. Il faut tapper sur le point faible.</p>
</blockquote>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Didier Lestrade — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2022-01-01 14:45</time> </div>https://blog.parisni.com/reading/sabato_le-tunnelLe Tunnel2023-08-25T00:06:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2023-08-24</time></span>
<span id="book-year" style="visibility:hidden;">1948</span>
<span id="book-author" style="visibility:hidden;">Enersto Sabato</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Le Tunnel</h1>
<h2>Citations</h2>
<p>p.17</p>
<blockquote>
<p>Il suffit d'observer les familles nombreuses où se répètent certains traits, certains gestes, certaines intonations.
Il m'est arrivé d'etre amoureux d'une femme (sans me déclarer bien sùr) et de fuir, effrayé par l'éventualité de
connaìtre ses sœurs. J'avais vécu quelque chose d'horrible en une autre occasion : le visage d'une femme m'avait
semblé très intéressant, mais d'avoir fait la connaissance de sa sœur m'avait laissé longtemps honteux et déprimé. Les
memes traits qui chez l'une m'avait parus admirables se retrouvaient accentués et déformés chez sa sœur, un peu comme
dans une caricature. Et cette espèce de vision déformée de la premiere femme en sa sœur avait suscité en moi, en plus
de cette sensation, un sentiment de honte comme si j'étais en partie coupable du jour légèrement ridicule que sa sœur
jetait sur la femme que j'avais tellement admirée.</p>
</blockquote>
<p>p.45</p>
<blockquote>
<p>Je ne crains pas d'avouer que, quelquefois, je n'ai pas pu manger de la journée, ou peindre de toute une semaine pour
avoir relevé tel ou tel trais de caractère. C'est incroyable à quel point la cupidité, l'envie, la prétention, la
grossièreté, l'avidité et en général tout cet ensemble d'attributs qui forment la condition humaine, transparaissent
sur un visage, dans une démarche, dans un regard. Il me parait naturel qu'après une telle rencontre, on n'ait plus
envie de manger, de peindre ni mème de vivre. Pourtant, je tiens à ce qu'il soit bien clair que je ne tire nul orgueil
de cette caractéristique.</p>
</blockquote>
<p>p.111</p>
<blockquote>
<p>Alors je me suis trainé jusqu'à la salle de bains et je me suis mis tout habillé dans la baignore. L'eau froide à
commencé à me calmer et dans ma tète sont apparus peu à peu quelques faits isolés, toujours par bribe et sans liens
entre eux, comme les premiers objets qu'on voit émerger après une inondation.</p>
</blockquote>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Le Tunnel — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-08-25 00:06</time> </div>https://blog.parisni.com/people/paul-phillipsPaul Phillips2023-12-26T20:00:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2023-12-26</time></span>
<span id="people-birthyear" style="visibility:hidden;">1972</span>
<span id="people-deathyear" style="visibility:hidden;"></span>
<span id="people-profession" style="visibility:hidden;">Programmer</span>
<span id="reading-time" style="visibility:hidden;">less than a minute</span></p>
<h1>Paul Phillips</h1>
<ul>
<li>poker player</li>
<li>scala major contributor</li>
<li>author of below</li>
</ul>
<pre><code>WHITESPACE ODDITY
by Paul Phillips, in eternal admiration of David Bowie, RIP
[Verse 1]
Bound Ctrl to Major mode
Bound Ctrl to Major mode
Read inputrc and set extdebug on
Bound Ctrl to Major mode (Ten, Nine, Eight, Seven, Six)
Connecting readline, options on (Five, Four, Three)
Check the syntax, may terminfo be with you (Two, One, Exec)
[Verse 2]
This is Bound Ctrl to Major mode
You really must upgrade
both shell completion and keyboard firmware
Now it's time to leave the console if you dare
This is Major mode to Bound Ctrl
I'm starting ssh
And my echo's in a most peculiar way
And the glyphs look very different today
[Chorus]
For here
Am I tapping on a keyboard
Far beneath the world
Now my screen is blue
And there's nothing I can do
[Verse 3]
Though I'm past one hundred thousand lines
my scrollbar's very still
And I think the shell must know where core dumps go
Tell Fallout I love it very much - it knows
Bound Ctrl to Major mode
Your pipeline's dead, there's something wrong
Can you hear me, Major mode?
Can you hear me, Major mode?
Can you hear me, Major mode?
Can you ?
[Chorus]
Here am I hiding in the basement
Safe in my cocoon
Now my screen is blue
And there's nothing I can do
</code></pre>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Paul Phillips — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2023-12-26 20:00</time> </div>https://blog.parisni.com/reading/AkiraAkira2024-01-20T18:59:00+00:00<div id="generated-toc"> </div><hr> <div id="generated-toc"> </div>
<p><span>Published on <time id="post-date">2024-01-20</time></span>
<span id="book-year" style="visibility:hidden;">1982</span>
<span id="book-author" style="visibility:hidden;">Katsuhiro Otomo</span>
<span id="reading-time" style="visibility:hidden;">less than a minute
<span id="ystars" style="visibility:hidden;">★★★★★</span>
<span id="stars" style="visibility:hidden;"></span></span></p>
<h1>Akira</h1>
<p><img src="../IMG_20240120_182432.jpg" alt="">
<img src="../IMG_20240120_185617.jpg" alt=""></p>
<div id="last-modified"><p> <a href="mailto:parisni@riseup.net?subject=Akira — software lobotomy">React ?</a> </p>This page was last modified:
<time id="git-timestamp"> 2024-01-20 18:59</time> </div>