bobek.cza blog by Antonín Králhttps://bobek.cz2021-03-08T01:00:00+01:00Antonín KrálWeekend long read suggestions 2020/11https://bobek.cz/long-reads/2020-11/2020-03-13T01:00:00+01:002024-03-20T13:25:18+01:00Antonín Král<ul>
<li><p><a
href="https://a16z.com/2020/03/06/external-ceo-founders-startups-when-how-nicira-story/">Bringing
in an External CEO</a></p>
<p>Andreessen Horowitz' view on when, why and how to bring outside CEO
to your business.</p></li>
<li><p><a href="https://youtu.be/DnT-LUQgc7s">Considering Rust</a></p>
<p>Jon Gjengset's excellent talk about why you should consider Rust for
your next project.</p></li>
<li><p><a
href="https://akoutmos.com/post/elixir-logging-loki/">Structured logging
in Elixir using Loki</a></p>
<p>I am a big proponent of structured logging for long time. Alex is
doing a great job in describing how to start with structured logs within
your Phoenix/Elixir application. You don't need to end up using Loki to
benefit.</p></li>
<li><p><a href="https://github.com/mit-pdos/noria">Noria</a></p>
<p>Noria is a streaming data-flow system designed to act as a fast
storage backend for read-heavy web applications. It acts like a
database, but pre-computes and caches relational query results so that
reads are blazingly fast. Noria automatically keeps cached results
up-to-date as the underlying data, stored in persistent base tables,
change.</p></li>
<li><p><a
href="https://www.confluent.io/blog/configure-kafka-to-minimize-latency/">99th
Percentile Latency at Scale with Apache Kafka</a></p>
<p>Impact of per-broker resource parameters (like number of connections,
number of partitions, and the request rate per broker) on the observed
latency on Kafka messages at scale.</p></li>
<li><p><a
href="https://fasterthanli.me/blog/2020/a-half-hour-to-learn-rust/">A
half-hour to learn Rust</a></p>
<p>Obviously, you are not going to learn Rust in a half of hour. But I
really enjoyed this article by Amos. He did a great job of throwing as
much of Rust idioms at you as possible.</p></li>
<li><p><a
href="https://medium.com/@gajus/lessons-learned-scaling-postgresql-database-to-1-2bn-records-month-edc5449b3067">Lessons
learned scaling PostgreSQL database to 1.2bn records/month</a></p>
<p>Couple of practical recommendations for scaling up your PostgreSQL.
Also a good story on "how using the hosted SQL doesn't cut it".</p></li>
</ul>Weekend long read suggestions 2020/10https://bobek.cz/long-reads/2020-10/2020-03-06T01:00:00+01:002024-03-20T13:25:18+01:00Antonín Král<ul>
<li><p><a
href="https://parsers.me/us-court-fully-legalized-website-scraping-and-technically-prohibited-it/">US
court fully legalized website scraping and technically prohibited
it</a></p>
<p>Coverage of decision which took part of the LinkedIn vs hiQ Labs. The
appeals court also upheld a lower court ruling that prohibits LinkedIn
from interfering with hiQ’s web scraping of its site. This fundamentally
changes the balance of power in dealing with such cases in the
future.</p></li>
<li><p><a
href="https://gigaom.com/2014/06/30/the-dark-side-of-io-how-the-u-k-is-making-web-domain-profits-from-a-shady-cold-war-land-deal/">The
dark side of .io: How the U.K. is making web domain profits from a shady
Cold War land deal</a></p>
<p>Came to this (older) piece via <a
href="https://www.diagrams.net/blog/move-diagrams-net">recent
decision</a> by <code>draw.io</code> to rename to
<code>diagrams.net</code> a they are not happy with the sate of
<code>.io</code> TLD.</p></li>
</ul>
<h2 id="software-development">Software Development</h2>
<ul>
<li><p><a
href="https://www.slowtec.de/posts/2020-02-28-porting-javascript-to-rust-part-3.html">Porting
a JavaScript App to WebAssembly with Rust</a></p>
<p>3 part series mapping an exercise of rewriting React+Redux
application written in JavaScript to WebAssembly (WASM) with
Rust.</p></li>
<li><p><a
href="https://thoughtbot.com/blog/breaking-out-of-ecto-schemas">Breaking
Out of Ecto Schemas</a></p>
<p>Super short article about stepping out from <code>Ecto.Schema</code>
closer to raw SQL with <code>Ecto</code>. Neath trick is to use
<code>Map</code> in your <code>select:</code> statement, so
<code>Ecto</code> will return (list of) <code>Map</code> instead of a
list of lists.</p></li>
<li><p><a
href="https://deterministic.space/high-performance-rust.html">Cheap
tricks for high-performance Rust</a></p>
<p>Pascal shares some of the simple tricks to speed up your Rust
programs without really changing the source. Hints like properly setting
your target architecture, alternative allocator, release profiles and
more.</p></li>
<li><p><a
href="https://blog.cloudflare.com/when-bloom-filters-dont-bloom/">When
Bloom filters don't bloom</a></p>
<p>Marek needed to deduplicate large list of IP addresses, so he set
sail on the journey of getting better then <code>sort | unique</code>.
He shares some lessons learned about random memory access latency, power
of cache friendly data structures and Bloom Filters and finally "just"
hash table.</p></li>
<li><p><a href="https://github.com/rust-lang/rust/issues/69593">Moving a
method from struct impl to trait causes performance degradation</a></p>
<p>On very similar note as a previous one -- code alignment having a
significant impact on the performance.</p></li>
<li><p><a href="https://github.com/brickpop/flutter-rust-ffi">Starter
project for Flutter plugins willing to access native and synchronous
rust code using FFI</a></p>
<p>Flutter meets Rust, wow.</p></li>
</ul>
<h2 id="psql">psql</h2>
<ul>
<li><p><a href="https://habr.com/en/company/postgrespro/blog/490228/">On
recursive queries</a></p>
<p>I never really give a recursion though in the context of
SQL...</p></li>
<li><p><a href="https://pgtune.leopard.in.ua">PGTune</a></p>
<p><code>pgtune</code> on web. Simple site to give you <code>psql</code>
configuration to start with for different use-cases and server
configuration.</p></li>
</ul>
<h2 id="machine-learning">Machine Learning</h2>
<ul>
<li><p><a
href="https://github.com/tomohideshibata/BERT-related-papers">BERT
Related Papers</a></p>
<p>List of various papers related to BERT (Bidirectional Encoder
Representations from Transformers). BERT was released by Google as part
of their NLP research. But researchers are stepping forward and you can
find multi-modal applications as well.</p></li>
<li><p><a
href="https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html">Exploring
Transfer Learning with T5: the Text-To-Text Transfer Transformer</a></p>
<p>New publication by Google at the NLP space. They have published a new
model called Text-To-Text Transfer Transformer (T5). They have also
open-sourced a new pre-training dataset, called the Colossal Clean
Crawled Corpus (C4).</p></li>
<li><p><a
href="https://graphdeeplearning.github.io/post/transformers-are-gnns/">Transformers
are Graph Neural Networks</a></p>
<p>Drawing parallels between Transformers (key component of BERT) and
Graph Networks.</p></li>
</ul>Weekend long read suggestions 2020/09https://bobek.cz/long-reads/2020-09/2020-02-28T01:00:00+01:002024-03-20T13:25:18+01:00Antonín Král<ul>
<li><p><a
href="http://airpower.airforce.gov.au/APDC/media/PDF-Files/Air%20Force%20Publications/AF13-Applied-Thinking-for-Intelligence-Analysis.pdf">Applied
thinking for intelligence analysis</a></p>
<p>Guide from Australian Air Force. The key message is not surprising --
time pressure is bad for decision making. It increases changes of making
mistakes, ignoring analyst's bias or even lead to seizure of pieces of
important information to confirm the biasses / expected
results.</p></li>
<li><p><a
href="https://blog.ycombinator.com/a-standard-and-clean-series-a-term-sheet/">A
Standard and Clean Series A Term Sheet</a></p>
<p>YC take on how "good" Series A Term Sheet should look like.</p></li>
</ul>
<h2 id="software-development">Software Development</h2>
<ul>
<li><p><a
href="https://www.poeticoding.com/distributed-phoenix-chat-with-pubsub-pg2-adapter/">Distributed
Phoenix Chat with PubSub PG2 adapter</a></p>
<p>How to leverage <code>pg2</code> (process groups) for PubSub over
multiple nodes.</p></li>
<li><p><a
href="https://gitlab.com/esr/reposurgeon/blob/master/GoNotes.adoc">Notes
on the Go translation of Reposurgeon</a></p>
<p>An experience report on a Python-to-Go translation of a program with
significant complexity, written in attempted conformance with the Go
community’s practice for grounding language enhancement requests not in
it-would-be-nice-to-have abstractions but rather in a description of
real-world problems.</p></li>
<li><p><a href="https://evrone.com/ferrum-ruby-chrome-driver">Ferrum: a
fearless Ruby Chrome driver</a></p>
<p>Ferrum is super simple to use chrome driver for writing your
tests.</p></li>
</ul>
<h2 id="infrastructure">Infrastructure</h2>
<ul>
<li><p><a
href="https://blog.getambassador.io/verifying-service-mesh-tls-in-kubernetes-using-ksniff-and-wireshark-454b1e3f4dc9">Verifying
Service Mesh TLS in Kubernetes, Using ksniff and Wireshark</a></p>
<p>You don't necessarily need to verify TLS on service mesh, but
knowledge of <code>ksniff</code> could be definitely helpful.</p></li>
<li><p><a
href="https://blog.acolyer.org/2020/02/26/meaningful-availability/">Meaningful
availability</a></p>
<p>Beautiful (you should subscribe to Adrian Colyer's blog) disection of
Google's paper about how they compute availability of services to better
reflect actual impact on the users.</p></li>
<li><p><a
href="https://levelup.gitconnected.com/design-for-failure-distributed-transaction-in-microservices-f026b25ba847">Design
for Failure — Distributed Transaction in Microservices</a></p>
<p>Kafka powered take on distributed sagas patter for implemented
distributed transactions.</p></li>
<li><p><a
href="https://engineering.shopify.com/blogs/engineering/circuit-breaker-misconfigured">Your
Circuit Breaker is Misconfigured</a></p>
<p>Good coverage of how to (and not to) configure your circuit breakers.
Damian is used context of <a
href="https://github.com/Shopify/semian">semian</a>, their open-source
implementation of circuit breaker (and bulkheading).</p></li>
</ul>
<h2 id="data-and-data-infrastructure">Data and Data Infrastructure</h2>
<ul>
<li><p><a
href="https://engineering.linkedin.com/blog/2020/open-sourcing-datahub--linkedins-metadata-search-and-discovery-p">LinkedIn's
DataHub</a></p>
<p>LinkedIn decided to open-source their tool DataHub, metadata search
and discovery platform for wast amounts of different datasets available
within the organization.</p></li>
<li><p><a href="https://github.com/mgartner/pg_flame">pg_flame</a></p>
<p>A flamegraph generator for Postgres EXPLAIN ANALYZE output.</p></li>
<li><p><a href="https://github.com/eulerto/wal2json">wal2json</a></p>
<p>JSON output plugin for changeset extraction.</p></li>
<li><p><a
href="https://github.com/Machine-Learning-Tokyo/AI_Curriculum">AI_Curriculum</a></p>
<p>Open Deep Learning and Reinforcement Learning lectures from top
Universities like Stanford University, MIT, UC Berkeley. Currently
covering the following topics:</p>
<ul>
<li>Introduction to Deep Learning</li>
<li>CNNs for Visual Recognition</li>
<li>NLP with Deep Learning</li>
<li>Deep Reinforcement Learning</li>
</ul></li>
</ul>The Power of Written StandupFrequently overlooked tool for distributed teamshttps://bobek.cz/blog/2021/written-standup/2021-03-08T01:00:00+01:002024-03-21T17:34:15+01:00Antonín Král<p>Written standup is a frequently overlooked tool for distributed
teams. It lets people catch up easily if they missed a particular
standup. It also produces a body of searchable and rich documents. In
addition, it helps you see what you accomplished during the day and feel
proud of yourself, which can be hard to do in knowledge jobs. It is best
to treat written standup as your logbook. Filling it during the day
removes the burden of remembering what you have done at the end of the
day. As well as it allows for faster pickup by others.</p>
<p>Here are some of the benefits of written standup:</p>
<ul>
<li><strong>Increased visibility:</strong> Written standup gives
everyone on the team visibility into what everyone else is working on.
It keeps everyone on the same page and working towards the same
goals.</li>
<li><strong>Improved accountability:</strong> Written standup helps to
improve accountability by making it clear what everyone is supposed to
be working on. Makes sure that everyone is contributing to the team's
success.</li>
<li><strong>Improved communication:</strong> Written standup helps to
improve communication by providing a forum for team members to share
updates and ask questions. This helps to keep everyone in the loop and
reduces the chances of misunderstandings.</li>
</ul>
<p>If you are a distributed team, I highly recommend using written
standup. It is a simple tool that can have a big impact on your team's
productivity and effectiveness.</p>
<h2 id="how-to-do-written-standup">How to Do Written Standup</h2>
<p>Written standup<span
class="sidenote-wrapper"><label for="sn-0" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-0" class="margin-toggle"/><span
class="sidenote">Can we call it standup, if you are sitting, when
writing it? Maybe it should be called <code>situp</code> :smile:<br />
<br />
</span></span> is a simple process. Here are the steps involved:</p>
<ol type="1">
<li>Simple automation creates a fresh standup document during the
night.</li>
<li>At the beginning of each day, each team member writes a short update
about what they plan to work on that day.</li>
<li>The updates are then shared with the rest of the team.</li>
<li>At the end of the day, each team member writes a short update about
what they've actually worked on and what they plan to work on the next
day.</li>
</ol>
<p>That's it! Written standup is a simple process that can have a big
impact on your team's productivity and effectiveness.</p>
<h2 id="tips-for-effective-written-standup">Tips for Effective Written
Standup</h2>
<p>Here are some tips for effective written standup:</p>
<ul>
<li>Keep your updates short and to the point.</li>
<li>Share links to the relevant resources (tickets, code, MRs).</li>
<li>Be specific about what you plan to work on and what you actually
worked on.</li>
<li>Be honest about your progress.</li>
<li>Be open to feedback from your teammates.</li>
</ul>
<p>With a little practice, written standup can become a valuable tool
for your team.</p>
<p><img src="standup_example.png" alt="Example of a written standup" />
<em>Example page from a written standup document of an IoT
project.</em></p>Backups with restic and rcloneEncrypted local and cloud backupshttps://bobek.cz/blog/2020/restic-rclone/2020-03-03T01:00:00+01:002024-03-21T17:38:51+01:00Antonín Král<p>I have started using <a href="https://restic.net/">restic</a> backup
about 2 years ago. I have learned recently, that some of my colleagues
are not aware of its existence or not aware of tight integration with <a
href="https://rclone.org/">rclone</a>. These two are a powerful combo,
allowing for encrypted local and cloud backups with easy.</p>
<p>I also needed to recover from backups multiple times already, so even
this part of backup strategy is working :)</p>
<h2 id="restic">Restic</h2>
<p>Restic is super <em>easy</em> to use. It is <em>secure</em> as it
assumes that storage of the backup is untrusted by default. Backups are
<em>encrypted</em> with AES-256 in counter mode and authentication is
done using Poly1305-AES. Checkout <a
href="https://blog.filippo.io/restic-cryptography/">Filippo's
article</a> for a deeper look into the restic security model. It is
quite <em>fast</em>. Restic needs under 4 minutes to go through my 366GB
home. <code>rsync</code> needed something like 20-30 minutes to do the
same job.</p>
<p>I am using restic with two storage backends -- <code>sftp</code> and
<code>rclone</code>. Mode of operation is still the same, only what
changes (from the user's perspective is URL).</p>
<h2 id="getting-started">Getting started</h2>
<p>After you install <code>restic</code>, you need to initialize your
storage repository. It is as simple as:</p>
<pre><code>~ ❯ restic -r sftp:bobek@klobouk:/klobouk/backups/restic init
enter password for new repository:
enter password again:
created restic repository 0507dc2c01 at sftp:bobek@klobouk:/klobouk/backups/restic
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.</code></pre>
<p>Where <code>sftp:bobek@klobouk:/klobouk/backups/restic</code> is a
repository URL. As you would guess</p>
<ul>
<li><code>sftp</code> is a protocol to be used (<code>sftp</code> in
this case)</li>
<li><code>bobek</code> is a username used for establishing
connection</li>
<li><code>klobouk</code> is a server name (ip address, FQDN, alias from
the ssh_config, basically anything you can ssh/sftp to)</li>
<li><code>/klobouk/backups/restic</code> is the destination path for the
repository on the server side</li>
</ul>
<h2 id="making-backup">Making backup</h2>
<p>Making a snapshot (incremental backup) is question of using
<code>backup</code> command. You can pass additional arguments, such as
<code>-e</code> for filtering out certain paths. Below is example of
backing up my home:</p>
<pre><code>~ ❯ restic -r sftp:bobek@klobouk:/klobouk/backups/restic backup -e ~/Movies ~/
Enter passphrase for key '/home/bobek/.ssh/id_rsa4096':
enter password for repository:
repository 0507dc2c01 opened successfully, password is correct
Files: 6941 new, 2045 changed, 2041192 unmodified
Dirs: 0 new, 1 changed, 0 unmodified
Added to the repo: 2.609 GiB
processed 2050178 files, 365.919 GiB in 3:44
snapshot 72217508 saved</code></pre>
<p>You can check your snapshots with <code>snapshots</code> command,
such as</p>
<pre><code>~ ❯ restic -r sftp:bobek@klobouk:/klobouk/backups/restic snapshots
enter password for repository:
repository 0507dc2c01 opened successfully, password is correct
ID Time Host Tags Paths
------------------------------------------------------------------
41a3d112 2019-11-16 17:23:12 bobek /home/bobek
7816d840 2019-11-23 08:14:47 bobek /home/bobek
7788586b 2019-12-01 21:58:40 bobek /home/bobek
// removed bunch of lines to improve readability of the article
fda51794 2020-03-02 22:37:36 bobek /home/bobek
72217508 2020-03-03 09:20:54 bobek /home/bobek
------------------------------------------------------------------
56 snapshots</code></pre>
<h3 id="pruning-restic-repository">Pruning restic repository</h3>
<p>Your backup repository will grow indefinitely as retention policy is
not part of the <code>backup</code> command. You can use
<code>forget</code> command for that. As you can see, I have not ran it
for a while. Let's fix that.</p>
<p>I am going to do it slightly differently this time and ssh to the
server hosting the repository, so I will make all IO operations local
(you can still use <code>sftp:...</code> with no problem).</p>
<pre><code>bobek@klobouk:~$ restic -r /klobouk/backups/restic forget --keep-daily 7 --keep-weekly 10 --prune --cleanup-cache
enter password for repository:
repository 0507dc2c01 opened successfully, password is correct
Applying Policy: keep the last 7 daily, 10 weekly snapshots
keep 15 snapshots:
ID Time Host Tags Reasons Paths
-----------------------------------------------------------------------------------
cadfc8cc 2020-01-05 23:38:23 bobek weekly snapshot /home/bobek
1f6bd29f 2020-01-11 22:56:46 bobek weekly snapshot /home/bobek
2cfaf041 2020-01-19 20:05:58 bobek weekly snapshot /home/bobek
b222742c 2020-01-26 23:59:14 bobek weekly snapshot /home/bobek
0dbcbc46 2020-02-02 21:18:13 bobek weekly snapshot /home/bobek
8c1ad6e8 2020-02-08 14:11:41 bobek weekly snapshot /home/bobek
1794a66f 2020-02-16 23:05:10 bobek weekly snapshot /home/bobek
bb3abe88 2020-02-23 23:22:54 bobek weekly snapshot /home/bobek
e4f2f56c 2020-02-24 21:16:06 bobek daily snapshot /home/bobek
9940a28b 2020-02-25 21:39:12 bobek daily snapshot /home/bobek
df5bab63 2020-02-26 21:23:02 bobek daily snapshot /home/bobek
b8d11888 2020-02-27 23:41:14 bobek daily snapshot /home/bobek
272d9e5e 2020-02-28 17:19:45 bobek daily snapshot /home/bobek
weekly snapshot
fda51794 2020-03-02 22:37:36 bobek daily snapshot /home/bobek
72217508 2020-03-03 09:20:54 bobek daily snapshot /home/bobek
weekly snapshot
-----------------------------------------------------------------------------------
15 snapshots
remove 41 snapshots:
ID Time Host Tags Paths
------------------------------------------------------------------
41a3d112 2019-11-16 17:23:12 bobek /home/bobek
7816d840 2019-11-23 08:14:47 bobek /home/bobek
// again, removed bunch of lines here
c8f29801 2020-03-02 07:12:00 bobek /home/bobek
4a7085c1 2020-03-02 21:03:36 bobek /home/bobek
------------------------------------------------------------------
41 snapshots
41 snapshots have been removed, running prune</code></pre>
<p>You can run <code>forget</code> without a <code>--prune</code> and it
will be quick as it only marks snapshots for removal. The actual removal
of data and compaction of the repository can be also achieved with a
separate <code>prune</code> command. Just be aware, that prune can take
a considerable amount of time.</p>
<pre><code>bobek@klobouk:~$ du -sh /klobouk/backups/restic
587G /klobouk/backups/restic
bobek@klobouk:~$ restic -r /klobouk/backups/restic prune --cleanup-cache
enter password for repository:
repository 5a4fef57 opened successfully, password is correct
counting files in repo
building new index for repo
[1:10:38] 100.00% 121521 / 121521 packs
repository contains 121521 packs (2995676 blobs) with 577.392 GiB
processed 2995676 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 15 snapshots
[7:16] 100.00% 15 / 15 snapshots
found 2294461 of 2995676 data blobs still in use, removing 701215 blobs
will remove 0 invalid files
will delete 27364 packs and rewrite 14807 packs, this frees 172.604 GiB
[2:27:26] 100.00% 14807 / 14807 packs rewritten
counting files in repo
[37:10] 100.00% 85892 / 85892 packs
finding old index files
saved new indexes as [82c9a37f 5acc102a d5daaa0c 4a5dee08 881d6187 e9dfa06e 5bf9bf27 a3b85e96 daf45279 b56cc75e 95ef84d2 7e39afc3 9f10a455 ea80a8e9 b65b4106 6def21e2 c6ea3aae c0b30616 bb03b1eb f9c4261c 2babd31c e7d80fa8 8a5b5e47 984d8a78 c3275bb3 071e64a9 c97bb60c f263321e 67629707]
remove 125 old index files
[26:01] 100.00% 42171 / 42171 packs deleted
done
bobek@klobouk:~$ du -sh /klobouk/backups/restic
412G /klobouk/backups/restic</code></pre>
<h3 id="recovery">Recovery</h3>
<p>Let's restore some data. Frequently, I don't need a recovery of
complete backup (like complete home), but rather some particular
directory or file. Something like -- I've just crashed my Firefox, lost
all the open tabs and want to restore the complete Firefox
directory.</p>
<ol type="1">
<li>Use <code>snapshots</code> to figure ID of snapshot we want to
recover from. Let's pick the latest from our example, which is
<code>72217508</code> at our example above.</li>
<li>User <code>restore</code> to get files from that snapshot. I can
look something like</li>
</ol>
<pre><code>~ ❯ restic -r sftp:bobek@klobouk:/klobouk/backups/restic restore 72217508 --target /tmp/restore-firefox --include /home/bobek/.mozilla/firefox
enter password for repository:
repository 0507dc2c01 opened successfully, password is correct
restoring <Snapshot 72217508 of [/home/bobek] at 2020-03-03 09:20:54.744399448 +0100 CET by bobek@bobek> to /tmp/restore</code></pre>
<p>And we have our files under
<code>/tmp/restore/home/bobek/.mozilla/firefox</code> as expected, with
a proper file mode bits etc.</p>
<h3 id="locking">Locking</h3>
<p><code>restic</code> locks the repo when working with it. It will tell
you when and who locked it. For example:</p>
<pre><code>~ ❯ restic -r sftp:bobek@klobouk:/klobouk/backups/restic snapshots
enter password for repository:
repository 0507dc2c01 opened successfully, password is correct
Fatal: unable to create lock in backend: repository is already locked exclusively by PID 18172 on klobouk by bobek (UID 1000, GID 1000)
lock was created at 2020-03-03 09:48:29 (1h37m6.789848817s ago)
storage ID 9e313963</code></pre>
<p>In cases, where you are sure that process which acquired lock is
already dead, you may need to use <code>unlock</code> command. It is
also a good idea to run <code>restic check</code> to verify consistency
of the repository afterwards.</p>
<h2 id="rclone">rclone</h2>
<p><code>rclone</code> is like <code>rsync</code> for cloud. It supports
usual suspects (Amazon S3, Google Drive, Dropbox) but also a lot of
others.</p>
<p>I will use Google Drive as an example. Just follow the <a
href="https://rclone.org/drive/">rclone GDrive documentation</a> to set
it up. You need to pick a name for your configuration. I have used
<code>gdrive</code>, so after everything is done, you can issues</p>
<pre><code>~ ❯ rclone ls gdrive:</code></pre>
<p>to list all files on your Google Drive. No need for mounting FUSE
based emulations etc. Also <code>rclone</code> behaves like
<code>rsync</code> and can keep two destinations in sync. So the
practical use-case, is synchronization between Dropbox and GDrive:</p>
<pre><code>~ ❯ rclone sync -P "dropbox:/Apps/Pragmatic Bookshelf/" "gdrive:/Pragmatic Bookshelf/"</code></pre>
<h2 id="restic-with-rclone">restic with rclone</h2>
<p>Finally -- let's setup backup to the cloud. And it is simple as:</p>
<ul>
<li><p>configuring <code>rclone</code> for the cloud storage of your
choice (<code>gdrive</code> in my case)</p></li>
<li><p>using <code>rclone</code> backend of the <code>restic</code> to
talk to the selected storage, for example:</p>
<pre><code>~ ❯ restic -r rclone:gdrive:restic init</code></pre>
<p>which will create <code>restic</code> folder at your
<code>gdrive</code>.</p></li>
<li><p>and using all the <code>restic</code> commands we have covered
before, like</p>
<pre><code>~ ❯ restic -r rclone:gdrive:restic backup -e .notmuch ~/Mail</code></pre>
<p>to backup your <code>~/Mail</code> folder to GDrive.</p></li>
</ul>
<h2 id="regular-backups">Regular backups</h2>
<p>Backups are useful only if you do them regularly. I have initially
place <code>restic</code> to my <code>crontab</code> to schedule
backups. But ended up with running backups manually after all as I need
to have it under a bit more control. Thus I have a script which I run
before getting to bed:</p>
<div class="sourceCode" id="cb12"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="co">#!/bin/bash</span></span>
<span id="cb12-2"><a href="#cb12-2" aria-hidden="true" tabindex="-1"></a><span class="bu">set</span> <span class="at">-e</span></span>
<span id="cb12-3"><a href="#cb12-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-4"><a href="#cb12-4" aria-hidden="true" tabindex="-1"></a><span class="va">SFTP_REPO</span><span class="op">=</span><span class="st">"sftp:bobek@10.0.0.250:/klobouk/backups/restic"</span></span>
<span id="cb12-5"><a href="#cb12-5" aria-hidden="true" tabindex="-1"></a><span class="va">GDRIVE_REPO</span><span class="op">=</span><span class="st">"rclone:gdrive:restic"</span></span>
<span id="cb12-6"><a href="#cb12-6" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-7"><a href="#cb12-7" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> i <span class="kw">in</span> <span class="st">"</span><span class="va">$@</span><span class="st">"</span></span>
<span id="cb12-8"><a href="#cb12-8" aria-hidden="true" tabindex="-1"></a><span class="cf">do</span></span>
<span id="cb12-9"><a href="#cb12-9" aria-hidden="true" tabindex="-1"></a> <span class="cf">case</span> <span class="va">$i</span> <span class="kw">in</span></span>
<span id="cb12-10"><a href="#cb12-10" aria-hidden="true" tabindex="-1"></a> <span class="ss">--prune</span><span class="kw">)</span></span>
<span id="cb12-11"><a href="#cb12-11" aria-hidden="true" tabindex="-1"></a> <span class="va">RESTIC_PRUNE</span><span class="op">=</span>true</span>
<span id="cb12-12"><a href="#cb12-12" aria-hidden="true" tabindex="-1"></a> <span class="cf">;;</span></span>
<span id="cb12-13"><a href="#cb12-13" aria-hidden="true" tabindex="-1"></a> <span class="pp">*</span><span class="kw">)</span></span>
<span id="cb12-14"><a href="#cb12-14" aria-hidden="true" tabindex="-1"></a> <span class="co"># unknown option</span></span>
<span id="cb12-15"><a href="#cb12-15" aria-hidden="true" tabindex="-1"></a> <span class="cf">;;</span></span>
<span id="cb12-16"><a href="#cb12-16" aria-hidden="true" tabindex="-1"></a> <span class="cf">esac</span></span>
<span id="cb12-17"><a href="#cb12-17" aria-hidden="true" tabindex="-1"></a><span class="cf">done</span></span>
<span id="cb12-18"><a href="#cb12-18" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-19"><a href="#cb12-19" aria-hidden="true" tabindex="-1"></a><span class="fu">ssh</span> root@10.0.0.250 <span class="st">'df -h /klobouk/'</span></span>
<span id="cb12-20"><a href="#cb12-20" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-21"><a href="#cb12-21" aria-hidden="true" tabindex="-1"></a><span class="bu">read</span> <span class="at">-s</span> <span class="at">-t</span> 5 <span class="at">-p</span> <span class="st">"Enter restic password, followed by [ENTER]"</span> <span class="va">RESTIC_PASSWORD</span> <span class="kw">&&</span> <span class="bu">export</span> <span class="va">RESTIC_PASSWORD</span></span>
<span id="cb12-22"><a href="#cb12-22" aria-hidden="true" tabindex="-1"></a><span class="bu">echo</span> <span class="st">""</span></span>
<span id="cb12-23"><a href="#cb12-23" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-24"><a href="#cb12-24" aria-hidden="true" tabindex="-1"></a><span class="ex">restic</span> <span class="at">-r</span> <span class="st">"</span><span class="va">$SFTP_REPO</span><span class="st">"</span> snapshots <span class="at">--cleanup-cache</span></span>
<span id="cb12-25"><a href="#cb12-25" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-26"><a href="#cb12-26" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> <span class="kw">[[</span> <span class="op">(</span><span class="st">"</span><span class="va">$RESTIC_PRUNE</span><span class="st">"</span> <span class="ot">=</span> true<span class="op">)</span> <span class="kw">]]</span></span>
<span id="cb12-27"><a href="#cb12-27" aria-hidden="true" tabindex="-1"></a><span class="cf">then</span></span>
<span id="cb12-28"><a href="#cb12-28" aria-hidden="true" tabindex="-1"></a> <span class="ex">restic</span> <span class="at">-r</span> <span class="st">"</span><span class="va">$GDRIVE_REPO</span><span class="st">"</span> forget <span class="at">--keep-daily</span> 14 <span class="at">--keep-weekly</span> 18 <span class="at">--prune</span> <span class="at">--cleanup-cache</span></span>
<span id="cb12-29"><a href="#cb12-29" aria-hidden="true" tabindex="-1"></a> <span class="ex">restic</span> <span class="at">-r</span> <span class="st">"</span><span class="va">$GDRIVE_REPO</span><span class="st">"</span> check</span>
<span id="cb12-30"><a href="#cb12-30" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-31"><a href="#cb12-31" aria-hidden="true" tabindex="-1"></a> <span class="ex">restic</span> <span class="at">-r</span> <span class="st">"</span><span class="va">$SFTP_REPO</span><span class="st">"</span> forget <span class="at">--keep-daily</span> 7 <span class="at">--keep-weekly</span> 10 <span class="at">--prune</span> <span class="at">--cleanup-cache</span></span>
<span id="cb12-32"><a href="#cb12-32" aria-hidden="true" tabindex="-1"></a> <span class="ex">restic</span> <span class="at">-r</span> <span class="st">"</span><span class="va">$SFTP_REPO</span><span class="st">"</span> check</span>
<span id="cb12-33"><a href="#cb12-33" aria-hidden="true" tabindex="-1"></a><span class="cf">fi</span></span>
<span id="cb12-34"><a href="#cb12-34" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-35"><a href="#cb12-35" aria-hidden="true" tabindex="-1"></a><span class="ex">rclone</span> sync <span class="at">-P</span> <span class="st">"dropbox:/Apps/Pragmatic Bookshelf/"</span> <span class="st">"gdrive:/Pragmatic Bookshelf/"</span></span>
<span id="cb12-36"><a href="#cb12-36" aria-hidden="true" tabindex="-1"></a><span class="ex">restic</span> <span class="at">-r</span> <span class="st">"</span><span class="va">$GDRIVE_REPO</span><span class="st">"</span> backup <span class="at">-e</span> .notmuch ~/Mail</span>
<span id="cb12-37"><a href="#cb12-37" aria-hidden="true" tabindex="-1"></a><span class="ex">restic</span> <span class="at">-r</span> <span class="st">"</span><span class="va">$SFTP_REPO</span><span class="st">"</span> backup <span class="at">-e</span> ~/Movies ~/</span>
<span id="cb12-38"><a href="#cb12-38" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb12-39"><a href="#cb12-39" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> poweroff</span></code></pre></div>Extensible version manager - ASDF - for the rescuervm/rbench/nvm/etc under one roofhttps://bobek.cz/blog/2019/asdf/2019-11-25T01:00:00+01:002024-03-20T13:25:18+01:00Antonín Král<p>I have been using <code>rvm</code> for a long time to manage per
project ruby versions (and appropriate gemsets as well). While that
works really well for ruby, it doesn't solve for other run-times.
<code>node.js</code> for example. While I am not a front-end developer,
node is a frequent dependency for building JS for Rails as well as
Phoenix. Some people solve the issue with bunch of Docker containers,
but that feel clumsy and slow for local development.</p>
<p>Fortunately, there is a project called <a
href="https://github.com/asdf-vm/asdf">asdf</a>. It can replace
<code>rvm</code> as well as <code>nvm</code> (node) or even
<code>virtualenv</code> (Python) and much more -- check <a
href="https://asdf-vm.com/#/plugins-all">available plugins</a>.</p>
<h2 id="installation">Installation</h2>
<p>Installation is very simple, just follow <a
href="https://asdf-vm.com/#/core-manage-asdf-vm?id=install-asdf-vm">documentation</a>.
When done, you want to install plugins. For example:</p>
<pre><code>asdf plugin-add ruby
asdf plugin-add elixir
asdf plugin-add python
asdf plugin-add erlang
asdf plugin-add golang
asdf plugin-add yarn</code></pre>
<h2 id="updating">Updating</h2>
<p>You can update plugins with</p>
<pre><code>asdf plugin-update --all</code></pre>
<p>and you can also update <code>asdf</code> with</p>
<pre><code>asdf update</code></pre>
<h2 id="install-vm">Install vm</h2>
<p>Having plugin installed is not yet sufficient for your work. You also
need to install the appropriate version of your vm you want to use. For
example, you can get all available ruby versions with</p>
<pre><code>asdf list-all ruby</code></pre>
<p>when you pick one, you want to install, you need to invoke
<code>asdf install <plugin> <version></code>. For
example:</p>
<pre><code>asdf install ruby 2.6.5</code></pre>
<p>After you have installed particular version, you can set it as your
default with</p>
<pre><code>asdf global ruby 2.6.5</code></pre>
<p>or you can set it "per project" with</p>
<pre><code>asdf local ruby 2.6.5</code></pre>
<p>Commands above will write to file <code>.tool-versions</code>. Global
variant will use fire in your home directory, while local one uses
current directory. Local configuration has precedence, thus you can
define different versions per project.</p>
<p>You can verify your current active version with</p>
<pre><code>asdf current
# or for the particular plugin
asdf current ruby</code></pre>
<h2 id="shims">Shims</h2>
<p>Shim is used instead of binary installed by a library. This sometimes
break and you need to <strong>reshim</strong> your binaries. For
example</p>
<pre><code>gem install middleman # will install Middleman
asdf reshim ruby # fixes ruby shims
which middleman # $HOME/.asdf/shims/middleman</code></pre>
<h2 id="legacy-version-files">Legacy version files</h2>
<p>You man want to add <code>legacy_version_file = yes</code> to your
<code>~/.asdfrc</code> as that will instruct asdf to follow classic
version files. Such as <code>.ruby-version</code>.</p>
<h2 id="gemsets">Gemsets</h2>
<p><code>rvm</code> has a feature called gemset, which allows for
separation of installed gems per environment. <code>asdf</code> doesn't
have such support (<a
href="https://github.com/asdf-vm/asdf/issues/312">asdf#312</a>, <a
href="https://github.com/asdf-vm/asdf-ruby/issues/25">asdf-ruby#25</a>).
But I have found that I don't really miss them at all as all of my
projects are using <code>Gemfile</code> / <code>bundler</code>
anyway.</p>
<p>There is a neat feature of <code>asdf-ruby</code> though -- you can
list gems, which will get installed to every Ruby version, by placing
them to <code>$HOME/.default-gems</code>. For example:</p>
<pre><code>bundler
pry</code></pre>
<h2 id="conclusion">Conclusion</h2>
<p><code>asdf</code> works great so far and I am super happy with it. It
significantly simplified my workflow in projects depending on multiple
languages (e.g. Elixir + node).</p>Tailwind CSS & Phoenixhttps://bobek.cz/blog/2019/tailwindcss-phoenix/2019-11-08T01:00:00+01:002024-03-20T13:25:18+01:00Antonín Král<blockquote>
<p><a href="https://tailwindcss.com/">Tailwind CSS</a> is a highly
customizable, low-level CSS framework that gives you all of the building
blocks you need to build bespoke designs without any annoying
opinionated styles you have to fight to override.</p>
</blockquote>
<p>I am no front-end developer by any stretch of imagination.
Utility-first CSS framework thus looks quite interesting as it allows
rather rapid composition of various elements. It also plays nicely with
other CSS frameworks, so you can start using it without rewriting rest
of your code. But This article is going to cover the green field Phoenix
project wanting to use it.</p>
<p>I have found similar tutorials on the Internets, but they have been
always missing something. Versions in time of writing are Elixir
<code>1.9.1</code>, Phoenix <code>1.4.1</code>, Tailwind
<code>1.1.2</code>, Webpack <code>4.4.0</code>.</p>
<h2 id="getting-started">Getting started</h2>
<h3 id="dependencies">Dependencies</h3>
<p>If you are starting from scratch, just create your Phoenix app with
something like <code>mix phx.new tailwind</code>. You can skip
installation of dependencies for now. Then install Tailwind and <a
href="https://github.com/postcss/postcss">PostCSS</a>:</p>
<div class="sourceCode" id="cb1"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="bu">cd</span> assets</span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="ex">npm</span> install tailwindcss <span class="at">--save-dev</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="ex">npm</span> install postcss-loader <span class="at">--save-dev</span></span></code></pre></div>
<p>That will install both packages and also adds them to
<code>package.json</code>. Next thing is to add Tailwind as a plugin to
PostCSS. You can do it through <code>postcss.config.js</code> file.</p>
<div class="sourceCode" id="cb2"><pre
class="sourceCode javascript"><code class="sourceCode javascript"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co">// assets/postcss.config.js</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a>module<span class="op">.</span><span class="at">exports</span> <span class="op">=</span> {</span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="dt">plugins</span><span class="op">:</span> [</span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> <span class="pp">require</span>(<span class="st">'tailwindcss'</span>)<span class="op">,</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="pp">require</span>(<span class="st">'autoprefixer'</span>)</span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a> ]</span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a>}</span></code></pre></div>
<p>Or, as Phoenix already ships with <code>webpack</code>, just modify
<code>webpack.config.js</code> directly. Relevant is css
<code>rules</code> section, which looks like</p>
<div class="sourceCode" id="cb3"><pre
class="sourceCode javascript"><code class="sourceCode javascript"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a>{</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a> <span class="dt">test</span><span class="op">:</span> <span class="ss">/</span><span class="sc">\.</span><span class="ss">css</span><span class="sc">$</span><span class="ss">/</span><span class="op">,</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> <span class="dt">use</span><span class="op">:</span> [MiniCssExtractPlugin<span class="op">.</span><span class="at">loader</span><span class="op">,</span> <span class="st">'css-loader'</span>]</span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a>}</span></code></pre></div>
<p>we will add PostCSS like shown at the following snippet:</p>
<div class="sourceCode" id="cb4"><pre
class="sourceCode javascript"><code class="sourceCode javascript"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a>{</span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a> <span class="dt">test</span><span class="op">:</span> <span class="ss">/</span><span class="sc">\.</span><span class="ss">css</span><span class="sc">$</span><span class="ss">/</span><span class="op">,</span></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a> <span class="dt">use</span><span class="op">:</span> [</span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a> MiniCssExtractPlugin<span class="op">.</span><span class="at">loader</span><span class="op">,</span> </span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a> <span class="st">'css-loader'</span><span class="op">,</span></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a> {</span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a> <span class="dt">loader</span><span class="op">:</span> <span class="st">'postcss-loader'</span><span class="op">,</span></span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a> <span class="dt">options</span><span class="op">:</span> {</span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a> <span class="dt">ident</span><span class="op">:</span> <span class="st">'postcss'</span><span class="op">,</span></span>
<span id="cb4-10"><a href="#cb4-10" aria-hidden="true" tabindex="-1"></a> <span class="dt">plugins</span><span class="op">:</span> [</span>
<span id="cb4-11"><a href="#cb4-11" aria-hidden="true" tabindex="-1"></a> <span class="pp">require</span>(<span class="st">'tailwindcss'</span>)<span class="op">,</span></span>
<span id="cb4-12"><a href="#cb4-12" aria-hidden="true" tabindex="-1"></a> <span class="pp">require</span>(<span class="st">'autoprefixer'</span>)<span class="op">,</span></span>
<span id="cb4-13"><a href="#cb4-13" aria-hidden="true" tabindex="-1"></a> ]<span class="op">,</span></span>
<span id="cb4-14"><a href="#cb4-14" aria-hidden="true" tabindex="-1"></a> }<span class="op">,</span></span>
<span id="cb4-15"><a href="#cb4-15" aria-hidden="true" tabindex="-1"></a> }</span>
<span id="cb4-16"><a href="#cb4-16" aria-hidden="true" tabindex="-1"></a> ]</span>
<span id="cb4-17"><a href="#cb4-17" aria-hidden="true" tabindex="-1"></a>}</span></code></pre></div>
<h3 id="use-tailwind-in-your-stylesheet">Use Tailwind in your
stylesheet</h3>
<p>We can remove <code>assets/css/phoenix.css</code> as we replace
<code>assets/css/app.css</code> with</p>
<div class="sourceCode" id="cb5"><pre
class="sourceCode css"><code class="sourceCode css"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="im">@tailwind</span> base<span class="op">;</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="im">@tailwind</span> components<span class="op">;</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="im">@tailwind</span> utilities<span class="op">;</span></span></code></pre></div>
<p>Tailwind's <a
href="https://tailwindcss.com/docs/installation/#2-add-tailwind-to-your-css">documentation</a>
suggests using <code>@import</code> instead <code>@tailwind</code>, but
that didn't work for me.</p>
<h3 id="optional-tailwinds-configuration">(Optional) Tailwind's
configuration</h3>
<p>Tailwind supports tweaking of its behavior (for example changing
look-and-feel of the standard theme) through <a
href="https://tailwindcss.com/docs/configuration">configuration
file</a>. You can easily generate configuration file with</p>
<div class="sourceCode" id="cb6"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="bu">cd</span> assets <span class="kw">&&</span> <span class="ex">npx</span> tailwind init</span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="co"># or npx tailwind init --full in case you want to see all default values</span></span></code></pre></div>
<p>And that's it, Tailwind should be working.</p>
<h2 id="example----flash-messages">Example -- flash messages</h2>
<p>Let's use Tailwind in your Application. Flash messages are currently
displayed with the following code
(<code>lib/surveys_web/templates/layout/app.html.eex</code>):</p>
<div class="sourceCode" id="cb7"><pre
class="sourceCode eex"><code class="sourceCode elixir"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="op"><</span>p class<span class="op">=</span><span class="st">"alert alert-info"</span> role<span class="op">=</span><span class="st">"alert"</span><span class="op">><%=</span> get_flash<span class="fu">(</span><span class="ot">@conn</span>, <span class="va">:info</span><span class="fu">)</span> %<span class="op">></</span>p<span class="op">></span></span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a><span class="op"><</span>p class<span class="op">=</span><span class="st">"alert alert-danger"</span> role<span class="op">=</span><span class="st">"alert"</span><span class="op">><%=</span> get_flash<span class="fu">(</span><span class="ot">@conn</span>, <span class="va">:error</span><span class="fu">)</span> %<span class="op">></</span>p<span class="op">></span></span></code></pre></div>
<p>Let's replace it with</p>
<div class="sourceCode" id="cb8"><pre
class="sourceCode eex"><code class="sourceCode elixir"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="op"><</span>p class<span class="op">=</span><span class="st">"alert bg-green-200 border border-green-300 text-green-900 px-4 py-3 rounded relative"</span> role<span class="op">=</span><span class="st">"alert"</span><span class="op">><%=</span> get_flash<span class="fu">(</span><span class="ot">@conn</span>, <span class="va">:info</span><span class="fu">)</span> %<span class="op">></</span>p<span class="op">></span></span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a><span class="op"><</span>p class<span class="op">=</span><span class="st">"alert bg-red-200 border border-red-300 text-red-900 px-4 py-3 rounded relative"</span> role<span class="op">=</span><span class="st">"alert"</span><span class="op">><%=</span> get_flash<span class="fu">(</span><span class="ot">@conn</span>, <span class="va">:error</span><span class="fu">)</span> %<span class="op">></</span>p<span class="op">></span></span></code></pre></div>
<p>We now see the empty alert boxes shown on the site. Let's add CSS to
hide them (<code>assets/css/app.css</code>):</p>
<div class="sourceCode" id="cb9"><pre
class="sourceCode css"><code class="sourceCode css"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="fu">.alert</span><span class="in">:empty</span> { <span class="kw">display</span>: <span class="dv">none</span><span class="op">;</span> }</span></code></pre></div>Facebook Workplace to Slackbuilding simple a service with Elixirhttps://bobek.cz/blog/2019/workplace2slack-index/2019-10-22T02:00:00+02:002024-03-20T13:25:18+01:00Antonín Král<p>We are using Slack extensively at <a
href="https://dtone.engineering">work</a>. But some parts of the company
like FB Workplace for posting updates. As we are <a
href="https://handbook.dtone.engineering">remote</a> company, we need to
keep these type of information accessible. This lead to development of
simple <code>workplace2slack</code> service. I've also got interested in
Elixir, so I took this pet project as a good motivation for starting
with it. Please note, that I'm quite new to the language, so do your
search before committing to stuff mentioned here.</p>
<p>Complete code is available at <a
href="https://github.com/bobek/workplace2slack">github</a>.</p>
<h2 id="getting-started">Getting started</h2>
<h3 id="initial-elixir-project">Initial Elixir project</h3>
<p>I have decided not to use Phoenix as I wanted to explore
<code>Plug</code> a bit more. Not sure, if that was necessary but just
Elixir and Plug turned to be more than enough for this little project. I
assume, that you have <a
href="https://elixir-lang.org/install.html">Elixir already
installed</a>.</p>
<p>Let's start with <code>mix new workplace2slack --sup</code> which
will create a new Elixir project with preconfigured <a
href="https://hexdocs.pm/elixir/Supervisor.html">supervision tree</a>.
As we are building <code>http</code> API server, we need to add means of
handling <code>http</code>. We will do that through <code>plug</code>
(which is pluggable set of processors handling your request, similar to
<code>rack</code> from Ruby world) and <code>cowboy</code> which is go
to <code>http</code> server in Elixir world. We add project dependencies
in <code>mix.exs</code>:</p>
<div class="sourceCode" id="cb1"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a> <span class="kw">defp</span> deps <span class="kw">do</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a> <span class="ot">[</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="va">:plug_cowboy</span>, <span class="st">"~> 2.0"</span><span class="fu">}</span>,</span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="va">:jason</span>, <span class="st">"~> 1.1"</span><span class="fu">}</span>,</span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a> <span class="ot">]</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span></code></pre></div>
<p>I have also added <a
href="https://github.com/michalmuskala/jason">Jason</a> which is one of
the Elixir JSON parsers. We will use it for (de-)serialization of
incoming and outgoing requests. Running <code>mix deps.get</code> wil
fetch all missing dependencies for you.</p>
<h4 id="setting-up-basic-routes">Setting up basic routes</h4>
<p>HTTP routing is facilitated with a router. We add one to
<code>lib/slackolixir/router.ex</code>:</p>
<div class="sourceCode" id="cb2"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="kw">defmodule</span> <span class="cn">Workplace2Slack</span><span class="op">.</span><span class="cn">Router</span> <span class="kw">do</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a> <span class="im">use</span> <span class="cn">Plug</span><span class="op">.</span><span class="cn">Router</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a> <span class="im">alias</span> <span class="cn">Workplace2Slack</span><span class="op">.</span><span class="cn">Workplace</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="im">require</span> <span class="cn">Logger</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> plug <span class="va">:match</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a> plug <span class="cn">Plug</span><span class="op">.</span><span class="cn">RequestId</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a> plug <span class="cn">Plug</span><span class="op">.</span><span class="cn">Logger</span></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a> plug <span class="cn">Plug</span><span class="op">.</span><span class="cn">Parsers</span>,</span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a> <span class="va">parsers:</span> <span class="ot">[</span><span class="va">:json</span>, <span class="va">:urlencoded</span><span class="ot">]</span>,</span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a> <span class="va">json_decoder:</span> <span class="cn">Jason</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a> plug <span class="va">:dispatch</span></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-14"><a href="#cb2-14" aria-hidden="true" tabindex="-1"></a> get <span class="st">"/health"</span> <span class="kw">do</span></span>
<span id="cb2-15"><a href="#cb2-15" aria-hidden="true" tabindex="-1"></a> send_resp<span class="fu">(</span>conn, <span class="dv">200</span>, <span class="st">"OK"</span><span class="fu">)</span></span>
<span id="cb2-16"><a href="#cb2-16" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span>
<span id="cb2-17"><a href="#cb2-17" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-18"><a href="#cb2-18" aria-hidden="true" tabindex="-1"></a> get <span class="st">"/workplace"</span> <span class="kw">do</span></span>
<span id="cb2-19"><a href="#cb2-19" aria-hidden="true" tabindex="-1"></a> <span class="cn">IO</span><span class="op">.</span>inspect conn</span>
<span id="cb2-20"><a href="#cb2-20" aria-hidden="true" tabindex="-1"></a> <span class="kw">case</span> conn<span class="op">.</span>query_params<span class="ot">[</span><span class="st">"hub.challenge"</span><span class="ot">]</span> <span class="kw">do</span></span>
<span id="cb2-21"><a href="#cb2-21" aria-hidden="true" tabindex="-1"></a> challange <span class="kw">when</span> challange <span class="op">></span> <span class="dv">0</span> <span class="op">-></span> send_resp<span class="fu">(</span>conn, <span class="dv">200</span>, challange<span class="fu">)</span></span>
<span id="cb2-22"><a href="#cb2-22" aria-hidden="true" tabindex="-1"></a> <span class="cn">nil</span> <span class="op">-></span> send_resp<span class="fu">(</span>conn, <span class="dv">200</span>, <span class="st">"OK"</span><span class="fu">)</span></span>
<span id="cb2-23"><a href="#cb2-23" aria-hidden="true" tabindex="-1"></a> _ <span class="op">-></span> send_resp<span class="fu">(</span>conn, <span class="dv">200</span>, <span class="st">"OK"</span><span class="fu">)</span></span>
<span id="cb2-24"><a href="#cb2-24" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span>
<span id="cb2-25"><a href="#cb2-25" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span>
<span id="cb2-26"><a href="#cb2-26" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-27"><a href="#cb2-27" aria-hidden="true" tabindex="-1"></a> match _ <span class="kw">do</span></span>
<span id="cb2-28"><a href="#cb2-28" aria-hidden="true" tabindex="-1"></a> <span class="cn">IO</span><span class="op">.</span>inspect conn</span>
<span id="cb2-29"><a href="#cb2-29" aria-hidden="true" tabindex="-1"></a> send_resp<span class="fu">(</span>conn, <span class="dv">404</span>, <span class="st">"not found"</span><span class="fu">)</span></span>
<span id="cb2-30"><a href="#cb2-30" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span>
<span id="cb2-31"><a href="#cb2-31" aria-hidden="true" tabindex="-1"></a><span class="kw">end</span></span></code></pre></div>
<p>Couple things happening here -- as you can see Router is Plug by
itself and also defines plug pipeline (<code>:match</code>,
<code>Plug.RequestId</code>, <code>Plug.Logger</code>,
<code>Plug.Parsers</code>, <code>:dispatch</code>). The really needed
are only <code>:match</code> and <code>:dispatch</code>. I am using
others to:</p>
<ul>
<li>inject a unique <code>request_id</code> or use one received in http
header <code>X-Request-Id</code> it will also be added as metadata to
Logger. This is done via <code>Plug.RequesId</code> plug.</li>
<li>enable logging of requests through <code>Plug.Logger</code>. This
will result in log messages like</li>
</ul>
<pre><code>07:43:00.314 request_id=fffe6acb5ab42be03aecef339a97f690 [info] POST /workplace
07:43:00.314 request_id=fffe6acb5ab42be03aecef339a97f690 [info] Sent 201 in 665µs</code></pre>
<ul>
<li><code>Plug.Parsers</code> will automatically parse request body and
set e.g. <code>body_params</code> to for example Map. So we will not
need to manually parse JSON bodies.</li>
</ul>
<h4 id="start-serving-http-requests">Start serving http requests</h4>
<p>Last missing thing is to run http server as an entry point for our
application. We do it at <code>lib/workplace2slack/application.ex</code>
by adding a new child to our supervision tree:</p>
<div class="sourceCode" id="cb4"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a>children <span class="op">=</span> <span class="ot">[</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="cn">Plug</span><span class="op">.</span><span class="cn">Cowboy</span>, <span class="va">scheme:</span> <span class="va">:http</span>, <span class="va">plug:</span> <span class="cn">Workplace2Slack</span><span class="op">.</span><span class="cn">Router</span>, <span class="va">options:</span> <span class="ot">[</span><span class="va">port:</span> <span class="cn">Application</span><span class="op">.</span>get_env<span class="fu">(</span><span class="va">:workplace2slack</span>, <span class="va">:port</span>, <span class="dv">4000</span><span class="fu">)</span><span class="ot">]</span><span class="fu">}</span>,</span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="ot">]</span></span></code></pre></div>
<h4 id="setup-elixir-releases">Setup Elixir releases</h4>
<p>I want to use freshly stabilized <a
href="https://elixir-lang.org/blog/2019/06/24/elixir-v1-9-0-released/">releases</a>.
Go ahead and initialize releases with <code>mix release.init</code> and
creating basic configuration file (I believe, it is needed to be sure,
that Gigalixir picks up the fact, that you want to use
<code>mix release</code>):</p>
<div class="sourceCode" id="cb5"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> echo <span class="st">"import Config"</span> <span class="op">></span> config/releases.exs</span></code></pre></div>
<p>You should be able to run <code>mix release</code> which will create
the release for your.</p>
<h4 id="testing-it-locally">Testing it locally</h4>
<p>We can now run our release with</p>
<div class="sourceCode" id="cb6"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="ex">_build/dev/rel/workplace2slack/bin/workplace2slack</span> start</span></code></pre></div>
<p>and make our first request</p>
<div class="sourceCode" id="cb7"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> curl http://localhost:4000/health</span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a><span class="ex">OK</span></span></code></pre></div>
<h3 id="deploying-to-gigalixir">Deploying to Gigalixir</h3>
<p>I am using a free tier of <a
href="https://www.gigalixir.com/">Gigalixir</a> for deploying this
project at the moment. We will create a <code>Dockerfile</code> in later
stage to be able to deploy it to our infrastructure. But Gigalixir
allows for super simple deployment (no affiliation :)). Just be aware,
that you can specify region and provider when creating application with
<code>gigalixir create</code>. I have not found a way how to change it
later, so be sure, that you have picked the right one (I am in EU, so I
went for <code>gcp</code> and <code>europe-west1</code> region).</p>
<p>You also need to tell Gigalixir, which version of erlang/elixir you
want to use. That is done via <code>elixir_buildpack.config</code> in
the root of your project:</p>
<div class="sourceCode" id="cb8"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="va">elixir_version</span><span class="op">=</span>1.9</span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a><span class="va">erlang_version</span><span class="op">=</span>21.3</span></code></pre></div>
<p>We are now ready to deploy to Gigalixir. The requirement is to be
able to respond to <code>/health</code> healthiness probes. Which we
already have in our Router. You can monitor logs from your application
with <code>gigalixir logs</code>.</p>
<h2 id="facebook-workplace">Facebook Workplace</h2>
<p>First you will need to setup integration on the Workplace side. Go to
<a href="https://my.workplace.com/work/admin/apps/">Integrations</a> at
your admin section and create a new custom integration. Allow
<code>Read group content</code> permission and configure webhook <img
src="fb_webhooks.png" alt="FB webhooks" /> Also note your
<code>App Secret</code>.</p>
<p>When you hit <code>Save</code>, Facebook will make a <code>GET</code>
request to your defined endpoint to verify that you are able to respond.
You can verify the shared challenge as well, which will get passed to
the endpoint, so you can verify that it is coming from your integration.
To confirm pairing, endpoint has to return value of
<code>hub.challenge</code>.</p>
<blockquote>
<p><strong>Note:</strong> This article is not trying to filter out
messages or route messages from different groups to different channels.
Our use-case for Workplace is so simple, that sending all message to one
Slack channel is enough. Described setup may lead to
<strong>disclosure</strong> of content from private groups!</p>
</blockquote>
<h2 id="slack-bot">Slack bot</h2>
<p>Got to your Slack and <a href="https://api.slack.com/apps">create a
new application</a>. You need to do two things at your
<code>OAuth & Permissions</code> screen:</p>
<ol type="1">
<li>Allow bot to post message with the <code>chat:write:bot</code>
scope. <img src="slack_scopes.png" alt="Slack scope" /></li>
<li>Install the application to your workspace and note value of
<code>OAuth Access Token</code>.</li>
</ol>
<h2 id="processing-messages-from-facebook-workplace">Processing messages
from Facebook Workplace</h2>
<p>Now we have a working service, which can receive events from the
Workplace. Next, we want to parse them, and later on send them to Slack.
The following code is probably beyond need for being split into module
and multiple methods. But let's live with that for now. Add a new method
to our router, which will handle <code>POST</code> from Workplace:</p>
<div class="sourceCode" id="cb9"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a>post <span class="st">"/workplace"</span> <span class="kw">do</span></span>
<span id="cb9-2"><a href="#cb9-2" aria-hidden="true" tabindex="-1"></a> <span class="co"># Verify that request came from Facebook</span></span>
<span id="cb9-3"><a href="#cb9-3" aria-hidden="true" tabindex="-1"></a> <span class="cn">Workplace2Slack</span><span class="op">.</span><span class="cn">HubSignature</span><span class="op">.</span>validate_request!<span class="fu">(</span>conn<span class="fu">)</span></span>
<span id="cb9-4"><a href="#cb9-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-5"><a href="#cb9-5" aria-hidden="true" tabindex="-1"></a> with %<span class="fu">{</span><span class="st">"entry"</span> <span class="op">=></span> <span class="ot">[</span>%<span class="fu">{</span><span class="st">"changes"</span> <span class="op">=></span> <span class="ot">[</span>change<span class="op">|</span>_<span class="ot">]</span><span class="fu">}</span><span class="op">|</span>_<span class="ot">]</span>, <span class="st">"object"</span> <span class="op">=></span> <span class="st">"group"</span><span class="fu">}</span> <span class="op"><-</span> conn<span class="op">.</span>body_params,</span>
<span id="cb9-6"><a href="#cb9-6" aria-hidden="true" tabindex="-1"></a> %<span class="fu">{</span><span class="st">"field"</span> <span class="op">=></span> <span class="st">"posts"</span>, <span class="st">"value"</span> <span class="op">=></span> %<span class="fu">{</span><span class="st">"community"</span> <span class="op">=></span> %<span class="fu">{</span><span class="st">"id"</span> <span class="op">=></span> _community_id<span class="fu">}</span>, <span class="st">"from"</span> <span class="op">=></span> %<span class="fu">{</span><span class="st">"name"</span> <span class="op">=></span> author<span class="fu">}</span>, <span class="st">"message"</span> <span class="op">=></span> message, <span class="st">"permalink_url"</span> <span class="op">=></span> permalink_url<span class="fu">}}</span> <span class="op"><-</span> change <span class="kw">do</span></span>
<span id="cb9-7"><a href="#cb9-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-8"><a href="#cb9-8" aria-hidden="true" tabindex="-1"></a> <span class="co"># Parsing out any image attachments to be added to Slack message</span></span>
<span id="cb9-9"><a href="#cb9-9" aria-hidden="true" tabindex="-1"></a> attachment_urls <span class="op">=</span> <span class="kw">case</span> change <span class="kw">do</span></span>
<span id="cb9-10"><a href="#cb9-10" aria-hidden="true" tabindex="-1"></a> %<span class="fu">{</span><span class="st">"field"</span> <span class="op">=></span> <span class="st">"posts"</span>, <span class="st">"value"</span> <span class="op">=></span> %<span class="fu">{</span> <span class="st">"attachments"</span> <span class="op">=></span> %<span class="fu">{</span><span class="st">"data"</span> <span class="op">=></span> attachments<span class="fu">}}}</span> <span class="op">-></span> <span class="cn">Enum</span><span class="op">.</span>flat_map<span class="fu">(</span>attachments, <span class="kw">fn</span> x <span class="op">-></span> <span class="cn">Workplace</span><span class="op">.</span>extract_image_url<span class="fu">(</span>x<span class="fu">)</span> <span class="kw">end</span><span class="fu">)</span></span>
<span id="cb9-11"><a href="#cb9-11" aria-hidden="true" tabindex="-1"></a> _ <span class="op">-></span> <span class="ot">[]</span></span>
<span id="cb9-12"><a href="#cb9-12" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span>
<span id="cb9-13"><a href="#cb9-13" aria-hidden="true" tabindex="-1"></a> images <span class="op">=</span> attachment_urls</span>
<span id="cb9-14"><a href="#cb9-14" aria-hidden="true" tabindex="-1"></a> <span class="op">|></span> <span class="cn">List</span><span class="op">.</span>flatten</span>
<span id="cb9-15"><a href="#cb9-15" aria-hidden="true" tabindex="-1"></a> <span class="op">|></span> <span class="cn">Enum</span><span class="op">.</span>map<span class="fu">(</span><span class="kw">fn</span> x <span class="op">-></span> %<span class="fu">{</span><span class="va">type:</span> <span class="st">"image"</span>, <span class="va">image_url:</span> x, <span class="va">alt_text:</span> <span class="st">"image"</span><span class="fu">}</span> <span class="kw">end</span><span class="fu">)</span></span>
<span id="cb9-16"><a href="#cb9-16" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-17"><a href="#cb9-17" aria-hidden="true" tabindex="-1"></a> <span class="co"># Assemble the Slack message, channel_id should really be in configuration</span></span>
<span id="cb9-18"><a href="#cb9-18" aria-hidden="true" tabindex="-1"></a> slack_msg <span class="op">=</span></span>
<span id="cb9-19"><a href="#cb9-19" aria-hidden="true" tabindex="-1"></a> %<span class="fu">{</span></span>
<span id="cb9-20"><a href="#cb9-20" aria-hidden="true" tabindex="-1"></a> <span class="va">as_user:</span> <span class="cn">false</span>,</span>
<span id="cb9-21"><a href="#cb9-21" aria-hidden="true" tabindex="-1"></a> <span class="va">channel:</span> <span class="cn">Application</span><span class="op">.</span>get_env<span class="fu">(</span><span class="va">:workplace2slack</span>, <span class="va">:slack_channel</span><span class="fu">)</span>,</span>
<span id="cb9-22"><a href="#cb9-22" aria-hidden="true" tabindex="-1"></a> <span class="va">link_names:</span> <span class="cn">true</span>,</span>
<span id="cb9-23"><a href="#cb9-23" aria-hidden="true" tabindex="-1"></a> <span class="va">parse:</span> <span class="st">"full"</span>,</span>
<span id="cb9-24"><a href="#cb9-24" aria-hidden="true" tabindex="-1"></a> <span class="va">blocks:</span> <span class="ot">[</span></span>
<span id="cb9-25"><a href="#cb9-25" aria-hidden="true" tabindex="-1"></a> %<span class="fu">{</span></span>
<span id="cb9-26"><a href="#cb9-26" aria-hidden="true" tabindex="-1"></a> <span class="va">type:</span> <span class="st">"section"</span>,</span>
<span id="cb9-27"><a href="#cb9-27" aria-hidden="true" tabindex="-1"></a> <span class="va">text:</span> %<span class="fu">{</span></span>
<span id="cb9-28"><a href="#cb9-28" aria-hidden="true" tabindex="-1"></a> <span class="va">type:</span> <span class="st">"mrkdwn"</span>,</span>
<span id="cb9-29"><a href="#cb9-29" aria-hidden="true" tabindex="-1"></a> <span class="va">text:</span> <span class="st">"New post from </span><span class="ot">#{</span>author<span class="ot">}</span><span class="st"> - </span><span class="ot">#{</span>permalink_url<span class="ot">}</span><span class="st">"</span>,</span>
<span id="cb9-30"><a href="#cb9-30" aria-hidden="true" tabindex="-1"></a> <span class="fu">}</span>,</span>
<span id="cb9-31"><a href="#cb9-31" aria-hidden="true" tabindex="-1"></a> <span class="fu">}</span>,</span>
<span id="cb9-32"><a href="#cb9-32" aria-hidden="true" tabindex="-1"></a> %<span class="fu">{</span></span>
<span id="cb9-33"><a href="#cb9-33" aria-hidden="true" tabindex="-1"></a> <span class="va">type:</span> <span class="st">"section"</span>,</span>
<span id="cb9-34"><a href="#cb9-34" aria-hidden="true" tabindex="-1"></a> <span class="va">text:</span> %<span class="fu">{</span></span>
<span id="cb9-35"><a href="#cb9-35" aria-hidden="true" tabindex="-1"></a> <span class="va">type:</span> <span class="st">"mrkdwn"</span>,</span>
<span id="cb9-36"><a href="#cb9-36" aria-hidden="true" tabindex="-1"></a> <span class="va">text:</span> <span class="cn">Workplace</span><span class="op">.</span>sanitize_message<span class="fu">(</span>message<span class="fu">)</span></span>
<span id="cb9-37"><a href="#cb9-37" aria-hidden="true" tabindex="-1"></a> <span class="fu">}</span>,</span>
<span id="cb9-38"><a href="#cb9-38" aria-hidden="true" tabindex="-1"></a> <span class="fu">}</span> <span class="op">|</span> images <span class="ot">]</span>,</span>
<span id="cb9-39"><a href="#cb9-39" aria-hidden="true" tabindex="-1"></a> <span class="fu">}</span></span>
<span id="cb9-40"><a href="#cb9-40" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-41"><a href="#cb9-41" aria-hidden="true" tabindex="-1"></a> <span class="cn">Logger</span><span class="op">.</span>info<span class="fu">(</span><span class="st">"</span><span class="ot">#{</span>permalink_url<span class="ot">}</span><span class="st"> by </span><span class="ot">#{</span>author<span class="ot">}</span><span class="st">"</span><span class="fu">)</span></span>
<span id="cb9-42"><a href="#cb9-42" aria-hidden="true" tabindex="-1"></a> <span class="co"># Following line will send a "job" to job queue, will be covered in next section</span></span>
<span id="cb9-43"><a href="#cb9-43" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="va">:send_message</span>, <span class="ot">[</span>slack_msg, <span class="cn">Logger</span><span class="op">.</span>metadata<span class="fu">()</span><span class="ot">[</span><span class="va">:request_id</span><span class="ot">]]</span><span class="fu">}</span> <span class="op">|></span> <span class="cn">Honeydew</span><span class="op">.</span>async<span class="fu">(</span><span class="va">:slack</span><span class="fu">)</span></span>
<span id="cb9-44"><a href="#cb9-44" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span>
<span id="cb9-45"><a href="#cb9-45" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-46"><a href="#cb9-46" aria-hidden="true" tabindex="-1"></a> send_resp<span class="fu">(</span>conn, <span class="dv">201</span>, <span class="st">"OK"</span><span class="fu">)</span></span>
<span id="cb9-47"><a href="#cb9-47" aria-hidden="true" tabindex="-1"></a><span class="kw">end</span></span></code></pre></div>
<p>Function <code>Workplace.extract_image_url</code> is a little helper
which parses out URLs of images attached to the original post. So those
can be then attached to Slack notification message. Check the <a
href="https://github.com/bobek/workplace2slack/blob/master/lib/workplace2slack/workplace.ex"><code>Workplace2Slack.Workplace</code></a>
module for the actual implementation.</p>
<h3 id="verifying-that-message-came-from-facebook">Verifying that
message came from Facebook</h3>
<p>Every message from Facebook is signed with signature being carried in
<code>X-Hub-Signature</code> http header. We definitely want to verify
it, otherwise we would start posting spam to our Slack.</p>
<p>First of all, we need to get access to original body of the
<code>POST</code> request. That is discarded by <code>Plug.Parser</code>
during processing. Fortunately, an option for defining a <a
href="https://hexdocs.pm/plug/Plug.Parsers.html#module-custom-body-reader">custom
body parser</a> has been recently added. We just need to extend our plug
pipeline in router as follows:</p>
<div class="sourceCode" id="cb10"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb10-1"><a href="#cb10-1" aria-hidden="true" tabindex="-1"></a> plug <span class="cn">Plug</span><span class="op">.</span><span class="cn">Parsers</span>,</span>
<span id="cb10-2"><a href="#cb10-2" aria-hidden="true" tabindex="-1"></a> <span class="va">parsers:</span> <span class="ot">[</span><span class="va">:json</span>, <span class="va">:urlencoded</span><span class="ot">]</span>,</span>
<span id="cb10-3"><a href="#cb10-3" aria-hidden="true" tabindex="-1"></a> <span class="va">body_reader:</span> <span class="fu">{</span><span class="cn">Workplace2Slack</span><span class="op">.</span><span class="cn">Plug</span><span class="op">.</span><span class="cn">CacheBodyReader</span>, <span class="va">:read_body</span>, <span class="ot">[]</span><span class="fu">}</span>,</span>
<span id="cb10-4"><a href="#cb10-4" aria-hidden="true" tabindex="-1"></a> <span class="va">json_decoder:</span> <span class="cn">Jason</span></span></code></pre></div>
<p>and define <code>CacheBodyReader</code> in
<code>lib/plug/cache_body_reader.ex</code>:</p>
<div class="sourceCode" id="cb11"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true" tabindex="-1"></a><span class="kw">defmodule</span> <span class="cn">Workplace2Slack</span><span class="op">.</span><span class="cn">Plug</span><span class="op">.</span><span class="cn">CacheBodyReader</span> <span class="kw">do</span></span>
<span id="cb11-2"><a href="#cb11-2" aria-hidden="true" tabindex="-1"></a> <span class="kw">def</span> read_body<span class="fu">(</span>conn, opts<span class="fu">)</span> <span class="kw">do</span></span>
<span id="cb11-3"><a href="#cb11-3" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="va">:ok</span>, body, conn<span class="fu">}</span> <span class="op">=</span> <span class="cn">Plug</span><span class="op">.</span><span class="cn">Conn</span><span class="op">.</span>read_body<span class="fu">(</span>conn, opts<span class="fu">)</span></span>
<span id="cb11-4"><a href="#cb11-4" aria-hidden="true" tabindex="-1"></a> conn <span class="op">=</span> update_in<span class="fu">(</span>conn<span class="op">.</span>assigns<span class="ot">[</span><span class="va">:body_raw</span><span class="ot">]</span>, <span class="op">&</span><span class="ot">[</span>body <span class="op">|</span> <span class="fu">(</span><span class="op">&</span><span class="dv">1</span> <span class="op">||</span> <span class="ot">[]</span><span class="fu">)</span><span class="ot">]</span><span class="fu">)</span></span>
<span id="cb11-5"><a href="#cb11-5" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="va">:ok</span>, body, conn<span class="fu">}</span></span>
<span id="cb11-6"><a href="#cb11-6" aria-hidden="true" tabindex="-1"></a> <span class="kw">end</span></span>
<span id="cb11-7"><a href="#cb11-7" aria-hidden="true" tabindex="-1"></a><span class="kw">end</span></span></code></pre></div>
<p>This copies original body into <code>body_raw</code> attribute of the
<code>conn</code> passed around. Which seems to be suboptimal solution,
which may be reworked later on.</p>
<p>The actual validator is then called at the very top of message
handled. My initial implementation was creating a custom
<code>Plug</code> to do so, but then I realized that I really need it
only for one call. Otherwise I would keep it and implement an option
like <code>except_path</code> to disable it for certain routes. Code
itself is straightforward and checks that <code>HMAC</code> signature of
body and shared secret (that's the <code>App Secret</code> you can get
from the Workplace integration screen) matches with value of the header.
You can check <a
href="https://github.com/bobek/workplace2slack/blob/master/lib/workplace2slack/hub_signature.ex"><code>Workplace2Slack.HubSignature</code></a>
for the actual implementation.</p>
<h2 id="sending-messages-to-slack">Sending messages to Slack</h2>
<p>This could have been easily implemented as a <a
href="https://elixir-lang.org/getting-started/mix-otp/genserver.html">GenServer</a>,
but I wanted to explore proper job queue a bit. <a
href="https://github.com/koudelka/honeydew">Honeydew</a> seems to be
quite popular within the community. Only gotcha is that default
in-memory queue utilizes Mnesia, so <a
href="https://github.com/koudelka/honeydew/issues/94">don't forget</a>
to add <code>:mnesia</code> into your <code>extra_applications</code>
within <code>mix.exs</code> otherwise it will not get included into the
release.</p>
<p>We will also need to add some dependencies, already mentioned
<code>honeydew</code> and <code>httpoison</code> for making
<code>http</code> calls towards Slack API. Just add the following to
<code>deps</code> in your <code>mix.exs</code>:</p>
<div class="sourceCode" id="cb12"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="va">:httpoison</span>, <span class="st">"~> 1.6"</span><span class="fu">}</span>,</span>
<span id="cb12-2"><a href="#cb12-2" aria-hidden="true" tabindex="-1"></a> <span class="fu">{</span><span class="va">:honeydew</span>, <span class="st">"~> 1.4"</span><span class="fu">}</span>,</span></code></pre></div>
<p>We just start a new queue and worker in our <a
href="https://github.com/bobek/workplace2slack/blob/master/lib/workplace2slack/application.ex"><code>application.ex</code></a>:</p>
<div class="sourceCode" id="cb13"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a> <span class="va">:ok</span> <span class="op">=</span> <span class="cn">Honeydew</span><span class="op">.</span>start_queue<span class="fu">(</span><span class="va">:slack</span><span class="fu">)</span></span>
<span id="cb13-2"><a href="#cb13-2" aria-hidden="true" tabindex="-1"></a> <span class="va">:ok</span> <span class="op">=</span> <span class="cn">Honeydew</span><span class="op">.</span>start_workers<span class="fu">(</span><span class="va">:slack</span>, <span class="fu">{</span><span class="cn">Workplace2Slack</span><span class="op">.</span><span class="cn">SlackWorker</span>, <span class="ot">[</span><span class="cn">Application</span><span class="op">.</span>get_env<span class="fu">(</span><span class="va">:workplace2slack</span>, <span class="va">:slack_token</span>, <span class="st">""</span><span class="fu">)</span><span class="ot">]</span><span class="fu">})</span></span></code></pre></div>
<p>and also define worker <a
href="https://github.com/bobek/workplace2slack/blob/master/lib/workplace2slack/slack_worker.ex">Workplace2Slack.SlackWorker</a>,
which just send the message to Slack via their API. And that's it.</p>
<h2 id="secrets-and-releases">Secrets and releases</h2>
<p>As you may guess, I had secrets (slack token and FB App Secret)
hard-coded in the application configuration. It turns out, that releases
make live configuration much easier then with previous version of
Elixir. Secret sauce is the fact that <code>config/releases.exs</code>
will get <em>evaluated</em> when the release is going to be started. So
the following (complete content of <code>config/releases.exs</code>)
will do exactly what you expect:</p>
<div class="sourceCode" id="cb14"><pre
class="sourceCode elixir"><code class="sourceCode elixir"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> <span class="cn">Config</span></span>
<span id="cb14-2"><a href="#cb14-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb14-3"><a href="#cb14-3" aria-hidden="true" tabindex="-1"></a>config <span class="va">:workplace2slack</span>,</span>
<span id="cb14-4"><a href="#cb14-4" aria-hidden="true" tabindex="-1"></a> <span class="va">slack_token:</span> <span class="cn">System</span><span class="op">.</span>fetch_env!<span class="fu">(</span><span class="st">"SLACK_TOKEN"</span><span class="fu">)</span>,</span>
<span id="cb14-5"><a href="#cb14-5" aria-hidden="true" tabindex="-1"></a> <span class="va">fb_app_secret:</span> <span class="cn">System</span><span class="op">.</span>fetch_env!<span class="fu">(</span><span class="st">"FB_APP_SECRET"</span><span class="fu">)</span></span></code></pre></div>
<p>If you are running on Gigalixir, you can set your env with</p>
<div class="sourceCode" id="cb15"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb15-1"><a href="#cb15-1" aria-hidden="true" tabindex="-1"></a><span class="ex">gigalixir</span> SLACK_TOKEN=<span class="st">"secret_slack_token"</span> FB_APP_SECRET=<span class="st">"secret_fb_workplace_app_secret"</span></span></code></pre></div>
<h2 id="result">Result</h2>
<p>So next time, somebody posts an update to Workplace such as</p>
<figure>
<img src="fb-post.png" alt="hard at work" />
<figcaption aria-hidden="true">hard at work</figcaption>
</figure>
<p>you will get a nice notification in your Slack channel</p>
<figure>
<img src="slack-post.png" alt="hard at work at slack" />
<figcaption aria-hidden="true">hard at work at slack</figcaption>
</figure>
<p>No need to have Workplace tab opened anymore!</p>Simple GIT identities (overrides in general) managementhttps://bobek.cz/blog/2019/git-identities/2019-07-26T02:00:00+02:002024-03-20T13:25:18+01:00Antonín Král<p>I need to be committing intro different repositories under different
user identities. To Open Source ones with my private email, while using
work email for work-related commits/repos.</p>
<p>Typically, you would use repo-specific identities. For example (my
work repo):</p>
<div class="sourceCode" id="cb1"><pre
class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="fu">git</span> config user.name <span class="st">"Antonin Kral"</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="fu">git</span> config user.email antonin.kral@dtone.com</span></code></pre></div>
<p>Now, that is error-prone and painful. I have discovered a neat
feature of git (since 2.13) which allows for conditional inclusion of
configuration files. Let's assume the following configuration files:</p>
<div class="sourceCode" id="cb2"><pre
class="sourceCode toml"><code class="sourceCode toml"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co"># ~/.gitconfig</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="kw">[user]</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="dt">email</span> <span class="op">=</span> <span class="dt">a.kral</span><span class="er">@</span><span class="dt">bobek.cz</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> <span class="dt">name</span> <span class="op">=</span> <span class="dt">Antonin</span> <span class="dt">Kral</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="kw">[includeIf</span> <span class="kw">"</span><span class="dt">gitdir:~/Sources/dtone/"</span><span class="kw">]</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a> <span class="dt">path</span> <span class="op">=</span> <span class="dt">.gitconfig_dtone</span></span></code></pre></div>
<div class="sourceCode" id="cb3"><pre
class="sourceCode toml"><code class="sourceCode toml"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="co"># ~/.gitconfig_dtone</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="kw">[user]</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> <span class="dt">name</span> <span class="op">=</span> <span class="dt">Antonin</span> <span class="dt">Kral</span></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> <span class="dt">email</span> <span class="op">=</span> <span class="dt">antonin.kral</span><span class="er">@</span><span class="dt">dtone.com</span></span></code></pre></div>
<p>And that's it. Every time I work with repo, which is cloned under
<code>~/Sources/dtone</code>, my identity will be automatically changed
to work one. This is not limited to identities; you can override
anything you want.</p>