<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>Eyayaw Beze</title>
<link>https://eyayaw.github.io/blog.html</link>
<atom:link href="https://eyayaw.github.io/blog.xml" rel="self" type="application/rss+xml"/>
<description>Eyayaw Beze&#39;s personal website</description>
<generator>quarto-1.10.18</generator>
<lastBuildDate>Sun, 06 Sep 2026 22:00:00 GMT</lastBuildDate>
<item>
  <title>jobrake for the job</title>
  <dc:creator>Eyayaw Beze</dc:creator>
  <link>https://eyayaw.github.io/posts/jobrake-for-the-job/</link>
  <description><![CDATA[ 




<p>I’ve been looking for a simple tool for getting new job postings from job boards, because I was tired of doomscrolling on the LinkedIn Jobs tab. Oh yeah, [sg]etting job alerts is equally overwhelming.<sup>1</sup></p>
<p>Who to blame, but I’ve not found any that is “reliable”, simple, and free.<sup>2</sup></p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Tip
</div>
</div>
<div class="callout-body-container callout-body">
<p>I spend bonkers amount of time trying to “automate” my job search, but less on actually searching. So, heed my advice on this matter. 😅</p>
</div>
</div>
<p>So <code>jobrake</code> (<a href="https://github.com/eyayaw/jobrake"><i class="bi bi-github"></i></a>) is an ambitious attempt to address these challenges.</p>
<p>It’s a simple CLI tool and Python package that fetches job postings from Indeed and LinkedIn and returns every result as the same small <a href="https://github.com/eyayaw/jobrake/blob/main/src/jobrake/models.py#L30-L60"><code>Job</code></a> dictionary. There is no login, no browser automation, just simple requests via <code>httpx</code>.</p>
<p>I intend <code>jobrake</code> to be simple and light<sup>3</sup>. It searches, normalizes, and returns jobs. That is it. It helps you swing the rake at the big piles—LinkedIn and Indeed. So, currently, it does not support other boards such as Glassdoor or ZipRecruiter.</p>
<section id="installation" class="level2">
<h2 class="anchored" data-anchor-id="installation">Installation</h2>
<p><code>jobrake</code> is not on PyPI so install it from GitHub with <a href="https://astral.sh/uv"><code>uv</code></a>. Please read the <a href="https://docs.astral.sh/uv/getting-started/installation/">installation guides</a> for your system.</p>
<blockquote class="blockquote">
<p><em>I assume unix-based systems hereafter, sorry Windows-(or dare I say WSL-) people, you’re on your own—read the uv docs, adapt all the code wherever necessary; you can do it!</em></p>
</blockquote>
<p>I personally use <code>jobrake</code> through the CLI, so you can install it as a tool, too. <code>uv</code> installs <code>jobrake</code> and adds it to your <code>${PATH}</code>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> tool install git+https://github.com/eyayaw/jobrake</span>
<span id="cb1-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># For ephemeral use</span></span>
<span id="cb1-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># uvx git+https://github.com/eyayaw/jobrake</span></span></code></pre></div></div>
</section>
<section id="a-first-rake" class="level2">
<h2 class="anchored" data-anchor-id="a-first-rake">A first rake</h2>
<p>Say you’re an <em>economist</em> looking for <em>data science</em> roles in the <em>Netherlands</em>, just like <mark>me</mark><sup>4</sup>.</p>
<div class="columns" style="font-size:0.9em;">
<div class="column">
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode sh code-overflow-scroll code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> linkedin <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb2-2">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--query</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"economist OR data scientist"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb2-3">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--location</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"amsterdam, netherlands"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb2-4">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--max-age</span> 24 <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb2-5">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--results</span> 5 <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb2-6">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--output</span> data/jobs-linkedin.csv <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb2-7">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--details</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>/dev/null</span></code></pre></div></div>
</details>
</div>
</div><div class="column">
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb3-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> indeed <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb3-2">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-q</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"economist OR data scientist"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb3-3">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-c</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"netherlands"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb3-4">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"amsterdam"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb3-5">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-a</span> 24 <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb3-6">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-n</span> 5 <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb3-7">        <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-o</span> data/jobs-indeed.json <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>/dev/null</span></code></pre></div></div>
</details>
</div>
</div>
</div>
<p><em>Note: If you’re new to CLI lingo, left and right are the same interface twice: long flags spelled out for LinkedIn, and their short aliases for Indeed. Remove the redirect (<code>2&gt;/dev/null</code>) to see progress updates and other info.</em></p>
<details>
<summary>
Expand for the results
</summary>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-1-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-1" aria-controls="tabset-1-1" aria-selected="true" href="">LinkedIn</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-2" aria-controls="tabset-1-2" aria-selected="false" href="">Indeed</a></li></ul>
<div class="tab-content">
<div id="tabset-1-1" class="tab-pane active" aria-labelledby="tabset-1-1-tab">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># The first three rows, some fields, description truncated</span></span>
<span id="cb4-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">duckdb</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--markdown</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;&lt;quack</span></span>
<span id="cb4-3"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">select</span></span>
<span id="cb4-4"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    date,</span></span>
<span id="cb4-5"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    title: printf('[%s](%s)', title, url),</span></span>
<span id="cb4-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    company: printf('[%s](%s)', company, company_url),</span></span>
<span id="cb4-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    location,</span></span>
<span id="cb4-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    employment_type,</span></span>
<span id="cb4-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    applicants,</span></span>
<span id="cb4-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    description: regexp_replace(description, '</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\s</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">+', ' ', 'g')[:500] || '...'</span></span>
<span id="cb4-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">from 'data/jobs-linkedin.csv'</span></span>
<span id="cb4-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">order by date desc</span></span>
<span id="cb4-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">limit 3;</span></span>
<span id="cb4-14"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">quack</span></span></code></pre></div></div>
</details>
<table class="caption-top table">
<thead>
<tr class="header">
<th>date</th>
<th>title</th>
<th>company</th>
<th>location</th>
<th>employment_type</th>
<th style="text-align: right;">applicants</th>
<th>description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>2026-09-07</td>
<td><a href="https://nl.linkedin.com/jobs/view/data-scientist-at-i4talent-detachering-4464088788">Data Scientist</a></td>
<td><a href="https://nl.linkedin.com/company/i4talent">i4talent detachering</a></td>
<td>Amsterdam, North Holland, Netherlands</td>
<td>full_time</td>
<td style="text-align: right;">25</td>
<td>Als Data Scientist bij i4talent werk je aan opdrachten bij organisaties in de Randstad. Je verdient tussen de €3.500 en €5.720 bruto per maand en komt in dienst bij i4talent, terwijl je als consultant werkt aan voorspellende modellen, machine learning, Python en cloudomgevingen zoals Azure of AWS. Je werkt op projecten waar data echt verschil kan maken. Denk aan forecasting, klantgedrag, risicoanalyse, optimalisatie of automatisering. Wij kijken samen welke opdracht past bij jouw ervaring, inter…</td>
</tr>
<tr class="even">
<td>2026-09-07</td>
<td><a href="https://nl.linkedin.com/jobs/view/short-term-forecasting-data-scientist-at-eneco-4462948463">Short Term Forecasting Data Scientist</a></td>
<td><a href="https://nl.linkedin.com/company/eneco">Eneco</a></td>
<td>Rotterdam, South Holland, Netherlands</td>
<td>full_time</td>
<td style="text-align: right;">25</td>
<td>Good to know: you can easily save this vacancy using the print button at the top of the page. After the closing date, this vacancy will be removed from our website. Combine operational responsibility with long-term data science and forecasting projects Develop and improve forecasting products that directly support short-term energy trading Work closely with traders in a fast feedback environment, where you monitor performance, answer questions, and resolve incidents Why choose Eneco? The energy …</td>
</tr>
<tr class="odd">
<td>2026-09-07</td>
<td><a href="https://nl.linkedin.com/jobs/view/senior-data-science-consultant-at-sia-4461879641">Senior Data Science Consultant</a></td>
<td><a href="https://fr.linkedin.com/company/sia-partners">Sia</a></td>
<td>Rotterdam, South Holland, Netherlands</td>
<td>full_time</td>
<td style="text-align: right;">25</td>
<td>Sia is a next-generation, global management consulting group born digital, augmented by data, enhanced by creativity, and driven by responsibility. We partner with clients to resolve challenges and capitalize on opportunities. We believe that in today’s world of change and disruption, optimism is a force multiplier. Job Description To support its development, Sia is recruiting senior consultants within its data science team whose vocation will be to take charge, in synergy with our different pra…</td>
</tr>
</tbody>
</table>
</div>
<div id="tabset-1-2" class="tab-pane" aria-labelledby="tabset-1-2-tab">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># The first three jobs, all fields, description truncated</span></span>
<span id="cb5-2"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'```json'</span></span>
<span id="cb5-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">duckdb</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-json</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;&lt;quack</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jq</span></span>
<span id="cb5-4"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">select * replace (regexp_replace(description, '</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\s</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">+', ' ', 'g')[:500] || '...' as description)</span></span>
<span id="cb5-5"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">from 'data/jobs-indeed.json'</span></span>
<span id="cb5-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">order by date desc</span></span>
<span id="cb5-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">limit 3;</span></span>
<span id="cb5-8"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">quack</span></span>
<span id="cb5-9"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">echo</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'```'</span></span></code></pre></div></div>
</details>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode json code-with-copy"><code class="sourceCode json"><span id="cb6-1"><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span></span>
<span id="cb6-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb6-3">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"indeed"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-4">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2911e6df5f3ebd6e"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-5">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://nl.indeed.com/viewjob?jk=2911e6df5f3ebd6e"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-6">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Commercial Data &amp; AI Lead"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-7">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Schuberg Philis"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-8">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Schiphol, NH, NL"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-9">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-07"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-10">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"description"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Helping organizations turn data ambition into decisions they can stand behind The Commercial Data &amp; AI Lead at Schuberg Philis is a commercially driven technologist who bridges proposition strategy and customer execution. Embedded in Greater Customer Teams (GCTs), the Commercial Data &amp; AI Lead translates customer challenges into compelling, standardized data and AI solutions, accelerates sales cycles through technical expertise, and ensures that what we propose is both desirable for customers an..."</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-11">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company_url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://nl.indeed.com/cmp/Schuberg-Philis"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-12">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company_logo"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://d2q79iu7y748jz.cloudfront.net/s/_squarelogo/256x256/44e5b2f3c02ec88d3444a9c6e2e00c00"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-13">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"city"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Schiphol"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-14">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"region"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NH"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-15">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"country_code"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NL"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-16">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"latitude"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">52.27497</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-17">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"longitude"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4.750813</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-18">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"posted_at"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-07 05:00:00"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-19">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"apply_url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://schubergphilis.com/careers/7652253003?greenhouse=1&amp;gh_jid=7652253003"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-20">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"employment_type"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">null</span></span>
<span id="cb6-21">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-22">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb6-23">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"indeed"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-24">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"a4e73cd1582140e9"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-25">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://nl.indeed.com/viewjob?jk=a4e73cd1582140e9"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-26">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Data Consultant"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-27">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Incentro"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-28">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Utrecht, UT, NL"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-29">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-07"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-30">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"description"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wil jij niet alleen met data werken, maar vooral begrijpen wat een organisatie nodig heeft en hierover adviseren? Hoe zorg je ervoor dat cateringbedrijven precies genoeg maaltijden bereiden zonder onnodige verspilling? Hoe optimaliseer je een logistiek netwerk zodat vrachtwagens minder kilometers rijden? Of hoe help je een organisatie om AI-ready te worden met een stevig datafundament? Dit zijn voorbeelden van vraagstukken waar jij als Data Consultant bij Incentro mee aan de slag kunt. Data comb..."</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-31">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company_url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://nl.indeed.com/cmp/Incentro"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-32">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company_logo"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://d2q79iu7y748jz.cloudfront.net/s/_squarelogo/256x256/78dcf2090eeff9c0e3df827d29eb63fe"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-33">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"city"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Utrecht"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-34">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"region"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"UT"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-35">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"country_code"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NL"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-36">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"latitude"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">52.06974</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-37">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"longitude"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5.076462</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-38">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"posted_at"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-07 05:00:00"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-39">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"apply_url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://careers.incentro.com/jobs/8328148-data-consultant"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-40">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"employment_type"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"part_time"</span></span>
<span id="cb6-41">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-42">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb6-43">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"indeed"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-44">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"c7e5ba843602ed0e"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-45">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://nl.indeed.com/viewjob?jk=c7e5ba843602ed0e"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-46">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Model &amp; Governance Specialist Anti-Financial Crime"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-47">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ASN Bank"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-48">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Utrecht, UT, NL"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-49">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-07"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-50">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"description"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Plaats Utrecht Werktijden 36 uur Education HBO € 5.179,72 € 6.906,29 Als Model &amp; Governance Specialist bij ASN Bank in Utrecht draag jij bij aan verantwoorde AI-inzet tegen financiële criminaliteit, met impact op mens én maatschappij. Wat ga je doen Jij speelt een sleutelrol in het verantwoord ontwikkelen en beheren van AI-modellen binnen het Anti-Financial Crime domein. Met jouw kennis van AI, modelrisico's en wet- en regelgeving zorg jij ervoor dat innovatie én integriteit hand in hand gaan. J..."</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-51">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company_url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://nl.indeed.com/cmp/Asn-Bank"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-52">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company_logo"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://d2q79iu7y748jz.cloudfront.net/s/_squarelogo/256x256/8384201f53ab16ae688d4779f9f5a1b6"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-53">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"city"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Utrecht"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-54">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"region"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"UT"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-55">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"country_code"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"NL"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-56">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"latitude"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">52.09083</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-57">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"longitude"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5.12222</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-58">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"posted_at"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-07 05:00:00"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-59">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"apply_url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://werkenbij.asnbank.nl/nl/nl/job/6977/Model-Governance-Specialist-Anti-Financial-Crime"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb6-60">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"employment_type"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"full_time"</span></span>
<span id="cb6-61">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span></span>
<span id="cb6-62"><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
</div>
</div>
</div>
</details>
<p>By default, results are compact JSON on stdout:</p>
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb7-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> linkedin <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb7-2">-q <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"data scientist"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb7-3">-l <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"london, england"</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-n</span> 5 <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span>/dev/null <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb7-4"><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">|</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jq</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-r</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'"```json\n\(.)\n```"'</span></span></code></pre></div></div>
</details>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode json code-with-copy"><code class="sourceCode json"><span id="cb8-1"><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"linkedin"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"4460492939"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://uk.linkedin.com/jobs/view/data-scientist-or-senior-data-scientist-at-hoare-lea-4460492939"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Data Scientist or Senior Data Scientist"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Hoare Lea"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"London Area, United Kingdom"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-03"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"linkedin"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"4437696372"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://uk.linkedin.com/jobs/view/data-scientist-ai-advanced-analytics-public-sector-at-ibm-4437696372"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Data Scientist - AI &amp; Advanced Analytics (Public Sector)"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"IBM"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"London, England, United Kingdom"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-03"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"linkedin"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"4451513631"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://uk.linkedin.com/jobs/view/data-scientist-maidenhead-%C2%A340-45k-at-solas-it-recruitment-4451513631"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Data Scientist - Maidenhead (£40-45k"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Solas IT Recruitment"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Maidenhead, England, United Kingdom"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-02"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"linkedin"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"4463387995"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://uk.linkedin.com/jobs/view/data-scientist-at-clearscore-4463387995"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Data Scientist"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ClearScore"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"London, England, United Kingdom"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-04"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"site"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"linkedin"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"id"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"4459327487"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"url"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://uk.linkedin.com/jobs/view/data-scientist-maidenhead-%C2%A340-45k-at-solas-it-recruitment-4459327487"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"title"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Data Scientist - Maidenhead (£40-45k)"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"company"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Solas IT Recruitment"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"location"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Maidenhead, England, United Kingdom"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"date"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2026-09-02"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
</section>
<section id="descriptions-and-other-details" class="level2">
<h2 class="anchored" data-anchor-id="descriptions-and-other-details">Descriptions and other details</h2>
<p>Every job result includes the same <strong>identity</strong> and <strong>summary fields</strong>, with <code>null</code> for unavailable summary values. <strong>Detail fields</strong> appear only when the job board supplies a value<sup>5</sup>. This keeps the common fields predictable, while keeping optional details compact.</p>
<p>The two sites differ in how much info they display upfront. Indeed includes every supported detail field in its search results. However, LinkedIn search cards contain only basic summaries, so <code>jobrake</code> “hydrates” them—fetches each posting page—when you ask for details such as the job description with <code>--details | -d</code>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb9-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> linkedin <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb9-2">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-q</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"economist"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb9-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-l</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"amsterdam, north holland, netherlands"</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\</span></span>
<span id="cb9-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-d</span></span></code></pre></div></div>
<p>The difference also affects speed. Indeed searches are fast because we use its mobile app API. LinkedIn’s guest search API is rate-limited, so <code>jobrake</code> uses a token bucket to pace requests. The LinkedIn scraper takes longer by design, so keep blinking at the progress bar.<sup>6</sup></p>
<p><code>jobrake</code> caches fetched details in your <mark>user cache directory</mark>. Repeated searches can reuse those details. If the cache is missing or damaged, <code>jobrake</code> fetches them again. Use <code>--no-cache</code> to disable caching.</p>
</section>
<section id="using-jobrake-from-python" class="level2">
<h2 class="anchored" data-anchor-id="using-jobrake-from-python">Using <code>jobrake</code> from Python</h2>
<p>Add it to your project dependencies with <code>uv</code>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb10-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># create a .venv (uv venv .venv) or init a project uv init</span></span>
<span id="cb10-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">uv</span> add git+https://github.com/eyayaw/jobrake</span></code></pre></div></div>
<p>The package entry point is asynchronous:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb11-1"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> asyncio</span>
<span id="cb11-2"></span>
<span id="cb11-3"><span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">from</span> jobrake <span class="im" style="color: #00769E;
background-color: null;
font-style: inherit;">import</span> scrape</span>
<span id="cb11-4"></span>
<span id="cb11-5"></span>
<span id="cb11-6"><span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">async</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">def</span> main():</span>
<span id="cb11-7">    jobs <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">await</span> scrape(</span>
<span id="cb11-8">        <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"indeed"</span>,</span>
<span id="cb11-9">        query<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"data scientist"</span>,</span>
<span id="cb11-10">        country<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"netherlands"</span>,</span>
<span id="cb11-11">        location<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"amsterdam"</span>,</span>
<span id="cb11-12">        results<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>,</span>
<span id="cb11-13">    )</span>
<span id="cb11-14"></span>
<span id="cb11-15">    <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> job <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> jobs:</span>
<span id="cb11-16">        <span class="bu" style="color: null;
background-color: null;
font-style: inherit;">print</span>(job[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"date"</span>], job[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"title"</span>], job[<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"company"</span>])</span>
<span id="cb11-17"></span>
<span id="cb11-18">asyncio.run(main())</span></code></pre></div></div>
</section>
<section id="what-if" class="level2">
<h2 class="anchored" data-anchor-id="what-if">What if …?</h2>
<div class="callout callout-style-default callout-warning callout-titled" title="A caveat">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Warning</span>A caveat
</div>
</div>
<div class="callout-body-container callout-body">
<p>Breaking in no time: Scrapers do not make promises, right? 🙃 <code>jobrake</code> pings unofficial(ly) (api) endpoints and relies on small HTML markups, so it lives at the mercy of LinkedIn and Indeed. Changes of API endpoints or markups can break it any day. If a request fails, it’s probably them, not us. <a href="https://github.com/eyayaw/jobrake/issues">File an issue</a>, and offer fixes.</p>
</div>
</div>
<p>The code is available at <a href="https://github.com/eyayaw/jobrake">github.com/eyayaw/jobrake</a>.</p>
<p>Happy searching, I mean, raking!</p>
<hr style="border: 0; height: 1px; width: 85%; margin: 3.5rem auto
  2.5rem; background: linear-gradient(to right, transparent, #aaa,
  transparent);">
</section>
<section id="appendix" class="level1">
<h1>Appendix</h1>
<section id="output-formats" class="level2">
<h2 class="anchored" data-anchor-id="output-formats">Output formats</h2>
<p>Explicit output-format selection:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb12-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> ... <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--format</span> json</span>
<span id="cb12-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> ... <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-f</span> jsonl</span>
<span id="cb12-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> ... <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-f</span> csv</span></code></pre></div></div>
<p>Files still infer their format from the extension.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb13-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> ... <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-o</span> jobs.csv</span>
<span id="cb13-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> ... <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-o</span> jobs.jsonl</span></code></pre></div></div>
<p>An explicit <code>--format</code> overrides that inference, i.e., you can do <code>-f jsonl -o jobs.txt</code> but you get JSON Lines in a <code>.txt</code> file.</p>
</section>
<section id="place-name-resolution" class="level2">
<h2 class="anchored" data-anchor-id="place-name-resolution">Place name resolution</h2>
<p><code>jobrake</code> can also resolve place names from the CLI. LinkedIn needs an unambiguous location to return results. Look up the place first, then use its display name in your search:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb14-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> places linkedin <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"amsterdam"</span></span>
<span id="cb14-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Then search with -l "Amsterdam, North Holland, Netherlands"</span></span></code></pre></div></div>
<p>For Indeed, include the country:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb15-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">jobrake</span> places indeed <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-c</span> netherlands <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"amsterdam"</span></span></code></pre></div></div>


</section>
</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>Indeed’s interface is annoying as hell anyway.↩︎</p></li>
<li id="fn2"><p>I tried <a href="https://github.com/speedyapply/JobSpy"><code>jobspy</code></a> for some time. However, the API is unstable and bundles unnecessary <em>downstream</em> <a href="https://github.com/speedyapply/JobSpy/blob/fda080a373e8226f3fd60635323f5da9af9892b1/pyproject.toml#L19-L28">dependencies</a> such as <code>pandas</code> and <code>numpy</code>. Unfortunately, it also appears not actively maintained. Nonetheless, <code>jobrake</code> owes a lot to <code>jobspy</code>.↩︎</p></li>
<li id="fn3"><p>It depends on <code>httpx</code> and <code>beautifulsoup</code>. I leave it to you if you need downstream poking with the job search data with <code>pandas</code> and co. I had to point this because <code>jobspy</code> depends on lots of such packages.↩︎</p></li>
<li id="fn4"><p><mark>Btw, seriously looking for a job. Hire me.</mark>↩︎</p></li>
<li id="fn5"><p>However, if you request CSV via <code>--format csv</code> or <code>--output jobs.csv</code>, all fields appear as columns.↩︎</p></li>
<li id="fn6"><p>Hey, let’s talk LinkedIn at its own pace, as slow as we can. 😅 As they say, “<em>If you want to go fast, go alone. If you want to go far, go together.</em>”↩︎</p></li>
</ol>
</section></div> ]]></description>
  <category>package</category>
  <category>jobrake</category>
  <category>python</category>
  <category>cli</category>
  <guid>https://eyayaw.github.io/posts/jobrake-for-the-job/</guid>
  <pubDate>Sun, 06 Sep 2026 22:00:00 GMT</pubDate>
  <media:content url="https://eyayaw.github.io/posts/jobrake-for-the-job/assets/cover.png" medium="image" type="image/png" height="76" width="144"/>
</item>
<item>
  <title>Reproducible research workflows</title>
  <dc:creator>Eyayaw Beze</dc:creator>
  <link>https://eyayaw.github.io/posts/reproducible-research-workflows/</link>
  <description><![CDATA[ 




<p>I gave <a href="https://eyayaw.github.io/rrw/">my talk</a> at the <a href="https://www.ams-institute.org/">AMS Institute</a> to research fellows at our all-hands meeting.</p>
<p>My old workflow went like this. Writing in Word, summary stats copied out of Excel, regression estimates and figures made in Stata or QGIS and pasted in by hand. Five or more tools and formats, and every number in the paper copied at least once. Then, say, Reviewer #2 wants one threshold changed, and you are staring at <code>analysis_v3_FINAL.do</code> next to <code>final_final_draft.docx</code>, trying to remember how you cleaned the data months ago.</p>
<p><em>Sound familiar?</em></p>
<p>My objective was to show colleagues that there is a better workflow, and how to adopt it slowly until it becomes a habit. I also wanted to emphasize literate programming: a single <mark>source of truth</mark> makes a research project highly reproducible, with the code and the prose intertwined, no hardcoded stats, and no copy-pasting of stuff.</p>
<p>I organized the talk around four pillars.</p>
<ol type="1">
<li><strong>Project organization</strong>: A self-contained repo per project, with data, code, and outputs together. Separate raw data from processed data, and write docs, a README. For a cumulative dissertation, each chapter can be a git submodule, but I used separate <a href="https://github.com/eyayaw/dissertation">repos</a>.</li>
<li><strong>Version control with Git &amp; GitHub</strong>: Use Git for tracking your work (code, prose, or docs). Use it even as a single author, with GitHub or locally.</li>
<li><strong>Dependency management</strong>: Our analysis depends on specific package versions, and package updates can break our code. <mark><em>It does not work on my machine!</em></mark> <a href="https://docs.astral.sh/uv/"><code>uv</code></a> (Python) and <a href="https://rstudio.github.io/renv/"><code>renv</code></a> (R) for recording versions of packages for cross-machine use.</li>
<li><strong>Literate programming</strong>: Code, results, and narrative in one document: no copy-paste errors, and results update automatically.</li>
</ol>
<p>I discussed literate programming with <a href="https://quarto.org">Quarto</a> in detail because it encompasses the other three and is highly relevant for reproducibility.</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p>The reproducibility crisis is real. See the <a href="https://datacolada.org/118">Data Colada</a> posts on the Gino case. It may take you down, from earning <a href="https://fortune.com/2023/08/02/harvard-professor-gino-dishonesty-investigation-falsified-research/">$1M+ a year</a> to a pleb! :D Journals and reviewers increasingly ask for code and data.</p>
</div>
</div>
<p>One source but several output formats:</p>
<div class="cell" data-layout-align="default">
<div class="cell-output-display">
<div id="fig-one-source" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-one-source-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div>
<pre class="mermaid mermaid-js" data-label="fig-one-source">flowchart LR
  project["project/
  ├── data/
  │   ├── raw/
  │   └── processed/
  └── code/
  #nbsp;   ├── download_data.py
  #nbsp;   └── clean_data.R"]

  project --&gt; article["article.qmd"]
  project --&gt; seminar["internal-seminar-2026.qmd"]

  article --&gt;|render| html["HTML"]
  article --&gt;|render| pdf["PDF"]
  article --&gt;|render| docx["Word"]

  seminar --&gt;|render| revealjs["reveal.js slides (HTML)"]
  seminar --&gt;|render| beamer["Beamer slides (PDF)"]
  seminar --&gt;|render| pptx["PowerPoint slides"]

  %% mermaid trims leading spaces in a label, so the code/ children start with #nbsp;
  classDef tree text-align:left,font-family:monospace,white-space:pre
  class project tree
</pre>
</div>
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-one-source-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure&nbsp;1: Write in Markdown, embed R or Python, render to HTML, PDF, Word, or slides.
</figcaption>
</figure>
</div>
</div>
</div>
<p>I demoed Quarto with trends of <a href="https://eyayaw.github.io/rrw/demo/woz_analysis.html">WOZ property values</a> across Dutch municipalities. Load data from CSV, make exploratory plots, run a regression with inline results, add cross-references and citations, and render to HTML and PDF.</p>
<p><em><a href="https://eyayaw.github.io/rrw/">Slides</a>, with the <a href="https://eyayaw.github.io/rrw/demo/woz_analysis.html">WOZ demo</a> and the talk <a href="https://github.com/eyayaw/rrw">repo</a>.<br>
Dissertation chapters: <a href="https://github.com/eyayaw/housing-supply-elasticity-in-germany">housing supply elasticity</a>, <a href="https://github.com/eyayaw/de-donut-effect">the German donut effect</a>, and <a href="https://github.com/eyayaw/the-monocentric-city-gradients-addis-ababa">Addis Ababa gradients</a>.</em></p>
<p>—— <strong>Thank you for stopping by!</strong> 💚💛❤️ ——</p>



 ]]></description>
  <category>talk</category>
  <category>r</category>
  <category>python</category>
  <guid>https://eyayaw.github.io/posts/reproducible-research-workflows/</guid>
  <pubDate>Wed, 28 Jan 2026 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Underwhelmed by stats::reshape</title>
  <dc:creator>Eyayaw Beze</dc:creator>
  <link>https://eyayaw.github.io/posts/underwhelmed-by-stats-reshape/</link>
  <description><![CDATA[ 




<p>When was the last time you used <a href="https://search.r-project.org/R/refmans/stats/html/reshape.html"><code>stats::reshape</code></a>? Probably never, in the presence of {tidyr}’s <a href="https://tidyr.tidyverse.org/articles/pivot.html"><code>pivot_*</code></a> and {data.table}’s <a href="https://rdatatable.gitlab.io/data.table/articles/datatable-reshape.html"><code>melt</code>/<code>dcast</code></a> reshaping (aka pivoting) functions.</p>
<p>A while ago, while perusing through the {data.table} <a href="https://rdatatable.gitlab.io/data.table/articles/">vignettes</a>, I learned about the <code>reshape</code> function. The {data.table} core team described it as “<a href="https://rdatatable.gitlab.io/data.table/articles/datatable-reshape.html#limitations-in-previous-meltdcast-approaches">It is an extremely useful and often underrated function.</a>”. As a member of the “dependency-frets” (as <code>reshape</code> is part of <a href="../base-r.html">base R</a>), I’ve been using it fairly since. However, it has some annoyances that might lead you to think that the function isn’t yet quite “fully ripe”, which could explain its “underratting”.</p>
<p>First, it’s very slow; no further comment is needed, but that could be a deal breaker for some.<sup>1</sup></p>
<p>Args of the <code>reshape</code> function are also a bit cryptic. I have to hit <a href="https://search.r-project.org/R/refmans/stats/html/reshape.html"><code>?reshape</code></a> time and again because it is not easy to remember what is what. What (or not) to pass to <code>varying</code> (a vector or a list (or a matrix)) and how to order its values? Like <code>varying=c("gdp_2020", "gdp_2021", "pop_2020", "pop_2021")</code>, or <code>c("gdp_2020", "pop_2020", "gdp_2021", "pop_2021")</code>? What does <code>v.names</code> (and frankly most args) do in <code>reshape</code> to wide vs.&nbsp;long? Would not it be helpful if <code>reshape</code> was broken down into two separate functions (<code>reshape_{long,wide}</code>)?<sup>2</sup> Also it is hard to tell which args are optional and which get ignored silently even when you explicitly provide them.<sup>3</sup></p>
<p>Futhermore, you have to “nicely” format your column names to unlock reshape to long’s potential: that is use a <code>sep</code> (defaults to <code>.</code>) between the time <code>varying</code> variables and the <code>timevar</code> (like <code>gdp.2020</code>, <code>pop.2020</code>).</p>
<div class="callout callout-style-default callout-warning callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Warning
</div>
</div>
<div class="callout-body-container callout-body">
<p>Do you really think about renaming your columns when reshaping? Don’t! 🙃</p>
</div>
</div>
<p>Even if you “nicely” format your column names say with <code>sep="_"</code>, your’re doomed if you have “_” somewhere in the column names as you can’t use perl-compatible regex to force the location of the <code>sep</code> (e.g., you have <code>gdp_ppp_2020</code> and you’re not allowed to use <code>sep="_(?=\\d{4})"</code> or <code>split(regexp = "_(?=\\d{4})", include = FALSE, fixed = FALSE)</code>). If you look at the <a href="https://github.com/wch/r-source/blob/trunk/src/library/stats/R/reshape.R">source</a>, this could have been done as <code>split</code> and <code>regexpr</code> (where <code>sep</code> and <code>split</code> are used) can handle perl-compatible regex with <code>perl = TRUE</code>.</p>
<p>While <code>reshape</code> is fundamentally built to deal with panel data, it does not handle “unbalanced” (panel) data. A special case that traps reshape (to long) is when a variable (say population size) is available for 2019 but missing for 2020 (for all units), while another variable (GDP) is measured for both 2019 and 2020 (for some or all units).<sup>4</sup> You may think this is a useless edge scenario, but both {data.table} and {tidyr} support it. This rare edge case may frequently <a href="https://stackoverflow.com/q/76167864">creep into</a> typical use cases, doesn’t it?</p>
<section id="reshaping-with-statsreshape" class="level1">
<h1>Reshaping with <code>stats::reshape</code></h1>
<p>Let’s quickly look at how we can use <code>reshape</code> to convert data from wide to long format and vice versa and touch on some of the issues along the way, with sample datasets from the World Development Indicators (WDI) database. The WDI datasets provide a perfect use case for reshaping because you can download the data in wide or long format.</p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Tip
</div>
</div>
<div class="callout-body-container callout-body">
<p>The official <a href="https://search.r-project.org/R/refmans/stats/html/reshape.html">documentation</a> and <a href="https://stat.ethz.ch/R-manual/R-devel/library/stats/doc/reshape.pdf">vignette</a> for <code>reshape</code> are great. I also find this <a href="https://jozef.io/r001-reshape/">blog post</a> by Jozef Hajnala excellent.</p>
</div>
</div>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># import data (source WBG WDI)</span></span>
<span id="cb1-2">wdi_data_wide <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read.csv</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"wdi_data_wide.csv"</span>)</span>
<span id="cb1-3">wdi_data_long <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read.csv</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"wdi_data_long.csv"</span>)</span>
<span id="cb1-4"></span>
<span id="cb1-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># constants</span></span>
<span id="cb1-6">data_url <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://databank.worldbank.org/id/8dfced2d"</span></span>
<span id="cb1-7">varying_patt <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^((?:pop|gdp).*)[_](</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4})$"</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># regex for measure vars</span></span>
<span id="cb1-8">varying_patt_vec <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^pop"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^gdp"</span>)</span></code></pre></div></div>
</details>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-3-contents" aria-controls="callout-3" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>About the sample WDI datasets
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-3" class="callout-3-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>I downloaded the data in two formats: “table” and “list”, which roughly correspond to wide and long formats, respectively. The data contains information on 2 indicators (“Population, total” and “GDP per capita, PPP (constant 2021 international $)”), for countries Korea, Rep., Ethiopia, and Germany for the years 2019-2020. Here is a direct <a href="https://databank.worldbank.org/id/8dfced2d">link</a> to the original dataset.</p>
<div class="cell" data-layout-nrow="2">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read.csv</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sample_wdi_data_table.csv"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">check.names =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb2-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb2-3">  knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>()</span>
<span id="cb2-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read.csv</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sample_wdi_data_list.csv"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">check.names =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb2-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb2-6">  knitr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kable</span>()</span></code></pre></div></div>
</details>
<div id="tbl-original-data" class="quarto-layout-panel anchored">
<figure class="quarto-float quarto-float-tbl figure">
<figcaption class="quarto-float-caption-top quarto-float-caption quarto-float-tbl" id="tbl-original-data-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Table&nbsp;1: Sample Dataset from WDI Database (Raw)
</figcaption>
<div aria-describedby="tbl-original-data-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div class="quarto-layout-row">
<div class="cell-output-display quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-original-data" style="flex-basis: 100.0%;justify-content: center;">
<div id="tbl-original-data-1" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-original-data-1-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(a) Head of the data in table (wide) format
</figcaption>
<div aria-describedby="tbl-original-data-1-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table id="tbl-original-data-1" class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 14%">
<col style="width: 57%">
<col style="width: 14%">
<col style="width: 14%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">Country Name</th>
<th style="text-align: left;">Series Name</th>
<th style="text-align: right;">2019</th>
<th style="text-align: right;">2020</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Korea, Rep.</td>
<td style="text-align: left;">Population, total</td>
<td style="text-align: right;">51764822.00</td>
<td style="text-align: right;">5.183624e+07</td>
</tr>
<tr class="even">
<td style="text-align: left;">Korea, Rep.</td>
<td style="text-align: left;">GDP per capita, PPP (constant 2021 international $)</td>
<td style="text-align: right;">46757.26</td>
<td style="text-align: right;">4.636160e+04</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Ethiopia</td>
<td style="text-align: left;">Population, total</td>
<td style="text-align: right;">115737383.00</td>
<td style="text-align: right;">1.189177e+08</td>
</tr>
<tr class="even">
<td style="text-align: left;">Ethiopia</td>
<td style="text-align: left;">GDP per capita, PPP (constant 2021 international $)</td>
<td style="text-align: right;">2437.97</td>
<td style="text-align: right;">2.516549e+03</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Germany</td>
<td style="text-align: left;">Population, total</td>
<td style="text-align: right;">83092962.00</td>
<td style="text-align: right;">8.316087e+07</td>
</tr>
<tr class="even">
<td style="text-align: left;">Germany</td>
<td style="text-align: left;">GDP per capita, PPP (constant 2021 international $)</td>
<td style="text-align: right;">62970.72</td>
<td style="text-align: right;">6.034266e+04</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
</div>
<div class="quarto-layout-row">
<div class="cell-output-display quarto-layout-cell-subref quarto-layout-cell" data-ref-parent="tbl-original-data" style="flex-basis: 100.0%;justify-content: center;">
<div id="tbl-original-data-2" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-subfloat-tbl figure">
<figcaption class="quarto-float-caption-top quarto-subfloat-caption quarto-subfloat-tbl" id="tbl-original-data-2-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
(b) Head of the data in list (long) format
</figcaption>
<div aria-describedby="tbl-original-data-2-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<table id="tbl-original-data-2" class="caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 15%">
<col style="width: 62%">
<col style="width: 6%">
<col style="width: 15%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">Country Name</th>
<th style="text-align: left;">Series Name</th>
<th style="text-align: right;">Time</th>
<th style="text-align: right;">Value</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Korea, Rep.</td>
<td style="text-align: left;">Population, total</td>
<td style="text-align: right;">2019</td>
<td style="text-align: right;">51764822.00</td>
</tr>
<tr class="even">
<td style="text-align: left;">Korea, Rep.</td>
<td style="text-align: left;">Population, total</td>
<td style="text-align: right;">2020</td>
<td style="text-align: right;">51836239.00</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Korea, Rep.</td>
<td style="text-align: left;">GDP per capita, PPP (constant 2021 international $)</td>
<td style="text-align: right;">2019</td>
<td style="text-align: right;">46757.26</td>
</tr>
<tr class="even">
<td style="text-align: left;">Korea, Rep.</td>
<td style="text-align: left;">GDP per capita, PPP (constant 2021 international $)</td>
<td style="text-align: right;">2020</td>
<td style="text-align: right;">46361.60</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Ethiopia</td>
<td style="text-align: left;">Population, total</td>
<td style="text-align: right;">2019</td>
<td style="text-align: right;">115737383.00</td>
</tr>
<tr class="even">
<td style="text-align: left;">Ethiopia</td>
<td style="text-align: left;">Population, total</td>
<td style="text-align: right;">2020</td>
<td style="text-align: right;">118917671.00</td>
</tr>
</tbody>
</table>
</div>
</figure>
</div>
</div>
</div>
</div>
</figure>
</div>
</div>
</div>
</div>
</div>
<section id="reshape-to-long-reshape...-direction-long" class="level2">
<h2 class="anchored" data-anchor-id="reshape-to-long-reshape...-direction-long">Reshape to long: <code>reshape(..., direction = "long")</code></h2>
<p>The dataset in wide format looks like:<sup>5</sup></p>
<div class="cell">
<div id="tbl-true-wide" class="cell quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-tbl figure">
<figcaption class="quarto-float-caption-top quarto-float-caption quarto-float-tbl" id="tbl-true-wide-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Table&nbsp;2: The data in true wide format (cleaned)
</figcaption>
<div aria-describedby="tbl-true-wide-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<div class="cell-output-display">
<table class="cell caption-top table table-sm table-striped small">
<colgroup>
<col style="width: 16%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 21%">
<col style="width: 21%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">country</th>
<th style="text-align: right;">pop_total_2019</th>
<th style="text-align: right;">pop_total_2020</th>
<th style="text-align: right;">gdp_pc_ppp_2019</th>
<th style="text-align: right;">gdp_pc_ppp_2020</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Korea, Rep.</td>
<td style="text-align: right;">51764822</td>
<td style="text-align: right;">51836239</td>
<td style="text-align: right;">46757.26</td>
<td style="text-align: right;">46361.596</td>
</tr>
<tr class="even">
<td style="text-align: left;">Ethiopia</td>
<td style="text-align: right;">115737383</td>
<td style="text-align: right;">118917671</td>
<td style="text-align: right;">2437.97</td>
<td style="text-align: right;">2516.549</td>
</tr>
<tr class="odd">
<td style="text-align: left;">Germany</td>
<td style="text-align: right;">83092962</td>
<td style="text-align: right;">83160871</td>
<td style="text-align: right;">62970.72</td>
<td style="text-align: right;">60342.663</td>
</tr>
</tbody>
</table>
</div>
</div>
</figure>
</div>
</div>
<p>To pivot this into a long format, we just collect the suffixed years in the (series) columns into a single column “year”, and each series makes (or gets collapsed into) a single column.</p>
<div class="callout callout-style-default callout-tip callout-titled" title="Reshape to long with other packages">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-4-contents" aria-controls="callout-4" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Reshape to long with other packages
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-4" class="callout-4-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>IMO, all three solutions are straighforward. We get a proper long format, a row represents a record for a country at a point-in-time, that is, this is in a typical panel (<a href="https://en.wikipedia.org/wiki/Panel_data">repeated-crossection or longitudinal</a>) data format.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-1-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-1" aria-controls="tabset-1-1" aria-selected="true" href=""><code>data.table::melt</code></a></li><li class="nav-item"><a class="nav-link" id="tabset-1-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-2" aria-controls="tabset-1-2" aria-selected="false" href=""><code>tidyr::pivot_longer</code></a></li><li class="nav-item"><a class="nav-link" id="tabset-1-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-1-3" aria-controls="tabset-1-3" aria-selected="false" href=""><code>duckdb::unpivot</code></a></li></ul>
<div class="tab-content">
<div id="tabset-1-1" class="tab-pane active" aria-labelledby="tabset-1-1-tab">
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">wdi_data_wide <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb3-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.table</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb3-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">melt</span>(</span>
<span id="cb3-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">measure.vars =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">measure</span>(value.name, year, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pattern =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^([a-z_]+)_(</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4})$"</span>),</span>
<span id="cb3-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">id.vars =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span></span>
<span id="cb3-6">)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>       country   year pop_total gdp_pc_ppp
        &lt;char&gt; &lt;char&gt;     &lt;int&gt;      &lt;num&gt;
1: Korea, Rep.   2019  51764822  46757.261
2:    Ethiopia   2019 115737383   2437.970
3:     Germany   2019  83092962  62970.716
4: Korea, Rep.   2020  51836239  46361.596
5:    Ethiopia   2020 118917671   2516.549
6:     Germany   2020  83160871  60342.663</code></pre>
</div>
</div>
</div>
<div id="tabset-1-2" class="tab-pane" aria-labelledby="tabset-1-2-tab">
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">wdi_data_wide <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb5-2">  tidyr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pivot_longer</span>(</span>
<span id="cb5-3">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cols =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>country,</span>
<span id="cb5-4">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_pattern =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^([a-z_]+)_(</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4})$"</span>,</span>
<span id="cb5-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_to =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">".value"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>)</span>
<span id="cb5-6">  )</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 6 × 4
  country     year  pop_total gdp_pc_ppp
  &lt;chr&gt;       &lt;chr&gt;     &lt;int&gt;      &lt;dbl&gt;
1 Korea, Rep. 2019   51764822     46757.
2 Korea, Rep. 2020   51836239     46362.
3 Ethiopia    2019  115737383      2438.
4 Ethiopia    2020  118917671      2517.
5 Germany     2019   83092962     62971.
6 Germany     2020   83160871     60343.</code></pre>
</div>
</div>
</div>
<div id="tabset-1-3" class="tab-pane" aria-labelledby="tabset-1-3-tab">
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb7-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">duckdb</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;&lt; EOF</span></span>
<span id="cb7-2"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">create or replace table wdi_data_wide as</span></span>
<span id="cb7-3"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">from read_csv('wdi_data_wide.csv');</span></span>
<span id="cb7-4"></span>
<span id="cb7-5"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">set variable measure_patt = '^([a-z_]+)_(</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">\d</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">{4})$';</span></span>
<span id="cb7-6"></span>
<span id="cb7-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">.mode markdown</span></span>
<span id="cb7-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">with unpivoted as (</span></span>
<span id="cb7-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  select</span></span>
<span id="cb7-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    country,</span></span>
<span id="cb7-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    regexp_extract(column_name, getvariable('measure_patt'), 2) as year,</span></span>
<span id="cb7-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    regexp_extract(column_name, getvariable('measure_patt'), 1) as measure,</span></span>
<span id="cb7-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    column_value as value</span></span>
<span id="cb7-14"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  from (</span></span>
<span id="cb7-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    unpivot wdi_data_wide</span></span>
<span id="cb7-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    on COLUMNS(getvariable('measure_patt'))</span></span>
<span id="cb7-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    into</span></span>
<span id="cb7-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      name column_name</span></span>
<span id="cb7-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      value column_value</span></span>
<span id="cb7-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  )</span></span>
<span id="cb7-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">)</span></span>
<span id="cb7-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">pivot unpivoted</span></span>
<span id="cb7-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">on measure</span></span>
<span id="cb7-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">using first(value)</span></span>
<span id="cb7-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">order by country, year;</span></span>
<span id="cb7-26"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">EOF</span></span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>|   country   | year |    gdp_pc_ppp    |  pop_total  |
|-------------|------|-----------------:|------------:|
| Ethiopia    | 2019 | 2437.97046055952 | 115737383.0 |
| Ethiopia    | 2020 | 2516.54907642544 | 118917671.0 |
| Germany     | 2019 | 62970.7160913279 | 83092962.0  |
| Germany     | 2020 | 60342.6627037391 | 83160871.0  |
| Korea, Rep. | 2019 | 46757.2614496182 | 51764822.0  |
| Korea, Rep. | 2020 | 46361.5957811718 | 51836239.0  |</code></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>According to the <a href="https://search.r-project.org/R/refmans/stats/html/reshape.html">doc</a>, the easiest way to reshape to long with <code>reshape</code> is by using “nicely” formatted column names: that is, use a <code>sep</code> (which defaults to <code>.</code>) between the time <code>varying</code> variables and the <code>timevar</code>, like <code>gdp_pc_ppp.2020</code> and <code>pop_total.2020</code>. This nice formatting has the added benefit that <code>v.names</code> (<code>gdp_pc_ppp</code>, <code>pop_total</code>) and <code>times</code> (i.e., the values for <code>timevar</code>) (2020), will be auto-guessed; otherwise, we need to supply them ourselves (which is error prone).</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">### Typical usage for converting from wide to long format:</span></span>
<span id="cb9-2"></span>
<span id="cb9-3"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">### If names of wide-format variables are in a 'nice' format</span></span>
<span id="cb9-4"></span>
<span id="cb9-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># reshape(data, direction = "long",</span></span>
<span id="cb9-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#         varying = c(___), # vector</span></span>
<span id="cb9-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#         sep)              # to help guess 'v.names' and 'times'</span></span></code></pre></div></div>
<p>So let’s rename our <code>varying</code> columns using a special <code>sep</code> and try reshaping to long as follows:</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1">sep <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"."</span></span>
<span id="cb10-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># let's replace "_" (just before the year) with the sep to "nicely" format the column names</span></span>
<span id="cb10-3">nicely_format_names <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(nms) {</span>
<span id="cb10-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sub</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_(?=</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d)"</span>, sep, nms, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">perl =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb10-5">}</span>
<span id="cb10-6"></span>
<span id="cb10-7">nice_nms <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nicely_format_names</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(wdi_data_wide))</span>
<span id="cb10-8"></span>
<span id="cb10-9">wdi_data_wide_long1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">reshape</span>(</span>
<span id="cb10-10">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setNames</span>(wdi_data_wide, nice_nms),</span>
<span id="cb10-11">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">direction =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"long"</span>,</span>
<span id="cb10-12">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">varying =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grepv</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sprintf</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^[a-z_]+%s</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4}$"</span>, sep), nice_nms),</span>
<span id="cb10-13">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">idvar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># if ommitted, an `id` column will be added `seq_along(data)`</span></span>
<span id="cb10-14">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timevar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># otherwise `time`</span></span>
<span id="cb10-15">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sep =</span> sep <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># the default is `.`</span></span>
<span id="cb10-16">)</span>
<span id="cb10-17"></span>
<span id="cb10-18">wdi_data_wide_long1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>()</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>                     country year pop_total gdp_pc_ppp
Korea, Rep..2019 Korea, Rep. 2019  51764822  46757.261
Ethiopia.2019       Ethiopia 2019 115737383   2437.970
Germany.2019         Germany 2019  83092962  62970.716
Korea, Rep..2020 Korea, Rep. 2020  51836239  46361.596
Ethiopia.2020       Ethiopia 2020 118917671   2516.549
Germany.2020         Germany 2020  83160871  60342.663</code></pre>
</div>
</div>
<p>That is a hell of unnatural detour for reshaping. What if we want to go along with our column names? Enter explicit-specification:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb12-1"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">### Typical usage for converting from wide to long format:</span></span>
<span id="cb12-2"></span>
<span id="cb12-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ...</span></span>
<span id="cb12-4"></span>
<span id="cb12-5"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">### To specify long-format variable names explicitly</span></span>
<span id="cb12-6"></span>
<span id="cb12-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># reshape(data, direction = "long",</span></span>
<span id="cb12-8"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#         varying = ___,  # list / matrix / vector (use with care)</span></span>
<span id="cb12-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#         v.names = ___,  # vector of variable names in long format</span></span>
<span id="cb12-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#         timevar, times, # name / values of constructed time variable</span></span>
<span id="cb12-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#         idvar, ids)     # name / values of constructed id variable</span></span></code></pre></div></div>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb13-1">wdi_data_wide_long2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> wdi_data_wide <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb13-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">reshape</span>(</span>
<span id="cb13-3">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">direction =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"long"</span>,</span>
<span id="cb13-4">    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># varying = lapply(c("^pop", "^gdp"), \(p) grepv(p, names(wdi_data_wide))),</span></span>
<span id="cb13-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">varying =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(</span>
<span id="cb13-6">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pop_total_2019"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pop_total_2020"</span>),</span>
<span id="cb13-7">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gdp_pc_ppp_2019"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gdp_pc_ppp_2020"</span>)</span>
<span id="cb13-8">    ),</span>
<span id="cb13-9">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">v.names =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pop"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gdp_pc_ppp"</span>),</span>
<span id="cb13-10">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">idvar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># otherwise column `ids` will be created</span></span>
<span id="cb13-11">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timevar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># otherwise "time"</span></span>
<span id="cb13-12">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">times =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2019"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"2020"</span>) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># otherwise auto-generated</span></span>
<span id="cb13-13">  )</span>
<span id="cb13-14"></span>
<span id="cb13-15">wdi_data_wide_long2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>                     country year       pop gdp_pc_ppp
Korea, Rep..2019 Korea, Rep. 2019  51764822  46757.261
Ethiopia.2019       Ethiopia 2019 115737383   2437.970
Germany.2019         Germany 2019  83092962  62970.716
Korea, Rep..2020 Korea, Rep. 2020  51836239  46361.596
Ethiopia.2020       Ethiopia 2020 118917671   2516.549
Germany.2020         Germany 2020  83160871  60342.663</code></pre>
</div>
</div>
<p>That is very verbose and error prone: (1) we need to specify <code>varying</code> “with care” (choosing the list form), as urged by the doc, (2) the values of <code>timevar</code> (named as <code>year</code>) which are controlled by <code>times</code> will be auto-generated instead of the actual values we see from the data, had we not provided them like so <code>times = c("2019", "2020")</code>, (3) not only do we need to know the unique values of <code>timevar</code>, but also their correct order.</p>
<p>Importantly, were we should note that we provided a list to <code>varying</code>. What if you accidentially gave a vector like <code>c(c("pop_total_2019", "pop_total_2020"), c("gdp_pc_ppp_2019", "gdp_pc_ppp_2020"))</code>? Well, that would be wrong because if <code>varying</code> takes on a vector, we should rather provide it in this order <code>c("pop_total_2019", "gdp_pc_ppp_2019", "pop_total_2020", "gdp_pc_ppp_2020")</code>.</p>
<blockquote class="blockquote">
<p>The correct order requires all columns corresponding to the same time to be contiguous; this is the same intrinsic column-major ordering in the matrix [ or list] […] form. It is best to avoid the atomic vector form of varying unless <code>v.names</code> is being omitted.<br>
— <a href="https://stat.ethz.ch/R-manual/R-devel/library/stats/doc/reshape.pdf">Using the reshape function</a> (The R Core Team, April 2025)</p>
</blockquote>
<p>Looks like passing a list to <code>varying</code> is safe and straighforward when it comes to the order of the time-varying variables. But as we saw we need to manually define the names/values of several variables (<code>v.names</code>, <code>timevar</code>, <code>times</code>). Before abandoning passing a vector for <code>varying</code>, since we actually want names/values auto-guessed, what if we tell <code>reshape</code> to split at <code>sep = "_"</code>? However, since our <code>varying</code> variables have <code>_</code>, this would be impossible to do without a perl-compatible regex option, which <code>reshape</code> hides from us. If you edit the function definition of <code>reshape</code> to allow for this (i.e, add <code>perl = TRUE</code>), we can use <code>sep = "_(?=\\d{4})"</code> and get the same result as the nicely formatted naming option above.</p>
<p>We can also play with <code>split</code> to provide a custom <code>sep</code> (<code>regexp</code>) as shown below but the names will have trailing <code>_</code>s (because <code>include = FALSE</code> fails), but we need to pass a vector of names for <code>varying</code> (to take advantage of auto-guessing, so much complication!).</p>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb15-1">wdi_data_wide_long3 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> wdi_data_wide <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb15-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">reshape</span>(</span>
<span id="cb15-3">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">direction =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"long"</span>,</span>
<span id="cb15-4">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">varying =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pop_total_2019"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gdp_pc_ppp_2019"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pop_total_2020"</span>,  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gdp_pc_ppp_2020"</span>),</span>
<span id="cb15-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timevar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>,</span>
<span id="cb15-6">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">idvar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span>,</span>
<span id="cb15-7">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">split =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">regexp =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4}$"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">include =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fixed =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>)</span>
<span id="cb15-8">  )</span>
<span id="cb15-9"></span>
<span id="cb15-10">wdi_data_wide_long3 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>                     country year pop_total_ gdp_pc_ppp_
Korea, Rep..2019 Korea, Rep. 2019   51764822   46757.261
Ethiopia.2019       Ethiopia 2019  115737383    2437.970
Germany.2019         Germany 2019   83092962   62970.716
Korea, Rep..2020 Korea, Rep. 2020   51836239   46361.596
Ethiopia.2020       Ethiopia 2020  118917671    2516.549
Germany.2020         Germany 2020   83160871   60342.663</code></pre>
</div>
</div>
<p>I would argue that this is a better approach (than the nicely renaming columns we saw above) and is more flexible and usable: we can force the location of the split without perl-compatible regex, and have auto-guessing of names/values.</p>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb17" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb17-1">wdi_data_wide_unbalanced <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> wdi_data_wide[, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setdiff</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(wdi_data_wide), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"gdp_pc_ppp_2020"</span>)]</span>
<span id="cb17-2"></span>
<span id="cb17-3">wdi_data_wide_unbalanced <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb17-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">reshape</span>(</span>
<span id="cb17-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">direction =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"long"</span>,</span>
<span id="cb17-6">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">varying =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grepv</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^[a-z_]+_</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4}$"</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">names</span>(wdi_data_wide_unbalanced)),</span>
<span id="cb17-7">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timevar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>,</span>
<span id="cb17-8">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">idvar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span>,</span>
<span id="cb17-9">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">split =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">regexp =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4}$"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">include =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fixed =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>)</span>
<span id="cb17-10">  )</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-error">
<pre><code>Error in `reshapeLong()`:
! 'varying' arguments must be the same length</code></pre>
</div>
</div>
<div class="callout callout-style-default callout-tip callout-titled" title="Unbalanced data handled by other packages">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-5-contents" aria-controls="callout-5" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Tip</span>Unbalanced data handled by other packages
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-5" class="callout-5-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-2-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-2-1" aria-controls="tabset-2-1" aria-selected="true" href=""><code>data.table:melt</code></a></li><li class="nav-item"><a class="nav-link" id="tabset-2-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-2-2" aria-controls="tabset-2-2" aria-selected="false" href=""><code>tidyr::pivot_longer</code></a></li><li class="nav-item"><a class="nav-link" id="tabset-2-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-2-3" aria-controls="tabset-2-3" aria-selected="false" href=""><code>duckdb::unpivot</code></a></li></ul>
<div class="tab-content">
<div id="tabset-2-1" class="tab-pane active" aria-labelledby="tabset-2-1-tab">
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb19" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb19-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># data.table solution</span></span>
<span id="cb19-2">wdi_data_wide_unbalanced <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb19-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.table</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb19-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">melt</span>(</span>
<span id="cb19-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">id.vars =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span>,</span>
<span id="cb19-6">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">measure.vars =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">measure</span>(value.name, year, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">pattern =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^([a-z_]+)_(</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4})$"</span>)</span>
<span id="cb19-7">  )</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>       country   year pop_total gdp_pc_ppp
        &lt;char&gt; &lt;char&gt;     &lt;int&gt;      &lt;num&gt;
1: Korea, Rep.   2019  51764822   46757.26
2:    Ethiopia   2019 115737383    2437.97
3:     Germany   2019  83092962   62970.72
4: Korea, Rep.   2020  51836239         NA
5:    Ethiopia   2020 118917671         NA
6:     Germany   2020  83160871         NA</code></pre>
</div>
</div>
</div>
<div id="tabset-2-2" class="tab-pane" aria-labelledby="tabset-2-2-tab">
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb21" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb21-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># tidyr solution</span></span>
<span id="cb21-2">wdi_data_wide_unbalanced <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb21-3">  tidyr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pivot_longer</span>(</span>
<span id="cb21-4">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">cols =</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>country,</span>
<span id="cb21-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_pattern =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"^([a-z_]+)_(</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\\</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">d{4})$"</span>,</span>
<span id="cb21-6">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_to =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">".value"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>)</span>
<span id="cb21-7">  )</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 6 × 4
  country     year  pop_total gdp_pc_ppp
  &lt;chr&gt;       &lt;chr&gt;     &lt;int&gt;      &lt;dbl&gt;
1 Korea, Rep. 2019   51764822     46757.
2 Korea, Rep. 2020   51836239        NA 
3 Ethiopia    2019  115737383      2438.
4 Ethiopia    2020  118917671        NA 
5 Germany     2019   83092962     62971.
6 Germany     2020   83160871        NA </code></pre>
</div>
</div>
</div>
<div id="tabset-2-3" class="tab-pane" aria-labelledby="tabset-2-3-tab">
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb23" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb23-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># duckdb solution</span></span>
<span id="cb23-2"></span>
<span id="cb23-3">duck_cmd <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> r<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"{</span></span>
<span id="cb23-4"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">duckdb &lt;&lt; EOF</span></span>
<span id="cb23-5"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  create or replace table wdi_data_wide as</span></span>
<span id="cb23-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  from read_csv('wdi_data_wide.csv');</span></span>
<span id="cb23-7"></span>
<span id="cb23-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  set variable measure_patt = '^([a-z_]+)_(\d{4})$';</span></span>
<span id="cb23-9"></span>
<span id="cb23-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  pivot (</span></span>
<span id="cb23-11"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    select</span></span>
<span id="cb23-12"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    country,</span></span>
<span id="cb23-13"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    regexp_extract(column_name, getvariable('measure_patt'), 2) as year,</span></span>
<span id="cb23-14"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    regexp_extract(column_name, getvariable('measure_patt'), 1) as measure,</span></span>
<span id="cb23-15"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    column_value as value</span></span>
<span id="cb23-16"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    from (</span></span>
<span id="cb23-17"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      unpivot wdi_data_wide</span></span>
<span id="cb23-18"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      on COLUMNS( * exclude(country, gdp_pc_ppp_2020))</span></span>
<span id="cb23-19"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      into</span></span>
<span id="cb23-20"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        name column_name</span></span>
<span id="cb23-21"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        value column_value</span></span>
<span id="cb23-22"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    )</span></span>
<span id="cb23-23"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  )</span></span>
<span id="cb23-24"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  on measure</span></span>
<span id="cb23-25"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  using first(value)</span></span>
<span id="cb23-26"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  order by country, year;</span></span>
<span id="cb23-27"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">EOF</span></span>
<span id="cb23-28"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">}"</span></span>
<span id="cb23-29"></span>
<span id="cb23-30"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">system</span>(duck_cmd, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">intern =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sep =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>┌─────────────┬─────────┬──────────────────┬─────────────┐
│   country   │  year   │    gdp_pc_ppp    │  pop_total  │
│   varchar   │ varchar │      double      │   double    │
├─────────────┼─────────┼──────────────────┼─────────────┤
│ Ethiopia    │ 2019    │ 2437.97046055952 │ 115737383.0 │
│ Ethiopia    │ 2020    │             NULL │ 118917671.0 │
│ Germany     │ 2019    │ 62970.7160913279 │  83092962.0 │
│ Germany     │ 2020    │             NULL │  83160871.0 │
│ Korea, Rep. │ 2019    │ 46757.2614496182 │  51764822.0 │
│ Korea, Rep. │ 2020    │             NULL │  51836239.0 │
└─────────────┴─────────┴──────────────────┴─────────────┘</code></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section id="reshape-to-wide-reshape...-direction-wide" class="level2">
<h2 class="anchored" data-anchor-id="reshape-to-wide-reshape...-direction-wide">Reshape to wide: <code>reshape(..., direction = "wide")</code></h2>
<p>Reshaping from long to wide is supposed to be straightforward. However, <code>reshape</code> handles one <code>timevar</code> at a time. This is because <code>reshape</code> is designed to fundamentally deal with longitudinal data. However, the sample long dataset (see Table&nbsp;1 (b)) is not quite in that form as we have <code>series</code> and <code>year</code> as <code>timevar</code>s. So we need to transform the dataset into a longitudinal form (<code>country</code>, <code>year</code>, <code>pop_total</code>, <code>gdp_pc_ppp</code>), which means we reshape to wide with <code>timevar="series"</code>. Then, we reshape this again with <code>timevar="year"</code>. Thus, the solution involves reshaping twice along the two dimensions (<code>series</code>, <code>year</code>). However, <code>data.table::dcast</code>, <code>tidyr::pivot_wider</code>, and <code>duckdb::pivot</code> handle this in one step.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-3-1-tab" data-bs-toggle="tab" data-bs-target="#tabset-3-1" aria-controls="tabset-3-1" aria-selected="true" href=""><code>reshapeWide</code></a></li><li class="nav-item"><a class="nav-link" id="tabset-3-2-tab" data-bs-toggle="tab" data-bs-target="#tabset-3-2" aria-controls="tabset-3-2" aria-selected="false" href=""><code>data.table::dcast</code></a></li><li class="nav-item"><a class="nav-link" id="tabset-3-3-tab" data-bs-toggle="tab" data-bs-target="#tabset-3-3" aria-controls="tabset-3-3" aria-selected="false" href=""><code>tidyr::pivot_wider</code></a></li><li class="nav-item"><a class="nav-link" id="tabset-3-4-tab" data-bs-toggle="tab" data-bs-target="#tabset-3-4" aria-controls="tabset-3-4" aria-selected="false" href=""><code>duckdb::pivot</code></a></li></ul>
<div class="tab-content">
<div id="tabset-3-1" class="tab-pane active" aria-labelledby="tabset-3-1-tab">
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb25" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb25-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 1 -&gt; reshape into longitudinal</span></span>
<span id="cb25-2">(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">wdi_data_longitudinal =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">reshape</span>(</span>
<span id="cb25-3">  wdi_data_long,</span>
<span id="cb25-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">direction =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"wide"</span>,</span>
<span id="cb25-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">idvar =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>),</span>
<span id="cb25-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timevar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"series"</span>,</span>
<span id="cb25-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">v.names =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"value"</span>,</span>
<span id="cb25-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sep =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_"</span></span>
<span id="cb25-9">))</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>       country year value_pop_total value_gdp_pc_ppp
1  Korea, Rep. 2019        51764822        46757.261
2  Korea, Rep. 2020        51836239        46361.596
5     Ethiopia 2019       115737383         2437.970
6     Ethiopia 2020       118917671         2516.549
9      Germany 2019        83092962        62970.716
10     Germany 2020        83160871        60342.663</code></pre>
</div>
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb27" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb27-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># step 2 -&gt; wide format</span></span>
<span id="cb27-2">wdi_data_longitudinal <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb27-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">reshape</span>(</span>
<span id="cb27-4">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">direction =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"wide"</span>,</span>
<span id="cb27-5">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">idvar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"country"</span>,</span>
<span id="cb27-6">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">timevar =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>,</span>
<span id="cb27-7">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sep =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_"</span></span>
<span id="cb27-8">  ) <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">-&gt;</span> wdi_data_long_wide</span>
<span id="cb27-9"></span>
<span id="cb27-10">wdi_data_long_wide <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># |&gt; setNames(sub("^value_", "", names(wdi_data_long_wide)))</span></span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>      country value_pop_total_2019 value_gdp_pc_ppp_2019 value_pop_total_2020
1 Korea, Rep.             51764822              46757.26             51836239
5    Ethiopia            115737383               2437.97            118917671
9     Germany             83092962              62970.72             83160871
  value_gdp_pc_ppp_2020
1             46361.596
5              2516.549
9             60342.663</code></pre>
</div>
</div>
</div>
<div id="tabset-3-2" class="tab-pane" aria-labelledby="tabset-3-2-tab">
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb29" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb29-1">wdi_data_long <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb29-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.table</span>() <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb29-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">dcast</span>(country <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> series <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> year)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>Key: &lt;country&gt;
       country gdp_pc_ppp_2019 gdp_pc_ppp_2020 pop_total_2019 pop_total_2020
        &lt;char&gt;           &lt;num&gt;           &lt;num&gt;          &lt;num&gt;          &lt;num&gt;
1:    Ethiopia         2437.97        2516.549      115737383      118917671
2:     Germany        62970.72       60342.663       83092962       83160871
3: Korea, Rep.        46757.26       46361.596       51764822       51836239</code></pre>
</div>
</div>
</div>
<div id="tabset-3-3" class="tab-pane" aria-labelledby="tabset-3-3-tab">
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb31" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb31-1">wdi_data_long <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb31-2">  tidyr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pivot_wider</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">names_from =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(series, year), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">values_from =</span> value)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 3 × 5
  country     pop_total_2019 pop_total_2020 gdp_pc_ppp_2019 gdp_pc_ppp_2020
  &lt;chr&gt;                &lt;dbl&gt;          &lt;dbl&gt;           &lt;dbl&gt;           &lt;dbl&gt;
1 Korea, Rep.       51764822       51836239          46757.          46362.
2 Ethiopia         115737383      118917671           2438.           2517.
3 Germany           83092962       83160871          62971.          60343.</code></pre>
</div>
</div>
</div>
<div id="tabset-3-4" class="tab-pane" aria-labelledby="tabset-3-4-tab">
<p>See https://duckdb.org/docs/stable/sql/statements/pivot</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb33" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb33-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">duckdb</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;&lt; EOF</span></span>
<span id="cb33-2"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">create or replace table wdi_data_long as</span></span>
<span id="cb33-3"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">from read_csv('wdi_data_long.csv');</span></span>
<span id="cb33-4"></span>
<span id="cb33-5"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">.mode markdown</span></span>
<span id="cb33-6"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">pivot wdi_data_long</span></span>
<span id="cb33-7"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">-- with a custom sep</span></span>
<span id="cb33-8"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">-- on series || '.' || year</span></span>
<span id="cb33-9"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">on series, year</span></span>
<span id="cb33-10"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">using first(value);</span></span>
<span id="cb33-11"><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">EOF</span></span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>|   country   | gdp_pc_ppp_2019  | gdp_pc_ppp_2020  | pop_total_2019 | pop_total_2020 |
|-------------|-----------------:|-----------------:|---------------:|---------------:|
| Ethiopia    | 2437.97046055952 | 2516.54907642544 | 115737383.0    | 118917671.0    |
| Korea, Rep. | 46757.2614496182 | 46361.5957811718 | 51764822.0     | 51836239.0     |
| Germany     | 62970.7160913279 | 60342.6627037391 | 83092962.0     | 83160871.0     |</code></pre>
</div>
</div>
</div>
</div>
</div>
<p>To conclude, this is not a diss at <code>reshape</code>. In fact, it is great even with the abovementioned idiosyncriasies, if we use it with care. But, I hope it gets better and more usable in future versions of R.</p>
<hr>
<p>—— <strong>Thank you for stopping by!</strong> 💚💛❤️ ——</p>
<hr>


</section>
</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>Consider using <a href="https://rdatatable.gitlab.io/data.table/articles/datatable-reshape.html">data.table</a>!↩︎</p></li>
<li id="fn2"><p>Oh wait, that was the case but <a href="https://stat.ethz.ch/R-manual/R-devel/library/stats/html/stats-defunct.html">removed</a> at some point.↩︎</p></li>
<li id="fn3"><p>For example, <code>times</code> is ignored if <code>v.names</code> is not provided in reshape to long (see <a href="https://stat.ethz.ch/R-manual/R-devel/library/stats/doc/reshape.pdf">here</a>).↩︎</p></li>
<li id="fn4"><p><a href="https://en.wikipedia.org/wiki/Panel_data#Example">Panel data is unbalanced</a> when some units are not measured for at least one time period.↩︎</p></li>
<li id="fn5"><p>When you download the data in “table” format, it is kinda in wide format (because the indicators are stacked in the <code>Series Name</code> column), as shown in Table&nbsp;1. However, I cleaned (including shortening the names of the series) the original dataset to transform it into “true” wide format shown in Table&nbsp;2.↩︎</p></li>
</ol>
</section></div> ]]></description>
  <category>r</category>
  <category>base-r</category>
  <category>tidyverse</category>
  <category>rdatatable</category>
  <category>duckdb</category>
  <guid>https://eyayaw.github.io/posts/underwhelmed-by-stats-reshape/</guid>
  <pubDate>Mon, 11 Aug 2025 22:00:00 GMT</pubDate>
</item>
<item>
  <title>Leveraging Gen AI</title>
  <dc:creator>Eyayaw Beze</dc:creator>
  <link>https://eyayaw.github.io/posts/leveraging-genai/</link>
  <description><![CDATA[ 




<p><em>Have you ever called an LLM from somewhere other than its chat window?</em> I motivated my talk with that question at my Ethiopian community gathering in Wageningen, Netherlands, and most of <a href="https://eyayaw.github.io/talks/leveraging-genai-ethio-wage-july-2025/">the talk</a> is my answer to it.</p>
<p>Most of us use LLMs daily but still carry the impression of the ChatGPT of November 2022, when they hallucinated a ton, knew nothing past their training cutoff, and came in one flavor from one provider. A lot has changed since then. Models are generally better, and the playing field has gotten crowded with providers and options, closed-source or otherwise: OpenAI, Anthropic, Google, xAI, Meta, Mistral, Alibaba, DeepSeek, and the open-weight models you can run yourself with <a href="https://ollama.com/">Ollama</a> or <a href="https://lmstudio.ai/">LM Studio</a>.</p>
<p>My objective was to show the community (WUR staff, postdocs, and PhD candidates, plus our kids and other family) that we can go beyond the chat window and put these models to productive use, and to stress <mark>calling them programmatically</mark>.</p>
<p>What makes these models useful is the context you supply yourself: a system instruction, your own documents, a tool the model is allowed to call. My own use cases are unglamorous. I fix my writing with them, and I draft cover letters from a system prompt that tells the model to act as a career coach, to say upfront when I’m not a good fit for the job, and to avoid clichés and flashy words.</p>
<p>I showed two demos about using LLMs for data extraction from unstructured data.</p>
<p>The first extracts (OCRs) the numbers from a <a href="https://nwcommons.nwciowa.edu/schuitemanethiopia/256/">photograph of a blackboard</a> chalked in Mettu in 1972, with Ethiopian medical statistics on it: square miles, population, hospitals, health centers, health stations, beds, and the USA in the next column for comparison. One prompt asking for valid CSV, safe for parsing, in tidy long format, and it comes back as something <a href="https://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html"><code>read.csv</code></a> takes without complaint.</p>
<p>The second is asking LLMs to extract data and return structured output. Hand the <a href="https://ai.google.dev/gemini-api/docs/structured-output">Gemini API</a> a <code>responseSchema</code> and an Amharic rental listing, the kind written in one breath with no punctuation and a phone number at the end, and you get back JSON: the price in ETB, the plot area in square meters, bedroom and bathroom counts, the broker’s commission, and the address both as written and transliterated. That is how a channel full of ads becomes a dataset.</p>
<p>I ended with questions to the audience. What are your use cases? Will this take our jobs? What role does it play in economic development back home? Is Ethiopia lagging behind wrt AI usage and literacy?</p>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p>At that time, July 13, 2025, Ethiopia <strong>was not</strong> on Anthropic’s list of <a href="https://www.anthropic.com/supported-countries">supported countries</a>.<br>
The tools keep getting cheaper and more capable, but not everywhere and not for everyone.</p>
</div>
</div>
<p><em>Slides: <a href="https://eyayaw.github.io/talks/leveraging-genai-ethio-wage-july-2025/">Leveraging Gen AI</a>, source <a href="https://github.com/eyayaw/talks/tree/main/leveraging-genai-ethio-wage-july-2025">on GitHub</a>.</em></p>
<p>—— <strong>Thank you for stopping by!</strong> 💚💛❤️ ——</p>



 ]]></description>
  <category>talk</category>
  <category>python</category>
  <guid>https://eyayaw.github.io/posts/leveraging-genai/</guid>
  <pubDate>Sat, 12 Jul 2025 22:00:00 GMT</pubDate>
</item>
<item>
  <title>Setting up a new Mac</title>
  <dc:creator>Eyayaw Beze</dc:creator>
  <link>https://eyayaw.github.io/posts/setting-up-a-new-mac/</link>
  <description><![CDATA[ 




<p>I finally treated myself to a new Mac Mini for completing my PhD. In the past, setting up a new machine meant hours of manual downloading, clicking through installers, and tediously configuring apps and tweaking default OS settings. However, over time I learned setup automation, so now I can get everything up and running quickly with minimal manual effort.</p>
<p>What have I learned? Writing a set of shell scripts that handle everything from app installation to config, and setting default settings. This post walks through my approach to programmatically setting up a new Mac and the tools that make it possible.</p>
<div class="quarto-layout-panel" data-layout-ncol="2">
<div class="quarto-layout-row">
<div class="quarto-layout-cell" style="flex-basis: 50.0%;justify-content: flex-start;">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://eyayaw.github.io/posts/setting-up-a-new-mac/mac-mini-closeup2.jpg" class="img-fluid figure-img"></p>
<figcaption>The new Mac Mini</figcaption>
</figure>
</div>
</div>
<div class="quarto-layout-cell" style="flex-basis: 50.0%;justify-content: flex-start;">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://eyayaw.github.io/posts/setting-up-a-new-mac/welcome-to-mac.jpg" class="img-fluid figure-img"></p>
<figcaption>Initial setup process completed, ready to hit <code>./bootstrap.sh</code></figcaption>
</figure>
</div>
</div>
</div>
</div>
<section id="tldr" class="level2">
<h2 class="anchored" data-anchor-id="tldr">TL;DR</h2>
<p><strong>Apps and CLI tools</strong>: I use <a href="https://github.com/Homebrew/brew">Homebrew</a> as my package manager to install everything from <code>1Password</code> and <code>Raycast</code> to dev tools like <code>Python</code>, <code>R</code>, and <code>DuckDB</code>. For Mac App Store apps, I use <a href="https://github.com/mas-cli/mas"><code>mas</code></a>.</p>
<p><strong>Configs</strong>: I manage my configs with <a href="https://www.gnu.org/software/stow/">GNU Stow</a>, which creates symlinks from my version-controlled <a href="https://github.com/eyayaw/dotfiles"><code>dotfiles</code></a> to the proper locations in my <code>${HOME}</code> directory.</p>
<p><strong>macOS defaults</strong>: I apply custom macOS settings through a script that changes some default options (like enabling list view in Finder and disabling “natural” scrolling).</p>
</section>
<section id="the-automation-process" class="level2">
<h2 class="anchored" data-anchor-id="the-automation-process">The automation process</h2>
<p>Package managers are the key to automation. I remember the days when I used to spend hours manually downloading and installing apps until I discovered tools like <a href="https://manpages.ubuntu.com/manpages/xenial/man8/apt.8.html"><code>apt</code></a> for Ubuntu and <a href="https://brew.sh"><code>Homebrew</code></a> for macOS. Another milestone was when I discovered about <a href="https://wiki.archlinux.org/title/Dotfiles">dotfiles</a>, configs that you can version-control and use across machines.<sup>1</sup></p>
<p>By combining package management with dotfiles, and macOS default settings, I created a system that helps me set up a new machine with all my favorite tools and configs in a fraction of the time it used to take me. There are three main components of this system: (1) package management with Homebrew, (2) GNU Stow-ed dotfiles, and (3) customizing macOS defaults.</p>
</section>
<section id="component-1-package-management-with-homebrew" class="level2">
<h2 class="anchored" data-anchor-id="component-1-package-management-with-homebrew">Component 1: Package Management with Homebrew</h2>
<p>macOS, unlike many Linux distros, doesn’t come with a built-in package manager. <a href="https://docs.brew.sh/">Homebrew</a> fills this (“the missing package manager”) gap, becoming the first tool I install on any new Mac:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb1-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">/bin/bash</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-c</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">$(</span><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">curl</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-fsSL</span> https://raw.githubusercontent.com/Homebrew/install/master/install.sh<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">)</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span></span></code></pre></div></div>
<p>Homebrew lets me install both GUIs (casks) and cli tools (formulae) and through simple commands. Here is how I install my favorite packages via Homebrew:</p>
<section id="core-tools" class="level3">
<h3 class="anchored" data-anchor-id="core-tools">Core tools</h3>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb2-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Password and secrets management</span></span>
<span id="cb2-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> 1password</span>
<span id="cb2-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install 1password-cli</span>
<span id="cb2-4"></span>
<span id="cb2-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Application launchers (with window management, clipboard history and other features)</span></span>
<span id="cb2-6"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> raycast alfred</span>
<span id="cb2-7"></span>
<span id="cb2-8"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> arc  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># modern browser</span></span>
<span id="cb2-9"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> karabiner-elements  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># for keyboard customization</span></span>
<span id="cb2-10"></span>
<span id="cb2-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Terminal environment</span></span>
<span id="cb2-12"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> ghostty  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># terminal emulator</span></span>
<span id="cb2-13"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install git <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># not to interfere with the system: macOS comes with a pre-installed version</span></span>
<span id="cb2-14"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install starship <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># shell prompts</span></span>
<span id="cb2-15"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install tmux  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># terminal multiplexer</span></span>
<span id="cb2-16"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install stow  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># for dotfiles management</span></span></code></pre></div></div>
</section>
<section id="command-line-utilities" class="level3">
<h3 class="anchored" data-anchor-id="command-line-utilities">Command-line utilities</h3>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb3-1"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install tldr  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># simplified man pages</span></span>
<span id="cb3-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Better alternatives to standard tools</span></span>
<span id="cb3-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install fzf <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># a fuzzy finder</span></span>
<span id="cb3-4"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install bat ripgrep fd  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># modern alts for cat, grep, find</span></span>
<span id="cb3-5"></span>
<span id="cb3-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Additional CLI tools</span></span>
<span id="cb3-7"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install gh  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># GitHub CLI</span></span>
<span id="cb3-8"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install mas  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># for Mac App Store apps</span></span></code></pre></div></div>
</section>
<section id="dev-environments" class="level3">
<h3 class="anchored" data-anchor-id="dev-environments">Dev environments</h3>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Programming languages</span></span>
<span id="cb4-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> r <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># the cask version is recommended, https://x.com/jimhester_/status/1374342249568481288</span></span>
<span id="cb4-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install python lua</span>
<span id="cb4-4"></span>
<span id="cb4-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Package managers and code quality tools</span></span>
<span id="cb4-6"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install uv ruff  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Python package manager and linter</span></span>
<span id="cb4-7"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> tap r-lib/rig <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">&amp;&amp;</span> <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> rig  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># R version manager</span></span>
<span id="cb4-8"></span>
<span id="cb4-9"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Editors and IDEs</span></span>
<span id="cb4-10"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> visual-studio-code zed  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># modern code editors</span></span>
<span id="cb4-11"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> visual-studio-code@insiders zed@preview  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># unstable versions</span></span>
<span id="cb4-12"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> rstudio positron  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># R &amp; Python IDEs by Posit</span></span>
<span id="cb4-13"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install neovim  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># for btw</span></span></code></pre></div></div>
<p>I also install typefaces (mostly coding fonts). My <a href="https://github.com/eyayaw/dotfiles/blob/main/scripts/install/fonts.sh">fonts installer script</a> copies my purchased fonts to <code>~/Library/Fonts/</code>, including <a href="https://monolisa.dev">MonoLisa</a> and <a href="https://usgraphics.com/products/berkeley-mono">Berkeley Mono</a>. The free ones can be installed via Homebrew, such as <a href="https://github.com/JetBrains/JetBrainsMono">JetBrains Mono</a> and <a href="https://github.com/githubnext/monaspace">Monaspace</a>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Fonts</span></span>
<span id="cb5-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> font-<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">{jetbrains-mono</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">monaspace</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">ibm-plex-mono}</span></span>
<span id="cb5-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">brew</span> install <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--cask</span> font-<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">{fontawesome</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">inter</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">open-sans</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">roboto}</span></span></code></pre></div></div>
<p>I also install Mac App Store apps, mostly not free:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Mac App Store apps (using mas)</span></span>
<span id="cb6-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">mas</span> install 441258766  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Magnet</span></span>
<span id="cb6-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">mas</span> install 424389933  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Final Cut Pro</span></span>
<span id="cb6-4"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">mas</span> install 1091189122  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Bear Notes</span></span></code></pre></div></div>
<p>As an academic, I use <a href="https://zotero.org">Zotero</a> for reference management (sometimes as read-later storage, and has an impressive PDF reader). For academic citations, I use <a href="https://retorque.re/zotero-better-bibtex">Better BibTex</a> to automatically generate citations keys in Zotero. I write a script to install this addon, instead of <a href="https://retorque.re/zotero-better-bibtex/installation/">manually installing</a> it.</p>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center" data-bs-toggle="collapse" data-bs-target=".callout-1-contents" aria-controls="callout-1" aria-expanded="true" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Tip
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-1" class="callout-1-contents callout-collapse collapse show">
<div class="callout-body-container callout-body">
<p>The <a href="https://retorque.re/zotero-better-bibtex/citing/formulas/index.html">citation key formula</a> I use is <code>authEtal(sep='_').lower+year.prefix(_)</code>, which generates citation keys like <code>glaeser_2010</code>, <code>glaeser_gyourko_2009</code>, and <code>glaeser_etal_2020</code> for single, two, and three or more authors, respectively.</p>
<p>You can read <a href="https://retorque.re/zotero-better-bibtex/citing/index.html#configurable-citekey-generator">this</a> on how to configure the citation key generator.</p>
</div>
</div>
</div>
<p>Lastly, I wrote a script that installs <a href="https://github.com/tmux-plugins/tpm">tpm</a>, a plugin manager for <a href="https://github.com/tmux/tmux">tmux</a>, inspired by typecraft-dev’s <a href="https://github.com/typecraft-dev/crucible/blob/main/install-tpm.sh">crucible</a>.</p>
<p>All of these installations are automated in a single script <a href="https://github.com/eyayaw/dotfiles/blob/main/scripts/install/install.sh"><code>scripts/install/install.sh</code></a>, which sources the relevant scripts. If you’re curious about in the full list of my favorite tools, see <a href="../../my-setup.html">here</a>.</p>
</section>
</section>
<section id="component-2-configuration-management-with-dotfiles" class="level2">
<h2 class="anchored" data-anchor-id="component-2-configuration-management-with-dotfiles">Component 2: Configuration Management with Dotfiles</h2>
<p>After installing apps, the next challenge is configuring them. This is where <a href="https://wiki.archlinux.org/title/Dotfiles">dotfiles</a> come in, which are configs (often hidden, starting with a dot, hence the name) that control how apps behave.</p>
<p>Dotfiles are simply text-based configs. Examples include: <code>.zshrc</code> for Zsh config, <code>.gitconfig</code> for Git preferences, and various configs for editors, terminal emulators, and other tools.</p>
<p>By storing these files in a version-controlled repository, we can easily track changes and sync configs across machines.</p>
<section id="managing-dotfiles-with-gnu-stow" class="level3">
<h3 class="anchored" data-anchor-id="managing-dotfiles-with-gnu-stow">Managing dotfiles with GNU Stow</h3>
<p>Managing my dotfiles was challenging, and I often lost them since they typically reside in the home directory, which I don’t sync with iCloud Drive. (Of course, you can!) Then I discovered <a href="https://www.gnu.org/software/stow/">GNU Stow</a>, through Dreams of Autonomy’s <a href="https://www.youtube.com/watch?v=y6XCebnB9gs">video</a>.</p>
<p>Stow creates symlinks from a repository (in my case located at <code>~/Developer/dotfiles/</code>) to the locations where apps expect to find their configs, usually in the home directory (i.e., <code>~</code>) and its subdirectories such as <code>.config</code>.</p>
<p>Here is how I structure my dotfiles repository:</p>
<pre><code>~/Developer/dotfiles/
├── zsh/            # Shell configs
│   ├── .zshrc
│   ├── .zshenv
│   └── .zsh_aliases
├── zed/            # Zed editor settings
│   └── .config/
│       └── zed/
│           ├── settings.json
│           └── keymap.json
├── git/
│   └── .gitconfig
└── ... other tools</code></pre>
<p>Notice how the directory structure (for each package) mirrors where these files would normally live in my home directory.This is key to how Stow works.</p>
<p>To apply these configs, I simply run:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb8-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># or run from anywhere by adding the `--dir` flag with the location of your dotfiles</span></span>
<span id="cb8-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># in my case `--dir=${HOME}/Developer/dotfiles`</span></span>
<span id="cb8-3"><span class="bu" style="color: null;
background-color: null;
font-style: inherit;">cd</span> ~/Developer/dotfiles</span>
<span id="cb8-4"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">stow</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--target</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">${HOME}</span> zsh ghostty karabiner git tmux zed R rstudio</span></code></pre></div></div>
<div id="screenshot-stow-in-action" class="quarto-figure quarto-figure-center anchored">
<figure class="figure">
<p><img src="https://eyayaw.github.io/posts/setting-up-a-new-mac/screenshot-dotfiles-symlinks-1.png" class="img-fluid figure-img"></p>
<figcaption>Symlinked dotfiles<br><em>This screenshot shows Stow in action, where configs in my home directory (left side) are pointing to their actual location in my dotfiles directory (right side)</em>.</figcaption>
</figure>
</div>
<p>As shown in the screenshot above, Stow creates symbolic links from the appropriate locations in my home directory to the files in my dotfiles repository. When I update a config file in my dotfiles, the change is automatically reflected in the corresponding config file and hence the package that uses the config, and vice versa.</p>
<p>For example, here is a simplified version of my Zed config which is located at <code>~/Developer/dotfiles/zed/.config/zed/settings.json</code>:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode json code-with-copy"><code class="sourceCode json"><span id="cb9-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb9-2">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"base_keymap"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"VSCode"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-3">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"vim_mode"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">true</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-4">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"relative_line_numbers"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">true</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-5">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"buffer_font_family"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"MonoLisa"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-6">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"buffer_font_size"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-7">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"ui_font_size"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-8">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"theme"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">{</span></span>
<span id="cb9-9">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"mode"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"system"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-10">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"light"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Zed Legacy: Solarized Light"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-11">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"dark"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"One Dark"</span></span>
<span id="cb9-12">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">},</span></span>
<span id="cb9-13">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"wrap_guides"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">[</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">72</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">80</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">,</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">120</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">]</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb9-14">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">"soft_wrap"</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"editor_width"</span></span>
<span id="cb9-15"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">}</span></span></code></pre></div></div>
<p>With Stow, this file lives in my dotfiles repository but appears to Zed as if it’s in the expected <code>~/.config/zed/</code> location.</p>
</section>
</section>
<section id="component-3-customizing-macos-defaults" class="level2">
<h2 class="anchored" data-anchor-id="component-3-customizing-macos-defaults">Component 3: Customizing macOS Defaults</h2>
<p>macOS comes with many default settings that I immediately change on a new system. Rather than clicking through System Preferences, I use a <a href="https://github.com/eyayaw/dot/blob/main/scripts/macos/set_defaults.sh">script</a> that modifies these settings programmatically, which are mostly adapted from Mathias Bynens’ <a href="https://github.com/mathiasbynens/dotfiles/blob/master/.macos">dotfiles</a>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb10-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Use list view in Finder by default</span></span>
<span id="cb10-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">defaults</span> write com.apple.finder FXPreferredViewStyle <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-string</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Nlsv"</span></span>
<span id="cb10-3"></span>
<span id="cb10-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Disable "natural" scrolling (which feels unnatural to many)</span></span>
<span id="cb10-5"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">defaults</span> write NSGlobalDomain com.apple.swipescrolldirection <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-bool</span> false</span>
<span id="cb10-6"></span>
<span id="cb10-7"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Faster keyboard repeat rate</span></span>
<span id="cb10-8"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">defaults</span> write NSGlobalDomain KeyRepeat <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-int</span> 1</span>
<span id="cb10-9"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">defaults</span> write NSGlobalDomain InitialKeyRepeat <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-int</span> 10</span>
<span id="cb10-10"></span>
<span id="cb10-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Show hidden files in Finder</span></span>
<span id="cb10-12"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">defaults</span> write com.apple.finder AppleShowAllFiles <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-bool</span> true</span>
<span id="cb10-13"></span>
<span id="cb10-14"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Disable automatic capitalization, smart quotes, etc.</span></span>
<span id="cb10-15"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">defaults</span> write NSGlobalDomain NSAutomaticCapitalizationEnabled <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-bool</span> false</span></code></pre></div></div>
<p>These commands modify the property list files that store macOS preferences. After running my settings script, I restart the machine to ensure all changes take effect.</p>
</section>
<section id="putting-it-all-together" class="level2">
<h2 class="anchored" data-anchor-id="putting-it-all-together">Putting it all together</h2>
<p>My complete setup process is orchestrated by a single <a href="https://github.com/eyayaw/dotfiles/blob/main/bootstrap.sh"><code>bootstrap.sh</code></a> script that:</p>
<ol type="1">
<li>Installs Homebrew</li>
<li>Installs all packages, fonts, and apps</li>
<li>Sets up plugin managers for tools like tmux</li>
<li>Configures macOS default settings</li>
<li>Deploys all dotfiles using Stow</li>
</ol>
<p>Here is the actual workflow:</p>
<ol type="1">
<li>Boot up the new Mac.</li>
<li>Clone my dotfiles repository: <code>git clone --depth=1 https://github.com/eyayaw/dotfiles.git ~/Developer/dotfiles</code>, macOS comes with a pre-installed version of Git.</li>
<li>Run the bootstrap script: <code>cd ~/Developer/dotfiles &amp;&amp; ./bootstrap.sh</code>.<sup>2</sup></li>
<li>After a reboot, everything is ready to go.</li>
</ol>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p>Installing Homebrew takes longer than I would like because of the Command Line Tools and other dependencies (I assume).</p>
<p>You may be asked for your password repeatedly, so you might need to stay nearby.</p>
<p>You can install Homebrew in <a href="https://docs.brew.sh/Installation#unattended-installation">unattended mode</a> by setting <code>NONINTERACTIVE=1</code>. If you want to install everything unattended, there may be a way through providing the password upfront (in bash <code>set -v</code> and additional <a href="https://github.com/mathiasbynens/dotfiles/blob/b7c7894e7bb2de5d60bfb9a2f5e46d01a61300ea/.macos#L9C1-L13C80">hack</a>), but I couldn’t get it to work in my setup.</p>
</div>
</div>
<p>The entire process takes less than an hour, most of which is just waiting for downloads to complete. When I return to the computer, all my tools are installed and configured exactly the way I like them.</p>
<p>While saving time is the most obvious benefit, this approach offers other advantages. My dev environment is familiar across machines, the scripts serve as documentation of <a href="../../my-setup.html">my setup</a>. I can easily add new tools or update configs. If something goes wrong, I can quickly restore my environment because my dotfiles are version-controlled.</p>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>Setting up a new machine used to be a gruelling, unproductive task. Now it’s a matter of running a single script and taking a coffee break. This automation has dramatically reduced the friction of moving to new machine and ensures I immediately have access to my dev tools on a fresh system.</p>
<p>The tools outlined here—Homebrew, GNU Stowed dotfiles, and defaults scripts—form a powerful trio for Mac setup automation. If you’re still setting up machines manually, I encourage you to explore automation. There are hundreds of <a href="https://github.com/search?q=dotfiles&amp;type=repositories">amazing dotfiles</a> on GitHub.</p>
<p>How do you set up your dev environment? I would love to hear about your approach and favorite tools in the comments!</p>
<p>—— <strong>Thank you for stopping by!</strong> 💚💛❤️ ——</p>
<hr>
<p>Expand the appendix if you want to skim through simplified versions of some of my configs.</p>
<details>
<summary class="h2">
Appendix
</summary>
<section id="karabiner-elements" class="level3">
<h3 class="anchored" data-anchor-id="karabiner-elements">Karabiner-Elements</h3>
<p>I use Karabiner-Elements for keyboard customization, with a “hyper key” + sublayer workflow. My <a href="https://github.com/eyayaw/karapyner">karapyner</a> repository includes Python scripts for configuring Karabiner with:</p>
<ul>
<li>Caps Lock as a dual-function key (Escape when tapped, Hyper Key when held)</li>
<li>Custom sublayers for different categories of actions:
<ul>
<li><code>hyper + b</code>: Browser commands</li>
<li><code>hyper + o</code>: Open apps</li>
<li><code>hyper + w</code>: Window management</li>
<li><code>hyper + r</code>: Raycast commands</li>
</ul></li>
</ul>
<p>For more on this approach, check out Max Stoiber’s excellent <a href="https://www.youtube.com/watch?v=j4b_uQX3Vu0">video</a>.</p>
</section>
<section id="r" class="level3">
<h3 class="anchored" data-anchor-id="r">R</h3>
<p>My R setup is made up of R default options and environment variables, and linting rules for the <a href="https://github.com/r-lib/lintr/"><code>lintr</code></a> package.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb11-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># .Rprofile</span></span>
<span id="cb11-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">options</span>(</span>
<span id="cb11-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">prompt =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"R&gt; "</span>,</span>
<span id="cb11-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">continue =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" "</span>,</span>
<span id="cb11-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">example.ask =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>,</span>
<span id="cb11-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">warnPartialMatchArgs =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>,</span>
<span id="cb11-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">repos =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(</span>
<span id="cb11-8">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rspm =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://packagemanager.rstudio.com/all/latest"</span>,</span>
<span id="cb11-9">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">CRAN =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://cran.r-project.org"</span></span>
<span id="cb11-10">  ),</span>
<span id="cb11-11">  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># data.table options</span></span>
<span id="cb11-12">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">datatable.print.colnames =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"auto"</span>,</span>
<span id="cb11-13">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">datatable.print.class =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>,</span>
<span id="cb11-14">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">datatable.optimize =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span></span>
<span id="cb11-15">)</span></code></pre></div></div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb12-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># .Renviron</span></span>
<span id="cb12-2"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">R_MAX_VSIZE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>48Gb</span></code></pre></div></div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb13-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># .lintr</span></span>
<span id="cb13-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">linters:</span> linters_with_defaults<span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">(</span></span>
<span id="cb13-3">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">assignment_linter</span> = NULL,</span>
<span id="cb13-4">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">line_length_linter</span> = line_length_linter<span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">(</span><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">110L</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">)</span><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">,</span></span>
<span id="cb13-5">    <span class="ex" style="color: null;
background-color: null;
font-style: inherit;">commented_code_linter</span> = NULL</span>
<span id="cb13-6">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">)</span></span>
<span id="cb13-7"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">encoding:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"UTF-8"</span></span></code></pre></div></div>
<p>Everything else can be found in the <a href="https://github.com/eyayaw/dotfiles">dotfiles repo</a>.</p>
</section></details>


</section>



<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>Scripts for application installation and changing default OS options are also part of dotfiles. See examples <a href="https://dotfiles.github.io/inspiration/">here</a>.↩︎</p></li>
<li id="fn2"><p>You may need to the change its permission with <code>chmod +x bootstrap.sh</code> or use <code>source bootstrap.sh</code>↩︎</p></li>
</ol>
</section></div> ]]></description>
  <category>macos</category>
  <category>config</category>
  <category>cli</category>
  <guid>https://eyayaw.github.io/posts/setting-up-a-new-mac/</guid>
  <pubDate>Thu, 24 Apr 2025 22:00:00 GMT</pubDate>
  <media:content url="https://eyayaw.github.io/posts/setting-up-a-new-mac/welcome-to-mac.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Base R</title>
  <dc:creator>Eyayaw Beze</dc:creator>
  <link>https://eyayaw.github.io/posts/base-r/</link>
  <description><![CDATA[ 




<p>You probably engaged in endless debates about {tidyverse} versus {base} R and/or {data.table}.<sup>1</sup> But what does “base R” actually mean? Clarifying this may unite the <a href="https://x.com/hadleywickham/status/903103150332280832">dichotomy</a> (or trichotomy), that we are all R. But, wait, we’re all R? What about “R or Python”? Oh no, don’t let the <a href="https://x.com/hadleywickham/status/1280340931657564160">holy war</a> escalate! Again, we’re all one . . . embrace both like me, like <a href="https://posit.co/blog/rstudio-is-becoming-posit/">RStudio</a> . . . in fact, become a polyglot.</p>
<p>Back to base R, it is typically used to refer to the set of packages that come or distributed with R<sup>2</sup>. The name “base” might be confusing, because of {base}, the fundamental package that “contains the basic functions which let R function as a language”. However, base R encompasses more than {base}. Pause and count how many base R packages you can name. . . .</p>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1">lib_path <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">normalizePath</span>(.Library)</span>
<span id="cb1-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] "/Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library"</span></span>
<span id="cb1-3">fields <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Title"</span>)</span>
<span id="cb1-4"></span>
<span id="cb1-5">installed_pkgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">installed.packages</span>(lib_path, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">fields=</span>fields) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.frame</span>()</span>
<span id="cb1-6"></span>
<span id="cb1-7">base_pkgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">subset</span>(installed_pkgs, Priority <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"base"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">select =</span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Package"</span>, fields))</span>
<span id="cb1-8">high_pkgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">subset</span>(installed_pkgs, Priority <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"high"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">select =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Package"</span>, fields))</span></code></pre></div></div>
</details>
</div>
<p>There are 14 base R packages, in R 4.6.1. Below is a complete list.</p>
<div class="callout callout-style-default callout-important no-icon callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon no-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Important</span>Base R Packages
</div>
</div>
<div class="callout-body-container callout-body">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sprintf</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"- %s --- %s"</span>, base_pkgs<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>Package, base_pkgs<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>Title) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|&gt;</span></span>
<span id="cb2-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cat</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sep =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span></code></pre></div></div>
</details>
<ul>
<li>base — The R Base Package</li>
<li>compiler — The R Compiler Package</li>
<li>datasets — The R Datasets Package</li>
<li>graphics — The R Graphics Package</li>
<li>grDevices — The R Graphics Devices and Support for Colours and Fonts</li>
<li>grid — The Grid Graphics Package</li>
<li>methods — Formal Methods and Classes</li>
<li>parallel — Support for Parallel Computation in R</li>
<li>splines — Regression Spline Functions and Classes</li>
<li>stats — The R Stats Package</li>
<li>stats4 — Statistical Functions using S4 Classes</li>
<li>tcltk — Tcl/Tk Interface</li>
<li>tools — Tools for Package Development</li>
<li>utils — The R Utils Package</li>
</ul>
<p>See <a href="https://search.r-project.org/R/refmans/utils/html/installed.packages.html">?installed.packages</a>.</p>
</div>
</div>
<p>Some of these packages I did not know exist, let alone knowingly use. So the list shows that base R includes packages attached to your R session by default (i.e.&nbsp;default packages), and other few important packages used behind the scenes. Therefore, in short, base R = {{base}, {default}, {other}} packages.</p>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1">default_pkgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">in_plain_r</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cat(getOption('defaultPackages'))"</span>)</span>
<span id="cb3-2">default_pkgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">strsplit</span>(default_pkgs, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" "</span>)[[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>]]</span></code></pre></div></div>
</details>
</div>
<p>The default packages are attached when R starts up. As of R 4.6.1 these are: {datasets}, {utils}, {grDevices}, {graphics}, {stats}, and {methods}. Obviously, add {base} to the list. You can also see them in the search path (prepended with <code>package:</code>):</p>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">in_plain_r</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cat(search())"</span>)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>[1] ".GlobalEnv package:stats package:graphics package:grDevices package:utils package:datasets package:methods Autoloads package:base"</code></pre>
</div>
</div>
<p>You can control the list of default packages by setting the environment variable <code>R_DEFAULT_PACKAGES</code> with a comma-separated list of packages. You may ask, why do we need to do that? Well:</p>
<blockquote class="blockquote">
<p>Changing the set of default packages is normally used to reduce the set for speed when scripting: in particular not using methods will reduce the start-up time by a factor of up to two. But it can also be used to customize R, e.g.&nbsp;for class use. Rscript also checks the environment variable R_SCRIPT_DEFAULT_PACKAGES; if set, this takes precedence over R_DEFAULT_PACKAGES.</p>
<p>— <a href="https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Default-packages">R manuals #Default-packages</a></p>
</blockquote>
<div class="callout callout-style-default callout-tip callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Tip
</div>
</div>
<div class="callout-body-container callout-body">
<p>To modify the list of default packages, set the env variable <code>R_DEFAULT_PACKAGES</code> in your <code>.Renviron</code>, or set the <code>defaultPackages</code> option in <code>.Rprofile</code> (see <a href="https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Default-packages">R manuals</a>). If you set <code>R_DEFAULT_PACKAGES=NULL</code>, then only {base} is loaded.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode sh code-with-copy"><code class="sourceCode bash"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ~/.Renviron</span></span>
<span id="cb6-2"><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">R_DEFAULT_PACKAGES</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span>base,utils,stats,methods</span></code></pre></div></div>
<p>Or,</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ~/.Rprofile</span></span>
<span id="cb7-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">options</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">defaultPackages=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"base"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"utils"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"stats"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"methods"</span>))</span></code></pre></div></div>
<p>As you can see below, only {base} is loaded when no default packages are provided.</p>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">system</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'export R_DEFAULT_PACKAGES=NULL;R --vanilla --no-echo --quiet -e "cat(search())"'</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">intern=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>[1] ".GlobalEnv Autoloads package:base"</code></pre>
</div>
</div>
</div>
</div>
<div class="cell">
<details class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1">other_base_pkgs <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setdiff</span>(base_pkgs<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>Package, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"base"</span>, default_pkgs))</span></code></pre></div></div>
</details>
</div>
<p>The other base R packages are {compiler}, {grid}, {parallel}, {splines}, {stats4}, {tcltk}, and {tools}. These (or some) may not be familiar to you (myself included) if you’re not a package developer. I have used {tools} for example to check package dependencies (e.g., <code>tools::package_dependencies("data.table", recursive = TRUE)</code>), and work with file paths (see my <a href="https://x.com/eyayaw/status/1488568897334915081">tweet</a>). I have also lightly used {grid} to add annotations to base plots. Have you ever used {tcltk}?</p>
<p>In sum, when people say “I use base R”, they probably mean {base} functions because the comparison is mostly about data manipulation/wrangling, not say visualization (base {graphics} vs {ggplot2}) or other tasks.<sup>3</sup> <a href="http://varianceexplained.org/r/teach_ggplot2_to_beginners/">Not true</a>? See examples <a href="https://x.com/hadleywickham/status/1231252596712771585">here</a>, <a href="https://rdatatable-community.github.io/The-Raft/posts/2024-02-18-dt_particularities-toby_hocking/">here</a>. In other words, if you use <code>df[df$x &gt; 1, ]</code> instead of say <code>dplyr::filter(df, x &gt; 1)</code>, or <code>data.table::as.data.table(df)[x &gt; 1, ]</code>, you’re a base R-er. However, base R “should” refer to the suite of built-in packages in R (14 as of R 4.6.1), like {tidyverse} which is a collection of diverse (but related) packages <a href="https://www.tidyverse.org/packages/">from everyday data analyses to specialized packages</a>.</p>




<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>If you want to read some, see <a href="https://blog.ephorie.de/why-i-dont-use-the-tidyverse">here</a>, <a href="https://x.com/TimTeaFan/status/1754964772603216171">here</a>, <a href="https://x.com/prisonrodeo/status/1612396078913011713"><del>here</del></a>, <a href="https://x.com/tylermorganwall/status/1599238413580132354">“a poem about the battle”</a>, and <a href="https://x.com/search?q=%22base%22%20(tidyverse%20OR%20data.table)%20(%23rstats)&amp;src=typed_query&amp;f=top">many more</a> in the bird site. Sorry for the X links. Most of the debates occurred during the #rstats Twitter era.↩︎</p></li>
<li id="fn2"><p>Packages that come built-in with R may include more than base R packages: <code>installed.packages(priority="high")</code> shows <code>base + recommended</code> packages.↩︎</p></li>
<li id="fn3"><p>Most, if not all, base R-ers use {ggplot2}, lovingly.↩︎</p></li>
</ol>
</section></div> ]]></description>
  <category>r</category>
  <category>base-r</category>
  <guid>https://eyayaw.github.io/posts/base-r/</guid>
  <pubDate>Sun, 06 Apr 2025 22:00:00 GMT</pubDate>
  <media:content url="https://eyayaw.github.io/posts/base-r/base-r.svg" medium="image" type="image/svg+xml"/>
</item>
<item>
  <title>Welcome to my Blog</title>
  <dc:creator>Eyayaw Beze</dc:creator>
  <link>https://eyayaw.github.io/posts/welcome-to-my-blog/</link>
  <description><![CDATA[ 




<p>This is the first post in my blog. Welcome!</p>
<p><img src="https://eyayaw.github.io/posts/welcome-to-my-blog/thumbnail.jpg" class="img-fluid"></p>
<p>I will blog about what I have learned in my <code>R</code> and <code>Python</code> journey. Stay tuned!</p>



 ]]></description>
  <category>meta</category>
  <guid>https://eyayaw.github.io/posts/welcome-to-my-blog/</guid>
  <pubDate>Sat, 30 Jul 2022 22:00:00 GMT</pubDate>
  <media:content url="https://eyayaw.github.io/posts/welcome-to-my-blog/thumbnail.jpg" medium="image" type="image/jpeg"/>
</item>
</channel>
</rss>
