The stuff I do

Using a systemd timer to monitor my laptop's battery

statox — Fri, 27 Mar 2026 00:00:00 +0000

I've been using i3 as my window manager for the past 7 years. It's great and you should check it out! However, one issue I have is that by default I don't get an obvious notification when my battery is low. My personal laptop is getting old and the battery significantly drains when it is put to sleep. So recently it happened several times that I woke it up, didn't check my battery, and had it die a few minutes later because it was on super low battery.

So I created a simple bash script which checks the battery level and uses notify-send to display a desktop notification to remind me to charge the computer. The question was how to run it:

I wanted to run it at startup and then every 5 minutes
I wanted that to be easily configurable from my dotfiles
I didn't want to use cron because setting cron automatically is always a bit wacky:
- I don't like the scheduling syntax
- I don't like the fact that I can't just put a crontab file somewhere in my ~/.config and have cron pick it up
- Debugging is not always straightforward, I have to handle redirection to a log file, add echo to my scripts to log start and stop time
- ...

So I finally looked into systemd timers, which I've been meaning to do for a long time but I was worried it would add too much complexity to my dotfiles system.

I was wrong.

The solution 🔗

To set up a systemd service at the user level you need 2 files:

monitor_battery.service which defines how the service runs, that's not much more than pointing to the path of the script (Here ~/.bin/monitor-battery):

[Unit]
Description=Low battery popup notification

[Service]
Type=oneshot
ExecStart=%h/.bin/monitor-battery

# notify-send requires a running D-Bus session

Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus

monitor_battery.timer which defines when the service runs, here 1 minute after boot and then every 5 minutes after that:

[Unit]
Description=Check battery level every 5 minutes

[Timer]
OnBootSec=1min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

After putting these files in ~/.config/systemd/ I just have to run 2 commands once to enable and start the service:

systemctl --user enable --now monitor_battery.timer
systemctl --user start monitor_battery.timer

As they are idempotent I can just add them to the script which sets up my dotfiles (which I run once in a while when I change my configs) and that's it!

Result 🔗

The good part is that now this script is managed like a regular systemd unit. So

I can use systemctl to start, stop, check the status of the service

Handling the logs is super easy:

I don't have to handle inconvenient shell redirections in my crontab (stuff like >> /var/log/monitor.log 2>&1)
Every echo in my script is sent to the logs automatically
By default systemd logs the startup and end times of the script so I don't need to pollute my script with additional logging
I can use journalctl --user -u monitor_battery.service to access the logs without having to find back the log file

Mar 27 11:42:05 hostname systemd[3972]: Starting monitor_battery.service - Low battery popup notification...
Mar 27 11:42:05 hostname monitor-battery[93724]: Fri Mar 27 11:42:05 AM CET 2026 level='95' status='Discharging' alerted='0'
Mar 27 11:42:05 hostname systemd[3972]: Finished monitor_battery.service - Low battery popup notification.
Mar 27 11:47:16 hostname systemd[3972]: Starting monitor_battery.service - Low battery popup notification...
Mar 27 11:47:16 hostname monitor-battery[97345]: Fri Mar 27 11:47:16 AM CET 2026 level='94' status='Discharging' alerted='0'
Mar 27 11:47:16 hostname systemd[3972]: Finished monitor_battery.service - Low battery popup notification.

Note how I should remove the echo "$(date) ... from the monitor-battery script to avoid duplicating the date

Moralité: Tailoring your tools to your needs is a good way to learn new stuff!

Resources 🔗

The monitor-battery script on Github
A StackExchange question "Cron vs. Systemd timers"
The Arch wiki page about Systemd timers and in particular its section As a cron replacement

Synchronized colorscheme toggling for WezTerm and Neovim

statox — Tue, 16 Dec 2025 00:00:00 +0000

TL;DR: I wanted a keybinding to toggle between light and dark colorschemes and have the change apply immediately to all my WezTerm windows and Neovim instances across multiple running processes, using pure Lua configurations. If you want to skip the context and jump straight to the implementation, go to the solution.

Recently I've been reworking my terminal setup. After years of using gnome-terminal with tmux, I made the jump to WezTerm and replaced most of my tmux workflow with WezTerm's built-in features. The switch has been great overall, and it was the opportunity to tackle an issue which had been bothering me for a long time: I couldn't easily toggle between light and dark colorschemes and have the change apply to both my terminal and all my Neovim instances simultaneously.

In this article I'll show how I solved this problem using a shared state file and some file watching magic. If you're using WezTerm and Neovim and want synchronized colorscheme switching, this might give you some ideas for your own setup.

The context 🔗

For colorschemes I've been using nightfox.nvim, specifically the nightfox variant for dark mode and dayfox for light mode. During the day I prefer a light theme, but when evening comes I want to switch to something easier on the eyes. One advantage of WezTerm is that by default it bundles a lot of colorschemes including the nightfox ones I was already using in Neovim.

I'm not using system-wide light/dark theme settings because I haven't integrated that well with my i3 window manager setup yet. That might change in the future, but for now I wanted something simpler that I could control directly from my terminal.

Here's what makes this tricky: I often have multiple WezTerm instances running at the same time. One for my frontend repo, another for backend, maybe a third for some side project. Each instance has several windows (the WezTerm name for tmux "tabs"), and potentially multiple Neovim instances spread across different windows. When I toggle between light and dark mode, I want all of them to update immediately.

The problem 🔗

What I needed was:

A keybinding in WezTerm to toggle between light and dark themes
All WezTerm windows (across multiple instances) to update their colorscheme
All Neovim instances to update their colorscheme automatically
Everything to happen immediately, without manual intervention
A pure Lua implementation without relying on shell commands for file operations

This seemed like it should be a common use case, but I couldn't find an existing solution that did exactly what I wanted.

What others have tried 🔗

I did some research and found that other people have faced similar issues, but the solutions weren't quite right for my needs:

This Reddit thread describes using a file to store the colorscheme name, but Neovim is responsible for changing the file. I wanted WezTerm to be in control. Also, their solution requires manually restarting WezTerm or reloading its config to apply changes.

This GitHub discussion has WezTerm updating the file, which is closer to what I wanted, but Neovim needs to be manually restarted to pick up the new colorscheme. Also, it relies on spawning a subprocess to run echo to write to the file. I wanted a pure Lua solution.

This blog post has an interesting approach where Neovim changes are based on the FocusGained autocommand event. The problem is that you need to remove and add focus to potentially all running Neovim instances for this to work, which is not ideal when you have many of them.

None of these solutions gave me the immediate, automatic synchronization I was looking for, so I decided to build my own.

My initial attempts 🔗

Before settling on the file-based approach, I tried a couple of other ideas that didn't quite work out:

First, I experimented with using an environment variable to share the colorscheme state between WezTerm and Neovim. The problem was timing and environment variables scope: I needed to set the environment variable before WezTerm reads its configuration, and then update it when toggling. I couldn't find a convenient way to make this work, especially since environment variables set in a running WezTerm instance wouldn't propagate to other already-running instances.

I also tried using WezTerm's window:set_config_overrides() API, which allows changing configuration for a specific window at runtime. This seemed promising at first, but the overrides only applied to the single window where the keybinding was pressed, not to all WezTerm instances. This meant I'd still need some way to communicate the colorscheme change to other instances, bringing me back to square one.

These experiments led me to realize that what I needed was a simple shared state mechanism that all instances could monitor independently. That's when I landed on the file-based approach.

The solution: High level overview 🔗

My approach uses a simple shared state file that both applications can read and watch for changes. Here's how it works:

The setup relies on a file at /tmp/colorscheme which contains a single line with either light or dark.

On the WezTerm side:

On startup, check for the file /tmp/colorscheme. If it doesn't exist, create it with a default value.
Read the file content and set the colorscheme accordingly (light → dayfox, dark → nightfox).
Provide a keybinding that reads the file, writes the opposite value, and reloads the configuration.

On the Neovim side:

On startup, read /tmp/colorscheme and set the appropriate colorscheme.
Set up a file watcher that monitors the file for changes.
When the file changes, automatically update the colorscheme.

The advantage of this approach is that WezTerm controls the toggle and all instances of both applications react immediately because they're all watching the same file.

Implementing the WezTerm side 🔗

The WezTerm implementation is split across a few modules to keep things organized. Let's walk through the key parts.

Managing colorscheme state 🔗

First, I created a colorscheme.lua module to handle reading, writing, and toggling the colorscheme state:

local module = {}

local light_colorscheme = 'dayfox'
local dark_colorscheme = 'nightfox'

-- Shared state file: written by WezTerm, watched by Neovim
local colorscheme_file_path = '/tmp/colorscheme'

module.default_colorscheme_mode = "light"

function create_colorscheme_file()
    local file = io.open(colorscheme_file_path, "w")
    file:write(module.default_colorscheme_mode)
    file:close()
end

function read_colorscheme_file()
    local file = io.open(colorscheme_file_path, "r")
    content = file:read()
    file:close()
    return content
end

function init_colorscheme()
    local success, content = pcall(read_colorscheme_file)

    if not success then
        wezterm.log_info('Could not read file content, using default')
        create_colorscheme_file()
        content = module.default_colorscheme_mode
    end

    if not (content == "light" or content == "dark") then
        wezterm.log_info('Invalid mode in file, using default', content)
        content = module.default_colorscheme_mode
    end

    if content == "light" then
        return light_colorscheme
    end

    return dark_colorscheme
end
module.init_colorscheme = init_colorscheme

The init_colorscheme() function does the heavy lifting: it reads the state file, handles cases where the file doesn't exist or contains invalid data, and returns the appropriate colorscheme name to use in the WezTerm configuration. The use of pcall() (protected call) is important here: it executes the read_colorscheme_file() function in protected mode, catching any errors (like the file not existing) and returning a success status instead of crashing. If the call fails, we create the file with a default value.

The file operations use Lua's standard I/O library: io.open() opens a file and returns a file handle, and file:read() reads its content. These are pure Lua functions, no shell commands needed.

See the complete colorscheme.lua file on GitHub

Toggling the colorscheme 🔗

The toggle function is straightforward. It reads the current mode, writes the opposite value to the file, and triggers a configuration reload:

function module.toggle_colorscheme(window, pane, config)
    local success, content = pcall(read_colorscheme_file)

    -- Lua's ternary-like expression: if content is "light", return "dark", else "light"
    local new_value = content == "light" and "dark" or "light"
    local file = io.open(colorscheme_file_path, "w")
    if file then
        file:write(new_value)
        file:close()
    end

    -- We reload the config which triggers setup_ui() which uses
    -- init_colorscheme() to setup the colorscheme based on the file content
    wezterm.reload_configuration()
end

When wezterm.reload_configuration() is called, it re-runs the entire configuration, which calls init_colorscheme() again and picks up the new value from the file.
Note Another approach is possible here: Rather than manually calling wezterm.reload_configuration() it is possible to use wezterm.add_to_config_reload_watch_list('/tmp/colorscheme'). With this config WezTerm will automatically reload its configuration when the file is changed. This could open the way to have a third party script controlling the colorscheme (for example a cronjob change the theme based on the time of the day).

Integrating into the main config 🔗

In my ui.lua module, I call init_colorscheme() during setup:

local colorscheme = require 'colorscheme'

function module.setup_ui(config)
    config.font_size = 11
    config.color_scheme = colorscheme.init_colorscheme()

    -- ... rest of UI configuration
end

And in the main wezterm.lua file, everything comes together:

local wezterm = require 'wezterm'
local ui = require 'ui'
local mappings = require 'mappings'

local config = wezterm.config_builder()

ui.setup_ui(config)
mappings.setup_bindings(config)

return config

See the complete ui.lua and wezterm.lua files on GitHub

Setting up the keybinding 🔗

Finally, in mappings.lua, I bind the toggle function to <leader>f:

local colorscheme = require 'colorscheme'

function module.setup_bindings(config)
    config.leader = { key = ' ', mods = 'CTRL', timeout_milliseconds = 1000 }

    config.keys = {
        -- ... other keybindings

        -- Toggle light and dark colorscheme
        {
            key = 'f',
            mods = 'LEADER',
            action = wezterm.action_callback(function(window, pane)
                colorscheme.toggle_colorscheme(window, pane)
            end)
        },
    }
end

With this setup, pressing Ctrl+Space (my leader key) followed by f will toggle between light and dark modes.

See the complete mappings.lua file on GitHub

This demonstration starts with an empty i3 workspace, I use super+Enter to start a new Wezterm instance which uses light mode. Then ctrl+space+f to switch to dark mode. Then I start a second Wezterm instance in a vertical split and toggle the theme for both instances a couple times more with ctrl+space+f

Implementing the Neovim side 🔗

The Neovim side is simpler but requires setting up a file watcher. I put everything in a colorscheme_watcher.lua file.

Setting up the file watcher 🔗

I adapted some code from fwatch.nvim to create a lightweight file watcher. The core functionality uses Neovim's built-in vim.loop (the Lua bindings for the libUV library that Nvim uses for networking, filesystem, and process management):

local uv = vim.loop

local function watch_with_function(path, on_event, on_error)
    local handle = uv.new_fs_event()

    local flags = {
        watch_entry = false,
        stat = false,
        recursive = false
    }

    local event_cb = function(err, filename, events)
        if err then
            on_error(error, unwatch_cb)
        else
            on_event(filename, events, unwatch_cb)
        end
    end

    uv.fs_event_start(handle, path, flags, event_cb)

    return handle
end

The vim.loop.new_fs_event() API provides access to the operating system's file watching capabilities (inotify on Linux, FSEvents on macOS, etc.). When the file changes, our callback function gets executed.

Reading and applying the colorscheme 🔗

I created functions to read the state file and update the colorscheme accordingly:

function read_colorscheme_file()
    local filepath = "/tmp/colorscheme"

    -- Read the first line of the file
    local content = vim.fn.readfile(filepath, '', 1)[1] or ""

    if string.match(content, "light") then
        return "light"
    end
    if string.match(content, "dark") then
        return "dark"
    end

    -- Default case
    return "dark"
end

function update_colorscheme()
    if read_colorscheme_file() == "light" then
        vim.g.colorsDefault = "dayfox"
        vim.g.colorsDiff = "dayfox"
    else
        vim.g.colorsDefault = "nightfox"
        vim.g.colorsDiff = "nordfox"
    end

    vim.cmd("colorscheme " .. vim.g.colorsDefault)
end

The vim.cmd() function executes a Vim command, so vim.cmd("colorscheme " .. vim.g.colorsDefault) is equivalent to running :colorscheme nightfox in normal mode, with the colorscheme name taken from our variable.

I use different colorschemes for diff mode, which is why I set both colorsDefault and colorsDiff variables.

Putting it all together 🔗

Finally, I set up the colorscheme on Neovim startup and register the file watcher:

-- Setup the colorscheme on startup
update_colorscheme()

-- Update the colorscheme when WezTerm updates the colorscheme file
do_watch('/tmp/colorscheme', {
    on_event = function()
        print('Colorscheme file has changed.')
        -- We wrap update_colorscheme in vim.schedule because the Vimscript function
        -- "readfile" must not be called in a fast event context
        vim.schedule(update_colorscheme)
    end
})

The vim.schedule() wrapper is important here. The file watcher callback runs in a "fast event" context where certain functions like readfile() are not allowed. By wrapping our update function in vim.schedule(), we defer its execution to a safe context.

See the complete colorscheme_watcher.lua file on GitHub which is then required in my init.lua file.

This demonstration starts with two WezTerm instances next to each other containing each two splits. In each instance there is a neovim instance and some command outputs (git status and man git). I use ctrl+space+f to toggle between light mode and dark mode. We can observe a slight delay before all applications colorscheme are updated.

How it all works together 🔗

Here's what happens when I press my toggle keybinding:

WezTerm's toggle function reads /tmp/colorscheme (let's say it contains "light")
It writes "dark" to the file
WezTerm reloads its configuration, which reads the file and sets color_scheme = 'nightfox'
All WezTerm windows across all instances update to the dark theme
Simultaneously, all running Neovim instances detect the file change via their watchers
Each Neovim instance executes its callback, which reads the new value and runs :colorscheme nightfox
All Neovim windows update to the dark theme

The whole process happens in a fraction of a second, giving the impression that everything updates simultaneously.

Reflections 🔗

I'm fairly happy with how this feature turned out. The implementation is simple and after a few days of use, it's working well. When I press the keybinding in any WezTerm instance, every terminal window and every Neovim instance updates immediately. No manual intervention needed, no focus juggling, no restarts. I should mention that I've only tested this on Ubuntu with i3, so your mileage may vary on other systems.

Using a simple text file for state management might seem primitive, but it's actually good enough for this use case. Both applications can easily read and write to it, the file system provides the notification mechanism through the watchers, and there's no complex IPC to set up or maintain.

Note that for my shell I use zsh+oh-my-zsh with fairly minimal colors configuration which makes that the shell remains readable with both colorschemes. I also have a couple color configurations happening in my gitconfig but again they work well with both colorschemes.

What about system-wide settings? 🔗

For now, I'm manually controlling my colorscheme with a keybinding, but WezTerm actually has built-in support for detecting the system appearance (light or dark mode). The documentation shows how to use wezterm.gui.get_appearance() to automatically switch based on your system settings.

I think my current setup would be fairly easy to adapt for this. Instead of using a keybinding to toggle the state file, I could run a background process that watches for system appearance changes and updates /tmp/colorscheme accordingly. That way, when I toggle dark mode in my system settings, both WezTerm and Neovim would follow along automatically. And that would also allow my browsers and other GUI applications to be synced with the terminal.

But for now, the manual toggle is working well for me, and I haven't dedicated time to integrate with system-wide settings yet.

Make it yours 🔗

If you want to try this approach in your own setup, all the code is available in my dotfiles repository. The main files you'll need are:

config/wezterm/colorscheme.lua: Manages the state file and toggle logic for WezTerm
config/wezterm/ui.lua: Integrates the colorscheme into WezTerm's configuration
config/wezterm/mappings.lua: Sets up the keybinding
config/nvim/lua/colorscheme_watcher.lua: Implements the file watcher and colorscheme updates for Neovim

You can easily adapt this to use different colorschemes, a different state file location, or even extend it to have an external scrip change the colorscheme file based on your location and the time of the day. The file-based approach is flexible enough to support a lot of different use cases. (Though it might not be wise to re-create a complete external configuration system outside the default config)

If you implement something similar or have ideas for improvements, I'd love to hear about it in the comments!

Looking for a new terminal emulator on Linux

statox — Sun, 05 Oct 2025 00:00:00 +0000

This article is a work in progress

Goal 🔗

Replace my current workflow of Gnome Terminal + tmux:

Get rid of tmux (Some graphic inconsistencies, one dependency in my dotfiles)
Keep my current keyboard centric workflow
Ideally choose a tool which works well with Wayland based WM.
Why not CPU or even GPU acceleration, NO electron based application
Easy to install and update
The terminal window can span the whole interface. No annoying "options" bar, scrollbar or fluff.

workflow to preserve 🔗

Settings

ctrl+space as a leader
"2 times bindings" where I press the leader, then press the mapping without having to maintain both pressed at the same time.

Splits

Create splits with leader+/ and leader+!
Move between splits with leader+hjkl
Move splits with leader+{}
Adjust splits size with leader+arrow keys
Close split with leader+x

Tabs

Create a tab with leader+c
Navigate tabs with leader+ctrl+hl
Move tabs around wiht leader+ctrl+left-right
A tab bar taking as few space as possible, ideally 1 terminal line

Copy mode

tmux select + copy mode easy to trigger and navigable with vim bindings
Enable copy mode with leader+escape

FZF integration

FZF used in my custom gd command should work well
FZF used in zsh ctrl-r ctrl-t should work well

Espanso should keep working

Kitty 🔗

Good

Easy installation with apt but version v0.32.2 outdated by 18 months. (Directly appears in my app launcher)
To recreate the ctrl+space + map mechanism we can't use kitty_mod in the config but the end result remains close
The tabs and windows navigations can be configured the same as my tmux bindings.

Neutral

I had to enable the visual bell, by default we get a sound.
Config file is not a standard format

Issues to solve:

The key repeat rate is too high, so I struggle to open some togglable menus, my backspace and enter keys are too sensitive.
The author wants to delegate the copy-paste mechanism to a pager so there is no great way to copy past the text of the terminal out of the box
The last run command is printed below the prompt before its results are displayed, don't know why
Almost all icons of :NvimWebDeviconsHiTest were displayed the first time I used the terminal, now there are not anymore I'm not sure why.

Issues impossible to solve:

The author seems pretty negative. I found a few Github issue where he seems to have strong opinion and be not very welcoming to suggestions
The copy past mode delegated to an external pager is very annoying. There is a Neovim plugin to use it as a pager, I'm not a fan of adding things to my editor config because my terminal can't handle selecting text.

Wezterm 🔗

Install via flatpak, dedicated apt repo or appimage (if installed via appimage need to create an entry in my app launcher)
Most of the icons of :NvimWebDeviconsHiTest are supported and gets a notification when some glyph can't be found in any font
Config file autoreload + config in lua
Managed to get the space screen space as tmux (Disabling "fancy tab bar" makes it even better than tmux)
The docs are really not clear
- ~I didn't find a simple way to test lua commands interactively with the cli or to use what I did in cli in the lua conf~ ctrl+shift+l opens the debug layout which gives a lua repl
- Using colorschemes isn't clear at first and seem to require running lua code
Colorschemes:
- Good my nvim themes *fox are available
- Dimming inactive tabs works out of the box

Issues to solve:

[x] Because of the autoreload of the config file and invalid config causes an error and I need a different terminal emulator to fix the config file. It keeps open the window with the previous config.
[x] It looks like I can't center the tab bar show it's not as good visually as tmux
- My workaround is to have the tab bar at the top the window. When it is at the bottom sometimes my hands on my laptop keyboard prevent me to see the bottom of my screen so I can't see the tabs (I should pay more attention to my posture too)
[x] At one point the whole UI became very slow, I'm not sure if its also affected other apps. I'm trying to disable nvim plugin render-markdown but I'm not sure it's really related.
- The issue comes from the interference with picom. I disabled the composer and it seems to solve the performance issue, but I don't have window transparency anymore. I might dig more but also I might also ditch picom completely.
- After a few more weeks I'm used not having picom so I'll consider that a problem solved. Last bit to solve is: I keep having to stop picom manually when I start my session eventhough I commented the line which starts picom in my i3 config file. I'll need to look deeper into it.
Apparently there are no layouts, so my tmux leader+n doesn't work out of the box. It seems to be possible to reimplement with custom lua functions
I have to setup LSP lua to work in config file to get a better discoverability of the available configs.
Espanso doesn't work out of the box for unicode characters: It works well with :espanso -> Hi there! but it fails for :+1: -> 👍
In nvim when two vertical splits are open, scrolling in one split messes up the rendering of the other split: The current buffer becomes unreadble because chunks of the screen are blank
I have trouble reproducing but regularly when a pane is open with nvim for too long (I guess?) it seems like the pane brokes, it looks like we just quit we nvim but we can't interact with the tab anymore. That might be a neovim plugin more than a wezterm one.
The styling of the terminal lacks a bit of configuration, for example I would like to be able to draw a border around my panes to indicate if they are zoomed in. That doesn't seem to be possible for now.
I need to improve the dark/light theme toggling feature I added in my config so that the toggle also works with nvim and ideally is bound to the system preference.
I have a major pet peeve on the bindings of the copy mode and how b goes back words but e goes to end of WORD, that's probably something I can solve with a remapping.

Update after about one month of usage 🔗

I made wezterm my daily driver and got rid of tmux and gnome terminal. This is overall a good experience.

No performance issue once I removed picom (a tradeoff I have made for more than a month now and I haven't missed it once so it's a sign that I just removed bloat, that's cool)
Configuration in lua is great
I got used to the tab bar it's clean
It integrated well with my i3, zsh, nvim setup

Python codegolf tips

statox — Mon, 20 Jan 2025 00:00:00 +0000

Resources 🔗

Useful code to copy/paste in codingame "fastest" challenges:

def log(*args): import sys; print(*args, file=sys.stderr)

Input parsing 🔗

To call create an input from calling the same function several times:

[input() for i in range(8)]

[i() for i in [input]*8]
[input() for _ in "1"*8]

Note that the list comprehension might be better in some cases. Example with The Descent where we need to read a list of integers and return the index of the highest value:

while m:=[input()for _ in "1"*8]:print(m.index(max(m)))
while 1:print(max([(input(),i)for i in range(8)])[1])

The list comprehension allows to create tuples with the value and the index. Iterating on "1"*8 would not give us the incrementing index.

Creating my homemade environmental sensors system

statox — Tue, 18 Jun 2024 00:00:00 +0000

This article is a work in progress

Supporting software architecture 🔗

The first component needed for this system is a logging system which allows me to

Receive HTTP calls from the sensors
Store and display applicative logs of these calls for debugging purpose
Store and display the metrics gathered by the sensors

To get these features I'm using my API and an Elasticsearch cluster that I setup earlier.

The API has an endpoint dedicated to reporting data from the sensors. The endpoint uses a JSON schema to validate the body of the POST requests it receives and only allows a specific list of fields to be sent. When a valid request is received, the data is sent via a JSON message to logstash with a specific message indicating that the log contains sensor data.

When logstash received this message, its pipeline detects the specific message and writes the log to a dedicated data stream meant to contain only environmental metrics. I can then use Kibana to create all the visualizations I need to this metric data.

I cut a few corners to get things working for now, but a few things could be reworked in this system:

I am currently implementing API key-based authentication to ensure only my sensors can call my API.
Having my sensors connected to the internet and calling my API directly is not an ideal security situation. Once my system is stable, a future iteration will involve having the sensors call a gateway on their local network, with the gateway responsible for sending the data to my API.
I probably could leverage the ELK stack's ability to handle metrics and avoid sending logs to logstash. This is a topic I need to dig.

Breadboard prototyping 🔗

I went through several iterations to get a working sensor. The goal was to create a device with the following features:

Read environmental data from one or several sensors. At a minimum, temperature and humidity, ideally also atmospheric pressure, and maybe as a bonus, luminosity, air quality, precipitation volume, ambient noise level, etc.
Send this data to a remote server via a Wi-Fi network.
Safely operate on battery power.

For the basis of my devices, I chose to use the Adafruit Feather HUZZAH ESP8266. I chose it for its ability to connect to a Wi-Fi network without hassle, the several input pins it offers, and its compatibility with the FeatherWing Proto, which allows creating a self-contained device without having to design and print a complete PCB.

VMA320 🔗

I created my first iterations on a breadboard, and the first sensor I tried was a VMA320, which I had laying around in my toolkit. The VMA320 returns the current temperature on its data pin with an analog output.

VMA320

temperature range: -55 °C to 125 °C
accuracy: ± 0.5°C
connection: analog output

The analog output of the VMA320 is proportional to the logic voltage we apply to it. Since the ESP8266 has a 3V logic, the output can go up to 3V, but the analog input pin of the ESP8266 can handle only up to 1V. So I needed to create a voltage divider to be able to read the output of the sensor.

This had two major drawbacks:

Because of the inaccuracies in my resistor values and the code I used to read from the voltage divider, I was not very confident in the data I got from the sensor.
The ESP8266 only has one analog input pin, which I knew I would need later on to monitor the battery.

With this first version I confirmed that I was able to

Upload code to the ESP8266
Interface with its pin
Debug my code when running on the SoC

The early prototype: Breadboard version of ESP8266 reading from a VMA320

The early prototype: Schematics version of ESP8266 reading from a VMA320

#include <math.h>

/*
 * Code for the VMA320 temperature sensor
 * https://www.velleman.eu/products/view/?country=be&lang=fr&id=435554
 */

#define SENSOR_PIN A0 // Input PIN: ADC (The only analogue pin on ESP8266)

// Based on https://stackoverflow.com/a/44932077/4194289
double analogToCelsius(int RawADC) {
    double Temp;
    Temp =log(10000.0/(1024.0/RawADC-1)); // for pull-up configuration
    Temp = 1 / (0.001129148 + (0.000234125 + (0.0000000876741 * Temp * Temp ))* Temp );
    Temp = Temp - 273.15;            // Convert Kelvin to Celcius
    return Temp;
}

double readTempCelsius() {
    int sensorvalue = analogRead(SENSOR_PIN);
    double celsius = analogToCelsius(sensorvalue);

    return celsius;
}

The code to read from the VMA320 is awkward

DHT11 and battery level 🔗

My second iteration replaced the VMA320 with a DHT11 I also had from an earlier attempt at this project. This allowed me to free the ESP8266 analog pin and to benefit from the more accurate sensor.

DHT11

temperature range: 0 °C to 50 °C
accuracy: ± 2°C

humidity range: 20% to 80%
accuracy: ± 5%

connection: digital output
sampling rate: 0.5Hz to 1Hz (Depending on the sources)

Early prototype: Breadboard version of ESP8266 reading from a DHT11

To read the sensor's data I used adafruit's DHT library which allowed me to greatly simplify my code:

#define DHTPIN 13      // Pin used to read the DHT sensor
#define DHTTYPE DHT11  // Can be changed for DHT22 sensors

// Initialize DHT sensor for normal 16mhz Arduino
DHT dht(DHTPIN, DHTTYPE);

void initDHT() {
    dht.begin();
}

float* readDHT() {
    // Reading temperature or humidity takes about 250 milliseconds!
    // Sensor readings may also be up to 2 seconds 'old' (its a very slow sensor)
    float h = dht.readHumidity();
    // Read temperature as Celsius
    float t = dht.readTemperature();

    float* result = new float[2];
    result[0] = t;
    result[1] = h;

    // Check if any reads failed
    if (isnan(h) || isnan(t)) {
        Serial.println("Failed to read from DHT sensor!");
    }

    return result;
}

With this version I also started using a Li-Po battery to power the system. This is the part that I was most worried about because Li-Po batteries tend to be explode pretty impressively when mishandled and I'm not keen on seeing that happening in my house. Using this type of battery requires a basic understanding of how they work to avoid pushing them into unsafe zones.

The Li-Po voltage is around 4.2V when fully charged. It then decreases to around 3.7V and remains stable at this voltage for a relatively long time. Subsequently, the voltage drops quickly. It is recommended to stop using the battery below 3.3V, and the battery's protection circuitry is designed to shut it down completely at around 2.5V.

To monitor the battery level, I created a voltage divider to scale the 4.2V voltage down to 1V, which I could then read from the ESP8266 analog pin. As shown in the schematic, I could also use my voltmeter to directly measure the battery's output voltage.

Voltage divider schematic to read battery level

After adding some code to convert the analog reading into a percentage of charge and send it to the server, I successfully created my first visualization with a graph showing the temperature, humidity, and battery level.

The first Kibana visualization

At this point the main loop of the arduino code was as follow:

Read from the sensor
Send the readings to the server
Use delay() to pause the program for 10 minutes
Repeat.

I decided to let the system run for several hours to observe its performance, and it seemed to work well. I got confused by my battery readings and ended up asking for help on the adafruit forum:

The battery continued operating at voltages much lower than I had expected.
- I learned that the shutdown voltage of the battery circuitry is not standardized and varies by manufacturer.
Despite the low battery voltage, the ESP8266 continued running.
- I learned that the board is more efficient than I had realized, and its ability to operate at lower voltages is a feature.

Even though my calculations for the battery percentage were slightly inaccurate, the discharge curve of my battery matched exactly what I had expected based on my research into Li-Po batteries. Therefore, both my voltage divider and raw readings were correct!

Battery tension dropping

From the breadboard to the protoboard 🔗

With my initial tests successful, I decided to transition my sensor setup from the breadboard to a more permanent solution that would be easier to move around the house and manipulate. To achieve this, I mounted the Feather on a mini breadboard and utilized the FeatherWing to establish connections to the sensor and implement the voltage divider for the battery.

Moving the project to a protoboard

DHT22 🔗

At this point, I replaced the DHT11 sensor with a DHT22 module. The DHT22 is essentially an upgraded version of the DHT11, and the module includes a convenient pull-up resistor.

DHT22

temperature range: -40 °C to 80 °C
accuracy: ± 0.5°C

humidity range: 0% to 100%
accuracy: ± 2-5%

connection: digital output
sampling rate: 0.5Hz

Adafruit has an interesting comparison page. Switching to the DHT22 was quite convenient: the DHT library functions the same way for both sensors, so it was simply a matter of connecting it to the same pins and changing the DHTTYPE constant in my code.

I also created a second device identical to the first one and let them run for several days to compare their readings. The following graph shows these readings:

Between the 22nd and the 26th, the sensors were placed at two opposite corners of the same room.
Between the 26th and the 30th, they were placed right next to each other.
After the 30th, they were placed in two different rooms in a fairly small apartment.

Similar curves for two different sensors

I am pleased with the results. Although my methodology is not the most scientific, I am fairly confident to say that:

Since both curves generally follow the same pattern, both sensors have similar reaction times and sensitivity.
The minimal differences observed when the sensors are next to each other suggest they are calibrated similarly.

With these results, I'm not certain about the absolute accuracy of the readings (e.g., whether the sensors accurately reflect the current temperature: when they indicate 24°C is it really 24°C or 22°C or 26°C?), but they should be meaningful and sufficient for my goal of understanding how temperature evolves in my apartment.

BME280 🔗

Another metric I wanted to track in my system is atmospheric pressure. For this, I used a BME280 sensor, which I connected to one of my devices that already had a DHT22. This allowed me to have both sensors in the same location and use a single ESP8266 to read from them.

I opted to use the BME280 in I2C connection mode, requiring only 4 pins to connect to the board: 2 for power supply and 2 for data reading. The I2C connection also means that the sampling rate is essentially as fast as I want now (not that it matters since I am still polling the sensors every 10 minutes).

BME280

temperature range: -40 °C to 85 °C
accuracy: ± 0.5°C

humidity range: 0% to 100%
accuracy: ± 3%

pressure range: 300hPa to 1100 hPa
accuracy: ± 1hPa

connection: I2C (also support SPI)
sampling rate: -

Protoboard with BME280

I then compared the readings of the BME280 with those of the two DHT22 sensors to check for calibration issues. The raw values show the BME280 as the pink curve, while the blue and green curves represent the two DHT22 sensors.

The BME280 is not calibrated the same way as the DHT22 (left: temperatures, right: humidity)

For the temperature, the BME280 closely follows the trend of the other two sensors but with an offset of about 0.6°C. Regarding humidity, the situation is more complex: there appears to be both an offset and a scaling issue. After adjusting the humidity curve using the formula 20 + (0.7 * last_value(document.humidity2, kql='"document.humidity2": *')), I obtained the following results:

However the readings can be transformed to better fit the DHT22 readings (left: temperatures, right: humidity)

Good enough for me! Again, I'm unsure about the absolute calibration of the sensors, but they are consistent with each other, so my measurements should be reliable.

I also really liked this article by Robert Smith about his process for calibrating thermometers. At some point, I intend to try a similar ice bath experiment.

Adding an enclosure 🔗

Now that I'm confident my sensors work well, I want to enclose them properly. I have two different needs: enclosures for the indoor sensors to tidy up cables and protect them from daily life, and enclosures for the outdoor sensors that need to be weatherproof.

Indoor enclosure 🔗

For my indoor enclosure I am considering two options:

Mastering Mecabricks to design a Lego enclosure and then using the Lego Pick a Brick to acquire the necessary bricks. However, this option is on hold until I improve my skills with the online tool.

Using a colleague's 3D printer to print a suitable enclosure. I have started to gather quotes for the project, but this is also on hold for now.

Behold my incredible technical drawing skills!

Outdoor enclosure 🔗

This enclosure is more complex because it needs to protect the board and the battery from significant humidity changes while allowing the sensor to be exposed to the environment for accurate readings. I have no experience with this type of project, but I intuition tells me that exposing electronic components and sensitive batteries outdoors requires some caution.

Therefore, I began by researching and educating myself:

These papers are super interesting but I ended up winging it: I bought a small plastic enclosure similar to this one with a toric joint for waterproofing and a weatherproof cable gland. The idea is to have the sensor outside of the box while the ESP8266 and its battery remain safe in a waterprood box.

To test this enclosure I used my two DHT22 sensors: One in the box and one not in the box:

Metrics in the weather-proof enclosure

I found that the box doesn't prevent temperature changes (as expected), but it does maintain the humidity level at what it was when sealed. This should be sufficient for my upcoming initial tests of the outdoor sensor.

Capteurs environnementaux maison

statox — Mon, 17 Jun 2024 00:00:00 +0000

This post is a copy-past of a personnal message sent to a relative to talk about my project. A more complete and technical version is available here.

L'idée est de mettre en place des capteurs qui me remontent régulièrement la température et l'humidité des différentes pièces de l'appartement et de l'extérieur de l'appartement, de corréler ces données aux données de stations météo France autour de chez nous et de visualiser l'évolution de ces données au cours du temps. Le but est d'une part de mieux comprendre comment garder l'appartement le plus frais possible pendant les canicules, mais surtout de m'amuser avec de l'électronique et en apprendre plus sur la météorologie!

L'expérience débute avec la réalisation de capteurs environnementaux "maison". D'abord la version la plus simple possible qui sera équipée de l'équivalent d'un arduino avec une puce wifi intégrée connecté à un capteur de température et d'humidité. Un œil avisé remarquera trois résistances branchées entre le positif et la masse avec un fil qui part de l'une d'entre elle pour aller jusqu'au circuit, le lecteur électronicien identifiera peut être un pont diviseur de tension: le système est alimenté par une petite batterie LiPo (à gauche de la photo), ces batteries ayant une fâcheuse tendance à exploser quand leur charge atteint un niveau trop bas il est nécessaire de surveiller le niveau de charge. Mais les entrées analogiques du circuit ne supportant une tension maximale que de 1V et la batterie délivrant entre 4.2V et 3.3V dans sa zone de sécurité, on a besoin d'un pont diviseur pour ramener la tension de la batterie à une tension lisible par le circuit.

The early prototype: Breadboard version of ESP8266 reading from a DHT11

Une fois ce premier montage fonctionnant on se félicite de ne pas avoir fait exploser la batterie et on essaie d'exploiter la donnée du capteur: Pour ce faire le circuit va relever la température et l'humidité toutes les 10 minutes et utiliser sa puce wifi pour appeler un serveur. C'est un serveur sur lequel le créateur travaille depuis plusieurs mois et dont la dernière fonctionnalité implémentée par ses soins permet de recevoir un appel HTTP contenant des informations et de les enregistrer dans une base de donnée spécialement conçue pour contenir des données "temporelles" (qui évoluent avec le temps).

On passera au lecteur les détails techniques et on retiendra que l'on commence a obtenir des graphiques de ce genre représentant l'évolution de la température et de l'humidité et de la charge de la batterie au cours du temps. Le lecteur attentif remarquera que les données ne semblent pas significatives pour l'instant: Le capteur n'a pas fonctionné en continu pendant l'expérience, les manipulations ont faussé les valeurs de température et la charge de la batterie est notée au delà de 100%. Le créateur y verra néanmoins un début prometteur!

The first Kibana visualization

Les étapes suivantes consisteront en deux avancées majeures: On passe du format "planche d'expérimentation" (breadboard) au format "planche de prototypage" (protoboard) qui s'enfiche directement sur les connections du circuit pour réduire l'encombrement du capteur. On remplacera également le capteur DHT11 par un capteur DHT22, qui a l'avantage d'être plus stable dans ses relevés, d'avoir une précision de température de ±0.5°C contre ±2°C pour l'ancien capteur DHT11, un plus large champ de relevés d'humidité (0-100% contre 20-80%) contre l'inconvénient de ne pouvoir donner des résultats significatifs que toutes les 2 secondes (contre une fréquence d’échantillonnage minimale de toutes les 1 secondes pour le DHT11) ce qui n'est pas un problème en soit vu que le système est conçu pour faire des relevés toutes les 10 minutes.

Moving the project to a protoboard

On aura réalisé 2 capteurs similaires pour commencer a comparer la précision des capteurs industriels achetés et on se félicitera de voir que sur la période du 25 au 30 mai où les capteurs sont resté placés l'un à coté de l'autre les valeurs sont très similaires!

Similar curves for two different sensors

À ce moment on commence à se poser la question du conditionnement des capteurs: Comment créer des boites qui permettent à la fois de protéger les capteurs des aléas d'un environnement ménager tout en leur permettant d'assurer au mieux leur fonction de relevés? Plusieurs pistes seront envisagées d'une part un collègue propose de mettre à disposition son imprimante 3d afin de créer des boites parfaitement adaptées au système, d'autre part le créateur envisage la conception de boites faites de pièces de Lego. Pour ce faire l'auteur fera ressortir ses anciens cours de dessin industriel et tentera de relever au mieux les cotes du projet. Néanmoins la partie de création de boites est pour l'instant à l’arrêt du fait d'un manque de temps pour relancer l'imprimante 3d et d'un créateur qui a remis à plus tard l'idée de maîtriser l'outil de conception d'assemblage de Lego en 3d pour concevoir la boite uniquement depuis un ordinateur et sans pièces Lego à disposition.

Handmade sensor quotes

À défaut de boite artisanales adaptées à l'environnement intérieur le créateur se lance ensuite dans la réalisation d'une boite "weather-proof" qui pourra accueillir le capteur en milieu extérieur, on s'appuiera alors sur un boîtier issu du commerce et équipé d'un joint torique dans lequel on réalisera une ouverture afin de faire passer un passe cable étanche. Le but étant d'assurer au circuit et à sa batterie un environnement à l'humidité stable tout en permettant au capteur d'être placé en extérieur afin de mesure au mieux son environnement:

Sensor in a weather-proof enclosure

Ce système sera testé via la méthode dite "de la bonne grosse douche bien chaude": On placera le capteur dans son boîtier étanche ("dev-sensor-2" courbe bleue) dans une salle de bain pendant la douche du créateur pendant qu'un second capteur non protégé ("dev-sensor" courbe verte) sera placé derrière la porte de la-dite salle de bain . Le lecteur pourra observer que si le boîtier étanche ne protège pas le circuit des changements de températeurs (courbe de gauche) l'humidité à l'intérieur du boîtier, elle, reste constante quand l'humidité à l'extérieur de la salle de bain augmente significativement: Réussite notre boiter étanche est bien étanche!

Metrics in the weather-proof enclosure

La suite consiste désormais à intégrer un nouveau type de capteur afin qu'une fois en extérieur les relevés puissent intégrer la pression atmosphérique, composante cruciale de la prévision météo. L'auteur profitera de l'occasion pour se documenter sur les systèmes météorologiques d'anticyclone, de dépression, de fonts chauds, froids et composites, des différents facteurs ayant une influence sur la pression atmosphérique, ainsi que sur les recommendations d'installations de stations météo amateur.

(En haut à gauche le nouveau capteur BME280 combinant pression, température et humidité -meme si 6 connections sont accessibles seulement 4 sont vraiment utilisées dans notre système-. L'ancien capteur DHT22 reste également connecté pour comparer les relevés)

Protoboard with BME280

En plus de ce nouveau capteur l'auteur aura pris soin d'incorporer à ses tableaux de mesures les informations fournies par les stations MétéoFrance installées dans Paris. En effet MétéoFrance met à la disposition du public un service permettant d'interroger ses bases de données en temps réel et de recevoir les mises à jour horaires de ses différentes stations. Toutes les stations ne renvoient pas les mêmes informations par exemple seulement Longchamp et Montsouris fournissent des relevés d'humidité:

Meteo-France temperature and humidity

Sur le graph de gauche le lecteur avisé s'interrogera sûrement sur la station renvoyant une température constamment inférieure a toutes les autres: cette station est en fait située au sommet les 330 mètres de la Tour Eiffel, ce qui explique une température en moyenne 3 degrés inférieurs à l'ensemble des autres stations qui se situent en moyenne a 1m20 du sol.

On récupère également les niveaux de précipitations de certaines stations (On notera que ces données semblent toutes arrondies à l'entier le plus proche, une interrogation en cours pour l'auteur, plus de détails dans un prochain épisode)

Meteo-France precipitations

Enfin on récupère la pression atmosphérique renvoyée uniquement par la stations de Montsouris (en jaune) et en la compare à la pression relevée par notre capteur maison. Le lecteur pourra s’enthousiasmer du niveau de précision du capteur qui suit de très près les variations de la station:

Atmospheric pressure

Mais si les variations sont suivies de près on notera que les valeurs remontées ont un écart constant de 4hPa ce qui nous amène à la question cruciale du moment: Comment les capteurs sont-ils calibrés et comment mesurer compenser leur défaut s'il existe?

Prenons par exemple les mesures d'humidité:

Humidity with error

Nous avons en tout 3 capteurs: 2 DHT22 (courbe verte et courbe bleue) et 1 BME280 (courbe rose). À première vue les allures sont similaires mais on remarque que le BME280 à l'air d'avoir une erreur d'environ 4.8%, on peut donc tracer la courbe 4.8% plus haut mais on se rend compte alors que le capteur n'a peut être pas qu'un défaut de décalage mais aussi un défaut d'échelle: Pourquoi les pics roses sont-ils plus hauts que les pics verts et bleus?

Humidity correction hypothesis

Les resources en ligne décrivent des protocoles expérimentaux pour tenter de quantifier ce genres de décalages. Une méthode possible consiste à remplir une bouteille Thermos de glace pilée réalisée a partir d'eau déminéralisée pour atteindre une température exacte de zéro degrés et à partir de cette température connue observer les relevés des capteurs et en déduire le biais. Ce genre d'expériences pourraient bien occuper les prochains weekends du créateur, qui ne manquera pas de tenir son lecteur informé des évolutions du projet!

Using an OpenSearch (AWS's ElasticSearch) cluster to monitor my API

statox — Sat, 27 Apr 2024 00:00:00 +0000

Notes about how I set up an OpenSearch cluster to manage the logs of my api.

Goal: Set up the cheapest ELK cluster possible to - Ingest the logs of my API - Maybe use as a tracking database for my habits

TODO:

Review costs after some usage. The idea was to benefit from the free tier, after one week it seems I don't get it.
Review UltraWarm nodes they might be usefull for my usage if I can't benefit from the free tier.

Setup the cluster in AWS 🔗

Everything done easily from the AWS console. I had to use "Standard create" to be able to setup a smaller cluster than what AWS recommends because I have very small needs.

Not using the default Easy mode

Specifying a single availability zone to limit the costs.

Using a single node both for master and data with the smallest possible EBS and minimum IOPS.

I also set up a custom endpoint so that I can reach my cluster from logs.statox.fr instead of the default long endpoint provided by ELK. The setup is simple: Create a certicate in ACM, setup a DNS verification to validate the certificate and use the certificate in the setup screen of OpenSearch.

The security is far from ideal but pretty convenient for a very-not-critical API like mine: The cluster is open to internet and the auth is done with ELK's built-in user store. (Note that the automatic software updates are enabled thank to the managed aspect of the cluster, I'll need to make sure it doesn't break stuff when I don't do anything).

Using a single node both for master and data with the smallest possible EBS and minimum IOPS.

TODO: Check if I can better restrict this policy.

By default the cluster has hourly snapshots. The cluster takes several minutes to start up, once it is up we can test that it answers properly:

DOMAIN_ENDPOINT='https://logs.statox.fr'
ELASTIC_USER='xxxxxxxx'
ELASTIC_PASSWORD='xxxxxxxx'


curl  -H 'Content-Type: application/json' \
      -XGET --user "$ELASTIC_USER:$ELASTIC_PASSWORD" \
      "$DOMAIN_ENDPOINT/_cat/health"

About ingestion 🔗

AWS har a service OpenSearch Ingestion which provides ingestion pipeline like Logstash would do in a regular ELK setup.

For now I don't use it because I want to avoid the additional costs but this has some big downsides, particularly I need to provide a timestemp fields when my clients send new logs, that sucks and is very error prone. Also the clients are responsible for sending the logs to the correct data stream, for now this is ok because I use the cluster only for my logging but when I start using it for other features I'll probably need to get to the ingestion part.

The UI 🔗

AWS replaces Kibana with OpenSearch Dashboards which is a rebranded version of Kibana. It is accessible to https://logs.statox.fr/_dashboard.

The main difference with a self hosted Kibana is the "Management" section of the menu which changes, for example there is no "Stack Monitoring" section because the monitoring is done directly in AWS.

Configuring the database 🔗

Goal: Ingest my logs on a rolling index (weekly rollover)

I am going to use data steams instead of regular index to make it easier to configure the roll over.

1. Create the index template. 🔗

This is the template of settings which will be applied to all the indices which will be used to create the data stream.

Important settings:

Specify that we are working with data streams
Specify the timestamp field as the time field (which will allow the proper indexing and temporal search).
We are not really using index pattern as with regular indices because we are using a data stream, so the index pattern property must have the name of the stream we will create later on. Note: No * character

No need to specify an index alias because the data stream will take care of that. And I have only one node so it doesn't make sense to have more than one primary shard or any replica shard.

The index mapping are useful to specify that a specific field is not a simple number but a timestamp or not a simple string but an IP. See the supported fields types.

2. Create the data stream 🔗

If created with the name used in the index template's index pattern field the stream will be associated to the template.

Once the stream is created we can start pushing data to it, that will automatically create the underlying index:

curl  -H 'Content-Type: application/json' -XPOST --user "$ELASTIC_USER:$ELASTIC_PASSWORD" "$DOMAIN_ENDPOINT/api.statox.fr/_doc" \
    -d '{"message": "Test message from bash", "timestamp": $(date +"%s")000}' # With a shitty hack to transform seconds to milliseconds

3. Create the Index pattern 🔗

This is the entity which tells to Dashboards (Kibana) which indices to search when you are in the Dicover tab.

The only thing needed when creating the pattern is to use the proper name api.statox.fr (Again no need for * thanks to the data stream which masks the complexity)

From there the logs are available in the Discover tab:

4. Set up the index policy 🔗

This is the final step to make sure the indices backing the data streams are regularly rotated (so that at one point they can be deleted to keep the volume of data below a certain threshold without dropping a unique index with all the logs).

An optional step 1 is to setup a Notification channel to be notified when the policy fails, settings up a notification on a slack webhook is super straight forward, just past the webhook URL and you're done.

By using the correct index pattern, the policy will be applied to all the indices of the data stream and will rollover them.

Conclusion 🔗

This kind of setup is something I'm used to do on a much bigger scale in my day job so I see the cracks in this current setup but at least it's working conveniently for my small needs.

It would be great to terraform the AWS part (but, la flemme)
I need to check if I can easily restore the automatic snapshots made by AWS and if that keeps the configurations I did in Kibana (in theory, yes)
Next I'll use this document store as part of the product of my api.

Automatically merging Dependabot PRs on Github

statox — Sat, 20 Apr 2024 00:00:00 +0000

I have Dependabot set on many of my repos but I often get too lazy to check the PRs and merge them. On my api repo this is an issue because I really want to keep dependencies up to date. Here is what I did to have Dependabot's PRs merged automatically as they are created.

The file lives there on Github

At no point there is a need to generate a github access token by yourself. Dependabot will use its own

Setup tests 🔗

For my api repo I use mocha to run the tests. I have 3 commands to be able to run the tests

npm run env            # This triggers a podman-compose command starting the containers
./src/tools/init-db.sh # Create the test database in the container
npm run tests:all      # Call mocha and run the tests

Setups in Github 🔗

Dependabot 🔗

Allow Dependabot to regularly create PRs with dependencies update.

In the Github repo settings: Security > Code security and analysis > Dependabot

Enable the Dependabot alerts

For better configuration a file github/dependabot.yml can be created in the repo, see the doc for more details.

Actions settings 🔗

Allow GitHub Actions to create and approve pull requests. This is needed because the workflow we will trigger will be responsible for approving the PR automatically.

In the Github repo settings: Code and automation > Actions > General > Workflow permissions

Enable the actions to modify PRs

Branch protection 🔗

In the Github repo settings: Code and automation > Branches > Add rule

The rule needs to apply to the main branch (i.e. the one we'll be merging to).

We need two checks enforced by the rule:

Require a pull request before merging and Require approvals: The approval will be given by github-action bot throught the workflow
Require status checks to pass before merging: The checks will be the success of the workflow itself and we'll make the workflow fail if the tests for the repo are not validated.

Enforce PR approval and checks

Setup the workflow 🔗

In the repo create a file for the workflow like .github/workflows/dependabot-auto-merge.yml

Trigger 🔗

We will trigger the workflow on PRs.

TODO Find a way to trigger only for dependabot PRs, for now all MR will be automatically merged if they pass the tests.

types: Using edited is useful to debug the workflow while setting it up: once the MR is open, you can edit the workflow, push, comment @dependabot rebase on the PR and the workflow will be re-run

name: Test and AutoMerge PRs

on:
    pull_request:
        types: [opened, synchronize, edited]
        branches: [main]

Permissions 🔗

This changes the permissions of the github token that dependabot gets when creating the PR, we need two additional permissions.

permissions:
    # This is needed to approve the PR
    pull-requests: write
    # This is needed to merge the PR https://github.com/cli/cli/issues/6695#issuecomment-1348430969
    contents: write

Without these permissions the calls the the gh cli in the next steps fail with errors similar to this:

Example of gh permission error

Filter dependabot 🔗

One way to have the whole workflow triggered only for dependabot's PR is to add a condition for the job:

jobs:
    test-and-auto-merge:
        if: github.actor == 'dependabot[bot]'
        runs-on: ubuntu-latest
        steps:
            # [...]

This could be improved to better factorize the code and have the tests running on all MR and the auto merge running only on dependabot PRs but since I'm the only one working on this repo and I don't use PRs for other reasons I will not bother with that.

Repo setup and tests 🔗

This is dependent on all repos though the important part is to run some tests. Here we need several setup steps:

Install python throught a marketplace Github action to be able to install podman-compose with pip
Install podman throught a marketplace Github action.
Install node to run build the project and run the tests.
Checkout the code, install the deps and run the tests.

If these steps fail the following one will fail too, so the PRs tests will fail and the code will not be merged.

jobs:
 test-and-auto-merge:
    runs-on: ubuntu-latest
    steps:
      - name: Install python 3
        uses: actions/setup-python@v5
        with:
          python-version: '3.x'

      - name: Install podman
        uses: gacts/install-podman@v1

      - name: Install podman-compose
        run: pip3 install podman-compose

      - name: Install node.js
        uses: actions/setup-node@v4
        with:
          node-version: 'latest'

      - name: Checkout code
        uses: actions/checkout@v2
        with:
          ref: ${{ github.head_ref }}

      - name: Install dependencies
        run: npm ci

      - name: Start podman environment
        run: npm run env

      - name: Init db
        run: ./src/tools/init-db.sh

      - name: Run tests
        run: npm run tests:all

Handle the PR 🔗

Three important steps:

Call the fetch-metadata action to get the $PR_URL variable referring to the current PR and used in the next steps.
Have the github-action bot approve the PR
Have the github-action bot set the auto merge setting for the PR, which will trigger the merge of the PR because all the checks will succeed.

jobs:
 test-and-auto-merge:
    runs-on: ubuntu-latest
    steps:
      # [...]
      # Setup and test steps
      # [...]

      - name: Dependabot metadata
        id: metadata
        uses: dependabot/fetch-metadata@v2
        with:
          github-token: "${{secrets.GITHUB_TOKEN}}"

      - name: Approve the PR
        run: gh pr review --approve "$PR_URL"
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          GH_TOKEN: ${{secrets.GITHUB_TOKEN}}

      - name: Auto-merge the PR
        run: gh pr merge --rebase --auto "$PR_URL"
        env:
          PR_URL: ${{github.event.pull_request.html_url}}
          GH_TOKEN: ${{secrets.GITHUB_TOKEN}}

Result 🔗

Once all of that is configured the Dependabot PRs should be automatically handled.

The check tab of the PR should have some results

The checks should point to a succesful action

The various bots should have acted on the PR

Notification 🔗

When all of that is succesful I get emails from Github both for when the bot approves the PR and for when it merges it.

Javascript codegolf tips

statox — Thu, 07 Sep 2023 00:00:00 +0000

Resources 🔗

A list of the various codingame problems with their stats: chadok.info

Type conversions 🔗

Decimal string to Number 🔗

When an input is expected to be a number examples usually use the following code:

n = parseInt(readline());

This is 10 characters just to parse a number from a string. Using the unary operator + on a string will cast it to a number. Example:

let s = '12';
let n = +s;

typeof n; // 'number'

let t = +s[0] + s[1]; // t=3

Note that if you need to add a variable which is a number to a string representing a number you will need to use the string first or add a whitespace before the + operator:

let s = '1';
let n = 1;

n + s; // string: '11'
n + +s; // number: 2

+s + n; // number: 2

Binary string to Number 🔗

parseInt('101', 2); // 5
Number('0b101'); // 5
+'0b101'; // 5

Number to binary string 🔗

See this SO answer

let a = 6;
a.toString(2); // '110'

// For negative numbers
let a = -6(a >>> 0).toString(2);

ASCII 🔗

'a'.charCodeAt(0); // 97
'A'.charCodeAt(0); // 65

Position in the alphabet 🔗

'a'.charCodeAt(0) - 97; // 0
'b'.charCodeAt(0) - 97; // 1
'c'.charCodeAt(0) - 97; // 2

'A'.charCodeAt(0) - 65; // 0
'B'.charCodeAt(0) - 65; // 1
'C'.charCodeAt(0) - 65; // 2

If not considering the case

parseInt(c, 36) - 10;

// Example
[...'abcdefghijklmnopqrstuvwxyz'].map((c) => parseInt(c, 36) - 10); // [0, 1, 2, 3..., 25];

Loops 🔗

For < while 🔗

for loops are most of the time the most byte efficient way to write loops.

while(1)
for(;;)

Declarations in `for` loop first statement 🔗

In javascript you can combine some statements with , and this can be used to put a lot of things. Say you need to read two values and then iterate between these two:

// 44 bytes
a=readline()
b=readline()
for(i=a;i<b;i++)

// 37 bytes
for(b=realine(a=readline();a<b;a++)

Here is an example of a code I made for a challenge using this kind of declarations:

for (r = readline, n = +r((m = +r((t = 0)) + 1)); m < n; m++) {
    for (j = T = 0, S = [...(m + '')], L = S.length; j < L; j++) T += S[j] ** L;
    t += T == m;
}
print(t);

`for (... of ...)` 🔗

When you need to iterate on a list the for (iterator of list) syntax is much shorter than for (i=0; i<list.length; i++), but also most of the time is beats functions like .forEach or .map because you don't need the arrow function and the iterator is available:

l.map(Number); // Good
l.map((c) => +c); // Better
for (c of l) t += +c; // Top

Variables declaration and initialization 🔗

Get rid of explicit declaration statements 🔗

Code golf solutions are often very short and not complex enough to require switching between different scopes. So most of the time you can loose the let, const and others var from your code.

You should always have in mind the implications of not implicit scopes but that saves you a lot of characters.

// 58 bytes
let n = 10;
for (let i = 0; i < n; i++) {
    console.log(i);
}

// 37 bytes
n = 10;
for (i = 0; i < n; i++) console.log(i);

Chained declarations 🔗

You've probably tried to do this in a project and had eslint yelling at you for doing that, but in codegolf that saves bytes:

a = b = 1;
a; // 1
b; // 1

Math 🔗

Replacing Math built-ins with bitwise operations 🔗

In math challenges using bitwise operations are often very powerful. This is not something every developer is used to but it's worth knowing them

Math.floor(n);
0 | n;
n | 0;

Math.round(n);
(n + 0.5) | 0;

Math.ceil(n);
n % 1 ? -~n : n;

Math.abs(n);
n < 0 ? -n : n;

Math.min(x, y);
x < y ? x : y;

Math.max(x, y);
y < x ? x : y;

Dividing by a power of 2 with bitwise operators 🔗

Shifting right divides and rounds the result by the power of two

(i / 2) | (0 === i >> 1);
(i / 4) | (0 === i >> 2);

Get rid of leading zeros 🔗

0.5 === 0.5; // true

Conditions 🔗

The ternary operator 🔗

The ternary operator is precious to avoid lengthy conditional expressions.

Always be mindful of what you use in your conditional expressions, sometimes putting them in another way can save a byte or two while preserving the same feature:

r = a < 10 ? a + '0' : a;
r = a > 9 ? a : a + '0';

r = a == b ? 'same' : a > b ? 'higher' : 'lower';
r = a > b ? 'higher' : a < b ? 'lower' : 'same';

Using `&&` and `||` 🔗

TODO Find some good examples for this one
Sometimes you need to execute something only if you match a condition and if statements are not always the shortest solutions. Instead you can use && to execute an instruction only when the first one returns true:

if(a%2==1)t+=a
(a%2==1)&&t+=a

Consider a problem where you need to find the nth digit in a string of all palindromic numbers concatenated.
You will need to generate the string 123456789112233445566778899101.... One way to do it is the following:

for (n = +readline((s = '')), i = 0; !s[n]; i++)
    if ([...(i + '')].reverse().join`` == i) {
        s += i;
    }
print(s[n]);

While the nth digit of the string doesn't exists continue to search for number which are palindromes.

for (n = +readline((s = '')), i = 0; !s[n]; i++) [...(i + '')].reverse().join`` == i && (s += i);
print(s[n]);

Codingame particularities 🔗

These tips are only useful on the codingame website.

Shorten `readline` calls 🔗

Codingame provides a readline() function to get the inputs of a problem. This function will read one line of standard input and return it as a string without the new line character.

If you need to use readline() more than one it will be cheaper to store it in a variable first:

// 27 bytes
n = readline();
m = readline();

// 24 bytes
r = readline;
n = r();
m = r();

This is only 3 characters saved but the more calls to readline() you need to write the more characters you save.

Execute code in the `readline` arguments 🔗

You can use javascript scope smartly to reduce the number of bytes needed to get the inputs. To do so keep in mind that if you pass a variable assignation as the argument of a function the variable is still available in your current scope.

For example let's say you need to read two strings n and m and to init a counter i to zero:

// 28 bytes
r = readline;
n = r();
m = r();
i = 0;

This can be shortened like this:

// 26 bytes
r = readline;
m = r((n = r((i = 0))));

⚠ The order of calls made to r() is important here: The first call made to the function should be the most nested.

Use tagged templates to shorten some function calls 🔗

See the MDN doc for tagged templates

Basically if you pass to a function a template literal with some placeholders like this:

myTagFn`str1 ${a} str2 ${b} str3`;

The function will get as parameters 1) the resulting array of splitting the template literal on each placeholders and 2) the different placeholders. E.g.:

const myTagFn = (strings, ...args) => {
    console.log(strings); // [ 'str1 ', ' str2 ', ' str3' ]
    console.log(args);    // [ 1, 2 ]
}

const a = 1
const b = 2
myTagFn`str1 ${a} str2 ${b} str3`

It turns out that many methods of the standard library work as tag functions which allows to remove the parenthesis around their arguments. For example:

'ab cd ef'.split(' ');
'ab cd ef'.split` `;

[1, 2, 3].join('\r');
[1, 2, 3].join`\r`;

Replace `console.log` by `print` 🔗

To pass your result to codingame validators you need to write it to the standard output. In regular javascript this is done with console.log however codingame's environment supports the deprecated print which works exactly the same:

console.log('valid');
print('valid');

Stop your program with an invalid command 🔗

In some cases where you need to stop your program immediately it is possible to use an invalid instruction rather than trying to use e.g. process.exit().

This work because codingame validate the standard output of your program but not that it ran without errors. This is a bogus example just to show my point: Imagine you need to iterate through a list looking for the first value matching a condition, knowing that several values will match it. You'll want to avoid printing the next values so you need to stop the program altogether. You could use process.exit() but using Z (assuming that you didn't define the variable) is much shorter:

if (/*condition*/)print(result),process.exit()
if (/*condition*/)print(result),Z

Programmatically update Zoom's virtual background

statox — Fri, 15 Jul 2022 00:00:00 +0000

Recently I created a script which automatically downloads NASA's Astronomy Picture of the Day and sets it as the wallpaper of my computer. This was fun but I also wanted to set the picture as my virtual background in Zoom so that ~~I have another pretext to bore my coworkers with space stuff~~ my teammates could also enjoy the cool astronomy pictures of the Nasa.

The issue was that when I dug into Zoom's documentation I couldn't find a reliable way to change my own virtual background without using the GUI. So I went with a hacky solution which seems to work well for my use case.

The solution is not perfect and I only tested it for my particular use case, but the result makes me happy so here it is:

When I couldn't find in the different SDKs and APIs offered by zoom a way to change the user's virtual background I first thought that I was missing something and decided to check the Zoom apps marketplace looking for apps which would already do what I want.

It turned out the only relevant apps I could find were ones which add new Virtual Background to the ones available in the app but never set a background for the user.

That meant that I had to find a different way, a hacky way 😈

On Linux, Zoom stores its data in $HOME/.zoom/data/ and in this directory there is a subdirectory named VirtualBkgnd_Custom which is quite interesting.

When the user sets a virtual background the image is actually copied to this directory and renamed with a random uuid.

The interesting thing is that Zoom doesn't do any validation or caching of any kind: You can just replace this image, restart the app and voila the virtual background has been changed.

cp myNewBackground.jpg $HOME/.zoom/data/VirtualBkgnd_Custom/{d04bd4b9-57d8-44a1-9cd7-31cea7945157}

So with a chunk of javascript and a tiny bit of shell I made this repo: A nodeJS program which downloads the picture and changes the wallpaper and the zoom background added to a cron table and the job is done. I took some shortcuts which probably make the tool only work on my machine, but maybe I'll make it more robust one day when I get a new computer or reinstall my system.

The stuff I do

Using a systemd timer to monitor my laptop's battery

The solution 🔗

Result 🔗

Resources 🔗

Synchronized colorscheme toggling for WezTerm and Neovim

The context 🔗

The problem 🔗

What others have tried 🔗

My initial attempts 🔗

The solution: High level overview 🔗

Implementing the WezTerm side 🔗

Managing colorscheme state 🔗

Toggling the colorscheme 🔗

Integrating into the main config 🔗

Setting up the keybinding 🔗

Implementing the Neovim side 🔗

Setting up the file watcher 🔗

Reading and applying the colorscheme 🔗

Putting it all together 🔗

How it all works together 🔗

Reflections 🔗

What about system-wide settings? 🔗

Make it yours 🔗

Looking for a new terminal emulator on Linux

Goal 🔗

workflow to preserve 🔗

Kitty 🔗

Wezterm 🔗

Update after about one month of usage 🔗

Python codegolf tips

Resources 🔗

Input parsing 🔗

Creating my homemade environmental sensors system

Supporting software architecture 🔗

Breadboard prototyping 🔗

VMA320 🔗

DHT11 and battery level 🔗

From the breadboard to the protoboard 🔗

DHT22 🔗

BME280 🔗

Adding an enclosure 🔗

Indoor enclosure 🔗

Outdoor enclosure 🔗

Capteurs environnementaux maison

Using an OpenSearch (AWS's ElasticSearch) cluster to monitor my API

Setup the cluster in AWS 🔗

About ingestion 🔗

The UI 🔗

Configuring the database 🔗

1. Create the index template. 🔗

2. Create the data stream 🔗

3. Create the Index pattern 🔗

4. Set up the index policy 🔗

Conclusion 🔗

Automatically merging Dependabot PRs on Github

Setup tests 🔗

Setups in Github 🔗

Dependabot 🔗

Actions settings 🔗

Branch protection 🔗

Setup the workflow 🔗

Trigger 🔗

Permissions 🔗

Filter dependabot 🔗

Repo setup and tests 🔗

Handle the PR 🔗

Result 🔗

Notification 🔗

Javascript codegolf tips

Resources 🔗

Type conversions 🔗

Decimal string to Number 🔗

Binary string to Number 🔗

Number to binary string 🔗

ASCII 🔗

Position in the alphabet 🔗

Loops 🔗

For < while 🔗

Declarations in for loop first statement 🔗

Declarations in `for` loop first statement 🔗

`for (... of ...)` 🔗

Using `&&` and `||` 🔗

Shorten `readline` calls 🔗

Execute code in the `readline` arguments 🔗

Replace `console.log` by `print` 🔗