Blogs on Lukasz Zagroba

Building Tickstream: Real-Time Crypto Analytics with an AI Assist

Sat, 18 Apr 2026 09:00:00 +0000

I built Tickstream over a few weekends. It’s a real-time crypto analytics pipeline: BTC and ETH prices stream in from Coinbase, anomalies get detected against a rolling baseline, a local LLM generates narrative for significant moves, and news events get correlated with price reactions after the fact. Everything is observable through a Grafana dashboard embedded in a landing page.

This post is a walkthrough of how the system works and an honest account of building it with AI assistance.

The Architecture

The system is a set of independent services, each responsible for one concern, communicating through an in-process event bus. This keeps the design modular: each piece can fail, retry, or be replaced without cascading through the rest.

 Coinbase WebSocket
        │
        ▼
  Price Consumer Verticle ─────────────────────┐
        │                                      │
        ▼                                      ▼
 Anomaly Detection Verticle            InfluxDB Verticle
        │                                      ▲
        ├──► Push Notification (ntfy)          │
        │                                      │
        └──► InfluxDB (anomaly regions) ───────┘

 Finnhub News ──► Sentiment (Ollama) ──► Price Impact ──► InfluxDB (events)
                                                |
                                                ▼
                                          SQLite storage
                                                │
                                                ▼
                                         Daily Briefing (ntfy)

Prices stream in from Coinbase’s WebSocket feed and flow in two directions: straight to storage, and through an anomaly detector that watches for significant moves against a rolling baseline. When something crosses the threshold, a push notification goes out and the anomaly is recorded alongside the price data.

News events are handled separately. A scheduler polls Finnhub at regular intervals, and each new item gets run through a local LLM for sentiment analysis before being matched against the price data in the surrounding window. The result is a record of what happened in the market alongside what was being said about it at the time — the correlation is more interesting than either signal alone.

Everything is observable through Grafana: price history, detected anomalies, annotated news events, and application metrics and logs. The whole thing runs in Docker Compose on a Hetzner CX32, with Caddy as the reverse proxy handling the landing page and Grafana panel embedding.

What I Learned

Vert.x is well suited to event-driven processing — the verticle model keeps concerns clean and the event loop handles high-frequency ticks without drama. The harder part was making it all stable: getting flows to restart on failure, wiring up retries with the right backoff, and making sure the observability was good enough to actually know the app was working as it should. Health checks, Prometheus metrics, and structured logs help keep an eye on the status of the app.

InfluxDB 3 was a deliberate choice to work with a timeseries database outside my usual stack. Grafana was the highlight. The flexibility of the dashboard builder — composable layouts, annotation overlays, variable-driven queries, Loki integration alongside metrics — is something I hadn’t fully appreciated before. The fact that it’s open source and self-hosted makes it even better.

AI-Assisted Development: What It Actually Looked Like

I used Claude Code throughout — it runs in the terminal, integrates naturally with Vim and my Omarchy setup, and keeps context on the full codebase across the session.

The best use was kick-starting unfamiliar territory. InfluxDB 3 and the Grafana dashboard format were both fairly new to me, and having a working starting point — even an imperfect one — was faster than reading the full API surface from scratch. The generated code for InfluxDB queries gave me something to run against and iterate on. Same with Grafana: the dashboard JSON was based on older schema versions and needed corrections, but it was a better starting point than a blank file. The problem is that the model’s training data skews toward older documentation and Stack Overflow answers. For anything touching a recently-changed API or a niche tool, going straight to the official docs is faster than iterating through confident wrong answers. The one frustrating limitation: Claude won’t give you direct links. Being pointed to the right documentation page would have saved real time on multiple occasions.

The bigger lesson was about architecture. I generated a first version quickly and then realised most of the structural decisions were wrong. Vibe coding doesn’t work — not because the generated code is bad line by line, but because you can produce plausible-looking architecture that solves the wrong problem. The anomaly detector was the clearest example: the first generated version looked reasonable, but when I read it carefully it didn’t actually match what I needed. The issue was that I hadn’t thought through what anomaly detection should mean for this feed — what constitutes an entry, an exit, where to anchor the baseline. Only after working that out myself was I able to describe the problem clearly enough to get good code. The model can implement a solution well; it can’t figure out what the solution should be.

This also puts new pressure on code review. It’s surprisingly easy to generate nonsense that looks correct, compiles, and passes tests — especially when the tests were also written by AI. The test suite validates the implementation’s own assumptions, not yours. Reading AI-generated code requires more attention than reading code a colleague wrote, not less — because a colleague usually has a mental model of the problem that leaks into the code in legible ways. Generated code can be locally coherent but globally wrong, and the only way to catch that is to actually understand what it’s doing.

Working with Local LLMs

Running Ollama locally is useful for a side project — no API costs, no data leaving the machine, and baking the model into the Docker image at build time means the container starts with no runtime dependency on a model registry. The harder problem is getting consistent, meaningful output. Out of the box, llama3.2:3b was pretty good at generating the daily briefing based on the news from the day, but when doing sentiment analysis it was skewing bearish on almost everything — I think that’s an inherent bias in the base model toward negative framing in financial contexts. I fine-tuned it to correct that, and overcorrected: it started reading bullish into news that clearly wasn’t. After another round it’s drifted bearish again. The scores are still inconsistent in a way that’s hard to reason about. Fine-tuning a small model for a specific analytical task is not as straightforward as it sounds, and I’m not convinced I’ve found the right approach yet. It’s something I want to dig into more seriously — the question of how to reliably adapt a general-purpose model to a narrow domain is genuinely interesting and I’ve only scratched the surface.

The Project as a Portfolio Artifact

The goal from the start was to demonstrate Kotlin stream processing in a real setting. The AI assistance made it faster to build, and occasionally introduced errors that I had to catch and fix. That’s about the right way to describe it. The judgment about what to build, whether it was working correctly, and where the design was going wrong — that stayed with me. Which is, I think, exactly where it should be.

The project page is at tickstream.lzag.dev.

AI Tug of War: Everyone's Shipping. Nobody's Building.

Mon, 23 Mar 2026 09:00:00 +0000

Everyone is offloading the hard parts. The pipeline is clogged. And the generation that was supposed to learn — didn’t.

There is a new dynamic spreading through engineering teams, and almost nobody is naming it clearly. It isn’t about productivity. It’s about offloading. Specifically, it’s about who can use AI to generate work that forces someone else to do the hardest part.

Junior developers can now produce an extraordinary volume of code. Pull requests come in at a pace that would have been unthinkable two years ago. The problem is that generating a PR and writing correct, reviewable code are not the same thing. Seniors are being buried under a growing mountain of AI-generated output that looks plausible on the surface but requires genuine expertise to evaluate. The pipeline isn’t moving faster — it’s clogging.

It Runs Both Ways

But let’s not pretend this is only a junior problem. Senior engineers who want to ship fast are doing the same thing in the other direction — using AI to crank out features and, when something inevitably breaks, deferring the investigation and cleanup to junior teammates. Meanwhile, managers can now generate entire sprint backlogs from a single brainstorm session. Tickets appear at volume, each one requiring someone lower in the chain to actually determine whether the idea is technically coherent. Nobody is accountable for the mess they’ve created upstream, because generating the mess required almost no effort.

This is the tug of war: every role has found a way to use AI to pass the genuinely difficult work to someone else. The result isn’t more velocity. It’s more noise at every layer of the organization, with the same or worse actual delivery — and, when something ships, the quality often suffers visibly.

The Same Disease in Your Feed

This pattern isn’t limited to code. Look at any professional content feed right now. The volume of long-form posts has exploded because turning a half-baked idea into a thousand words takes about ninety seconds with the right prompt. The posts look considered. They use all the right frameworks. But scroll through enough of them and something becomes clear: almost nothing new is actually being said. It’s the same ideas, recirculated, reworded, re-packaged. The marginal cost of producing content has collapsed to near zero, and so the feed has become a landfill of well-formatted emptiness.

Fake Productivity

Funny how there’s so much productivity, but no product.

We are now at the point where it is becoming possible to see clearly what the AI productivity boom actually was — at least in part. A lot of it was fake. Not fake in the sense that nothing got built. Things got built. But the output wasn’t meaningfully better, the code wasn’t meaningfully more correct, and the ideas weren’t meaningfully more original. What grew was volume. Volume of PRs, volume of tickets, volume of posts, volume of job applications. Noise, not signal.

The tools for filtering that noise have not kept pace. Code review cannot be fully automated in any way that provides real safety guarantees. Human judgment is still the bottleneck, and human reviewers are being asked to process an order of magnitude more input than they were before, with roughly the same number of hours in the day. Something has to give — and what gives is either quality or people’s sanity.

The Generation That Didn’t Learn

There is a longer-horizon problem that deserves more attention than it’s getting. Engineering skill is built through friction. You learn a technology by fighting with it — by hitting the wall, reading the stack trace, understanding why the abstraction leaked, internalizing the behavior. That process is slow and sometimes painful, and it is also irreplaceable.

A significant portion of the junior developers entering the field right now are not going through that process. They are going through a different one: write a prompt, get code, debug it with another prompt, ship it. This can produce results fast enough to pass code review in many organizations. It does not produce engineers who understand what they built.

The consequences of that are deferred, not avoided. Systems will break. Incidents will happen. When they do, you need engineers who can reason about what the system is actually doing — not just engineers who can prompt their way toward a solution in ideal conditions. The bill for skipping the hard work of learning is coming. It is just not due today.

Where This Leaves Us

The organizations that figure this out fastest will be the ones that raise the bar on what counts as a reviewable contribution — not to be obstructionist, but because high standards are genuinely the only defense against a pipeline flooded with plausible-looking slop. That means investing in tooling, yes, but more fundamentally, it means rebuilding a culture that takes correctness seriously as a shared responsibility rather than something to be deferred upstream or downstream.

Individually, the question is simpler but harder to answer honestly: where do you actually stand? Are you the one generating the noise, or the one with the judgment to evaluate it? Because in a world where content and code can be produced at near-zero cost, the scarce thing — the actually valuable thing — is the judgment to know what is good.

That judgment doesn’t come from prompts. It still comes from doing the work.

Exploring APT

Sat, 06 Sep 2025 23:38:06 +0000

As a software developer, I like to really understand the tools I use every day to feel more in control and aware. I’ve been using APT for a long time to handle updates, dependencies, and installations on my Debian-based systems, but I’ve never looked into how it actually works. Learning about things like how it uses GPG keys to check repositories or manages software sources could come in handy for tasks like setting up Docker containers or solving issues with software updates.

intro

APT (Advanced Package Tool) is a powerful, high-level package management system used by Debian-based Linux distributions like Ubuntu. It’s a suite of tools, including apt-get and apt-cache, that simplifies software management by relying on the lower-level dpkg tool to install, configure, and remove software packages.

At the heart of Debian-based systems like Ubuntu are .deb packages, which function as software installers. These packages are archives containing two main parts: the application’s files, laid out in the exact directory structure they’ll be installed in, and a DEBIAN directory holding crucial metadata. This metadata, particularly in the control file, lists the package’s version, description, and dependencies. This is where the apt package manager comes in; it reads this information to automatically handle installations, upgrades, and the complex web of dependencies, ensuring that all necessary software components are in place for a smooth and functional system.

Applications rarely exist in isolation and often depend on other programs or libraries to function correctly. Manually tracking and installing every single dependency would be a nightmare and this is where APT comes in. When you ask APT to install a package, it intelligently reads the package’s metadata, identifies all its dependencies, and automatically installs them from its configured sources.

configuration

So, where does APT get all this software? From repositories. A repository is essentially a remote server (or even a local directory) that hosts a collection of .deb packages and crucial metadata files. Your system knows which repositories to check by reading configuration files. The primary file is /etc/apt/sources.list, and additional repository sources are placed as separate .list files in the /etc/apt/sources.list.d/ directory. You can view your configured sources using the cat command on these files.

Repositories contain index files (like Packages.gz), which act as a catalog for all the available software. When you run the sudo apt update command, APT downloads the latest versions of these index files from all configured repositories. It then builds a local cache of available packages, their versions, and their dependencies on your machine.

When you later run a command like sudo apt install <package-name>, APT consults this local cache to find the package and figure out exactly what needs to be downloaded and installed to make it work.

gpg authentication

Using GPG with APT ensures the security and authenticity of software packages. GPG keys are used to verify the digital signatures of a repository’s metadata and packages, confirming they haven’t been tampered with and come from a trusted source. Repository maintainers sign their Release files with a private GPG key. When you run apt update, APT downloads this signed file and uses the corresponding public GPG key (which you’ve trusted and added to your system) to validate the signature. This establishes a chain of trust, protecting against malicious packages by ensuring that the software you’re about to install is exactly what the repository intended for you.

components and suites

When you look at a repository entry in your sources.list file, you’ll see terms like main, universe, stable, or focal. These are components and suites.

Components: These are categories of software within a repository, usually grouped by licensing or support level. For example, Ubuntu uses main (fully supported), universe (community-maintained), restricted (proprietary drivers), and multiverse (software with legal restrictions). Most third-party repositories will simply use a main component.

Suites: Suites: These refer to a specific release of your operating system, telling APT which set of packages matches your system. For example, if you’re using Ubuntu 22.04, the suite is jammy. Other sources might use suites like stable or nightly to signal which version of the software you want to track.

source file syntax

APT source files have two main syntaxes: the traditional one-line format and the newer deb822 format. The one-line format is a single entry that’s easy to read, while the deb822 format is a more structured, multi-line format with distinct fields. The most comprehensive documentation can be found in the sources.list man page.

One-line Format
This is the classic syntax where each repository is on a single line. It’s often found in .list files.

Example:

deb http://us.archive.ubuntu.com/ubuntu/ jammy main restricted

deb822 Format
This is a newer, more readable syntax that uses a stanza-based structure, similar to a .deb package control file. It’s often used in .sources files in the sources.list.d/ directory. In this format, each piece of information has a clear label.

Types: deb
URIs: http://us.archive.ubuntu.com/ubuntu/
Suites: jammy
Components: main restricted

useful commands

Below I’m listing the most useful commands with short descriptions for reference.

apt update

Refreshes the local package cache. This command retrieves the latest index files from all configured repositories but does not install or upgrade any packages. It’s a critical first step to ensure your system knows what software versions are available.

apt search

The apt search command is a powerful tool for discovering new software, not just checking for a given name. It searches the local package cache for packages and descriptions that match a given name or keyword. This is useful when you’re looking for a specific type of software but don’t know the exact package name.

For example, if you wanted to find different web servers available in your repositories, you could run:

apt search web server

This would return a list of packages where either the name or the description contains the words “web” and “server”, allowing you to browse different options like apache2, nginx, or lighttpd.

apt install

Downloads the .deb packages to /var/cache/apt/archives/ and then calls dpkg to extract and install them on the system. It also ensures that the packages are installed in the correct order to satisfy the dependencies.

apt clean

Clears out the local cache of downloaded package files from /var/cache/apt/archives/. This is useful for freeing up disk space, especially in storage-constrained environments like Docker containers or after you’ve finished a large batch of installations.

apt list

Lists all packages known to APT from the local cache. You can use flags like –installed to see only the packages currently on your system or –upgradable to see what can be upgraded. It’s also powerful when combined with tools like grep to find specific packages.

Diving into PHP: the fun of custom extensions

Tue, 24 Jun 2025 00:07:51 +0000

Inspired by a chapter on compiling from source in How Linux Works, I decided to practice a bit by building a PHP extension. Compiling programs from source is a valuable skill, and what better way to practice than by extending everybody’s favourite language, PHP? This post explores creating a simple PHP extension to add a function that wraps a string in a decorative border and experiment with creating a new class.

What are PHP extensions?

PHP extensions are compiled libraries that enhance PHP’s core functionality. Written in C or C++, they integrate with the PHP engine to provide new features, boost performance, or connect to external systems.

Writing a PHP extension is useful when you need to:

Add unique functionality not in PHP’s standard library.
Improve performance for compute-intensive tasks.
Interface with custom or low-level libraries.
Optimize specific application workflows.

Why not just create a standard PHP library, you ask? Well, creating an extension gives us access to pretty low level PHP plumbing, so when we want to squeeze every little bit of performance from the language that might be the best way. And didactically it’s also a great way to learn how PHP works behind the scenes.

Project goal

The goal is to create a PHP extension with:

A printLuLu(string, length=10) function that prints a string framed by alternating = and - lines.
A PrintLu class with two methods:
- printWrap(string, length=10): Prints a string with the same border.
- getLuLen(string): Returns the string’s length.

Getting started

I followed tutorials from PHP Internals Book and Zend’s guide. The former is detailed but slightly outdated, while the latter is more recent but less in-depth.

PHP vs. Zend Extension

PHP extensions add user-facing functionality, while Zend extensions modify the engine itself. Since I’m adding a simple function and class, a PHP extension suffices. Extensions can be compiled as shared libraries or statically linked into the PHP binary. I’m going with static linking for this one.

Setting up

First, I cloned a forked PHP source repository: php-src-fork and build PHP from it:

sudo apt install -y pkg-config build-essential autoconf bison re2c libxml2-dev libsqlite3-dev
./buildconf
./configure --enable-debug
make -j4

Generating the extension skeleton

PHP provides a script to create an extension template that generates boilerplate code and saves time on setup:

./sapi/cli/php ext/ext_skel.php --ext print_lu

Generated code can be cound in ext/print_lu.

Navigating the code

PHP extensions make heavy use of macros that simplify many tasks. However, these macros can make code hard to understand sometimes. To check out what gets replaced I can run the preprocessor that would inline the macros code:

gcc -E -P -Imain -I. -IZend -ITSRM ext/print_lu/print_lu.c > comp.c && clang-format -i comp.c

This creates a formatted comp.c file with expanded macros.

Writing tests

PHP’s testing framework is well-documented. The skeleton includes sample tests, so it’s easy to figure out the structure. I’m writing 4 tests:

Check if the print_lu extension is loaded.
Test printLuLu output.
Test PrintLu::printWrap.
Test PrintLu::getLuLen.

--TEST--
Check if print_lu is loaded
--EXTENSIONS--
print_lu
--FILE--
<?php
echo 'The extension "print_lu" is available';
?>
--EXPECT--
The extension "print_lu" is available

--TEST--
print_lulu Basic test
--EXTENSIONS--
print_lu
--FILE--
<?php
printLuLu("Hello, World!");
printLuLu("Hello, World!", 4);
?>
--EXPECT--
=-=-=-=-=-
Hello, World!
=-=-=-=-=-
=-=-
Hello, World!
=-=-

--TEST--
printWrap Basic test
--EXTENSIONS--
print_lu
--FILE--
<?php
$printLu = new PrintLu();
$printLu->printWrap("Hello, World!", 4);
?>
--EXPECT--
=-=-
Hello, World!
=-=-

--TEST--
getLuLen Basic test
--EXTENSIONS--
print_lu
--FILE--
<?php
$printLu = new PrintLu();
$len = $printLu->getLuLen("Hello World");
var_dump($len);
?>
--EXPECT--
int(11)

Run tests with:

make test TESTS="ext/print_lu"

Initially, tests were skipped because the extension wasn’t loaded, so to enable it, I rebuilt PHP:

make clean
./buildconf
./configure --enable-print_lu --enable-debug
make -j4

This made the first test pass.

Implementing the function

Let’s implement the first function. We need to add code to print_lu.c as well print_lu.stub.php, removing the old test functions and adding the new definitions. The convenient thing is that a lot of boilerplate code will be generated for us.

File print_lu.c defines the inner workings of the function using the PHP_FUNCTION macro as well as param parsing macros:

PHP_FUNCTION(printLuLu)
{
    zend_long line_len = 10;
    char *var;
    size_t var_len;
    size_t i;

    ZEND_PARSE_PARAMETERS_START(1, 2)
            Z_PARAM_STRING(var, var_len)
            Z_PARAM_OPTIONAL
            Z_PARAM_LONG(line_len)
    ZEND_PARSE_PARAMETERS_END();

    for (i = 0; i < line_len; i++) {
        php_printf("%c", i % 2 == 0 ? &#039;=&#039; : &#039;-&#039;);
    }
    php_printf("\n");

    php_printf("%s\n", var);

    for (i = 0; i < line_len; i++) {
            php_printf("%c", i % 2 == 0 ? &#039;=&#039; : &#039;-&#039;);
    }
    php_printf("\n");

    RETURN_NULL();
}

File print_lu.stub.php defines the function interface:

function printLuLu(string $string, int $len = 10): void {}

This makes our second test pass

Implementing the class

I added the PrintLu class to print_lu.stub.php:

class PrintLu {
    public function printWrap(string $string, int $len = 10): void {}
    public function getLuLen(string $string): int {}
}

The class methods use the PHP_METHOD macro:

PHP_METHOD(PrintLu, printWrap)
{
    zend_long line_len = 10;
    char *var;
    size_t var_len, i;

    ZEND_PARSE_PARAMETERS_START(1, 2)
        Z_PARAM_STRING(var, var_len)
        Z_PARAM_OPTIONAL
        Z_PARAM_LONG(line_len)
    ZEND_PARSE_PARAMETERS_END();

    for (i = 0; i < line_len; i++) {
        php_printf("%c", i % 2 == 0 ? '=' : '-');
    }
    php_printf("\n%s\n", var);
    for (i = 0; i < line_len; i++) {
        php_printf("%c", i % 2 == 0 ? '=' : '-');
    }
    php_printf("\n");

    RETURN_NULL();
}

PHP_METHOD(PrintLu, getLuLen)
{
    char *var;
    size_t var_len;

    ZEND_PARSE_PARAMETERS_START(1, 1)
        Z_PARAM_STRING(var, var_len)
    ZEND_PARSE_PARAMETERS_END();

    RETURN_LONG(var_len);
}

PHP_MINIT_FUNCTION(print_lu)
{
    zend_class_entry *print_lu_ce = register_class_PrintLu();
    return SUCCESS;
}

And now we have all the tests passing!

Next steps

This is a fairly simple example, but shows that starting writing a simple PHP extension is not overly complicated. Having some basic knowledge of this process allows us to inspect other extensions and understand their inner workings. There’s much more that can be done with these extensions, but that’s too much to cover in one post. I recommend reading the linked sources for more info and experimenting.

Debugging connections with strace and packet analysis

Tue, 20 May 2025 22:37:37 +0000

Debugging TCP Connections: Tools and Techniques

This post explores tools and techniques for debugging TCP connection management, focusing on strace, tshark (Wireshark’s CLI counterpart), ss, and comparisons with alternatives like tcpdump and netstat. We’ll analyze single-threaded and multi-threaded servers, as well as a Vert.x-based server-sent events (SSE) setup, to understand TCP behavior, system calls, and packet exchanges. The code and configurations are available in the following repositories:

Webserver project: github.com/lzag/webserver/tree/strace
SSE server and client: github.com/lzag/sse-tcp-server, github.com/lzag/sse-tcp-client

Tools Overview

strace

strace is a powerful tool for tracing system calls made by a program. It’s invaluable for debugging TCP connections by revealing how a server interacts with the kernel, including socket binding, polling, and data transfer. Its output can be overwhelming without filters, so here are key options for effective use:

-Y: Print command names for PIDs.
-y: Decode file descriptors (FDs).
-T: Show time spent in each system call.
-tt: Include timestamps.
-s <size>: Set maximum string size (default 32).
-a 40: Align output for readability.
--syscall-limit: Limit the number of syscalls displayed.
Filtering: Use %net for network-related calls and %desc for file descriptor-related calls to focus on TCP-relevant syscalls. You can also filter by specific syscalls, groups, regex, FD numbers, status (successful/unsuccessful), or paths.

Timing data from -T is critical for identifying slow syscalls, making strace ideal for performance optimization.

tshark

tshark is the command-line interface for Wireshark, offering detailed packet capture and analysis. It’s user-friendly with well-formatted output, making it a preferred alternative to tcpdump. Key options include:

-i <interface>: Specify the network interface (e.g., lo for loopback).
-f "<filter>": Use Berkeley Packet Filter (BPF) syntax to capture specific packets (e.g., tcp port 9090).

tshark reveals packet-level details like connection setup (SYN-ACK), data transfer, and teardown (FIN-ACK), helping diagnose issues like slow responses or connection delays.

ss

ss is a modern replacement for netstat, offering more options for analyzing socket states. It’s particularly useful for monitoring TCP send and receive queues. A sample command to monitor connections on port 8080, refreshing every second:

watch -n 1 'ss -tn state all "( sport = :8080 or dport = :8080 )"'

This command filters TCP connections by port and displays queue sizes, helping identify bottlenecks in data flow.

Alternatives

tcpdump: A lightweight packet capture tool, but its output is less formatted than tshark, making it harder to parse.
netstat: A traditional tool for socket analysis, now largely replaced by ss due to its limited options and slower performance.

Analyzing a Threaded Webserver

Using the webserver project (GitHub branch: strace), we’ll compare single-threaded and multi-threaded versions to understand TCP connection handling. The setup uses Docker Compose to run the server and client, with strace and tshark executed inside containers.

Setup Commands

To initiate the client and capture packets run these command in 3 separate terminals:

INDEX_FILE=single.php docker compose up or INDEX_FILE=multi.php docker compose up
docker compose exec php-server php client.php
docker compose exec php-server tshark -i lo -f "tcp port 9090"

strace Analysis

The server is wrapped in strace to monitor network (%net) and file descriptor (%desc) syscalls. The sequence of operations includes:

Socket Binding: The server binds to a port.
Polling: The server polls for incoming connections.
Request Handling: Reads client requests and writes responses.

For the single-threaded server, strace shows accept calls with non-blocking flags and timings for each syscall. This helps identify slow operations. In the multi-threaded version, PHP uses pselect for polling, which slightly alters the syscall pattern but doesn’t significantly change the connection flow (open images in new tab to see the details).

The timing data (-T) highlights which syscalls are performance bottlenecks, guiding optimization efforts. For example, a slow read or write syscall might indicate buffer issues or network delays.

tshark Analysis

tshark captures the packet exchange:

Connection Setup: SYN-ACK pattern establishes the TCP connection.
Data Transfer: HTTP GET request and response.
Teardown: FIN-ACK closes the connection.

In our test, the response took 5 seconds in single-threader server and 10 seconds in multi-threaded one, since we extended the sleep for the test. Comparing single- and multi-threaded servers, tshark shows similar packet patterns.

Debugging SSE with Vert.x

The SSE scenario (GitHub: sse-tcp-server, sse-tcp-client) uses a Vert.x server to stream events, allowing us to debug TCP send/receive queues and flow control. We’ll use strace, tshark, and ss to analyze the setup.

strace Analysis

strace reveals that Vert.x uses epoll for socket polling, which is more efficient than PHP’s select.

ss Analysis

Using the ss command, we monitor TCP queues:

watch -n 1 'ss -tn state all "( sport = :8080 or dport = :8080 )"'

As the SSE client connects, the receive queue fills up initially, followed by the send queue as the server streams events.

When the server’s TCP window fills, tshark shows zero-window packets, indicating flow control.

As the window opens, window update packets appear, and streaming resumes. This demonstrates TCP’s built-in flow control in action.

tshark Analysis

tshark captures the SSE streamand ss monitors the send and receive queues for server and client:

tshark -i lo -f "tcp port 8080"

Initial Events: A few events are sent successfully.
Window Full: Zero-window packets halt transmission.
Recovery: Window updates allow streaming to resume.

By adjusting server settings like the TCP send buffer size or send queue size, we can observe their impact on queue behavior and streaming performance.

Key Insights

strace: Essential for tracing syscalls and identifying performance bottlenecks. Use filters (%net, %desc) to focus on TCP-related calls.
tshark: Provides detailed packet-level insights, ideal for debugging connection setup, data transfer, and teardown.
ss: Monitors TCP queues, revealing flow control dynamics and queue bottlenecks.

Sources

10 Essential Strategies for High-Performance API Design

Sat, 17 May 2025 02:59:43 +0000

API Performance Optimizations: Techniques and Insights

Optimizing API performance is critical for delivering fast, scalable, and reliable services. Below I summarize key strategies with practical insights to enhance your API’s efficiency.

Server-Side Caching

Caching reduces server load and accelerates responses by storing frequently accessed data. Cacheability determines which responses can be cached based on their idempotency and freshness. GET requests are typically cacheable, while POST or PUT requests often aren’t due to side effects.

Whole Response Caching: Store entire API responses in an in-memory store like Redis or Memcached. For an e-commerce API, cache a product catalog to avoid repeated database queries, with an expiration time like 1 hour to balance freshness and performance.
Fragment Caching: Cache specific parts of a response, such as a user profile summary or navigation menu, in dynamic applications. This avoids reprocessing unchanged sections.

Caching Headers

HTTP caching headers enable clients and intermediaries, like browsers or CDNs, to reuse cached data, reducing server requests.

Cache-Control: Defines caching rules, like max-age for freshness. For example, public, max-age=3600 allows caching for 1 hour.
ETag: A unique identifier for a resource’s version. Clients send the ETag; if it matches, the server returns a 304 Not Modified, avoiding redundant data transfer.
Last-Modified: Indicates the resource’s last update time. Clients check if it has changed since the provided timestamp, also avoiding redundant data transfers.

Database Optimizations

Database queries often bottleneck APIs. Optimize them with these techniques:

Eager Loading: Fetch related data in one query to avoid the N+1 query problem, where one query for a resource (e.g. users) triggers additional queries for related data (e.g. posts). For example, when retrieving a list of users, include their posts in a single query instead of separate queries per user, reducing query count from N+1 to one.
Indexing: Add indexes to frequently queried columns, like a user ID in a posts table to speed up searches.
Connection Pooling: Use a pool of reusable database connections to reduce overhead during high traffic.

Response Size Optimizations

Smaller responses lower latency and bandwidth usage.

Selective Attribute Loading: Return only requested fields. In a REST API allow clients to specify fields via query parameters or use GraphQL for precise data selection.
Compression: Enable Gzip or Brotli to compress payloads like JSON, reducing transfer size.
Pagination: Limit results per page, such as 20 items per request, to prevent oversized responses.

Workload Parallelization

Concurrency boosts throughput by processing tasks simultaneously. For instance, when aggregating data from multiple external APIs, handle requests in parallel using threads or asynchronous tasks to reduce response time.

Rate Limiting

Rate limiting prevents abuse and ensures fair usage. Set a limit, like 100 requests per 15 minutes per client, to protect your API from overload while maintaining availability.

Asynchronous Processing

Offload resource-intensive tasks, like file processing or report generation, to background workers. Use a queue system or background job framework to keep API responses fast.

Content Delivery Networks (CDNs)

CDNs cache content closer to users, reducing latency. For APIs with cacheable responses, configure a CDN to store JSON payloads, minimizing origin server load.

Load Balancing

Distribute traffic across multiple servers for reliability and scalability. Use a load balancer to route requests based on server health or load, preventing bottlenecks.

Protocol optimizations

Modern protocols can significantly boost API performance over traditional HTTP/1.1. Upgrading to HTTP/2 or HTTP/3 enhances speed by allowing multiple data streams in a single connection and compressing headers. HTTP/3 further improves performance on unstable networks, such as mobile connections. Alternatively, gRPC offers a high-performance option outside standard HTTP, using compact binary formats like Protocol Buffers instead of JSON. This reduces data size and speeds up processing, especially for complex requests.

By applying these optimizations - caching, database tuning, response size reduction and more - you can significantly improve your API’s performance, delivering a seamless user experience and optimizing resource utilization.

My Coding Philosophy

Tue, 29 Apr 2025 12:11:31 +0000

Not long ago, someone asked me about my coding philosophy. I mumbled something semi-coherent at that time, but the question stuck with me and got me thinking - what is my coding philosophy? Having a clear credo to guide me through crunch times would be invaluable.
After some soul-searching, I boiled it down to five principles that keep me grounded and productive.

Solve Real Problems

I focus on tackling actual issues - bugs, unmaintainable code, or performance bottlenecks, rather than chasing hypothetical problems. Theoretical debates, like Vim vs. Emacs (Vim, obviously), often waste time when multiple valid solutions exist and personal preference is the deciding factor. By addressing concrete challenges as they arise, I keep progress on track and development rooted in reality.

Master the Fundamentals

Great programmers command the basics: data structures, algorithms, and core programming concepts. These foundations enable creative problem-solving and adaptability across tools and languages. For example, studying concurrency in Kotlin deepened my understanding of Swoole coroutines - same principles, different context. Mastery of essentials, like writing clear, reusable functions, takes deliberate practice, not just years of experience. It’s about drilling the essentials until they’re second nature. Skip this, and you’re building castles on sand, no matter how flashy your tech stack. I make sure that my foundation is rock-solid.

Pursue excellence

Excellence is diving deep into your tools - including reading source code, studying official docs, and knowing their quirks, not hacking together code that “just works” while your IDE winks at you knowingly. It’s easy to hack together spaghetti code and scrape by with surface-level knowledge, especially with AI tools at hand. But I’m not content with shallow knowledge and I always seek a profound understanding of whatever I’m working with.

Cultivate judgement

Judgment grows from experience, like learning to avoid “quick fixes” that spawn a dozen bugs. It’s about choosing the right technology, deciding which code to refactor, and picking tech battles wisely while learning from mistakes. This blend of technical knowledge, practical experience, and honed intuition takes time and practice to develop. I refine my judgment by questioning my assumptions, drawing on expertise and lessons from past errors.

Focus on Outcomes

In the end, it’s about shipping it. Success isn’t measured by how many design patterns you can name, but about delivering tangible results, whether it’s a side project or a work ticket. Intellectual rabbit holes are fun, but they’re no match for a working product. A coder who delivers reliably, even if their code isn’t poetry, outshines the genius lost in micro-optimizations. I strive to build a reputation for dependability by delivering consistent, impactful results.

Ace Technical Arguments with S.O.F.T.W.A.R.E.

Fri, 18 Apr 2025 12:42:19 +0000

Technical disagreements are a fundamental part of a programmer’s role. Whether debating frameworks, architectures, or the eternal tabs vs. spaces dilemma (spaces, clearly), these discussions can feel like high-stakes battles. However, programming is a collaborative effort - no one owns the codebase unless they’re single-handedly building the next unicorn in their garage. For those craving total control, a side project offers the freedom to code like a visionary artist, chasing bold tech dreams (or nightmares). In a team setting, though, it’s about sharing the sandbox and finding common ground. To navigate these debates effectively, I’ve distilled a practical approach into the S.O.F.T.W.A.R.E. framework, inspired by sales techniques but tailored for technical discussions.

Study Requirements

Before engaging in a debate, uncover the root of the disagreement. Is your colleague fixated on performance, stressed about deadlines, or driven by personal preferences? Ask open-ended questions like, “What drives your preference for this approach?” or “What risks are you aiming to avoid?” If they’re overlooking critical factors, such as financial implications, gently share the broader context. Align the discussion with company goals - if you’re unsure of those, do some reconnaissance. This step is akin to analyzing a codebase before diving in, preventing costly missteps. Sometimes, there’s no real disagreement; clarifying requirements may reveal you’re already aligned. Always define the problem, outline objectives, and explore all possible options, including those you might not have considered.

Overcome Objections

With a clear understanding of requirements, identify objections to your proposed solution. Play devil’s advocate to stress-test your ideas and anticipate concerns like complexity, cost, or past failures. Validate their worries: “I understand your concerns about integration risks.” Then, counter with practical solutions. For instance, if a manager fears repeating a failed integration, propose incremental changes to minimize disruption. Break the solution into small, manageable steps and back your case with benchmarks or proof-of-concepts (PoCs). Empirical evidence cuts through subjective debates, much like debugging with hard data instead of guesswork.

Forge Relationships

Build trust before disagreements escalate. Acknowledge your colleague’s expertise: “Your optimization last sprint was impressive.” Use collaborative language: “We both want a scalable system, don’t we?” A reputation for sound technical decisions earns goodwill, making persuasion easier. Avoid pushing a personal agenda - solutions should prioritize the team’s and company’s goals, not resume-driven development. Picking battles that benefit the organization carries more weight than championing pet preferences. Strong relationships create a foundation for constructive debates and smoother resolutions.

Tout Benefits

Don’t just pitch technical details (e.g., “It’s microservices!”). Focus on outcomes that resonate with your audience: “This isolates services, cutting debugging time significantly.” Tailor your pitch to their priorities - maintainability for engineers, cost-efficiency for business stakeholders, or faster delivery for project managers. For example, if a manager doubts a new tool, highlight how it reduces development time, backed by data if possible. Numbers speak louder than opinions, so tap into financial metrics or performance stats when relevant. Touting benefits is like showcasing a product’s impact, not its code, and requires hitting the right notes for your audience.

Work Out a Win-Win

Treat technical disagreements as opportunities for collaboration, not battles. Aim for a hybrid solution: “Let’s adopt my backend approach but retain your frontend design.” If compromise isn’t feasible, ensure the other party feels heard: “I’ll document your concerns for review if issues arise.” A win-win preserves team harmony and fosters goodwill. This approach mirrors designing software where everyone benefits from the outcome.

Alleviate Risk

Every technical solution carries risks, and depending on your company’s risk tolerance, this can be a significant concern. Poor tech choices can be costly or even catastrophic. Mitigate fears by proposing low-risk trials: “Let’s test this on one module for a sprint.” Offer to demo or prototype to address concerns about complexity or learning curves. Like a free trial in sales, a risk-free pilot builds confidence and lets data drive the decision. If your solution succeeds, great; if not, pivot without drama. Alleviating risk is about creating a safe environment to test ideas, much like trialing software before full adoption.

Reference Experts

Strengthen your case with credible sources: “This pattern reduced latency by 30% in [project].” Cite industry standards, whitepapers, or open-source examples to add weight to your argument. If the debate stalls, consult a neutral third party, like a senior engineer or external expert, for an unbiased perspective. This reduces perceived risk and grounds the discussion in evidence, similar to referencing trusted authorities in software development. Leveraging others’ experiences saves time on building PoCs and highlights potential outcomes, making your case more compelling.

Evaluate | Escalate | Evacuate

After reaching an agreement, follow up: “How’s the new setup working? Any tweaks needed?” If the solution succeeds, share credit: “Thanks for supporting this - it’s performing great!” If it fails, own the mistake to maintain credibility: “My bad, let’s fix it.” This also helps identify if your advice was ignored, providing leverage for future discussions. Evaluating outcomes is like conducting a post-mortem on a software release, ensuring continuous improvement.

If agreement proves impossible, escalate to a tech lead or engineering manager, presenting all gathered insights clearly and objectively. However, if your input is consistently disregarded and the environment feels misaligned, consider a change. Not every team or company is the right fit - finding a workplace that matches your work style can reduce conflicts and make debates less frequent and more productive.

Final Thoughts

Stay calm and don’t take technical debates personally. Not every decision is rational; some stem from irrational preferences. Agree on success criteria upfront - performance, cost, or timelines - to minimize opinion-driven conflicts. When clarity is lacking, compromise or escalate, but always keep the team’s shared goals in focus. Above all, stay professional, keep coding, and approach debates with a collaborative mindset.

Rate limiter experiments

Wed, 26 Mar 2025 12:59:17 +0000

What is a Rate Limiter?

A rate limiter is a mechanism designed to control the frequency of requests or actions in a system. It ensures that a service doesn’t get overwhelmed by too many requests in a short period, protecting it from overuse or abuse. Imagine a bucket that holds a limited number of tokens - each request takes one, and if the bucket runs dry, further requests are delayed or denied until it refills. Rate limiters are widely used for:

Preventing Overload: Keeping servers stable under high traffic.
Security: Mitigating denial-of-service (DoS) attacks by capping request rates.
Fair Usage: Ensuring equitable resource access among users or applications.

You might use a rate limiter in APIs, web services, or microservices to maintain performance and reliability. For a broad overview, see Rate Limiting on Wikipedia. I also drew inspiration from practical examples like Rate Limiters at Stripe and How a Rate Limiter Took Down GitHub, which highlight real-world applications and pitfalls.

Circuit Breaker

While rate limiters control request rates, circuit breakers handle failure resilience. A circuit breaker monitors a service for failures (e.g., timeouts, errors) and “trips” to block requests when a threshold is crossed, preventing cascading failures across a system. Once the service recovers, it “closes” to resume normal operation. Think of it as a safety switch in your home’s electrical system - cutting power to avoid overload.

They’re critical in distributed systems where one failing component can drag down others. I explored this concept alongside rate limiting because both contribute to system resilience, a topic well-covered in tools like Resilience4j, which I investigated during my experiments.

Concurrency Limiter

Before diving into my solutions, I also tackled a concurrency limiter. Unlike a rate limiter (which caps requests over time), a concurrency limiter restricts the number of simultaneous active requests. It’s perfect for scenarios where a service can only handle a limited number of concurrent resource-heavy operations. I implemented this alongside my rate limiter because both address resilience: one prevents overload over time, the other in the moment.

Why a Hands-On Approach?

I decided to build a rate limiter from scratch because abstract discussions only get you so far. Coding a proof of concept (PoC) forces you to wrestle with real-world details - headers, concurrency, exceptions - that don’t surface in theory. It’s the best way to clarify technical expectations and refine requirements. Plus, I wanted to explore how rate limiting ties into broader resilience strategies, like circuit breaking, in a tangible way. Resources like An Alternative Approach to Rate Limiting and Rate Limiting in Node.js fueled my curiosity to experiment hands-on.

Implementation Challenges: Going Distributed

A rate limiter in a single app is straightforward - you can track requests in memory with a counter. But in a distributed system, where multiple apps or instances serve requests, that approach falls apart. Each app might have its own count, leading to inconsistent limits. For true rate limiting, you need a centralized system that all instances can query, ensuring a unified view of request rates.

This is where distributed design becomes critical. I found Understanding Rate Limiting (YouTube) incredibly helpful for grasping these challenges. It emphasizes the need for a shared store - like Redis - to synchronize limits across services, a strategy also detailed in Scaling GitHub’s API with a Sharded, Replicated Rate Limiter in Redis, which I referenced for inspiration.

My Solutions

To tackle these challenges, I leaned heavily on Redis for its speed and atomicity, guided by Redis’s rate limiting patterns. Here’s how I approached it:

Centralized Storage: I stored all rate-limiting state in Redis, making it accessible to any app in the system. This ensures consistent limits across distributed instances.
Scripts in Redis: I embedded the core logic (e.g., token bucket refills, counter updates) in Redis Lua scripts. This minimizes network calls - everything happens in one atomic operation on the server side. For example, my token bucket checks tokens, updates counts, and sets expiration in a single script.
Concurrency Limiter: I implemented this with a simple counter (decremented when requests end) and a timestamp set for cleanup, all managed in Redis with termination middleware to avoid leaks.
Refill Strategy: I opted for an application-scheduled Redis script (schedulePeriodic) over periodic mset calls or Redis-side cron jobs. This gave me control over errors, visibility into timing (via getNextTimer), and atomic execution without excessive network overhead.

All scripts and configs are in my rate-limiter repo. The trade-off? Managing Lua scripts adds complexity, but their simplicity - akin to Simple Rate Limiting Algorithm Gist - keeps bugs at bay, and the performance gains are worth it.

Rate Limiter Algorithm Descriptions

Token Bucket

This algorithm maintains a fixed number of tokens (maxRequests) in a bucket that refills at a constant rate. Each request consumes one token, and if tokens remain (remaining >= 0), the request is allowed. The bucket is refilled by a Lua script executed periodically by the app. Learn more about its mechanics at Token Bucket Algorithm.

Timestamp Token Bucket

An enhanced token bucket that uses timestamps to calculate token replenishment dynamically. It tracks the last request time and adds tokens based on elapsed time relative to the window size, capping at maxRequests. Requests are allowed if tokens remain after consumption. In the current implementation, the user starts accruing tokens from the first request.

Fractional Token Bucket

A variation of the token bucket that divides the maxRequests into smaller fractional tokens (e.g., 1/60th). Tokens replenish incrementally over time based on elapsed time, allowing finer granularity. Requests are permitted if fractional tokens are available.

Leaky Bucket

This algorithm treats requests as water flowing into a bucket that leaks at a constant rate (maxRequests/windowSize). It tracks the number of requests (count) and removes them over time. If the bucket isn’t full (count <= maxRequests), the request is allowed.

Fixed Window Counter

This divides time into fixed windows (e.g., seconds) and tracks requests per window using a concatenated key. Each request decrements the counter, and the window resets after expiration. Requests are allowed if the counter hasn’t hit zero.

Sliding Window Log

This logs each request’s timestamp in a Redis sorted set and removes entries outside the sliding window. It counts requests within the window, allowing new ones if the total is below maxRequests. The log expires after the window duration.

Sliding Window Counter

This approximates a sliding window by dividing it into sub-windows (e.g., 1/60th of windowSize) and tracking request counts per sub-window. Old sub-windows are pruned, and requests are allowed if the total count across the window is below maxRequests.

Comparison table

Algorithm	Granularity	Storage Method	Time Handling
Token Bucket	Coarse	Single key (SETEX)	Fixed expiration
Timestamp Token Bucket	Medium	Hash (HSET)	Timestamp-based
Fractional Token Bucket	Fine	Hash (HSET)	Timestamp-based
Leaky Bucket	Medium	Hash (HSET)	Timestamp-based
Fixed Window Counter	Coarse	Concatenated key (SETEX)	Window index
Sliding Window Log	Fine	Sorted set (ZADD)	Timestamp-based
Sliding Window Counter	Medium	Hash (HSET)	Sub-window index

Wrapping Up: It’s All About Resilience

This project was about more than just rate limiting - it’s about building resilient systems. Rate limiters prevent overload, concurrency limiters manage real-time capacity, and circuit breakers handle failures. By implementing these hands-on, I learned how they interplay and how distributed systems demand careful design. Check out the code at my rate-limiter repo and try it yourself.

The importance of testing in software development

Fri, 21 Mar 2025 10:52:58 +0000

To test or not to test

Testing is the bedrock of solid software development. It’s not just about squashing bugs, it’s about crafting a foundation that keeps your code dependable, maintainable, and a breeze to work with. From my own experience, I’ve noticed not every developer buys into testing. Some rock a YOLO mindset, thriving on the thrill of tossing bugs into the wild and patching them later, rather than building a sturdy base of tested code. Skip the tests, and you’re flirting with a hulking monolith — devs start piling on new code instead of tweaking what’s there, terrified of shattering some vital piece. Left unchecked, this spirals into an untamable dumpster fire. To me, writing tests feels like a no-brainer, yet I keep running into devs who resist the idea. Here’s a rundown of the most common objections I’ve heard:

Common Objections to Testing (and Counter-Arguments)

“REQUIREMENTS FLIP AND TESTS EXPLODE”

Tests should break when requirements evolve — they’re there to flag when logic no longer aligns with expectations. If the business rules stay put, but you’re still rewriting tests for every new feature, that’s a sign your tests suck, likely because they’re overly focused on implementation details instead of the big-picture behavior. The problem isn’t testing itself — it’s how you’re doing it.

“WRITING TESTS TAKES MORE TIME THAN DOWNLOADING WINDOWS UPDATES”

Track your bugs religiously. We often lowball how long bug fixes take, and honest tracking reveals the real cost in time. If your bug-fix rate’s at zero, I’ll eat my words, but I’ve never seen a reliable codebase without solid test coverage. Plus, manual testing during dev? Total time vampire. Automating it speeds things up and makes code friendlier for the next person — no more secret voodoo in one dev’s head. As a manager be careful of rewarding the “bug hero” who swoops in for quick fixes, ignoring the devs churning out stable stuff.

“TESTS ON, BUGS STILL LAUGHING”

Testing’s an art you refine over time. You learng what’s absolutely crucial to test and what you can comfortably skip. I’ve seen tests so over-mocked they’re just flexing on a lone if statement in a sea of code. It takes practice - don’t expect gold right out of the gate.

“TESTING’S NOT MY GIG”

Some developers view testing as a QA task, but this separation increases effort . You’re already running the code to check it why not write a test to do that instead of poking it manually? Also, track the times when a ticket comes back with “QA failed,” and you have to restart development and test everything again, manually.

“NO TESTS, NO TOOLS, NO CLUE, NO HOPE”

In startups, speed often trumps quality. Fair enough to get to market, but as a company grows, bugs get pricier. Early on, it’s all about shipping; later, customers demand polish. Without tests, the code balloons, and there’s never a perfect moment to bring in tools or training. Managers, track defects and delivery hiccups. Carve out time for key tests and build team skills gradually - it’s a future-proof investment. Reward stable, tested code, too.

“MANAGEMENT COULDN’T CARE LESS”

They often don’t - until disaster strikes. Devs need to step forward, putting risks in language they get: measure time spent on bugs, note late deliveries, and tie it to financial impact. Management may not comprehend the code’s state, imagining tests cover it or underestimating fixable flaws. If after all they’re unbothered by chaos and you’re driven to fix it, you’re at a crossroads - either suck it up or jump ship. Code quality is a strategic decision - not all prioritize excellence, weighing it as a cost versus a long-term gain.

“TESTS ARE FOR WIMPS, YOLO”

YOLO’s cool if your shop’s a madhouse. Otherwise, I bet you’re just posturing, stacking code like a Jenga tower, chanting “don’t touch it” while it wobbles. Refactoring? What’s that?

“MY CODE’S A GOD - TESTS CAN’T TOUCH IT”

Tests? Pfft - those are for peasants, not your celestial genius, right? If you’re really the Michelangelo of coding, maybe I’m wrong, but really? (Track those bugs and prove it.)

“TESTS SUMMON THE CODE KRAKEN”

That’s a valid one! Still, I say it’s a risk worth taking. Pro tip: skip testing on a full moon.

How to Prioritize Testing

If you’re convinced by my counter-arguments, you might want to dive head-first into blanket-covering your codebase with tests, but hold on. Often, managers push full-on TDD with 100% coverage. That usually doesn’t lead to better-quality code - but neither does 0%. Figuring out this number varies for each codebase and takes judgment, especially if we’re starting from a low point. So, how do we decide what to test and what not?

Focus on Critical Paths

What’s the core functionality? Identify the features or flows users rely on most (e.g. login, checkout). If these break, your app’s dead in the water. Also, follow the money: if your code ties directly to revenue (e.g. payment processing, order fulfillment), test that first. A bug there hurts more than a glitch in the settings page.

Target High-Risk Zones

Bug-prone code: Look at git history or bug trackers — where do fixes cluster? Test the stuff that’s broken before.
Complex logic: Prioritize functions with lots of conditionals, loops, or edge cases. Complexity breeds bugs.
Refactoring candidates: If you plan to rework a module soon, tests give you confidence to make bold changes without breaking things.

Develop Awareness During Coding

With time, you can build a mental map of the codebase, so whenever the opportunity arises you can spot the problematic areas, quick wins and the best targets.

Why Testing is a Must-Have

Documenting Behavior

Tests act like living documentation. They show exactly how your code is supposed to work, making it clear for anyone (including future you) who needs to understand it. No more guessing what a function does - just run the tests.

Confidence in Refactoring

Want to clean up that messy codebase? With a good test suite, you can refactor fearlessly. Tests confirm your changes didn’t break anything, so you’re not holding your breath every time you deploy.

Detect Regressions and Cut Debugging Time

A test suite catches regressions fast - those sneaky bugs that can creep in even when you ’re doing only a tiny tweak’.” Instead of spending hours (or days) debugging, you pinpoint the issue in minutes.

Writing Testable Code: The How-To

Testing’s more than tossing assertions around — it’s about crafting code that’s made to be tested. Here’s how to make it happen:

Make Testing a Habit

Start writing tests from day one. It’s a grind that pays off, forcing you to nail design and edge cases early.

Cut Dependencies

Fewer dependencies mean fewer headaches. Reduce reliance on third-party libraries or pesky side effects like logging.

Keep It Simple

Write small, focused functions and use straightforward constructors. Complexity is the enemy of testability — simple code is easier to verify.

Stick to the Law of Demeter

Don’t dig too deep into object structures (e.g. obj.getA().getB().doSomething()). Stick to “tell, don’t ask” to keep your code loosely coupled and testable.

Avoid Hidden Dependencies and Global State

Global variables or sneaky side effects are testing kryptonite. Make dependencies explicit (e.g., pass them in) so your tests can control the environment.

Favour Generic Methods

Write reusable, flexible methods over hyper-specific ones. Generics are easier to test because they’re not tied to one-off scenarios.

Favour Polymorphism Over Conditionals

Swap if/else chains for polymorphic behavior (e.g. interfaces or subclasses). It’s cleaner, more extensible, and lets tests target specific implementations.

Choose Composition Over Inheritance

Inheritance can lock you into rigid hierarchies. Composition gives you flexibility, making it simpler to mock or swap out parts during testing.

Final Thoughts

Testable code isn’t just about passing tests — it pushes you to write better code. You get modular, manageable systems that make sense. Sure, it takes discipline, but the time you save debugging and the confidence you gain refactoring make it worth every second.

Automating Redis cluster setup

Tue, 18 Mar 2025 00:52:39 +0000

Introduction

This post serves as a companion to my repository, which provides an automated setup script for deploying a Redis cluster. Designed to streamline the process of configuring a clustered environment, the script enables users to quickly establish a functional Redis cluster with minimal manual intervention. In this guide, I’ll explain the purpose behind this project, detail the testing process for various configurations, and highlight key Redis capabilities uncovered during development. Additionally, I’ll clarify the differences between standalone and clustered Redis to provide context for when clustering is advantageous.

Purpose of This Project

The motivation for this work stems from my interest in distributed systems and resilient architectures. Redis is renowned for its performance as an in-memory data store, but its clustering feature offers a pathway to scalability and fault tolerance that standalone instances cannot achieve. This project is an experiment in harnessing Redis clustering to distribute data across nodes, manage failures, and simplify deployment through automation. By creating this setup, I aimed to explore Redis’ advanced features—such as sharding, replication, and dynamic scaling—while ensuring the process remains accessible and repeatable.

Standalone vs. Clustered Redis: Key Differences

Redis can operate in two primary modes—standalone and clustered—each suited to different use cases:

Standalone Redis

Description: A single instance hosting the entire dataset in memory.
Advantages: Simple to configure (start with redis-server), offers peak performance for moderate workloads, and avoids network latency.
Limitations: Constrained by the memory and processing capacity of a single machine. Without replication or persistence, a failure results in data loss or downtime.
Use Case: Ideal for small applications or caching where simplicity and low latency are priorities.

Clustered Redis

Description: Multiple nodes sharing the dataset, with data sharded across 16,384 slots and optional replication for redundancy.
Advantages: Scales horizontally by adding nodes and handles larger datasets and traffic.
Limitations: Introduces complexity in setup and management, along with potential network overhead between nodes.
Use Case: Suited for large-scale applications requiring fault tolerance, distributed data, and sustained performance under heavy load.

Testing Configurations and Exploring Redis Capabilities

To assess the automated setup script and probe Redis’ clustering features, I conducted targeted tests comparing throughput and resilience across different configurations. The experiments utilized containerized Redis instances. Below are the setups, methodologies, and findings:

Standalone vs. Clustered Throughput

Configuration: A standalone Redis instance (single container) versus a 6-node clustered setup (three master containers, 1 replica each).
Test: Executed 1,000,000 write operations (SET keyN valueN) on each setup using a benchmarking script, measuring completion time and throughput (operations per second).

Vertx:
Standalone mode: 7.22 seconds with 138k/sec throughput
Clustered mode: 13.33 seconds and 75k/sec throughput

PHP:
Standalone: 6.7 seconds and 150k/sec throughput
Clustered mode: 13.72 seconds and 73k/sec throughput

Outcome

The standalone instance achieved higher throughput for small datasets due to zero network overhead, completing the writes faster. The clustered setup, while slower per operation due to slot hashing and inter-node communication, demonstrated inferior performance for 1M operations.
Having investigated a bit more on the topic it makes sense to set up clustered Redis if the volume of operations reaches around 100k or the data doesn’t fit into memory of a single instance. Otherwise standalone Redis provides better performance. That calculations could also be influenced by other factors like network latency where cluster can parallelize operations. However load for reads can also be distributed with replication which is possible in both standalone and cluster setups.
Insight: Standalone Redis excels for low-latency, single-node workloads, whereas clustering provides scalability for larger datasets and higher concurrency.

Judgement in software development

Wed, 12 Mar 2025 12:39:16 +0000

Judgment in software development is clutch—it’s what separates the code monkeys from the architects. (Grok)

Role of judgement in software development

In software development, judgment is about anticipating more than just today’s requirements—it’s about preparing for tomorrow’s unpredictability. You’re coding not only for current specifications but also for future challenges: will the system scale under load, will it fail unexpectedly, or will it create problems later? This foresight requires balancing intuition with analysis, much like strategizing in a game where the next move isn’t fully visible. Development involves a constant tension between order—such as clean code and design patterns—and chaos, driven by deadlines, bugs, and changing requirements. Judgment provides the balance, guiding you to enforce structure when it’s needed and to tolerate imperfection when time or resources demand it, adapting to uncertainty rather than resisting it.

Judgment goes beyond technical decisions; it reflects the developer’s character. It’s not just about expertise but also about personal qualities—patience to refactor a messy module, humility to recognize a flawed approach, and courage to challenge impractical demands. The code you write mirrors these traits, revealing both your strengths and limitations through the choices you make. Whether it’s opting for a quick fix or investing in a robust solution, these decisions shape not only the software but also how you grow as a professional, tying technical skill to individual integrity.

Beyond numbers

In software development, relying solely on metrics can mislead, but avoiding them entirely risks paralysis. For instance, skipping numbers on testing coverage might leave a codebase with no clear standards, while obsessing over a metric like cyclomatic complexity can waste time on stable, unchanged classes that don’t need refactoring. It’s better to have an imperfect measure, like tracking bug frequency over weeks, than no measure at all, as this reveals whether a system truly “works” or just limps along with constant fixes. Numbers provide a starting point, but without judgment to interpret them and adjust for deteriorating systems or shifting priorities, they lose their value, leaving decisions to navigate the gray area of “it depends”.

Judgment also helps developers separate a task’s inherent business complexity from accidental complexity introduced by technical choices. For example, a payment processing feature might be complex due to regulatory requirements—unavoidable business rules—but poor judgment, like overengineering with an unneeded microservices split, can pile on accidental complexity, such as latency from extra network calls or debugging across services. Metrics like deployment time or error rates can hint at this, but only judgment discerns whether the numbers reflect the task’s nature or self-inflicted issues. By logging decisions—like choosing a simpler stack—and reviewing outcomes, developers can refine their ability to minimize unnecessary complexity, focusing effort where the business demands it most.

Judgement calls

Every day, developers face judgment calls that weave together trade-offs and team dynamics. Prioritizing speed, quality, and cost means knowing when to release a functional feature rather than perfecting it endlessly—for instance, deciding whether to optimize an algorithm now or wait until it slows the system. It’s also about understanding the team, choosing tools like a framework based on their skills and the company’s existing investments, rather than chasing the latest trend. Debugging often relies on instinct—like suspecting a race condition—when logs alone aren’t enough, while scope creep demands the discipline to say “no” or “later” to keep the project on track. Tech debt, meanwhile, is a calculated risk: judgment determines when a shortcut makes sense for a deadline and when it’s wiser to address it early, preventing a cascade of issues later. Below examples highlight the daily decisions developers navigate worldwide.

Framework Adoption

When building a web app, Next.js offers efficient React rendering and long-term scalability, but if your team consists mostly of junior developers skilled in vanilla JavaScript and jQuery, judgment is key. You must decide between adopting Next.js, which requires a learning curve, or using Express.js, which aligns with their current abilities. A practical choice might be to select Express.js for faster delivery and plan to train the team on Next.js later, balancing immediate needs with future growth.

Microservices or Monolith

Splitting a legacy app into microservices allows independent deployments and improved scaling, but with a team of five developers—only two familiar with Docker—judgment favors a monolith. A monolith is simpler to manage with limited operational skills, keeps latency lower by avoiding network calls between services, and reduces code complexity since there’s no need to coordinate multiple interdependent components. Pushing microservices could overburden the team, introducing challenges like Kubernetes configuration they can’t handle, along with higher latency from service-to-service communication and increased complexity in debugging distributed logic.

Testing Depth

Deciding between unit tests for every function or integration tests for major workflows depends on team capability. If your team excels at test-driven development, you might aim for 80%+ unit test coverage to catch edge cases early. However, if many struggle with writing mocks, judgment favors integration tests—covering fewer scenarios with broader scope—to suit their skills, even if it means some unit-level issues might slip through.

Legacy Code Overhaul

Faced with a tangled PHP codebase—tightly coupled and untested—a full rewrite in Go might seem appealing, but judgment considers the team: one developer knows Go, while three are experienced in PHP. Rewriting would delay progress significantly, so a better approach is incremental refactoring—adding tests and decoupling modules—leveraging their PHP expertise while gradually improving the code’s structure.

Caching Strategy

Redis could accelerate slow database queries for your API, but if only one busy team member has experience with it, judgment favors a simpler in-memory cache, like a Kotlin HashMap. Though less efficient than Redis, it’s faster for the team to implement, meeting current needs without overtaxing their limited expertise, while leaving space to transition to a more robust solution later.

Deadline-Driven Shortcuts

The sprint’s closing, and the user authentication module still relies on a temporary database table. Do you leave it as-is to meet the deadline, or push the sprint to migrate it to a permanent schema? If your team has a strong track record of addressing technical debt post-launch, judgment opts for shipping now, confident they’ll refactor later. With a less reliable team, delaying is safer—otherwise, the makeshift table lingers, risking data inconsistencies down the line.

Improving judgement

1. Analyze Past Decisions with Metrics
After a project, measure the results of your choices. For example, if you chose a monolith over microservices, track deployment frequency (once daily vs. hourly) and bug rates (5% higher due to coupling) using tools like Jira or Sentry. Numbers make the impact clear, like a 20% slowdown from skipping Redis caching, helping you learn what works. Review quarterly.
2. Shadow the Masters
Work with an experienced developer on a complex task, such as optimizing a query, and study their reasoning—like why they picked a JOIN over a subquery. You’ll pick up their decision-making process which builds your own judgment.
3. Use Experimental Builds for Data
Build a small system twice—like a CRUD API with Flask and then Spring Boot—and compare build time (4 hours vs. 8 hours) and scalability (100 requests/second vs. 300 requests/second). Testing both options gives you concrete data, so you can compare metrics, not feelings.
4. Formalize Judgment with Decision Criteria
Before deciding, like choosing PostgreSQL over MongoDB, list factors—schema structure, team experience (SQL: 80%, NoSQL: 20%), query needs (OLTP vs. OLAP)—and score them, then check results later with post-mortems. A clear score (Postgres: 85, Mongo: 60) helps you make reasoned choices.
5. Broaden Technical Exposure
Learn new tools on a regular basis, such as Kafka for event streams, and build a simple pipeline (1000 messages, 500 messages/second throughput) to understand its trade-offs compared to options like RabbitMQ. Knowing more tools helps you make informed decisions.
6. Know Your Team
Before starting a project, document each team member’s experience with key technologies and review how those skills align with the work, such as a backend-focused sprint requiring strong database or API expertise. After, check metrics like code review turnaround, feature delivery velocity, or tech debt tickets to see if gaps affected progress. Recognizing that skill gaps bogged down tasks helps you adjust future plans, like pairing novices with experts.
7. Iterate Under Pressure
In a short deadline, like 48 hours, pick a solution—such as in-memory caching—deploy it, measure the result (latency from 200ms to 50ms), and adjust if needed (e.g., memory errors). Working under constraints teaches you to balance speed and quality effectively. Document the fix later.
8. Read The Classics
Study foundational books like “Design Patterns” or “Refactoring” which offer proven principles to improve your decision-making skills. Apply a concept—such as replacing a switch statement with polymorphism—and measure the impact, like reducing cyclomatic complexity from 15 to 8. This gives you concrete data to understand why these practices matter

Software development is a gamble

Real-world decisions, like choosing in-memory caching over Redis, are gambles. Judgment grows when you make a choice and act, not when you overanalyze. Software is never truly “done”—it’s a living system shaped by human decisions. Good judgment means accepting imperfection, understanding that every line of code balances the ideal against the practical. You won’t have all the information upfront, so select a caching strategy, deploy it, and refine it later. Focus on reviewing actual results, not theoretical possibilities, and avoid stalling in pursuit of perfection.

Notes from Docker Up & Running

Sat, 08 Mar 2025 22:02:13 +0000

Docker Up & Running

Recently I was taking a deep dive into the workings of Docker with the help of Docker: Up & Running. These notes list useful commands and info for later use, covering ways to handle, check, and tune containers and images.

Monitoring and Stats

docker stats <container-name>

This one shows a live feed of resource usage stats for running containers, like CPU percentage, memory use, and network I/O. It acts as a real-time performance checker in the terminal, highlighting resource demands or spikes—handy for keeping an eye on container health without extra tools. The output refreshes constantly to track trends, and pairing it with glances gives a broader view of system and Docker stats together.

docker top <container-name>
Running this lists active processes inside a container, similar to a ps command on a standard server, showing details like PID, user, and the command in use. It’s a quick way to check what’s running inside, spotting any odd processes without needing to log into the container, offering a direct look at operations through the Docker API.

docker system events
This command streams live events from the Docker daemon, such as container starts, stops, or exec calls, providing a real-time log of system activity. It’s useful for tracking what’s happening or catching potential security issues, like unexpected exec commands, by leaving it running in a terminal to watch events as they occur.

Searching and Exploring Images

docker search <term>
This one queries Docker Hub or another registry for images matching the term, returning names, descriptions, star counts, and official status flags. It’s a simple way to browse available images without downloading, showing how the client reaches out through the daemon to external services, all from the terminal.

Inspecting Containers

docker container diff <container-name>
This command lists filesystem changes in a container since it started, marking files as A (added), C (changed), or D (deleted)—like a change tracker. It shows what’s been altered, such as new logs or updated files, making it easy to debug filesystem effects without entering the container, pulling data straight from the daemon.

cd /var/lib/docker/containers/<hash>
Navigating to /var/lib/docker/containers/<hash> (where <hash> is a container’s ID from docker ps -a) is a shell step, not a Docker command, reaching the host directory where the daemon keeps container data like logs and configs. It gives a raw look at a container’s files, letting you check logs or configs to troubleshoot issues, skipping container entry.

docker inspect <container-name>
This command retrieves comprehensive details about a container, including environment variables, log driver configuration (e.g., blocking or non-blocking), and other attributes, output as a single JSON object. This facilitates an in-depth analysis of a container’s setup, proving valuable insights for troubleshooting.

docker service inspect <service-name> --pretty
This command provides detailed information about a Docker service, such as those managed in Swarm mode, formatted in a human-readable structure. This offers an efficient method to examine service configuration and scaling parameters, retrieving data directly from the daemon in a clear, accessible presentation without requiring additional interpretation.

Image History and Optimization

docker image history
This one lists all build layers of an image, with full commands, sizes, and timestamps, helping figure out why an image might be too big. It shows Docker’s layers stack up—deleting files doesn’t shrink them, but multi-stage builds can—making it clear how to trim images for less network strain.

System and Context Insights

docker context list
Running this spits out a list of all configured Docker contexts—like local or remote daemons you’ve set up to work with. It’s a quick way to see which server the CLI is talking to, keeping things straight when juggling multiple environments, all without digging through config files.

docker system info
This command dumps a bunch of useful stats about the Docker setup, like how many containers are running, total images, and system details. It’s a handy snapshot to check the daemon’s workload or see what’s chewing up space, delivered straight to the terminal.

Daemon Configuration and Metrics

Metrics Endpoint Config

Adding {"experimental": true, "metrics-addr": "0.0.0.0:9323"} to /etc/docker/daemon.json sets the daemon to share metrics on port 9323, accessible via the API for detailed runtime data. Using Prometheus in a container with --network host, you can pull stats like CPU or memory use, giving a close look at how the daemon manages container resources, useful for performance checks.

Logs

To configure the maximum log size for Docker containers, you can use the –log-opt flag with the max-size option when running a container. For example, to set the maximum log size to 10 megabytes, you would use the following command:

docker run --log-opt max-size=10m my-app:latest
Alternatively, you can configure the logging options in the docker-compose.yml file under the logging section for a service. Here is an example configuration:

    logging:
      options:
        max-size: "10m"
        max-file: "5"

To apply these settings globally, you can set the log-opts in the /etc/docker/daemon.json file. Here is an example configuration:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

After making changes to the daemon.json file, you need to restart the Docker service for the changes to take effect.

Existing containers will not automatically adopt the new logging configuration unless they are recreated. To update the logging driver for an existing container, you would need to stop and recreate the container with the desired logging options.

Coding challenge – HTML web server in PHP

Sun, 02 Mar 2025 16:58:44 +0000

I wanted to learn a bit more about concurrency and network programming, so I built two TCP socket servers in PHP as a coding challenge. It’s a practical way to practice and get a grip on managing multiple connections at once. I’m not going for anything complex—just experimenting and picking up the basics step by step. The code can be found in my repository: github.com/lzag/webserver

I found it quite useful to go through the Beej’s guide to network programming to understand the inner workings of sockets programming and have a better idea of how the PHP wrappers work underneath. I added comments in the code linking to relevant sections of the guide. Another helpful resource is Mastering Swoole PHP, which explains concurrency concepts and includes some network programming examples.

To run the server I built a Docker image with the necessary extensions:
PHP Parallel extension
PHP Sockets extension
The former contains PHP wrappers around socket system calls and the latter is to achieve parallel execution.

I also added a Locust container to the composee file with some basic configuration that allows me to test the performance of the server.

There are 2 versions of the server - single and multi-threaded.

Single threaded server

First, I’m building a non-blocking single threaded socket server in PHP. I’m setting up a TCP server that listens on port 9090, manages multiple client connections concurrently, and handles incoming data asynchronously. This setup lets me explore concurrency concepts like non-blocking I/O and signal handling, managing multiple clients without relying on threads or forks.

I create a TCP socket with socket_create(), using IPv4 (AF_INET) and a streaming protocol (SOCK_STREAM).

$socket = socket_create(AF_INET, SOCK_STREAM, 0);

I bind the socket to 0.0.0.0 on port 9090, allowing connections from any interface. Any error can be retrieved using a combination of socket_last_erorr and socket_strerorr.

if (socket_bind($socket, '0.0.0.0', 9090) === false) {
    echo "socket_bind() failed: reason: " . socket_strerror(socket_last_error($socket)) . "\n";
    exit;
}

I set the socket to listen for incoming connections using socket_listen().

if (socket_listen($socket) === false) {
    echo "socket_listen() failed: reason: " . socket_strerror(socket_last_error($socket)) . "\n";
    exit;
}

I switch the socket to non-blocking mode with socket_set_nonblock() so it doesn’t hang on I/O operations and maintain an array of clients to track all active client sockets.

socket_set_nonblock($socket);
$clients = [];

I register a SIGINT signal handler with pcntl_signal() to shut down gracefully when I interrupt it (e.g., with Ctrl+C). Normally Docker compose would send SIGTERM, but I configure it to send SIGINT like the Linux command line.

pcntl_signal(SIGINT, function () use (&$running) {
    $running = false;
    echo "Caught signal, shutting down...\n";
});

I run an infinite loop until I stop it:
I check for new connections with socket_accept() and add them to my $clients array. I loop through connected clients, reading their data with socket_read() in non-blocking mode. I process their input, sending back an HTTP 200 OK response, and close their connection if they send “quit. I dispatch pending signals with pcntl_signal_dispatch() and add a tiny delay (usleep(1)) to avoid overloading the CPU.

while ($running) {
    if ($conn = socket_accept($socket)) {
        if ($conn !== false) {
            echo "New connection is accepted\n";
            socket_set_nonblock($conn);
            $clients[] = $conn;
            $id = sizeof($clients) - 1;
            echo "Connection #[$id] is connected\n";
        }
    }
    if (count($clients)) {
        foreach ($clients as $id => $conn) {
            if ($input = socket_read($conn, 512)) {
                $input = trim($input);
                echo "Connection #[$id] input: $input\n";
                $ack = "HTTP/1.0 200 OK\r\nContent-Length: 0\r\n\r\n";
                socket_write($conn, $ack, strlen($ack));
                if ($input == 'quit') {
                    socket_close($conn);
                    unset($clients[$id]);
                    echo "Connection #[$id] is disconnected\n";
                }
            }
        }
    }
    pcntl_signal_dispatch();
    usleep(1);
}

When I shut it down, I close all active client connections in $clients and terminate the listening socket.

Multi-threaded server

In the second version I’ve upgraded my original non-blocking socket server to use multiple threads with PHP’s parallel extension. While the core setup (listening on port 9090, non-blocking I/O, and signal handling) remains similar, I’ve introduced significant changes to handle requests concurrently with a thread pool and processing of HTTP-like inputs.

I create a thread pool with 10 threads using parallel\Runtime to process client requests in parallel, replacing the single-threaded approach of the first version.

$poolSize = 10;
$pool = [];
for ($i = 0; $i < $poolSize; $i++) {
    $pool[] = new Runtime();
}

I define a closure to handle HTTP requests, parsing the method, path, and protocol, serving files from /var/www/html, and returning formatted HTTP responses (e.g., 200 OK, 404 Not Found), instead of just echoing input and sending a static 200 OK. Sleep is there to simulate processing time.

$processRequest = function ($input): string {
    sleep(1);
    $docRoot = "/var/www/html";
    @list($method, $path, $protocol) = explode(' ', $input, 3);
    if (!isset($method) || !isset($path) || !isset($protocol)) {
        return "HTTP/1.1 400 Bad Request\r\n\r\n";
    }
    if ($path == '/') {
        $path = $docRoot . "/" . "page.html";
    } else {
        $path = $docRoot . $path;
    }
    if (!file_exists($path)) {
        $response = "HTTP/1.1 404 Not Found\r\nRequested path: $path\r\n\r\n";
    } else {
        $responseBody = file_get_contents($path);
        $response = "HTTP/1.1 200 OK\r\n";
        $response .= "Content-Type: text/html\r\n";
        $response .= "Content-Length: " . strlen($responseBody);
        $response .= "\r\n\r\n";
        $response .= $responseBody;
        $response .= "\r\n\r\n";
    }
    return $response;
};

I limit the number of concurrent clients to 100, rejecting new connections with a 503 Service Unavailable response if the limit is reached, unlike the first version which accepted unlimited clients.

if (count($clients) >= 100) {
    $refusal = "HTTP/1.1 503 Service Unavailable\r\nContent-Length: 20\r\n\r\nServer is full, sorry";
    socket_write($incoming_conn, $refusal, strlen($refusal));
    socket_close($incoming_conn);
    echo "Refused a connection - client limit (100) reached\n";
} else {
    echo "Connection is accepted\n";
    $clients[] = $incoming_conn;
}

I replace the simple polling loop with socket_select() to efficiently monitor multiple sockets (the server socket and clients) for readability, improving scalability over the first version’s manual checks.

$readfds = array_merge($clients, [$socket]);
$writefds = NULL;
$errorfds = [];
$select = socket_select($readfds, $writefds, $errorfds, 0, 1000);

I offload request processing to the thread pool by submitting tasks to the thread pool with run(), storing the results in an array, instead of handling requests inline like in the first version.

if ($input = socket_read($conn, 8192)) {
    $input = trim($input);
    $futures[$connkey] = $pool[array_rand($pool)]->run($processRequest, [$input]);
}

I check completed futures in a separate loop, write their responses to clients, and close connections immediately after, unlike the first version where I kept connections open until a “quit” command.

foreach ($futures as $key => $future) {
    if ($future->done()) {
        $response = $future->value();
        $conn = $clients[$key];
        echo "Conn #[$key] writing response...\n";
        socket_write($conn, $response, strlen($response));
        unset($futures[$key]);
        socket_close($conn);
        unset($clients[$key]);
    }
}

I modify the main loop to continue running until both $running is false and all $futures are resolved, ensuring all threaded tasks complete before shutdown, unlike the first version which stopped immediately.

while ($running || !empty($futures)) {
    // ... (rest of the loop)
}

These changes shift my server from a basic, single-threaded echo service to a multi-threaded, HTTP-capable server with connection limits and efficient I/O handling, leveraging parallelism for better performance.

Testing in Locust shows the difference in peformance: the first version only can achieve 1 RPS vs the other close to 10 RPS. In real life we would avoid blocking the tread, but for this simple exercise that’s good enough.

Notes from Grokking Concurrency

Sat, 01 Mar 2025 14:05:16 +0000

Introduction

I wrote this summary of Grokking Concurrency to take notes and help me remember the main ideas about parallel programming. The book covers things like multitasking and synchronization with examples and explanations that I want to understand better. It’s a way for me to keep track of what I’m learning so I can use it later.

Sequential Programming: Pros and Cons

Sequential programming refers to executing tasks one after another in a single-threaded, linear fashion.

Pros:

Its simplicity makes it easy to design, implement, and debug. The straightforward flow eliminates the need to manage complex interactions between tasks.

Cons:

Limited Scalability: Sequential programs can only scale vertically (e.g., by upgrading to a faster CPU), which has physical and economic limits. Horizontal scaling (adding more processors) isn’t possible without parallelism.
Resource Inefficiency: System resources, such as CPU cores, memory, and I/O, are underutilized because tasks wait idly for others to complete, introducing significant overhead.

Transitioning to Parallelism

To process tasks independently, they must be inherently separable—i.e., their execution shouldn’t depend on the outcome of other tasks. While any parallel task can theoretically be executed sequentially, the reverse isn’t always true. Some tasks can be partially parallelized, but this introduces synchronization points—moments where parallel threads must coordinate. Synchronization adds overhead (e.g., locking mechanisms or waiting times), which can negate the performance benefits of parallelization if not carefully managed. In extreme cases, the overhead may outweigh the gains, making parallelization impractical.

Amdahl’s Law

Amdahl’s Law quantifies the theoretical speedup of a program when parallelized. It states that the maximum speedup is limited by the fraction of the code that must remain sequential. Mathematically, if ( P ) is the proportion of a program that can be parallelized and ( N ) is the number of processors, the speedup ( S ) is:

S = 1 / ((1 - P) + (P / N))

Where:
S = Speedup
P = Fraction of program that can be parallelized
N = Number of processors
Thus, a parallel program’s performance is ultimately bottlenecked by its slowest sequential component. For example, if 20% of a program is sequential (1 - P = 0.2), the maximum speedup, even with infinite processors, is 5x.

CPU Execution Cycle

Understanding how CPUs execute instructions is key to concurrency:
Fetch: The CPU retrieves an instruction from the cache (or memory, if not cached).
Decode: The instruction is interpreted by the control unit.
Execute: The Arithmetic Logic Unit (ALU) performs the computation or operation.
Store: Results are written back to cache or memory.
Fetching from main memory is significantly slower than from cache (often by orders of magnitude), so modern CPUs use multiple cache levels (L1, L2, L3) to minimize latency and keep the processor busy. Efficient concurrency design leverages this by optimizing data locality and minimizing cache misses.

CPUs vs. GPUs

CPUs: Designed for general-purpose computing, CPUs use a MIMD (Multiple Instruction, Multiple Data) architecture, supporting a wide range of instructions with moderate parallelism (e.g., via multi-core designs). They excel at complex, sequential tasks.
GPUs: Built for parallel processing, GPUs use a SIMD (Single Instruction, Multiple Data) architecture, executing the same instruction across many data points simultaneously. They’re ideal for tasks like graphics rendering or matrix computations but are limited to a narrower instruction set.

Process States

A process transitions through distinct states during its lifecycle:
Created: The process is initialized in memory, allocated resources, and prepared for execution.
Ready: It resides in a queue, waiting for CPU assignment.
Running: The process is actively executing on the CPU.
Terminated: Execution completes, and the operating system frees associated resources (e.g., memory, file descriptors).

Operating System Abstractions: POSIX Threads (Pthreads)

The OS provides abstractions to manage program execution:
File Descriptors: In Unix-like systems, these are handles to I/O resources (files, pipes, sockets). They enable safe, efficient interaction between processes and the system.
Threads: Lightweight units of execution within a process. Unlike processes, threads share the same memory address space but maintain their own stack and instruction stream. Introduced to reduce the overhead of inter-process communication (IPC), threads are scheduled independently by the OS.
Advantages: Lower memory usage and faster context switching compared to processes.
Disadvantages: Shared memory necessitates synchronization to prevent data corruption, adding complexity.

Inter-Process Communication (IPC)

IPC enables data exchange between processes. Common methods include:
Shared Memory: Processes access a common memory region. It’s fast but requires synchronization to avoid race conditions.
Message Passing: The OS manages communication channels, copying data between user and kernel space via system calls. This is slower than shared memory due to the overhead of data copying.
Pipes: A unidirectional communication channel implemented as a file descriptor. Pipes block until data is available, making them simple but limited. This is a fairly good explanation of the working of pipes.
Message Queues: Structured buffers for sending/receiving messages, offering more flexibility than pipes.
Unix Domain Sockets: Similar to network sockets but optimized for local communication, supporting both stream and datagram modes.

Multitasking

Multitasking enables concurrent execution of multiple tasks, but it’s a runtime feature managed by the OS, not the hardware. Applications are classified as:
CPU-Bound: Limited by processing power (e.g., computations).
I/O-Bound: Limited by input/output operations (e.g., disk or network access).

Types of multitasking:

Preemptive: The OS scheduler allocates CPU time slices to tasks, interrupting them as needed for fairness and responsiveness.
Cooperative: Tasks voluntarily yield control, relying on programmer discipline (less common in modern systems due to reliability issues).

Task Decomposition

Task decomposition breaks an application into independent, parallelizable units based on functionality:

Identify separable tasks.
Create a dependency graph to map task relationships.
Apply techniques:
Pipeline Pattern: Tasks process data in stages, like an assembly line.
Data Decomposition: Split data into chunks processed independently.
Fork/Join, Map/Reduce, Map Patterns: Tasks execute autonomously with no side effects, then combine results.
Determine granularity: Fine-grained decomposition increases parallelism but also communication overhead—a critical trade-off.

Synchronization and Race Conditions

A race condition occurs when multiple tasks access a shared resource, and the outcome depends on execution order. Synchronization ensures orderly access:
Mutexes: Locks that only the owning thread can release, protecting critical sections.
Semaphores: Generalized locks with a counter, allowing multiple resource instances. They can be manipulated by different processes.
Atomic Operations: Operations (e.g., incrementing a variable) that complete indivisibly, invisible to other threads in a partial state. An atomic operation ensures a memory update is uninterrupted, preventing race conditions in concurrent environments

Deadlocks

Deadlocks arise when tasks wait indefinitely for resources held by each other. Solutions include:
Arbitrator: A central authority resolves conflicts.
Resource Hierarchy: Assigns an order to resource acquisition, preventing circular waits.

What I Liked About the Book

Grokking Concurrency does a great job of making tricky ideas easy to understand with clear explanations and hands-on code examples. I really liked how the detailed code examples showed every concept in the book—they made things really clear for me. It teaches the basics of multitasking, keeping things in sync, and designing for parallel work, which are super important for today’s programming.

Setting up MySQL replication with Docker

Sat, 01 Mar 2025 02:41:39 +0000

Introduction

Setting up MySQL replication is a powerful way to enhance the reliability, scalability, and performance of your database system. In simple terms, MySQL replication is a process where data from one MySQL database server (known as the primary or master) is automatically copied to one or more additional servers (called replicas or slaves). This creates a live backup of your data, ensuring that if the primary server fails, a replica can step in with minimal downtime. Beyond redundancy, replication also allows you to distribute read-heavy workloads across multiple servers, improving efficiency and speed for applications that rely on quick data access.

Importance of DB replication

Why is this important? In today’s world, where uptime and fast performance are non-negotiable, replication provides a safety net against hardware failures, network issues, or unexpected outages. It’s a cornerstone for businesses running critical applications—like e-commerce platforms, content management systems, or analytics tools—that can’t afford to lose data or leave users waiting. Plus, with a well-configured setup, you can scale your database infrastructure as your needs grow, all while keeping everything in sync. In this post, we’ll walk through the steps to set up MySQL replication, so you can unlock these benefits for your own projects.

Inner works of MySQL replication

MySQL replication works by copying and syncing data from a primary server (master) to one or more replica servers (slaves). The primary server logs all changes—like inserts, updates, or deletes—in a binary log. The replicas then read this log and apply those changes to their own databases in near real-time. A key component, the I/O thread, pulls the log data from the primary, while an SQL thread on the replica executes the updates locally. This setup keeps the data consistent across servers, enabling redundancy and load balancing.

Setting up replication with Docker Compose

The source and each replica must be configured with a unique ID (using the server_id system variable).
The binlog_do_db setting in MySQL controls which databases are included in the binary log on the primary server during replication. When you specify a database with this option (e.g., binlog_do_db=my_database), only changes to that specific database—like table updates or inserts—are recorded in the binary log.
On the replicata there are settings that can limit the statements to specifc dbs or tables:

replicate-do-db: Tells replication SQL thread to restrict replication to specified database.
replicate-do-table: Tells replication SQL thread to restrict replication to specified table.
replicate-ignore-db: Tells replication SQL thread not to replicate to specified database.
replicate-ignore-table: Tells replication SQL thread not to replicate to specified table.

In addition, each replica must be configured with information about the source’s host name, log file name, and position within that file.

Here is a repo where I set up replication using Docker. Just by running <code>docker compose up you can have a MySQL db with a replica for testing: github.com/lzag/docker-mysql-master-slave

Thoughts on software architecture and decision-making

Sat, 04 Jan 2025 14:05:17 +0000

Recently, I read the book Software Architecture and Decision-Making by Srinath Perera. It contains a lot of content that is available in other software architecture books but offers some unique angles on certain topics.

I think the author is right to point out that you need to design deeply things that are hard to change, especially database schemas. For businesses, data is gold, and making changes to it comes with enormous risks. Therefore, changes are very rare and done only when absolutely necessary. The same applies to customer-facing APIs (or even internal APIs). These are usually contracts that other parts of the system adhere to, so changes to them come at a big cost. Facebook was famous for following the adage “move fast and break things,” but after they annoyed developers with constantly breaking APIs, they changed their attitude to “Move fast with stable infrastructure.”

The author also emphasizes the business environment where the architect operates. The skill level of the team is important when designing a system, as well as the required time to market.

Often, naive implementations are the best to get the business going. The architect must be the person who, using their judgment, takes calculated risks and assumes responsibility for the outcome; otherwise, the whole team becomes blocked.

Something that I haven’t encountered previously is a chapter discussing mental models explaining performance. I strongly believe that good mental models are essential in enabling good judgment when making software decisions, so seeing a chapter dedicated to mental models explaining performance was great. I’ve been spending a great deal of time trying to understand how computers work in order to build better and more performant systems.

Another useful part is the one talking about the importance of testing and enforcing behavior from a management perspective. It’s important for a designer or manager to understand the team’s motivations and design processes in a way that makes it hard to avoid writing tests. Often in startups, this is omitted because they focus on developing as fast as possible. The role of a designer is to understand the importance of this and create incentives that make people want to write tests; otherwise, it might be easier to pass on the buck. Demanding excellence is also very important. Some companies don’t enforce excellence standards because they cause stress to the developers. We all make mistakes, so we shouldn’t blame the developers if something happens, but it’s important to enforce certain standards and strive to raise them for the team. It’s vital to remove anything that the developer doesn’t control from the equation, but they need to be held responsible if they don’t meet certain diligence standards. Otherwise, there is no incentive for the rest of the team to hold themselves to a higher standard.

One of the big issues in tech is mindlessly copying architectures from popular companies like Uber or Netflix without considering the actual problems that should be solved, often resulting in building antipatterns like distributed monoliths. The book points this out in the microservices chapter and offers alternatives like repos-based teams. The same goes for building event-driven and asynchronous software. That requires developers to really understand how to write such systems, and without that knowledge, the systems produced might have additional complexity without the actual performance boosts.

Overall, the book gives a very good overview of many leadership considerations when leading software projects without going too much into technical details, so I would highly recommend it.

hello world

Mon, 01 Jan 2024 00:00:00 +0000

New site, new stack. Ditched WordPress for Hugo + Cloudflare Pages.

Posts are Markdown files, resume is built from LaTeX, everything is in git. No database, no PHP, no plugin updates at 2am.

Removing bloatware from Android phones

Sun, 20 Feb 2022 18:55:42 +0000

Most Android phones come with many preinstalled apps that you might not ever use, but they do take up valuable space and clutter the interface. If you try to uninstall them the way you would any other apps you will find that it’s not possible. So is there a way to get rid of them?

It’s actually not too complicted to uninstall them using the right tool which are available for every major operating system.

Keep in mind that while you can reinstall the packages if you change your mind later, but uninstalling them will wipe out all the application data and might sometimes interfere with correct functionning of other applications. Please make sure that you back up all your important data before starting to play around

To install the necessary tools on Linux you can use the command:

sudo apt update && sudo apt install android-tools-adb android-tools-fastboot

On Windows and MacOS you can use brew or choco package managers to install the tools.
Windows:

choco install adb

MacOS:

brew install android-platform-tools

Now that you have the tools on your computer you need to enable USB debugging on your phone. First, go to Settings -> About Phone and tap on the build number seven times until you unlock the developer mode. Then go to Settings -> System -> Advanced -> Developer Options and enable USB debugging.

When you connect your phone to the PC via USB you should get a notification popup to confirm the incoming debugging connection from your PC. Your need to confirm it in order to proceed.

To verify that the device is connected you can use the command:

adb devices

Now your can list all the packages installed for your user.

adb shell pm list packages --user 0

You can use the grep command to filter only specific types of packages, for example:

adb shell pm list packages --user 0 | grep google

This would look for google pacakges. On Xiaomi phones you can search for ‘miui’ which should return their preinstalled packages.
If you would like to remove the package for a user (Gmail app in this case):

adb shell pm uninstall -k --user 0 com.google.android.gm

To find out the name of the package you can go the the play store page of an app and look at the URL. You should find the name of the package in the id parameter. This would be the link to the Gmail app:

https://play.google.com/store/apps/details?id=com.google.android.gm

To reinstall the app:

adb shell cmd package install-existing com.google.android.gm

You can also choose to only disable the apple for the user:

adb shell pm disable-user --user 0 com.google.android.gm

To enable:

adb shell pm enable --user 0 com.google.android.gm

Overview of PHP code quality tools

Sat, 18 Dec 2021 21:05:15 +0000

General

Below tools provide a comprehensive set of rules and analyse the code from different angles.

PHPStan

PHPStan focuses on finding errors in your code without actually running it. It catches whole classes of bugs even before you write tests for the code. It moves PHP closer to compiled languages in the sense that the correctness of each line of the code can be checked before you run the actual line.

Psalm

Very similar to PHPStan. Also can specify levels and there is probably a lot of overlap in terms of analysis, but could also be a useful tool.

PHP Mess Detector

It is a spin-off project of PHP Depend and aims to be a PHP equivalent of the well known Java tool PMD. PHPMD can be seen as an user friendly and easy to configure frontend for the raw metrics measured by PHP Depend.

PHPCodeSniffer

Detects violations of the defined coding standards and helps to keep the code base clean and easy to read.

PHPMetrics

PhpMetrics provides metrics about PHP project and classes, with beautiful and readable HTML report.

PHP Insights

The perfect starting point to analyse the code quality of your PHP projects.

Exakat

Exakat looks very promising and comprehensive, but the biggest drawback is buggy documentation and small community. Generates reports in various formats, including HTML and analyses many areas like security or compatibility with specific versions of PHP.

Specialised

Below deal with specific areas of code quality.

PHP Copy/Paste Detector (PHPCPD)

phpcpd is a Copy/Paste Detector (CPD) for PHP code. Does what the name says. It discovers where simple copy paste was used or a duplicate code. Makes it possible to see where it can be wrapped up into a function.

PHPMND (PHP Magic Number Detector)

This tools is pretty specific: it can help you to find magic numbers your code.

Enlightn Security Checker

PHP Assumptions

churn-php

Helps discover good candidates for refactoring by finding classes that change often.

Testing

These tools test the code base or improve the quality of written tests

PHPUnit

PHPUnit is a programmer-oriented testing framework for PHP.

InfectionPHP

PHP Mutation Testing Framework

Others

Rector

Rector is a CLI tool written in PHP. It can instantly upgrade old PHP code and handle automated refactorings. It’s fast and precise - changes 5000 files under minute.

SonarLint

Deptrac

PHP Parallel Lint

Phan

PHPLint

dePHPend

PhpDeprecationDetector

Analyser of PHP code to search usages of deprecated functionality in newer interpreter versions - deprecations detector. Doesn't support PHP 8.0 yet.

How does browser fingerprinting work?

Tue, 16 Mar 2021 01:46:53 +0000

What is browser fingerprinting?

Nowadays, most people are familiar with cookies - information stored in browsers that helps websites keep track of our settings and track our actions online. Many users know how to delete cookies or navigate in private mode hoping that this will grant them privacy and prevent companies from following their actions online.

However, there is a technique that allows to track the user online without the need for cookies called browser fingerprinting. One might think that just based on some general system settings it would be hard to tell computers from one another, but the blend of our systems’ settings makes them quite unique and it’s usually possible to tell them apart.

Browsers automatically send many bits of information about a machine whenever they make a petition to the website server. These technical non user-specific details usually help with communication between the browser and the server, such as serving a correct version of the website for specific browser. One of the most common pieces of information shared is the user agent. For example, user agent for a Windows 10 machine that uses Edge browser would looks something like: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246’. This string identifies the operating system and the specific version of the browser. This information is automatically sent in the headers when a website is loaded.

Other pieces of information that the browser can access are: screen size, timezone, system fonts, language, video and audio hardware available etc. Fingerprinting can also perform operations with audio or video hardware to create unique hashes. Using a protocol called WebRTC it can even be possible to discover user’s IP behind a VPN. Even though fingerprinting doesn’t contain any personally identifiable information, it allows to identify users browsing on different website without his knowledge or consent.

Can browser fingerprinting be prevented?

Most browsers can be configured to make fingerprinting much harder, but it’s hard to prevent it completely, since browsers might need access to many of the functions that make fingerprinting possible just to display websites correctly or user advanced functionalities like videochats.

If you’re interested in fingerprinting you can check out: https://coveryourtracks.eff.org/. This website will tell you which techniques can be used to identify your browser and how unique your signature is. It’s quite eye-opening to see what information about your computer is available every time you access a website and how unique it is.

Difference between virtual machines and containers

Mon, 24 Aug 2020 14:29:43 +0000

Definition

Virtualization and containerization might seek to achieve the same goals but they are different concepts.

The virtual machine loads an OS on its own it can be any type of OS that you would like. The virtual machine has its own hard drive, network interfaces etc. that are then mapped to the physical hardware of your machine. So you can easily run a Windows or macOS machine on you Ubuntu if you’d like to. Virtualization however is different from emulation, so you’re still limited to the hardware that you have available.

Containerization is the process of isolating the application dependencies into a container. It uses kernel functionalities (like control groups and namespaces) that allow for resource isolation. Since you’re not running a full guest OS you won’t be able to run a Windows container on a Linux machine and vice versa. Docker didn’t invent the concept of containers, but is the most popular solution at the moment and when people are talking about containers they usually refer to this solution.

Software required

In order to run a virtual machine on your computer you need a hypervisor. As per Wikipedia a hypervisor is computer software, firmware or hardware that creates and runs virtual machines. There are two different types of hypervisors: type-1 or native ones run directly on the computer hardware while type-2 or hosted ones run on the host OS as a process. Examples of hypervisors a Microsoft Hyper-V, VirtualBox or VMware Player.

The docker containers on the other hand are executed with the Docker engine that runs on the host OS.

Resource isolation

The virtual machines operate within a sandbox, meaning that they are completely isolated and self-contained. The containers share the operating system while virtual machines run an OS on their own. Resource isolation is one of the advantages that virtual machines have over containers.

Resources consumption

Most companies choose Docker over VMs because of much smaller resource footprint. Virtual machines might also need to have the resources assigned to them permanently while they’re operating, containers allow for more flexibility and optimization.

There is also less redundancy when using containers - you don’t have to install the full guest operating system, just components that you need for your application.

Each VM runs not just a full copy of an operating system, but a virtual copy of all the hardware that the operating system needs to run. The guest OS has its own memory management and virtual device drivers. As a result containers are smaller that VMs and have superior startup times than virtual machines. In many cases you might double the number of applications run and the speed when using containers. However, this might not be true in all situations. Current technology of virtualization reduced the overhead of CPU and memory usage and in some configuration a virtual machine might run faster than a Docker container [1]1.

The biggest advantage of containers is that you can create a lightweight, portable and above all consistent operating environment when you’re developing, testing and deploying applications.

Coding trivia 1

Sun, 09 Aug 2020 00:27:29 +0000

Random interesting stuff that I learned last week

ER diagrams

Entity Relationship diagrams are important concept in database design. They help to conceptualize the relationship between tables in a database. I was looking for a quick refresher on the ER diagrams. These two videos did the job: Part 1 and Part 2

Unix sockets

You know when you’re trying to connect to MySQL and an error shows up that the socket file is missing? That’s because in MySQL the socket connection is enabled by default on localhost. What are UNIX socket files? These are files that enable two application to communicate with each other on the same machine, without the need for networking.
Here is an in-depth explanation of how they work:
www.techdeviancy.com/uds.html
And here’s a quick video explanation:
www.youtube.com/watch?v=uXve8WYiGts
More on the details of sockets:
stackoverflow.com/questions/14919441/principle-of-unix-domain-socket-how-does-it-work

The sticky bit

In some cases you might want to set a folder to be writable to the whole world, bu limit the scope of changes that every user could do to the files. Sticky bit limits the activity in the folder so that only the owners of the files the overall directory or root user can rename or delete . This means that users won’t be able to mess with other user’s files and makes sense to set it in collaborative directories.

Adding it is easy with:
chmod +t test;

Wikipedia has a good explanation of the concept

Creating a custom keyboard layout in Linux

Sun, 14 Jun 2020 16:38:15 +0000

Sometimes you might want to modify the keyboard layout, for example to add non standard characters that you’re often using or if you’re writing in 2 different languages you might combine all their special characters in one keyboard layout. In my case I bought a keyboard that had the escape key mapped by default to the Home Page special key and I found irritating having to press the Fn key every time I needed to use the escape button.

I’m using the Lubuntu Linux distribution that uses the X keyboard extension to enable multiple keyboard layouts.

First, run the xev command and filter only the keyboard events for a clearer log:

xev -event keyboard

When pressing a key it shows the keycode that is being sent and also the symbol of they key that is interpreted. In my case, using the standard US layout, the key 180 maps to the X86HomePage. The xev tool logs both the KeyPress and KeyRelease event:

With this information we can look for the key symbol that we can use to modify the layout. In the directory “/usr/share/X11/xkb/keycodes” we can find the mapping in the evdev file.

cat /usr/share/X11/xkb/keycodes/evdev | grep 180

The output shows that the keycode 180 is mapped to the symbol :

 <I180> = 180;   // #define KEY_HOMEPAGE            172

Now, there is an easy way to map the key to the escape button:

xmodmap -e "keycode 180 = Escape"

To make the changes persistent we can save the command in the ~/.Xmodmap file

However, we than can also define the layout in the “/usr/share/X11/xkb/symbols” folder

The easiest way to go about it is to copy the one of the files that you can build upon. For example the standard US layout and then remove the contents of the layout.

One of the layouts marked with default will be the main one and it also can have other variants.

default  partial alphanumeric_keys modifier_keys
xkb_symbols "basic" {

    name[Group1]= "Custom Keyboard Layout";

    include "us(basic)";
    include "eurosign(5)";

    key <I180> {        [ Escape                        ]        };
};

In the code we’re first importing the US layout, then adding the Euro sign on the 5 button and finally mapping the I180 key to the Escape button. We can save the file as “cust” for example.

After we save the file we should enable the configuration in the rules folder base.lst file, otherwise the configuration will not be available.

We’re looking for the layout section and adding the name of the file with the description of the Layout:

! layout
  cust           Custom keyboard layout

And with this the layout should be one of the choices in the layout selector.

Setting up the fixture in PHPUnit

Sat, 06 Jun 2020 18:17:50 +0000

When running our tests we should begin with a know stage, called the fixture, and for that we might need to initialize variables etc. And after we finish the tests we might have some cleaning up to do.

If you need to have custom code run before you execute them. PHPUnit allows you to configure a specific file to be run before the test execution.

This directive in your phpunit.xml configuration file will execute the bootstrap.php fiel before the tests: :

<phpunit bootstrap="tests/bootstrap.php" >
    <!-- ... -->
</phpunit>

The bootstrap file can also be specified in the command line argument:

phpunit --bootstrap bootstrap.php

If every method is the class needs the same setup we might use provided functions to save ourselves from repeating the code.

PHPUnit has setUp() and tearDown methods that run before and after every method. We might use these for example to initialize variables.

Additionally, setUpBeforeClass() and tearDownAfterClass methods are being run before any method in the class has been executed and after all of them finish, respectively. These might be useful to setting up for example a database connection. However, one has to keep in mind that that when tests share a fixture and are not fully independent it makes them harder to debug.

Setting up WordPress development environment with Docker

Sun, 24 May 2020 23:48:50 +0000

One of the biggest challenges in web development is to have a stable development environment and assuring that the website works when deploying on different servers.

Docker helps us build isolated containers that give a this stable and predictable environment and saves many headaches when debugging our website. Thanks to the preconfigured images available on Docker Hub and the Docker compose it’s a breeze to set up a development environment.

This post assumes that you already have Docker and Docker compose isntalled on your system. On Linux Docker Compose doesn’t come installed with Docker so make sure you install it separately.

Create a directory to put the docker compose file into and then paste the following into the docker-compose file:

version: '3.3'

services:
   db:
     image: mysql:5.7
     # declaring the volumes so we have persistent data
     volumes:
       - wp_data:/var/lib/mysql
     restart: always
     environment:
       MYSQL_ROOT_PASSWORD: password
       MYSQL_DATABASE: wordpress
       MYSQL_USER: wordpress
       MYSQL_PASSWORD: wordpress

   wordpress:
     container_name: docker_wordpress_dev
     depends_on:
       - db
     # using the latest wordpress image
     # Docker will automatically look it up on Docker Hub
     image: wordpress:latest
     # binding the loopback port 8000 on the host to port 80 on the docker container
     ports:
       - "127.0.0.1:8000:80"
     restart: always
     environment:
       WORDPRESS_DB_HOST: db:3306
       WORDPRESS_DB_USER: wordpress
       WORDPRESS_DB_PASSWORD: wordpress
       WORDPRESS_DB_NAME: wordpress
# we need to declare the volumes that we will use
volumes:
    wp_data:

After running the “docker-compose up -d” command you should be able to access WordPress on by entering localhost:8000 in your browser. This way you should should be able to bring up the installation page of WordPress:

After completing the setup steps you’re ready to start working with WordPress.

If you’d like to access the source code you can the following command, which will bring you to the main folder with your WordPress installation:

docker exec -it docker_wordpress_dev bash

The database volume persists after the container is stopped. You can check that running the command “docker-compose stop” to stop the containers. Whenever you reinitialize the container you will be able to pick up where you left, so whatever changes you mad to your WordPress installation in Docker will persist. Also the database will persist in the wp_data volume.

Thanks to the persisting database volume, even if you remove the container you should be able to spin the WordPress site back up from the same image.

Downloading and installing custom Linux kernels

Tue, 19 May 2020 03:59:04 +0000

You might want to install a different kernel to which you have supplied with your current version of Linux. This might be because another version might work better with your hardware.

The Linux kernel is the core of the operating system that facilitates interaction between the hardware components and software.

You can install a different version of the kernel with the apt command:

sudo apt install linux-image

The command will present you with several options and you need to type the specific one you would like to install.

After installing the kernel you will be able to choose an additional option from the Linux boot menu.

In many cases the version you would like to install is not available for install from the command line and you would need to compile is from the source.

You need to go to to kernel.ubuntu.com/~kernel-ppa/mainline/ to download the deb packages for the version you would like to try.

After downloading the correct files for you version and architecture you need to install them with the following command and the reboot your system:

sudo dpkg -i linux*.deb

In order to be able to boot, you might need to sign the binaries, disable validation for the kernel binary or disable Secure Boot altogether in BIOS.

Setting custom PHP code sniffing rules

Fri, 15 May 2020 02:03:58 +0000

Code sniffing is the practice of checking the code for compliance with some pre-determined standards. It’s very important to run your code through a sniffer because it helps with maintenance and ensures better code quality.

The most common standards in PHP are PSR-0, PSR-1, PSR2 and PEAR. These rules might specify details such as depth and type of indentation, placement of braces, spacing, variables names etc. Depending on the standard, they might go into a lot of detail about how the structure of your code should look like.

A popular tool for sniffing code in PHP is the PHP CodeSniffer (phpcs).

When running it it you can add arguments that specify the folder with the files, ignored directories, coding standards etc. When working on different projects for different clients it might be tiresome to have to specify them differently for every sniff. Phpcs lets you create an archive that specify custom rules and arguments to use. Whenever you run phps it will first look into the current directory for the file and if found, apply the configuration.

The file should be be in an xml format. The most useful options that you can specify are:

The file tag adds files and folders to be sniffed.

<file>file.php</file>

The “exclude-pattern” tag lets you exclude specific directories or files from sniffing:

<exclude-pattern>*/tests/Core/*/*Test\.(js|css)$</exclude-pattern>

The “rule” tag lets you add ruleset by its name and also exclude specific rules if you would not like to apply them:

<rule ref="PEAR">
   <exclude name="PEAR.NamingConventions.ValidFunctionName"/>
</rule>

The arg tag lets you specify arguments that should be passed to the phpcs executable, just like the arguments from the command line. The names and acceptable values are sames as on the CLI, with a couple of arguments not permitted to be used there.

 <arg name="basepath" value="."/>

Using custom ruleset files is very useful when you would like to track the quality of your code and your projects must conform to different standard. Defining them before starting a project will surely help you produce cleaner code.

How to deploy a React app in a subdirectory on a WordPress site

Thu, 14 May 2020 11:10:45 +0000

The case here is pretty specific, but maybe somebody will be in the situation like me. I have a WordPress site and I wanted to deploy a React app to include in my portfolio and serve it from a subdirectory.

When deploying a React app in a website subdirectory there are some hurdles to overcome. First of all, WordPress redirects most requests to the main index.php file, so when you’re trying to load the app the server might direct the request there and you will get an error. To resolve this you need to edit the .htaccess file so that the requests are correctly routed. Below is the code that you should add in your access file , before the rewrite conditions of WordPress:

RewriteCond %{REQUEST_URI} ^/subdirectory/.*
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^.*$ /subdirectory/index.html [L]

If the request URL will start wit the subdirectory and is not an existing file (RewriteCond rules), the request will be routed to the main index file (RewriteRule). The [L] flag tells the server not to process any more rules - otherwise it will execute the following ones as well.

Secondly, the React Router has a handy “basename” attribute that specifies that base of the URLs.

<Router basename="/subdirectory" >                
  <Route path="/my-account" component={Form} />
 <Route path="/about" component={About} />
</Router>

If we don’t include the attribute the paths would be relative to the main directory and would break. With the baseame, they will all include the base path.

And that’s pretty much it. With those 2 simple tweaks you can make your React app work from a subdirectory on a WordPress site.

Under the URL https://lzag.dev/goodreads-app you can find a React app that I built.

What are .htaccess files?

Wed, 13 May 2020 18:39:11 +0000

If you were working with WordPress you probably noticed that the installation includes a .htaccess file in its main directory.

Whate are these files?

The .htaccess file set specific directives for the server that should be executed when an Apache server is handling requests. In the default case of WordPress it makes the server point most of the requests to the main index.php file, which then loads all the necessary scripts. The server needs to be configured to look for and honor the directives included there. That can be achieved using the AllowOverride directive that is specified by by directory. Access files allow you to set different directives per directory without the need to touch the main server configuration.

Example:

<Directory "/path/from/where/the/files/are/served">
    AllowOverride All
</Directory>

In this case the server will include all the directives that are premitted to be set there.

Other options are:

"None" - the server will not even look for .htaccess files
directive type - one or more of the directive groupings such as "AuthConfig" or "FileInfo"

The configuration is applied to the directory where the access file is found and all subdirectories. However, subdirectories might contain their own access file that override configuration from higher up, because it’s being applied in the order that the files are found.

The default name of the access files is .htaccess, but this can be changes to something else using the AccessFileName directive:

AccessFileName ".config"

Docker volumes vs bind mounts

Tue, 12 May 2020 20:14:12 +0000

By default a docker container won’t store any persistent data, so anything that we write to its writable layer won’t be available once we stop it.When running Docker containers we might be interested in persisting certain data. This can be achieved by using either volumes or bind mounts. Below I’m describing main differences between the two.

Bind mounts are directories on the host filesystem mounted onto a Docker container. These can be modified outside of Docker.

Volumes on the other hand are managed by Docker only and you can use Docker cli commands to manage them directly. They are a preferred way to persist data.

There are named volumes created by the “volume” command that can be referenced by their name and there are volumes that are created by the VOLUME instruction from the docker files. The latter are identified by a hash and stored in the same Docker volumes folder.

When mounting volumes you specify the name instead of the paths you would like to mount. Volumes don’t increase the size of docker containers. If you would like to keep your own directory structure you should use bind mounts. Volumes make your project more portable, because with bind mounts you would need to rebuild your directory structure when migrating and you must ensure that all the security settings like user permissions are setup correctly.

Different ways of running PHP on an Apache server

Tue, 12 May 2020 00:24:59 +0000

When the server receives a request from the user it can serve the file directly, but sometimes it needs to translate the code using the appropriate application. Otherwise it would be showing the users the source code of the website. A server like Apache needs to be configured to handle specific content with an appropriate application like PHP.

A common set of rules of passing the information between the server and the application is the Common Gateway Interface. This standard lets programmers write applications in different languages that can plug into the server and interact with it. The interface guarantees consistency of data passed between the server and the application. An alternative to a CGI application is Microsoft’s Active Server Page (ASP).

CGI

A CGI program is one that conforms to the above specified protocol. The advantage of using CGI over a server module is keeping the execution of the application separate from the server.

If CGI is installed on the server a cgi-bin directory might be added to the public directory of the website and every page from that directory is treated as a CGI script, so the server runs the files through a relevant application instead of showing the code directly to the user. The server can also be configured to treat files with specific extensions as CGI files.

If CGI is installed on the server, the specific cgi-bin directory is also added there, for example home/user/public_html/cgi-bin. CGI scripts are stored in this directory. Each file in the directory is treated as an executable program. When accessing a script from the directory, the server sends request to the application, responsible for this script, instead of sending file’s content to the browser. After the input data processing is completed, the application sends the output data to the web server which forwards the data to the HTTP client.

When PHP is run in a CGI mode the server knows the location of the executable file and the loads it, with all its settings from the php.ini file every time it needs to interpret a php file. With this kind of workflow comes a lot of repeated work. The advantage is slightly better security, because the code is isolated from the server and the ability to load the php settings directly, without having to restart the server.

Fast CGI

The original CGI wasn’t very efficient and the FastCGI interface was introduced to deal with some of thess performance issues. One of the differences with the original is that the process spawned to handle the request is kept alive longer, so it can handle also some future requests.

FPM

FPM which stands for FastCGI Process Manager is a FastCGI implementation which has some features geared toward heavily trafficked websites. It maintains a pool of workers that can handle PHP requests and should avoid the memory overload from PHP in Apache processes. You can check out a list of features on the php.net website.

Apache Module

In case of using an Apache Module every child process that Apache spawns will contain its own copy of PHP interpreter. The downside of this solution is that the footprint of every spawned process is higher, so it consumes more server resources. In order to apply new PHP settings a server restart is required, but on the other hand PHP-specific settings can be included in the .htaccess files.

Finding out PHP server interface

Sun, 10 May 2020 22:14:04 +0000

There are many ways of executing a PHP script. For example you might run it directly from a command line:

php <name_of_the_script>

Other option, readily available for tesing since PHP 5.4, is to spin up the internal PHP server and see the result in a browser:

php -S localhost:3000

If you create a simple page with the function php_sapi_name you will be able to check out the interface that is being used to run the script.

In the first case the out put should be “cli” in the second one “cli-server”. If you run the file on a server it could output “apache2handler” or something similar if you’re using Apache server.

Installing Let’s Encrypt SSL certificate

Sun, 10 May 2020 00:40:23 +0000

Https has become a standard for websites and there’s no excuse not to have it enabled on the website, especially that it’s now possible for free.

Making your website more secure with a free Let’s Encrypt certificate is very easy with Certbot.

The prerequisites are to have SSH access to your server that is already online and serving the site over http. In the example I’m setting up the certificates with Apache.

You would need to add the repository for Certbot first:

    sudo apt-get update
    sudo apt-get install software-properties-common
    sudo add-apt-repository universe
    sudo add-apt-repository ppa:certbot/certbot
    sudo apt-get update

After Certbot and the Apache plugin are installed you need to install the certificate with the server:

sudo apt-get install certbot python3-certbot-apache
sudo certbot --apache

During the installation you will need to answer some configuration questions like which domains that you would like to include, but they are pretty self-explanatory.

And that’s it! Your server is ready to serve the site over https.