Hetzher vs AWS

I moved a business of ~100 FTEs from AWS to Hetzner once. Aside from the migration cost, the price was roughly 25% of AWS. I left many years ago, the business switched frameworks since then but they stayed on Hetzner.

Thinking again of this old story I now realize that the biggest gain was not monetary, but human. For years, that business could retain skilled engineers who had the opportunity to work closer to bare metal, caring about the nitty-gritty technical details of backups, failover and high availability.

And they did not even cost much. That they had so much leeway in designing the system instead of "relying on the cloud" was a major retainer.

PHP must die

One thing PHP got right is its 'PHP must die' mode of operation where every request spawns a new process (ok, not a process anymore but still an isolated execution environment that lasts until the response is served.

Java, Ruby, C#, etc are mostly used for application servers and this massively complicates everything.

For an engineer, it feels like one-process-per-request model, even with php-fpm and other optimizations should be much harder to scale than Java applications servers, but the reality is different.

I've spent many years doing PHP and Java-based development. Often times in parallel, having PHP projects on the side and earning money in Java development.

At some point, I had a stint of several years doing exclusively PHP-based work.

Every single time, PHP projects I worked on were serving orders of magnitude more people orders of magnitude cheaper than Java-based projects.

And when I say "orders of magnitude", it is not an exaggeration but sad truth.

I am now in the Java land again, because pay is better. My current project has already burnt several millions serving 20 secretaries that handle 1000 individual cases each year.

My mind is with the PHP project I sold several years ago when it was serving 120,000 people at the cost of a few hundred euros per year + my labor, which was essentially free, but if I could slap my usual daily rate on it, it would still be somewhere between 20,000 and 30,000€/year.

Even this blog is written using Drupal 7, a 14-years old PHP software that still works perfectly well and needs little maintenance. Go find a similar Java piece of software.

The strange origin of ISO20022 abbreviations

One of my favorite watercooler stories is about the origin of ISO20022 abbreviations.

ISO20022 is the XML-based format to exchange messages in inter-banking networks like SWIFT or SEPA., It replaced the old, ASN.1 inspired format. You can see both side by side here:

Notice the strange element names. Cdtr stands for Creditor, Nm for Name... for the most part, the abbreviations are easy to guess, but the obvious question is, why do they use abbreviations to start with?

Turns out, the standard first used unabbreviated element names, but once SWIFT and banks started implementing it, multiple complaints were raised about the increased traffic and storage space. The complaints bubbled up to the Standards Department and were put in a panel discussion during the yearly SWIFT conference. Where suits decided that the best way to fix the situation is to abbreviate the element names.

The rumor is, most abbreviations were penned on the spot in a way that created multiple naming clashes. That is, one can not automatically convert between abbreviated and unabbreviated XML without taking into account the context.

Anyway. The conferenced ended with the statement that the problem of message size has been solved.

And this is how we landed with words like Ccy forever ingrained into the banking vocabulary.

As any tech-savvy person may guess, the best way to make the XML smaller is to compress it. At the time, there was already the XML Fast Infoset standard which, among other things, defined how to optimize the compressor vocabulary by deducing it from the XML Schema, leading to higher compression rates.

Belgian diplomatic mail goes straight to NSA, right?

Ever since I worked on mailing lists, newsletters and transactional mails I have a habit of looking at mail server logs.

So did I today after sending a mail to the Belgian Foreign Ministry.

Turns out, my mail first passes through the servers of an US-based conglomerate that owns RSA Security, a company that has been publicly caught implementing backdoors for NSA.

Here's the exim mainlog:

diplobel.fed.be R=dnslookup T=remote_smtp H=primary.emea.email.fireeyecloud.com [3.123.5.10]
X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 DN="C=US,ST=California,O=Musarubra US LLC,CN=mx.emea.email.fireeyecloud.com" C="250 2.0.0 3yW4eTU-47539-06g0D5C1032540783A662d84a39 mail accepted for delivery"

Yes, 3.123.5.10 is hosted in Germany, but so what?

The curse of Active Directory

I broadly divide organizations into

  1. Active Directory based
  2. Other

When I say Active Directory based, I do not mean Microsoft. Price Waterhouse Coopers for instance is not Active Directory based but they have Windows notebooks and use Google Workspace.

Active Directory promotes a certain way of decision making that bonds all such companies together. The management attend the same events, receive the same vendors, buy the same solutions from the same suppliers, and in general tend to structure their businesses in similar ways.

People around me are so used to Active Directory companies, that they do not see it, but I had a chance to work in a Silicon Valley style unicorn, local and European businesses that are not Active Directory based, and the difference is huge.

A company that by choice or by accident has no Active Directory also had a chance to reinvent its organisation, not just its IT. Such companies differ more from each other than Active Directory companies. Adaptation, improvisation are their competitive advantage.

The Worst Kind of Programmer

After 25 years of my career I came to understand that one particular type of programmers is the source of many problems in our industry. Here is the story of a project that was nearly destroyed by two such programmers. One of them was leading frontend development, the other – backend. While the rest of the team was slacking off because business requirements were in the works, these two chaps were working hard.

The frontend lead bootstrapped an Angular mono-repository with NX, Rx and other trendy technologies of the moment. The corporate environment did not lend itself easily to NX, but he pushed hard to solve or circumvent infrastructure problems by working with the domain expert. He set up a workflow that, while having Angular as the base framework, bore little resemblance to the official Angular tutorials.

The backend lead was no less motivated to bootstrap the Spring Boot project according to the corporate guidelines while adding a bit of a personal touch in the form of the Vavr library and a highly sophisticated hierarchy of JPA entities with multiple levels of inheritance, discriminators and generators. He then sprinkled it over with a hierarchy of validators based on booth Spring Bean Validation and a third-party validation framework and polished with a trendy testing framework.

The rest of the team observed them in admiration.

On coding interviews

I am one of those senior engineers who regularly code and dare I say produce optimal solutions quickly. I still fail at coding interviews, and I attribute it to the gamification fatigue and age.

Please understand me. I have a leetcode account but I can not force myself into solving puzzles in my free time after solving them all day long at work. And puzzles at work are more interesting, because they are real people's problems.

I avoid board games for similar reasons. Whenever I sit in and play, I enjoy being in the flow, the conversation, and the intellectual challenge. But dragging me in is damn hard.

On the other side, I am much more into sports nowadays: running, swimming, skiing and hiking, as if I was making up for my younger years spent in front of a screen.

I feel like I have to carefully choose where to apply my intelligence because I am not as resourceful as I was. But I am wiser.

Would you hire me? No. Would I be a net positive for you business even if I cost double your younger self? Probably yes.

On the winning strategy in a core team

I was once hired for a core team job. Between my hire and the first day, the core team manager resigned and I was reassigned, so I did not have much chance to work with him.

Instead, I landed in a middle of a drama. The previous core team manager was a servant leader, delivering bare minimum and helping on administrative and organizational matters. He was the loosing party, preparing to quit and passing affairs. By virtue of being hired by this guy, I was considered his protégé by the new core team manager.

That new manager was operating by decrees, and his first decrees were naturally targeted at everyone who could have undermined his authority, including me. Instead of helping he was setting standards and mandating bills of materials. Older product teams mostly ignored him, but new ones were suffering. By the time I left, the guy had a promotion.

Lesson learnt? Core teams win by imposing, not helping.

What do to if your website was marked as malicious by an antivirus or security vendor?

Aside from the obvious JavaScript injections and framework exploits, check for

  • multiple redirects
  • \u-encoded JavaScript code
  • broken or too permissive CORS along with suspicious backend name

Then, check different website URLs on Virustotal URL checker and use yaronelh/False-Positive-Center to report false positives.

Enjoy.

How to do a healthcheck in a Docker image without curl?

There is a tendency to strip docker images from wget and curl. Luckily, perl is still available. If you have trouble configuring spring boot healthcheck, think of perl and its IO::Socket module shipped with perl core:

FROM openjdk:11-jre-slim
HEALTHCHECK --start-period=10s --timeout=3s --retries=5 \
 CMD perl -e "use IO::Socket; $sock = IO::Socket::INET->new(Proto => 'tcp', PeerAddr => 'localhost', PeerPort => '8888') or die $@; $sock->autoflush(1); print $sock 'GET /actuator/health HTTP/1.1' . chr(0x0a) . chr(0x0d) . 'Host: localhost:8888' . chr(0x0a) . chr(0x0d) . 'Connection: close' . chr(0x0a) . chr(0x0d) . chr(0x0a) . chr(0x0d); while (my $line = $sock->getline ) { if ($line =~ /UP/) {exit;} }; close $sock; exit 1;"
Tags: 
docker
perl
curl
wget

Pages