I had previously used Arq for backing up our home network. It is awfully powerful and easy to use. I especially appreciate that it lets you run backups to numerous cloud providers. Unfortunately, it only runs on Windows and Mac. I could keep using it by letting my main Windows computer back up the contents (by reading from a shared drive), but something running on the Linux server itself would be much better.

After a round of searching, I settled on Restic. Restic is a lot like Arq, except that it runs on Windows, Mac, and Linux. It’s also free and open source. Unfortunately, it is command-line only, so it takes a bit more work to set it up.

The Linux setup was easy, but it took a couple of tricks to get it working the way I wanted on Windows…

Setup on Linux

There is lots of good documentation for setting up Restic backups on Linux. This guide also shows setting up an S3 bucket in AWS: https://restic.readthedocs.io/en/stable/080_examples.html#setting-up-restic-with-amazon-s3

Because I use Debian for my home server, I used systemd Services and Timers to run the backups on a schedule. It was easy to get going, and works great. I highly suggest using systemd-creds to keep your repository key encrypted in your systemd service files.

You should also make sure to run this command after installing Restic: sudo restic self-update. One of the weaknesses (and strengths) of Debian is that the package feeds are slow to take feature updates. The version apt installed still worked, but it was pretty old, and there were some important updates in that time.

Installing on Windows

Setting it up on Windows isn’t very different from Linux. It might be intimidating for an average Windows user, but a power user should have no trouble. winget makes it particularly easy:

winget install --exact --id restic.restic --scope Machine --force

Configuring Regular Backups on Windows

I spent a bit of time experimenting and iterating before I settled on the following configuration. It’s not a requirement for using Restic, but I like the way it works.

First of all, I like to keep the configuration and cache files in ProgramData. You can use these commands to create the directories required for the scripts below:

# Create the directory structure:
New-Item -ItemType Directory C:\ProgramData\ResticBackup\Settings -Force
New-Item -ItemType Directory C:\ProgramData\ResticBackup\Cache -Force
New-Item -ItemType Directory C:\ProgramData\ResticBackup\Logs -Force

I also use text files for listing the includes and excludes. This way I can easily look the list up or adjust it without touching the scheduled task or PowerShell code.

"C:\Users\$Env:USERNAME
C:\ProgramData\ResticBackup\Settings
" | Out-File C:\ProgramData\ResticBackup\Settings\Includes.txt

"# Code
node_modules
C:\Code\**\bin
C:\Code\**\obj
C:\Code\**\build
.vscode
.nuget\packages
.idea\libraries

# User directories
C:\Users\**\Temp
C:\Users\**\Google\Chrome
C:\Users\**\INetCache
C:\Users\**\MicrosoftEdge\Cache
C:\Users\**\Mozilla\Firefox\**\cache*
" | Out-File C:\ProgramData\ResticBackup\Settings\Excludes.txt

This is the PowerShell script that runs the backup. Write the contents to C:\ProgramData\ResticBackup\Settings\Invoke-ResticBackup.ps1:

#Requires -Version 7
param
(
    [Switch]
    $WhatIf
)

Set-StrictMode -Version 3.0
$ErrorActionPreference = 'Stop'

$LogPath = "C:\ProgramData\ResticBackup\Logs"
$LogFilePath = "$LogPath\$(Get-Date -Format 'yyyy-MM-dd_HH-mm-ss')_Backup.txt"
Start-Transcript -Path "$logFilePath" -UseMinimalHeader

$ResticRepositoryKeyFile = "C:\ProgramData\ResticBackup\Settings\ResticRepositoryKey.$Env:USERNAME.txt"
if (-not (Test-Path $ResticRepositoryKeyFile))
{
    throw "repository key file not found for user: $ResticRepositoryKeyFile"
}

$backupParams =
    $(
        '--repo', 's3:s3.amazonaws.com/<your-bucket-name>',
        '-o', 's3.region=<your-region>',
        '-o', 's3.storage-class=ONEZONE_IA'
    )

$Env:AWS_PROFILE = 'restic'
$Env:AWS_SHARED_CREDENTIALS_FILE = "C:\Users\$Env:USERNAME\.aws\credentials"

if ($WhatIf)
{
    $backupParams += '--dry-run'
}

$Env:RESTIC_PASSWORD_COMMAND = "pwsh -NoProfile -NonInteractive -Command '(Get-Content `"C:\ProgramData\ResticBackup\Settings\ResticRepositoryKey.$Env:USERNAME.txt`" -Raw).Trim() | ConvertTo-SecureString | ConvertFrom-SecureString -AsPlainText'"
$Env:RESTIC_PROGRESS_FPS = '0.016666'

restic backup @backupParams `
    --files-from "C:\ProgramData\ResticBackup\Settings\Includes.txt" `
    --iexclude-file "C:\ProgramData\ResticBackup\Settings\Excludes.txt" `
    --cache-dir 'C:\ProgramData\ResticBackup\Cache' `
    --use-fs-snapshot `
    --no-scan `
    --verbose `
    2>&1 | Out-String -Stream | Tee-Object "$LogPath\restic-progress.txt"

You can use this command to schedule the task (running as user SYSTEM, twice daily):

$action = New-ScheduledTaskAction -Execute 'pwsh.exe' `
    -Argument "-NonInteractive -NoProfile -File `"C:\ProgramData\ResticBackup\Settings\Invoke-ResticBackup.ps1`""

$trigger1 = New-ScheduledTaskTrigger -Daily -At 7:10am
$trigger2 = New-ScheduledTaskTrigger -Daily -At 7:10pm
$settings = New-ScheduledTaskSettingsSet -ThrottleLimit 1 -MultipleInstances IgnoreNew -DontStopIfGoingOnBatteries
$principal = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest

$task = New-ScheduledTask -Action $action -Trigger $trigger1,$trigger2 -Settings $settings -Principal $principal

Register-ScheduledTask -TaskPath 'Backup' -TaskName 'Restic S3 Backup' -InputObject $task -Force

Storing Passwords Securely on Windows

If you were paying close attention above, you may have noticed the username variable in the repository key file. That’s because the key is stored in an encrypted format (via ConvertFrom-SecureString), and the encryption key it uses is different for each user of the computer.

ConvertFrom-SecureString has a -Key parameter so that you can use the same encrypted string for multiple users. The problem with this approach is that you then have to protect the key used to encrypt the secret, and end up with the same problem again.

This mechanism is a little annoying, but it keeps the key safe. It would be a lot simpler to store your unencrypted repository key in a text file and let Restic use this instead. I don’t recommend this, however, because anyone who can access that file could then read your backups.

I use this script to write my own copy of the encrypted repository key:

$ResticRepositoryKey = Read-Host -AsSecureString -Prompt 'Restic Repository Key'
$ResticRepositoryKey | ConvertFrom-SecureString `
    | Out-File "C:\ProgramData\ResticBackup\Settings\ResticRepositoryKey.$Env:USERNAME.txt"

Saving the key so that the system user can decrypt it is a little more challenging. There is no way (that I’ve found) to encrypt a secret for another user with ConvertFrom-SecureString, so you have to run the code that encrypts it as that user. Because you can’t log in as SYSTEM directly, I use a scheduled task. The running task needs access to the key, however, so I temporarily write the unencrypted key to a file.

Write the following to Save-EncryptedResticRepositoryKeyForCurrentUser.ps1:

#Requires -Version 7
Set-StrictMode -Version 3.0
$ErrorActionPreference = 'Stop'

$LogPath = 'C:\ProgramData\ResticBackup\Logs'
$LogFilePath = "$LogPath\$(Get-Date -Format 'yyyy-MM-dd_HH-mm-ss')_SaveEncryptedKey_$Env:USERNAME.txt"
Start-Transcript -Path "$logFilePath" -UseMinimalHeader

$ResticRepositoryKey = (Get-Content 'C:\ProgramData\ResticBackup\Settings\UnencryptedRepositoryKey.txt' -Raw).Trim() `
    | ConvertTo-SecureString -AsPlainText

$ResticRepositoryKey | ConvertFrom-SecureString `
    | Out-File "C:\ProgramData\ResticBackup\Settings\ResticRepositoryKey.$Env:USERNAME.txt"

This script writes the unencrypted repository key to a file, then configures a scheduled task to encrypt it as the SYSTEM user. It then runs the scheduled task, waits a few seconds for it to finish, and (crucially) deletes the unencrypted key file.

$ResticRepositoryKey = Read-Host -AsSecureString -Prompt 'Restic Repository Key'
$ResticRepositoryKey | ConvertFrom-SecureString -AsPlainText `
    | Out-File 'C:\ProgramData\ResticBackup\Settings\UnencryptedRepositoryKey.txt'

$scriptPath = "C:\ProgramData\ResticBackup\Settings\Save-EncryptedResticRepositoryKeyForCurrentUser.ps1"
$action = New-ScheduledTaskAction -Execute 'pwsh.exe' `
    -Argument "-NonInteractive -NoProfile -File `"$scriptPath`""
$settings = New-ScheduledTaskSettingsSet -ThrottleLimit 1 -MultipleInstances IgnoreNew -DontStopIfGoingOnBatteries
$principal = New-ScheduledTaskPrincipal -UserId "SYSTEM" -LogonType ServiceAccount -RunLevel Highest
$task = New-ScheduledTask -Action $action -Settings $settings -Principal $principal
Register-ScheduledTask -TaskPath 'Backup' -TaskName 'Encrypt Restic Repository Key for SYSTEM User' -InputObject $task -Force

Start-ScheduledTask -TaskPath 'Backup' -TaskName 'Encrypt Restic Repository Key for SYSTEM User'

Start-Sleep -Seconds 5
Remove-Item 'C:\ProgramData\ResticBackup\Settings\UnencryptedRepositoryKey.txt'

Running Restic Manually

If you want to run any manual Restic commands (viewing backups, retrieving files, pruning history, etc.), and you don’t feel like pasting your repository key every time, you can use this helpful one-liner:

$Env:RESTIC_PASSWORD_COMMAND = "pwsh -NoProfile -NonInteractive -Command '(Get-Content `"C:\ProgramData\ResticBackup\Settings\ResticRepositoryKey.`$Env:USERNAME.txt`" -Raw).Trim() | ConvertTo-SecureString | ConvertFrom-SecureString -AsPlainText'"

It allows Restic to decrypt and use your encrypted repository key every time it needs it. This is the same mechanism used in the backup script.

Regarding S3 credentials

I am storing my backups in AWS S3, and using the AWS CLI’s own configuration mechanism to store the AWS API key for me. It doesn’t seem to be encrypted, which is not my favourite, but I’ve decided that this is okay for me. The key is associated with a user that has only the permissions needed to write backups into the one S3 bucket I use for them.

If you want to be extra cautious, you could use a trick like I use for the repository key.

Other Considerations

For a fully-functional, self-sustaining backup system, there are a few more things you need to set up:

• regular pruning of old/unnecessary data
• some kind of monitoring to tell you if the backups stop running
• a routine/process to periodically check that you can actually pull files from your backup
• if the backup destination supports it, some mechanism to lock/version files for a set amount of time will ensure that your backups can’t be destroyed secretly (a common trick used in ransomware attacks)

Would anyone be interested in me writing about these things? Comment below with any particular questions you have.

Results

I’ve been using this to back up my personal laptop for over a year now, and it’s been working great. It’s fast and effective. It works even when I’m travelling. And I don’t notice a performance hit if it runs while I’m working.

Arq was good to me, but I’m glad I switched.

A lack of trust can have negative effects beyond just making people miserable. It can discourage exploration and discovery. It can force painful and inefficient workarounds. It can also cause data to be duplicated, forgotten, corrupted, or lost entirely.

In the age of autonomous agents this problem is bound to get worse, so let’s talk about how to do it better.

Quick version

An easy way to test your background interactions is to imagine that you are the background process, and your significant other asked you to do the task. Assuming you want to please your significant other, what would be the best service you could provide?

If my wife asked me to buy milk at the store, I would buy the milk, and put it in the fridge. I wouldn’t interrupt her in the middle of something important to tell her the milk is in the fridge. I’d just put it there, and she would expect it to be there later.

If the store was out of milk, however, I’d probably let her know. Similarly, I wouldn’t call her at work to tell her about it on the spot. I’d save it until the next convenient opportunity to bring it up.

If there was some problem selecting the milk, like I wasn’t sure what kind she wanted, or they were out of the kind she wanted, I might call her. I might just make a choice. I might skip it and wait until the next time I’m at the store too. This will depend on the urgency of the request, and the cost of getting it wrong. Buying the wrong kind of milk is a bit wasteful, but if we were talking about ordering plane tickets, buying tickets to the wrong city would be a serious problem.

To bring this back to a software example, take a look at email clients. Email is asynchronous by nature, and a lot of effort has gone into refining the experience over the decades.

Aside: You may not think email applies to your situation, but according to Zawinski’s Law, you’ll be adding email to your application eventually.

If I push send on a new email, it’s nice to get some immediate acknowledgement that the email will be sent. If everything goes normally, this is enough.

If the email system can’t deliver the email, it will usually give some kind of feedback. It usually isn’t an in-your-face warning. A little red dot on the outbox is probably enough. Some indication of what it means if a user hovers or clicks on it helps with discoverability. If the client can eventually send the email, the dot can silently disappear. A very busy person might never notice, and that’s okay.

If something was wrong with the email such that it could never be delivered (ex: a non-existent email address), something a little harder to ignore makes sense. An in-your-face error message is still not great. A lot of email services send an email back to you explaining the situation. This is great because you’re (likely) already checking your emails, and you can choose when you deal with it.

There are numerous failure modes in email, and if you haven’t worked on email processing yourself, you probably aren’t aware of most of how fraught a system it really is. This is a sign of a mature user experience. You haven’t hit many edges because you’ve been steered away from them subtly. When problems do occur, though, the feedback feels natural and proportionate.

Long Version

I now submit to you: my attempt at a list of suggestions to consider when designing background processes.

1) Be Transparent

Background process can be awfully opaque, even in some well established applications. When a user is requesting some work, it’s important that they understand what they are doing. If they mess something up because the system did something they didn’t anticipate, they will lose trust in the system.

1.1) It should be obvious when starting an asynchronous task

Users should have some indication that the thing they’re requesting will not be instantaneous. If it will be a long time before it gets done, this should be obvious too. There are a lot of great, subtle ways to hint at this. This should be enough for most cases.

1.2) The user should know what they’re going to get, and/or it should be easy to undo

If a button in an application is a “heavy” button, meaning that it does a bunch of stuff, or can’t be easily reversed, I naturally feel nervous. Messing things up for myself is annoying, but messing them up for my team can be a lot more scary. This is especially true when I’m not familiar with the system I’m using. For any changes that aren’t obvious, it’s helpful to get a breakdown of exactly what will happen.

Alternatively, if the operation is quick and easily reversible, that might be good enough. It’s sometimes a lot easier to show the result than to describe it. However, the longer the process takes, the harder it will be for the user to notice what changed. Try-and-undo works best for changes that are easily discoverable.

2) Be Dependable

A big part of handing work over to a computer is so that you don’t have to keep track of it any more. This kind of delegation can be very freeing, but it only works if the application delivers reliable results.

2.1) Don’t drop the ball

When a user requests something, make sure you do it, or if you can’t, make sure you tell them that you couldn’t do it. This could take a lot of different forms, and some tuning to get it right, but it’s an essential part of being useful.

2.2) Fail quickly when possible

If something is requested that will obviously never work, you should try to let the user know as soon as possible. Even better if you can notify them before accepting the task. If a user is notified right away, they’ll still have the context in their head to fix it. Or if they are delegating a more important task, they will know they might need to change their plans. A great example of this is when sending an email with no recipients.

2.3) Let users check-up on long-running tasks

Allowing users to check on the things they requested can also improve trust. This is especially important for tasks that can take a long time, or are of critical importance. Sometimes people want to check up on what’s going on, or check to see if something is broken.

2.3.1) Finding completed tasks

Similarly, being able to find work that has completed successfully can also be helpful. The sent folder in most email clients is a great example of this. If there is ever a question about something being done, looking it up is awfully convenient.

2.3.2) Allow cancelling work that hasn’t completed

If you do have a way to see queued or incomplete work, and it’s possible, allowing someone to cancel it is also nice. If they realize they forgot something, or made a mistake, being able to stop it and fix it early allows them to do better work.

This assumes that cancelling a task is possible. If it’s not possible, then obviously you shouldn’t make that option available.

3) Handle Issues with Civility

Error handling is always tricky in software design. It’s especially tricky with processes that don’t have constant attention from the user. Getting this right is another essential part of maintaining trust.

3.1) The intrusiveness of a notification should be proportional to its urgency

If a user has had enough time to move on to another task, you should only interrupt them for feedback if it’s likely they’ll want to drop what they’re doing to deal with it. Email is an example where it can probably wait. If someone is hailing a ride, and something has changed the driver’s arrival time significantly, you should probably let them know urgently, and give them some choices about what happens next (such as cancelling or rescheduling).

3.2) Make the current state of things clear

When a user is notified of a problem, they’re going to have to decide what to do about it. If work was half-done, things were partially sent, or something needs to be manually cleaned up, they should know this. Expecting them to know, or to hunt around and figure it out themselves isn’t very generous. If some kind of retry or automatic correction is going to happen asynchronously, it’s essential that this be communicated as well.

Making atomic changes is nice when it’s possible, because not doing anything is usually better than doing something half-way. Of course, it’s not always feasible. If you do have this behaviour, make sure your users know about it. It will give them more confidence with the system.

Alternatively, it can be helpful to give the user an easy way to find the affected data / records. If they have to do manual cleanup, this will at least help them find it all.

3.3) Let the user manage failure notifications

Providing asynchronous failure notifications with some bit of visual feedback can be a great way to go for low-urgency issues. When you do, however, make sure that there is a way for the user to clear the notification. If you want a user to respond when they see the red dot, make sure you take it away when there is nothing to do. Otherwise, if they get accustomed to seeing the dot when there is nothing to fix, they may start to ignore it when it there really is something to do.

Similarly, for tasks that can be important but may not necessarily be urgent, once they’ve been notified of a failure, they may want to keep it in a list somewhere so they are reminded to deal with it later.

Email is a great example of this done well. When a message can’t be delivered, popping it into their inbox lets them keep it as long as they want. They can delete it if they respond to it right away, forward it to someone if they need to ask for help, or just leave it there forever.

4) Be Considerate of Multi-user Impacts

It’s always important to consider the impacts of actions in multi-user systems, but this is especially true with background processes.

If it’s normal for tasks to run for a very long time (hours, days, etc), you might need a system that multiple users can use to monitor the work and respond to any failures. You don’t want urgent issues to get ignored when someone has taken a lunch break or a vacation. If immediate action isn’t necessary, you could also have a staged notification: only notify the broader team if the person who started the job hasn’t dealt with it in some amount of time. This can get pretty complicated though, so use it carefully, and test it a lot.

If you do allow multiple users to respond to a failure, you might also need some way for them to coordinate who is responding to the issue.

When tasks are very expensive to run, or there are scarce resources to run them, you may also need some sort of prioritization or rate limiting system. However, these kinds of mechanisms can cause unpleasant side effects when systems are run beyond their capacity, so consider them carefully before adding them in.

5) Be Autonomous with guard rails

A big benefit of running work asynchronously is that it frees up the user to do other things. The more you require interaction to get the task done, the less valuable this becomes. That being said, doing something undesirable in failure scenarios is often worse than doing nothing. So it’s important to only implement autonomous error handling when the desired outcome is certain.

5.1) Make the obvious correction when possible

Sometimes the way to resolve an issue is obvious. An easy example is when someone is trying to delete something that no longer exists. Maybe you treat that as a success, or a success with a warning. The user requested that the thing no longer exist, and now it doesn’t.

This assumes, of course, that there isn’t some other implication to the delete being already requested by someone else. An inventory management system is an example where it might be a more serious problem.

5.2) Allow the user to override bad judgements

If you do make corrections on behalf of the user that could be questionable, make sure there is a way for them to find them and fix them if they’re wrong. Replication conflicts is a common example of this. It’s great when you can auto-merge, but sometimes it’s wrong, and if there is no way for the user to find the conflicting changes, it may be hard for them to make it correct.

5.3) Guard against infinite loops

Automatic retry is a great solution to some kinds of problems, but you should always make sure there is some finite limit to how much you’ll retry. Just like queue base systems need dead-letter queues, background tasks need some way to say that it’s unlikely that a task will ever complete if the user doesn’t intervene.

Also make sure that you don’t retry immediately over and over again. If some system down the line is having a temporary hiccup, this just increases the stress on it when it may already be overwhelmed. Incremental back-off and circuit-breaker mechanisms can make a components drastically better citizens of their systems.

Conclusion

Background processes may be mostly invisible, but they become quite obvious when they behave poorly. Design your background interactions mindfully, because delighting users is what we should all be striving to do.

GitHub Spark is a new feature in GitHub Copilot that helps you build web applications quickly using AI. The Spark platform is based on React with Tailwind CSS. It also provides user authetication (via GitHub account), and user-specific data storage (via Azure).

At the time of this writing it seems like it’s still an early version of the platform, so depending on when you’re reading this, you may need to do your own research.

The Project

I’ve been looking for a good health tracking app for years, but so far been disappointed with everything I’ve found. They are either too simple, too limited, or too expensive. There are some free ones, but I haven’t found one I like. Even if I did, the free ones are often making their money selling your health data. So I tried making my own with Spark.

The kind of app I’m looking for will provide reminders for medications, track how much I’ve taken and when, and record a few symptoms of interest. With this kind of data available, I can take hard numbers and graphs to my doctor appointments. It’s definitely on the nerdy end of the data collection fad, but I’ve found it helpful and insightful.

Short Version

It took a couple of four-hour sessions, but by the end of it, I had an app that I’m still using today. I like it better than any product I’ve seen. It has a few annoying bugs, some visual inconsistencies, and the layout could be a lot better. Honestly though, it looks and performs better than many cheap mobile apps I’ve seen in the store.

The most interesting part for me is the Spark platform, which is both a strength and a weakness. It’s comforting to have built-in authentication and storage mechanisms instead of depending on AI slop for these core functions. It also takes care of auto-update notifications. It makes it possible to get many simple apps off the ground quickly.

The problem, though, is that once you’re in the platform, you’re in the platform. Spark only makes web apps built on the Spark platform, and you don’t have any other choices for where the data goes or how it gets synced. It also doesn’t make mobile apps. It works fine as a pinned link on my phone, but this limits it’s ability to be used for ad-supported consumer apps.

Notifications

I’ve not yet seen a single notification from my app, even though Spark has assured me that it works. It’s also assured me that it’s fixed it, twice now. Each time, it’s told me that it knows for sure why it hasn’t been working. So this hasn’t inspired a lot of confidence.

This could be a limitation of the platform, or the platform it’s running in, but it is a glaring defect. Even though it can’t do one of the main things I want from the app, I still prefer it over others I’ve used.

To be clear, I haven’t spent much time debugging or diagnosing the issue myself. I might be able to get it to work if it was a significant issue for me, but so far, it hasn’t made the top of the list.

Design Experience

Spark generated a working app that met most of the requirements after just a few minutes. It wasn’t great, but it was usable. Unfortunately, my experience went downhill from there.

The app had a few tabs for the different areas of functionality, but they all worked differently. To make them consistent, I had to keep asking Spark to change things. I found myself getting more and more explicit because I was developing a preference, and I wanted to make sure it took the best approach. This is not what it’s like working with great developers, it’s more like how the relationship goes when I work with sloppy ones.

It produced a clean, modern looking app, but it had a bunch of usability issues. Information density was terribly low, and several iterations trying to improve it resulted in little improvement. I ended up accepting it as it was and moving on to more important things.

One part of the experience I liked was the way the app is constantly running in a test mode while Spark is working on it, so you can keep playing around and testing. Unfortunately, the changes were often so slow that I found myself needed to document all the things I wanted to fix somewhere else. I couldn’t just walk through all the little visual and usability issues and move on, I had to enter them one at a time and wait while it figured out what to change.

The long time between changes quickly became a real drag on the experience. Even simple changes like, “Make the save buttons on every tab have the same padding and style” could take 5 minutes. That’s not long enough to do anything else, but way too long to maintain a state of flow.

Rapid Prototyping & Evaluating Requirements

The work was ultimately very different from normal programming. I wasn’t building an app as much as I was designing a product. It was easy to try things out, to get my hands on it, and see what works. But I had arms-length control in a way that I’m not used to.

To be clear, Spark makes all the code available to you, and editable, but I’m not sure how well it handles it if you start changing code directly. Except for very minor tweaks to wording, I didn’t want to risk losing the AI experience to continue making more fundamental changes.

I think this project turned out well for me because of a few factors:

• I was building something I wanted to use myself. I have a very strong understanding of the problem and requirements. Every reaction I had was a real user reaction. This wouldn’t be true if I was trying to make software for an industry I’m not a part of.
• I had already tested several similar applications, so I had seen a few different approaches to the problem. This allowed me to articulate what I wanted up front, taking the best parts from everything I’d used.
• I’ve been designing and building user interfaces for a long time, so I know the common patterns and have a specific sense of what I like. Also, like any developer, I am particularly well practiced at describing what I don’t like. In this case, it’s actually a benefit.

I don’t think you need to be able to write code to build an app using Spark, but I think it would be hard to build a good one without practice studying them. Spark wasn’t able to suggest approaches, or push back when I asked for things it couldn’t do well, or do at all. I had to give it meticulously detailed descriptions and then inspect everything thoroughly.

That being said, the ability to build, test, and iterate helped me to figure out what worked and what didn’t in an amazingly short amount of time.

Another non-trivial factor was how tiring it was to work with Spark. Between waiting and making decisions (which is surprisingly tiring), I would get tired really fast. I found it hard to motivate myself to go back for the second round to finish the project.

Looking at it from the outside, building a functional-enough app in just 8 hours of labour is an amazing feat. But even as someone who loves using software to help people, I don’t feel any excitement about using it again.

Since those first two sessions I’ve since gone back and tweaked some features and fixed some bugs. I think I made a pretty big improvement with just 30 minutes, and I didn’t feel worn out afterward. Perhaps more short stints spaced out would be a better way to work with this sort of tool.

Next Steps

It was an interesting experience, and helped me prove to myself that a home-made tracking app is something I would actually use. I don’t think I could turn this one into a product though. I’m not even comfortable releasing it freely as open source. It might be possible to hack my code out of Spark since it’s mostly powered by React. It might even be possible to cram in a shim to change where the data goes. My developer instincts tell me that it wouldn’t be worth the trouble it could cause.

I will probably make another attempt at building a similar app with AI, but I won’t use the Spark platform. I’d like to have a bit more control (or any at all) over how the data is stored, how users log in, and to be able to make real mobile applications with platform-native features.

I might use Spark again if I want to try some ideas out quickly, or show someone a prototype. It might be okay for some in-house tools too, but the user-centric data storage would be a dealbreaker for many common scenarios.

It’ll be interesting to see how this technology evolves. I’m sure the AIs will get faster and better over time. The Spark platform could add a native mobile runtime environment and expanded features for managing data. Until it does improve, though, I can’t see myself using it for any serious projects.

I was accessing the containers via the aws ecs execute-command --interactive --comand "bash" mechanism. If you’re not familiar, this is a quick way to get a terminal inside an ECS container running in any cluster. It’s especially helpful when you have containers that do not receive incoming traffic from the internet. It is almost like SSH, but it is not exactly SSH. And one of the limitations is that there is no equivalent to scp. There is also no ability to forward ports, so you can’t use it to access SSH either.

You could assign a public IP to every container you run so that it’s accessible via SSH. But, why would you? Not only does it cost more (though only a little bit), it is increasing the attack surface of your system.

The easiest way to get a couple of text files onto an ECS container is to copy and paste their contents into a file through the terminal window. This only works with plain text files, but for scripts this is usually sufficient. For example:

cat >myscript.sql <<EOF
paste your contents here
EOF

I tried this at first, but about 5 files in I started to wonder if I was still pasting the right contents into the right file. There is a kind of mindless haze the sets in when doing repetitive operations, and it invites mistakes. I’ve also got a nurtured sense of laziness that pushes me to find new ways to solve problems.

After a bit of fiddling, I discovered that it’s possible to compress a bunch of files into a giant base64 blob, which can easily be pasted into the terminal session, and unpacked inside it. This means you’re only doing one paste operation, and binary files can be transmitted too! Here is how to do it:

First, you should connect to the destination machine. You can use a command like this with the AWS CLI. Note that you need to be authenticated and configured to use the correct AWS region first. You’ll also need to look up the task id, which can also be done from the command line, but I find the AWS console faster.

aws ecs execute-command --cluster mycluster --interactive --command "bash" --container mydatabase --task 6932e628a2ca4a15b53d9e25d9bd3193

The main reason to do this first is so that you only have to copy and paste the base64 blob once. Guess how many times I’ve copied the blob, only to realize I need to use the clipboard for the task id, and then have to select and copy the blob again afterwards?

Next, from a different terminal window on the source machine, you use tar and base64 to convert the desired files into a blob of base64:

cd location/of/files
tar -czf - *.sql | base64

This will dump all the content to the terminal. Depending on your environment, you may be able to send it directly to a clipboard. I can’t figure out how to do it in my WSL environment, so I just select it and copy it manually. It may help to use a clear command first so that you can find the start of the blob more easily.

Back on the destination machine, stick the base64 data into a temporary file:

cat >pasted.b64 <<EOF
paste contents here
EOF

Now you can unpack the contents into the current directory with this command, and continue with your task.

base64 -d pasted.b64 | tar -xzf -

You could also use the same technique in reverse to move a bunch of text files from an ECS container back to your local machine. Or even between two different ECS containers.

Or maybe some nice person will write an aws-ecs-scp command to do all the hard work for us.

Robux are a type of token that you can use to buy stuff in the various Roblox worlds. There are links to buy things everywhere, in your face, all around you, always. This is because Robux are a way for creators in the game to earn money. It’s unfortunately difficult for younger children to understand exactly what the big numbers beside cool things mean, or what the impact of buying them is, so I suggest considering how you want to approach this as a parent before dumping your kids into it.

The default mechanism for buying Robux in a kids account is for you to put your credit card number into their account. Any time they buy Robux, they have to promise that they are, in fact, the adult owner of the card. I am skeptical that a wall of text will effectively dissuade an eager child from impulse-buying something cool, so I don’t recommend using this method.

You can, however, easily purchase Robux gift cards in stores and online. I get mine on Amazon. There are a lot more choices for amounts to spend, and the price seems to be a touch better than buying them on the various gaming platforms.

I suggest the “Roblox Digital Gift Code”, which is fulfilled immediately via email. All you have to do is hand the code to your kid. Once they enter the code in their account, they can use the Robux immediately.

When I’m working at home, I connect to this workstation using Remote Desktop (RDP). It’s a fantastic protocol for accessing a Windows machine across a network. It’s built in to the operating system, and it’s generally pretty smooth. Most times I can’t even tell that I’m using it.

When I’m working remotely, however, accessing that workstation under the desk is a lot harder.

You could share the RDP port on the workstation through your router, but I don’t recommend this. Windows isn’t really hardened to be on the open internet. I suggest instead exposing an SSH server, authenticating with a certificate (and not allowing any other methods), and using that to tunnel RDP through to your Windows workstation nestled safely within your firewall.

In this post I’ll walk you through setting that up.

Quick version

You can use this command to set up the SSH connection with RDP being routed through it:

ssh user@fully_qualified_ddns_name -p ssh_port -i ~/.ssh/home_server -L 127.0.0.2:3389:windows_private_ip:3389

Substitute the values:

• user - your username on your home server (where SSH is running)
• fully_qualified_ddns_name - the full ddns name of yours router (ex: myhome.duckdns.org)
• ssh_port - the public port you use to expose SSH to the internet (ex: 2299)
• home_server - the certificate file you set up for SSH authentication
• windows_private_ip - the private network ip (from the point of view of your home server) of the Windows workstation you want to connect to

Once the connection is established, open the standard Remote Desktop client, and connect to: 127.0.0.2. Then log in to the workstation with your normal credentials.

How it works

Setting up SSH

I’m running sshd (the service that hosts SSH) on a little home server I use to run a number of services in my home. I’m using Debian, so I was able to install sshd easily using the built-in package manager. There are SSH servers for all the platforms, so you don’t need to use Debian if you don’t want to.

If you do use Debian however, this is how I’ve got mine configured:

sudo groupadd server-access

CURRENT_USER=$(whoami)
sudo usermod -aG server-access "$CURRENT_USER"

sudo echo '
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
MaxAuthTries 4

AllowGroups server-access
' > /etc/ssh/sshd_config.d/secure.conf

sudo service ssh restart

This creates a group called server-access, and adds the current user to it. Later on, we’re allowing anyone in this group to access the sshd server. We are also requiring all users to authenticate with a certificate. You can add as many users to this group as you’d like.

Creating a certificate and adding it to a user

Run this command to generate a certificate on the computer you’re connecting from. Note that it should work in both Linux and modern versions of Windows. Probably on Macs too, but I haven’t tried.

ssh-keygen -t rsa -f ~/.ssh/home_server

You can use any filename you like for the certificate, but remember what you choose because you’ll need it later. If you like, you can also add a comment inside the file with -C "home server". This can help keep them straight if you have multiple certificates.

This command generates two files: the private key (the name you chose), and the public key (the name you chose with .pub added to the end). Open the public key file and copy its contents. You can use cat ~/.ssh/home_server.pub to read the file in a console.

If you’re using a Debian machine for your Home Server, you can find (or create) a file at ~/.ssh/authorized_keys, and paste the whole contents of the .pub file as a new line. If you’re setting this up for multiple users, make sure to create one of these files for each home directory. I suggest different keys for each user.

If you have multiple computers to connecting to your home network all for the same user, I also suggest using a different key pair for each connecting computer. This way, if one of your computers is lost or stolen, you can delete its key while continuing to use the others. If you’re doing this, I also suggest adding comments to the end of public keys in the authorized_keys file so you can tell them all apart.

Now you can test your SSH authentication. Running this inside your home network is fine.

ssh user@home_server_ip -i ~/.ssh/home_server

If it works, you should be logged in to the home server.

Exposing SSH to the Internet

You have to set up port forwarding on your router/firewall to allow SSH on your home server to be accessible from outside your home. Look to the documentation for your particular router if you need help with this.

The default port for SSH is 22. You should be forwarding to port 22 at the local ip address of your home server. I don’t recommend, however, using port 22 as the public port on the outside of your router. There are a lot of robots crawling every IP address on the internet. Since port 22 is a commonly used port, it could be a target for automated attacks.

It’s ideal to choose a port about 2000 as this is well past the range of the most commonly used protocols on the internet. You can refer to a listing of commonly used port numbers (such as https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers) to make extra sure it’s not something standard.

Set up dynamic DNS for your router

Depending on your Internet Provider (ISP), your home network may get an IP address that never changes, or you might have one that changes every day. Even if it happens infrequently, Dynamic DNS can make connecting to your home environment a lot simpler.

Many consumer routers have a Dynamic DNS (DDNS) option built in. If your router does, this will likely be the easiest method. If not, you can set up up a service like ddns-updater on your home server. I personally run it inside a Podman container.

Once this is working, you’ll now have a convenient name to reach your home network instead of needing to look up its public ip every time you want to connect.

Connecting to SSH

With all of the above configured, you should now be able to connect to SSH from outside your network. Depending on your router, you may not be able to test this from inside your home network.

When I want to test external connections, I enable hotspot mode on my phone and connect the client computer I’m using to its wifi. If you don’t want to do this, it could also be a good excuse to visit your local coffee shop.

You can use this command to connect to the SSH server:

ssh user@external_ip_address -p ssh_port -i ~/.ssh/home_server

Enable RDP on your workstation

You need to enable Remote Desktop on the Windows workstation before you can use it. If you haven’t done this already, open Settings and search for Remote Desktop.

You’ll also need to know your username and password to log into Windows.

Connecting RDP to localhost

If your client computer happens to be a Windows computer, you have to use a trick to get around a weird limitation in the Microsoft Remote Desktop Client.

If you try to connect it to 127.0.0.1, you get this error:

Your computer could not connect to another console session on the remote computer because you already have a console session in progress.

Unfortunately, you really want to open sharing on localhost (as opposed to the wifi ip) so that only RDP connections from on your computer will be able to reach the SSH tunnel. This is especially important when using public networks like a coffee shop wifi.

One little-known trick is that all the IP addresses in the 127.0.0.1/8 block (anything starting with 127.) are all valid localhost IP addresses. The check in the Microsoft RDP client only looks for 127.0.0.1, so you can use 127.0.0.2 and get around it.

To forward the port through the SSH connection, you add this to the SSH command: -L 127.0.0.2:3389:windows_private_ip:3389

The first 3389 is the port that’s opened up on your local computer. You don’t have to use 3389 here, but it’s convenient. If you had multiple workstations inside your home network you can add -L multiple times with a different local port for each one.

The second 3389 is the port on the computer you’re connecting to. This will pretty much always be 3389, so don’t try to change it.

Putting it all together

Refer to Quick version near the top of this post for all the steps together.

With everything set up, you should now be able to work remotely with ease and peace of mind. Maybe it’s time to finally do some real work on your laptop from that coffee shop?

The 5500 can make a lot of drinks. I couldn’t find a description of what the differences are, so I made my own list. It took months for me to try every drink. This was partially because I prefer iced drinks in hot weather, but also because I didn’t want to waste good coffee.

Front Panel

More Drinks → Hot

More Drinks → Iced

All of the drinks with Iced in the name ask you to put ice in the cup before you can adjust the settings and press start.

One of the most significant differences between some of the drink variations is when milk is steamed, either before or after the espresso is pulled. All of the drinks which steam milk after the espresso are in the More Drinks menu. This is a bit annoying for me, since I prefer the Caffè Latte to the Macchiato.

(I realize complaining about the order of espresso and milk is pretty snobby, even for me.)

It was a fun experiment. I was most surprised by the Iced Latte setting. I thought it would be weird steaming milk into ice, but it worked great. The foam was nice and cold by the time the espresso was done, but it still kept a fair bit of its froth. This is now my favourite iced drink.

Minecraft has a built-in store with a wide assortment of extension, skins, and even whole worlds. Some of them are free, but most require some form of credit.

I spent a lot of time figuring out how to buy our first addon… It was far from straightforward, and I didn’t find any good resources that explained the process from beginning to end. So that’s what I’m going to share with you here.

My son uses Minecraft Bedrock Edition, running on a PS5. It probably isn’t so difficult if you use any other platform, but on the PlayStation, in particular, purchasing credits was a challenge.

Minecoins and Minecraft Tokens

Most platforms use Minecoins for purchases in the store. On PlayStation, however, you need to buy Minecraft Tokens. You can’t mix-and-match, or use one on the other. A lot of games are like this. It can be pretty confusing the first time you run into this, and they don’t go out of their way to tell you the difference when they ask for your credit card number.

As long as you buy credits from within whichever platform you’re using, you may not even notice this oddity. But in the case of PlayStation, since it was impossible to buy Minecoins for my son’s account, we had to find another solution.

Limitations of Child Accounts on PlayStation

All the gaming platforms have some kind of child mode protections, and PlayStation is no exception. This applies to launching games, but it especially applies to purchasing things in the PlayStation store.

Most of the things we purchase in the PlayStation store are straightforward. I purchase them with my adult account, and they become available for my son who uses the same console.

Any Minecraft Tokens I purchase, however, won’t be available to him. The only way for him to use them is to buy them with his own account. Unfortunately, because the game is rated for 10 and older, and my son is not yet 10, he isn’t able to access any Minecraft items in the PlayStation store.

For most things on the PlayStation, such as games and other content, I can control the level to which my son is restricted, and I can make exceptions for specific games. The store, however, appears to use his birthdate. There is no way that I’ve found to get around it.

The error message is incredibly unhelpful too. It took a lot of searching just to figure out what was happening. This is the error message he gets:

Content not found This content can’t be found.

Some people online suggested setting up fake adult accounts for kids to use, but this is messy. I don’t like to fool around with fake identities where money is involved. If there is ever a problem, it might be impossible to get help.

Sharing AddOns Between Platforms

You can log in to a Microsoft account from Minecraft on any platform. This turns out to be very helpful for kids accounts. As long as the add-on is available, it will be in your library on any platform you happen to be using.

This means that you can purchase Minecoins and Add-ons on a different device (such as a phone), and then use them on the PlayStation. The store on Android is much easier for my son to use. He clicks purchase, a message pops up on my phone, and I can approve or decline it.

The downside to this approach is that you need to have two copies of Minecraft, and since it isn’t free on either platform, this costs extra money. It was worth it for us. If you’d rather not purchase a second copy, I suggest you stick to a non-PlayStation version of the game where it’s easier to purchase Minecoins.

It took me hours and hours to purchase the first add-on for my son, but now that we’ve figured this trick out, it only takes seconds.

The digital display has two major advantages over the whiteboard. We can access it from anywhere, even outside the house. And it can include dynamic elements that are updated automatically, such as the weather, our calendars, and various bits of information from our home automation system.

The most useful and transformative feature has been the addition of a digital shopping list. I can quickly add an item from the kitchen panel, or from my phone, and my wife can do the same. When either of us is at the store, we can see the list and check off items as we buy them. I can’t tell you how many times we used to be missing a key ingredient because the paper list got lost or left at home. That doesn’t happen any more.

We’re using an older Android tablet for the display, which I don’t recommend. The tablet is connected permanently to a USB charging cable. On the recommendation of Everything Smart Home, I have it connected to a smart plug that toggles the power on and off to protect the battery. I’m not sure how necessary this is with the tablet.

I’m using the popular commercial Fully Kiosk Browser application on the tablet. It seems to be the best of class, and recommended my numerous people trying to do the same thing. I have found it to be a bit glitchy, but the application also reports that many features are not available because of the older Android version my tablet is running.

The dashboard itself is generated with Home Assistant, a free open-source application for home automation. I originally ran it on a Raspberry Pi I won in a contest, but I now it’s running in a VM on a Proxmox server that runs a bunch of household services. It worked fine on the Pi, but the SD card got worn down. Home Assistant is notoriously hard on its storage medium.

My dashboard includes:

• Four family calendars in a tabular layout (one calendar for each of us, and a shared one for combined obligations), and a couple of web calendars for public holidays and school events
• A link to the shopping list (with a count of items currently in it)
• The weather: current conditions and forecasts for the next few days
• A plot of indoor air temperatures over the last few hours
• An air quality reading from our neighbourhood

I’d like to put more information on the panel, and adjust how some of it is displayed. Home Assistant is extensible, but I’ve so far avoided writing my own extensions. Maybe one of these days I’ll take a crack at it. The information we have now is already useful for a bunch of situations.

Previous Attempts

Before Home Assistant, I used a home automation platform called Hubitat. This was a great platform for home automation, though it is both closed source, and has fewer integrations available. I did find it to be more reliable for automation. But also, the dashboard, at the time that I tried it, was hard to configure and looked pretty terrible.

On the tablet itself I also tried a free open source application called WallPanel. It was not very reliable. I’m not sure how much of this is because of the older Android version on my tablet.

Summary

Having a panel mounted on the wall is a great way to keep the family organized, but it takes a fair bit of work to set it up. It is absolutely useful, but as of right now, I wouldn’t recommend it for anyone but the most technically capable, or those with a lot of free time.

There are a lot of things I could write about, but I’m just going to share one Japanese word. It is my favourite word. If I could wave a magic wand and import it into English, I would do so with little thought.

The word is: Ganbaru

In Japanese, it can be spelled either as 頑張る, or phonetically as がんばる. You can look at the definition in Jisho.org for more info or the pronunciation, but it should be pretty easy for an English speaker to pronounce it well enough from its romaji.

Ganbaru is a verb that has a bunch of meanings all bundled together:

• Good luck
• You can do it
• I believe in you
• Don’t give up
• Hang in there
• Keep going
• Give it everything you have

It is normally used in its imperative form: 頑張って (ganbatte). You can add a suffix: 頑張ってください (ganbatte kudasai), which is like adding please in English.

The phrase can be very positive and friendly, as you would expect, but it can also be considerably more snarky, depending on context. If someone asks you to help with something crazy, it can just as easily (and still fairly politely) tell them that they are on their own. Kind of like “good luck with that” in English.

Now that you know my favourite word, and my hopeless wish to bring it into English, you have the perfect response to give me…

He asked me, “Is our app scalable?” After a moment of thought, I answered, “Yes.”

I spent the next few years remembering that conversation, wondering what would have happened if I had asked what he meant by scalable.

We used a cloud-based architecture with automated deployments. We could go a lot further scaling horizontally, and if that failed, we could easily generate new environments. If a lot of customers were to show up with cash in hand, it would be no problem for us to put them on the system.

That was, unfortunately, not what he meant. He was involved in a possible deal with a single giant customer, and he wanted to know if we could support them. That was a much harder thing for us to achieve.

He couldn’t tell me why he was asking. I also don’t think that one conversation caused any significant issues or miscalculations… but since then I am reminded of that moment any time someone uses the world scalability.

For that reason, I’ve been cataloging the different ways that the word is being used. Here are the meanings I’ve found:

Lots of users

I think this is what most people mean when they say scalable. In the land of B2C (Business to Consumer), this is certainly a problem that businesses want to have.

Cloud architectures can handle this kind of growth pretty easily. You need a database that’s twice as big? Sure, let me hit this button here and it’ll be provisioned in a minutes. Need a fourth server? No problem, it’ll be up in 5 minutes.

Compared to the olden times of dedicated rack space, this is magical. You can skip a lot of testing and planning when resources can be brought online in minutes instead of weeks.

It should be noted that the cloud providers charge a premium for this convenience, but if your income is doubling, it usually isn’t a problem to double your infrastructure costs as well.

Rapid addition of users

This is similar to lots of users, and similarly, cloud architectures can make it easier to handle, but there is a limit to how far you can push it.

For example, imagine working on a video game. It’s only been out for a week, so your player counts are still in four-digits. Now imagine your company has placed an ad in the Superbowl, and it is a success. Within a few hours you are approaching 6-digit user counts, and more people are trying to sign up.

At a small company, this would be a make-or-break moment. Your cloud architecture could help, but adding 100x capacity is maybe not something you can do with a couple of button clicks in the cloud console.

You could (and probably should) provision some extra servers before running a major advertising campaign, but you can’t go overboard there either. You have to pay for the resources you’ve activated even if the paying customers don’t show up.

This is a much more complicated problem to deal with, but there are a lot of things you can do. Here are a few off the top of my head:

• use serverless back-end technologies that scale automatically, such as document databases and cloud functions
• put as much of the logic into the client as you can. You only need to scale the components in the back-end, so the smaller they are, the less work you have to do
• lean on established third-parties for critical and sensitive stuff like payment processing and user authentication. Even if they take a bigger cut than roll-your-own alternatives, they can also reduce your risks.
• use more static content and less dynamic content. CDNs cost less than custom servers, deliver bits faster for a global audience, and are designed to handle traffic spikes
• use a smaller-scale test campaign before the big one to get a better idea of the potential interest in a larger campaign
• build in mechanisms that allow you to temporarily lower the server load if things get bad. For example, you could use a configuration service to increase client polling intervals, or temporarily disable more computation-intensive features
• add a waiting list at the starting page in case everything else fails

Saving money when traffic drops

A lot of B2B (Business to Business) software experiences the significant majority of its load between 8am and 5pm from Monday to Friday. B2C stuff can be spread out a bit more, but it tends to have natural rhythms too.

If you’re a fresh start-up, scaling up quickly is the priority. A few years into the journey, however, accountants start to ask about the cost per customer, and shrinking that number becomes increasingly important. Running idle servers in the middle of the night is wasteful, and you pay for it out of your profits.

Serverless cloud technologies can help here, as can environment orchestration technologies. Kubernetes is excellent at handling this, but it requires a significant investment of time and resources to use it.

Autoscaling is a simple concept, but there are lots of challenges to using it successfully. Orchestration code can have bugs like any software. Now imagine deploying a bug that adds an extra server every minute even though it’s unnecessary. Now imagine deploying that bug on Friday afternoon before a long weekend. How much money will you have spent before you check the operations dashboard on Tuesday morning?

These systems need checks, alerts, testing, and staff available to deal with problems. If you only need two servers for your entire customer base, you shouldn’t be worrying about this. If you’re spending millions-per-year on cloud resources, it’s probably worth hiring a team of IT experts to cut that down%.

Very big customers

As I mentioned in the introduction, I discovered this particular meaning in a memorable way. This is more of a risk in the B2B space.

It can feel really wonderful when a huge company shows up and offers to drop some serious cash on the table. If things are tight, it can be hard for executives to refuse this kind of deal. It’s hard even when times are good.

Unfortunately, large customers can be very different from small customers, and it can affect a lot more than your server architecture. For example:

• any entity in your data model that scales linearly with the customer size needs a UI/UX that supports it, such as:
  • drop-down lists with searching / filtering / paging capabilities
  • list pages with good filtering / paging capabilities
  • dashboard pages that operate smoothly when the quantity of data behind them jumps massively
• the queries powering routine reports might hit the database a lot harder when your biggest customer runs them
• if the customer can’t be naturally divided into pieces (ex, state, region, store), you need to make sure one environment can handle everything. If it can be split, you need to make sure some admin features work across multiple environments
• large customers often have more complicated requirements around user permissions and security controls
• if a customer makes up more than 50% of your company revenue, they can make it really difficult to refuse feature requests, even absurd ones

It’s hard to make an excellent product for a single audience, but it’s considerably harder to make an excellent product for two. When you support a bunch of small customers and one big one, you effectively have two audiences. This will strain your product managers, designers, developers, testers, trainers, and so on.

If you’re the kind of person who’s reading my blog, however, I assume this kind of decision won’t be yours to make… so if you end up in this position, you’ll have to find a way to make it work anyway. Good luck. :)

Sustainable practices

As a technically focused person, this one rubbed me the wrong way at first. I’ve had to accept that some people use the word scalable a bit more poetically.

That being said, it is still important to remember that the impacts of heavy utilization can be personal as much as they are technical. It doesn’t matter how much money the shareholders are making, at some point your employees are going to start quitting when they can’t sleep at night or take weekends off.

Summary

Shared vocabulary is a powerful way to communicate big ideas efficiently, but it always has the danger of communicating different ideas just as quickly. This is why we should use examples, use case statements, and diagrams to make sure we are understood clearly. If the difference between one meaning and the other can be measured in person-years of labour, it’s worth your time to make sure you’re talking about the same thing!

Many of the tools I use are free and open source, but some of them are not. Make sure to review and purchase licences where you need to. I’ve used most of these things fairly heavily, and am comfortable recommending all of them. It’s also easy to remove anything you don’t want to use.

I start with these commands because it makes the rest of the script easier to run. Windows Terminal is more comfortable for running lots of scripts, and BitWarden is my password manager of choice. Obsidian is the app where I keep my copy of these instructions, and Notepad++ is always helpful. Run this as administrator in whatever terminal application is available:

winget install --id chocolatey.chocolatey --source winget
winget install --id Microsoft.Powershell --source winget
choco install -y microsoft-windows-terminal bitwarden obsidian notepadplusplus

My next step is to config Windows Terminal and pin it to the taskbar. I make PowerShell (the newer Core version) the default. I also make a clone that includes the “Run as Administrator” option. This makes it a little easier to run administrative scripts. I also make these the first two options in the list, but it currently requires editing the settings file.

After this, I can do most of the setup with a few commands. I’ve used a group of PowerShell variables and expansion operators to group the apps I install. It’s better to run one choco command with everything than to run a bunch of separate ones (less fussing while it does its business). It should be straigtforward to understand how these work if you want to add/remove items or entire sections

Next, and now in Windows Terminal, PowerShell, as administrator, I run:

winget install -e --id Google.Chrome
winget install -e --id WinDirStat.WinDirStat

$hardware = @("razer-synapse-3", "logitech-options", "samsung-magician")
$utilities = @("Firefox", "restic", "7zip", "irfanview", "irfanviewplugins", "vlc", "winscp")
$editors = @("scrivener", "paint.net", "audacity", "inkscape")
$entertainment = @("spotify", "steam", "epicgameslauncher")
$development = @("git", "git-lfs", "dotnet-sdk-9", "podman-cli", "jetbrains-rider", "webstorm", "Python", "dbeaver", "beyondcompare", "nvm", "awscli", "jq")
$misc = @("zoom", "veracrypt", "calibre")

choco install -y @hardware @utilities @editors @entertainment @development @misc

# show hidden files and file extensions
Set-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced" -Name "Hidden" -Value 1
Set-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced" -Name "HideFileExt" -Value 0

# This disables the incredibly annoying Ctrl+Shift and Left Alt + Shift input language hotkeys
New-ItemProperty "HKCU:\Keyboard Layout\Toggle" -Name 'Language Hotkey' -Value 3 -PropertyType String -Force
New-ItemProperty "HKCU:\Keyboard Layout\Toggle" -Name 'Hotkey' -Value 3 -PropertyType String -Force
New-ItemProperty "HKCU:\Keyboard Layout\Toggle" -Name 'Layout Hotkey' -Value 3 -PropertyType String -Force

# Add Roaming binary folder (in OneDrive) to my Path variable. I keep a few utilities in this directory such as Procmon64, and a couple of PowerShell scripts I like to keep handy
$UserPath = [System.Environment]::GetEnvironmentVariable('PATH', [System.EnvironmentVariableTarget]::User) | Out-String -NoNewline
if ($UserPath -notmatch 'OneDrive\\Roaming\\WindowsBin')
{
    if ($UserPath -notmatch ';') { $UserPath += ';' }
    $UserPath += '%USERPROFILE%\OneDrive\Roaming\WindowsBin;'
    [System.Environment]::SetEnvironmentVariable('PATH', $UserPath, [System.EnvironmentVariableTarget]::User)
}

# More Development stuff
wsl --install
dotnet dev-certs https --trust

Install-Module -Force posh-git
Install-Module -Name AWS.Tools.Installer
Install-AWSToolsModule AWS.Tools.EC2,AWS.Tools.S3

git config --global user.email REDACTED@users.noreply.github.com
git config --global user.name Jesse
git config --global diff.tool bc
git config --global difftool.bc.path 'c:/Program Files/Beyond Compare 5/bcomp.exe'
git config --global merge.tool bc
git config --global mergetool.bc.path 'c:/Program Files/Beyond Compare 5/bcomp.exe'
git config --global core.editor "'C:/Program Files/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"

# Add some aliases and adjust some settings in my PowerShell profile
New-Item -Type Directory -Path (Split-Path $PROFILE) -Force
"
New-Alias npp 'C:\Program Files\Notepad++\notepad++.exe'
New-Alias bc 'C:\Program Files\Beyond Compare 5\BComp.exe'
New-Alias ll Get-ChildItem

Import-Module posh-git
`$GitPromptSettings.EnableFileStatus = `$false

Import-Module PSReadLine" | Out-File $PROFILE -NoClobber

After this, there are still a few manual steps I take. I could probably automate these, but I haven’t yet:

• Ensure BitLocker is enabled, and back up the recovery keys in my password manager.
• Set OneDrive to download all files locally
• Adjust keyboard layout settings as I like them
• Put a shortcut for Borderline on the Task Bar. It is a little utility that moves and/or resizes any windows that are outside or across monitor boundaries. This is unfortunately common when running an external display and multiple scale factors.
• Adjust Power & battery settings:
  • Turn my screen off after: 10 minutes
  • When plugged in, make my device sleep: Never
• Restore the following files and directories:
  • $HOME\.ssh
  • $HOME\Downloads
  • $HOME\AppData\Roaming\Notepad++\shortcuts.xml
  • C:\Windows\System32\drivers\etc\hosts
• Set up my backup system and test that it’s working
• Set a reminder to clean up backups for any retired systems after a couple of months

According to my records, I spent 28 hours working on the decision. This breaks down to about 22 hours testing the eight finalists, 4 hours researching options, at a summary level, and 2 hours of planning. Of the time spent testing, I spent between 30 and 240 minutes each for 7 of the 8 finalists. I spent 8 hours testing Obsidian, though most of that was testing and improving the processes I ended up using.

I started this task thinking purely about about improving my working notes and personal task list, but as I dug further in, it became apparent that I could centralize many of the different organizational tools I use. That process is still under way as of this writing, but I must say, it feels pretty nice every time I delete another productivity app.

Purpose

This document is a decision record explaining my choice for replacing my organizational system. Theoretically, it is intended for myself as a way to guide my thinking, but in reality, I am only writing it out in this form to share on my blog. The real document I used as the first draft for this post is similar, but a lot rougher, and with other unimportant details jumbled in.

Some of this text is inserted purely as commentary for the blog post, and would not normally be included in a decision record document. I have marked such text with italics, this paragraph included.

Context

• My personal backlog is spread across two email accounts, Trello, Remember The Milk, a few Evernote notes, and occasionally project management systems used by my customers. I would like to have one place where I can see all of my priorities so that I can easily find the next most important thing to work on, and to ensure important tasks don’t get dropped.
• I am spending a fair bit of time fiddling with cards in Trello, trying to update them, reset them, and change the dates on them. I don’t feel like I ever have a good view of my priorities there. I am also worried that important tasks are getting dropped because of it’s limitations.
• Remember The Milk is great for managing recurring tasks, but it doesn’t work for anything more complicated than simple chores. Some of my projects have a lot of research notes, open questions, and subtasks that are best kept all together. Trello can handle more complexity in a card, but I still find it limiting for a project that is more than a couple days of work.
• Keeping a backlog spread between notes in Evernote and Trello lists causes unnecessary work reconciling the two. I also end up searching both when I want to find notes from something I’ve done in the past.

Decision

• Try using Obsidian
  • Use a minimal number of vaults
  • Set up the Tasks plugin
  • Create a daily note template that includes incomplete tasks scheduled for that day and a view of my calendar
• Migrate one type of information at a time. Migrate new types as I get more accustomed to Obsidian.

Options Evaluated

Finalists

I tested each of these applications before choosing Obsidian.

The table below is what I call a decision matrix. I put options across the top, and questions down the side. I don’t particularly like tables for this sort of data, but so far, they’re the best solution I’ve found. A nice feature of this format is that I can keep expanding the list of questions as I learn more about the available options and the problem I’m trying to solve.

I don’t go out of my way to answer every question for every option. When an option stacks up enough limitations or blockers, or I just have a feeling that it’s not what I’m looking for, I focus my attention on something else. The tabular format makes it easy to see when I’ve done this, but also makes it possible to pick up where I left off if all the other options prove to be worse.

I like to use background shading in the cells (green, yellow, and red) to mark answers that are notable strengths, weaknesses, or blockers. For the sake of this blog post, I’ve added emoji to the cells instead.

Obsidian Alternatives Evaluated

Obsidian was a bit different from most of the other options that made it to the finalist stage. Once it began to stand out as a frontrunner, I did a quick search for apps similar to Obsidian, and tested the following two as a result:

• Amplenote - included in matrix above
• Affine.pro - disqualified pretty quickly because there was no mobile option and had a very small library of plugins

Initial Scan for Options

The following were discovered and briefly researched to decide if they should be evaluated further.

I only evaluated these options on a few criteria that were easy to look up. This allowed me to narrow the field quickly, as opposed to testing each one in detail. I went in interested in network-style tools, and excited by the prospect of something open source, extensible, and based on a transparent data format.

Because of the limitations of tables, and because I usually have a lot more options that questions in this phase, I’ve swapped the table axes from my usual for a decision matrix.

Summary

It was a bit of a process to do the research, but I’m glad that I did. Moving all my organization systems has the potential to be time consuming. The greater the cost of making a mistake, the more up-front effort is justified. This also gave me an opportunity to explore various options and styles and get better informed on how I could organize everything.

Now that I’m a few weeks into the switch, I’m still loving Obsidian. I have a few more things to migrate, but generally it has been better than my expectation in every regard.

I’ll never forget the day. About a week before the planned outage, we were in a big meeting of all the teams. We were talking through our shutdown script, moving things around and making sure everyone agreed on the order. It seemed like we could shut down smoothly if we followed the steps, and one false move could break everything. About 30 minutes into the meeting, someone came running in to announce that the whole system had just gone down.

We rushed back to our desks and started checking. It wasn’t all down, but a bunch of things had gone down at once, and one of those things was our main firewall appliance.

We reached out to the data centre operators to get more information. The planned outage was for an upgrade to one of the two power feeds, and they had electricians preparing the new lines in advance. One of those electricians had accidentally hit a live wire with his drill, knocking out one of the feeds.

Normally this wouldn’t be a problem because of the second, redundant feed. Unfortunately, this was when we learned that the data centre operators had not always connected each device to the separate power sources correctly. Our firewall, for example, had both of its plugs connected to the affected feed. Our database servers and NAS were powered with redundancy, though, so some parts of the system were able to keep working.

We immediately asked for the firewall to be fixed, and as we figured out each remaining server that was down, we asked for them to be corrected as well. Many customers in the data centre had similar issues, so the operators were overwhelmed, and responses got slower and slower. Things were almost working for us again when suddenly everything went down again.

Fifteen minutes later, we got an email (sent half an hour earlier) that the data centre had to be completely shut down. The building’s air conditioning did not have a redundant power supply, and the building had reached 60°C (140°F). Cutting the building power was the only choice they had; the operators weren’t able to safely enter the building anymore.

Several hours later, the data centre was restored. We didn’t know what state things were in, and the operators didn’t have a lot of time to help us, so they just turned everything on, and we took it from there.

It was a late night, but we had everything going by the end of the same day. Thanks to diligent administration, redundant storage, and our design to support regular incremental upgrades, we had no meaningful data loss, and most systems worked correctly on startup. The worst of it was a couple of behind-the-scenes services with interdependencies that had to be restarted a few times.

It was a sobering experience. We had a happy ending, but it could have gone very differently if one of the key systems hadn’t been designed or configured correctly. We had excellent, experienced people on our staff, and that helped a lot, but even experienced people make mistakes.

Regular testing is the only way to know for sure that your redundancy plans are good enough. If it’s something your business needs, you had better make sure you’re testing it. Using upgrade procedures that kill processes without shutdown warnings is a great way to make this a regular practice. Netflix’s Chaos Engineering approach, which involves routinely and automatically breaking things in production, demands major respect also. Recovery drills are tedious and time-consuming, but they too are an important part of ensuring system continuity.

About a year ago, after conducting the household organization surveys, I started using the whiteboard as a physical weekly calendar for my family. I used some thin black masking tape to make a grid that would withstand the whiteboard eraser. I also got some small magnetic dry-erase rectangles to simplify rescheduling common recurring tasks.

The biggest benefit of the mounted calendar has been the process of updating the board once a week. It forces me to look through all the calendars. This is when I’ll notice scheduling conflicts and allows me to make adjustments to keep the week flowing smoothly.

Although it’s the biggest benefit, the need to manually update the board is also the biggest drawback. Since we primarily use online calendars, the board can quickly get out of date when things change, and for us, they can change a lot as the week progresses. It also doesn’t include the weather, which is very handy when planning outdoor activities.

When the board was a novelty, my wife and I would regularly stop to look, but now that we’re used to it, it takes intention to notice it. This means we can’t use the whiteboard to communicate changes to the schedule.

The whiteboard calendar hasn’t been worthless, but it hasn’t radically improved our organization either. A larger family might have better results. If you’re curious, a piece of paper taped to the wall would be easier to start out.

The whiteboard itself has been good. It’s been great to have a central place to write notes, draw pictures, and otherwise play around. The marble tracks have been tremendously fun as well, and we can set them up over top of our schedule.

There is a lot of subtlety to working with time series data, especially if you are concerned with scientific accuracy. One surprising challenge, which I’ll be discussing in this post, is storing and transmitting the data efficiently.

At my previous company, the data we dealt with was relatively low frequency (typically a reading every 5-15 minutes), but some of the data sets went back into the 1800s! Although the data is simple in structure, the size can get surprisingly big when you need to move around millions of points.

Test application

I built a simple app to measure the transmission size of the data in the different variations discussed in this post. You don’t need to look at it to follow along, but you might find it interesting. The source code is available here: https://github.com/jessemcdowell/time-series-encoding-test

The test application is written in TypeScript, and uses node, Express, and Axios. It took some fiddling to be able to control compression, and to measure the transmission size (as opposed to the total size of the data, which would normally be more important). The numbers below also include other transmission overhead like headers and chunk encoding. This will skew the numbers a bit, but the results should still be more than good enough to illustrate the differences between approaches.

In application

A typical way to store a point in application memory is as an array of simple structures/objects. With an 8-byte integer for the timestamp, and an 8-byte floating point for the value, you can represent each point with 16 bytes. This will have enough precision for most use cases.

For one million points packed in an efficient array, this would consume 16 million bytes of memory. I’m going to use this as a baseline for evaluating other options.

This is an efficient, convenient way to handle the data in an application, but chances are you will want to write it to disk, or return it from a web service. It is possible to send this binary representation over the wire, but there are some drawbacks to consider. I’ll come back to this.

JSON

JSON is a common way to transmit data between server and client or server and server. A simple representation might look like the following:

[
    {
        "time": "2024-01-01T00:00:00.000Z",
        "value": 0.54188
    },
    {
        "time": "2024-01-01T01:00:00.000Z",
        "value": 0.32624
    },
    {
        "time": "2024-01-01T02:00:00.000Z",
        "value": 0.76939
    }
]

This is just three points, and you can see that it’s quite a few bytes. If you exclude the whitespace (included above for readability), it’s still 157 bytes with each point taking 52 bytes. This means 1 million points takes about 52 million bytes (just over 51MB)!

Here’s how it breaks down:

• 19 bytes - object wrapper, commas, and property names
• 26 bytes - time string and its quotes
• 7 bytes - the value

52 bytes is a lot more than the 16 we need in application memory. This is more than 200% overhead! It also has lower precision than the in memory version can support. It is possible to use a less precise timestamp format, or to use smaller names for the properties (like t and v). Those would help a bit, but we can do better.

CSV

CSV has a lot less wrapping characters, and doesn’t need to repeat the property names for each point. Here are the same values as the JSON example:

time,value
2024-01-01T00:00:00.000Z,0.54188
2024-01-01T01:00:00.000Z,0.32624
2024-01-01T02:00:00.000Z,0.76939

The total is 109 bytes (or 112 with CRLF line endings). The header is just 11 bytes, and can be omitted if desired. Each point is 33 bytes, which is better than the 52 of JSON, but more than double that of our 16 bytes in memory (106% overhead). 1 million points takes just under 36 million bytes.

There are other advantages to CSV as well. It’s easy to produce, easy to parse, and can handle multiple values or other types of data for each timestamp easily. It’s more readable in debugging tools, and can be consumed easily in many applications, more easily than JSON. It has capacity for extensibility via adding columns, though it can’t naturally handle structured data the same way JSON can.

Custom binary format

A time series can be stored or transmitted in a binary format very similar to the way it would be stored in application memory. You can encode it however you want, but my simple implementation takes 16 million bytes which is considerably better than both text formats discussed above. There is some added complexity to this approach, however. In particular, you will need your own versioning strategy if you ever need to change the structure.

You may also have some difficulty getting your web server and client to transmit and receive the data correctly. An easy way to get around this is to re-pack the binary data into something like Base64 and send that through a normal string property. This is especially helpful if you have a bunch of different time series or other related data you want to return in the same response. The drawback to this approach is that Base64 adds about 33% overhead, so the same million points from before would take 21.3 million bytes instead.

Unpacking precise timestamps in JavaScript

Another bit of complexity you might encounter with binary data, or any kind of data really, is the inherent limitation of JavaScript when dealing with 64-bit integers. For most applications this probably won’t matter. At my previous company, however, it was a significant and memorable challenge. Some of our features required quoting exact timestamps back to the server, and even a single millisecond of lost precision broke things.

Another challenge was different timestamp representations on the client and server. Our server (C# and C++ based) used a single 64-bit integer to represent 100-nanosecond intervals. Our client (JavaScript) used two number properties (one to store seconds, and a second to store nanoseconds). When unpacking the binary data, every timestamp had to be divided by 10000 for seconds, and the remainder multiplied by 100 for nanoseconds. Doing this with floating points in native JavaScript was very slow. It took some frustrating research and experimentation, but I got the performance gains I needed using a bit of Web Assembly.

Standardized binary format

Another option is to use a common binary format to transmit the data. MsgPack is very similar to JSON, but a bit smaller and faster. It also has the same drawback with time series data: it includes the key names with every single point. A MsgPack equivalent of the million points take around 28 million bytes. This is 76% overhead compared to the 16 million bytes in memory.

Protobuf (the binary protocol used in gRPC) does not include keys in the data, but it is harder to use. You have to define a schema that is kept is sync between your client and server. The same million points takes around 14 million bytes in Protobuf. This is a modest 11% less than the custom binary format, but with more safeguards and flexibility.

Compressed binary

Gzip compression is an easy way to reduce the size of the data being transmitted. Even better, this is a feature you probably already have enabled on your web servers and clients.

I tested a few configurations of compressed binary data to see how they performed. One interesting discovery was that the compression was noticeably better if I arranged the data in sets (time1-time2-time3-value1-value2-value3 instead of pairs such as time1-value1-time2-value2-time3-value3. I assume this is because of how gzip handles adjacent similarities. Here are the results for similar 1-million point blocks:

• Binary in any arrangement with no compression: 16 million bytes
• Binary pairs (time-value-time-value) with gzip: 10.3 million bytes
• Binary sets (time-time-value-value) with gzip: 8.7 million bytes
• Binary pairs, encoded in Base64, with gzip: 10.8 million bytes
• Binary sets, encoded in Base64, with gzip: 10.1 million bytes

I also tested MsgPack and Protobuf with gzip. Even though they are arranged in pairs, they performed slightly better:

• MsgPack: 9.8 million bytes
• Protobuf: 8.2 million bytes

One drawback to sets over pairs is that you would need all the points available to send them in sequence. If you had a very large amount of points and wanted to stream them incrementally, you would need to develop your own chunk-capable format. This is also a drawback of MsgPack which writes the size of each array at its start. I don’t know if Protobuf has this limitation too.

Compressed CSV and JSON

Compression is usually pretty effective wherever you have lots of repeating sequences, and that is certainly the case with JSON and CSV time series data. The same millions points packed in JSON took 7.5 million bytes when gzipped. That’s a compression factor over 7 times as compared to the plain JSON, and more than 2x better than straight binary!

A million points of CSV compressed to 6.9 million bytes, which is even better. The compression factor is not as impressive, but the overall size is smaller. This is the smallest of all the options in this post.

Downsampling

Another great way to reduce the size of a time series is to send fewer points. Downsampling (sometimes called Decimation) is a technique that excludes some or most of the points, and it can have a tremendous impact on the size of the data and the performance of your applications.

If you use this technique for data that is being graphed, the user might not even be able to notice. For example, If you are displaying a year of data in a spacious 1200-pixel wide area, you only have 3.3 pixels of width per day. If you have a point per hour, most of those 24 points won’t change the image at all and can be safely excluded. If you had a point every minute, you can exclude almost all the points!

There are a few ways to do this. The best technique will depend on your use case and the correct interpolation method for the data. I’m not going to get into those in this post. The result of downsampling is still a time series, so you should be able to use any of the techniques already discussed to transmit the downsampled data efficiently.

Conclusion

Time series data can be larger than you expect, but there are lots of ways to store and transmit them more efficiently. For most projects, I would start with CSV compressed by gzip, and use downsampling where possible. For cases with more strenuous requirements, there are plenty of other options to consider.

My testing and discussion were purely focused on the size of the data. Fewer bytes will generally lead to better performance, but in some situations, serialization speed or memory utilization might be more important. If that matters to you, you should work with an architect to find the best approach.

I quit my job with the dream of building my own product, but I started without any particular ideas about what to build. It didn’t take long before I was considering working on this exact problem. Not only was it something I would benefit from, but it was also something I would find interesting, and could enjoy building and maintaining by myself.

My first urge was to start coding. I had a clear idea of what I wanted, and my head was spinning with potential designs. However, years of experience have taught me the importance of starting with solid research, so I started there instead.

The first thing I discovered was that there are already several products specifically for this problem. I tried a few out, and while they were not all fantastic, a couple of them were pretty good. There were some differences, but they mostly all had the same core features: a shared calendar, a shared task list, and a shared shopping list.

Many had been around for years. Some were free. The fact that none of them seemed to be particularly successful was my first hint that there was a problem.

My next step was putting together a survey and sending it out via my social networks. I got 67 responses. It’s certainly not enough for any kind of scientific validity, but it was enough to notice some trends. I found many people who had the same kinds of problems I did, and most wanted to use technology to help. Desire is normally a good thing for an entrepreneur to find, but there had to be a reason that quality products that had existed for years weren’t working.

In the survey, I also asked participants if they would participate in a one-on-one virtual interview in exchange for a gift card. Most agreed. I selected a handful, and set up some meetings.

One of the people I spoke with had used and abandoned two of the best products I’d tried. I asked him about it, and his answers are the reason I decided to abandon this problem. He thought technology was a great idea, but he couldn’t make his partner use it. He tried using it on his own, but it didn’t help, it just became another chore for him to manage.

I had originally approached this problem like business software, but there is a fundamental difference between a workplace and a family: it’s easier to motivate an employee than a member of your family. It’s not enough to make something that appeals to one member of the family, a successful app would have to be intrinsically motivating to everyone in the household. This is an extremely high bar to clear.

If I take a holistic view of household organization, I don’t think another piece of software is the most effective solution. A shared calendar is the main thing a family needs, and there are some good free options. If you need more, there are a few free apps already. If you want to stay offline, you can use a whiteboard, or even a piece of paper hung on the wall. All that leaves for me is to show people how to use these things, but writing a self-help book or teaching a class isn’t what I want to work on.

The interviews weren’t a complete waste, however. Some of the people I talked to had very organized households, and those conversations were insightful. Here were a few ideas that stood out to me:

• Having a weekly planning ritual makes sure everyone knows what’s happening, and creates an opportunity to discuss issues. This seems to be especially helpful for larger households, households with multiple generations, and/or households with children old enough to contribute.
• Planning week by week (as opposed to day-by-day) simplifies the process and makes family routines more predictable.
• Having a calendar that everyone can see, be it through software or posted on the wall, gives everyone the same view of the week.
• Assigning chores to specific people, and making them accountable for those chores, helps to make sure that those chores get done.

The exercise of researching the market, talking to people, and learning about the problem was interesting and enjoyable. It cost me some time and a bit of money, but I have no regrets. Maybe some of it will help with my next idea.

Any online spreadsheet is a great way to share a small amount of data. Users can filter, sort, make formulas, export the data, or even write their own integrations. It’s also possible (depending on how changes are handled) to manually annotate or override data that’s been exported. Implementing all of this in a custom web page is possible, but some of it is pretty tricky.

The project was a success, and I’m happy with the results. Working with the Google Sheets API was not as straightforward as I had expected though. It was hard to find documentation, or even many examples showing how to use it. I’m not sure why this is. In this post I’ll share some of the things I learned.

Google.Apis.Sheets.v4 Package

The Google.Apis.Sheets.v4 package is the official .NET client library for the Google Sheets API. You can install it with NuGet.

It has official documentation, but I found it to be almost useless. I recommend the Google Sheets API documentation instead. It has much more information about how the API works and what the arguments mean. Methods and properties in .NET client library match the Rest API documentation exactly, I assume this is a result of the client library being auto-generated.

My first tip is to make sure you’re using the exact name for an API endpoint. For example, these two APIs, although seemingly similar, take different arguments, and behave totally differently:

• service.Spreadsheets.BatchUpdate()
• service.Spreadsheets.Values.BatchUpdate()

Another thing to be aware of is that the methods above don’t actually call the service, they create a request object. They almost look like they can be used in a fluent style, but this is only possible for the simplest get calls. I found it much easier to assign the request object into a variable so that I could set its properties directly. For example:

var range = "!A1:A3";
var change = new ValueRange
{
    Range = range,
    MajorDimension = "ROWS",
    Values = [["one", "two", "three"]]
};
var request = service.Spreadsheets.Values.Update(change, "SHEET_ID", range);
request.ValueInputOption = SpreadsheetsResource.ValuesResource.UpdateRequest.ValueInputOptionEnum.RAW;
var result = await request.ExecuteAsync();

Notice how I passed an array of arrays to the Values property? The API uses lists of lists for most values. This is necessary when the data includes multiple rows, but a bit annoying when updating single rows or cells. Fortunately it’s pretty easy to make an inline Array in modern C#, and an Array implements IList.

Each request object does have a Configure() method that allows you to set properties inline, but I found it even more verbose than working with the request object directly.

Authentication

There are a bunch of ways to authenticate to the Google APIs, but I found the choices overwhelming. For my purposes (a background service that needs to authenticate as a single user without a human present) a service account seemed like the best choice.

You can generate a service account in the Google Cloud Console. You’ll need a project. Also make sure to enable the Google Sheets API for your project. The UI in the console changes all the time, so you’ll be better off looking up instructions in the official documentation.

Once the account is created, you can download the JSON key file, and use this in your client code like this:

var credential = GoogleCredential
    .FromJson("<your JSON here>") // or use .FromFile() and pass the path
    .CreateScoped(SheetsService.ScopeConstants.Spreadsheets);

using var service = new SheetsService(new BaseClientService.Initializer
    {
        HttpClientInitializer = credential
    });

When retrieving this file, also take note of the service account email address (the property client_email in the JSON key file). You will need to share any google documents with this email address for the account to have access to them.

Spreadsheet, Sheet, and Range Identifiers

There are three kinds of identifiers you need to be aware of (documentation here):

• Spreadsheet - This is the top-level document identifier. You’ll find it in the URL for the Google Sheets Spreadsheet.
• Sheet - This is the tab within the Spreadsheet, also identified by the gid in the URL. It’s usually passed as the first part of the range (before the !). I’ve found that an empty string refers to the first (auto-generated) sheet in the document (gid 0), or you can use the name of the sheet. I haven’t had success passing the gid (instead of the name) even though the documentation says it’s supported.
• Range - Most of the API calls require a range. This refers to a rectangular group of Cells, and there are two ways to specify them:
  • A1 notation is the most familiar and most supported. For example Sheet1!A1:D1 will refer to the first four Cells in the first Row of the Sheet named “Sheet1”.
  • R1C1 notation is another syntax that can be used with some api methods. The above example would look like Sheet1!R1C1:R1C4 in R1C1 notation. It can be pretty handy when doing computed updates because you don’t have to convert the column index to a letter/letters. Unfortunately, it doesn’t seem to be supported by all the API methods.

If you need to compute the column letter(s) for a given index, you can use this code:

private static string GetColumnNameFromIndex(int columnIndex)
{
    const int numberOfLetters = 26;
    if (columnIndex >= numberOfLetters)
        return GetColumnNameFromIndex(columnIndex / numberOfLetters - 1) +
               GetColumnNameFromIndex(columnIndex % numberOfLetters);

    const char firstColumnLetter = 'A';
    return ((char)(firstColumnLetter + columnIndex)).ToString();
}

Update Types

There are three kinds of changes you can make to a sheet:

• Update - this is the simplest, allowing you to update a single range of cells. Use these sparingly. Batch updates will be much more efficient where they’re possible.
• Batch Update - this method takes an array of range update blocks and processes them all together.
• Append - This method allows you to stick rows on the end of the document. It requires a range, but it’s okay to use row 1. It will automatically stick the new values at the end of the sheet for you.

ETags

All change requests and get responses include an ETag property. In an ideal world, you would include the ETag from your get request when you send an update so that any changes that have occurred since are taken into account. Unfortunately, this property never seems to be set during get requests, and I couldn’t figure out how to make it work.

In all of my testing, the APIs were responding quickly enough that it likely wouldn’t matter. If you have a much longer process between gets and updates, you may need to figure this out. Please comment below if you do!

Performance / Update Size

I couldn’t find any documentation on the limits of the API, so I did some rough testing instead. I found that a batch update with 10,000 cell updates in it (each as a separate change in the request) took only 1.9 seconds. This goes way beyond what my client could ever need.

Conclusion

Google sheets are a great way to share data from an automated process. Although it took a little while to figure out how to read and write data at all, once I had it working, the Google Sheets API worked great. Even better, my client was thrilled with the results.

There are a number of IRC clients available. On windows, mIRC is popular and easy to set up, but it shows its age. IRCCloud is very convenient, especially if you want to use a mobile device, but it lacks some common features, and you are limited to two servers on the free account. There are other options, of course, but they all have limitations.

If you like doing things the hard way, you can use WeeChat like me. It’s powerful, has a bunch of plugins, and importantly, is still maintained. It also has a relay protocol that allows remote clients (like one on your phone) to read and respond to messages when you need to step away. It is, however, a terminal application, so it can take some getting used to.

WeeChat is built for Linux, but it is still very possible to run it on modern Windows. I’m confident you can get it working with the following steps.

Running WeeChat in a Podman Container

The best method I’ve found for running WeeChat is inside a Podman container. Podman is a container engine similar to Docker, but free on Windows. Setting it up to run WeeChat is rather easy:

1. (Recommended, but not required): Install Windows Terminal: winget install Microsoft.WindowsTerminal
2. Download and install the Podman CLI from the Podman Releases page. See the Podman installation instructions for more info.
3. Create a directory for storing WeeChat configuration and logs. I use $HOME/OneDrive/WeeChat in the examples here, but you can use any location you like
4. Run the following in Windows Terminal (PowerShell). Or better yet, save it as a script somewhere in your path for easy use later:

   if ((podman machine info | Out-String).Contains('machinestate: Stopped'))
   {
       podman machine start
   }
   
   podman run --rm -it --pull newer --env 'WEECHAT_HOME=/home/user/weechat' -v $HOME/OneDrive/WeeChat:/home/user/weechat --tz local weechat/weechat:latest-alpine weechat @args
   

And that’s it. You should now be running WeeChat.

Some notes about the startup script:

• The first block of the script runs the podman machine start command for you automatically if needed. This saves you a step the first time you run podman after each reboot of your computer
• The container is being re-created and deleted every time it’s run (because of podman run and --rm), and new images are automatically downloaded (because of --pull newer). You could create a container and start it each time, but continually recreating it this way ensures that you get updates automatically when new images are published
• the left half of -v specifies where data will be saved on your computer (outside the container). This is important, because everything else inside the container will be reset every time you run this script
• The --env command forces WeeChat (via a supported environment variable) to write everything (configuration and logs) into a single folder inside the container which allows the file mapping with a single -v
• (Added 2024-10) weechat @args at the end of the command line allows the PowerShell script to pass arguments through to WeeChat. I have been using this to sometimes send the --no-connect option when I want to prevent auto-connect or auto-join.

Why not use WSL directly

It is absolutely possible to run WeeChat in WSL, but I prefer the Podman approach because:

• it’s easier to expose the WeeChat Relay port - I found it difficult to share a port for an app in WSL to my local network
• you get updates to WeeChat automatically without running apt update; apt upgrade
• you are certain no state is being secretly left behind inside the WSL environment
• you don’t have to worry about breaking WeeChat if you break WSL, and don’t lose anything if you decide to reset it
• it uses less space. The normal Ubuntu image used by WSL is pretty big, which is wasteful if you aren’t using it for anything else

Using WeeChat Relay and a mobile client

One of the main reasons I went with WeeChat was the WeeChat Relay protocol and clients like weechat-android. This allows you to keep WeeChat running on your main computer, but access it remotely to check for or respond to messages. I find this handy when I need to do a chore or take a break. Because IRC depends on persistent connections, it is hard to use it (reliably) on a mobile device directly. And because WeeChat is exposing the relay, people don’t even know that you’re switching devices.

You can set up the relay service with the following steps. There are also detailed instructions in the documentation if you have specific questions.

1. Make up a password
2. Use the following commands in WeeChat to store the password, and use it for your relay server:

   /secure set relay new_password
   /set relay.network.password "${sec.data.relay}"
   

3. Use the following command in WeeChat to enable WeeChat Relay:

   /relay add weechat 9500
   

4. Run the following in PowerShell (as Administrator) to allow WeeChat Relay through the Windows firewall, and to proxy any traffic from the local network to the relay in the container:

   New-NetFirewallRule -DisplayName "WeeChat Relay" -Direction Inbound -Protocol TCP -LocalPort 9500 -Action Allow
   
   netsh interface portproxy set v4tov4 listenport=9500 listenaddress=0.0.0.0 connectport=9500 connectaddress=127.0.0.1
   

5. Add the following to the podman run command to expose the relay port from the container: -p 9500:9500. The complete command will now be:

   podman run --rm -it --pull newer --env 'WEECHAT_HOME=/home/user/weechat' -v $HOME/OneDrive/WeeChat:/home/user/weechat --tz local -p 9500:9500 weechat/weechat:latest-alpine weechat @args
   

6. Install weechat-android on your Android device. I haven’t found an iOS equivalent. If you have one you’d recommend, please post it in the comments.
7. Configure your client. You can use the following settings in weechat-android:
   • Connect type: Plain Connection
   • Relay host: IP address of the computer running WeeChat (inside your network)
   • Relay port: 9500
   • Relay password: The password you created earlier

As long as WeeChat is running on your computer, and the client is connected to the same network, you should now be able to connect to the relay.

If you want to access WeeChat relay from outside your home, you’ll have to set up some sort of vpn to your home network. I do not suggest opening this port to the internet directly without setting up TLS encryption first.

Setting up WeeChat Relay with TLS

WeeChat supports TLS for the relay protocol, which makes it much safer for connections going over the internet. However, it requires a DNS record that points to your WeeChat instance, and a self-signed certificate that matches the name. This appears to be a security limitation in Android itself, so you’ll have to deal with it if you want TLS.

For my own purposes, I stuck an A record in one of my domains that points to the local fixed IP address of my main computer. If you’re also using the relay protocol over your home network, you could use a dynamic DNS service like Duck DNS that allows you to specify the IP address manually. Dynamic DNS will also be helpful if you are opening the port to the internet from your router.

Once you have DNS figured out, generating a self-signed certificate can be done relatively easily.

1. Run the following in PowerShell to start an interactive shell in a temporary container:

   podman run --rm -it -v $HOME/OneDrive/WeeChat:/weechat --entrypoint sh alpine/openssl
   

   • Note: Make sure the directory mapping (on the left) matches the location on your host machine where you store your WeeChat data.
2. Run the following with the DNS name you set up:

   export HOSTNAME=yourweechatrelay.duckdns.org
   

3. Run the following to generate a self-signed certificate in the WeeChat directory:

   mkdir -p weechat/tls
   openssl req -x509 -newkey rsa:4096 -nodes \
       -keyout weechat/tls/relay.pem -out weechat/tls/relay.pem \
       -days 3650 -subj "/CN=$HOSTNAME" -addext "subjectAltName=DNS:$HOSTNAME"
   

4. Run exit to stop the container
5. Run the following in WeeChat to replace the relay server with a TLS-enabled relay server

   /relay del weechat
   /set relay.network.tls_cert_key "${weechat_config_dir}/tls/relay.pem"
   /relay add ssl.weechat 9500
   

From weechat-android, you can change the Connect type to “WeeChat SSL”. The next time you connect, you’ll be asked to verify the certificate. After this all communication between WeeChat and your client should be encrypted.

I like whiteboards so much that I bought a stack of them, and an artist’s portfolio bag to carry them around. I imagined popping out a few drawings at a meeting, or handing boards around for a creative session. I still love the idea, but in more than twenty years I’ve never even taken them out of the house.

These days I use a Rocketbook for most of my creative explorations. I received one as a gift at work a few years ago, and have been in love with it ever since. It looks and performs like a normal notebook, but it’s easy to scan the pages, and the pages can be reused after erasing them with a wet cloth. This is achievable because of some clever design and a specially optimized app.

If I’m working at home, or travelling, the Rocketbook is very convenient. It’s also good when I’m working in an office and need to sketch something. More than just replacing a whiteboard, it can be helpful for notes. Entering notes in a laptop is usually better, but if I have a couple of quick notes, or don’t have a laptop handy, the Rocketbook will do the job.

A whiteboard is still the best tool for an in-person meeting. Having the drawings out in the open where anyone can grab a pen and add to it is very convenient. Online meetings that involve collaboration don’t work very well with a whiteboard, or a Rocketbook for that matter. In these situations I prefer a virtual whiteboard. It does, however, also require some preparation before the meeting to prevent wasting precious time in the meeting.

If you’re interested in a Rocketbook for yourself, I suggest looking at the Rocketbook Fusion first. It has a mix of lined pages for notes and grid pages for drawings. I like the executive size for its portability. The only hitch is that you need a hard surface to write effectively. This isn’t a problem at a desk, but you may want to get a hard cover if you need to take notes on the move. If you’re doing a lot of design work at a desk, a letter-sized Rocketbook Matrix may be a better choice.

I’m going to be writing about the technology stack I’m using to run my company, and why I made the choices I did. There is a lot more to a company than its technology, but I’m not an expert in setting those up.

I made these choices for my company, keeping in mind my needs and values. I would probably recommend something similar for another small company, but it would depend. I didn’t perform an exhaustive search for each decision. Most options are going to be good enough, and even if I make a poor choice, I can replace most of these with a few hours of work while my company is small. I did put some effort into it though, and I’m happy to share that with you.

Here is some more of the context that guided my decisions:

• Pragmatic Potato Software is a small (single employee) software architecture and development consulting company based in Burnaby, Canada.
• I want to minimize the time I’m spending managing infrastructure because I’d rather spend that time helping customers and potentially earning income.
• I am pretty technically capable, and can handle setting up and hosting my own infrastructure. I also don’t need to worry much about usability: I am happy to use a code editor when I want to update the text on my website.
• I have modest expectations for traffic and customer interactions, so I should be able to use free or low-cost services for most things. I would like to keep my costs down.

Domain Registration: Cloudflare and Porkbun

For any kind of web presence, you need a domain name. Of all the registrars I’ve used, my favourite is Cloudflare Registrar. They famously don’t make any profit on their domain offering, so it is also the best price you can get. I used them for the pragmaticpotato.com domain.

Because I have a Canadian company, I also registered pragmaticpotato.ca, and have it set up to redirect to the .com site. At the time of this writing Cloudflare can’t register .ca domains, so I am using Porkbun for that one.

Website: Hugo static site hosted on Cloudflare Pages

I’ve written before about my preference for static websites. They are the simplest thing that can possibly work, and that’s often a good starting point. I can’t imagine anything I’d want to do on this website that’d require a dynamic back end, so this should be good enough for a long time.

I originally planned to use Hexo for the website because I already use it for this blog, but I had trouble finding a template that I liked for a business landing page. I ended up using Hugo. It is very similar to Hexo, but I found several templatse that I liked.

Having used both, I now prefer Hugo. One feature I appreciate is its ability to extend templates through partials rather than modifying the templates directly. This allows some pretty direct customizations, and should make it easier to take updates in the future. I used this functionality to tune the page metadata for search engines and social links. In the template I chose the templates was broken up into several partials to make these sorts of customizations trivial.

The hardest part of putting the website together was writing the text, and I went through several iterations to get it just the way I wanted. Then my friends told me it was garbage, so I rewrote most of it again!

I’m using Cloudflare Pages to host the content. It’s very similar to GitHub Pages (which I use for this blog), but I like that it’s a bit more configurable. I also found the build system a little easier to set up. I liked it enough that I will probably end up moving this blog over at some point too.

Contact Form: FabForm

I’m using FabForm for the contact form on the website. It is a simple service that was trivial to set up. I also find the one-time pricing much more reasonable than the higher (over time) monthly fees of other popular options. Using this allows me to accept messages on the website without needing a dynamic back-end.

Stock Photography: Unsplash and Squoosh

I used a few stock photos on the website to make it feel more engaging. Unsplash has a massive library of free, high-resolution photos taken by professionals. I made a point of including an attribution page, but that isn’t necessary with their licence. I used Squoosh to resize and compress the images for efficient load times.

I also hired a professional photographer to take some pictures of me to make the website more personal. This was a particularly fun process, and I’m thrilled with the results. The photographer was very reasonable. My new clothing was the most expensive part.

Website Analytics: Cloudflare Web Analytics

I am using Cloudflare Website Analytics for the website, mostly because it was easily available. I also appreciate that they collect minimal information, and in a way that doesn’t require popups to comply with modern privacy laws.

Email: Zoho Workplace

Microsoft 365 and Google Workspace are the two big players for email and office application, and I did consider each of them. I ended up going with Zoho Workplace because it is much cheaper and comes bundled with more stuff.

Setting it up was an easy process. Even setting up email for my custom domain was easy. I had it all up and working in less than an hour.

I don’t recommend trying to run your own email server, even for medium-sized businesses. Email used to be an open system that anyone could participate in, but with all the spam and especially with the spam prevention mechanisms today, it can be difficult to keep your emails out of spam boxes.

Document Editing & Storage: Zoho Workplace & Evernote

I’m gradually moving my (few) stored files over to Zoho WorkDrive. I was using a little bit of Google and a bit of Microsoft, but ultimately found their free options too limiting. The final straw for me was editing a contract. I spent a couple of hours trying to get the numbered lists and cross-section links right in the free online Microsoft Word and Google Docs before giving up. Zoho Writer has limitations too, but it proved the easiest to use.

Almost all of my work notes and planning content is in Evernote, which I’ve been using for many years. It’s not great for some kinds of content (like sheets), but it is extremely searchable, which makes it fast for pretty much all other information.

Task Management: Trello & Evernote

I’ve been using Trello for years to organize myself. I use it primarily to track the things I’m working on in the short term. It’s especially easy to slap cards into a list and put them in a sensible order. I mix personal and professional tasks in the same list to reduce friction as I go through the day.

For my longer term backlog, I’m using notes in Evernote. It isn’t a great tool for project management, but it’s good enough for now.

Video Conferencing: Zoho Meeting

Because I’m already using Zoho Workplace, I get Zoho Meeting included, and it works great. It looks and performs a lot like Zoom, but I don’t have to pay extra for it. It’s not quite as smooth as Google Meet (which has a particularly streamlined experience), but it has a few more features that I sometimes want.

Appointment Scheduling: Zoho Bookings

Zoho also has Zoho Bookings which allows customers to schedule meetings with me themselves. This can save a lot of emailing when someone needs to talk to me. I have it integrated with my Google calendar (which I use for my personal and family schedules), so it can automatically avoid conflicts with my other obligations.

Bookkeeping: Outsourced

Another key part of a business is keeping appropriate financial records. I spent some time looking into software for managing this, but I have decided so far not to bother. My accountant is happy to take care of this for me, and they can do it far faster than I can. Even though I’m the kind of person that likes to keep records up to date, and to see the numbers for myself, I would rather spend that time helping customers instead.

Cross-cutting concerns are the requirements that span multiple features. This includes things such as error handling, logging, profiling, security policy, and so on. We often want these things to be consistent, and we occasionally want to make systematic adjustments to them. Unfortunately, because of their nature, they can be challenging to centralize.

There are a bunch of tools and techniques for dealing with cross-cutting concerns, and I’ve made a list of them. I recommend thinking carefully about which approach(es) you want to use. It may be challenging to switch approaches once one has become widely used in your code base.

Once you abstract away some of these big requirements, you start to think about similar techniques every time you repeat a pattern. This is a deep rabbit hole to go down, but I can’t say it’s entirely wrong to plunge the depths. Things like classes that report changes, hash functions, even writing property getters and setters can be convenient to automate. This kind of code is often not fun to write, and because we often put little thought into it, honest mistakes can creep in. Worse, it’s eve more tedious to test these sorts of things, and the bugs they cause can be tough ones to reproduce and find.

I am focusing on the broader-scoped cross-cutting concerns in this post, but the same tools and techniques can be applied to many places where you need to implement the same behaviour across multiple bits of code.

Aspect Oriented Programming (AOP) tools

Aspect Oriented Programming is an approach that tackles cross-cutting concerns head on. The way they work varies by framework, but essentially they inject behaviour by either wrapping functions or modifying compiled code. You attach behaviours either using language features (ex: Attributes, Annotations), or through pattern matching (ex: namespace, class name, function name, function signature). There are multiple tools available in many common languages, so you should have no trouble finding something for your project.

When they work, these tools can be magical. Slapping a security attribute on a function is far easier than interrogating some security service and writing your own error messages. It also helps us keep our code cleaner: the account balance class does need to be secure, but security isn’t really a core part of the logic, and it’s nice to keep them untangled.

The problem with this approach is that it can be a little too magical. A developer joining the team could find it difficult to understand why some behaviour is happening where there is no apparent code causing it. It is also not always straightforward to debug this kind of code when it doesn’t work (but some tools perform better than others). It can be even more confusing when some shared logic is executed where it’s not supposed to.

If you’re going this route, there are two flavours to consider: some tools modify the compiled code, and others wrap objects at runtime (often in cooperation with your dependency injection system).

Wrapping objects at runtime is helpful because they can sometimes wrap code you didn’t write. They may also make it possible to dynamically add or change behaviours, either via configuration files or your own code. For example, you could inject a policy that wraps all database calls with performance counters, or do this only in a test environment.

Working at (or near) compile time can be a good choice too. Behaviour inserted this way has the benefit of access to the internal bits of your classes, making more powerful policies possible.

Extract into a helper function

If you don’t want magical code, you can still centralize repeated logic with your own helper functions. This gives you one place to make changes, and is easier for new developers to follow. For logic that needs to wrap your code (ex: profiling), you can write a helper function that takes a lambda or delegate. For example (implementation simplified for illustration):

public static void Main()
{
    Profile("Application Startup", () =>
    {
        // startup code here
    });
}

public static void Profile(string description, Action profiledFunction)
{
    var executionTime = Stopwatch.StartNew();
    try
    {
        profiledFunction();
    }
    finally
    {
        Log($"{description} took {executionTime.ElapsedMilliseconds}ms");
    }
}

The difficulty with this approach is that it can be deceptively challenging to design a good helper function. The implementation can be changed easily so long as you don’t need to change the function signature. For example, if you are writing your own logging helper, what arguments do you require? Do you want the class name? Do you want a stack trace? Do you want to support string formatting inline? Every additional argument adds more effort for developers using the helper function, however, if you don’t require enough information, you may not be able to make that desired change to your centralized logic without making expensive sweeping changes anyway.

This technique may be best for cases where you have a bit of duplication in a handful of areas. If your helper function is being called from hundreds or thousands of places, you may want a technique that is a bit more robust.

Use an established library

Picking a library and using it can be a good approach for common concerns. Logging is an example where this makes sense. There are lots of fantastic, well-established libraries available. They have mature APIs, tested and refined in thousands of apps. Every framework I’ve seen has been configurable for most common use cases. Many also support plugins for more specialized needs.

Logging is also the kind of thing that developers want to interact with: it can be helpful to throw a bit of extra information into the logs when you’re trying to track down an elusive bug. For comparison, if you are wrapping your code with logging using an aspect oriented framework, you may not have a natural way to pass specific information in to a log message.

Compared to aspect oriented programming, however, you do have to invoke the library whenever you want to use it. If you wanted to log all calls to your service layer, you would have to call your logging framework for all your service methods (or use one of the other automatic techniques in this list).

Another drawback to this approach is that you can become stuck with the library you choose because the cost of replacing it will be prohibitive. This may or may not be the kind of thing you have to worry about though. I have seen more than zero teams switch logging frameworks in an established project, but I’ve ever seen an example where it was necessary.

Use a common interface library

If you want the stability of a mature API, but really might need the ability to change your implementation, you could look for a common interface intermediary. There are lots of examples of community-maintained interface projects, and even some built into base frameworks. With these, most of your code gets implemented referencing the common interface, but the actual implementation is wired up through a plugin. Java’s Apache Commons Logging is a great example of this.

There are also more advanced interfaces that do more than wrap another implementation. OpenTelemetry is a fantastic library for recording telemetry, traces, and logs. Its greatest benefit is that you can add one or more plugins to expose this data to your various monitoring tools such as a privately hosted Prometheus system, or a commercial observability platform.

A common interface is a particularly useful tool when they are so widely used that common libraries start to use them too. For example, parts of .Net Core are now instrumented using Open Telemetry, so you can pick up valuable data in many helpful areas just by adding a plugin.

When choosing one of these interfaces, make sure you’re picking something that’s widely used in your component ecosystem. If you are forced to use two (either because you have components that use difference ones, or you decide to change), you might be able to make a simple bridge to stick them together. It is far nicer to pick the right library the first time though.

Another thing to be careful about is that these abstractions can sometimes be complicated. Using an abstract authorization interface could have a lot of bells and whistles you don’t need, or aren’t even supported in your implementation. The more complicated the interface is, the more likely you’ll find diverging usage patterns across your code, and they may not universally work. This erodes some of the benefits of using a common library in the first place.

Put it in a base class

Inheritance is the classic object-oriented way to share logic, and it can be very effective. Features like abstracts and virtual functions make it easy to deal with exceptions, which many approaches to centralizing logic have trouble with. It’s also quite helpful having the compiler to enforce the rules for you, making it safer to change the structure in the future.

The problem with inheritance is that where some is good, more is definitely not better. It is a powerful tool, but you can hit its limits pretty fast. Most languages only support single inheritance, and the ones that support multiple inheritance regret it. This means that you can only inject one set of behaviours into a class. If you want logging on some classes, but logging and authorization on others, you might need multiple layers of base classes to pile the features on or off. If you have a handfull of concerns, you could find yourself making lots of base classes to pick and choose where they go. This makes this technique generally unsuitable for the classic cross-cutting concerns. Another limitation of inheritance is sometimes it isn’t available to you: you might need your classes to derive from another type to participate in some other system.

Another challenge with using base classes for shared code appears when you have dependencies. You either need to include your dependencies in every constructor, or do some tricky stuff with statics to gain access to them. For example, if you are using a base class for authorization, you may need to pass a reference to your authorization service from every implementation of the base class. If you ever wanted to make a change that required a new dependency, you’d have to go and change the constructor for every implementation. This isn’t fun if it’s widely used.

This approach can, however, work well for small-scale repeating concerns. One example that comes to mind is as a foundation for a plugin architecture. Abstract methods are a great way to ensure implementations provide the entry points you require. If you go this route, I suggest keeping your utilities out of the base class to avoid versioning problems.

Put the logic higher or lower in the stack

Sometimes you can find a single place higher or lower in the stack to insert some shared logic. For example, for a common requirement to log errors in a web service, your web framework might support adding some code that intercepts every request. This code can check for and log errors before responses are returned. I generally prefer this kind of approach for error handling: logging general errors all over the place takes a lot of effort. It’s much nicer when you can let most errors bubble up the stack and know that the application will record and report them correctly.

You can also sometimes use this technique lower in the stack. For example, you might be able to wrap a database connection to add some generic profiling logic for your database interactions.

Use code generation tools

Some advanced IDEs have tools for generating code. This makes it easy to insert the exact same code over and over. It also makes it possible to share the patterns across a team.

The code that’s generated will be duplicated. This will make some kinds of changes to the pattern harder, but for some cross-cutting concerns that may not be as important. Even so, if the code is identical, you may be able to get pretty far into a sweeping change with carefully crafted compare-and-replace commands.

This approach may be best in places where you want some common code, but you can’t use one of the other more automatic approaches for whatever reason.

Generate boilerplate with AI-powered tools

We can quickly generate code with one of the new AI-powered tools, and they should have no trouble generating boilerplate for us along with everything else. This could seem like an attractive option for dealing with cross-cutting concerns, but I am skeptical.

Generative AIs can generate boilerplate, but they will be making it up every time they write code. You would need to provide appropriate instructions for the kinds of boilerplate you want, and remembering to do this is not that different from adding it yourself every time anyway. Even if you do, there is no guarantee that the generated code will be consistent. It may be mostly cosmetic differences, but the threat of a functional difference is always looming.

If you ever needed to change the pattern across your code base, you would have all the duplication, but subtle differences (even cosmetic ones) could make more automatic approaches impossible. Scanning all your code manually to change your error handling approach would not be fun.

It’s hard to say how these tools will perform in the future. Maybe they’ll get much better at including required boilerplate in a particular way. For now, I recommend avoiding this approach to this particular problem.

Use automated tests to enforce consistency

Sometimes there is no good way to share the implementation of a cross-cutting concern. In these cases, it may make sense to write an automated test that makes sure that the policy is present. It may sound a bit daunting, but it can actually be pretty easy to write a test that finds every class in your project and checks it. You may be able to use reflection to find and call methods. For some scenarios, it may be easier to write a test that looks at the source files directly.

Some concerns are so important that it may be worthwhile to write these sweeping tests even if you can implement the implementation automatically using one of the other techniques in this list. Authorization is a great example: you might want to test that all of your service classes fail with the right error if no permissions are present.

When writing these kinds of tests, I find it helpful to put a list of exceptions in a constant nearby. For example, you would want your login service to work for a user that hasn’t logged in yet. There may not be any other exceptions to the rule, but you can easily add them to the list if they appear.

Do nothing

Depending on the dynamics of your team and the kind of concerns your dealing with, it may not be possible or even worthwhile to use one of these techniques. For example, an open source project may not want to enforce strict coding requirements on unpaid volunteers. Automatic handling of a concern may be too complicated for most contributors to understand, or make it too difficult to set up a development environment.

So what happens in this worst case scenario? You get inconsistent implementation of a requirement, inconsistent implementations when the requirement is implemented, and it becomes harder to change the behaviour across the system. On the other hand, you don’t have the overhead of these approaches either. You don’t have any magic code, or any unexpected tricks hidden in middleware.

For something like a prototype, or for a small internal project that’s unlikely to grow into something bigger, maybe this is good enough.

Make your intentions understood

Good commit messages and review titles are important. It may be (and in fact, should be) obvious from your code change what you’re trying to do, but a good message is still important. There are lots of cases where someone needs to scan the change list, and good messages make this a lot easier. It also helps your reviewer understand what you’re doing.

The key to a good commit message is describing your intention. “Added some code” may be correct, but it’s not helpful. “Check for invalid email addresses” explains why you added the code. Your audience is a future developer trying to figure out why a feature broke, and what they care about is getting a quick picture of what you could have changed.

A little humour can be fun sometimes, but be careful with this. Small tasteful jokes are fine, but sarcasm or jokes that obscure your meaning will just slow others down.

Group changes by intention, one intention at a time

I tend to meander through changes as opposed to attacking them linearly, and I think this is pretty common. I run into small things while I’m wandering through the code, and I want to fix them right away. I could take a note and come back later, but this sometimes feels like moving backwards. The compromise that I normally go with is fixing the side-thing anyway, but presenting it separately.

If the change is very small (ex: a typo in a private method name - the fix will only affect a couple lines of code in one file) it may be okay to tuck it in. If it’s only a few changes but spread across a few files, putting it in its own commit is better.

The worst offender is the sweeping change. No matter how much a typo in a widely used function name might bother, adding a couple of hundred lines of noise to the review is absolutely going to slow it down. A new, separate review is a great way to present these. It’s a lot easier to verify simple changes if they are the only kind of change present. With Git and GitHub, it’s pretty easy to rebase a single sweeping change commit into its own branch and put it up for for review immediately.

Some “simple” fixes are not as simple as they seem, and could introduce a bug. If there is any doubt at all, it may be worth separating these changes so that they get separate scrutiny in the test and documentation pipelines. This usually means creating a new issue in your issue tracker. If they’re small enough, and if your tools allow it, you might still be able to include the change (in its own commit) in your same code review.

Break up large changes into multiple reviews

Sometimes a single change is big enough that you may want to break it into smaller pieces. For most cases, multiple commits may be enough, but for the largest changes you may want multiple reviews. This can be especially kind for a reviewer, or helpful if a change spans multiple areas of expertise.

One common and sensible rule is that every review should be able to build and pass tests before it is merged. That needs to be the first consideration when breaking a change down into smaller reviews. It can be temping to break up reviews by folder, or application layer, or some other simple delineation, but it will make it harder for reviewers to understand what is happening.

Avoid high-noise, low-value changes

Reordering code, rearranging imports, or running automatic reformatting functions can create a lot of changed lines in a review tool. Worse, it can sometimes be mistaken for removal and re-addition of code, making it harder to notice any changes made at the same time. This also makes it harder to track the true history for your lines of code. Sometimes there is enough value to justify the noise, but make the decision consciously.

Establish shared hygiene to reduce noise

Using an explicit .editorconfig is an easy starting point. Automatic reformatting tools can save a lot of time. The noise should be minimal if everyone follows the same rules.

There are some other practices that can help reduce noise:

• use trailing commas in lists (if your language supports it) - this prevents the last line from changing when a new line is added
• keep large lists in alphabetical order - this makes it easier to find code, and prevents the temptation to regroup and reorder things
• keep code files small (ex: one class per file)
• keep auto-generated code in separate files from custom code (ex: partial classes)

Rebase with care

Avoid rebasing when a review is underway. Sometimes a rebase is necessary, but it can disrupt the change history that reviewers are in the middle of processing. GitHub can sometimes be pretty bad with this. It’s okay to rebase before anyone has looked at your review. If you have no choice, see if you can delay the rebasing until all feedback has been addressed and accepted.

Review your work before asking others

Quickly scan your changes before sharing them with others. I have caught innumerable typos or accidental changes this way. You could leave them in the review, but it wastes time for your reviewers, and it makes you look sloppy. It can also add an unnecessary iteration to the review process if there was no other feedback.

Understand your review tool

There can be subtle differences in the reviewing experience depending on the tools and process your team is using. Take time to understand how your tool works, and make sure you follow the processes your peers prefer.

I’m not here to complain about companies taking back free offerings. I don’t like the change, and I wish they’d given me more than three days of notice, but they are a businesses, and businesses need to make money. It is a good reminder though: even if something is free to use, it still takes time and effort to integrate, to maintain, and you may occasionally need to throw it out and find a replacement.

Dependencies are inescapable in this business. We write programs using frameworks, written and compiled in computer languages, executing in operating systems that run on chips and boards designed and manufactured by others. It would be insane to build bespoke implementations of these for most purposes. Fortunately, these lowest level components are designed to minimize breaking changes.

Most software relies on a lot more dependencies, and that is often a sensible choice. There are costs to building things yourself even when they are simple: design, testing, and upkeep. Upgrading components to get security updates is a pain, but making your own vulnerabilities that never get detected or fixed has drawbacks too.

So if there are drawbacks to depending on third parties and drawbacks to building everything yourself, what should you do? It depends! Either way: Before adding a dependency, consider its maturity, stability, and how actively its supported. If the choice is hard to change (ex: a logging framework that will be referenced throughout your code), check again, and harder! Before building it yourself, consider the true cost, including testing and maintenance.

When I talk about being honest, I actually mean being candid. Candor means both being honest and also communicating in good faith. Many fantasy authors have written about characters who never lie but are also fundamentally dishonest. Candor eliminates this loophole. I push myself to communicate everything that feels important, even if it’s uncomfortable or to my own disadvantage.

Why would I do this? Honestly, I don’t know. I learned about candor from a business book, and the idea of it resonated so strongly with me that I decided to make it part of what I am. It has been a difficult road, but it has also been good for me. Saying the difficult can sting in the moment, but it builds trust over time. Trust is very important when working in a team. It’s helped bring attention to problems. It’s also helped me to strengthen the important friendships in my life.

Being honest may not get you a lot of friends but it’ll always get you the right ones. – John Lennon

Being honest is hardest when emotions are involved. Sometimes I am worried about hurting feelings, or sometimes I am frustrated or upset with the way things have gone. In these moments it is easier, and in our culture sometimes even expected, that we say something polite and positive even if we feel otherwise. Even if it’s well-intentioned, this approach diminishes trust and allows problems to fester. Even worse, these hard feelings can amplify the longer we keep them unvoiced.

Blurting out our first thoughts is not a winning strategy either. Care needs to be taken to make sure what we say is honest, but also polite.

The way I do this is to re-frame my opinion so that I see the situation in a constructive way. I’m not pretending that I feel that way, I convince myself to feel that way. This is essential to being candid and professional at the same time. If this sounds hard, you’re right, it is. But I believe it is absolutely worth the effort.

Here are some questions I can ask myself to find another opinion:

• Am I misunderstanding the intentions of the other person?
• Is the other person pushing beyond their comfort zone?
• Is the failure a result of a system that allows mistakes to go undiscovered?
• Is it possible the other person has personal circumstances that are influencing their capacity?
• Are my feelings a result of my own personal baggage?

With practice, this gets faster and easier, but sometimes it’s harder. Stepping away for a bit can help. Talking about it with someone else can help. Or sometimes I need to consider the source of the feelings and what that says about me. There are also ways to express being upset or disappointed sincerely that are less offensive.

Sometimes feedback is best given privately. I had a boss who often said, “Give praise in public and criticism in private.” I think this is good advice for the most part. People can get defensive and upset if they feel like they’re being shamed. That will not improve the working environment. If you are giving feedback, make it as constructive as you can. Focus on actions that happened and their consequences, giving concrete examples whenever you can. Generalizations and judgements about the other person should be avoided. A lot has been written on this subject by people who are much better at it than me. If you are in a management position, I highly recommend taking the time to study and practice these skills.

There are times when feedback goes sideways and tempers can come out. This is a good time to take a break; 30 minutes for the nervous system to calm down is great. Make sure to talk in private, and talk in person if you can. It isn’t fun when things get to this point, but it can happen when people care about what they’re working on. It isn’t fun, but it is a good sign.

If things go really wrong, make the effort to apologize once the dust has settled, and try to move through it. Repair also builds trust and improves relationships, so even if you get it wrong sometimes, don’t lose hope.

There are unfortunate times when being honest isn’t possible. I try very hard not to keep things secret just because it’s easier, but sometimes there is no choice. One example that comes to mind is hiring and firing decisions. These situations can be hard, but they are unavoidable when you get to a certain level of influence in an organization.

Operating this way at work and in life has been difficult, but it has also been very freeing. I don’t have to carry the baggage of the truths I haven’t shared, and I attract the kind of people who want honest feedback. It has helped surface more problems and make my life and my work better. I have no intention of changing.

How I switched

It seems pretty reckless in retrospect, but I was young then, and it seemed like a good idea. So one day I decided to go for it. I changed the keyboard setting and dropped suddenly from about 60 words per minute to less than 1.

If that wasn’t jarring enough, I decided to also break my hunt-and-peck typing habit at the same time. I had an older mechanical keyboard with a uniform key size, so I was able to re-arrange the key covers. I moved all of them into random positions except F and J because of their little bumps that are essential for touch typing. This way I had no choice but to memorize the new positions. It was disruptive and frustrating, but it was also very effective.

I was up to a workable speed after a couple of days. I could match my QWERTY speed after a couple of weeks.

Once I was comfortably using the new layout, I moved all the key covers to their positions in the Dvorak layout, including F and J. A dot of epoxy on the new starting keys (U and J) allowed me to keep touch typing.

Faster typing & reduced wrist strain

I can type faster with the Dvorak layout, and considerably faster than most of my peers. I don’t measure often, but I have recorded myself at 160 words per minute. I can only approach these speeds when I’ve been writing a lot of text for several days in a row, which hasn’t happened very often. It turns out that typing fast doesn’t actually help me much at all.

Almost all of my typing as a programmer is in short spurts. An email here, a chat message there. Writing source code is not much faster on Dvorak because of all the symbol characters used. I am not a slow programmer, but typing speed only becomes a limiting factor when I’m writing the kind of low-value boilerplate code I should be avoiding in the first place.

The most helpful benefit I got from switching was learning to touch type, but that doesn’t require a switch to the Dvorak layout.

The biggest benefit I get from the layout itself is reduced wrist strain. The more efficient layout means my fingers don’t need to move as much. It’s hard to quantify the improvement, but even as an invincible teenager, I noticed it immediately.

Availability of Dvorak layouts

When it comes to using a computer, the Dvorak layout has been implemented pretty much everywhere. Even in the early 90s, it was readily available on Windows, Linux, and Apple systems.

There are also keyboards that can do the mapping at the physical level. The one I tried had a switch that alternated between QWERTY and Dvorak. The computer it was attached to would have no idea when you were using Dvorak. Although these eliminate some mapping problems, I don’t find them that helpful. I also don’t want to carry a special keyboard around wherever I want to type.

I have tried printing the Dvorak layout onto my keyboards, but this has also been impractical. I have yet to find an ink or paint that can withstand more than a month of typing. I don’t look at the keys when typing, so even if I did, it wouldn’t help me. It might help a stranger using my computer, but it would still be excruciatingly difficult for them. If someone else needs to use my computer the first thing they should do is change the layout back to QWERTY.

I don’t bother with any of this nowadays. I have a normal QWERTY keyboard on my laptop, using just the Dvorak setting in the OS.

QWERTY is inescapable

There are keyboards everywhere nowadays, physical and virtual, and almost all are in the QWERTY layout. I have one on my phone, one on my TV’s menu, one on my gaming console, and even one on my printer. Most of these devices have no way to change the layout.

I occasionally want to use a computer that isn’t my own, such as when I’m pair programming, or helping my dad with his computer. I could change the layout in these cases, but it’s often too much of a nuisance.

For these reasons, I had to re-learn how to type on QWERTY shortly after switching to Dvorak. I’m unsure how or why, but I developed an ability to use both depending on where I’m looking - QWERTY if I’m looking at the keyboard or Dvorak if I’m looking away. It’s as if I have two different sets of muscle memory for typing and my eyes control which is engaged. This has been a benefit, but it causes some weird side effects too.

The most bizarre is that I find it nearly impossible to use shortcut keys one-handed because I have trouble finding the Drovak letter if I need to look at the keys. Apple computers have a layout called “Dvorak - QWERTY ⌘” that switches the layout back to QWERTY for command sequences. It solves the problem, but it isn’t available in Windows.

I use the QWERTY layout on touch screens like my phone for the same reason, even though the Dvorak layout is often available. It was even worse back when I was using a Microsoft Surface; because I sometimes had a physical keyboard, I had to change the layout back and forth whenever I needed the touch keyboard.

Cut, Copy, and Paste

One of the most infuriating challenges with the Dvorak layout has to be the shortcut keys for Cut, Copy, and Paste. X, C and V are excellent keys for QWERTY users because you can press them easily using only your left hand. This is especially great if you use your mouse with your right hand. On the Dvorak layout, C is where the QWERTY I is, and V is where the QWERTY period key is. This is a significant disadvantage.

I can switch back to QWERTY temporarily if I have a lot of pasting to do, but I’ll invariably need to modify a couple of words here or change the punctuation there. Switching back and forth constantly doesn’t end up being easier.

Login screens and shared sessions

Login screens on shared computers can be a real pain. It was an advantage at first - it was a fantastic way to stop my sister from using my computer when we lived with our parents - but it is much less useful in the workplace.

The behaviour is different in every operating system and often inconsistent. For example, if you lock your session in Windows, the unlock screen will use the keyboard layout of the session you locked. If you reboot your computer instead, the login screen will use the system default.

Most modern operating systems don’t require typing a username when logging in. This means the only characters you type are masked password characters, so you often can’t tell when the layout is wrong. Newer operating systems show the layout on the login screen, but the indicator is too subtle, even when I know to look for it. I often don’t check the layout until after one or two failed attempts. In a strict workplace, it doesn’t take very many before an account gets locked.

Another variation of this crops up when connecting to remote computers or virtual machines. Every technology can be different, but most transmit the hardware key codes rather than the ASCII character values. This means you must change the OS-level layout setting on the system you’re connecting to.

Windows Remote Desktop (RDP) sets the layout for you when you start a session. It takes the value from the session where you start the RDP client, which is usually helpful. However, it doesn’t change automatically when you connect to an existing session, nor does it switch back to the system default when you leave a session idle. If I’m sharing an account with someone (for example, a test server on a local network; sharing accounts in production is bad!), I could get either keyboard layout when I connect. I am used to dealing with this, but it has wasted a lot of time for my colleagues when they’ve intercepted my old sessions.

Video games

Some video games suffer from a similar problem. Games from the olden days tended to work on hardware key codes, but newer web-based games usually use ASCII codes. A and D for left and right is universal, but it sucks on a Dvorak layout. The A is in the same place, but the D is over in the same position as the QWERTY H. Don’t even get me started on W and S. The best way to deal with this is to change the keyboard mappings in the game, but I usually switch the keyboard layout back to QWERTY because it’s easier.

Video games with in-game text chat are the most difficult. I need to be in QWERTY mode to play, then switch back to Dvorak mode to write a message. To make matters worse, the newer language/layout switcher (Windows + Space) thrusts you out of full-screen games because it has a dialog window. Some games handle both gracefully (using ASCII codes for text and key codes for gameplay), but it’s not universal.

Japanese IME and the Dvorak layout

Another unusual drawback appeared when I started learning Japanese. Japanese mixes 4 alphabets: the Latin alphabet, two phonetic alphabets with 46 characters each, and about 100,000 characters imported from traditional Chinese. A keyboard with every possible character would be obviously impractical, so text is written phonetically and converted as you go. This is done with an IME (Input Method Editor), an operating system feature that intercepts keystrokes and translates them into the appropriate characters.

Because the IME is a kind of keyboard itself, it replaces the keyboard mapping. This means there is no built-in way to use the Dvorak layout with Japanese writing. It used to be possible to change a registry key to replace the keyboard layout, but this option seems to no longer work with the updated Japanese IME that ships with Windows 11. There are instructions available to use the old IME, but who knows how long that’ll be available.

Typing Japanese via the QWERTY layout is painful for me. The IME is constantly popping up suggestions, and since I can’t touch type on the QWERTY layout, I have to keep looking up and down to use it properly. This is no longer much of a problem for me since I’ve stopped practicing Japanese, but it would be if I ever picked it back up. This is one of the few reasons I would consider a physical Dvorak keyboard.

Final thoughts

I am surprised by what I’ve written. I had imagined a small blurb about how I switched and then re-sharing the Japanese IME registry hack (which I’ve since discovered no longer works). I did not realize I had so much to say about my experiences! Would I recommend the Dvorak layout to others? No, I would not, and it surprises me that I say that. I can’t even think of a situation where the benefits would be important enough to justify the disadvantages.

Would I recommend Dvorak users switch back to QWERTY? I feel a sadness I don’t understand when I consider the answer for myself, but I think the answer is yes.

With a bit of intentionality and the right support systems, code reviews can become a catalyst for positive change that are absolutely worth the time we put into them.

Why we do code reviews

There are a lot of reasons for code reviews, but the one I find the most valuable is that they promote shared ownership of the code. Sharing code forces teams to get aligned on architecture and coding practices. This gives you more flexibility to distribute work, reducing bottlenecks and guarding business continuity. The developers who work on your critical systems will appreciate the ability to take vacations as well.

Another advantage is that reviews improve code quality simply by being part of the development process. Developers can be lazy people (ask me how I know), but we get a little extra incentive to tidy things up when we know someone else will be looking. Since we spend much more time reading our code than writing it, this will improve efficiency in the long run.

Reviews can also be a more efficient way to onboard developers, especially for the kinds of knowledge that are hard to articulate. For a small team with little planned growth or churn, writing a huge body of documentation may not be a great use of time. Of course, learning through review feedback doesn’t always feel very nice when you’re on the receiving end, but this is another reason why it’s valuable. Making it normal to receive constructive feedback without shame is essential if you want a team where members push themselves and grow.

First, automate everything you (effectively) can

One simple way to reduce the time you spend on code reviews is to automate as much as you can. Some things require a human touch, but some other things are much easier for computers.

Compilers are typically very easy to automate, and it can take humans lots of time to detect compilation errors. If you automate only the compilation of any proposed changes, you can prevent a lot of build breaks and wasted time. This is helpful even with a team of one.

You should be running your unit tests automatically as well. Not only will this make sure any new tests pass, but it’ll prevent failing tests from getting merged back to the main codebase, which can in turn disrupt other developers.

Longer-running automation tests or e2e tests may be too slow or expensive to run on every proposed change, but you should consider the tradeoffs. Even the most well-intentioned team can find it irresistible to ignore a failing test when a tight deadline is looming. Once you have a couple of tests that fail regularly, it doesn’t take long before confidence disappears and decay sets in.

There are lots of tools for checking basic code formatting and whitespace. These are also good to automate and should be set up early in any project. I have seen so much time wasted in reviews talking about white space and proper nesting. Let the machines give feedback in minutes instead of wasting developer time on it. Make sure you have an .editorconfig that matches the rules you’re checking so that your IDEs get most code correct before the review starts.

What to look for when reviewing code

I try to focus mostly on readability and maintainability. Do I understand the intention of the change? If I needed to change this code in 6 months would I be able to find it and figure out what it’s doing? Do the names being used describe what they are?

I don’t check if the change meets its requirements unless I happen to notice it. Code is not a great level to test functionality at, and I expect the author to test this themselves, and a further manual test to be done once the change is in a build. I do check that any commit messages adequately describe what the change is.

I also like to check that a good number of tests have been written, and any complicated or risky code has sufficient coverage. If I’m honest though, I don’t typically check the contents of test methods. I find most tests difficult to read, and I expect the developer to have checked that they work already.

Self-editing and maintaining a positive culture

Before I post a code review comment I like to ask myself a few questions:

• Is the change important? Have I communicated that?
• Is it clear what needs to be changed?
• Is it reasonable to expect the author to make the change?

At this point in my career, I find that a lot of my code review comments are more instinctual than formed logically. I don’t think it’s a bad thing to operate this way, but instincts can drift or misfire sometimes. Even if something feels wrong it might not be a problem, or it might not be important. I use my checklist to validate my instincts. It’s not uncommon for me to add a bunch of comments while doing a review, then go back and delete some of them before hitting the send button.

Sometimes I have concerns about the underlying approach or the broader scope of work the change is a part of. In these cases, I check myself carefully to make sure my feedback is important. If I don’t want to be a part of every decision made by the team, I have to trust the team to make them without me. Big changes are also more expensive, and the code review step is pretty late in the process. Sometimes the stakes are high enough though.

As with all business communication, it is important to be mindful of tone in code reviews. I always try to assume positive intent and try to reflect that in my comments. If I have a lot of feedback or serious feedback, I might speak with the author directly. A quick face-to-face or video chat makes it easier to diffuse hard feelings.

Another thing I avoid is the “I would have done it like…” pattern. At best it’s a poor justification for a change, at worst it implies superiority. It is much better to use “I suggest doing X because Y”.

Suggesting alternatives

The easiest way to ensure a comment is reasonable is to provide an alternative myself. For example, I won’t leave a comment about something’s name unless I can offer a better name, and give a good reason why the old name doesn’t make sense.

I also like to use the diff feature in GitHub comments to show the actual change if it’s reasonable to do so. This often doesn’t work for naming (because there will be references that also have to be changed), but it’s great for little stuff like typos in user-facing strings. It is also sometimes the easiest way to communicate an improved algorithm or propose new function signatures.

Number of reviewers

Sometimes it’s important to get lots of eyes on a critical change. Sometimes it’s not important but people do it anyway. Regardless of the reason, it can be especially painful to get code approved when multiple reviewers have to approve. In these circumstances, and especially when I’m late to the review, I temper my feedback further than usual so that the code review won’t go around in circles forever.

The worst is when two reviewers disagree on the best approach and argue about it in the code review. Whenever this happens I ask the two reviewers to talk to each other directly to resolve the issue. It can be beneficial for the author to join in as well.

I generally recommend just one reviewer for most code reviews as this gets the most benefit with the least cost. I also suggest making sure the whole team is doing reviews. You could assign them all to the most senior member of the team, but doing reviews is a valuable teaching experience and not all code requires an expert opinion. If a code review does require an expert opinion, I prefer to ask those experts to focus on specific areas but let someone else handle the broader review. Having new team members join code reviews is also an easy way to expose them to team practices and any work in progress.

Don’t say anything if you don’t need to

Sometimes the code is good enough, especially if the change is small and clear. If it’s good enough, resist the urge to nitpick and mark it as reviewed.

Accepting feedback on my feedback

When someone puts comments on the code I’ve put up for review, I take those comments seriously. Sometimes I don’t agree, but I put in extra effort to make sure I maintain an appreciative and curious tone. Even bad feedback is often grounded in good reasons. It might be a hint that another other part of the change isn’t clear. If I believe the comment is an error, I respectfully point it out and provide references if I can.

When I’m reviewing, I often make suggestions without testing them. I also go from memory, and sometimes I make mistakes. When someone challenges a comment of mine, I take it just as seriously. This isn’t a sign of disrespect, it’s just the nature of working on hard problems in a team.

I have been using Spotify happily for music for years now, so I decided to give it a try. I didn’t do any research or look for other options. I had some confidence that it would be a reasonable option however because Spotify has been putting a lot of effort and money into its podcast offering.

Is Spotify a good replacement for Google Podcasts

Short version:

If you want something that will work on a wide variety of devices, has a huge library of content, and is likely to be around for a while, then Spotify is a good choice. If you want a simple replacement that behaves the same as Google Podcasts, you may have better results elsewhere.

Long version:

In terms of the availability of the app on multiple devices, availability of podcasts, and sound quality, Spotify performs excellently. They are a huge company with lots of content, and everything works great.

I can listen to podcasts on my phone, on my computer, in my car (via Android Auto), or on my kitchen speaker (via my Google Nest Mini). I can even listen to part of an episode on any of these devices and finish it on another device. It has remembered my position every time, something which was often not the case with Google Podcasts.

The biggest disadvantage has been the disruption of my listening workflows.

When I wanted to listen to a podcast on Google Podcasts, I would open the app, pick an item out of the Queue, and play it. When it finished, it would usually (depending on how I started it) start on the podcasts at the top of my Queue. If I wanted to listen to a few in a row without touching my phone (for example, while working in the garden), I could pick the next couple podcasts I wanted to hear and put them at the top of the list. Spotify’s “Your Episodes” playlist does not allow custom ordering, but I have found another way to do it.

The other annoying tidbit is that using the Spotify app or a Podcast app isn’t enough to specify what I want to hear. If I was in the middle of a podcast when I get into my car, I’ll be listening to a podcast there too. It’s not a big deal on the phone, but it can be pretty annoying to change this through the Android Auto interface.

How to migrate your data to Spotify

It took me less than 30 minutes to migrate my subscriptions and queue manually. There is supposed to be an export added to Google Podcasts, but I assume it’s not yet available because I couldn’t find it. I also never found an import in Spotify, so maybe it wouldn’t matter.

To do it manually, just search for and click “Follow” on any podcasts you had subscribed to. Next, search for and click the plus button on any episodes that were in your queue. I suggest using a computer for this, or at least two different devices. If you didn’t already know: Google Podcasts has a web version that gives you access to everything you’ll need.

My Spotify Podcast workflow

When I want to play an episode, I use “Add to Queue” instead of the play button. I can then use the next track button to switch from my current song to the podcast. This makes it easy to queue up a couple of specific podcasts, and makes the player switch automatically back to music when they are all finished. If I want to switch immediately back to music, the next track button will save my place and move on.

I don’t like to play podcasts directly from the “Your Episodes” playlist because you can’t control the order and it will jump automatically to whatever comes next. Playing from within each podcast’s page is okay if you want it to go from one episode to the next.

I’ve pinned the “Your Episodes” playlist, which makes it easier to start an episode. In the mobile version, this playlist has a settings menu. Setting “Remove played episodes” to “After playing” will make this list behave the most like Google Podcasts’ queue.

I occasionally look at the “New Episodes” playlist and save any episodes that interest me. This playlist only appears in the mobile version, but it is extremely helpful because it shows the latest episodes from all the podcasts I follow. I don’t pin this playlist however, since Spotify is so stingy with its pins.

Pragmatic Potato is helping customers with software architecture and development. We can help with architectural investigations, software design, development process, interviewing, and even development. You can find out more about the kind of services we’re offering on our spiffy new website: pragmaticpotato.com/services/.

If you’re interested, contact me through my website. Let’s have a quick conversation and see if we can help you.

Some of you may know that I’m also working on a product. I am keeping these efforts in stealth mode for now, but you can sign up to be notified when I’m ready to say more.

What is an Architectural Decision Record?

An Architectural Decision Record (often abbreviated ADR) is just as the name implies: a document that describes an architectural decision. It can include various details, but should at a minimum describe the problem being solved and the solution that was chosen. It can be a simple text document with a page or two of text, or it can be longer with specific sections that are important to the team.

Why use Architectural Decision Records?

The most obvious benefit of architectural decision records is that they record your important technical decisions. Years later, if you’re considering replacing Framework X, you can see the original analysis that went into its selection. There may have been good reasons your new candidate wasn’t chosen. Or more likely, the decision may have been based on assumptions that are now known to be invalid.

My favourite reason for using Architectural Decision Records is that they are an easy way to encourage good architectural habits. For example, forcing a developer to write down the problem they are trying to solve makes it harder to add technologies with no purpose (yes, this happens). Asking for alternatives considered makes sure that alternatives were actually considered. Even considering if an ADR is necessary is a valuable exercise.

The process of writing and reviewing ADRs also makes technical decisions more visible, and accessible to more of the team. With these simple guardrails in place, you can hand a decision off to an intermediate developer and know that your most senior developers will still be involved. Distributing work like this helps prevent bottlenecks, but it also gives everyone a chance to grow their skills.

Having a formal decision-making process is especially helpful for big teams where decisions can impact a lot of people. Or if a team has a partially involved architect. When I was an architect with responsibilities for multiple teams, I found this process to be an excellent way to stay in the loop, and a great entry point to get involved when needed.

When should I use an Architectural Decision Record process?

This is really two different-but-not-so-different questions: when should a team have an ADR process, and what kind of changes should require an ADR process? The answer to both is effectively the same thing: when is a decision architecturally significant?

The way I answer this question is to ask what the impact would be if the decision is wrong. If there is a low risk of serious consequences and I can easily change to a different solution, then I make a good-enough choice and move on.

Here is an incomplete list of reasons why a decision might be architecturally significant:

• A lot of code would need to be rewritten if the decision gets changed. For example, when selecting a:
  • programming language
  • base framework
  • automated test framework
• Mistakes could have serious impacts. For example, if the project/change has anything to do with:
  • the privacy of sensitive data
  • security
  • financial transactions
  • life-critical systems
  • civic infrastructure
• The software is used by many different teams, and a change could impact many of them. For example: something exposed in the api of an open-source library or shared module used by multiple teams
• The software could be used and maintained for many years or have a lot of different people working on it
• The software is difficult to change if there is a mistake. For example: firmware

If any of the above are true for your project generally, then implementing a formal ADR process could be helpful.

When you have an ADR process, you should also provide some guidance about when it needs to be used. For the same reasons, it makes sense to base it on the potential impact of a wrong decision. You can use an arbitrary replacement cost for your definition (for example: “more than a week of effort to replace”), or list potential impacts like privacy or security risks.

Even without a formal ADR process, individual contributors can also benefit from using one of these templates. Filling out a good template makes sure you truly understand your problem and that you don’t skip any important steps. This is helpful even without soliciting feedback. I use this technique myself sometimes. This is also effectively what my Architectural Report template is.

Can I introduce an Architectural Decision Record process to an existing project?

Yes, but it’s harder for the team to appreciate the value of it.

You get all the benefits of standardizing the decision-making process, but it’s hard to realize the value of documentation without a significant majority of your important decisions documented. Retroactively documenting decisions isn’t very practical either: without tremendous notes or exceptional memory, the results can easily lack sufficient value.

If you have a strong reason to improve the quality of future decisions, and the team believes in it, then you have a shot at adding an ADR process to an existing project. I wouldn’t recommend it otherwise.

What Architectural Decision Record template should I use?

You can find several decent templates with a quick internet search. Joel Parker Henderson has helpfully posted a list of Architecture Decision Record templates and some other helpful resources in GitHub, which I’ve used in the past.

My favourite template is the Decision record template by Michael Nygard. I appreciate its simplicity as a starting point, and it’s trivial to add other sections as you need them.

The best template will depend ultimately on the benefits you want to get from your ADR process. For example, if you’re building life-critical systems you might include a section for safety testing measures to make sure they are considered with every decision. If your organization is concerned with copy-left licensing, you can include a section for any new dependencies and their licences.

Where should I store Architectural Decision Records?

The best place to store these documents is the place your team will look for them. If you have a wiki where all your technical documentation goes, that is a good choice. Some people like to put these records into source control, and that’s a good choice too. Whatever location you choose, make sure it has good search facilities and some kind of change history. A mechanism for giving feedback in the document itself is nice as well.

One advantage to source control is that you can use your familiar code review tools and processes to manage them. One drawback to source control is that it encourages simple text formats, and that makes it harder to include diagrams or other rich content.

If you do put your ADRs into source control, think about which repository they belong in. Some people like to keep them in the same repository as the code they relate to, but this won’t work if your project spans multiple repositories. If you use branches heavily, it may be confusing to see the documents in different while working. When in doubt, create a new repository for your technical documentation.

Another detail to consider is the scope of the decisions being made. If your team maintains multiple pieces of software you might have different (and conflicting) decisions for each of them. Make sure it’s obvious what your documents apply to.

Maintaining Architectural Decision Records

Decision records can get stale with time. If you do start using them, remember to also occasionally go through and clean them up.

To help with this, many teams use an append-only approach. If a decision gets replaced, a new document will be created, and the old decision will get a note inserted marking it as deprecated. Decision records can be numbered to make cross-referencing them easier.

Summary

Architectural Decision Records can improve your architectural thinking without much effort, and they have the added benefit of leaving useful documentation behind. If you add them early in a project, they can improve the quality of the decisions your team makes.

Static vs Dynamic websites

WordPress is a very popular blogging platform. You can use it hosted on WordPress.com, or any number of other web providers that offer it. Because it’s open source, you can also host it yourself easily. In my case, I had a virtual server running in GoDaddy where I maintained the installation myself.

Hexo is a static website generator. When you want to publish changes, you run a command that causes it to spit out a bunch of static html, css, and so on that you can then host anywhere you like. These tools can store the raw content any way they like, but Hexo and all the other generators I’ve played with use a directory structure with simple markdown files.

Dynamic websites are generated by software running on the web server. Ignoring any caches or other optimizations, every web request triggers a call to the running application, which in turn makes database requests and invokes any loaded plugins before a page gets sent back to the browser. For medium-sized websites, this isn’t noticeable because the server will be constantly warmed up and ready to respond. For very small websites like mine that go idle, a fresh web request can take more than 30 seconds to respond while everything is initialized. This is not ideal.

Static websites have fantastic performance. The server only has to return the pre-computed content, and the providers that do this at scale do it very well. Even if I only get one hit a month, that page will still load very quickly, likely even faster than a warmed-up WordPress instance. It depends on the hosting platform, but static websites can typically also handle massive traffic spikes without any extra effort, and at a much lower cost. Your small self-hosted WordPress site, by comparison, could fall over when one of your posts goes viral.

Another benefit to static websites is their security. Because it’s just plain files, there is much less attack surface. As long as your account is secure, it should be impossible for an attacker to modify the content. Even if someone did, the content and site generator can be kept separate, so the site can easily be replaced with fresh, clean content. With a WordPress site, you have the running server, the WordPress software, and the database all running in the cloud. They are prone to bugs and attacks like any other cloud infrastructure, and they need to be updated regularly to keep them secure.

Comments and contact pages on a static website

The biggest drawback to static websites is that they are, well, static. There are a few tricks you can use, but some features are much more difficult to implement. For a simple blog like mine, this wasn’t a problem.

Comments are a common blog feature that is trivial for a dynamic site to implement. For this blog, I recently set up utterances. It uses GitHub Issues (in a free public repository) like a database to store comments, and retrieves / renders them on each page with a bit of client-side javascript. This way I don’t need to run hexo generate every time someone posts a comment.

A reader does need a GitHub account to add a comment with this system, but this seems like a reasonable limitation given my audience. There are other static-capable comment systems available, each with different styles and storage mechanisms. There are commercial offerings too, which may or may not be more expensive than using a hosted WordPress site for your blog to begin with.

I don’t have a contact form on this site, but I easily set one up on another static website using getform. For the frequency of messages I expect there, it should also be free.

It gets unfortunately difficult to push much further than this on a static website. Some things can be done with client-side javascript or advanced application firewalls, but these can get complicated and/or expensive pretty quickly. It may be worth it if you want the scalability advantages of a static site, but if you just want a cheap website with a couple of dynamic features, a hosted WordPress site may be your best bet.

HTTPS and custom domains

The main reason I left WordPress on GoDaddy was the desire to move my blog to HTTPS. Nowadays it’s not that hard to get a free certificate from Let’s Encrypt, but the certificates don’t last very long, so it’s best to automate the process. At the time I changed, automating this wasn’t possible with the package I had from GoDaddy. I could have changed to a more expensive virtual machine option or bought a proper certificate. I went with option 3 instead.

HTTPS requires a certificate, and the certificate has to contain the domain name of the website you’re using it for. That means that you need a custom certificate to go with your custom domain name. The good news is that many hosting services can procure certificates on your behalf, and some will do it for free.

GitHub Pages is one of these services, and it was easy to set up. There were a couple of steps necessary so they could validate that I had control of the domain, but now they just take care of it for me. They can also provide redirects from HTTP to HTTPS, and from www to the apex website. GitHub Pages has good documentation if you want to set this up for yourself.

Price of static websites

Because GitHub Pages is free (for public repositories, at the time of this writing), the only thing I’m paying for is the domain name and the DNS service, both of which I get from AWS. The .ca domain costs $13 USD per year, and the Route53 hosted zone costs me $0.90 USD per month (the minimum possible since I don’t exceed the initial usage limits). This is much less than I was paying GoDaddy for a virtual server.

There are a few equivalents to GitHub Pages that are also free. There are also cheaper options than AWS for domain registration and DNS.

If you don’t want the hassle of setting this all up, there are lots of hosted WordPress solutions around, and other blogging/website platforms too. Some of these are even free, but the free ones each have their own limitations. I haven’t searched exhaustively, but I don’t know of any free options that support custom domains and control over advertising.

Creating, editing, and publishing posts

It can take a bit of effort to get a static site generator set up and configured with a nice theme, but things get easier once you have it all working. You still need to run a few commands in a terminal, which will be too much for some people. Even the mildly technically challenged should be able to keep themselves going with a few basic commands written down. This is what I use:

When I want to start a new post, I use the hexo new command:

hexo new draft 'This Blog: Hexo-generated static site hosted on GitHub Pages'

This command generates a markdown file in the draft folder with some initial “front-matter”. I can then edit the content in any text editor I wish. These days I use VS Code. Markdown is a very simple mostly-text format for making pages that is very easy to learn. Front-matter is a bit of data about the page that go at the top of the document such as the title, publish date, tags, categories, and so on. You can see an example of what this looks like in the source for the Hexo documentation.

If I want to see how a page will look (for example: if I’m using some tricky markdown syntax), I use hexo server --drafts. This hosts a website on my computer that looks exactly like the blog will when I publish it. I also sometimes use commands like hexo list tag --draft to make sure I’m using the same tags across multiple posts.

Once I’m happy with a post, I run the hexo publish command:

hexo publish 'This Blog: Hexo-generated static site hosted on GitHub Pages'

The main thing this does is move the file and its attachments from the drafts folder into the posts folder. It also renames the file and updates the front-matter with the published date and time. You could do this manually, but the command is quick and reliable. Then I need to re-generate the static content for publishing:

hexo generate

This goes through all the content files and configuration and generates everything needed for the website. The new post gets its own folder and html file, but it will also cause changes to the front page and a number of peripheral pages. The theme I’m using supports paging, so it also pushes the post at the bottom of the front page back to page 2, the bottom post from page 2 to page 3, and so on. There are also the tag, category, and archive pages that get updated similarly. It might seem like a lot, but I assure you, the software doesn’t mind.

After this, it’s a simple matter of uploading the content to my hosting provider. In the case of GitHub Pages, this means copying it to the website repository, commiting, and then pushing the branch. Nowadays I use a PowerShell script to generate, copy, commit, and push in a single step.

I use Git and GitHub to manage my hexo folders because I am very comfortable with these tools. This isn’t necessary though. You could just as easily store your hexo files on a cloud drive. You could store it on your local hard drive too, but I highly recommend using somewhere that has automatic backups and some form of change history.

Because I’m using drafts for incomplete posts, I commit and push any changes I’ve made regularly. The drafts aren’t included in the generated site, so I can keep a few sitting around while publishing other posts or changes. I originally went a little bonkers with a branch-per-post but found it made my process unnecessarily complicated.

You can simplify publishing your site further. Some static page providers can automatically run the generate step for you. I chose not to do this for my blog because I didn’t want my drafts and their edit history to be visible in the public repository (as a public repository is required to use Pages for free). There are other options that don’t have this limitation.

Converting from WordPress to Hexo

Most of the work of migrating was achieved using a WordPress export file and the Hexo migration tool:

hexo migrate wordpress wordpress-export.xml --paragraph-fix --import-image original

It took a bit of fiddling after this to get everything working perfectly. I had some issues with my code snippets (I was using a gist plugin in WordPress), and some of the attachment references didn’t work with my theme. The nice thing about working with markdown files is that it was really easy to search for patterns once I found them. With Git and branches, I could also quickly test fixes and revert them if they didn’t work.

I didn’t have a lot of content when I made the switch, but it only took me a few hours.

Summary

Static websites can be cheaper, perform better, and require less maintenance than self-hosted options. They have their limitations, but for a simple blog like mine, it was an easy choice that I’m still happy with.

For a while there I was getting hammered by ads from Grammarly, and in a fit of frustration, or maybe I was avoiding yet another hour of editing, I signed up. I didn’t start with the free trial, I went straight to Grammarly Premium expecting that it would be great.

Within an hour I had installed every app and plugin I could. A couple of days later most of them were removed or almost completely disabled. I cancelled my subscription before the first month was over.

I’m getting ahead of myself though. Let me first explain why I thought it would be so helpful.

The quickest posts I’ve written here have taken me a couple of hours where the more detailed ones can take 10 to 20 hours to complete. About 75% of that time is spent editing the posts. Given the amount of time I’m spending editing, I assumed a tool like Grammarly could help me free up some time.

I ran the numbers for a few recent posts to illustrate:

It seemed like magic the first time I ran a blog post through Grammarly. The app filled up with suggestions, and many of them were helpful. Spelling mistakes, grammar errors, and little formatting mistakes were easy to spot and eliminate.

Here’s an example of a paragraph I edited one of the first times I used Grammarly on a recent blog post:

Unfortunately, Grammarly made a lot of other suggestions too, and you can see some of them in the example above. It tried to eliminate passive voice and remove emphasis from important points. It frequently suggested replacements for words (to reduce duplication) that would ruin common industry expressions. Worst of all, it felt like it was removing my personal style from the blog. Every time I skipped a suggestion I also had this little nagging voice in my head suggesting that maybe it was right and my style of communicating was wrong.

Even if I skipped the tone suggestions and stuck purely to grammar and formatting errors, it didn’t speed up my editing process very much. I call it editing, but it might be more correct to call it rewriting. My first drafts have typos and grammatical errors, but they are also often disorganized and lacking a coherent point. AI is going to have to advance a lot further to fix that for me!

Grammarly is still helpful with annoying tasks like checking for “its” and “it’s”, and it’s definitely better than a standard spell checker for catching typos and grammatical errors. I don’t typically enjoy editing, but I enjoy this kind of mindless checking the least of all.

Having switched to the free version of Grammarly, I get far fewer suggestions about tone and word choice, which I find better. The app does still share those suggestions occasionally in an attempt to upsell me, but I don’t find it too intrusive.

I don’t use any of the apps or plugins now. I found them to be exceedingly intrusive. The Windows app added a giant green G button that covered text boxes, and a lot of times it was covering things I wanted to see. The VS Code plugin covered my rough drafts in red and blue lines, and my programmer muscle memory couldn’t just ignore them. The browser plugin was the least intrusive, but still too noisy and often not helpful.

Now I use Grammarly primarily via its web interface, and only as a final pass at the end of the editing process.

The technology is promising, and it has helped a bit with some of the more mechanical editing I do. Until it gets a lot better, though, I will only be using it for very specific tasks. I’d have to see some pretty major improvements before I’d consider Grammarly Premium again.

Take a rest

The easiest way I’ve found to get back on track is to take a break. When in the office this was often making a cup of tea or eating a snack in the break room. It could have been a ten-minute walk around the block, but if I’m honest, I have rarely tried this.

Working from home, I typically load the dishwasher or throw a load of laundry in the wash. If I have time, I might do a bit of yoga or exercise.

My favourite is a quick shower which seems to reset thoughts and stimulate creative thinking. Rinsing and drying my face in a bathroom sink is not nearly as effective, but in an office, it may be better than nothing.

If I have the time, or if I’m particularly stuck, a night of sleep can do wonders. I almost always have something else just as urgent to work on, so I’ll usually switch to that for the rest of the day.

Rubber ducking and alternatives

Every now and then I get so thoroughly stuck that no amount of taking breaks is going to help. Or sometimes a problem is urgent enough that waiting a day to sleep would be irresponsible. That’s when I bust out my rubber duck, at least metaphorically. I don’t actually have or want a rubber duck because there are several similar alternatives that work as well or better.

The one I use most often is to write an email asking an imaginary colleague for advice. If you treat it like a real work email, it forces you to rethink where you are from someone else’s perspective, and can often unlock some insights or surface some tidbits that got overlooked along the way. Writing an email also forces you to check your facts and go through your numbers again. This is harder when a rubber duck is waiting impatiently for you to finish your sentence. Another advantage is that you can use it in an environment where talking would be a nuisance to others.

Sometimes writing an email isn’t the best way to communicate a problem though. I have also drawn diagrams on whiteboards with similar results. There are lots of ways to communicate a problem, the key is that you make the effort to reframe it for the benefit of someone not directly involved.

If you are really really stuck, or your problem is really really important, it might be worthwhile to explain it to a real person. This can be a better first step if you’re working in areas you don’t know very well. It can also be helpful if you’re new to a team as it helps you demonstrate trust for your new peers and accelerates team building.

Remove all distractions

It’s certainly not the first thing I want to try, but when a problem is particularly stressful or unpleasant, I can sometimes find myself working on other things on the side of my desk. It may be listening to and editing a playlist, or maintaining an idle chat with a friend. It could also be changing that load of laundry I threw in earlier as a small break. However I got there, if I’m really stuck, removing these extra things is another important step.

Literally changing your perspective

If I’ve been looking at a piece of code long enough, sometimes I need to change how I’m looking at it to really see it. An easy way to do this is to change the font, or sometimes even just the font size. Even if the content hasn’t changed, rearranging the letters a tiny bit seems to force my brain to process it like it’s something new.

If I’m feeling particularly drained or frustrated with a problem, I have also found it helpful to move to another place for a bit. This is easiest with a laptop, or you may be able to use Remote Desktop from an empty meeting room. In a pinch, I’ve taken a pad of paper to a coffee shop to see what happens and been pleasantly surprised.

Confirm the problem is important

If you are really, really, really stuck on a problem, maybe there is a good reason. It may be beyond your current capability. It may also not be important enough to justify more effort. A quick chat with your manager shouldn’t hurt. If it is dangerous to ask your manager, you may want to find a new manager, but that’s a topic for another post.

Ask for help

If your problem is important enough, and you’ve made an effort and failed to get yourself unstuck, it’s not inappropriate to ask for help. Pair programming is an excellent technique for tackling complicated problems.

Searching or asking online for an answer could also help, but I’ve not personally had much luck here. Your results may vary.

The name of the things you build is the first and most frequent way that your work will be represented to others. It is the very first of the first impressions. A catchy name can generate interest in you and bring more attention to your hard work. A name that shows your personality can make you more approachable and attract people who see the world like you do.

This is especially important in larger companies. There is so much more activity and so many other people trying to move forward that maximizing the impact of your first impressions really helps. I got into this habit when I worked in a big company, and I’ve carried it with me ever since. It has served me well.

The technique I use is a pretty standard design technique: get as many options as I (reasonably) can, then pick the best one. I often go through hundreds or thousands of candidates when choosing a name. When it’s important enough, I’ve gone through more.

Even though I have been through the process several times now, I sometimes find it overwhelming at the start. It goes by quickly once I get started, though, and I can usually find a pretty spectacular name for anything in a couple of hours.

To generate candidates, I like to list out all the things I want the name to convey. I have had good luck with lists of words in relevant domains. For example, when naming a new module for an environmental monitoring tool, I got a great name from a list of meteorological terms.

Once I have some good candidates, I check my finalists against a few important criteria. I want a name to:

• be relevant to the problem being solved
• have a playful spirit
• focus on the positive
• be respectful of all cultures
• be easy to say, spell, and pronounce
• not be confusable with common programming keywords or industry terms

Playful is a part of my personality; you can take or leave that, but I highly suggest staying positive. Even if it’s meant in fun, using negative imagery in your name can give people the wrong idea about you. The point of this exercise is to get more people interested, not to push people away.

Being respectful of all cultures is becoming more important as the internet continues connecting the whole world together. Some words can have unexpected or unpleasant connotations for people who live (or have lived) in other cultures. Some words commonly used in Western culture should be retired now. Names from nature, science, geography, or popular fiction are usually safe. References to religions, cultures, or historical figures or groups should be avoided.

After focusing on a name for hours, it’s not uncommon to develop tunnel vision. Before pasting a name all over my files and objects, I like to get an external perspective. Asking a boss or a peer can save a lot of time if your name isn’t quite as great as you thought.

Once the name is set, all that’s left is living up to it.

About six weeks before a game I was working on was scheduled to be feature complete, we discovered our routine couldn’t handle our load targets. The rate at which players were being removed from the matchmaking queue started dropping during load tests. Things got bad quickly once it fell below the rate at which we inserted them. Not only would this cause a bad user experience if we didn’t fix it, but it made it impossible for us to drive enough traffic to our game servers to test that they could handle the projected load. The company wasn’t going to release a game that could crash if it was successful, so we had to fix this issue, and we had to fix it quickly.

To make matters worse, we could only reproduce the bug in our production system. We had to simulate hundreds of thousands of users to hit it, which just wasn’t possible on a developer machine. Fortunately, our production system was available for testing. Unfortunately, a team on another continent was running the load tests. It took us a day to deploy a new version and a couple of days for them to run the test. With time zone differences expanding the handoffs, the fastest we could iterate was about once a week.

As I’ve described before, I like to find a bug, understand it, and then fix it. In this case we couldn’t figure it out by looking at the code, even with many smart people trying. This is one of those rare cases where I had to try fixing a bug before understanding it.

The week of iteration time also meant I had about a week to review the results, consider multiple possibilities, and sprinkle logging and profiling code all over the place, so I did that, too.

I plugged away at it week after week, but the problem was stubbornly difficult to find. The failing load tests were visible to our management chain, so I also had a fair number of questions about when it would be fixed (which I couldn’t estimate), how we were fixing it, and what we would do if we couldn’t fix it. I was also helping the rest of the team with their deliverables, which had the same deadline, so it was stressful for everyone. Nevertheless, I stuck to my process, narrowed down the location of the problem, and tested my fixes as best as possible in my development environment. Eventually, I figured it out.

The problem originated in a database query we used to find matches. It used a cartesian join, which takes exponentially longer to complete as more rows are added (O(n²) complexity). At some queue length, the processing time increases from milliseconds to seconds, and then it quickly increases from seconds to minutes. It also didn’t help that our contrived load test meant thousands of users were all attempting to make matches with the same power rating, thus making it harder to distinguish suitable matches.

After all the time it took to find the problem, it only took a day to fix it. I replaced the algorithm with one that processed matches linearly, giving it O(n) complexity.

About a week later, I got an email confirming that the issue was fixed. Since the report came in overnight, it was the first thing I read in the morning. It was exhilarating, but the day is still bittersweet in my memory. About an hour later, I attended a last-minute team-wide meeting where we were all let go. The game was never released.

It makes sense then that teams want to ensure that code is sufficiently covered with tests. Nobody wants to count tests every time they review a PR, so tools are added that check it automatically. It’s then a small step to set a coverage target, and suddenly you have a machine checking every PR for tests. This all makes sense to me, and it was my first instinct too. I don’t recommend this approach any more.

The problem with test coverage tools is that they can’t (at least, can’t yet) measure the quality or value of a test. They instead measure the quantity of code that the tests exercise. This can encourage a misplaced focus on building lots of low-value tests. For example, consider the following piece of relatively standard web service code:

public GetResult get(GetRequest request) {
    return service.get(request);
}

Here is a standard test for this function:

@Test
public void getCallBusinessLayerGetAndReturnResult() {
    var request = new GetRequest();
    var expected = new GetResult();
    when(mockBusiness.get(request)).thenReturn(expected);

    var actual = service.get(request);

    verify(mockBusiness).get(request);
    assertEquals(expected, actual);
}

And now imagine thousands of tests like this, all testing very similar functions.

The unit test is testing the expected behavior of the service function, but what are the chances of there being a bug in a function this simple? I think it’s far more likely that the test would be written incorrectly than the service function.

The worst case scenario would be if all these service functions and all of these tests were generated by copy-paste and modification. Then it is much more likely that the wrong values get pasted into the test and the implementation at the same time. Unfortunately this kind of boilerplate code is almost always generated with copy and paste because it’s fast and and easy.

You could make an argument that you should avoid architectures that encourage lots of boring boilerplate code. I agree with that idea, but in my experience most teams are not mature enough to design systems that prevent it. It is easy to follow simple service-business-repository patterns blindly, and to be honest, for most software this is good enough.

Again: I do think unit tests are a good thing, and good test coverage is essential to get good value from them, but unit tests also have a cost. Unit tests often need to be changed when code is being changed. If you have lots of low-value tests testing lots of simple methods, you can quickly get overwhelmed trying to make non-trivial changes. Unit tests are supposed to make it safer to go faster… but poorly written tests can do the exact opposite too.

Of course there is an exception to every rule. If you are writing the software for my bank, or for medical equipment, or for self driving cars, please enforce 100% coverage and use several other tools to enure an extremely high quality. For most of us though, not all of the tests are actually worth the effort of writing them, and I don’t want to edit the coverage percentage every time I make a change.

This template works for simple reports that are only a couple of pages, but can easily be adjusted or expanded for more complicated or much larger documents.

The template has these sections:

• Purpose
• Context
• Recommendation
• Paths Not Taken
• Questions and Answers

I typically start a new problem by creating a new document and sticking in these headings. Next, I work to understand and document the context of the problem. Once that’s complete, I can design and describe my answer.

Surfacing Context

It can be tempting to jump straight into solutioning, however, and I cannot emphasize this point enough, you cannot design a good solution to a problem if you don’t understand the problem. I enjoy armchair architecture as much as anyone, but a good architect chooses the best solution for the circumstances, not just the first solution they thought of.

I always start with the first two sections of this document. This is usually only a page of text, but it really helps to frame and validate the problem from a business / customer perspective before starting the technical design.

Purpose Section

This first section has just a few brief sentences. It starts by explaining the purpose of the document, and then the scope of the document, and then a description of who it’s written for. It can seem a bit superfluous, but these details are important to remember when writing the rest of the document. It can also help a reader’s understanding if they know the context in which the document was written.

Example

This document recommends an approach to boiling the ocean. It is intended to be used during discussions with potential partners for the project. It is written for the CTO and the PMO.

Context Section

This is the most important section, and sometimes the hardest thing to get right. I only put in a few nuggets of information, but they need to be important. They serve as a sales pitch for upstream stakeholders and simultaneously guide and rationalize the decisions in the following sections. Even if the context seems obvious it is good to write it out.

I write some bullet points to articulate the importance of the problem and any significant constraints to the design. I keep it to just a few items, ideally between 3 and 6. If there are too many it becomes harder to process, if there are too few I probably shouldn’t be spending my time on the problem.

I like to polish each point a bit, but I also avoid flowery or superfluous language. Neutral defensible facts laid out clearly are a powerful tool for building alignment. It’s also good to write these in a way that makes sense to stakeholders who are not technically focused.

When I’m faced with an unclear problem, I will sometimes stick together some context statements based on my best guesses and send them out for feedback. Cunningham’s Law applies to all of design - it’s much easier to get corrections to a bad answer than ask for an answer in a vacuum. There are more specific techniques to help extract and validate this kind of information, but they take a lot more time from stakeholders so I try to use them only when necessary.

Example

• Over 80% of internationally traded goods travel by sea. Marine freight is exposed to many dangers including navigational challenges, extreme weather, and piracy.
• The sea level has been steadily rising due to human-caused climate change. As the ocean continues to rise, estimates anticipate hundreds of millions to billions of people will be displaced unless additional measures are taken.
• The earth is simultaneously affected by an international housing shortage and the continual loss of essential farm land.
• Eliminating the world’s oceans can simplify or make new solutions available to all of the problems above, which could in turn create numerous opportunities for raising and generating capital.

Document Title

A descriptive title is an important part of any document. I don’t want to spend too long figuring this out up front, but I do make sure I have a good one before any document gets circulated. Once it gets sent out, changing the name can break links and make it harder for people to find.

I also like to include the date in the file name and title. These documents are snapshots of the best information available at the time they are written. Since this is usually before a project has started, they lose a lot of their value once things get underway. A date in the title makes it clear when the document is an archeological artifact.

I always use the yyyy-mm format for dates because it’s precise enough, and the metric system is a good thing.

I could update the document as the project goes along, but I don’t see much value in this. The document has served its purpose once the decision is accepted and things start. If we need some sort of onboarding guide or architectural documentation, the parts of this document that are valuable for that effort can be easily copied and updated as needed.

Example

How to Boil the Ocean - 2023-08

Stakeholder Check-in

Once I have the above sections completed I like to check in with my upstream stakeholders.

Sometimes this process can take a bit of time. Depending on the size and urgency of the problem I might start on the technical design while collecting feedback. Parts of the design could be invalidated if the context changes, but usually any effort will still have nuggets that can be reused.

When I don’t get any significant feedback it can be a sign that the problem is either not well understood or not important to anyone. I might then schedule some workshops to dig in further before continuing.

Describing a Solution

There is a lot that goes into answering a technical question, and the way I go about it depends on a lot of factors.

Before I get too far into it, I like to think about how much I should be collaborating on the answer. Even if the answer seems obvious to me, it is sometimes valuable to involve the people who’ll be building the solution to ensure better buy-in once the project starts.

It’s also important to think about the acceptable amount of risk. If the cost of getting the solution wrong is huge, it may be necessary to do more thorough research or build and test some prototypes. If I’m just putting some rough estimates together to help plan the roadmap I will put in considerably less effort.

After all the thinking and validating work is done, I dump the best design into the document.

Recommendation Section

This is where I describe the answer to the question. I sometimes change the heading depending on the purpose of the document. If I’m preparing a thorough design for a complex system I might include multiple sub-headings and a few diagrams. For documents used more for resource planning I might just provide a brief architectural summary and a table of high-level tasks with t-shirt resolution estimates.

There are lots of ways to communicate designs. I do spend some time thinking about how best to do this. If I can use more efficient tools that are suitable for the audience and purpose of the document I can produce less. Smaller documents are easier to read, and easier to update when the feedback comes in.

Example

Use barges with nuclear fission reactors to boil the ocean water. As the sea level drops, the barges can be moved to ensure they stay submerged in water.

The steam generated by the reactors can also be used to generate electricity. Electricity not used for the operation of the project can be sold, adding an additional revenue stream.

Paths Not Taken Section

I’ve found it is often helpful to list some of the other possibilities that were considered but not chosen. Not everyone cares about this, but it can answer a lot of “what about X” questions from some personality types. It can also be helpful if you look back at a document years later. It’s not the most important part of the document though, so I try not to put too much effort into it. Bullet points and rough notes are usually enough.

Example

• Use a giant laser from space.
  • The laser beams would be a hazard to air and space traffic.
• Accelerate global warming by burning a lot of oil.
  • The required oil would make this approach significantly more expensive than other options.
  • This option could generate too much negative PR.
• Pump the water into on-land boiling stations.
  • Cost of pumping would be significant.
  • Hoses and pumping stations would need to be continually added as the sea level drops.
• Use nuclear fusion reactors to generate heat.
  • The technology is not mature enough at the time of this writing. If it does become feasible during the project we could update the design for any new barges being constructed.

Questions and Answers Section

There will often be some things that I want to communicate that don’t fit anywhere else in the document. A Question and Answers section is really versatile, and it’s an easy place to add more information when responding to feedback.

Example

• Is boiling the ocean really a responsible thing to do?
  • That is outside the scope of this document.
• How will the boiling stations deal with salt accumulation?
  • This will need to be investigated further if we proceed with this project.

Customization

Just like in software development, I find it best to use the simplest document that can possibly do the job. The less I write, the easier the document is to read, and the more time I have to work on my next project.

If I don’t need one of the sections above I’ll remove it. I can also add new sections as I need them. I have added sections with estimate tables, rudimentary project plans, optional features, glossary of terminology, and so on.

Template

I’ve created a standalone page with the example from this post so you can use it as a template for your own work. You can find it under Resources, Architectural Report Template. If you do use it I suggest replacing the contents before sharing it to prevent any embarrassment.

Acknowledgements

This template was inspired by the Decision record template by Michael Nygard which I also recommend for its intended purpose.

If you want to know more about how to write architectural reports, gather information, or validate designs, I highly recommend the book Design It!: From Programmer to Software Architect by Michael Keeling. I learned a lot from this book, and have referred to its contents many times since.

Always use a NoSQL database so your app can scale.

NoSQL databases can be more scalable, but schema-on-read has other drawbacks. NoSQL databases are much less capable of transactional changes. Relationships are difficult or impossible. Designing schemas to be efficient is much harder, and requires more up-front knowledge about your problem. NoSQL databases are sometimes the right tool for the job, but they are not the right tool for every job.

Monoliths are bad. Build Microservices.

Microservices have advantages, but they are harder to build. They are harder to deploy. They require more tools, processes, and governance to keep them running and working together properly. If they aren’t carved up just right they can create cross-team dependencies and performance bottlenecks. All that being said, they are ane excellent tool for very large organizations to decouple and decompose development teams.

Never use reflection because it’s slow.

Reflection can be a heavy operation, but it can also give you extremely valuable information. If it does prove to be a performance problem, there are also other optimizations which can help, or at worst, you can try to restrict lookups to development and test environments. Stack traces in exceptions can be extremely helpful in finding bugs. Even if they do cost a bit to collect them, the developer time wasted figuring out bugs, and the unhappy customers waiting for fixes may cost you more.

I have heard these kinds of absolute statements many times in my career. To be fair: building software is hard and there is a lot to learn, and simple statements are a comfortable simplification. The problem is that few absolutes exist.

It doesn’t help that so much information in our industry is presented this way. Blogs that pronounce something bad get clicks. Vendors don’t like to tell you their drawbacks. Everyone wants you to agree with them.

This kind of blind adherence to arbitrary rules is bad engineering. We should work to understand the advantages and disadvantages of our technical choices, and understand when it is important to get them right.

Since I started with a quote, I’ll end with another:

When you believe in things that you don’t understand Then you suffer Superstition ain’t the way

– Superstition by Stevie Wonder

One of the first and most important lessons I learned as an architect is that you can’t design a good architecture without a good understanding of its requirements. You can design a system in a vacuum, it’s also much easier to do it this way, but it’s far less likely to serve the organization. Gathering, validating, and documenting technical requirements is tough work, but an essential part of being an architect.

The best way to discover needs and requirements is to talk to people who know them. There is quite a lot of ways to approach this, and a lot has been written on this subject. For me, I knew I needed to chat with my stakeholders early and often, and private meetings was the simplest technique that could possibly work.

In my first one-on-one meetings I explained my role, and more importantly, why my role would be beneficial to each of them. For the product manager I explained that if I was good at my job I would help the development team go faster (because of less rework) and to reduce the technical risks of higher value projects before they were scheduled. For the product owner and development leads I explained that I would find and encourage new technologies that helped the team go faster, and help to keep projects moving smoothly.

I didn’t have any difficulty getting time on a regular schedule. Even with a 3-month release cycle, there was always enough to fill a meeting every two weeks with each of my main stakeholders. I always came prepared with questions, and after a while my stakeholders starting bringing questions for me too.

I think I have pretty good communication skills, but when you need to explain the value of technical work to people who are primarily non-technical it can be a lot more difficult. It’s even more daunting because the people I was meeting with were far, far better presenters and public speakers than I am.

I started noticing that I was getting very similar questions from different stakeholders, and so I was able to re-use my answers. And with time, I started getting better at it. Regular conversations certainly helped, but talking about the same topics over and over seemed to make an even bigger difference. Every time I answered a similar question I had a chance to refine my best points and try out new ones.

I was fortunate to work in an environment with a lot of trust: I could speak my mind honestly, and I didn’t really need to be persuasive to make sure my ideas were heard. Even still, the better I got, the more confident I felt. Portraying confidence is important in a leadership position - if a leader looks worried it can generate more stress for those around them.

After a few years of these meetings, I am convinced that they were a big part of my success as an architect. I’m also convinced that they’ve helped me improve as a communicator.

Repeating similar conversation has been so valuable, that I’ve started having practice conversations just for the sake of it. Rough notes and research is still my preferred place to start, but this type of preparation starts to have diminishing returns after a while. If I really need to nail a conversation, a practice conversation is an efficient way to improve my messages. All it takes is a bit of time from a trusted colleague, or if that’s not possible, a rubber duck can do the job too.

I went through my usual bag of tricks: searched recent changes, searched for insert statements, tried to create empty users manually (and couldn’t). Nothing worked, and it was looking pretty hopeless.

I knew I wasn’t going to make any progress if I couldn’t narrow down the cause. The problem was only occurring for one customer, but it was also only occurring around once a week. I ended up making a patched version of our application that regularly scanned the number of users and threw up a notification if a new one had appeared. I also added some custom logging in a few conspicuous areas, and sent it to the customer.

The issue appeared again, but this time we were able to collect the logs and analyze them. I was able to determine that the problem was occurring for a specific user, and I was able to figure out which application had caused it. The weird thing: this application didn’t have a feature for creating users! We were able to question the user, and it turned out they had a new computer, and it had a brand new mouse.

It’s important to understand that this application was build using Microsoft Access (hey, don’t judge). One of the best features of Access in the data binding, saving your from writing a lot of boilerplate code. All you have to do is write a query and bind the controls on the form to the desired fields.

One of the (many) challenges with Access is that it is sometimes a bit too clover. It could convert a select statement into insert / update / delete statements automatically. If you navigated the form past the last row of the query (even if your query only selected a single row) it would go into creation mode.

The form where the row was created had all the keyboard navigation shortcuts blocked, as was the standard practice, but the user was still able to trigger it. This is where the new mouse comes in. This problem occurred around the time when mice just started shipping with scroll wheels, and in Access it automatically triggered row navigation.

We weren’t able to disable this behavior, but I was able to rewrite the query and modify the form so that a new row couldn’t be generated any more. We also updated our installation instructions to ban mice with scroll wheels in case the same problem could be triggered in any of our hundreds of other forms.

The normal “Kubernetes way” to upgrade an application is by changing the Deployment resource. With its default RollingUpdate strategy it will delete a pod with the old definition, start a pod with the new definition, wait for it to be healthy, then repeat continuously until the change is fully applied.

This process wouldn’t work for us, and it was not obvious how we could automate one that did. Our application is tied to its schema version; new versions of the app won’t run on the old schema, old versions of the app can’t run on the new schema, and the schema migrator won’t start if it detects any running applications. We would have preferred to use a rolling update without downtime, but it wasn’t possible to make our application support this in our timelines. I expect it will eventually be implemented, but it will require several changes and a significant testing effort.

The process we wanted to automate was:

1. Shut down the old version of the application
2. Run the schema migrator
3. Start the new version of the application

Or putting it into Kubernetes terms:

1. Delete all the pods
2. Run the schema migrator job
3. Create new pods (with new image tag)

We tried a few different approaches, but the solution we ultimately chose was using ArgoCD with its sync phases and waves feature. There were a few unexpected challenges, but we were still happy with the results.

What is ArgoCD and What Are Sync Phases and Waves?

ArgoCD is a powerful open source tool that lets you deploy Helm charts to a Kubernetes cluster. The charts and their settings are pulled from a configurable source, in our case GitHub. This allowed us to store all our Kubernetes configuration as code. We wanted better visibility and consistency in our infrastructure and this tool makes that possible. Being able to add “GitOps” is an added bonus.

ArgoCD is based on a custom resource called an Application. An application represents a single installation of a Helm chart. Its resource includes a source (where to retrieve the chart), a destination (where to install the chart), and any parameters to apply to the chart. You can automate more complicated scenarios with charts containing Applications, resulting in Applications containing Applications. You can also use the ApplicationSet resource to generate multiple Application objects. We combine all of these to push out a lot of similar but not exactly the same applications in several environments.

ArgoCD periodically compares the source (code) and destination (cluster state). If there are any differences, the application gets marked as out of sync. ArgoCD can synchronize all the changes in an application automatically or with the press of a button. It also has a nice user interface that shows all the applications, their state, and some other useful information.

If your application can be deployed all at once as a simple Helm chart, ArgoCD can easily do this. For a more complicated deployment process like ours, we used the sync phases and waves feature; we added special annotations in a few our resource definitions to control the order ArgoCD applies their changes. It looks like this:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/sync-wave: "5"

How We Automated our Deployment Process

Our process uses these steps:

1. PreSync / -1 - Create necessary secrets, service accounts, etc.
2. PreSync / 10 - Create job: run a script that sets the replica count to 0
3. PreSync / 20 - Create job: run the schema migrator docker image
4. Sync (default) - Create the deployment, services, and everything else

The PreSync / 10 step ensures the application is stopped before continuing. It checks that the deployment exists so it won’t fail on the very first run, then it sets the replica count to 0. The pods get deleted pretty quickly after this change is applied.

The schema migrator job runs next. It can upgrade the schema of an existing database or create a new one if one doesn’t already exists. Once it completes, all the rest of the resources are created in the Sync step. The deployment resource sets the new application version and restores the replica count. Pods start getting created, and pretty soon we have a fully working application.

If the schema migrator job fails, Kubernetes will execute any retries as per the job definition. If the job can’t complete, the synchronization cycle stops and gets marked as failed in ArgoCD. A human can then make any necessary changes and trigger another synchronization cycle.

We ran this process hundreds of times in our development environment and several more in our production environments. The ArgoCD part of it always worked correctly. Since some of our applications had an installation per tenant, we also ran several in parallel with no issues. We did have a few deployments fail, but they were all caused by infrastructure issues or application bugs. That won’t be different from any other Kubernetes system.

Branching Strategy

The GitOps approach to managing our environments brought some significant benefits, but it also made it more challenging to test changes to our charts. For example, some application changes would require add or dropping startup parameters in the pod definition. We had to be especially careful that it wouldn’t break an environment if the new chart was applied before the application version was updated. It was possible to deal with simple changes like this using conditional blocks in the Helm templates, but it got a lot more messy when we were updating community charts for our logging or monitoring infrastructure, or changing the shared ingress definitions.

To ensure we didn’t have any accidents we moved to a branching strategy. We now use three branches:

• develop - for our development environment. This is where most of our chart changes occur. It’s also where we test daily builds of our applications
• staging - for our staging environment. We use this to test new charts and applications before a production release
• production - for all of our production environments

Changes to the charts get tested in the development environment. They then get merged to the staging branch just before a major release. As part of the production release cycle we merge the same changes from the staging branch to the production branch, making sure that only tested changes get deployed.

To ensure there is no configuration drift, we also occasionally merge changes from the staging and production branches back to the development branch.

We did find branching a bit difficult to use, especially for parts of our team that had less experience with Git. Even with this difficulty, we found the added safety worthwhile.

The Good: Our Upgrade Process

Our upgrade run list was beautifully simple:

1. Make sure the environment is healthy and all the Application resources are in a healthy state
2. Merge any changes from the previous branch to the target branch (ex: develop to staging, or staging to production).
3. For each installation, modify the version (docker image tag) in the appropriate configuration yaml files and merge those.
4. Find the applications marked as out of sync in ArgoCD and trigger synchronization cycles. Wait for them to finish.

Once we started the synchronization cycle, ArgoCD would start applying the changes. A few minutes later the new version would start up and the web services would start responding to requests again.

The Bad: Scaling Pods Without Downtime

The biggest drawback of this approach was the inability to make minor configuration tweaks to our production system through the code. ArgoCD uses a synchronization cycle to apply changes from the source. This was great when we were changing the version of the application and the schema migrator needed to run, but it wasn’t so great when we needed to add a little memory or increase the replica count to keep everything working smoothly.

In these cases we had to make changes to the Kubernetes resources directly, bypassing ArgoCD. This meant the Application resources would be marked as out of sync until we made the same changes in the code. If we forgot this step, the changes would get stomped during the next synchronization cycle.

ArgoCD has a feature to ignore certain state differences in a resource. This is great when you’re using autoscaling or other Kubernetes automation features. We couldn’t use it though because it also sometimes prevented ArgoCD from applying the changes to raise the replica count from 0 in the Sync phase.

This is only an issue when deploying applications that need downtime while the chart is being applied. Many applications that are designed to run it Kubernetes will keep working throughout this process, and ArgoCD can handle this just fine.

The Ugly: Automatic Synchronization

ArgoCD is capable of triggering synchronization cycles automatically when it detects changes. This is helpful if you want to ensure your cluster’s state is always identical to your committed code, which is the ultimate in a GitOps workflow. The drawback is that it also means a synchronization cycle can be triggered whenever changes are detected. Since our process involves downtime, we didn’t want this to happen unintentionally in our production environments.

The other problem we ran into with automatic synchronization was that it made it harder to test minor configuration changes in our development environment. If we added or removed a bit of memory to measure the impact, ArgoCD would quickly reset it back. We could add parameters to allow these things to be configured, but that increased the complexity of the charts and made them harder to read. It also meant we had to remember te remove the same changes again later.

The setting for automatic synchronization is specified on a per-application basis via the resource definition, so you can make this behavior optional for some of your applications. We decided to use manual synchronization in our staging and production environments for everything but the top-level charts. This allowed us to control when changes were applied.

Conclusion

Using ArgoCD for automating the more complicated upgrade process worked well for us, even with its challenges. I would recommend this solution to others.

Another strong reason to use ArgoCD is that it is an excellent tool to use even if you don’t need to control the synchronization process. It was a great platform for us to deploy newer Kubernetes-native applications, and it was convenient to use the same tool for everything. It also left us in a position where we could iterate gradually to a simpler deployment process with our legacy applications.

On the other hand, when you have poor alignment you can see all sorts of problems. When teams are misaligned they can undermine each other’s efforts. When you are misaligned with your manager you can find yourself being over managed or left out to dry when things get rough. When you are not aligned with the goals of your organization you miss opportunities to demonstrate your skills and advance your career.

Alignment is such an important part of being successful in an organization that it’s important to understand what it looks like and regularly assess how healthy it is. If it starts to degrade, you should work to fix it, or you will find it gets harder and harder to be successful.

Ensuring good alignment with your manager is the most important. A good manager should help you find work that is rewarding, encourages development, and furthers the company’s goals. This is a hard job, and even the best manager in the world can’t do this if they don’t know what you enjoy and what your goals are. Make sure you are talking regularly with your manager and you have a good relationship. I can’t overstate how important it is to have a manager you trust that is helping you grow.

A manager that is honest with you when you’re doing poorly is also important. It’s impossible to grow without pushing yourself, and when you tackle new challenges it’s almost certain that you will make mistakes. If you don’t get negative feedback when you need it you can end up wasting a lot of time and energy, and ultimately damage your reputation.

In Vancouver / Canadian culture a lot of managers (myself included) find it difficult to deliver negative feedback, but the alternative is so much worse. A manager that doesn’t communicate honestly and candidly may seem like they are supporting you and agree with your direction. This is like wearing a fake seat belt; when things go bad you will have no protection.

If you have a manager that can’t give you guidance because they don’t understand what you’re doing, you need to make sure they at least agree with the choices you are making. Be very careful here though, as some managers will say they agree with you without really meaning it, especially when the choices are complicated. You have to have make sure they really understand what you are asking and the implications of it. If they aren’t invested in the decisions, they may not back them up when you need it.

Another challenging situation is when your manager is too far removed from your work to have an opinion about it. In these cases you have to depend on the other kinds of alignment and do your best to manage and sell yourself. This becomes more likely the higher up you go in an organization.

Good alignment with your organization’s goals is also important, but this is impossible without a culture that supports it. People at all levels of an organization have to make decisions every day, communicate with customers and partners, and are constantly representing the brand. You and everyone else should know what the organization is trying to achieve and how you are personally contributing to it. When everyone isn’t working in the same direction it is easier to have miscommunications and disputes between teams.

The best teams I’ve worked on update everyone with recurring company or department-level meetings. Even if some of the information isn’t immediately or personally relevant, it all still soaks into the subconscious. Even just the exercise of gathering and presenting to the broader audience is valuable; it forces team leaders to understand the value they provide and makes sure they are measuring and delivering it.

Some companies don’t see the value of keeping everyone informed. If that’s the case where you work, you will have to take matters into your own hands. At the very least, make sure you read the company emails. Ask questions when you can, and try to always understand the direction your company is going.

Even when you have good alignment with your manager and a clear understanding of the organization’s goals, you may sometimes find yourself under a manager that isn’t well aligned themselves. Even if you are a perfect employee, the success of your team will reflect on you, and the team won’t be seen as successful if it isn’t helping the organization. The teams that are doing the best from this point of view tend to have the best bonuses and the best growth opportunities.

It’s not always possible to control the team you work on, sometimes the only way to improve things is to leave your company entirely. Of course there are lots of factors that go into a decision like that… but if you don’t feel that you, your team, and your company are going in the same direction, or if you can’t tell, you should at least be aware that it could be hurting your opportunities to grow and advance.

An unconference, sometimes called an open spaces conference, is a participant-driven event where attendees choose the topics of discussion and provide the content themselves. They are meant to be open and inviting, and build interpersonal relationships. This year was no exception.

The event started with a brief introduction and some ground rules, and then attendees began pitching sessions. Anyone could pitch a session, and then organize it however they wanted. This year I pitched a session about how to start a software company. I got on stage, gave my name, and explained what I wanted to talk about.

After all the sessions were pitched, attendees voted on the sessions they want to attend. Organizers then assigned them to rooms. All the sessions can usually be accommodated, so the voting is only used to make sure that the number of interested attendees can fit in the rooms they are assigned to.

For my session, I used a fishbowl format. I put 5 chairs on the stage. Only people in chairs could ask or answer questions, but anyone in the audience could take a chair at any time. The group on the stage had to ensure that one chair was always empty. What you end up with is an intimated discussion with an audience.

A few experienced founders attended my session, and quite a few people who wanted to or had already started software companies were present too. I asked my questions, other people asked their questions, and we got a lot of great answers. I took 3 pages of notes that will absolutely be helpful in my endeavors.

The best part of this event is learning about what other companies in town are doing, what’s working for them, and where they’ve had problems. Traditional software conferences tend to have more vendor-sponsored presentations where everything is a sales pitch. These are valuable too, but the unconference is a better way to get a balanced opinion.

I can’t wait to attend again next year. `

When the pattern is used well it is almost invisible, and yet we should be thinking about it all the time. Sample code and simple apps often show exception handling as rote boilerplate that writes out stack traces and swallows errors. This is not a good example to be setting.

Most modern languages have an exception type and a throw statement. I’ll be using C#/.Net terminology for this post, but the same or similar terms and patterns exist in Java, JavaScript, TypeScript, Python, and many other languages.

Most scripting languages (shell script, Windows batch, and sometimes PowerShell) and some low level languages like C don’t have exceptions. In these cases you have to check return codes every time you call a function or an external application, and it sucks. You don’t have to look far to find scripts filled primarily with error handling code. For the rest of us, there is something much easier and better.

The Exception Pattern

Here is an example of the exception pattern being used well:

public string GetMagicString()
{
    using var reader = new StreamReader("magic.txt");
    var magicString = reader.ReadLine();
    if (String.IsNullOrEmpty(magicString))
        throw new InvalidOperationException("magic.txt did not contain any text");
    return magicString;
}

If you don’t already know how exceptions work you can read about them in Microsoft’s documentation.

You may have noticed that no exceptions are being handled in the function, which is the point. This depends on the built-in exception that will be thrown being good enough. This is because I expect the file to be created by the installer in normal deployments. I could wrap this in a try/catch block and do something, but what would I do? If you were about to say “return null” then you lose 10 points. You should always favour exposing problems quickly and clearly rather than ignoring them or pushing them down the line.

If I couldn’t expect the file to exist, if say magic.txt was a special override file that was only created in certain circumstances, I would write the function like this:

public string GetMagicString()
{
    try
    {
        using var reader = new StreamReader("magic.txt");
        var magicString = reader.ReadLine();
        if (String.IsNullOrEmpty(magicString))
            throw new InvalidOperationException("magic.txt did not contain any text");
        return magicString;
    }
    catch (FileNotFoundException)
    {
        return DefaultMagicString;
    }
}

Notice that I am only catching the specific file not found exception. I could have caught any exception, but that has its own dangers. Another possible exception is the UnauthorizedAccessException. If that happens, I’d rather notify the user than silently ignore the file.

I added a bit of guard code that can throw the InvalidOperationException. Even if it’s an unlikely problem, I prefer to surface problems like this earlier. They typically save time later when debugging. I’m keeping this code in the second example even while I have access to a default value because I think an empty file is more likely a mistake than intentional. It could be intentional for some use case I don’t know about, but it is always easier to ease restrictions than it is to add new ones in a deployed application, so I’d rather be more strict early.

The using var declaration is a relatively new C# feature that generates an implicit using block. It causes reader to be disposed before the function exits, be it successfully or a because of a thrown exception. Because the stream is opening and potentially locking a file, we want to make sure it is cleaned up quickly. Cleaning up resources in all cases is something we should always be mindful of as well, but in most modern languages you can use features like using to make this task similarly invisible.

Global Error Handling

So where do exceptions get handled? Generally you should have an error handler somewhere, as high up the stack as is reasonable. You only have to write it once in one place, and reducing repetition has a lot of advantages. Your error handler can do any fancy stuff you want it to, and you can change the behavior of your error handling without making sweeping changes across your application.

Most of our applications are iterating on some atomic unit of work. Web servers iterate over web requests. Background processors iterate over messages in a queue or scheduled tasks. It makes sense to implement error handlers at the level where these are invoked, especially if there is a good chance that another similar unit of work could succeed.

In a web api the error handler can usually be added as a wrapper around every request through some hook in the framework. Common implementations will log the exception and return a 500-level status code. Fancier implementations will return more nuanced codes depending on the type of the exception (like return a 400 for ArgumentException or anything derived from it). Your web framework of choice probably already does this for you.

Some errors, however, aren’t recoverable. For example, without complicated retry logic, many applications will be in a bad state if their startup logic fails somehow. You might have one global handler for all your startup code, but its behavior should probably be to output some useful diagnostic information and exit. There may be other errors that indicate poor application health beyond startup, but this is often pretty challenging to deal with reliably.

.Net also has a few special exceptions that can’t be caught normally. Most exceptions mean that the operation you attempted failed, but these uncatchables indicate that the runtime environment could now be in a bad state. The dreaded StackOverflowException is an example of this. In these cases exiting and getting restarted is the only safe thing to do. The framework is going to do that regardless of your error handling so you don’t need to add special logic for it.

Throwing Good Exceptions

The best thing to do when you detect unexpected conditions is often throwing an exception. The exception message should clearly describe what the problem is. Remember that you or one of your peers will be reading this message about 6 months from now when a test fails or a bug gets reported. Sometimes I like to include advice about fixing the problem in the exception, but use this sparingly; it’s much less helpful when the advice has become outdated and sends users on wild goose chases.

It sometimes makes sense to create a new exception type for your error cases. I can’t remember where I once read that exceptions should have the same level of abstraction as the interface that throws them… It’s not terrible advice, but I don’t think it’s worth the trouble for most cases. If you’re building a framework it might make sense. If you’re checking for an empty file after reading its contents it may not.

One case where I do create custom exceptions is when I’m writing a unit test. For really trivial stuff (like argument null) you probably don’t need a unit test, but for anything that should be tested, use a custom exception. If you’re asserting on any exception being thrown your test could be marked as passing when it should be failing. If you derive your custom exception from a built-in or another more common exception, your callers won’t normally need to handle your specific type, but they could if they wanted to.

If you are not creating your own type, try to use a specific and appropriate exception type, either built-in or custom. InvalidOperationException and ArgumentException give better hints the developer in the future than a simple Exception.

It’s also a good practice to include the inner exception whenever you generate a new exception in response to a failure lower in the stack. These details can be very helpful if the new exception is ever thrown unexpectedly.

What About Performance?

One common argument against throwing exceptions is that it causes a performance hit. It’s true, it does takes time to capture the stack trace, and allocating a new object on the heap isn’t free. It depends on your circumstance of course, but I think in 99% of scenarios this cost is too miniscule to matter, and certainly far cheaper than the wasted time of developers who can’t find bugs.

If your exceptional case is something you expect to hit often inside a tight loop though, it may actually matter. The TryFunction pattern is the common alternative in .Net. The drawback is that because your return value is typically boolean you can’t easily add new failure cases as safely as you can with exceptions. You could include more information in another output parameter, but that can also impact your callers whenever the list changes. It also means every caller needs to check the function response, or even worse, they can forget to and the app can continue running in a weird state without realizing the call failed.

What About Leaking Security Sensitive Information?

Another argument I’ve heard against good error handling is to prevent leaking information to attackers, but I think this is mostly bad advice too. For any software that is distributed, a malicious user can easily find tools to look inside and see how it’s wired. For server-side code you should at least share the error details with yourself in your own log files. If you need to withhold information from potential hackers then do that in your global error handler.

There are plenty of unintended features (bugs) in games that became beloved. Attack combos were an accident in Street Fighter II, but they became so popular that they are a part of basically every fighting game now. Rocket jumps are another example. The internet is full of examples.

Sometimes very glitchy games can be fun too, especially for a certain audience. Speed runners sometimes use glitches to lower their times. People love games for all sorts of reasons beyond just beating them and getting high scores. At the end of the day the goal of a game is to entertain more than be correct.

The next time a bug comes across your desk, maybe ask yourself if fixing it would make your app less fun.

I asked some questions and tried my usual tricks. There wasn’t an error message or anything helpful in the logs. There didn’t seem to be any obvious place to start, so I did something crazy: I searched the entire codebase for the world DELETE.

It seemed likely to me that the user was getting deleted by the application, so there had to be a delete statement somewhere that was the cause.

One of the biases I’ve noticed in myself is that I tend to drastically over-estimate the time it will take to perform mundane tasks. In reality, even if there were a few thousand delete statements in the code (and there weren’t), it would still be possible to review them all in an afternoon. Using a good IDE that has a preview for search results and a bit of care with search terms it was possible to get through them even faster.

I found the offending delete statement. The bug was actually a feature!

The customer had re-installed the software at some point and couldn’t find their licence key. The software wouldn’t work without a licence unless it was put into “demonstration mode.” Demonstration mode caused users to be deleted after the trial period ended.

We issued them a new licence key and changed the demonstration mode feature so it wouldn’t be so destructive if it ever got turned on in the future.

I’ve fixed a lot of bugs in my career, and to be honest with you, I usually enjoy the process. These days I am typically assigned the super urgent bugs that nobody else can figure out, and I kind of like it that way. I don’t get me wrong, I don’t like the bugs being there, but I enjoy being helpful and figuring out tough problems. I also think my successes have helped improve my reputation which is always a good thing.

This can take a lot of hard work, but I’ve made the job considerably easier through a set of tricks and a process I’ve developed over my career. I never thought of these as particularly special, or as a secret weapon, but I realize now that they aren’t really taught. They are important though, so I wrote them down.

This is the process I generally follow when I’m approaching a bug. I will obviously not go through a formal process for the super simple and obvious ones, but for the toughest nuts, all the steps here will help me get it over the finish line.

Step 1: Read the Bug Report

When a bug report comes across my desk, I read it and make sure that I understand it. Fixing a bug properly takes time and carries its own risks, so I want to be sure I’m fixing the right bug.

If I can’t understand the bug report, this is also a good place to stop and request more information. I wrote another post about writing good bug reports.

Step 2: Reproduce the Problem

The most important step to fixing a bug is reproducing it. Even if the problem seems obvious, even if I am feeling particularly lazy, I force myself to do it anyway. If I can’t, I have no way to test my fix later. It also means I won’t be able to use a debugger to figure out what is happening when it fails.

Sometimes it’s possible to write an automated test before finding the problem, and if I can, I will. It makes debugging and testing considerably faster. In complicated software this can even be faster than starting up all the required pieces to test an issue manually.

It can be difficult to reproduce an intermittent issue, but I still put in the effort for the same reasons. I’ve sometimes had luck wrapping unit tests in a for loop, but it won’t help if the causes are environmental.

There have been rare occasions when I’ve had to attempt fixing a bug before I can reproduce it, but I will only let this happen in extreme circumstances. When doing this kind of thing I also make sure to communicate it clearly with everyone involved.

Step 3: Find the Problem

This can be the most infuriating part of fixing bugs, but it is essential. There are a bunch of techniques that can work here. For this post I’ll stick to the tricks I use the most.

The first thing I do is read the error details again. If there is an error message in the bug report it can carry a lot of clues. It’s tempting to skim over it, and I have made this mistake plenty of times myself. Now I make sure I not only read the error message, but also understand it. This has saved me a lot of time. System errors can be especially helpful since they tend to be specific. Stack traces are also incredibly valuable. If I don’t understand what an error code means I look it up.

In the absence of a stack trace, I like to narrow down where the problem is occurring. I start by visualizing all the steps through the system that are involved in the malfunctioning operation, then I choose some point in the middle. Using my trusty debugger, I test if it’s failed at that point. I continue along in a binary search pattern until I have narrowed down the problem to a specific spot.

For example: if I’m fixing a bug in a web application where a user’s name is getting saved incorrectly I can start by checking the web request from my browser’s developer tool. If the request is wrong, I know the bug is on the client side. If the request is correct, the bug will be somewhere in the server. Assuming it’s in the server, I might set a breakpoint between my business layer and repository layer to narrow it down further. Continuing in this way I can find the exact location of the bug quickly and reliably.

It works for more than just software problems too. I’ve used the same approach to diagnose load balancer problems, problems with components in Kubernetes, computer hardware, even electrical wiring.

In some cases it’s easier to figure out what change introduced the bug instead. I will use the same kind of binary search pattern but checking out commits between releases. A quick build time and a unit test makes this a lot less painful. I’ve never had a bug where it was a good fit, but you can also try the git bisect command to automate the process fully.

If I ever get stuck in my investigation, I go back to the beginning and re-check all my assumptions. Did I read the bug right? Was the process in a healthy state when it occurred? Am I looking at the right version of the code? Did the feature ever work? Did I misclassify a success or failure when I did the binary search? Even if all my assumptions were correct, going through the problem again can sometimes spark new theories for investigation. This is also where I start when someone else asks me for help with their bugs.

If all else fails, or even if it hasn’t, searching on the internet can help. This is especially true for third party components or services. Be careful though, I am finding internet resources to be increasingly less helpful but your results may vary. Even though I’ve wasted a lot of time following red herrings from random internet threads, this is still sometimes the best option available.

Step 4: Test My Theory

I find it easy to jump to conclusions when I’m debugging, but my experience has taught me to approach bugs with a scientific kind of skepticism. Once I am pretty sure I know what the problem is, if it’s not a trivial change, I like to isolate it and prove to myself that I understand it.

I will write an automated test that reproduces the problem if at all possible. It might seem quicker to fix the bug first, but starting with the test will make the process simpler. Writing a failing test proves that you understand the bug, and it also proves that the test will fail if you don’t successfully fix it.

If you can’t write a test because that isn’t your team’s practice then you have my sympathies. On the bright side, you’ll be getting a lot more experience fixing bugs!

Step 5: Fix the Problem

I will remove some bad code and / or put some more good code in.

Step 6: Look for Similar Bugs

Sometimes a bug indicates a pattern of bugs. Before I fling my fix back out into the world I like to do a little research to see if the same mistake has been made in other places.

For example: if I was fixing a bug caused by a query operator that isn’t supported by an older database server version, I can do a quick search to see if the operator was used anywhere else. Since I’ve already figured out how to fix it, I can fix them all at the same time and eliminate a whole bunch of bugs.

Step 7: Test My Fix

Since I almost always have automated tests, I can check if I’ve fixed the problem pretty easily.

I’ll usually do a manual test as well to make sure I really have fixed the issue. Sometimes a bug has more than one cause, or is repeated in more than one place. To be honest, I often find this step tedious, so I have to remind myself why it’s important. A lot of time can get wasted if there was some aspect I missed. Also, if I’m going to put my name on a fix, I want people to be able to depend on it actually being fixed.

Step 8: Understand the Bug’s Impact

It depends on the bug, but if there is some damage left behind, I make sure to consider its impact. Sometimes this takes a bit of experimentation, but it is important. Sometimes a cleanup script is necessary, or sometimes manual steps can be provided to correct the issue. At the very least I want to make sure I can tell my stakeholders what the impact was.

Step 9: Bug Retrospective (Post-mortem)

Once in a while I take a bit of time to reflect on the bugs I’ve encountered. How did the bug escape in the first place? Is it likely that similar bugs will be introduced again? Can I introduce tools or change processes to make this class of bug less likely to occur?

Some organizations have a formal post mortem process for impactful issues. This is a great way to ensure a team is learning from its mistakes. I have introduced this process in a few of my teams and highly recommend it.

Even for bugs with less impact it can be worth spending a bit of time thinking about this. It’s not always feasible to prevent some types of bugs, but as craftspeople we should be trying!

Before a bug is fixed, it needs to be reported. Unfortunately it’s not uncommon to receive incomplete reports. We can spend a lot of time hunting and making guesses, and sometimes that’s enough, but if we can’t figure out the problem it’s pretty hard to fix it. This can be especially unfortunate when the stakes are high, and oddly, this is when it also seems to be the most common.

So what does a bad bug report look like?

The website crashed.

I get this kind of report now and then, even from experienced members of our support teams. I almost always need to send it back to get more information.

If I got this report instead, I wouldn’t (usually) need more information:

When I open the store website I get a 502 Bad Gateway error.

When I’m trying to understand a bug, I need to know four things:

• How did you encounter the bug? (Steps to Reproduce)
• What happened? (Actual)
• What should have happened? (Expected)
• Where did you encounter the bug? (Location)

Here is another example of a bug that would be hard to figure out:

I can’t delete a user.

It only answers one of the questions (Expected). I might be able to guess what the user tried to do (Steps to Reproduce), but there could be multiple ways to do this. It doesn’t tell me what application was being used (Location) or what happened (Actual). Was there an error message? Did the application close unexpectedly? Did the computer explode in a ball of fire? I can help with most of these, but they all get approached differently.

If you want a gold star, this is a more formal way to report the same bug, and the format I typically use myself:

Steps to Reproduce: 1. Go to the user list 2. Click a user 3. Click delete * Receive successfully deleted notification 4. Return to user list Actual: * User appears in the list * When I click on the user again, I get the error “User not found” Expected: * User does not appear in the list Location: * Admin site: https://{someserver}/admin/users

How did you encounter the bug? (Steps to Reproduce)

The first step to fixing a bug is trying to reproduce it. Exact steps can help a lot with this. Sometimes it isn’t enough, but it’s even more helpful when there are other factors involved.

What happened? (Actual)

A detailed, unemotional description of what happened is the most important part of a bug report. If there in an error message, give us the exact text. It may not make sense to you, but this can tell us a lot about what is going on. If possible, copy and paste the exact text of the error. A screenshot can also be really helpful.

Expressions like “crashed” or “failed” or “doesn’t work” sound good, but they don’t tell us what happened. You don’t need to use technical words if you don’t know them, but you should avoid interpreting the information. For example “an error message appeared that said …” is much more helpful than “it failed”.

What should have happened? (Expected)

Sometimes this is obvious. Generally when someone reports an error message we understand they expect not to get an error. Sometimes the expected behavior is not so obvious.

Keep in mind that programmers are trained in programming, and are usually not experts in the fields where our programs are being used. For example, if there’s a problem with the way tax is being calculated in an accounting application, the people looking at the bug may know less than you do on the subject. For technical bugs and miscalculations it can also be helpful to include how you determined the result you expected.

Where did you encounter the bug? (Location)

It’s not always obvious, but sometimes companies have more than one piece of software, or more than one version in circulation.

If you’re using a website, include the URL. If it’s a visual bug, maybe include the browser you’re using, it’s version, and the operating system you’re using it on. If it’s a mobile app, include the kind of phone you have. If the app has a version number include that too.

In the early days of my career, I got new responsibilities, promotions, and raises fairly regularly. It took a bit of luck, a lot of hard work, and a few years (but not very many years), to work my way up to a senior developer position. Senior means different things at different places, but eventually I got to a place where there was no easy next step, and I had a good number of peers in exactly the same position.

I can now proudly tell you that I have broken past the wall. About a year ago I achieved my goal and got promoted to the role of system architect. The following are the things that made the difference for me.

Choose the career path you want

There seems to be three distinct paths forward from senior developer:

• people and project management
• technical leadership
• focused technical expert

The focused technical expert path only exists in companies that need experts focused in very specific areas such as hardware manufacturers or OS / platform companies. If you are at the top of one of these specialties, you probably already work for the company where you’ll be getting promoted, and this post won’t really apply to you.

Both management and technical leadership are fine choices, but you should decide which is right for you. Here are some important differences to help you make your choice:

• Technical leadership positions are harder to get. There aren’t as many of them, and it seems they are more sought after.
• Management positions will eventually strangle out all time for development. At some point you may not be able to go back.
• Technical leadership positions require making harder technical choices based on research and experimentation.
• Management positions require making harder business decisions based on company and customer priorities.

Either way, they are both big changes from full-time development. You will be leaning heavily on your soft skills, spending more of your time in meetings and writing a lot of documents. You will typically have less control over the things you focus on, and your success will depend more heavily on the work of others.

If all this sounds awful to you, it is entirely fine to stay where you are. You can spice things up by changing projects or companies. You may want to scale up your impact some day, but until then, do what you love. The only unfortunate consequence is that you will likely be stuck in your current salary range.

Develop your hard skills and your technical breadth

Both leadership and management positions will favour generalists because you’ll typically have responsibility for far more parts of your system. If you’re after a management position, a basic understanding of different problem areas will be enough. If you want to be a technical leader, make sure you have some success stories in a few areas. If you’ve only ever worked in the back end, for example, see if you can get some time working in the front of the house.

Technical leaders are going to need to be up to date on all the latest technologies and be able to discuss them intelligently. If learning and working with new things isn’t possible where you work, it may be time to consider changing companies. Contracting helped me expand my portfolio dramatically in just a few years, but it’s not a path I’d necessarily recommend to anyone.

Find the right company and manager to help you advance

It’s always important to have a good relationship with your manager, but this is especially true if you want to advance your career. Make sure your manager knows what your goal is, and make sure they are supportive. A good manager will tell you what areas you need to improve, represent you when the right opportunity appears internally, or vouch for you if you find one outside.

Make sure your company has the role you want. Very small companies will probably not need a software architect, and may not have many openings for new managers. If you’re growing quickly it may appear in the future, but ask your manager. If you’re valued enough, they might even create the position for you when it makes sense.

If you’re in a place where advancing isn’t possible, or you aren’t getting the support you need, you may have to make the difficult choice and move on. When considering new roles, think about how they can help you work toward your goal. It’s also totally fine, maybe even advantageous, to tell a potential employer what you’re working toward, and ask if it’s possible with them.

Develop your soft skills

In your new role you’re going to need to juggle more priorities, build consensus with more people, and convince everyone that it’s all under control. You won’t need to be amazing at this on your first day, but you won’t get the job if they don’t think you can do it.

Here are some things to work on:

• Find your own personal organization system and get comfortable with it.
• Study development process and team management. There are lots of great books, blogs, videos, and classes.
• Give more presentations, and put effort into improving them.
• If you aren’t already, see if you can become someone’s manager.
• Try coaching / mentoring someone who is struggling.
• Run a study group or a book club.

Develop your personal brand

I think a lot of us developers lack awareness of our personal brand, but it’s an important part of your career, and it’s importance increases the higher up you go. Make sure people want to work with you, and try to get some high-profile successes under your belt. These will all help you earn a promotion, and give you essential credibility once you get it.

The short version: Make good tea with boiling water, then let it cool slowly before serving.

The long version:

Any decent black tea should work, but my favourite is loose Twinings Earl Grey (Amazon.com). The loose version tastes quite different than their tea bags, so do put in the effort to find it.

You will also need a fine strainer to remove the tea. I use a cloth tea sock (Amazon.com).

Also, be careful that you brew the tea in a container that can handle boiling water. A heat-safe glass pitcher is nice, but a regular metal pot from your kitchen will do just as well. Once it’s down to room temperature, you can transfer it to any container you like.

I add 25 mL of loose tea for every 1 L of boiling water. Make sure the water is actually at a rolling boil. The hot tap on your water cooler is not going to work for this.

Set a timer for 5 minutes. At about 2 minutes, stir the tea. At 5 minutes, remove the tea strainer gently without stirring. The tea seems to produce the most flavour about two minutes in, and the most bitterness at the end of the steeping. Stirring only the once yields the smoothest flavour for me.

Let the tea cool slowly to room temperature, then put it in the fridge to cool the rest of the way. Adding ice while it’s hot can disturb the flavour, and dilutes the tea.

Once it’s cold, pour a glass, add ice if you like, and enjoy.

You can experiment with different teas. Green or white tea should work, but follow their proper brewing instructions. Only black tea should be brewed with boiling water. My second favourite mix is using a plain black orange pekoe, inserting a couple springs of fresh mint after removing the tea.

Tuckman’s Stages of Group Development describe what happens when a team is formed. His theory has four stages: forming, storming, norming, and performing. As I stared writing this post, I noticed that the stages I was describing lined up fairly well with his. It’s important to note that I’m talking about joining an existing team, where he talks about a team being formed entirely from new people.

I will relate the stages I’ve noticed using his labels, but leave it as an exercise for the reader to compare to his more detailed descriptions.

1) Forming

When I first join a team, everyone is as clueless about me as I am about them. Anything I can do to facilitate interaction here is valuable. I like to set out a couple trinkets on my desk, things that show my personality and interests. This creates openings for others to start conversations, which can help to break the ice. It can be awkward and uncomfortable as the “new guy”, but getting through this sooner really helps speed things up.

I also tend to ask a lot of questions while I’m getting my bearings. The hard part here is gathering information without judging or commenting too much. Honesty is important, but this early in the process, it can be more damaging than helpful. It’s easy to forget that things were built with constraints and assumptions I wasn’t there to experience.

That being said, looking things over with a fresh set of eyes can uncover many interesting things. So that I don’t lose track of them, I write every idea and observation down and revisit them later.

2) Storming

After a month or two, I start getting used to what’s going on, and the problems the team is solving. The things that seemed strange before may now be routine, but some will still get in my way. My productivity improves, but I don’t feel like a full member of the team yet.

This is when I start going through my list of suggestions. I remove the items that no longer make sense, and then prioritize the remainder. I work through my list slowly, applying gentle pressure, trying to ask questions of various people on the team. Sometimes the questions receive insightful answers, other times they provoke good changes. Occasionally they uncover issues with team dynamics or lost political battles. All of these outcomes are valuable in different ways.

Essentially, I am trying to develop my voice. By waiting until I’ve been in the team a bit, I have a much firmer foundation to launch from. Being honest, and being able to disagree respectfully are necessary to fully contributing to the team. Just remember to take it slowly. If you come on too strong too suddenly you can alienate your coworkers.

With time, everyone gets used to me and my style. Once they realize that I can challenge opinions without judging their source, things become a lot smoother.

3) Norming

After a while, I run out of questions. At this point, I will have a pretty good understanding of the architecture, the history and opinions that shaped it, and have a rough idea where people stand on the important issues. Others should have a good idea where I stand too.

This is when I start fine-tuning my personal processes, and trying to resolve anything that’s still holding me back. I expect to have a good relationship with my manager by this point. I would be bringing up more serious issues earlier, but now I want to start bringing up everything else. If our relationship strong enough, this is when I would try to fix that also.

Sometimes I never make it to this stage. I might be spending too much energy arguing, or feel like my contributions are not appreciated. If I don’t make it here within a few months, and have no clear path to improve things, I know it’s time to brush up my resume.

4) Performing

With good working relationships, enough context about the problem space, and all my major issues resolved, I can now focus on getting some work done.

As a young adult, I tried loose tea on the advice of a friend. It was a totally different drink. Black tea tasted rich and warming. Earl grey had a wonderfully soothing aroma. Green tea had a nourishing earthy taste that made me feel good when I drank it. More importantly, there was very little bitterness, so I could skip the milk and sugar, and enjoy the flavours even more.

The reason why loose tea tastes better is basic chemistry. Tea is damaged by exposure to air. The tea in tea bags is ground to a fine dust, then spread out into thin mesh pouches. The larger surface area accelerates the ageing process, robbing the flavour and aroma.

It may be true that tea bags are easier than loose tea, but with a little equipment they aren’t much easier. Most of the time, I use a simple strainer that sits in the cup like this one (amazon.ca, amazon.com). Measure some loose tea into the basket, pour boiling water over it, and remove the basket once it’s finished steeping. The lid can be flipped to hold the wet basket, so you don’t even need to stay near the sink.

At work, or when I’m travelling, I use a tea thermos with a built-in strainer made by a local company. I put dark teas in the strainer part, steep with the container upside down, then flip and remove both lids to drink. It’s easy to use, and it makes it easy to carry hot tea around without spilling or making a mess.

Loose teas can be harder to find, but there are many great sources online. If there is a tea store in your neighbourhood, going in and selecting a few teas can be a lot of fun. Except for the very highest grades, most good teas aren’t too expensive, especially when compared to premium tea bags.

Once awakened, I started taking a lot of pleasure in hunting down and trying new teas and brewing equipment. I have been researching and experimenting for more than a decade now, and I still feel like I’ve barely scratched the surface.

Sometimes I am forced to drink tea from tea bags while I’m in a restaurant or visiting family. Some tea bags that are better than others, but all of them seem lacking compared to my own stash at home. It’s enough that I’ll never forget why I put in the extra effort to enjoy my cup of tea.

I enjoyed Angular. It was straightforward to use, and allowed me to bang out a lot of functionality without much cumbersome boilerplate code. Jasmine, the testing framework set up in the bootstrap source, was also pretty slick. I really liked how I could nest a bunch of test blocks inside of each other to reuse common setup code.

The only serious bump I ran into was getting multiple controllers to work together.

My goal was to build a wizard-style flow. A user enters a bit of info, hits a button, then enters more info. The answers in one step affect the questions in future steps, or might cause steps to be added or taken away. I wanted the controller to trigger the navigation, and I wanted to pass state when it did.

View-led navigation would have been easy: add a link whenever you like. A user clicks and the new controller is loaded. This kind of navigation is great for keeping controllers ignorant of each other. I suspect this approach is better for search engine indexers as well. It wasn’t ideal for me.

I searched the Internet and read a bunch of documentation hoping to find an easy answer. What I found was a bunch of other people asking the same questions. When I finally decided to build it myself, the solution was easier than I expected:

'use strict';

// var module = angular.module('...', []);
module.service('navigation', function($location) {
  var storage = null;

  return {
    navigate: function(path, data) {
      storage = {
        path: path,
        data: data
      };
      $location.path(path);
    },

    getNavigationData: function() {
      if ((storage == null) || (storage.path != $location.path())) {
        throw 'navigated without passing data';
      }

      var data = storage.data;
      storage = null;
      return data;
    }
  };
};

To navigate, I used code like this:

module.controller('Page1Controller', ['$scope', 'navigation', function($scope, navigation) {
  $scope.navigate = function (navigationParameter) {
      navigation.navigate('/Page2', navigationParameter);
  };
}]);

In the destination controller, I fetched the navigation parameter like this:

module.controller('Page2Controller', ['$scope', 'navigation', function ($scope, navigation) {
  $scope.navigationParameter = navigation.getNavigationData();
}]);

It was easy to test this code. Whenever I expected a controller to navigate, I checked the value of $location.path(). To pass navigation parameters into controllers, I just called the navigate method before the controller was created in the setup block.

Unfortunately this solution breaks the back button. Because the browser triggers backward navigation, the navigation parameter won’t be set when the controller tries to load. This wasn’t something I needed, so I left alone.

Assumptions

The patterns I’m describing assume a common architecture: you have some boxes in the front that have your traffic balanced between them, and some stuff behind them that’s not balanced. For the purposes of this discussion, it doesn’t matter if it’s one or more apps in the front, if you have a service tier, or what kind of data storage you use. Anything that’s redundant is going to be called distributed, and anything that’s not is going to be called central. You need some way to track user sessions, and the ability to detect disconnects within a few minutes. Some of these graphs also depend on a load balancer that’s configured to keep a single user session on the same distributed server.

Healthy

This is how a healthy system should look. I’m showing two views, one broken into a couple regions, and another just showing the total within a shorter time span. It’ll be a lot easier to show these patterns on the zoomed in graph, so I’ll use that as a base line for the following examples.

I was surprised when I first saw the smooth wave-like pattern a connected user graph makes. These examples are a pure sine wave because it was easy to produce, but it’s pretty close to what I’ve seen on real systems. The waves might get a little higher and wider on weekends, but it’s always a smooth line when things are normal.

The numbers and data shown are totally fabricated, and do not represent any of the systems I’ve worked on. My focus here is on the disruptions to the lines. I have observed all of the patterns I’m showing in real production environments, some of them numerous times.

Central Component Malfunction / Failure

This is the worst case for a server system, and as you can see, the results are drastic. You can tell it’s a central component because the number of connected users drops very close to zero. You’ll also note that I show the connected users shooting back above the norm. This happens when users try to reconnect once or multiple times when the system becomes unresponsive. This is a pattern you will notice during most malfunctions.

Distributed Component Malfunction

This is a much more common occurrence in a server system; a server starts to malfunction without losing the ability to respond to network traffic. The load balancer doesn’t detect a failure, but users have serious trouble using the app. You will see a serious fluctuation in the graph as users start disconnecting and reconnecting, slowly getting pushed to new working servers. This is one of the reasons why it’s important to have sticky sessions on your load balancer.

Distributed Server Failure

When one of your distributed servers fails outright, you see a much more sudden gouge, but proportional to the number of servers you have. The graph returns to normal fairly quickly once the load balancer detects the bad server, and users finish logging back in.

Application Overloaded

Performance limits can be distinguished because they get worse as the number of connecting users increases. This example shows a hard wall, but the severity you observe will depend on how your system is breaking down. The key indicator is the subtle twitching that gets progressively worse as the pressure builds. The deep downward spikes occur as various parts of the system start throwing large quantities of errors.

Central Component Performance

This is what it looks like when a central component starts to have a performance issue. When it occurs off peak, it’s a good clue that some critical system is acting up. If it’s not obvious what’s wrong, here are a few things you can check: failed hard drives in your storage system, hardware errors in your system logs, unusual latency with a heavily-used APIs, or perhaps someone is running an ad-hoc query on the production database.

Denial-of-Service Attack

Denial of Service attacks are awful, and unfortunately effective things. They look different than network gear failures because attackers have trouble ramping up load generators quickly. Once they do get going though, your networking gear will usually start failing, and nothing will get through until they stop. Most DOS attacks are network-level, so you shouldn’t see increased activity or connections before or during the attack.

Television Ad

Advertising should increase your number of connected users. If your ad hits lot of people at the same time, such as a TV ad, you’ll see a bump like this. There will be a spike just as the ad airs, a bit of hang time, then it starts to trickle down to normal. The size of the bump will depend on the effectiveness and reach of your ad.

Television Event

This is one of my favourite patterns. This is what happens if there is a big event that your audience is interested in. An example would be a sports site during the Super bowl. You see a dip while it airs, then you go back to normal when it ends. If people don’t like the event, the line might start returning to normal sooner.

Monitoring Failure

A flat line like this is almost never real, except maybe during deliberate maintenance windows. If it’s not obvious why you’re flat, you should check that your monitoring and graphing systems are collecting data correctly.

A member of our operations team was installing a Windows service I’d built to monitor some stuff in our production environment. I’ve made a few windows services in my day, and installed them many times on many machines. I’d even installed this one on my development machine with no issue. In our staging environment, however, this is what we got:

C:\Install\TheService>C:\Windows\Microsoft.NET\Framework64\v4.0.30319\InstallUtil.exe TheService.exe Microsoft (R) .NET Framework Installation utility Version 4.0.30319.1 Copyright (c) Microsoft Corporation. All rights reserved.

Exception occurred while initializing the installation: System.BadImageFormatException: Could not load file or assembly ‘file:///C:\Monitoring\Service\TheService.exe’ or one of its dependencies. An attempt was made to load a program with an incorrect format.

We checked the likely things: the framework version, the platform the app was built for, even re-copying the files in case they somehow got corrupted. When these didn’t work, we started trying more radical things: forcing all assemblies to 32 bit, even running the service as an executable to see if there was some error in the app.

In my defence, we are both experienced engineers, and I’m not the only person who missed it. Look closely at the command line we used:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\InstallUtil.exe

Long version: Service applications in Visual Studio 2010 are 32 bit by default, and this is a reasonable default for them to have. We were trying to install the 32 bit service with the 64 bit version of InstallUtil. InstallUtil loads the target assembly to access it’s installation instructions, but you can’t load a 32 bit assembly from a 64 bit application (or vice versa). If you try to, you get a BadImageFormatException.

Short version: Two numbers derailed my entire afternoon.

It would have been nice if the error message from InstallUtil was a little more specific, but I suppose this isn’t a common problem. At least I got a good reminder about the importance of checking the small details when the big ones aren’t bearing fruit.

When you change from trying to do everything to doing what’s most important, you need to decide what important means. Figuring this out for myself has proven difficult. I still don’t have all the answers, but I am constantly making progress.

Deciding what isn’t important has been easier. I am abandoning Themis, my attempt at an open source project. I no longer have a need for it, and it will take a lot more effort to get it finished than I originally expected. I would be delighted if someone wanted to take it on, but there isn’t enough there for it to be likely.

Writing in this blog has also been pushed down the priority list. I intend to keep going, but instead of forcing out a steady pace of content, I’ll wait until I have ideas that I have a strong desire to share.

One of the worst things about working in the software business is the high rate of change. If you only focus on your job and stop paying attention, you could wake up one day unemployed with an obsolete skill set. Keeping up to date is a lot of work, and it becomes harder every year.

Some skills hold up better over time. Solid work habits are priceless, and can help you advance your career even where technical skills are lacking. Work management, software design, and coding practices are good subjects to study as well; they don’t age nearly as fast as specific technologies, and they can make a big impact on the quality of your work. Unless you become a manager however, you’re still going to need to study technical stuff.

The only way to really learn a tool is to use it, so study at home isn’t always enough. Some employers will deliberately favour new technologies to keep skills fresh and employees happy, but this is not the standard. If your company starts to lag behind industry trends, you might consider changing jobs while it’s still possible.

There are some tricks that help me stay up to date: I listen to podcasts while exercising or doing simple chores. Public transit is a great opportunity to read while commuting. A smart phone helps me squeeze in a few extra blogs and tweets when I’m waiting for something. This is primarily how I follow industry trends.

To learn specific technologies, I’ll spend an hour or two a few times a week reading or watching videos. I need to play with something to understand it, so I spend an odd evening banging away at some pet project.

My study used to be limited to an occasional book or seminar, but increasing my study time, and opening myself to podcasts, blogs, and twitter has made a huge difference in my career. I have more to contribute in design meetings. I have opinions on various technologies I haven’t even used. It takes more energy for sure, but I feel like I am a better developer for it, and I don’t regret it one bit.

This business is a challenging one, but exciting as well. If you are the kind of person that loves to learn, explore new technologies, and push yourself to constantly look at problems in new ways, then embrace the change and you will do well.

What I’ve learned, however, is that it takes more than time to get things done. After about a month, pretty much all I’d achieved was completing a couple video games, and putting all the things I wanted to do into one very long list.

My first thought was that I wasn’t focusing, so after listening to a podcast by Scott Hanselman about organization, I decided to try the Pomodoro Technique. This is a fairly simple system where you try to focus completely on a task for 25 minutes at a time, shutting out all distractions.

After a bit of Internet searching with no results, I started making my own pomodoro timer application. I had something working in a couple hours, barely holding myself back from adding sounds and animations. It would have been a lot faster, but I wasn’t completely focused on the task.

Armed with my timer, ready to experience my first burst of productivity, my wife came home from work. I told her excitedly how I’d built this magical thing, and how I’m going to start getting so much done. Then she asks me one simple question, “Why didn’t you use the timer from the kitchen?” This is when I realized that my problem was focus on a whole other level. I was doing work, but I wasn’t doing the right work.

This is a common problem in the software world: getting overwhelmed by a giant list of tasks, doing a little bit of everything, and completing nothing. I’ve experienced this first hand, and have been using effective tools for dealing with it for years, but I had never considered applying them to my personal life.

I tried a few things next, but the best help came from the book Getting Results the Agile Way, which was also mentioned in the podcast. It combines agile philosophy with some ideas about balancing life priorities and energy levels. I’m not committed to the entire process, but the parts I am trying have already made a big difference. By focusing on three goals a day, I’m now managing to plow through even dauntingly large tasks. Though I’m not doing everything on my list, I’m getting the most important things done, and because I feel that I’m making progress again it’s motivating me to work harder.

Taking the time off has been really good for me. Beyond organizing my personal priorities, I’ve also been organizing my main workspaces: my office, and my kitchen. You could say the experience is like rebooting a computer; it took a bit of time, but now I feel fresh and full of life, ready to tackle something new.

I was recently working on a new application. After three months in the field, users were starting to complain about performance issues. We had done some limited performance tuning for the first release, and more as part of the second release, but new issues were popping up as more data got entered into the system. We could have continued fixing issues as they came up, one release at a time, but we wanted to get ahead of the problem, and the client wanted to know that the system would remain usable without developer intervention for a few years at least.

The nature of the business was such that the rate of data entry and number of concurrent users shouldn’t change much over time. This meant that increasing the amount of data would be a good approximation of how the system would look in the future. How do you increase the amount of data in a normalized relational database? It would be difficult to enter realistic test data manually, and unrealistic test data can cause unrealistic test results. Bad results means we waste time fixing issues that wouldn’t appear, and never get a chance to detect issues that will.

I proposed creating a T-SQL script that would double all the existing data. Running it once would simulate six months of usage, running it again simulates a year, twice more and we’re simulating four years of usage. Once it was approved, it took me a couple days to finish, about half of that going to testing and debugging. The result is the most impressive T-SQL script I’ve ever written.

The script I wrote didn’t do the actual duplication, it combined hand-entered metadata with data from sys.tables and sys.columns to generate a much bigger script. Not only was it faster to write this way, but it made it easier to fix issues, and it allowed the script to be reused after columns were added to the database.

I had one table variable with a list of tables to copy, and another defining the relationships between foreign keys and their source tables. Most foreign keys were named the same in all tables that they appeared in, so a single mapping was often enough for all of them. Most of the mappings could be determined from the list of tables itself, so I only had a few other relationships and special column rules that had to be entered manually.

Another factor that helped was our use of guids for all primary keys. Because the can be determined before inserting a row, it was possible to generate the mapping from old to new key at the start of the script. I could also use a single insert statement for each table, and the order of execution only mattered where foreign key constraints existed.

The results were tremendous. We found a bunch of issues we wouldn’t have found otherwise, and had a fairly solid indication of how the application would behave years in the future.

It came up at work during a discussion about interview questions. My colleague was both proud and surprised to have been the only person interviewed for his position that had known what it really means.

A critical path is a project management term that refers to the chain of dependant tasks that will take the longest amount of time in a project plan. For example, if I were to make a project plan for my breakfast on Sunday morning, it would look like this:

In this plan, making pancakes is the critical path. With an unlimited amount of people, and an unlimited amount of kitchen space, and all the resources and tools we would need, the fastest that we could produce my Sunday morning breakfast is the time it takes to make pancakes.

I can steam milk, flip pancakes, and toss a couple eggs in the pan all at the same time, but grinding coffee beans and mixing the pancake batter has to be done one at a time. This breakfast takes me just a few minutes longer than the critical path, so it’s not worth hiring a sous chef.

The time of a software projects is usually shaped more by the number of developers. We usually have a lot of features to deliver, and most of them are not directly dependant on each other. What you can see is a part at the beginning and end of a project where it’s hard to keep multiple people working, but that’s a whole other thing.

When a critical path is defining the time line for your project, it’s important to identify it; any step that gets delayed affects the entire chain. You should start these tasks as soon as possible, and do what you can to stop them from getting blocked.

The decision to leave my job of three and a half years wasn’t an easy one. I was working with some fantastic people, building the best software we could, but at the end of the day I wasn’t learning. The work was challenging, but the challenges were mostly non-technical. In this business, if you’re not moving forward you may as well lay down and die. I couldn’t change my situation, so I said my goodbyes and left.

Instead of settling in somewhere else for another long haul, I felt it was time for a change of pace. Now I get to hop from organization to organization working on new problems and broadening my skill set in the most efficient way possible. I might settle down if a good opportunity comes along, but for now I’m enjoying the freedom.

I’m currently working on a fun little contract, but I’ll be looking for my next opportunity soon.

If you want help designing or building awesome software that delights users, we should talk. I’d be happy to put my experience to work building you exactly what you need.

int Average(int a, int b)
{

}

It’s a great question because it’s a simple problem, and anyone that comes through the door should know the formula without having to ask. It’s interesting how people react to it sometimes, but I chalk it up to interview jitters. About half the people interviewed answer like this:

int SimpleAverage(int a, int b)
{
    return (a + b) / 2;
}

It’s a good first answer too, except that it doesn’t always work.

“What will be the result of your function if it’s called with the values Int32.MaxValue and Int32.MaxValue?”

The answer I want to hear is that it won’t work. Some people think (like I did), that it will throw a System.OverflowException, but it actually returns -1. The default for C# projects is to skip overflow checking for arithmetic.

This is another answer that I get from time to time:

int SystemMathAverage(int a, int b)
{
    return System.Math.Average(a, b);
}

I respect this answer; it indicates that a developer wants to leave low-level details to the framework, which is a perfectly reasonable attitude when developing business-class software. Unfortunately there are two problems with this answer: it defeats the purpose of the test, and it doesn’t exist in the framework. There is the Linq average statement which can be used like this:

int LinqAverage(int a, int b)
{
    int[] inputArray = new int[] { a, b };
    return (int)inputArray.Average();
}

After the first attempt, I indicate that I want a function which will work correctly for all values of a and b. The way candidates tackle it tells you a lot about them as a developer. Here are some behaviours I’ve noticed:

• Very few candidates stop to consider a solution before answering.
• About 3/4 of candidates try to use an if statement.
• About 1/4 of candidates write the “if” on the board before deciding what to use it for.
• Occasional candidates will challenge the necessity for handling high values.

A clean correct answer is hard to pull off on the spot, especially under pressure. I’ve never seen it, and I wouldn’t be surprised if I never do. This doesn’t bother me either. The key thing I’m looking for is the ability to understand questions and discuss the source code. These skills are essential to working in a team.

How did I come up with such a brilliant interview question? I didn’t. Once upon a time in a company far far away, someone asked me the very same thing. I was pretty proud of my answer too, until I tested it. This was what I came up with:

int JessesAverage(int a, int b)
{
    return (a / 2) + (b / 2) + (((a % 2) + (b % 2)) / 2);
}

I wrote a test harness in some spare time after the last round of interviews. Here’s how each of the answers measure up:

Results for SimpleAverage(): Near Zero: 121 of 121 passed. End of Range: 0 of 3 passed. Failed Values:

• SimpleAverage(2147483647, 2147483647) returns -1, but should return 2147483647
• SimpleAverage(-2147483648, -2147483648) returns 0, but should return -2147483648
• SimpleAverage(2147483647, 2147483645) returns -2, but should return 2147483646

Results for LinqAverage(): Near Zero: 121 of 121 passed. End of Range: 3 of 3 passed.

Results for JessesAverage(): Near Zero: 109 of 121 passed. End of Range: 3 of 3 passed. Failed Values:

• JessesAverage(-4, 1) returns -2, but should return -1
• JessesAverage(-4, 3) returns -1, but should return 0
• JessesAverage(-3, 4) returns 1, but should return 0
• JessesAverage(-2, 1) returns -1, but should return 0
• JessesAverage(-1, 2) returns 1, but should return 0
• JessesAverage(-1, 4) returns 2, but should return 1
• JessesAverage(1, -4) returns -2, but should return -1
• JessesAverage(1, -2) returns -1, but should return 0
• JessesAverage(2, -1) returns 1, but should return 0
• JessesAverage(3, -4) returns -1, but should return 0
• JessesAverage(4, -3) returns 1, but should return 0
• JessesAverage(4, -1) returns 2, but should return 1

This was the best answer I could come up with after some experimentation. I’m not sure why, but the need for type conversion bugs me.

int ConversionAverage(int a, int b)
{
    return (Int32)(((Int64)a + (Int64)b) / 2);
}

Results for ConversionAverage(): Near Zero: 121 of 121 passed. End of Range: 3 of 3 passed.

One thing that makes the experience considerably more enjoyable for me is the team of coworkers that come in to help. There’s a handful of us, each with our own responsibilities. I deploy the applications while someone else monitors the database and another person tests the system to make sure it’s running normally. We back each other up, help out where we can, and make decisions together when they need to be made quickly.

Pressure is never a desirable thing in a work environment, but one benefit is that it quickly builds trust between anyone facing it together. The people I work with are really fantastic: smart, dedicated, and fun. We come from different cultures, like different kinds of food, have different hobbies and different tastes in music, but we still find things to talk about, reasons to laugh and smile.

I’ve been trying ways to make releases more fun. We had a pot luck breakfast once, and a release soundtrack from team favourites another time. This time I made breakfast burritos, and as a joke, a doughnut salad topped with an espresso sauce. We played a few songs during the waiting periods, and had as much fun as anyone could that early in the morning.

Thanks to the people involved, this release went well despite a few hiccups.

I’ve used this pattern of processing an established or complex data format into a simple proprietary one before with good results. It allows you to isolate the complexities of a data format in the code that processes it, keeping the rest of your code clean. Like everything, there are cases where I wouldn’t use it, but for this kind of scenario it should work well.

Now that I’m building this neutral representation, my next thought is how it will be saved to the database. I don’t need a database at this stage of the project, so part of me wants to ignore the decision until later, but the whole point of the system is to persist and interact with these types, so it’s important that I get it right. If I make simple POCO classes now, and start writing a bunch of code that uses them, I might have to change a lot of code later if I want to switch to types generated by Entity Framework. I could write my own custom code to read and write my own types from the database, then I can use any type I want without restrictions, but it would be a waste to write this plumbing code myself when an ORM can do it faster and better.

Creating a table design now wouldn’t be a simple matter either. I don’t know enough about the needs of the scheduling service to know which parts of the VCalendar format to bring over. If I try to guess now I know I’ll bring over a bunch of stuff that I don’t need, but starting with a simple table and adding fields every time I need one is no fun either. Adding a database to a system is like attaching a ball and chain, and I want to wait until I can be sure I have my model correct before I do it.

I need to make a decision, so here it is: I’m going to keep building my own types, and try to keep them. When it comes time to hook up a database, I’ll play around with the new POCO support in Entity Framework 4, and if that fails I’ll try another ORM tool. I may need to change my model a bit to suit the ORM, but I’m hoping that it wont need to change much.

We used to have a system where anyone that could fix a problem would get a text message, and whoever was closest to a computer would respond. It worked most of the time, but there were some issues. The team creating the tickets would often be unsure of which component was broken, so the issues would hit lots of people who would be unable to help, and it also exposes a quirk of human behaviour that makes us less likely to help when more people are available.

Thankfully the whole plan is being reworked. Now a single volunteer will carry a phone that gets all the support tickets. They will respond to issue within a reasonable amount of time, and if they can’t solve them themselves, find other people to help. The company has also added decent incentives to be on call, and is investing in tools to make the job easier.

As part of the preparation, I’ve been asked to do presentations and create supporting documentation for all of the first responders. I enjoy doing the presentations, and everyone seems to be benefiting from the content, but it’s a lot more work than you might expect. The group of people I’m working with is comprised mostly of heads-down developers. They’re smart people, and all wanting to learn, but most don’t have any IT experience.

My first session covered these topics:

• how to think and act like a seasoned IT pro

• a brief description of our hosting environment, servers , and where the various applications are deployed

• how to recognize network problems, hardware problems, and OS problems

• the symptoms to expect when any of our key systems fail

• how to safely restart our key systems

It took me about an hour and a half to prepare, and two hours to present. I’ve spent about a day of effort writing it down, but would guess that I’ve covered less than half of the material I talked about. To be fair, the written material is laid out quite differently from the presentations; I am trying to capture a series of troubleshooting guides that can be used in an emergency.

Are you curious about the difference between a developer and an IT pro? The most significant difference I see is that we developers are used to an environment where we can test changes quickly and safely: change a line of code, hit F5, and see what happens. IT pros take the opposite approach because the cost of a bad change in a live production environment can be devastating. They need to be certain what any change will do before it’s made, and be prepared to roll it back if there’s any hint of trouble. It can be frustrating for a developer to work this way because it’s a serious effort to make even simple changes, but it’s this mentality that keeps the world’s servers running.

Outlook invitations are sent in the VCalendar format, content type “text/calendar”. The standard was published as RFC 2445 in 1998. It describes a standard layout for calendar data in the VCard format, which is described in RFC 2425.

VCard is a simple text based format with a nested structure that’s similar to XML. It uses colons and semicolons instead of angle brackets, supports attributes, and has standard representations for several common data types. Here is an example of the same data represented in both formats (content from RFC 2425):

I looked around for a library to do the job,but decided to build my own. Writing this kind of code isn’t a good way to deliver business value, but it sure is fun. A bit of time with a white board and a couple of evenings coding, and I have a mostly functional VCard parser ready to go.

When I am setting out to build something like a parser, I like to consider the designs of other established libraries that perform a similar function. As I’ve already mentioned, VCard is a lot like XML in structure, so XML parsers are a great place to look for inspiration. I built a structure that loosely mimics the XmlDocument model, an object tree that holds the entire document in memory. It won’t perform as fast as using something like an XmlReader, but it makes it easier to handle variations in document format with polymorphism. Since the VCard documents I’m processing should never be too big, it won’t be much of a performance burden anyway.

The object structure that holds VCard data looks like this:

At first I wanted to have a property like Value, but there is no reliable way to know the type of a VCard value without knowing the structure of the document, and at this level I don’t want to be coupled to the structure of the document. Instead, I will be adding a series of methods to get and set the EscapedValue as a specific type.

The VCardSimpleValue class was a late addition to the model. I needed a way to hold parameter values (the equivalent of attributes in XML), but since they can’t have parameters of their own I didn’t want to pick up the Parameters collection. I also considered making a type separate from VCardValue for the parameters, but both these classes will need the ability to read and write the escaped value, and I don’t want to write that code twice.

I’ve added the first unit tests to the solution, checking the parser against a number of examples in the specifications and real snippets extracted from emails sent by Outlook. I’ve also added tests for a number of failure cases in the parser such as groups without an ending or values lines without a value delimiter.

My next addition will be parsers for the value types and a stub in the test harness that replies to emails with some info about the original request.

The code at the time of this post is available here: https://github.com/jessemcdowell/themis/tree/8e0c428df0e33145df0323d6e9ae617114dd0f85

Fixing the problem was easy: Set all the assemblies to compile for the 32-bit platform, check it in, build it, ship it.

Except that my web service now looks like this:

Server Error in ‘/MyApp’ Application.

Could not load file or assembly ‘MyCompany.MyApp.MyAssembly’ or one of its dependencies. An attempt was made to load a program with an incorrect format.

This application has two parts on the server: a windows service that processes emails, and a web service for interacting with them. The two parts share a database and a bunch of business logic, so they also share some assemblies. The windows service works great now that I’ve converted all the assemblies to 32-bit, but the web service wont start.

Although the error message is a bit misleading, it wasn’t that hard to find the problem. When IIS is running in 64-bit mode (as it tends to do on 64-bit servers), all of its application pools will be 64-bit, and a 64-bit application pool cant load a 32 bit assembly. These are all the ways I can think of to fix it:

• IIS 7 can run an application pool in 32-bit mode
• IIS 6 can be configured to run in 32-bit mode, or you can run a second instance in 32-bit mode, though I don’t imagine both instances could share the same ports
• Write 32-bit and 64-bit versions for any platform-specific operations using conditional compilation, then build two versions of your assemblies
• Rewrite the malfunctioning parts of your app so you don’t have any platform-specific code
• Compile most of your assemblies for ‘Any CPU’, but compile entry-point assemblies (such as my service executable) for 32-bit

A configuration-based solution isn’t ideal based on the number and management practices of our environments. Finding and removing platform dependence would be the best technical solution, but it would be risky and time consuming. Switching some of my assemblies back to ‘Any CPU’ did the job with a minimal amount of impact.

My wife bought a pack of index cards. This is what a normal person would do. To me, it seems like defeat. Why use paper when there’s a computer in the room? While she’s at the store, I grab a white board and sketch out a data model.

I used WPF for a UI, two simple view models, and Entity Framework 4 to store the lists of characters, words, and phrases. I started out with SQL Compact Edition 3.5 as a database because that was the one that was there. That didn’t work out so well, so on the advice of a colleague I switched my app to SQLite.

This is what I found:

SQL CE 3.5

SQLite

Pros

• Can be created easily from the VS designer

• Can view and modify data inside Visual Studio

• Supports Identity Columns

• Easy to deploy

• Lots of good tools available for managing databases

• Can be used from non .Net and even non Windows applications

Cons

• Doesn’t support Identity Columns in EF

• Requires extra installation on client computers

• Generating tables from a model is painful

• Many model changes require regenerating the database

• Doesn’t map Integer type correctly

• Doesn’t support foreign key constraints

• Generating tables from a model is somewhat painful

SQL Compact Edition 3.5

Before I start dumping on SQL CE, I should say that I was trying the 3.5 version, not the fancy new 4.0 version. Maybe they’ve fixed all these issues.

Creating the model was easy, but as I went through a few iterations of the data model I quickly became frustrated with the inability to change the database to match the new model. I’m sure it’s possible to change it, but you have to write the scripts yourself and hope that you’ve named everything exactly right. Entity Framework created a handy database-generation script, but it won’t run directly on as SQL CE database because it doesn’t support the GO statement. Copy and pasting each section of the script is annoying, but still better than typing all the names manually.

The real shock was adding data from inside my app. Entity Framework can’t generate data in a SQL CE database if the primary key is an identity. The best workaround I could find was to use a regular old integer for a primary key and search the model for the highest key value every time an entity is created. Not fun at all.

SQLite

It took a bit of fooling around to switch my database to SQLite. Most of the creation script ran on the database, though the types had to be changed a bit to match the database. All integers in SQLite are actually Int64 in .Net, but the Entity Framework maps them to Int32 in the model. Modifying the mapping files manually is annoying, but not hard to figure out.

Once the tables were set up and working, it was easy to copy the data over. The database tool I’m using for SQLite can import CSV files, even the multi-byte Japanese characters came across correctly.

The only thing left is to study.

You can get a .Net library for SQLite here: https://sqlite.phxsoftware.com/

It’s able to integrate with Visual Studio, even the Express version, but it didn’t work well for me. With a good database management tool you don’t need it. I’ve been using SQLite Administrator, which you can get here: https://sqliteadmin.orbmu2k.de/

They really nailed the faster and cleaner parts. I didn’t find the user interface in IE8 too busy, but seeing the improvement I’m happy for the extra space.

Plunking a site onto my task bar like it were an application is pretty awesome. There are a couple of websites that I’ll do this for without hesitation.I wonder how long it’ll take to see adoption of the magic html jump list task extensions. I’m not terribly optimistic myself; I’m hoping someone will find a way to add tasks and maybe an overlay to sites that don’t have them built in.

In spite of all of that, sadly, I had to remove it.

The most severe issue I found was with the Live Mesh Remote Desktop service. My mouse cursor kept flicking to the the busy cursors and back again a few times a second no matter what application I was using.Also, myCPU was running at about 26% where it normally hovers closer to 2%. Closing the browser doesn’t fix it either.

It took a bit of digging around in Resource Monitor, but I was able to find the problem. It wasn’t a single app running the CPU, it was dozens of processes starting and terminating every second. That explains the mouse cursor. The two apps in question were the Windows error reporting service and the Live Mesh Remote Desktop service.

The solution was easy: set the Mesh Remove Desktop service to disabled. You won’t be able to access the desktop, but using everything else won’t suck. I don’t use Mesh desktop sharing enough that it’s a problem for me, but anyone who does may want to try it’s new replacement.

I could have lived with the disabled desktop sharing if it were the only problem, but a few hideous rendering problems in Google Calendar, and WordPress failing to save my posts was a deal breaker. The compatibility mode doesn’t seem to be compatible enough, and tweaking the settings in the developer console (F12) is too much of a nuisance to do every day.

IE9 came off pretty easy once I figured out how to do it. It’s listed as an update, not as an application, so you need to click on the the “View installed updates” thingy of thePrograms and Features menu.

I hope they get the bugs worked out soon; it’s only been a day and I’m starting to miss it already.

The Name

The first order of business was to choose a name. It helps to know the name before you create a project. I’m also hoping to put the source in a public repository right away so that I can talk about the coding process while I build it.

Themis comes from Greek mythology. According toWikipedia, she “is the embodiment of divine order, law, and custom.” That seems somewhat applicable to the subject at hand. I know thatGreek mythology isn’t too creative, but we can always change the name later if anyone has a better idea.

Application Structure

This system should be very simple. It’ll have a windows service to check for invitation emails, a web page to view schedules and configure the service, and a database to hold all the information so that both apps can access it. For simplicity, I want to put most of the code into a single assembly.I want to the web UI and the service runner to be a thin veneer over the main assembly.

Application Design

Although this is just a home project I do want it to be good quality, even if my name weren’t going to be pasted into every source file with the license. To achieve this, I will be using unit tests in as many areas as I can add them; not only to catch my mistakes but to help others understand and maintain the system after I’m gone. To make unit testing easier, I want to use anIoC / DIcoding style.

I’m going to use Entity Framework to keep the application code ignorant about the database. I don’t want to be tied to a particular database system, though I suspect most installations would use a sqlce database. I could use any ORM, but I’ve been wanting to try EF.

Processing Mail

Now that enough of the design (and some of the non-coding elements) have been straightened out, I can start to really focus on the first part of the project:Processing email.

I want to build the email handling logic as a group of small components,each responsible for a single aspect of the process. Aside from some shared types to hold email or request data, and the controller that knows how to orchestrate them, none of the components will know anything about each other. Even then the controller won’t know which types contain the actual implementation thanks to the IoC container.

By building each part in isolation I get a few good advantages. First, I’m able to test each component completely and separately from the rest of the system. Second, it becomes possible to replace any part of the chain with nothing more than a configuration change, which will be especially handy if anyone wants to fork the code. Third, I can reuse the individual components for other parts of the system, and they are likely to require less refactoring before they can be reused.

This is the new coding style: you build a system with complimentary building blocks loosely hooked together. You end up with a lot of classes, but you avoid the pain of complex inheritance trees.

The Next Step

Now that I’ve figured out my approach, I see that my first step is too big for an afternoon. Instead, I’m going to tackle it in three smaller steps:

1. Receive any email and send a simple response back.
2. Receive event invitations by email and send a human-readable response listing the details of the invitation.
3. Receive event invitations by email and send an acceptance email. (The original requirement.)

All I have left is to make a good cup of tea, turn up the music,and bang some keys.

I don’t want to write my own POP3 library, I’ve done that before and it’s much more complicated than it appears. Buying a commercial component isn’t going to happen either. That left me with one option: To scour the Internet for an open source library.

Most of my career has been spent in companies that are paranoid about using third-party code. Finding something was easy enough (though I must say there are surprisingly few .Net POP3 libraries around), and testing a couple libraries didn’t take long at all. The most difficult part was figuring out how to use an open source library properly.

My favourite of the contenders is released under the GNU LGPL license. This shouldn’t impose any awkward restrictions. I have additional freedom because I’m planning to release my project as open source, but I can see how this would be more difficult for a company building a commercial product.

Since I was reading about licenses, it seemed as good a time as any to pick one for my project. After a lot of reading, I decided to go with the GNU GPL license. I want my project to be freely available, and I don’t want anyone to use it to make a profit without me being involved. If I were unsure this would be the safest choice anyway, because I always have the ability to switch to a less restrictive license later.

Now if I can just find an IoC container, a unit testing framework, and a mocking framework that are also compatible with GPL, Ill be ready to get started.

Hardware

It took me quite a while to choose a new device. I did look at the iPod Nano, but they couldn’t store quite as much as I wanted.The iPod classic on the other hand is way more storage thanI need, and I prefer the lighter weight, longer battery life, and the improved durability of flash storage.

The Microsoft Zune is another device I considered, but the windows software for that is even worse than iTunes. For example,songs can have one of three ratings: heart, broken heart, or no rating. I like that they’re trying to simplify the interface, but I really feel like I’m not their target audience.That said, if Zune Pass worked in Canada I might have gone with them anyway.

The Creative Zen X-Fi gave me everything I wanted. It’s small and light, has 32GB of flash storage and can be easily expanded, and it’s priced competitively. It has an FM radio, and buttons on the outside. I know touch screen is in these days, but I can change songs without even taking it out of my pocket.

Software

Replacing iTunes was another big job. I’ve built up a detailed system of smart play lists and ratings over the years and change scares me. I’m willing to try something new, but I need an app with enough features to do the job.

My search was quickly reduced to two main contenders, Winamp and Media Monkey. Both had automatic play lists, song ratings, and file type conversion features.

I ended up downloading both and doing several tests before choosing Media Monkey. Both applications were comparable for features, but Media Monkey was a little better with syncing, and a little more straightforward for organising songs and play lists.

Converting to Media Monkey was easy, and all the data (like ratings, play counts, album art) came over fine. I did have some trouble with the comments getting scrambled in several tracks, but it seems like iTunes was causing that well before copying the data over. All told, I had all the data over and fixed up after a half day of elbow grease.

Verdict

My only real problem so far has been with the Zen. I’ve had to reboot the device a couple times when it’s hiccupped during a huge sync operation or failed to come back out of sleep.The UI can be a bit difficult sometimes as well. I won’t get into details, but let me just say that converting a WEP key to stars while typing with 9 buttons isn’t making my life any easier.

All in all, I’m pretty happy with the change so far. Just getting the weight of my old 20 GB monster out of my pocket has been nice.

These seem to be the things that can lead you into my scenario:

• Windows 7
• Visual Web Developer 2010 Express (which installs SQL Server 2008 Express)

These are the things I ran into trying to install SQL Management Studio (both from an SQL 2008 standard disk, and a download of the express version from the Microsoft web site):

• A warning that the installer was not compatible with Windows 7. The warning leads you to the SQL 2008 SP1 installer, but that can’t install management studio.
• The installer fails to load with this error: “Invoke or BeginInvoke cannot be called on a control until the window handle has been created."(If you keep retrying you can eventually get past this error, but it seems to be possible at three different stage of the install process.)
• A reboot is required before proceeding, even when running the installer immediately after a reboot.
• When given a list of modules to add, SQL Management Studio isn’t an option on the list.

What a frustrating evening!

The thing that finally worked turned out to be pretty simple though:

1. Launch the Microsoft Web Platform Installer (which you need to install to have Visual Web Developer 2010 Express, as far as I know).
2. Go to the Web Platform tab, and click “Customize” on the Database option.
3. Check “SQL Server 2008 R2 Management Studio Express”, and click Install.
4. Stand back and let the web platform installer do it’s thing.

I wish I’d tried that the first time.

This approach gives me several advantages. It’s not an issue in this case since I don’t have any deadlines to meet, but if this were a professional project and a deadline had to be pushed up, I am more likely to have a product that does the most essential things. If I had a fixed deadline and the project starts running long, I’m more likely to have a working product that’s just missing a few less important features. In an ideal world those don’t happen, but at the end of the project the parts that were risky and/or essential have had the most time to be tested and fine-tuned as the application matures.

I could just charge in and write some code, and sometimes the results can turn out okay, but in my experience it never saves time. If anything, it just increases the chance that the project will fail. On the other hand, I don’t want to do too much design up front. I will know more and more about the problem space the further I get into the project, so the further back I can push parts of the design, the more likely I am to get them right, or at least be close,the first time.

After a couple days of thinking and fooling around with the list, here is what I end up with in the general order I plan to approach them:

1. Receive event invitations by email and send an acceptance email.
2. Record accepted invitations, and send a rejection email when a new invitation conflicts with an existing one.
3. If an event is cancelled, remove it from the record.
4. If an event is rescheduled, try to accept the new time and remove the previous time.
5. Receive invitations for multiple resources.
6. Users can view the schedule for resources.
7. Administrators can view, add, edit, and remove resources or change other options for the service.
8. Save free/busy information for all resources in a format compatible with MS Outlook.
9. The schedule for a resource can be retrieved in the iCal format.

I’ve been wanting to play around with Entity Framework a bit, so this should be a good candidate. Perhaps I can slap a UI on it using Silverlight and Ria Services.

To be honest, creating my own blog wasn’t foremost in my thoughts until fairly recently. Scott Hanselman had a couple of interesting videos on his blog that finally convinced me to do it.

I hope you like it.