Devops

I recently needed to copy a few small text files into a container running in AWS Fargate. I assumed it would be an easy matter of copying it through the terminal session, but that is, unfortunately, not the case.

I was accessing the containers via the aws ecs execute-command --interactive --comand "bash" mechanism. If you’re not familiar, this is a quick way to get a terminal inside an ECS container running in any cluster. It’s especially helpful when you have containers that do not receive incoming traffic from the internet. It is almost like SSH, but it is not exactly SSH. And one of the limitations is that there is no equivalent to scp. There is also no ability to forward ports, so you can’t use it to access SSH either.

I have a Windows workstation sitting under my desk at home. This is where I do all of my professional development. I like this setup because it keeps my work completely separate from my personal stuff. If I’m honest, though, my main reason for buying it was because I poured a glass of water over my development laptop. But now that I have it, I highly recommend it.

When I’m working at home, I connect to this workstation using Remote Desktop (RDP). It’s a fantastic protocol for accessing a Windows machine across a network. It’s built in to the operating system, and it’s generally pretty smooth. Most times I can’t even tell that I’m using it.

Although it was years ago, I still remember the conversation vividly. I was having my regular check-in with the CTO. It was also sunny outside, and the view from his office was particularly nice that day.

He asked me, “Is our app scalable?” After a moment of thought, I answered, “Yes.”

I spent the next few years remembering that conversation, wondering what would have happened if I had asked what he meant by scalable.

I used to use Getform for a contact form on my consulting company website. I recently received an email from them announcing that their free tier was dropping from 50 submissions per month to a lifetime limit of 25. This makes it useless for anything more than a trial, and their lowest tier is a more expensive than other similar options.

I’m not here to complain about companies taking back free offerings. I don’t like the change, and I wish they’d given me more than three days of notice, but they are a businesses, and businesses need to make money. It is a good reminder though: even if something is free to use, it still takes time and effort to integrate, to maintain, and you may occasionally need to throw it out and find a replacement.

I recently worked on a project to move a complicated legacy application onto Kubernetes. It was quite an undertaking, but in the end we were successful. One of the biggest challenges was figuring out how to automate our legacy deployment process, one where the whole application has to be stopped completely for schema upgrades to run.

The normal “Kubernetes way” to upgrade an application is by changing the Deployment resource. With its default RollingUpdate strategy it will delete a pod with the old definition, start a pod with the new definition, wait for it to be healthy, then repeat continuously until the change is fully applied.

I’ve spent the last several years working on multi-user server systems in two different companies. Both those companies had a giant monitor hanging off a wall showing a graph of connected users. It won’t give you detailed diagnostic information, but it is a good indicator for the health of your servers, and your product generally. If you learn to notice certain patterns in your user graph, it can also save you precious time when things go wrong.

I had a frustrating issue at work this week: one that was easy to fix, but embarrassingly difficult to find. I came pretty close to giving up, which is not a solution I often explore, but in the end we figured it out and got everything working.

A member of our operations team was installing a Windows service I’d built to monitor some stuff in our production environment. I’ve made a few windows services in my day, and installed them many times on many machines. I’d even installed this one on my development machine with no issue. In our staging environment, however, this is what we got:

We did one of our monthly releases at work this week. Releases can be stressful and frustrating, and take a lot of methodical preparation to get right. It can be thankless work too; the only time a user notices a release is when it goes badly. We do our releases early on a week day to minimize impact, so if anything does go wrong, there’s not many bodies around to help out. It’s not much fun, but it’s important work that needs to be done.

My company runs a 24/7 site with a substantial number of users and connections to partner systems all over the world. We do what we can to make the system fault tolerant, but problems can still appear at any time of day or night. Ideally we would have a technical support team that’s staffed around the clock, but that not in the cards for now.

We used to have a system where anyone that could fix a problem would get a text message, and whoever was closest to a computer would respond. It worked most of the time, but there were some issues. The team creating the tickets would often be unsure of which component was broken, so the issues would hit lots of people who would be unable to help, and it also exposes a quirk of human behaviour that makes us less likely to help when more people are available.

I found my first 64-bit bug at work. I was moving a windows service built for the ‘Any CPU’ to a 64-bit server. It started fine on the new server, and gave no indication of poor health in the logs, but one key function was malfunctioning. I’m not exactly sure what the cause is, but I know that the hash of any binary file was resulting in the same value. The service does some direct memory manipulation which is a likely culprit.