Containerization (even for small things) makes modern infrastructure a LOT easier.
Containerization (even for small things) makes modern infrastructure a LOT easier.
Real talk, you don’t have the luxury of being an idealist right out of university. Your goal is to get a job. When you’re in that job you will likely not have the luxury of being an idealist either.
When you have enough experience making practical, reasoned decisions, then you can stand on principals.
For context, I have been in this business for nearly 20 years. The people I have personally worked with who have resisted things on philosophical grounds ALWAYS get left behind. I’ve seen it with systemd, the cloud, and now I’m seeing it again with kubernetes. You cannot escape the collective inertia of an entire industry.
Obviously there are still thresholds… I would never work for someone like Raytheon. You have to draw lines somewhere but saying you aren’t going to work for a company that does user behavior tracking is short sighted and impractical.
We had a service that compiles a dataset once per quarter. The total size is ~30gb. We were starting a container, storing it on an EFS volume, and mounting like any other disk.
Every time a pod started it would need to read this data into memory so we would get quick initial start-up time but the time to be ready for traffic still took a while.
Since we didn’t need to update it very often, we decided to just package the compiled dataset into the container and skip the EFS volume. We updated the image pull policy to ifNotPresent
so it cut egress traffic pricing from EFS to zero. Now there is a cost to pull the image from ECR but that’s only if the pod is being scheduled onto a node it hasn’t been run on before. There was no noticable change in behavior or performance and we saved a bunch on cost.
Sometimes the big, dumb option is the right choice.
Thing is, I had a reachable goal which made it easier for me to learn and feel good as I had a tangible result.
IMO, this exact thing is what separates the people who succeed and those who give up. If you are only approaching the code as some abstract concept then it will never work. Anyone learning this stuff needs to understand that the code is more like a hammer to a carpenter than anything else… It’s a very physical tool used for doing a real job. If you don’t have any nails to hit, you’re not going to get anything done.
We focus a lot more on production than the average developer. It’s our job to make sure whatever devs build is run quickly, efficiently, safely, and scalably.
You will work with a lot of kubernetes, Argo, terraform, Prometheus, grafana. You’ll design build pipelines and software rollout strategies. You plan for zero downtime migrations and upgrades, database maintenance… You’ll have your hands in everything from capacity planning to security to cost optimization to developer support… User permissions, infrastructure, networking, observability… You will write RFCs and setup POCs for new tools. You define and track error budgets and figure out how to keep your org under those projections. When there is an outage you will be involved in writing post mortems.
The days are so varied and unpredictable that it keeps things interesting. The landscape changes so often you’re never really stuck doing the same thing over and over.
I genuinely love it.
EDIT: The SRE Podcast from Google is actually really great for learning about this world. The first season talks about what you’ll be doing and why (based around the SRE O’Riley book). The second season talks about what to expect in different stages of your career progression.
Don’t program (as much). Point yourself towards DevOps, SRE, and/or Platform Engineering. You’ll be designing complex systems and will have your hands in dozens of different tech stacks.
Sometimes I think a straight dev job would be interesting but I legitimately love the SRE space.
An RFC that essentially boiled down to saying, in excruciating detail, that I am qualified for the job I was hired for and that I can be trusted not to break the website.
I remember reading about COBOL devs being able to earn pretty solid incomes. I thought, “Let’s check this out.” I found a site that did common things in different languages and compared them. Reverse string in ruby. 1 line. Reverse string in COBOL. 40-50 lines. “Ehh… Maybe I don’t want to learn this after all.”
I have really enjoyed the small projects I have written in rust but, being in the SRE space, it would be irresponsible and selfish to use anything other than bash, python, or go. It feels like the overwhelming majority of tools I use these days have been written in go.
I had this just the other day actually. I am in SRE and the overwhelming majority of the code I write is terraform, instructions for a Dockerfile or CI pipeline, or just some random ass bash to compile information… I don’t actually do that much coding at all and what I do end up doing is pretty rudimentary.
EVERY job interview I go on though wants to do some leetcode style code puzzle. The one I got the other day I just said to the guy, “I honestly don’t know how to do this. The code I write isn’t fancy or clever. It’s mostly just to get things done.” We worked through it together though and I understood the logic by the end but they were mostly holding my hand. What I was doing was throwing out ideas and trying to work out pros/cons with the interviewer. That was enough apparently because he green-lit me for the next round…
These type of interview questions really annoy me because they are not representative of the job in any way. In addition to work, I also have a life that does not involve computers. After putting in a 40 hour week on engineering stuff, grinding leetcode over a weekend is a hard sell.
Is there anything he can’t do? What an inspiration!
Would love to see some base salaries posted along with the responses. If you’re getting paid shit base maybe this is how they make up for it?
I’m in SRE. No on call benefits at all. Base salary is 175k USD plus 20% annual bonus.
I would love an Onion for software. This was great.
Certs are a waste of time tbh. If you have 8 years of experience, you should have more than enough to fill out a resume already.
An AWS cert is almost certainly even more useless for you specifically unless you wanted to get into devops/sre and do systems design. I have been in sre for a very long time and have never even heard of anyone writing tooling in Java. That section of the industry is entirely dominated by go, python, and (more often than anything else) bash for really quick automation.