PROFESSIONAL ACHIEVEMENTS

I don’t speak for any past or current employer. These are all my opinions.

STCC

STUDENT

• RECIPIENT OF OUTSTANDING ACADEMIC ACHIEVEMENT AWARD
• STUDENT SUPPLEMENTAL INSTRUCTOR (PYTHON) FOR 2 YEARS
• MODERATOR OF STCC DISCORD SERVER (300+ STUDENTS)

ADJUNCT PROFESSOR

• OVER 100 STUDENTS TAUGHT (LINUX)
• OVERHAULED LINUX CURRICULUM TO PREPARE STUDENTS FOR WORKFORCE

I’ll write more on this when I have time

RED HAT

I’ll write more on this when I have time

NORTHEASTERN UNIVERSITY RESEARCH COMPUTING

SYSTEMS ENGINEER

ARCHITECTED EXPLORER CLUSTER

When I first started in 2023, I began my work on a cluster called Discovery. It was an older CentOS 7 cluster with the EOL date approaching quickly. The senior sysadmin and mentor of mine would often have me jump on a call and open a tmate session where he would describe to me the environment, where it is hurting, and how a better solution would solve it. The new cluster would be called Explorer, it would be newer, and free from the sins of the past done by admins predating myself and the senior sysadmin.

Before Explorer was built the senior sysadmin had moved across the atlantic to be with family.

After he left, I was sat down by the AVP and was told that I would be the one to bring Explorer into reality. With increased pressure by the team and the university at large to get away from a dying OS, there wasn’t much time to think about the cluster and it was time to build.

DEPLOYED END TO END, TOP TO BOTTOM (PROCUREMENT, NETWORKING, SCHEDULER)

Prior to my sit-down, I had begun building some of the infrastructure for Explorer in an empty cabinet in the datacenter.

Initially we were going to change the networking model from ACCESS-AGG-CORE to SPINE-LEAF, however, at that point I didn’t have time or the understanding to rearchitect the network. So I plugged into the existing access switch and started from there.

• NODE 1 - Slurm DB
• NODE 2 - Slurm Controller
• NODE 3 - DNS
• NODE 4 - Login Node 1
• NODE 5 - Login Node 2

With the bare bones there and operational, I had figured I’d use the existing infrastructure built before me to give me a headstart.

• Provisioner Node
• Proxy
• Gateway
• Bastion Host

Building a new cluster is scary enough as it is (months into my first full-time role, fresh off the RedHat Apprenticeship).

Not to mention 2 days in as the only admin, attempting to update our 8+ PB storage cluster with the assurance of the vendor that nothing would go wrong. It did. Long story.

I was risk-averse.

The safe bet was a 3 year project moving every single node from Discovery to Explorer. The new cluster would slowly devour the old one. Like an ouroboros, Ship of Theseus, whatever.

Though this seemed like the best idea at the time. It also meant that I would be the sole admin of two clusters until Explorer was finished.

• 2 Database Nodes
• 2 Controllers
• 2 Configs
• 2 DNS servers
• 2 OOD Instances
• 4 Login Nodes

That’s two clusters I would standup every year when we powered them down for an annual outage.

Keeping track of SSL certs for both OOD instances.

Maintaining the configs of both clusters.

This gave me an incredible learning experience, seeing what worked where, and how the new cluster could improve on the old one.

None of it was easy, Explorer wasn’t very stable in the beginning. Things would go down and need standing up, surprise failures, misbehaving users on the login nodes. Each flaw was slowly ironed out.

Moving compute required an entire rewrite of the PXE and Ansible stack, this wasn’t too difficult because the feedback loop was quick.

I would PXE boot a machine, watch the anaconda install, see where it failed, change, ipmitool chassis bootdev pxe ipmitool chassis power cycle, repeat.

The goal was to have the kickstart get everything ready (partition disks, rename network interfaces, download ansible-pull) before we would do the heavy lifting with Ansible (Ansible being easier to maintain than a kickstart file). From there Ansible would change everything about the node that wasn’t an Explorer compute node. The definition of Explorer Compute Node would change over time, so I would have to backtrack now and again to unify the state.

Lots of custom facts to tailor the work and automate what I could out of necessity.

Over time, as I got help from my friends where I had my internship (Techsquare) we were able to start transitioning nodes at a faster pace. Following the 51% rule, eventually Explorer became the primary cluster of Northeastern Research.

A couple of wrenches were thrown into the machine (policy, cooling capacity, cabinet real estate). But the majority of the work has been done.

DISTRIBUTING THE OWNERSHIP

Now that the work has transitioned, I’m doing my best to delegate as much of the daily care and feeding work as I can. Especially the unseen, not pretty stuff the team doesn’t really see.

Writing a few scripts, training, documentation.

The goal is to make it all SAD.

• Standardized
• Automated
• Documented

I had to do some guessing while I’ve been administrating. Not all the admins from 14 years ago were writers.

I intend on leaving everyone with enough information to have an idea of how to keep the system alive.

UNSEEN TRIUMPH

It’s kind of bittersweet because my greatest and proudest professional achievement was created in the absence of witnesses.

Facing the difficulty of learning how SSL certs work because it just failed and 3,000+ users are waiting on you to ‘get it’. Or being the person everyone turns to for answers. It’s a hard feeling to categorize.

There are always optimizations to make. I think about decisions I’ve made 2 years ago and whether or not they were the correct ones. My newer, more informed self questions the kid building a massive computer.

However, I’m happy that grants are awarded and students are using it. I don’t get to see the user side much but I hope it’s a happy place. I hope that some college student discovers they really want to make Linux a career. If that happens, then it’s a win.

CLUSTER - AS OF 2025

MGHPCC

NEUHPC

REAR DOOR HEAT EXCHANGER BUILDOUT FOR NURC

This was a 1-year undertaking for Northeastern University.

A lot of notifying of researchers, downtime and reorganizing of servers across the datacenter.

The Techsquare team and I:

• Moved over 200 servers to new locations.
• Decommissioned over 120 servers (CMC Chassis).
• Communicated closely with facilities, working off their go-ahead to rack and stack.
• Draining racks at a time for testing or transitions.

TRANSITIONING DATA STORAGE HARDWARE FROM BOSTON DATACENTER

About one month of phone calls, dates, shipping information.

Shuffling information across 5 different parties to get NU’s system out of the Boston datacenter and finally into MGHPCC.

• Worked closely with Storage provider (1) for them to properly powerdown and package the system.
• Logistics (2) to move the system.
• Coordinating info with MGHPCC (3)
• Navigating access for everyone with Boston Datacenter (4)
• Keeping NU in the loop (5)

Eventually we got the thing in and were able to integrate it into the system.