Table of Contents
Linux and NoMachine Rebuild
Problem
The current Linux/NoMachine setup has two main issues. An increase in VSCode sessions has increased the total resource consumption on the linux nodes leading to partial/whole system outages.
Any solution other than rebuilding the current system as is will involved parallelizing the setup to reduce the impact any one failed node will have on other students using the cluster.
Most potential solutions suggest breaking apart the NoMachine and SSH services as it is easier (and cheaper) to parallelize the SSH systems where as NoMachine is licensed per server and CPU core.
Solutions
Current setup plus user limits
Per user memory and cpu limits can be enforced using cgroups/systemd.
Benefits
- Easier to setup
- Fewer nodes to manage
- Should keep one user from taking down entire cluster
Drawbacks
- Does not limit impact by any one node becoming unavailable
- Limits ability of extra resources bursty workloads
- NoMachine head node remains particularly vulnerable to outages (would take down all NX sessions)
Notes
- A more thorough use of Ansible would be recommended to effectively manage updates
New SSH VMs
Build new KVM based VMs in Proxmox
Benefits
- Setup is closer to current system and would involve fewer unknowns
- SSH/VSCode connections would no longer impact NoMachine
Drawbacks
- Limited ability to parallelize ssh before management of nodes becomes more difficult
Kubernetes Based Shared Containers
Build a SoCS Linux Docker Container and deploy to Kubernetes. These containers would be shared, similar to the current environment, but Kubernetes offers the opportunity to run many more nodes in parallel than a VM based setup could be effectively managed.
Benefits
- Potential to auto-scale cluster to more responsively meet the load
- SSH/VSCode connections would no longer impact NoMachine
- Container based setup could also be distributed to students
Drawbacks
- Potentially more complex setup with more unknowns
Notes
- Will need to determine best cluster ingress configuration. Metallb? Traefik? HAProxy? Something else?
Container SSH
Use ContainerSSH to allow one kubernetes container per student. Students would SSH to the cluster as they currently do, however they would instead be routed to their own dynamically provisioned container.
Benefits
- Completely removes impact of one student an another user's environment
- Container based setup could also be distributed to students
Drawbacks
- Under relatively inactive development - new and potentially unstable
- Complex setup for authentication server