===== Linux and NoMachine Rebuild =====

==== Problem ====

The current Linux/NoMachine setup has two main issues. An increase in VSCode sessions has increased the total resource consumption on the linux nodes leading to partial/whole system outages. 

Any solution other than rebuilding the current system as is will involved parallelizing the setup to reduce the impact any one failed node will have on other students using the cluster.

Most potential solutions suggest breaking apart the NoMachine and SSH services as it is easier (and cheaper) to parallelize the SSH systems where as NoMachine is licensed per server and CPU core.

==== Solutions ====

=== Current setup plus user limits ===

Per user memory and cpu limits can be enforced using cgroups/systemd.

== Benefits ==
  * Easier to setup
  * Fewer nodes to manage
  * Should keep one user from taking down entire cluster

== Drawbacks ==
  * Does not limit impact by any one node becoming unavailable
  * Limits ability of extra resources bursty workloads
  * NoMachine head node remains particularly vulnerable to outages (would take down all NX sessions)

== Notes ==
  * A more thorough use of Ansible would be recommended to effectively manage updates

=== New SSH VMs ====

Build new KVM based VMs in Proxmox

== Benefits ==
  * Setup is closer to current system and would involve fewer unknowns
  * SSH/VSCode connections would no longer impact NoMachine

== Drawbacks ==
  * Limited ability to parallelize ssh before management of nodes becomes more difficult

=== Kubernetes Based Shared Containers ===

Build a SoCS Linux Docker Container and deploy to Kubernetes. These containers would be shared, similar to the current environment, but Kubernetes offers the opportunity to run many more nodes in parallel than a VM based setup could be effectively managed. 

== Benefits ==
  * Potential to auto-scale cluster to more responsively meet the load
  * SSH/VSCode connections would no longer impact NoMachine
  * Container based setup could also be distributed to students

== Drawbacks ==
  * Potentially more complex setup with more unknowns

== Notes ==
  * Will need to determine best cluster ingress configuration. Metallb? Traefik? HAProxy? Something else?

=== Container SSH ====

Use [[https://containerssh.io/|ContainerSSH]] to allow one kubernetes container per student. Students would SSH to the cluster as they currently do, however they would instead be routed to their own dynamically provisioned container.

== Benefits ==
  * Completely removes impact of one student an another user's environment
  * Container based setup could also be distributed to students

== Drawbacks ==
  * Under relatively inactive development - new and potentially unstable
  * Complex setup for authentication server