Document toolboxDocument toolbox

(v2.2.0.2) Integrating with Slurm Cluster

Owned by Yajing Wang
May 28, 2024
8 min read

The following manual steps for EAR will be replaced with a simplified workflow for command line users and alternatively, the Mangement Console (Web UI) will be able to replace most of these steps, as well.

Connect to Your Slurm Head Node

During Early Access, integreation requires a handful of commands and root or sudo access on the slurm controller, where slurmctld runs.

  1. Get a shell on the head node and navigate to the slurm configuration directory, where slurm.conf resides.

    1. [ root@HEAD ~ ]# cd $SLURM_CONF_DIR

  2. Make a subdirectories here:

    1. [ root@HEAD /etc/slurm ]# mkdir -p assets/json [ root@head /etc/slurm ]# cd assets/json [ root@head /etc/slurm/assets/json ]#

Pull Down the Default Slurm Environment Assets as a JSON Payload:

  1. We will need the private MGMT_SERVER_IP, and the packages for jq and curl:

    1. yum install jq curl

  2. Pull down from the MGMT_SERVER default assets for customization:

  3. The asset will look like:

Edit the Slurm Environment JSON for Your Purposes:

  1. Copy default-slurm-env.json to something convenient like env0.json.

  2. Line 2 : "EnvName" is set to slurm by default, but you can specify something unique if needed (implication on get slurm.tgz later) .

  3. Lines 5-17 can be modified for a single pool of identical compute resources or they can be duplicated and then modified for each “hardware” configuration or “pool” you choose. When duplicating, be sure to add a comma after the brace on line 17, except when it is the last brace, or the final pool declaration.

    1. PoolName: This will be the apparent hostnames of the compute resources provided for slurm.

      1. It is recommended that all pools share a common trunk or base in each PoolName.

    2. PoolSize: This is the maximum number of these compute resources.

    3. ProfileName: This is the default profile name, az1: If this is changed, you will need to carry the change forward.

    4. CPUs: This is the targeted CPU-core limit for this "hardware" configuration or pool.

    5. ImageName: This is tied to the AMI that will be used for your compute resources. This name will be used in subsequent steps.

    6. MaxMemory: This is the targeted memory limit for this "hardware" configuration or pool.

    7. MinMemory: reserved for future use; can be ignored currently.

    8. UserData: This string is a base64 encoded version of user_data.

      1. To generate it:

        1. cat user_data.sh | base64 -w 0

      2. To decode it:

        1. echo "<LongBase64EncodedString>" | base64 -d

      3. It’s not required to be perfectly fine-tuned at this stage; it will be refined and corrected later.

      4. You may format user_data.sh in the usual ways:

        1. or

    9. VolumeSize: reserved for future use; can be ignored currently.

  4. Lines 24, 25, 26 should be modified for your slurm environment and according to your preference for the partition name.

    1. BinPath: This is where scontrol, squeue, and other slurm binaries exist.

    2. ConfPath: This is where slurm.conf resides.

    3. PartitionName: This is for naming the new partition.

  5. All other fields/lines in this asset can be ignored.

Validate and Push the Customized Environment to the MGMT_SERVER

  1. Validate the JSON asset with jq:

    2. You will see well-formatted JSON if jq can read the file, indicating no errors. If you see an error message, that means the JSON is not valid.

  2. When the JSON is valid, the file can be pushed to the MGMT_SERVER:

Pull Down the Default Profile Assets as a JSON Payload:

  1. The default is named az1.

  2. Copy it to faciliatate customization, leaving the default for future reference.

  3. The asset will look like this:

Edit the Profile JSON for Your Purposes:

  1. Tagging instances created by the backend is controlled by two sections, depending on the function of the asset:

    1. Controllers are On-Demand instances that manage other instances. By default, they are tagged as seen on lines 6-9, above, and 1-4 below.

      2. To add additional tags, duplicate lines 1-4 as 5-8 below (as many times as you need), noting that an additional comma is added on line 4.

      4. Don’t forget the comma between tags.

    2. Workers will be created by Controllers as needed and they can be On-Demand/Reserved instances or Spot. By default, they are tagged as seen on lines 26-30, above, and 1-4 below:

      2. Add as many tags as needed.

      4. Don’t forget the comma between tags.

  2. Line 11 - InstanceType: Controllers do not generally require large instances.

    1. In terms of performance, these On-Demand Instances can be set as c5.xlarge or m5.xlarge with no adverse effect.

  3. Line 20 - MaxControllers : This will define an upper bound for your configuration.

    1. Controllers will manage upto 80 workers.

    2. The default upper bound is 800 nodes joining your production cluster: notice line 20 "MaxControllers": 10,.

    3. If you plan to autoscale past 800 nodes joining your production cluster, MaxControllers should be increased.

    4. If you want to lower that upper bound, MaxControllers should be decreased.

  4. Line 21 - ProfileName: This is used for your logical tracking, in the event you configure multiple profiles.

  5. Lines 31-34 - InstanceTypes here in the Worker section, this refers to On-Demand instances – if there is no spot availability, what instances do you want to run on.

  6. Lines 38-43 - SpotFleetTypes : here in the Worker section, this refers to Spot instance types – because of the discounts, you may be comfortable with a much broader range of instance types.

    1. More types and families here, means more opportunities for cost optimization.

    2. Priorities can be managed by appending a : and an integer, e.g. m5:0 is a higher priority than c5:1.

  7. Line 48 - Hyperthreading: This is reserved for future use, can be ignored currently.

  8. Line 52 - NodeGroupName : This string appears in Controller Name tagging <profile>-NGN-count

  9. All other field/lines can be ignored in the asset.

Validate and Push the Customized Profile to the MGMT_SERVER

  1. Validate the profile with the quick jq test.

  2. Push the changes live.

Download Scheduler Assets from the Management Server

  1. We will need the private MGMT_SERVER_IP:

Compute AMI Preparation

An AMI is required that is capable of running your workloads. Ideally, this AMI is capable of booting quickly.

  1. The AMI-ID should be based on a slurm compute node from your cluster, capable of running your workloads.

  2. The AMI should be created by this account.

  3. The AMI should not have product codes.

  4. The Image Name was specified in the environment set up previously and will be used in this command.

  5. Additionally, we can pass -s script.sh if troubleshooting is required.

Validation of Migratable VM Joined to Your Slurm Cluster

The script test_createVm.sh exists for a quick validation.

  1. The hostname specified with -h xspot-vm01 is completely arbitrary.

  2. The Image Name specified with -i <IMAGE_NAME> should correspond to the Image Name from the parse_helper.sh command and the environment setup earlier.

  3. The -u user_data.sh is available for any customization that may be required: temporarily changing a password to faciliate logging in, for example.

  4. The script will continuously output updates until the VM is created. When the VM is ready, the script will exit and you’ll see all the fields in the output are now filled with values:

  5. This step is meant to provide a migratable VM so that sanity checking may occur:

    1. Have network mounts appeared as expected?

    2. Is authentication working as intended?

    3. What commands are required to finish bootstrapping?

    4. Et cetera.

  6. Lastly, slurmd should be started at the end of bootstrapping so the submitted job can begin.

  7. To remove this temporary VM:

    1. Replace VM_NAME with the name of the VM , -h xspot-vm01 example above.

  8. The above steps may need to be iterated through several times. When totally satisfied, stash the various commands required for successful bootstrapping and overwrite the user data scripts in the assets directory.

Finalize Integration with Slurm

  1. Move xspot.slurm.conf, xspot_resume.sh, xspot_suspend.sh, and the user data scripts to the SLURM_CONF_DIR.

  2. <handle XSPOT_NODENAME better: intuitive hostname is less confusing that random hash>

    1. edit xspot_resume.sh and add sed for every pool:

  3. Edit your slurm.conf:

    1. Add include statement to pick up xspot.slurm.conf:

  4. Introducing new nodes into a slurm cluster requires restart of the slurm control deamon:

  5. Integration steps are complete and a job submission to the new partition is the last validation:

    1. As a user, navigate to a valid job submission directory and launch a job as normal, but be sure to specifiy the new partition:

      1. sbatch -p NewPartitionName < job-script.sh