v2.3.0.0 Adding or Modifying Pools Slurm

It’s common to refine or otherwise modify configurations over time. Soon there will be a CLI tool to obviate the need for the following manual steps, which are the current steps required for reconfiguration after the initial integration has been pushed into production.

1. Navigate to the exostellar directory where the configuration assets reside:

   1. cd ${SLURM_CONF_DIR}/exostellar

2. Setup a timestamp folder in case there’s a need to rollback:

   1. PREVIOUS_DIR=$( date +%Y-%m-%d_%H-%M-%S )
      mkdir ${PREVIOUS_DIR}

3. Place the contents of the exostellar directory in the timestamp directory:

   1. mv * ${PREVIOUS_DIR}

4. Make a new json folder and copy the env.json and profile.json:

   1. mkdir json
      cp ${PREVIOUS_DIR}/json/env0.json ${PREVIOUS_DIR}/json/profile0.json json/
      cd json
      mv env0.json env1.json
      mv profile0.json profile1.json

5. Edit env1.json as needed, e.g.:

   1. Add more pools if you need more CPU-core or Memory options availble in the partition.

   2. Increase the node count in pools.

   3. Environment Configuration Information for reference.

6. Likely, profile0.json may not need any modification.

   1. Profile Configuration Information for reference.

7. Validate the JSON asset with jq:

   1. jq . env1.json

8. You will see well-formatted JSON if jq can read the file, indicating no errors. If you see an error message, that means the JSON is not valid.

9. When the JSON is valid, the file can be pushed to the MGMT_SERVER:

   1. curl -d "@env1.json" -H 'Content-Type: application/json' -X PUT http://${MGMT_SERVER_IP}:5000/v1/env

10. If the profile was changed, validate it with the quick jq test.

    1. jq . profile1.json

11. Push the changes live:

12. curl -d "@profile1.json" -H 'Content-Type: application/json' -X PUT http://${MGMT_SERVER_IP}:5000/v1/profile

13. Grab the assets from the MGMT_SERVER:

14. curl -X GET http://${MGMT_SERVER_IP}:5000/v1/xcompute/download/slurm -o slurm.tgz

    1. If the EnvName was changed (above in Edit the Slurm Environment JSON for Your Purposes - Step 2 ), then the following command can be used with your CustomEnvironmentName :

    2. curl -X GET http://${MGMT_SERVER_IP}:5000/v1/xcompute/download/slurm?envName=CustomEnvironmentName -o slurm.tgz

15. Unpack them into the exostellar folder:

16. tar xf slurm.tgz -C ../
    cd ..
    mv assets/* .
    rm assets

17. Edit resume_xspot.sh and add sed command snippet ( | sed "s/XSPOT_NODENAME/$host/g" ) for every pool:

    • user_data=$(cat /opt/slurm/etc/xvm16-_user_data | base64 -w 0)

    • becomes:

    • user_data=$(cat /opt/slurm/etc/exostellar/xvm16-_user_data | sed "s/XSPOT_NODENAME/$host/g" | base64 -w 0)

18. Introducing new nodes into a slurm cluster requires restart of the slurm control deamon:

    1. systemctl restart slurmctld

19. Integration steps are complete and a job submission to the new partition is the last validation:

    1. As a user, navigate to a valid job submission directory and launch a job as normal, but be sure to specifiy the new partition:

       1. sbatch -p NewPartitionName < job-script.sh