Document toolboxDocument toolbox

(v2.2.0.0) Troubleshooting Guides

Owned by Yajing Wang
May 18, 2024
1 min read

This troubleshooting guide is designed to help you identify and resolve common issues in your Infrastructure Optimizer environment. Using this guide, you can quickly pinpoint problems and implement solutions to ensure smooth operations.

Logs

For diagnosing issues related to the Management Server, you will primarily rely on the following log locations:

  • /var/log/messages

  • /var/log/munge/munged.log

  • /var/log/slurm/aws.log

  • /var/log/slurm/slurmctld.log

For any debugging relating to the Controller and the Worker, the logs can be found at:

  • /xcompute/slurm/aws/xcompute-daemon/data/

  • /xcompute/logs/

    • Within this directory you will find the messages directory and an xspot directory. This xspot directory will contain X-Spot specific logs, in addition to logs for the jobs and workers spawned by that controller.

Pack all logs:

tar -czvf exostellar-logs-$(date +%Y-%m-%d).tar.gz /var/log/messages /var/log/munge/munged.log /var/log/slurm/aws.log /var/log/slurm/slurmctld.log /xcompute/logs /xcompute/slurm/aws/xcompute-daemon/data

 

Â