HPC Lead System Administrator

Position Summary

Serve as the Lead for the team ensuring smooth operation of the Linux cluster consisting of 300+ GPU/CPU compute nodes including parallel filesystems and high-performance network. This is partly technical and partly people leading role which involves supervision of 3-4 experienced HPC system administrators. The role involves development, implementation and supervision of standard operating procedures for the system and the team.

Major Responsibilities

• System operation and upgrade planning to meet laboratory and customer requirements
• Workload scheduler policy development and implementation
• Support of high-performance filesystems
• Network infrastructure management including TCP/IP and HPC networks
• Use of scripting languages for nodes automation and configuration management
• Hardware failures and spare part management
• Build effective relationships with staff, faculty and students through the Core Labs
• Manages multiple or significant projects which may require the use of sophisticated project planning techniques
• Plans, schedules, conducts, or coordinates detailed phases of the work of a major project or in a total project of moderate scope
• Identifies technical training needs for staff attached to the area
• Serve as a resource and as a member to respond to security and safety incidents
• Creates opportunities to enhance technical methodology or content through expansion of existing, or development of, new efforts; may extend technology into new application areas; contributes or leads in major intellectual development activities
• Provides innovative problem-solving approaches to enhance organizational capabilities; uses peer network to expand technical capabilities and identify new research opportunities
• Understands broad strategic objectives and contributes to them; nurtures and maintains relationships with major customers
• May initiate new project concepts; develops technical proposals and makes presentations to potential customers
• Will supervise several scientists, engineers or technicians on assigned work; provides major input to staffing of overall project teams; builds teams and staff to optimize efficiency and cost effectiveness
• Identifies and evaluates candidates for open positions; mentors/trains staff in development of technical, project and business development skills

Competencies

• SLURM workload manager including GPU scheduling
• Parallel filesystems (Weka IO, Lustre)
• TCP/IP and high performance networks (Infiniband)
• Proficient in scripting languages (i.e. Bash, Python, Ruby)
• Familiar with configuration management tools (Puppet)
• Proficient documentation skills
• Will have working level contact with users and suppliers
• Demonstrates an analytical and systematic approach to problem solving
• Takes the initiative in identifying and negotiating appropriate development opportunities
• Demonstrates effective communication skills in written and oral English
• Works effectively with other teams in the Supercomputing Laboratory
• Plans, schedules and monitors own work (and that of others) competently within limited deadlines and according to relevant legislation and procedures
• Ability to work successfully in a highly collaborative research environment
• Uses discretion in identifying and resolving complex problems and assignments
• Performs a broad range of work, sometimes complex and non-routine, in a variety of environments
• Maintain expert-level knowledge in most of the laboratory systems, including high performance computing systems administration, high performance storage administration, or high performance network administration

Qualifications and Experience

• Bachelor of Science (or equivalent) in a relevant discipline plus 10 years’ experience, OR Master of Science (or equivalent) in a relevant discipline plus 7 years’ experience OR Doctor of Philosophy (or equivalent) in a relevant discipline plus 5 years’ experience.