„PRACE User Support” változatai közötti eltérés

Innen: KIFÜ Wiki
(The most simple commands)
(The most simple commands)
299. sor: 299. sor:
  
 
<code>
 
<code>
     test.q       
+
     parael.q
     uv.q  
+
    serial.q       
 +
     test.q  
 
</code>
 
</code>
  
316. sor: 317. sor:
 
     queuename                      qtype resv/used/tot. load_avg arch      states       
 
     queuename                      qtype resv/used/tot. load_avg arch      states       
 
     <nowiki>-------------------------------------------------------------------------------- </nowiki>
 
     <nowiki>-------------------------------------------------------------------------------- </nowiki>
     test.q@uv                     BIP  0/1/30           800.15          linux-x64               
+
     test.q@cn.32                     BIP  0/3/24           3.15          linux-x64               
 
     905 1.00000 PI_SEQ_TES stefan      r    06/04/2011 09:12:14    1               
 
     905 1.00000 PI_SEQ_TES stefan      r    06/04/2011 09:12:14    1               
    <nowiki>-------------------------------------------------------------------------------- </nowiki> 
 
    uv.q@uv                        BIP  0/802/1110      800.15          linux-x64 
 
  
 
</code>
 
</code>
330. sor: 329. sor:
 
     queuename                      qtype resv/used/tot. load_avg arch      states       
 
     queuename                      qtype resv/used/tot. load_avg arch      states       
 
     <nowiki>--------------------------------------------------------------------------------- </nowiki>   
 
     <nowiki>--------------------------------------------------------------------------------- </nowiki>   
     test.q@uv                      BIP  0/0/30           600.42           linux-x64      
+
 
 +
     parallel.q@cn31                BIP  0/24/24           22.3           linux-x64          
 
     <nowiki>--------------------------------------------------------------------------------- </nowiki>  
 
     <nowiki>--------------------------------------------------------------------------------- </nowiki>  
     uv.q@uv                        BIP  0/598/1110      600.42           linux-x64     
+
 
 +
     test.q@cn32                    BIP  0/24/24          23.5           linux-x64     
 
        
 
        
 
     <nowiki>############################################################################ </nowiki>
 
     <nowiki>############################################################################ </nowiki>

A lap 2013. július 10., 14:56-kori változata

User Guide to obtain a digital certificate

This document gives a short overview about how to require a digital certificate from NIIF CA for users, if the pre-registration form has been filled.

This guide is valid only for the Hungarian users.

If you are from a foreign country, and would like to get a certificate, here you can found your country's certification authority.


Installing NIIF CA root certificate

The first step is to download the "root certificate" ("NIIF CA Root Certificate" part), in the the format, which is known for the used browser or other SSL-using program. The browser asks wether to install/accept the certificate or not - accept or install the certificate in any cases. In addition, activate or allow the option which permits the browser to use the certificate to authenticate websites. Without that, it is not possible to reach the CA's web interface with secure protocol (https). The downloaded/installed certificate can be found in the certificate management modul of the browser.


Request a certificate

Request a certificate with openssl

  • Sing in into the certification registration website of the NIIF CA with our email address and password stored in the directory.
  • This site uses secure protocol (https), which the browser often indicates with a warning window - they should be acknowledged implicitly.
  • In the opening page - which is the public web surface of the CMS certificate management software - choose the "OpenSSL kliens kérelem benyújtása (PKCS#10)" (request an OpenSSL client) option. This leads to the datasheet, which must be filled in accordance with the printed datasheet. First, according to the purpose of the request, the corresponding field must be choosen (CSIRT, GRID, NIIF felhasználó, Független kutató, HBONE).
  • Copy the public part of our certificate in the field "PKCS#10". You can find a user guide about How to create a PKCS#10 certificate with openssl, which suitable for the NIIF CA requirements below.
  • A Challenge and a Request passwords must be given - both of them must be at least 8 characters long. Note them, because they needed for cancellation the certificate, or for the personal authentication.
  • Fill the other fields (name, email address, phone, organisation), and if there is anything, the CA operator should know, fill the last field with it. If everything is done, after a last check, click on the Elküld ("send") button on the bottom of the page.
  • In case of a successful PKCS#10 key-uploading, a page is opening with the confirmation of the successful certification request.

User Guide to create a PKCS#10 digital certification request with openssl

This paragraph gives a short overview about how to require a digital certificate from NIIF CA for users using openssl with the PKCS#10 format.The latest version of the openssl program can be downloaded from: Windows, Linux.

1. Download the openssl configuration file
To generate the CSR, there is a prewritten niif_ca_user_openssl.cnf file on the NIIF CA website.
The following modifications must be done in the config:


#purpose of the certificate

1.organizationalUnitName = Organizational Unit Name
1.organizationalUnitName_default = GRID # For example: GRID, HBONE, General Purpose
2.organizationalUnitName = Second Organizational Unit Name
2.organizationalUnitName_default = NIIF # For example: BME, ELTE, SZFKI, SZTAKI, NIIF, ...
commonName = Common Name (YOUR name) # User Name.
commonName_max = 64A


2. Create PKCS#10 reqquest
  • No existing secret key:

Run the

   openssl req -newkey rsa:1024 -config ./niif_ca_user_openssl.cnf -out new_csr.pem 

command, and answer the appearing questions at the prompt. The Institute (NIIF CA) and country (HU) datas should not be changed, or the request is going to be invalid. The certification request and the corresponding private key will be saved in the new_csr.pem and privkey.pem files. To gain acces to the private key, during the generating given "pass phrase" password must be used. In case of a forgotten password, the certificate will be unusable.

  • Existing private key (extend)

If there is an existing, previously generated private key (it must be at least a 1024 bit RSA key), which can be found in the old_key.pem file, then the following command creates the CSR

   openssl req -new -key ./old_key.pem -config ./niif_ca_user_openssl.cnf -out new_csr.pem


Personal Authentication

After the successful registration on the website, please visit the NIIF CA Registration Office personally with the copy of the pre-registration datasheet, the Request password and an ID document (ID card, passport).

Address:

NIIF Iroda
(RA Administrator)
Victor Hugo Str. 18-22.
H-1132 Budapest, HUNGARY
email: ca (at) niif (dot) hu
RA opening hours: Monday, 14:00 - 16:30 (CET)

During the authentication, the colleagues of the Registration Office verify the datas of the certificate and the user, and after the successful identification, they take the next steps in order to create the certification (it is not needed to wait for it).


Downloading the certificate

An email is going to arrive after the valid certificate has been completed (to the given email address during the request), and clicking on the URL in the email, the certificate can be downloaded. The saved certificate does not contain the private key.

If the certificate is installed into the browser, it is advised to export it with the private key in PKCS#12 format, so there will be a common backup with the private key and the certificate. Handle this backup carefully! If the private key lost, or gets into unauthorized hands, immediately request a certificate cancellation at the registration interface "Tanúsítvány visszavonása" (certificate cancellation) or at the Registration Office, and inform the concerned people!

Access with GSI-SSH

A user can access to the supercomputers by using the GSI-SSH protocol.

It requires a machine with a Globus installation that provides the gsissh client.

The needed credentials (these mean the private and public keys) must be created before entering the machine with the

   grid-proxy-init

or

   arcproxy

commands.

By default, the proxies are valid for 12 hours. It is possible to modify this default value with the following commands:

   arcproxy -c validityPeriod=86400

or

   grid-proxy-init -hours 24

Both of the previous commands set the validation of the proxies to 24 hours. Using the arcproxy, the validation time must be given in seconds.


To enter the site, the

   gsissh -p 2222 login.budapest.hpc.niif.hu

command has to be used.


GridFTP file transfer

In order to use GridFTP for file transfer, one needs a GridFTP client program that provides the interface between the user and a remote GridFTP server. There are several clients available for GridFTP, one of which is globus-url-copy, a command line tool which can transfer files using the GridFTP protocol as well as other protocols such as http and ftp. globus-url-copy is distributed with the Globus Toolkit and usually available on machines that have the Globus Toolkit installed.


Syntax

   globus-url-copy [options] sourceURL destinationURL

  • [options] The optional command line switches as described later.
  • sourceURL The URL of the file(s) to be copied. If it is a directory, it must end with a slash (/), and all files within that directory will be copied.
  • destURL The URL to which to copy the file(s). To copy several files to one destination URL, destURL must be a directory and be terminated with a slash (/).


Globus-url-copy supports multiple protocols, so the format of the source and destination URLs can be either

   file://path 


when you refer to a local file or directory or

   protocol://host[:port]/path


when you refer to a remote file or directory.

globus-url-copy is supporting other protocols such as http, https, ftp and gsiftp as well.


  • Example:

   globus-url-copy file://task/myfile.c gsiftp://login.budapest.hpc.hu/home/task/myfile.c 

This command uploads the myfile.c file from the locak task folder to the remote machine's home/task folder.


Command line options for globus-url-copy [options]

  • -help Prints usage information for the globus-url-copy program.
  • -version Prints the version of the globus-url-copy program.
  • -vb During the transfer, displays: (1) number of bytes transferred (2) performance since the last update (every 5 seconds) (3) average performance for the whole transfer


The following table lists parameters which you can set to optimize the performance of your data transfer:

  • -tcp-bs <size>Specifies the size (in bytes) of the TCP buffer to be used by the underlying GridFTP data channels.
  • -p <number of parallel streams> Specifies the number of parallel streams to be used in the GridFTP transfer.
  • -stripe Use this parameter to initiate a “striped” GridFTP transfer that uses more than one node at the source and destination. As multiple nodes contribute to the transfer, each using its own network interface, a larger amount of the network bandwidth can be consumed than with a single system. Thus, at least for “big” (> 100 MB) files, striping can considerably improve performance.

Usage of the Sun Grid Engine scheduler

Basically the SGE is a scheduler, which divides the resources, computers into resource partitions. These are called queues. A queue can’t be larger than a physical resource; it can’t expand its borders. SGE registers a waiting list for the resources managed by itself, to which the posted computing tasks are directed. The scheduler searches for the resource defined by the description of the task and starts it. The task-resource coupling depends on the ability of the resources and the parameters of the tasks. In case the resources are overloaded, the tasks have to wait while the requested processor and memory becomes available.


The detailed documentation of the SGE can be found here.


SGE version on all HPC sites: Open Grid Scheduler (OGS/GE 2011.11p1)


The most simple commands

The most simple SGE command is the display of the cluster data:

   qhost

A possible outcome of this command can be:

HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
global - - - - - - -
cn01 linux-x64 24 5.00 62.9G 8.6G 0.0 0.0
cn02 linux-x64 24 0.01 62.9G 1.2G 0.0 0.0
cn03 linux-x64 24 0.03 62.9G 1.5G 0.0 0.0


The first two columns define the names and types of the computers, which are in the cluster. The NCPU column shows the number of the available processor cores. LOAD shows the computer’s load for the moment (this value equals with the value demonstrated by the uptime UNIX command). The rest of the cells are: overall physical memory, the actual used memory, the available swap-memory, and the used swap. The global line marks all the information in total regarding the cluster.

We can have a look at the available queue-s with the following command:

   qconf -sql

One probable outcome of the command:

   parael.q
   serial.q       
   test.q 


To get more info about the state of the system use

   qstat -f

It shows which jobs run in which queues, and you can also get detailed info about the queues themselves (state, environment). The command can be used without the -f switch too, but it is less informative, since in this case only the jobs’ states will appear. The command’s outcome:

   queuename                      qtype resv/used/tot. load_avg arch       states       
   -------------------------------------------------------------------------------- 
   test.q@cn.32                      BIP   0/3/24           3.15           linux-x64               
   905 1.00000 PI_SEQ_TES stefan       r     06/04/2011 09:12:14     1               

The first column of this table shows the name of the row, the second column marks the type (B-batch, I-interactive, C-checkpointing, P-parallel environment, E-error state). The third part of the column shows how many jobs can be run at the same time in the row. All in all, these values fit to the number of overall processor cores in the system. The second item of the column shows the free compartments at the moment.

If a running (scheduled) job is to be found in the queue, it is directly next to the name of the row, like the recent "PI_SEQ_TES", which runs in the test.q row. The tasks waiting for the resources, because it is overwhelmed or the preliminary conditions are not prompt, appear behind the sum row, listed as pending jobs. For example:

   queuename                      qtype resv/used/tot. load_avg arch       states       
   ---------------------------------------------------------------------------------   
   parallel.q@cn31                BIP   0/24/24           22.3           linux-x64           
   ---------------------------------------------------------------------------------  
   test.q@cn32                    BIP   0/24/24           23.5           linux-x64    
      
   ############################################################################ 
    - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS       
   ############################################################################ 
       905 0.00000 PI_SEQ_TES stefan       qw    06/04/2011 09:12:04     1         



Each task is given an identifier, which is a number (a job ID, or j_id), this is followed by the job’s priority (0 in both cases), then the job’s name, and the user who posted the job, and the qw marks, that the job is waiting for the queue. Finally the date of the registration for the waiting queue is next

When a job finishes running, this is created: jobname.ojobnumber in our actual catalog, which contains the error messages and stapled outputs created by the program..

Job submission

Back then, the SGE scheduler was designed to be able to operate different types of architectures. That’s why you can’t post binary files directly, only scripts, like the

   qsub script.sh

command. The script describes the task, the main parameters of it, and its running. For example in the following script, the described hostname.sh task:

   #!/bin/sh       
   #$ -N HOSTNAME       
   /bin/hostname 


can be posted with the following command:

   qsub hostname.sh


The scripts can be used for separating the different binaries:

   #!/bin/sh         
   case `uname` in           
    SunOS) ./pi_sun           
    FreeBSD) ./pi_bsd         
   esac

With the following command, we can define the queue where the scheduler puts the job:

   qsub -q serial.q range.sh


The command qsub can be issued with a number of different switches, which are gathered in the following table:

Parameter Possible example Result
-N name -N Flow The job will appear under this name in the queue.
-cwd -cwd The output and the error files will appear in this actual catalog.
-S shell -S /bin/tcsh The shell in which the scripts run.
n} -j y Joining the error and the output in one file.
n} -r y After a restart, should the job restart too (from the beginning).
-M e-mail -M stefan@niif.hu Scheduler information will be sent to this address about the job.
-l -l h_cpu=0:15:0 Chooses a queue for the job where 15 minutes of CPU time could be ensured. (hour:minute:second)
-l -l h_vmem=1G Chooses a computer for the job where 1 GB memory is available. In the case of parallel jobs its value is extended with the required number of slots. If this parameter is not given, the default setting will be the number of the maximum memory cores set up in the computers.
-l -l in Consuming resources, complex request. (This will be defined in the documentation written for the system administrators)
-binding -binding linear:4 Chooses 4 CPU cores on the worker node-on and assignes in a fix way. Further information: here.
-l -l exclusive=true Demand of exclusive task execution (another job will not be scheduled on the chosen computers). It can be used in the following sites: Szeged, Budapest és Debrecen.
-P -P niifi Chooses a HPC project. This command will list the available HPC projects: qconf -sprjl
-R -R y Resource reservation. This will cause that bigger parallel jobs will get higher priority.


qsub command arguments can be added to the ~/.sge_request file. If this file exists then it will be added to the qsub arument list.

Sometimes we want to delete a job before its running. For this you can use the

   qdel job_id

command.

   qdel 903

The example deletes the job number 903.

   qdel -f 903

It can delete the running jobs immediately.

For pending and then continuing jobs, use qmod {-s,-us}.

   qmod -s 903       
   qmod -us 903 


The previous one suspends the running of number 903 (SIGSTOP), while the latter one allows (SIGCONT).


If there is a need to change the features (resource requirements) of a job put into the waiting list, it can be done with the command: qalter


   qalter -l h_cpu=0:12:0 903 


The previous command alternates the hard-CPU requirements of the job number 903 (h_cpu) and changes it to 12 minutes. The switches of the qalter command are mainly overlap the ones of the qsub command.


In a special case, we have to execute the same task, but on different data. These tasks are the array jobs. With SGE we can upload several jobs to the waiting. For example in the pi task shown in previous chapter, it can be posted multiple times, with different parameters, with the following script:array.sh


   #!/bin/sh       
   #$ -N PI_ARRAY_TEST       
   ./pi_gcc `expr $SGE_TASK_ID \* 100000` 

The SGE_TASK_ID is an internal integer used by the SGE, which created values for each running job. The interval can be set up when posting the block:


   qsub -t 1-7 array.sh 


meaning that the array.sh program will run in seven issues, and the SGE_TASK_ID will have the value of 1, 2, ..., 7 in every running issue. The qstat -f shows how the block tasks are split:

   ---------------------------------------------------------------------------------  
   test.q@uv                       BIP   0/0/30         8.15       linux-x64        
   ---------------------------------------------------------------------------------   
   uv.q@uv                         BIP   0/7/1110        8.15      linux-x64         
   907 1.00000 PI_ARRAY_T stefan       r     06/04/2011 10:34:14     1 1          
   907 0.50000 PI_ARRAY_T stefan       t     06/04/2011 10:34:14     1 2          
   907 0.33333 PI_ARRAY_T stefan       t     06/04/2011 10:34:14     1 3          
   907 0.25000 PI_ARRAY_T stefan       t     06/04/2011 10:34:14     1 4          
   907 0.20000 PI_ARRAY_T stefan       t     06/04/2011 10:34:14     1 5          
   907 0.16667 PI_ARRAY_T stefan       t     06/04/2011 10:34:14     1 6          
   907 0.14286 PI_ARRAY_T stefan       t     06/04/2011 10:34:14     1 7    

It is clear, that behind the tasks there are their array index with which we can refer to the components to the task. For example, in the case of block tasks, there is a possibility to delete particular parts of the block. If we want to delete the subtasks from 5-7 of the previous task, the command

   qdel -f 907.5-7 

will delete chosen components, but leaves the tasks 907.1-4 intact. The result of the running is seven individual files, with seven different running solutions:

It can happen; that the task placed in the queue won’t start. This case the:

   qstat -j job_id 

command will show the detailed scheduling information, containing which running parameters are unfulfilled by the task.

The priority of the different tasks only means the gradiation listed in the pending jobs. The scheduler will analyze the tasks in this order. Since it requires the reservation of resources, it is not sure, that the tasks will run exactly the same order.

If we wonder why a certain job won’t start, here’s how you can get information:

   qalter -w v job_id

One possible outcome

   Job 53505 cannot run in queue "szeged.q" because it is not contained in its hard queue list (-q)       
   Job 53505 (-l NONE) cannot run in queue "cn46.szeged.hpc.niif.hu" because exclusive resource (exclusive) is already in use       
   Job 53505 (-l NONE) cannot run in queue "cn48.szeged.hpc.niif.hu" because exclusive resource (exclusive) is already in use       
   Job 53505 cannot run in PE "mpi" because it only offers 0 slots       
   verification: no suitable queues

You can check with this command where the jobs are running:

   qhost -j -q

   HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS 
   ------------------------------------------------------------------------------- 
   global                  -               -     -       -       -       -       - 
   cn01                    linux-x64      48 41.43  126.0G    3.0G     0.0     0.0    
   serial.q             BI    0/42/48            
   120087 0.15501 run.sh     roczei      r     09/23/2012 14:25:51 MASTER 22     
   120087 0.15501 run.sh     roczei      r     09/23/2012 15:02:21 MASTER 78     
   120087 0.15501 run.sh     roczei      r     10/01/2012 07:58:21 MASTER 143     
   120087 0.15501 run.sh     roczei      r     10/01/2012 08:28:51 MASTER 144     
   120087 0.15501 run.sh     roczei      r     10/04/2012 17:41:51 MASTER 158     
   120340 0.13970 pwhg.sh    roczei      r     09/24/2012 23:24:51 MASTER 3     
   120340 0.13970 pwhg.sh    roczei      r     09/24/2012 23:24:51 MASTER 5     
   120340 0.13970 pwhg.sh    roczei      r     09/24/2012 23:24:51 MASTER 19     
   120340 0.13970 pwhg.sh    roczei      r     09/24/2012 23:24:51 MASTER 23     
   120340 0.13970 pwhg.sh    roczei      r     09/24/2012 23:24:51 MASTER 31     
   120340 0.13970 pwhg.sh    roczei      r     09/24/2012 23:24:51 MASTER 33     
   120340 0.13970 pwhg.sh    roczei      r     09/26/2012 13:42:51 MASTER 113     
   120340 0.13970 pwhg.sh    roczei      r     10/01/2012 07:43:06 MASTER 186     
   120340 0.13970 pwhg.sh    roczei      r     10/01/2012 07:58:36 MASTER 187     
   ... 


Queue types

parallel.q - for paralel jobs (jobs can run maximum 31 days)


serial.q - for serial jobs (jobs can run maximum 31 days)


test.q - test queue, the job will be killed after 2 hours



Getting information on the waiting line’s status:


   qstat -g c  


   CUSTER QUEUE         CQLOAD   USED    RES  AVAIL  TOTAL aoACDS  cdsuE   
   -------------------------------------------------------------------------------- 
   parallel.q           0.91    460       0    44     504    0      0  
   serial.q             0.84    200       0    40     240    0      0 
   test.q               0.00      0       0    24      24    0      0


Running PVM job

To run the previously shown and translated gexample application, we need the following task-describing gexample.sh script:

   #!/bin/sh       
   #$ -N GEXAMPLE      
   ./gexample << EOL       
   30       
   5       
   EOL 

We can submit this with the following command:

   qsub -pe pvm 5 gexample.sh 

The -pe pvm 5 command will tell to the SGE to create a PVM parallel computer machine with 5 virtual processors, and run the application in this.

   uv.q@uv                        BIP   0/5/1110               5.15     linux-x64          
   908 1.00000 GEXAMPLE   stefan       r     06/04/2011 13:05:14     5 

Also note that after the running two output files were created: one containing an attached standard error and standard output (GEXAMPLE.o908), another describing the working method of the (GEXAMLE.po908). The latter one is mainly for finding errors.


Running MPI jobs

All computers are set up with several installations of the MPI system: vendor-specific MPI implementations, and MPICH system too. The default setup is the vendor-specific MPI.

Running in the MPI environment is similar to the PVM environment. Let’s have a look at the example shown in the previous chapter connectivity. A very simple task which tests the MPI tasks’internal communication. Use the following connectivity.sh script to run it:

   #!/bin/sh       
   #$ -N CONNECTIVITY 

   mpirun -np $NSLOTS ./connectivity 

Here, the $NLOTS variable indicates that how many processors should be used in the MPI environment. This equals with that number what we have reuired for the parallel environment.

The job can be submitted with the following command:

   qsub -pe mpi 20 connectivity.sh 

With this command we instruct the scheduler to create a parallel MPI environment containing 20 processors, and reserve space for it in one of the queues. Once the space is available, the job starts:

   uv.q@uv                        BIP   0/20/1110               20.30     linux-x64               
   910 1.00000 CONNECTOVI stefan       r     06/04/2011 14:03:14    20      

Running the program will result in two files: the first one (CONNECTIVITY.o910) is the overlap of the result of the already run program standard output and standard error, while the second one (CONNECTIVITY.po910) is for the follow-up of the operation of the parallel environment. If the running is successful, this file is empty. The command -pe mpi 20 can be given in the script too with the directive #$ -pe mpi 20

Important notes: you should use mpirun.sge by SGI MPT on the Debrecen supercomputer when you run a job under SGE. This can automatic parse which machines have been selected by SGE.

This way you can check that you are using SGI MPT or not:

DEBRECEN[service0] ~ (1)$ type mpirun

mpirun is hashed (/opt/nce/packages/global/sgi/mpt/2.04/bin/mpirun)

DEBRECEN[service0] ~ (0)$ type mpirun.sge

mpirun.sge is hashed (/opt/nce/packages/global/sgi/mpt/2.04/bin/mpirun.sge)

DEBRECEN[service0] ~ (0)$

You should use mpirun binary directly if you are using SHF3 environment or you would like to use a more complex MPI run. However, you need to parse the SGE's PE_HOSTFILE environment variable in this case.

Running OpenMP jobs

There are applications that either use the solutions of the operation system for multi-threaded program execution, or use a special library designed for this, like OpenMP. These applications have to be instructed how many threads they can use. The matrix multiplication algorithm presented in the previous chapter can be described with the following omp_mm.sh script

   #!/bin/sh       
   #$ -N OPENMP_MM      

   ./omp_mm 

it can be submitted with this command which will use 6 threads

   qsub -pe openmp 6 omp_mm.sh


Checkpointing support

At the moment the system doesn’t support any automatic checkpointing/restarting mechanism. If it is need, the application has to take care of it.