condor_ssh_to_job create - an ssh session to a running job
condor_ssh_to_job[-debug] [-name schedd-name] [-pool pool-name] [-ssh
ssh-command] [-keygen-options ssh-keygen-options] [-shells shell1,shell2,...]
[-auto-retry] [-remove-on-interrupt] cluster | cluster.process |
condor_ssh_to_jobcreates an sshsession to a running job. The job is specified
with the argument. If only the job clusterid is given, then the job processid
defaults to the value 0.
condor_ssh_to_jobis available in Unix HTCondor distributions, and works with two
kinds of jobs: those in the vanilla, vm, java, local, or parallel universes,
and those jobs in the grid universe which use EC2 resources. It will not work
with other grid universe jobs.
For jobs in the vanilla, vm, java, local, or parallel universes, the user must
be the owner of the job or must be a queue super user, and both the
condor_scheddand condor_starterdaemons must allow condor_ssh_to_jobaccess. If
no remote-commandis specified, an interactive shell is created. An alternate
sshprogram such as sftpmay be specified, using the -sshoption, for uploading
and downloading files.
The remote command or shell runs with the same user id as the running job, and
it is initialized with the same working directory. The environment is
initialized to be the same as that of the job, plus any changes made by the
shell setup scripts and any environment variables passed by the sshclient. In
addition, the environment variable _CONDOR_JOB_PIDS is defined. It is a
space-separated list of PIDs associated with the job. At a minimum, the list
will contain the PID of the process started when the job was launched, and it
will be the first item in the list. It may contain additional PIDs of other
processes that the job has created.
The sshsession and all processes it creates are treated by HTCondor as though
they are processes belonging to the job. If the slot is preempted or
suspended, the sshsession is killed or suspended along with the job. If the
job exits before the sshsession finishes, the slot remains in the Claimed Busy
state and is treated as though not all job processes have exited until all
sshsessions are closed. Multiple sshsessions may be created to the same job at
the same time. Resource consumption of the sshdprocess and all processes
spawned by it are monitored by the condor_starteras though these processes
belong to the job, so any policies such as PREEMPT that enforce a limit on
resource consumption also take into account resources consumed by the
condor_ssh_to_jobstores ssh keys in temporary files within a newly created and
uniquely named directory. The newly created directory will be within the
directory defined by the environment variable TMPDIR . When the ssh session is
finished, this directory and the ssh keys contained within it are removed.
See the HTCondor administrator's manual section on configuration for details of
the configuration variables related to condor_ssh_to_job.
An sshsession works by first authenticating and authorizing a secure connection
between condor_ssh_to_joband the condor_starterdaemon, using HTCondor
protocols. The condor_startergenerates an ssh key pair and sends it securely
to condor_ssh_to_job. Then the condor_starterspawns sshdin inetd mode with its
stdin and stdout attached to the TCP connection from condor_ssh_to_job.
condor_ssh_to_jobacts as a proxy for the sshclient to communicate with sshd,
using the existing connection authorized by HTCondor. At no point is
sshdlistening on the network for connections or running with any privileges
other than that of the user identity running the job.If CCB is being used to
enable connectivity to the execute node from outside of a firewall or private
network, condor_ssh_to_jobis able to make use of CCB in order to form the
The login shell of the user id running the job is used to run the requested
command, sshdsubsystem, or interactive shell. This is hard-coded behavior in
OpenSSHand cannot be overridden by configuration. This means that
condor_ssh_to_jobaccess is effectively disabled if the login shell disables
access, as in the example programs /bin/trueand /sbin/nologin.
condor_ssh_to_jobis intended to work with OpenSSHas installed in typical
environments. It does not work on Windows platforms. If the sshprograms are
installed in non-standard locations, then the paths to these programs will
need to be customized within the HTCondor configuration. Versions of sshother
than OpenSSHmay work, but they will likely require additional configuration of
command-line arguments, changes to the sshdconfiguration template file, and
possibly modification of the $(LIBEXEC)/condor_ssh_to_job_sshd_setup script
used by the condor_starterto set up sshd.
For jobs in the grid universe which use EC2 resources, a request that HTCondor
have the EC2 service create a new key pair for the job by specifying
ec2_keypair_filecauses condor_ssh_to_jobto attempt to connect to the
corresponding instance via ssh. This attempts invokes sshdirectly, bypassing
the HTCondor networking layer. It supplies sshwith the public DNS name of the
instance and the name of the file with the new key pair's private key. For the
connection to succeed, the instance must have started an sshserver, and its
security group(s) must allow connections on port 22. Conventionally, images
will allow logins using the key pair on a single specific account. Because
sshdefaults to logging in as the current user, the -l <username>option
or its equivalent for other versions of sshwill be needed as part of the
remote-commandargument. Although the -Xoption does not apply to EC2 jobs,
adding -Xor -Yto the remote-commandargument can duplicate the effect.
- Display brief usage information and exit.
- Causes debugging information to be sent to stderr , based
on the value of the configuration variable TOOL_DEBUG
- Specify an alternate condor_schedd, if the default (local)
one is not desired.
- Specify an alternate HTCondor pool, if the default one is
not desired. Does not apply to EC2 jobs.
- Specify an alternate sshprogram to run in place of ssh, for
example sftpor scp. Additional arguments are specified as ssh-command.
Since the arguments are delimited by spaces, place double quote marks
around the whole command, to prevent the shell from splitting it into
multiple arguments to condor_ssh_to_job . If any arguments must contain
spaces, enclose them within single quotes. Does not apply to EC2
- Specify additional arguments to the ssh_keygenprogram, for
creating the ssh key that is used for the duration of the session. For
example, a different number of bits could be used, or a different key type
than the default. Does not apply to EC2 jobs.
- Specify a comma-separated list of shells to attempt to
launch. If the first shell does not exist on the remote machine, then the
following ones in the list will be tried. If none of the specified shells
can be found, /bin/shis used by default. If this option is not specified,
it defaults to the environment variable SHELL from within the
condor_ssh_to_job environment. Does not apply to EC2 jobs.
- Specifies that if the job is not yet running,
condor_ssh_to_job should keep trying periodically until it succeeds or
encounters some other error.
- If specified, attempt to remove the job from the queue if
condor_ssh_to_job is interrupted via a CTRL-c or otherwise terminated
- Enable X11 forwarding. Does not apply to EC2 jobs.
% condor_ssh_to_job 32.0
Welcome to firstname.lastname@example.org!
Your condor job is running with pid(s) 65881.
% gdb -p 65881
Connection to condor-job.tonic.cs.wisc.edu closed.
To upload or download files interactively with sftp:
% condor_ssh_to_job -ssh sftp 32.0
Connecting to condor-job.tonic.cs.wisc.edu...
sftp> get outputfile.dat
This example shows downloading a file from the job with scp. The string
"remote" is used in place of a host name in this example. It is not
necessary to insert the correct remote host name, or even a valid one, because
the connection to the job is created automatically. Therefore, the placeholder
string "remote" is perfectly fine.
% condor_ssh_to_job -ssh scp 32 remote:outputfile.dat .
This example uses condor_ssh_to_jobto accomplish the task of running rsyncto
synchronize a local file with a remote file in the job's working directory.
Job id 32.0 is used in place of a host name in this example. This causes
rsyncto insert the expected job id in the arguments to condor_ssh_to_job.
% rsync -v -e "condor_ssh_to_job " 32.0:outputfile.dat .
Note that condor_ssh_to_jobwas added to HTCondor in version 7.3. If one uses
condor_ssh_to_jobto connect to a job on an execute machine running a version
of HTCondor older than the 7.3 series, the command will fail with the error
Failed to send CREATE_JOB_OWNER_SEC_SESSION to starter
condor_ssh_to_jobwill exit with a non-zero status value if it fails to set up an
ssh session. If it succeeds, it will exit with the status value of the remote
command or shell.
Center for High Throughput Computing, University of Wisconsin-Madison
Copyright (C) 1990-2015 Center for High Throughput Computing, Computer Sciences
Department, University of Wisconsin-Madison, Madison, WI. All Rights Reserved.
Licensed under the Apache License, Version 2.0.