9

I am writing a script that locates a special type of file on my system and i want to check if those files are also present on a remote machine.

So to test a single file I use:

ssh -T user@host [[ -f /path/to/data/1/2/3/data.type ]] && echo "File exists" || echo "File does not exist";

But since I have to check blocks of 10 to 15 files, that i would like to check in one go, since I do not want to open a new ssh-connection for every file.

My idea was to do something like:

results=$(ssh "user@host" '
  for file in "${@}"; do
    if [ -e "$file" ]; then
      echo "$file: exists"
    else
      echo "$file: does not exist"
    fi
  done
' "${files_list[@]}") 

Where the file list contains multiple file path. But this does not work, as a result, I would like to have the "echo" string for every file that was in the files_list.

2
  • FYI, there's a great example of how to do this by defining a local function and using declare (aka typeset) in a heredoc on Stack Overflow at stackoverflow.com/questions/76583980/…. It's a good alternative to just scp-ing a script.
    – cas
    Commented 2 days ago
  • Beware TOCTOU here Commented yesterday

6 Answers 6

8

Use connection multiplexing:

ssh -S foo.sock -fNM thehost
ssh -S foo.sock thehost test1
ssh -S foo.sock thehost test2
ssh -S foo.sock thehost test3
ssh -S foo.sock -O exit thehost

This will use multiple command channels over the same SSH connection.

See also ControlPath and ControlPersist in the ssh_config manual.

You can also extend this to scp/sftp; those don't have the same -S but they still accept the multiplex socket path via -o ControlPath= instead.

ssh -S foo.sock -fNM thehost
if ! ssh -S foo.sock thehost "[ -f /path/to/foo ]"; then
    scp -o ControlPath=foo.sock ./foo thehost:/path/to/foo
fi
if ! ssh -S foo.sock thehost "test -f /path/to/bar"; then
    scp -o ControlPath=foo.sock ./bar thehost:/path/to/bar
fi
ssh -S foo.sock -O exit thehost

(Also possible with rsync but slightly more complex. At that point you might find it easier to define the ControlPath ~/.ssh/%r@%h:%p via ~/.ssh/config instead.)

6

In addition to all the other answers, without going into detail about the test script/command right now, you could also mount the remote directory via SSHFS/FUSE and then run the check locally on your client.

5

Another variation on a similar theme. You can wrap the mechanics in a function with a clean interface and pass the filenames via stdin under the hood. Using Bash's read -d '' to read NUL-separated filenames here (to allow for arbitrary filenames):

fileExistsOn()
{
    local rHost=$1
    shift

    printf '%s\0' "$@" |
        ssh -x "$rHost" '
            while IFS= read -r -d "" rFile
            do
                [ -e "$rFile" ] && printf "%s\0" "$rFile"
            done
        '
}

Then you call it like this

fileExistsOn remoteHost /etc/hosts /etc/passwd /etc/nothing

Example output is NUL-separated data, so you can process that in whatever way works for you. One demonstration approach is to pipe into tr '\0' '\n' to get newline-separated items:

/etc/hosts
/etc/passwd
4

The command and arguments given to ssh will be handled like this (from the OpenSSH ssh(1) manual):

If a command is specified, it will be executed on the remote host instead of a login shell. A complete command line may be specified as command, or it may have additional arguments. If supplied, the arguments will be appended to the command, separated by spaces, before it is sent to the server to be executed.

This means that you will have to arrange your command in such a way that adding the list of filenames to the end results in a valid shell command. You also need to ensure that each filename is properly quoted.

Assuming your shell on the local host is bash, you may do this:

ssh "$remote" '
t () {
    for name do
        [ -e "$name" ] && printf "%s exists\n" "$name"
    done
}
t' "${list[@]@Q}"

Or, as a "one-liner":

ssh "$remote" 't () { for n do [ -e "$n" ] && printf "%s exists\n" "$n"; done; }; t' "${list[@]@Q}"

This runs a POSIX shell fragment on the remote machine, which starts by defining a function, t. The function iterates over its arguments and writes out which ones correspond to existing names. The function is called after defining it, and ssh will append the elements of the list generated by "${list[@]@Q}" as arguments to the function call.

The expansion "${list[@]@Q}" is carried out by the local bash shell and will quote each element in the list list (see Stéphane's comments below for caveats).

An example where I connect to a system where only the mbox entry in the given list exists:

$ list=( 1 2 3 mbox "my * mbox" )
$ printf '%s\n' "${list[@]@Q}"
'1'
'2'
'3'
'mbox'
'my * mbox'
$ ssh "$remote" '
t () {
    for name do
        [ -e "$name" ] && printf "%s exists\n" "$name"
    done
}
t' "${list[@]@Q}"
mbox exists

If the local shell is the zsh shell, you would use ${(qq)list} in place of "${list[@]@Q}".

1
  • 2
    Better ${(qq)list} in zsh which uses a safer and more portable form of quoting than $list:q, bash's "${list[@]@Q}" only works safely if the remote shell is also bash and in the same locale (and depending on locale encoding you may also need same libc/bash and version thereof, and same locale definition) Commented yesterday
3

You can't do for file in "$@"; do something; done argument1 argument2, that isn't valid syntax. It has nothing to do with ssh, you can see it with a simple command:

$ for file in "$@"; do echo "F: $file"; done file1 file2
bash: syntax error near unexpected token `file1'

The $@ array holds the positional parameters. These are arguments passed to the shell when it is launched, so they have no meaning here. What you want to do is define an array with your files and pass that to your command. Something like this:

results=$(ssh "user@host" '
  files_list=(file1 file2 file3 file4 file5); 
  for file in "${files_list[@]}"; do 
    [[ -e $file ]] && 
     echo "$file" exists 
  done
')
8
  • Well, it kinda has to do with ssh, since the ssh client concatenates all the arguments it gets to a single string instead of passing the command line arguments separately and intact. Doing the similar thing with a shell, like sh -c 'for file in "$@"; do ... done' sh file1 file2 file3... should work.
    – ilkkachu
    Commented 2 days ago
  • Sure, @ilkkachu, $shell -c 'whatever' args would work, but not for ...; done args, that's my point. The positional parameters array has meaning when referring to the arguments given to a shell but not to something like a for loop as the OP was doing.
    – terdon
    Commented 2 days ago
  • well, yeah, but it's not like they're just concatenating the shell scriptlet and the filenames themselves; instead they are trying to pass the filenames as separate args, exactly as would be right with any other program. It just doesn't work with SSH in the way one might hope.
    – ilkkachu
    Commented 2 days ago
  • Oh, so you think the OP expected that the list of files would be treated as arguments to ssh somehow?
    – terdon
    Commented 2 days ago
  • um, Isn't that exactly what they're doing? They have ssh "user@host" '[shell script]' "${files_list[@]}" there, and that array expansion "${array[@]}" passes each array element as a separate arg to ssh, right? (Pretty much the same way "$@" works, and what happens in for file in "${array[@]}")
    – ilkkachu
    Commented 2 days ago
3

What @terdon said is correct (in that a shell for loop doesn't and can't take args), but you can do this if:

  1. You run bash -c from ssh, rather than running the for loop directly

  2. Remember that, just like when you run bash -c from find ... -exec, the first (zeroth) non-option arg to bash -c after the command string itself is the name of the process as it will be seen in process table (ps). i.e. it's a mandatory dummy argument. Details can be found in man bash, the -c option is documented near the top of the man page.

  3. You don't mind perpetrating some seriously ugly quoting (but that's pretty much guaranteed when running any non-trivial command via ssh). In particular, the '\'' trick (end single-quotes, escaped single-quote, start single-quotes again) to emulate single-quotes inside single-quotes.

  4. Your filenames don't contain whitespace (more precisely, characters in the remote host's $IFS which defaults to <space><tab><newline>). Working around this is possible but will require even more ugly quoting. You'd be better off piping a NUL-separated list of filenames into your ssh one-liner, and maybe use something like perl -0 to run the one-liner (or script file) on the remote host rather than bash. BTW, one of the reasons why perl is a good choice is because it has a plethora of quoting operators (e.g. see perldoc -q q and perldoc -q qq), which eliminates a huge number of extremely annoying shell quoting problems without ugly hacks like '\''.

For example:

files=(/etc/profile /etc/xyzzy)

ssh "user@host" 'bash -c '\''
for file; do
  if [ -e "$file" ]; then
    echo "$file: exists"
  else
    echo "$file: does not exist"
  fi
done'\''' sh "${files[@]}"
/etc/profile: exists
/etc/xyzzy: does not exist

Alternatively, write a script that does what you want, scp it to the remote host (/tmp or somewhere you have write privs), and then run the script with ssh. Use ssh bash /tmp/yourscript yourargs if you had to write it to a filesystem mounted with noexec. There's two extra steps, but it solves all manner of annoying shell quoting issues.

And, finally, a perl -0 example using the qq and q quoting operators as well as perl's version of the C ?: ternary operator:

files=(/etc/profile /etc/xyz "/tmp/this file isnt here")

printf '%s\0' "${files[@]}" |
  ssh "user@host" 'perl -0lne "
     printf qq(%s: %s\n),
       \$_,
       -e \$_ ? q(exists) : q(does not exist)"'
/etc/profile: exists
/etc/xyz: does not exist
/tmp/this file isnt here: does not exist

With this version, I managed to avoid needing to use '\''. In exchange, I had to escape the $ scalar variable sigil prefix as \$ in the perl one-liner to protect it from the remote bash. If the one-liner used a lot of scalar variables, this would be annoying. Even if I used double-quoting for the local host's part of the ssh command, and single-quoting on the remote, escaping would still be necessary. The only way to avoid that would be to scp a script file before running it with ssh.

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.