Commands tagged mirror (3)

  • Mirror a remote directory using some tricks to maximize network speed. lftp:: coolest file transfer tool ever -u: username and password (pwd is merely a placeholder if you have ~/.ssh/id_rsa) -e: execute internal lftp commands set sftp:connect-program: use some specific command instead of plain ssh ssh:: -a -x -T: disable useless things -c arcfour: use the most efficient cipher specification -o Compression=no: disable compression to save CPU mirror: copy remote dir subtree to local dir -v: be verbose (cool progress bar and speed meter, one for each file in parallel) -c: continue interrupted file transfers if possible --loop: repeat mirror until no differences found --use-pget-n=3: transfer each file with 3 independent parallel TCP connections -P 2: transfer 2 files in parallel (totalling 6 TCP connections) sftp://remotehost:22: use sftp protocol on port 22 (you can give any other port if appropriate) You can play with values for --use-pget-n and/or -P to achieve maximum speed depending on the particular network. If the files are compressible removing "-o Compression=n" can be beneficial. Better create an alias for the command. Show Sample Output


    4
    lftp -u user,pwd -e "set sftp:connect-program 'ssh -a -x -T -c arcfour -o Compression=no'; mirror -v -c --loop --use-pget-n=3 -P 2 /remote/dir/ /local/dir/; quit" sftp://remotehost:22
    colemar · 2014-10-17 00:29:34 0
  • website recursive offline mirror with wget


    4
    wget --mirror --convert-links --adjust-extension --page-requisites --recursive --no-parent www.example.com
    aysadk · 2019-03-24 21:13:55 1
  • sets the option set ftp:ssl-allow false, excludes some files I don't want to synchronize


    0
    lftp -u user,pass -e "set ftp:ssl-allow false; mirror --exclude settings.php --exclude .htaccess serverdir devdir" serverhost
    aufumy · 2015-09-08 21:25:41 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

return external ip
I dont have curl or links installed, so I use wget with write file as standard out.

make computer speaking to you :)
you can listen to your computer, but don't be carried away

monitor a tail -f command with multiple processes
when using named pipes only one reader is given the output by default. Also, most commands piped to by grep use a buffer which save output until tail -f finishes, which is not convenient. Here, using a combination of tee, sub-processes and the --line-buffered switch in grep we can workaround the problem.

Rename many files in directories and subdirectories
This is probably overkill, but I have some issues when the directories have spaces in their names. The $ find . -type d -print0 | while read -d $'\0' dir; do xxx; done loops over all the subdirectories in this place, ignoring the white spaces (to some extend). $ cd "$dir"; echo " process $dir"; cd -; goes to the directory and back. It also prints some info to check the progress. $ find . -maxdepth 1 -name "*.ogg.mp3" -exec rename 's/.ogg.mp3/.mp3/' {} \; renames the file within the current directory. The whole should work with directories and file names that include white spaces.

Write comments to your history.
A null operation with the name 'comment', allowing comments to be written to HISTFILE. Prepending '#' to a command will *not* write the command to the history file, although it will be available for the current session, thus '#' is not useful for keeping track of comments past the current session.

Trim linebreaks

vi a remote file
notice the double slash

Make a ready-only filesystem ?writeable? by unionfs
First look into /etc/modules if you have unionfs (or squashfs) support. If not, add the modules. UnionFS combines two filesystems. If there is a need to write a file, /tmp/unioncache will be used to write files (first create that directory). Reads will be done where the file is found first. http://tldp.org/HOWTO/SquashFS-HOWTO/creatingandusing.html

put all lines in comment where de word DEBUG is found

Get a brief overview of how many files and directories are installed
To start, you first need to make sure updatedb has been run/updatedb, and initialized the db: $ su -l root -c updatedb This locate command is provided through the mlocate package, installed by default on most GNU/Linux distributions. It's available on the BSDs as well. Not sure about support for proprietary UNIX systems. The output is self-explanatory- it provides an overview of how many directories and files are on your system.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: