Commands tagged mirror (3)

  • Fastest segmented parallel sync of a remote directory over ssh

    Mirror a remote directory using some tricks to maximize network speed. lftp:: coolest file transfer tool ever -u: username and password (pwd is merely a placeholder if you have ~/.ssh/id_rsa) -e: execute internal lftp commands set sftp:connect-program: use some specific command instead of plain ssh ssh:: -a -x -T: disable useless things -c arcfour: use the most efficient cipher specification -o Compression=no: disable compression to save CPU mirror: copy remote dir subtree to local dir -v: be verbose (cool progress bar and speed meter, one for each file in parallel) -c: continue interrupted file transfers if possible --loop: repeat mirror until no differences found --use-pget-n=3: transfer each file with 3 independent parallel TCP connections -P 2: transfer 2 files in parallel (totalling 6 TCP connections) sftp://remotehost:22: use sftp protocol on port 22 (you can give any other port if appropriate) You can play with values for --use-pget-n and/or -P to achieve maximum speed depending on the particular network. If the files are compressible removing "-o Compression=n" can be beneficial. Better create an alias for the command. Show Sample Output


    4
    lftp -u user,pwd -e "set sftp:connect-program 'ssh -a -x -T -c arcfour -o Compression=no'; mirror -v -c --loop --use-pget-n=3 -P 2 /remote/dir/ /local/dir/; quit" sftp://remotehost:22
    colemar · 2014-10-17 00:29:34 0
  • website recursive offline mirror with wget

    website recursive offline mirror with wget


    4
    wget -e robots=off --mirror --convert-links --adjust-extension --page-requisites --recursive --no-parent www.example.com
    aysadk · 2019-03-24 21:13:55 0
  • lftp backup from server to dev for a drupal site

    sets the option set ftp:ssl-allow false, excludes some files I don't want to synchronize


    0
    lftp -u user,pass -e "set ftp:ssl-allow false; mirror --exclude settings.php --exclude .htaccess serverdir devdir" serverhost
    aufumy · 2015-09-08 21:25:41 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Display Spinner while waiting for some process to finish
alternatively, run the spinner for 5 seconds: timeout 5 bash -c 'spinner=( Ooooo oOooo ooOoo oooOo ooooO oooOo ooOoo oOooo); while true; do for i in ${spinner[@]}; do for j in seq 0 ${#i}; do echo -en "\b\b"; done; echo -ne "${i}"; sleep 0.2; done; done'

check open ports without netstat or lsof

Convert CSV to JSON
Replace 'csv_file.csv' with your filename.

Get ElasticSearch configuration and version details
Replace localhost:9200 with your server location and port. This is the ElasticSearch's default setup for local instances.

Replace multiple file extensions with a single extension
The above is just a prove of concept based around the nested bash substitution. This could be useful in situations where you're in a directory with many filetypes but you only want to convert a few. $ for f in *.bmp *.jpg *.tga; do convert $f ${f%.*}.png; done or you can use ls | egrep to get more specific... but be warned, files with spaces will cause a ruckus with expansion but the bash for loop uses a space delimited list. $ for f in $(ls | egrep "bmp$|jpg$|tga$"); do convert $f ${f%.*}.png; done I'm guessing some people will still prefer doing it the sed way but I thought the concept of this one was pretty neat. It will help me remember bash substitutions a little better :-P

List just the executable files (or directories) in current directory
A bit shorter ;)

Dump a web page
Useful to browse dangerous web sites.

geoip information
Not my script. Belongs to mathewbauer. Used without his permission. This script gives a single line as shown in the sample output. NOTE: I have blanked out the IP address for obvious security reasons. But you will get whatever is your IP if you run the script. Tested working in bash.

rsync with progress bar.
transfer files from localhost to a remotehost.

Find the processes that are on the runqueue. Processes with a status of
Want to know why your load average is so high? Run this command to see what processes are on the run queue. Runnable processes have a status of "R", and commands waiting on I/O have a status of "D". On some older versions of Linux may require -emo instead of -eo. On Solaris: ps -aefL -o s -o user -o comm | egrep "^O|^R|COMMAND"

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: