Commands tagged mirror (3)

  • website recursive offline mirror with wget


    4
    wget --mirror --convert-links --adjust-extension --page-requisites --recursive --no-parent www.example.com
    aysadk · 2019-03-24 21:13:55 0
  • Mirror a remote directory using some tricks to maximize network speed. lftp:: coolest file transfer tool ever -u: username and password (pwd is merely a placeholder if you have ~/.ssh/id_rsa) -e: execute internal lftp commands set sftp:connect-program: use some specific command instead of plain ssh ssh:: -a -x -T: disable useless things -c arcfour: use the most efficient cipher specification -o Compression=no: disable compression to save CPU mirror: copy remote dir subtree to local dir -v: be verbose (cool progress bar and speed meter, one for each file in parallel) -c: continue interrupted file transfers if possible --loop: repeat mirror until no differences found --use-pget-n=3: transfer each file with 3 independent parallel TCP connections -P 2: transfer 2 files in parallel (totalling 6 TCP connections) sftp://remotehost:22: use sftp protocol on port 22 (you can give any other port if appropriate) You can play with values for --use-pget-n and/or -P to achieve maximum speed depending on the particular network. If the files are compressible removing "-o Compression=n" can be beneficial. Better create an alias for the command. Show Sample Output


    2
    lftp -u user,pwd -e "set sftp:connect-program 'ssh -a -x -T -c arcfour -o Compression=no'; mirror -v -c --loop --use-pget-n=3 -P 2 /remote/dir/ /local/dir/; quit" sftp://remotehost:22
    colemar · 2014-10-17 00:29:34 0
  • sets the option set ftp:ssl-allow false, excludes some files I don't want to synchronize


    0
    lftp -u user,pass -e "set ftp:ssl-allow false; mirror --exclude settings.php --exclude .htaccess serverdir devdir" serverhost
    aufumy · 2015-09-08 21:25:41 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Rsync remote data as root using sudo
If your user has sudo on the remote box, you can rsync data as root without needing to login as root. This is very helpful if the remote box does not allow root to login over SSH (which is a common security restriction).

List all duplicate directories
Very quick! Based only on the content sizes and the character counts of filenames. If both numbers are equal then two (or more) directories seem to be most likely identical. if in doubt apply: $ diff -rq path_to_dir1 path_to_dir2 AWK function taken from here: http://stackoverflow.com/questions/2912224/find-duplicates-lines-based-on-some-delimited-fileds-on-line

Check if network cable is plugged in and working correctly
with 'mii-tool -w eth0' you can watch the interface for changes of the link status

get all bookmarks from all profiles from firefox
for i in $(ls /home/marco/.mozilla/firefox/*\.*/places.sqlite); do sqlite3 $i "SELECT strftime('%d.%m.%Y %H:%M:%S', dateAdded/1000000, 'unixepoch', 'localtime'),url FROM moz_places, moz_bookmarks WHERE moz_places.id = moz_bookmarks.fk ORDER BY dateAdded;"; done

Display condensed log of changes to current git repository
Assuming you are working within a git repository, you can run the above command & see what has changed in reverse chronological order, with one commit per line. Other formatting variations to 'oneline' include 'short', 'medium', 'full', 'fuller', 'email' or 'raw'.

Simultaneously running different Firefox profiles
After running $ firefox -ProfileManager and creating several different profiles, use this command to run multiple Firefox profiles simultaneously.

Edit a script that's somewhere in your path.
Often I need to edit a bash or perl script I've written. I know it's in my path but I don't feel like typing the whole path (or I don't remember the path).

tell if a port is in use
if you don't do --numeric-ports, netstat will try to resolve them to names

Remove old unused kernels from Red Hat Enterprise Linux 5 & Fedora 12/13
Install using yum install yum-utils Options include: --oldkernels Remove old kernel and kernel-devel packages --count=KERNELCOUNT Number of kernel packages to keep on the system (default 2) use package-cleanup --help for a complete list

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: