Commands tagged xargs (148)

  • Liked command 4077 so I improved it, by doing all text manipulation with sed. "Run this as root, it will be helpful to quickly get information about the loaded kernel modules." THX mohan43u Show Sample Output


    1
    lsmod | sed -e '1d' -e 's/\(\([^ ]*\) \)\{1\}.*/\2/' | xargs modinfo | sed -e '/^dep/s/$/\n/g' -e '/^file/b' -e '/^desc/b' -e '/^dep/b' -e d
    marssi · 2009-11-17 22:51:08 4
  • There is a limit to how many processes you can run at the same time for each user, especially with web hosts. If the maximum # of processes for your user is 200, then the following sets OPTIMUM_P to 100. OPTIMUM_P=$(( (`ulimit -u` - `find /proc -maxdepth 1 \( -user $USER -o -group $GROUPNAME \) -type d|wc -l`) / 2 )) This is very useful in scripts because this is such a fast low-resource-intensive (compared to ps, who, lsof, etc) way to determine how many processes are currently running for whichever user. The number of currently running processes is subtracted from the high limit setup for the account (see limits.conf, pam, initscript). An easy to understand example- this searches the current directory for shell scripts, and runs up to 100 'file' commands at the same time, greatly speeding up the command. find . -type f | xargs -P $OPTIMUM_P -iFNAME file FNAME | sed -n '/shell script text/p' I am using it in my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html especially for the xargs command. Xargs has a -P option that lets you specify how many processes to run at the same time. For instance if you have 1000 urls in a text file and wanted to download all of them fast with curl, you could download 100 at a time (check ps output on a separate [pt]ty for proof) like this: cat url-list.txt | xargs -I '{}' -P $OPTIMUM_P curl -O '{}' I like to do things as fast as possible on my servers. I have several types of servers and hosting environments, some with very restrictive jail shells with 20processes limit, some with 200, some with 8000, so for the jailed shells my xargs -P10 would kill my shell or dump core. Using the above I can set the -P value dynamically, so xargs always works, like this. cat url-list.txt | xargs -I '{}' -P $OPTIMUM_P curl -O '{}' If you were building a process-killer (very common for cheap hosting) this would also be handy. Note that if you are only allowed 20 or so processes, you should just use -P1 with xargs. Show Sample Output


    1
    echo $(( `ulimit -u` - `find /proc -maxdepth 1 \( -user $USER -o -group $GROUPNAME \) -type d|wc -l` ))
    AskApache · 2010-03-12 08:42:49 6

  • 1
    find . -name '*.?pp' -exec grep -H "string" {} \;
    rthemocap · 2010-07-14 15:10:23 5
  • This can be useful for transforming command-line args into input for xargs (one per line). This can also be done with ls if the args are filenames, but that's getting awfully close to Useless Use of Cat territory (http://partmaps.org/era/unix/award.html).


    1
    each() { (IFS=$'\n'; echo "$*") }
    BobbyTables · 2010-10-02 06:51:44 6
  • Grabs the cmdline used to execute the process, and the environment that the process is being run under. This is much different than the 'env' command, which only lists the environment for the shell. This is very useful (to me at least) to debug various processes on my server. For example, this lets me see the environment that my apache, mysqld, bind, and other server processes have. Here's a function I use: aa_ps_all () { ( cd /proc && command ps -A -opid= | xargs -I'{}' sh -c 'test $PPID -ne {}&&test -r {}/cmdline&&echo -e "\n[{}]"&&tr -s "\000" " "<{}/cmdline&&echo&&tr -s "\000\033" "\nE"<{}/environ|sort&&cat {}/limits' ); } From my .bash_profile at http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html Show Sample Output


    1
    cd /proc&&ps a -opid=|xargs -I+ sh -c '[[ $PPID -ne + ]]&&echo -e "\n[+]"&&tr -s "\000" " "<+/cmdline&&echo&&tr -s "\000\033" "\nE"<+/environ|sort'
    AskApache · 2010-10-22 02:34:33 14
  • Increase the modification date for the files selected with the find command.


    1
    find . -type f | while read line; do NEW_TS=`date -d@$((\`stat -c '%Y' $line\` + <seconds> )) '+%Y%m%d%H%M.%S'`; touch -t $NEW_TS ${line}; done
    angleto · 2010-11-18 14:03:32 3
  • recursively search dir for a a particular file type, search each file for a particular text. Show Sample Output


    1
    find /name/of/dir/ -name '*.txt' | xargs grep 'text I am searching for'
    erickb · 2011-01-05 15:20:40 3
  • for when find . -print | grep -v .svn | xargs doesnt cut it.


    1
    find . -type f ! -iwholename \*.svn\* -print0 [ | xargs -0 ]
    alustenberg · 2011-03-21 16:45:35 6

  • 1
    find . -type d -name .svn -prune -o -type f -print0 | xargs -r0 ...
    depesz · 2011-03-21 21:36:57 4
  • This example command fetches 'example.com' webpage and then fetches+saves all PDF files listed (linked to) on that webpage. [*Note: of course there are no PDFs on example.com. This is just an example]


    1
    curl -s http://example.com | grep -o -P "<a.*href.*>" | grep -o "http.*.pdf" | xargs -d"\n" -n1 wget -c
    b_t · 2011-06-09 14:42:46 5
  • Grabs the Apache config file (yielded from httpd) and returns the path specified as DocumentRoot. Show Sample Output


    1
    httpd -V | grep -i SERVER_CONFIG_FILE | cut -f2 -d'"' | xargs grep -i '^DocumentRoot' | cut -f2 -d'"'
    dcpesses · 2011-08-13 20:45:05 5
  • Written for Mac OSX. When you are working in a project and want to open it on Github.com, just type "gh" and your default browser will open with the repo you are in. Works for submodules, and repo's that you don't own. You'll need to copy / paste this command into a gh.sh file, then create an alias in your bash or zsh profile to the gh.sh script. Detailed instructions here if you still need help: http://gist.github.com/1917716


    1
    git remote -v | grep fetch | sed 's/\(.*github.com\)[:|/]\(.*\).git (fetch)/\2/' | awk {'print "https://github.com/" $1'} | xargs open
    brockangelo · 2012-04-15 20:48:46 20
  • first grep all href images then sed the url part then wget


    1
    curl -s $1 | grep -o -i '<a href="//images.4chan.org/[^>]*>' | sed -r 's%.*"//([^"]*)".*%\1%' | xargs wget
    bugmenot · 2013-07-22 10:33:55 6
  • Sorts by latest modified files by looking to current directory and all subdirectories Show Sample Output


    1
    find . -name '*pdf*' -print0 | xargs -0 ls -lt | head -20
    fuats · 2013-10-03 21:58:51 9
  • Use GNU Parallel: short, easy to read, and will run one job per core.


    1
    parallel convert {} {.}.png ::: *.svg
    unixmonkey74668 · 2014-04-12 06:39:02 7
  • bash brace expansion, sequence expression Show Sample Output


    1
    echo {-1..-5}days | xargs -n1 date +"%Y-%m-%d" -d
    grault · 2014-07-22 17:56:07 8

  • 1
    find -L /path/to/check -type l | xargs rm
    sn0w · 2015-11-10 12:42:00 11
  • Neither of the others worked for me. This does.


    1
    curl http://url/rss | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$' | xargs wget -c
    dakira · 2016-05-29 12:07:21 21
  • this is good for variables if you have many script created files and if you want to know which one is the last created/changed one..


    1
    find . -type f -print0 | xargs -0 stat -c '%y %n' | sort -n -k 1,1 | awk 'END{print $NF}'
    emphazer · 2018-05-14 08:47:41 155
  • First the find command finds all files in your current directory (.). This is piped to xargs to be able to run the next shell pipeline in parallel. The xargs -P argument specifies how many processes you want to run in parallel, you can set this higher than your core count as the duration reading is mainly IO bound. The -print0 and -0 arguments of find and xargs respectively are used to easily handle files with spaces or other special characters. A subshell is executed by xargs to have a shell pipeline for each file that is found by find. This pipeline extracts the duration and converts it to a format easily parsed by awk. ffmpeg reads the file and prints a lot of information about it, grep extracts the duration line. cut and sed cut out the time information, and tr converts the last . to a : to make it easier to split by awk. awk is a specialized programming language for use in shell scripts. Here we use it to split the time elements in 4 variables and add them up. Show Sample Output


    1
    find . -print0 | xargs -0 -P 40 -n 1 sh -c 'ffmpeg -i "$1" 2>&1 | grep "Duration:" | cut -d " " -f 4 | sed "s/.$//" | tr "." ":"' - | awk -F ':' '{ sum1+=$1; sum2+=$2; sum3+=$3; sum4+=$4 } END { printf "%.0f:%.0f:%.0f.%.0f\n", sum1, sum2, sum3, sum4 }'
    pingiun · 2019-03-01 20:21:48 40
  • Directly download all mp3 files of the desired podcast


    1
    curl http://radiofrance-podcast.net/podcast09/rss_14726.xml | grep -Eo "(http|https)://[a-zA-Z0-9./?=_%:-]*mp3" | sort -u | xargs wget
    pascalvaucheret · 2021-08-09 13:40:26 178

  • 1
    xargs -I% -P10 curl -sL "https://iconfig.co" < <(printf '%s\n' {1..10})
    wuseman1 · 2022-06-02 02:13:14 355

  • 0
    find . -name "*.txt" | xargs sed -i "s/old/new/"
    noblejasper · 2009-09-24 15:02:31 3
  • Nice command to create a list, you can create too with for command, but this is so faster. Show Sample Output


    0
    seq 10 |xargs -n1 echo Printing line
    Waldirio · 2009-10-15 11:05:35 7
  • A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster. Parallel is from https://savannah.nongnu.org/projects/parallel/


    0
    find -not -empty -type f -printf "%s\n" | sort | uniq -d | parallel find -type f -size {}c | parallel md5sum | sort | uniq -w32 --all-repeated=separate
    unixmonkey8046 · 2010-01-28 08:40:18 4
  •  < 1 2 3 4 5 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Find and copy scattered mp3 files into one directory
No problem with word splitting. That should works on many Unix likes.

Open Remote Desktop (RDP) from command line having a custom screen size
This example uses xfreerdp, which builds upon the development of rdesktop. This example usage will also send you the remote machine's sound.

Put readline into vi mode
This lets you use your favorite vi edit keys to navigate your term. To set it permanently, put "set editing-mode vi" in your ~/.inputrc or /etc/inputrc.

Working random fact generator
extension to tali713's random fact generator. It takes the output & sends it to notify-osd. Display time is proportional to the lengh of the fact.

Cut out a piece of film from a file. Choose an arbitrary length and starting time.
With: -vcodec, you choose what video codec the new file should be encoded with. Run ffmpeg -formats E to list all available video and audio encoders and file formats. copy, you choose the video encoder that just copies the file. -acodec, you choose what audio codec the new file should be encoded with. copy, you choose the audio encoder that just copies the file. -i originalfile, you provide the filename of the original file to ffmpeg -ss 00:01:30, you choose the starting time on the original file in this case 1 min and 30 seconds into the film -t 0:0:20, you choose the length of the new film newfile, you choose the name of the file created. Here is more information of how to use ffmpeg: http://www.ffmpeg.org/ffmpeg-doc.html

Netcat Relay
This is an example of using 3 hosts, in a netcat relay. first host connects to middle host 1 -> 2 Second hosts redirects to target host 1 -> 2 -> 3 I hope this makes sense.

find files ignoring .svn and its decendents
for when find . -print | grep -v .svn | xargs doesnt cut it.

ASCII art of yourself
Use libcaca to render ascii chars on the webcam input... or don't.

list files recursively by size

Run a script in parrallel over ssh
Runs a local script over ssh assuming ssh keys are in place. -P argument prints results to stdout. # Uses - https://code.google.com/p/parallel-ssh/


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: