Commands tagged xargs (148)

  • Liked command 4077 so I improved it, by doing all text manipulation with sed. "Run this as root, it will be helpful to quickly get information about the loaded kernel modules." THX mohan43u Show Sample Output


    1
    lsmod | sed -e '1d' -e 's/\(\([^ ]*\) \)\{1\}.*/\2/' | xargs modinfo | sed -e '/^dep/s/$/\n/g' -e '/^file/b' -e '/^desc/b' -e '/^dep/b' -e d
    marssi · 2009-11-17 22:51:08 4
  • There is a limit to how many processes you can run at the same time for each user, especially with web hosts. If the maximum # of processes for your user is 200, then the following sets OPTIMUM_P to 100. OPTIMUM_P=$(( (`ulimit -u` - `find /proc -maxdepth 1 \( -user $USER -o -group $GROUPNAME \) -type d|wc -l`) / 2 )) This is very useful in scripts because this is such a fast low-resource-intensive (compared to ps, who, lsof, etc) way to determine how many processes are currently running for whichever user. The number of currently running processes is subtracted from the high limit setup for the account (see limits.conf, pam, initscript). An easy to understand example- this searches the current directory for shell scripts, and runs up to 100 'file' commands at the same time, greatly speeding up the command. find . -type f | xargs -P $OPTIMUM_P -iFNAME file FNAME | sed -n '/shell script text/p' I am using it in my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html especially for the xargs command. Xargs has a -P option that lets you specify how many processes to run at the same time. For instance if you have 1000 urls in a text file and wanted to download all of them fast with curl, you could download 100 at a time (check ps output on a separate [pt]ty for proof) like this: cat url-list.txt | xargs -I '{}' -P $OPTIMUM_P curl -O '{}' I like to do things as fast as possible on my servers. I have several types of servers and hosting environments, some with very restrictive jail shells with 20processes limit, some with 200, some with 8000, so for the jailed shells my xargs -P10 would kill my shell or dump core. Using the above I can set the -P value dynamically, so xargs always works, like this. cat url-list.txt | xargs -I '{}' -P $OPTIMUM_P curl -O '{}' If you were building a process-killer (very common for cheap hosting) this would also be handy. Note that if you are only allowed 20 or so processes, you should just use -P1 with xargs. Show Sample Output


    1
    echo $(( `ulimit -u` - `find /proc -maxdepth 1 \( -user $USER -o -group $GROUPNAME \) -type d|wc -l` ))
    AskApache · 2010-03-12 08:42:49 6

  • 1
    find . -name '*.?pp' -exec grep -H "string" {} \;
    rthemocap · 2010-07-14 15:10:23 5
  • This can be useful for transforming command-line args into input for xargs (one per line). This can also be done with ls if the args are filenames, but that's getting awfully close to Useless Use of Cat territory (http://partmaps.org/era/unix/award.html).


    1
    each() { (IFS=$'\n'; echo "$*") }
    BobbyTables · 2010-10-02 06:51:44 6
  • Grabs the cmdline used to execute the process, and the environment that the process is being run under. This is much different than the 'env' command, which only lists the environment for the shell. This is very useful (to me at least) to debug various processes on my server. For example, this lets me see the environment that my apache, mysqld, bind, and other server processes have. Here's a function I use: aa_ps_all () { ( cd /proc && command ps -A -opid= | xargs -I'{}' sh -c 'test $PPID -ne {}&&test -r {}/cmdline&&echo -e "\n[{}]"&&tr -s "\000" " "<{}/cmdline&&echo&&tr -s "\000\033" "\nE"<{}/environ|sort&&cat {}/limits' ); } From my .bash_profile at http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html Show Sample Output


    1
    cd /proc&&ps a -opid=|xargs -I+ sh -c '[[ $PPID -ne + ]]&&echo -e "\n[+]"&&tr -s "\000" " "<+/cmdline&&echo&&tr -s "\000\033" "\nE"<+/environ|sort'
    AskApache · 2010-10-22 02:34:33 14
  • Increase the modification date for the files selected with the find command.


    1
    find . -type f | while read line; do NEW_TS=`date -d@$((\`stat -c '%Y' $line\` + <seconds> )) '+%Y%m%d%H%M.%S'`; touch -t $NEW_TS ${line}; done
    angleto · 2010-11-18 14:03:32 3
  • recursively search dir for a a particular file type, search each file for a particular text. Show Sample Output


    1
    find /name/of/dir/ -name '*.txt' | xargs grep 'text I am searching for'
    erickb · 2011-01-05 15:20:40 3
  • for when find . -print | grep -v .svn | xargs doesnt cut it.


    1
    find . -type f ! -iwholename \*.svn\* -print0 [ | xargs -0 ]
    alustenberg · 2011-03-21 16:45:35 6

  • 1
    find . -type d -name .svn -prune -o -type f -print0 | xargs -r0 ...
    depesz · 2011-03-21 21:36:57 4
  • This example command fetches 'example.com' webpage and then fetches+saves all PDF files listed (linked to) on that webpage. [*Note: of course there are no PDFs on example.com. This is just an example]


    1
    curl -s http://example.com | grep -o -P "<a.*href.*>" | grep -o "http.*.pdf" | xargs -d"\n" -n1 wget -c
    b_t · 2011-06-09 14:42:46 5
  • Grabs the Apache config file (yielded from httpd) and returns the path specified as DocumentRoot. Show Sample Output


    1
    httpd -V | grep -i SERVER_CONFIG_FILE | cut -f2 -d'"' | xargs grep -i '^DocumentRoot' | cut -f2 -d'"'
    dcpesses · 2011-08-13 20:45:05 5
  • Written for Mac OSX. When you are working in a project and want to open it on Github.com, just type "gh" and your default browser will open with the repo you are in. Works for submodules, and repo's that you don't own. You'll need to copy / paste this command into a gh.sh file, then create an alias in your bash or zsh profile to the gh.sh script. Detailed instructions here if you still need help: http://gist.github.com/1917716


    1
    git remote -v | grep fetch | sed 's/\(.*github.com\)[:|/]\(.*\).git (fetch)/\2/' | awk {'print "https://github.com/" $1'} | xargs open
    brockangelo · 2012-04-15 20:48:46 20
  • first grep all href images then sed the url part then wget


    1
    curl -s $1 | grep -o -i '<a href="//images.4chan.org/[^>]*>' | sed -r 's%.*"//([^"]*)".*%\1%' | xargs wget
    bugmenot · 2013-07-22 10:33:55 6
  • Sorts by latest modified files by looking to current directory and all subdirectories Show Sample Output


    1
    find . -name '*pdf*' -print0 | xargs -0 ls -lt | head -20
    fuats · 2013-10-03 21:58:51 9
  • Use GNU Parallel: short, easy to read, and will run one job per core.


    1
    parallel convert {} {.}.png ::: *.svg
    unixmonkey74668 · 2014-04-12 06:39:02 7
  • bash brace expansion, sequence expression Show Sample Output


    1
    echo {-1..-5}days | xargs -n1 date +"%Y-%m-%d" -d
    grault · 2014-07-22 17:56:07 8

  • 1
    find -L /path/to/check -type l | xargs rm
    sn0w · 2015-11-10 12:42:00 11
  • Neither of the others worked for me. This does.


    1
    curl http://url/rss | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$' | xargs wget -c
    dakira · 2016-05-29 12:07:21 21
  • this is good for variables if you have many script created files and if you want to know which one is the last created/changed one..


    1
    find . -type f -print0 | xargs -0 stat -c '%y %n' | sort -n -k 1,1 | awk 'END{print $NF}'
    emphazer · 2018-05-14 08:47:41 155
  • First the find command finds all files in your current directory (.). This is piped to xargs to be able to run the next shell pipeline in parallel. The xargs -P argument specifies how many processes you want to run in parallel, you can set this higher than your core count as the duration reading is mainly IO bound. The -print0 and -0 arguments of find and xargs respectively are used to easily handle files with spaces or other special characters. A subshell is executed by xargs to have a shell pipeline for each file that is found by find. This pipeline extracts the duration and converts it to a format easily parsed by awk. ffmpeg reads the file and prints a lot of information about it, grep extracts the duration line. cut and sed cut out the time information, and tr converts the last . to a : to make it easier to split by awk. awk is a specialized programming language for use in shell scripts. Here we use it to split the time elements in 4 variables and add them up. Show Sample Output


    1
    find . -print0 | xargs -0 -P 40 -n 1 sh -c 'ffmpeg -i "$1" 2>&1 | grep "Duration:" | cut -d " " -f 4 | sed "s/.$//" | tr "." ":"' - | awk -F ':' '{ sum1+=$1; sum2+=$2; sum3+=$3; sum4+=$4 } END { printf "%.0f:%.0f:%.0f.%.0f\n", sum1, sum2, sum3, sum4 }'
    pingiun · 2019-03-01 20:21:48 40
  • Directly download all mp3 files of the desired podcast


    1
    curl http://radiofrance-podcast.net/podcast09/rss_14726.xml | grep -Eo "(http|https)://[a-zA-Z0-9./?=_%:-]*mp3" | sort -u | xargs wget
    pascalvaucheret · 2021-08-09 13:40:26 179

  • 1
    xargs -I% -P10 curl -sL "https://iconfig.co" < <(printf '%s\n' {1..10})
    wuseman1 · 2022-06-02 02:13:14 356

  • 0
    find . -name "*.txt" | xargs sed -i "s/old/new/"
    noblejasper · 2009-09-24 15:02:31 3
  • Nice command to create a list, you can create too with for command, but this is so faster. Show Sample Output


    0
    seq 10 |xargs -n1 echo Printing line
    Waldirio · 2009-10-15 11:05:35 7
  • A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster. Parallel is from https://savannah.nongnu.org/projects/parallel/


    0
    find -not -empty -type f -printf "%s\n" | sort | uniq -d | parallel find -type f -size {}c | parallel md5sum | sort | uniq -w32 --all-repeated=separate
    unixmonkey8046 · 2010-01-28 08:40:18 4
  •  < 1 2 3 4 5 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Find Files That Exceed a Specified Size Limit

drop first column of output by piping to this
An advantage is that this doesn't modify remained string at all. One can change {0,1} with {0,n} to drop several columns

computes the most frequent used words of a text file
using $ cat WAR_AND_PEACE_By_LeoTolstoi.txt | tr -cs "[:alnum:]" "\n"| tr "[:lower:]" "[:upper:]" | sort -S16M | uniq -c |sort -nr | cat -n | head -n 30 ("sort -S1G" - Linux/GNU sort only) will also do the job but as some drawbacks (caused by space/time complexity of sorting) for bigger files...

Find all active ip's in a subnet

Display the history and optionally grep
Place this in your .bash_profile and you can use it two different ways. If you issue 'h' on its own, then it acts like the history command. If you issue: $ h cd Then it will display all the history with the word 'cd'

Normalize volume in your mp3 library
This normalizes volume in your mp3 library, but uses mp3gain's "album" mode. This applies a gain change to all files from each directory (which are presumed to be from the same album) - so their volume relative to one another is changed, while the average album volume is normalized. This is done because if one track from an album is quieter or louder than the others, it was probably meant to be that way.

Show total size of each subdirectory, broken down by KB,MB,GB,TB

Show the command line for a PID with ps
Show the command line for a PID with ps

Calculate days on which Friday the 13th occurs (inspired from the work of the user justsomeguy)
Friday is the 5th day of the week, monday is the 1st. Output may be affected by locale.

Show IP Address in prompt --> PS1 var
when working with many machines in a computer lab need to know the IP addr is very large, this is a simplistic solution to make things easier


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: