Commands tagged xargs (148)

  • Liked command 4077 so I improved it, by doing all text manipulation with sed. "Run this as root, it will be helpful to quickly get information about the loaded kernel modules." THX mohan43u Show Sample Output


    1
    lsmod | sed -e '1d' -e 's/\(\([^ ]*\) \)\{1\}.*/\2/' | xargs modinfo | sed -e '/^dep/s/$/\n/g' -e '/^file/b' -e '/^desc/b' -e '/^dep/b' -e d
    marssi · 2009-11-17 22:51:08 4
  • There is a limit to how many processes you can run at the same time for each user, especially with web hosts. If the maximum # of processes for your user is 200, then the following sets OPTIMUM_P to 100. OPTIMUM_P=$(( (`ulimit -u` - `find /proc -maxdepth 1 \( -user $USER -o -group $GROUPNAME \) -type d|wc -l`) / 2 )) This is very useful in scripts because this is such a fast low-resource-intensive (compared to ps, who, lsof, etc) way to determine how many processes are currently running for whichever user. The number of currently running processes is subtracted from the high limit setup for the account (see limits.conf, pam, initscript). An easy to understand example- this searches the current directory for shell scripts, and runs up to 100 'file' commands at the same time, greatly speeding up the command. find . -type f | xargs -P $OPTIMUM_P -iFNAME file FNAME | sed -n '/shell script text/p' I am using it in my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html especially for the xargs command. Xargs has a -P option that lets you specify how many processes to run at the same time. For instance if you have 1000 urls in a text file and wanted to download all of them fast with curl, you could download 100 at a time (check ps output on a separate [pt]ty for proof) like this: cat url-list.txt | xargs -I '{}' -P $OPTIMUM_P curl -O '{}' I like to do things as fast as possible on my servers. I have several types of servers and hosting environments, some with very restrictive jail shells with 20processes limit, some with 200, some with 8000, so for the jailed shells my xargs -P10 would kill my shell or dump core. Using the above I can set the -P value dynamically, so xargs always works, like this. cat url-list.txt | xargs -I '{}' -P $OPTIMUM_P curl -O '{}' If you were building a process-killer (very common for cheap hosting) this would also be handy. Note that if you are only allowed 20 or so processes, you should just use -P1 with xargs. Show Sample Output


    1
    echo $(( `ulimit -u` - `find /proc -maxdepth 1 \( -user $USER -o -group $GROUPNAME \) -type d|wc -l` ))
    AskApache · 2010-03-12 08:42:49 6

  • 1
    find . -name '*.?pp' -exec grep -H "string" {} \;
    rthemocap · 2010-07-14 15:10:23 5
  • This can be useful for transforming command-line args into input for xargs (one per line). This can also be done with ls if the args are filenames, but that's getting awfully close to Useless Use of Cat territory (http://partmaps.org/era/unix/award.html).


    1
    each() { (IFS=$'\n'; echo "$*") }
    BobbyTables · 2010-10-02 06:51:44 6
  • Grabs the cmdline used to execute the process, and the environment that the process is being run under. This is much different than the 'env' command, which only lists the environment for the shell. This is very useful (to me at least) to debug various processes on my server. For example, this lets me see the environment that my apache, mysqld, bind, and other server processes have. Here's a function I use: aa_ps_all () { ( cd /proc && command ps -A -opid= | xargs -I'{}' sh -c 'test $PPID -ne {}&&test -r {}/cmdline&&echo -e "\n[{}]"&&tr -s "\000" " "<{}/cmdline&&echo&&tr -s "\000\033" "\nE"<{}/environ|sort&&cat {}/limits' ); } From my .bash_profile at http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html Show Sample Output


    1
    cd /proc&&ps a -opid=|xargs -I+ sh -c '[[ $PPID -ne + ]]&&echo -e "\n[+]"&&tr -s "\000" " "<+/cmdline&&echo&&tr -s "\000\033" "\nE"<+/environ|sort'
    AskApache · 2010-10-22 02:34:33 14
  • Increase the modification date for the files selected with the find command.


    1
    find . -type f | while read line; do NEW_TS=`date -d@$((\`stat -c '%Y' $line\` + <seconds> )) '+%Y%m%d%H%M.%S'`; touch -t $NEW_TS ${line}; done
    angleto · 2010-11-18 14:03:32 3
  • recursively search dir for a a particular file type, search each file for a particular text. Show Sample Output


    1
    find /name/of/dir/ -name '*.txt' | xargs grep 'text I am searching for'
    erickb · 2011-01-05 15:20:40 3
  • for when find . -print | grep -v .svn | xargs doesnt cut it.


    1
    find . -type f ! -iwholename \*.svn\* -print0 [ | xargs -0 ]
    alustenberg · 2011-03-21 16:45:35 6

  • 1
    find . -type d -name .svn -prune -o -type f -print0 | xargs -r0 ...
    depesz · 2011-03-21 21:36:57 4
  • This example command fetches 'example.com' webpage and then fetches+saves all PDF files listed (linked to) on that webpage. [*Note: of course there are no PDFs on example.com. This is just an example]


    1
    curl -s http://example.com | grep -o -P "<a.*href.*>" | grep -o "http.*.pdf" | xargs -d"\n" -n1 wget -c
    b_t · 2011-06-09 14:42:46 5
  • Grabs the Apache config file (yielded from httpd) and returns the path specified as DocumentRoot. Show Sample Output


    1
    httpd -V | grep -i SERVER_CONFIG_FILE | cut -f2 -d'"' | xargs grep -i '^DocumentRoot' | cut -f2 -d'"'
    dcpesses · 2011-08-13 20:45:05 5
  • Written for Mac OSX. When you are working in a project and want to open it on Github.com, just type "gh" and your default browser will open with the repo you are in. Works for submodules, and repo's that you don't own. You'll need to copy / paste this command into a gh.sh file, then create an alias in your bash or zsh profile to the gh.sh script. Detailed instructions here if you still need help: http://gist.github.com/1917716


    1
    git remote -v | grep fetch | sed 's/\(.*github.com\)[:|/]\(.*\).git (fetch)/\2/' | awk {'print "https://github.com/" $1'} | xargs open
    brockangelo · 2012-04-15 20:48:46 20
  • first grep all href images then sed the url part then wget


    1
    curl -s $1 | grep -o -i '<a href="//images.4chan.org/[^>]*>' | sed -r 's%.*"//([^"]*)".*%\1%' | xargs wget
    bugmenot · 2013-07-22 10:33:55 6
  • Sorts by latest modified files by looking to current directory and all subdirectories Show Sample Output


    1
    find . -name '*pdf*' -print0 | xargs -0 ls -lt | head -20
    fuats · 2013-10-03 21:58:51 9
  • Use GNU Parallel: short, easy to read, and will run one job per core.


    1
    parallel convert {} {.}.png ::: *.svg
    unixmonkey74668 · 2014-04-12 06:39:02 7
  • bash brace expansion, sequence expression Show Sample Output


    1
    echo {-1..-5}days | xargs -n1 date +"%Y-%m-%d" -d
    grault · 2014-07-22 17:56:07 8

  • 1
    find -L /path/to/check -type l | xargs rm
    sn0w · 2015-11-10 12:42:00 11
  • Neither of the others worked for me. This does.


    1
    curl http://url/rss | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$' | xargs wget -c
    dakira · 2016-05-29 12:07:21 21
  • this is good for variables if you have many script created files and if you want to know which one is the last created/changed one..


    1
    find . -type f -print0 | xargs -0 stat -c '%y %n' | sort -n -k 1,1 | awk 'END{print $NF}'
    emphazer · 2018-05-14 08:47:41 148
  • First the find command finds all files in your current directory (.). This is piped to xargs to be able to run the next shell pipeline in parallel. The xargs -P argument specifies how many processes you want to run in parallel, you can set this higher than your core count as the duration reading is mainly IO bound. The -print0 and -0 arguments of find and xargs respectively are used to easily handle files with spaces or other special characters. A subshell is executed by xargs to have a shell pipeline for each file that is found by find. This pipeline extracts the duration and converts it to a format easily parsed by awk. ffmpeg reads the file and prints a lot of information about it, grep extracts the duration line. cut and sed cut out the time information, and tr converts the last . to a : to make it easier to split by awk. awk is a specialized programming language for use in shell scripts. Here we use it to split the time elements in 4 variables and add them up. Show Sample Output


    1
    find . -print0 | xargs -0 -P 40 -n 1 sh -c 'ffmpeg -i "$1" 2>&1 | grep "Duration:" | cut -d " " -f 4 | sed "s/.$//" | tr "." ":"' - | awk -F ':' '{ sum1+=$1; sum2+=$2; sum3+=$3; sum4+=$4 } END { printf "%.0f:%.0f:%.0f.%.0f\n", sum1, sum2, sum3, sum4 }'
    pingiun · 2019-03-01 20:21:48 40
  • Directly download all mp3 files of the desired podcast


    1
    curl http://radiofrance-podcast.net/podcast09/rss_14726.xml | grep -Eo "(http|https)://[a-zA-Z0-9./?=_%:-]*mp3" | sort -u | xargs wget
    pascalvaucheret · 2021-08-09 13:40:26 176

  • 1
    xargs -I% -P10 curl -sL "https://iconfig.co" < <(printf '%s\n' {1..10})
    wuseman1 · 2022-06-02 02:13:14 351

  • 0
    find . -name "*.txt" | xargs sed -i "s/old/new/"
    noblejasper · 2009-09-24 15:02:31 3
  • Nice command to create a list, you can create too with for command, but this is so faster. Show Sample Output


    0
    seq 10 |xargs -n1 echo Printing line
    Waldirio · 2009-10-15 11:05:35 7
  • A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster. Parallel is from https://savannah.nongnu.org/projects/parallel/


    0
    find -not -empty -type f -printf "%s\n" | sort | uniq -d | parallel find -type f -size {}c | parallel md5sum | sort | uniq -w32 --all-repeated=separate
    unixmonkey8046 · 2010-01-28 08:40:18 4
  •  < 1 2 3 4 5 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Adding Prefix to File name
Adding course name prefix to lecture pdfs

Rename all subtitles files with the same name of mp4 files in same folder
Use this command if you want to rename all subtitles for them to have the same name as the mp4 files. NOTE: The order of "ls -1 *.mp4" must match the order of "ls -1 *.srt", run the command bellow to make sure the *.srt files will really match the movies after run this command: paste -d:

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

Google Translate
$ translate [output-language] [source-language] 1) "some phrase" should be in quotes 2) [output-language] - optional (default: English) 3) [source-language] - optional (default: auto) $ translate "bonjour petit lapin" hello little rabbit $ translate "bonjour petit lapin" en hello little rabbit $ translate "bonjour petit lapin" en fr hello little rabbit

Calculate days on which Friday the 13th occurs (inspired from the work of the user justsomeguy)
Friday is the 5th day of the week, monday is the 1st. Output may be affected by locale.

Check if the LHC has destroyed the world
This says if the LHC has destroyed the world. Run it in a loop to monitor the state of Earth. Might not work reliable, if the world has actually been destroyed.

Use socat to emulate an SMTP mail SERVER
Lots of scripts show you how to use socat to send an email to an SMTP server; this command actually emulates an SMTP server! It assumes the client is only sending to one recipient, and it's not at all smart, but it'll capture the email into a log file and the client will stop retrying. I used this to diagnose what emails were being sent by cron and subsequently discarded, but you can use it for all sorts of things.

find the rpm package name that provides a specific file
For Linux distributions using rpm (eg Mandriva), this command will find the rpm package name that provides a file.

Redirect STDIN
Several times, I find myself hitting my up arrow, and changing the search term. Unfortunately, I find myself wasting too much time typing: $ grep kernel /var/log/messages Redirecting STDIN allows me to put the search term at the end so I less cursor movement to change what I'm searching for: $ < /var/log/messages grep kernel If you're using the emacs keyboard binding, then after you press your up arrow, press CTRL+w to erase the word. If this has already been submitted, I couldn't find it with the search utility.

dd with progress bar and statistics
Will automatically take the size of the file but longer, usefull only if in an function.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: