Curl download file from list of urls

However, this paradigm is a very bad idea, especially if all of your URLs come from the same server -- you're not only going to be spawning another curl instance, but will also be establishing a new TCP connection for each request, which is highly inefficient, and even more so with the now ubiquitous https.

Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Asked 9 years, 8 months ago. Active 1 year, 7 months ago. Viewed 87k times. Finch Finch 1, 3 3 gold badges 15 15 silver badges 15 15 bronze badges. Thanks, bkconrad. Add a comment. Active Oldest Votes. Your xargs may work differently. This seems to work, but it's only giving me a byte HTML file containing the name of the file, not the actual file contents.

Ah, I see. There was a redirect involved so I needed to add the -L option to curl. Plus W10 powershell aliases curl to Invoke-WebRequest which is crudely similar but decidedly not the same. WoodrowShigeru WoodrowShigeru 7 7 bronze badges. It's a bit inefficient to do one curl per line, imo. Can you make one that uses the '-K -' option? Libre Sr. Libre 31 2 2 bronze badges. The -Z option can be combined with -K - to read lots of URLs from a file and download them a lot more quickly than running multiple curl commands MaxWaterman every url config should be in its own line that is enough.

Max Waterman Thank you for the -Z option very useful for mass file download. Dean Householder Dean Householder 3 3 bronze badges. Sign up or log in Sign up using Google. Sign up using Facebook. Out of the box, without any command-line arguments, the curl command will fetch a file and display its contents to the standard output.

Fetching a file and display its contents is all well and good, but what if you want to actually save the file to your system? You can check on things with the cat command:. Execute the following command to download the remote robots. Now use the cat command to display the contents of do-bots.

If you happened to try to fetch the robots. You can verify this by using the -I flag, which displays the request headers rather than the contents of the file:. It only takes a minute to sign up. Connect and share knowledge within a single location that is structured and easy to search.

I have a file that has all the urls from which I need to download. However I need to limit one download at a time. Is this possible using curl? Or should I use anything else. This is possible using curl within a shell script, something like this but you'll need to research appropriate options for curl etc for yourself. Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?

Today we'll be walking you through the basics of using curl to download a file. Our sample file will be the readme for blcli, BitLaunch's command-line interface , which is hosted on GitHub.

In the case of our readme, the complete command would like this:. So what if we want to use cURL to save a file to our server? For that, we must use the -O option:.