command line tool and library for transferring data with URLs (curl)

— ny_wk

Disclosure: some links above are affiliate links — if you buy through them I may earn a small commission at no extra cost to you. Thanks for supporting the channel!

Arre, junior, chai peete peete ek zabardast tool ke baare mein baat karte hain jo har DevOps engineer, developer, aur network enthusiast ke toolkit mein *pakka* hona chahiye: **curl**. This versatile command line tool and library for transferring data with URLs is the unsung hero behind countless internet operations, allowing you to interact with web services, download files, test APIs, and troubleshoot networks directly from your terminal, making it an indispensable asset for automation and debugging. Imagine a world where you need to fetch data, upload files, or simply poke a web server, but you don't want to open a browser ya koi heavyweight GUI client. *Wahaan pe curl kaam aata hai, yaar.* It's not just a command; it's the `libcurl` library at its core that powers thousands of applications, from cars and smart TVs to mobile phones and enterprise software. This means that every time you interact with a major web service, there's a good chance `libcurl` is doing the heavy lifting behind the scenes. It's truly the internet transfer backbone for applications affecting billions of humans daily. ### What Exactly is cURL? Your Universal Data Transfer Swiss Army Knife At its heart, **cURL (Client URL)** is two things: a command-line tool and a powerful library called `libcurl`. Both are designed for one primary purpose: transferring data using URL syntax. Simple, right? But the depth and breadth of what it can do are phenomenal. Think of it this way: when you type `curl` in your terminal, you're interacting with the command-line client that uses `libcurl` to perform its operations. This library is what makes it so ubiquitous. It supports an astonishing array of protocols, far beyond just HTTP and HTTPS. We're talking about: * **File Transfer:** FTP, FTPS, SFTP, SCP, FILE (local files). * **Web Protocols:** HTTP, HTTPS, HTTP/2, HTTP/3 (the latest internet protocols!), Gopher (old school, but still there!), DICT. * **Email Protocols:** SMTP, SMTPS, POP3, POP3S, IMAP, IMAPS. * **Directory & Messaging:** LDAP, LDAPS, MQTT. * **Streaming & Remote Access:** RTMP, RTMPS, RTSP, SMB, SMBS, Telnet, TFTP. *Dekho,* this extensive protocol support means `curl` isn't just for web development. It's for embedded systems communicating via MQTT, for transferring files securely across different servers, for scripting email checks, and even for legacy system interactions. It’s like having a universal translator for network communication. Beyond just protocols, `curl` supports a plethora of features essential for modern data transfer: * **Security:** SSL certificates are fully supported, ensuring secure communication. You can even specify client certificates for mutual TLS authentication. * **HTTP Verbs & Data:** Full support for HTTP POST, HTTP PUT, HTTP DELETE, and other methods. You can send form-based data, JSON payloads, or even raw binary data with ease. * **Authentication:** From basic username/password (Basic, Digest, CRAM-MD5, NTLM, Negotiate, Kerberos) to more complex schemes, `curl` has got your back. * **Advanced Networking:** Proxies (HTTP, SOCKS), proxy tunneling, HTTP/2 and HTTP/3 support for faster connections, cookies management, and even file transfer resume (a godsend for large files or unreliable networks). So, when we say `curl` is used in command lines or scripts to transfer data, that's just the tip of the iceberg. Its `libcurl` heart beats in your mobile phone fetching app updates, your smart TV streaming content, your router updating its firmware, and literally thousands of backend services making API calls. It's truly a testament to free and open-source software, built by thousands of contributors and supported by awesome sponsors. The project follows well-established open-source best practices, meaning it's robust, secure, and constantly evolving. ### Installing cURL: Getting This Powerhouse on Your System The good news is, for most modern operating systems, `curl` is either pre-installed or incredibly easy to set up. *Chalo, dekhte hain kaise install karte hain.* #### Linux: The Home Ground for cURL Most Linux distributions include `curl` by default because it's so fundamental. To quickly check if it's there: ```bash curl ``` If you see output like "curl: try 'curl --help' or 'curl --manual' for more information," it's installed! If you get a "command not found" error, no worries, installation is a breeze: * **For Ubuntu/Debian-based systems:** ```bash sudo apt update sudo apt install curl ``` `apt update` refreshes your package lists, and `apt install curl` fetches and installs the latest stable version available in your distribution's repositories. * **For CentOS/RHEL-based systems:** ```bash sudo yum install curl ``` `yum` is the package manager here. On newer RHEL/CentOS versions (8+), `dnf` has replaced `yum` in many contexts, though `yum` might still work as an alias. * **For Fedora systems (and newer RHEL/CentOS systems):** ```bash sudo dnf install curl ``` `dnf` is the next-generation package manager, offering better dependency resolution and performance. Whichever command you use, once installed, you'll have `curl` ready to roll. #### macOS: Apple's Gift to Developers Good news for Mac users: macOS ships with `curl` preinstalled. Apple regularly updates it with OS updates, so you usually have a reasonably current version. However, if you're a developer who needs the absolute latest features, such as experimental HTTP/3 support or very recent security patches, you might want a more cutting-edge version. For this, **Homebrew** is your best friend. 1. **Install Homebrew (if you haven't already):** ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` This command uses the *existing* `curl` on your system to download and run the Homebrew installation script. 2. **Install the latest `curl` via Homebrew:** ```bash brew install curl ``` Homebrew will install the latest version and symlink it into your PATH, often making it the default `curl` your shell uses, overriding the system one. #### Windows: Catching Up Fast! For Windows users, `curl` has made significant strides in native support. * **Windows 10 (version 1803 and above):** Great news! `curl` now ships by default in the Command Prompt and PowerShell. You can just open `cmd` or PowerShell and type `curl`. *Matlab, no installation required!* * **Older Windows Versions or Specific Needs:** For those on older Windows versions, or if you need a specific build/version not shipped by default, the `curl` project provides official Windows binaries. 1. **Download the ZIP:** Visit the official `curl` website's download section (e.g., `curl.se/windows`). Look for the "MinGW" builds for compatibility. 2. **Extract:** You'll get a folder like `curl--mingw`. Let's say it's `curl-7.62.0-win64-mingw`. 3. **Move to a sensible location:** A common practice is to move it directly to `C:\`. So, you'd have `C:\curl-7.62.0-win64-mingw`. 4. **Add to PATH Environment Variable:** This is the crucial step so Windows can find `curl` when you type its command. * Press `Windows key + R`, type `systempropertiesadvanced`, and press Enter. This opens "Advanced System Properties." * Click the "Environment Variables..." button. * In the "System variables" section, find and double-click the `Path` variable. * **For Windows 10:** Click "New" and add `C:\curl-7.62.0-win64-mingw\bin`. * **For older Windows:** Append `;C:\curl-7.62.0-win64-mingw\bin` to the end of the "Variable value" field. Make sure to include the semicolon at the beginning to separate it from existing paths. * Click "OK" on all open dialogs. 5. **Verify:** Open a *new* Command Prompt or PowerShell window (existing ones won't have the updated PATH). Type `curl`. You should see `curl: try 'curl --help' or 'curl --manual' for more information`. Congratulations! Now that `curl` is installed and ready, *chalo, iski shaktiyan dekhte hain*. ### cURL Basic Usage: Your First Steps into the Terminal Web The most fundamental way to use `curl` is to simply provide a URL. This tells `curl` to perform an HTTP GET request to that URL and print the response to your standard output (your terminal). ```bash curl example.com ``` This command will fetch the HTML content of `example.com` and display it directly in your terminal. It's like doing "View Page Source" in a browser, but without the browser! This is useful for quickly inspecting webpage content, checking if a server is responding, or fetching raw API output. #### Downloading Files: `-O` and `-o` One of the most common tasks for `curl` is downloading files. It offers two primary options: 1. **Download and save with the same name (`-O`, uppercase 'O'):** If you want to download a file and save it in your current directory with its original filename, use `-O`. ```bash curl -O https://example.com/downloads/testfile.tar.gz ``` This command will download `testfile.tar.gz` and save it as `testfile.tar.gz` in the directory where you ran the command. 2. **Download and save with a custom name (`-o`, lowercase 'o'):** When you need to rename the file as it's downloaded, or specify a different path, use `-o` followed by your desired filename/path. ```bash curl -o my_archive.tar.gz https://example.com/downloads/testfile.tar.gz ``` Here, `testfile.tar.gz` will be saved as `my_archive.tar.gz`. You can also specify a full path: `curl -o /tmp/downloads/my_archive.tar.gz ...`. While the official documentation might caution against it for simplicity and clarity, `curl` technically allows downloading multiple files in a single command using multiple `-O` or `-o` flags. However, for complex scenarios, it's often cleaner to loop or use separate commands in a script. #### Getting HTTP Headers: `-I` or `--head` When you're debugging web services or just want to quickly check the metadata of a resource without downloading the entire content, getting HTTP headers is super useful. ```bash curl -I example.com # OR curl --head example.com ``` This command sends an HTTP HEAD request, which means the server only sends back the HTTP headers, not the body of the response. You'll see things like: * **HTTP status code:** `HTTP/1.1 200 OK` (success), `404 Not Found`, `301 Moved Permanently`. * **Content-Type:** `text/html`, `application/json`, `image/jpeg`. * **Server:** What web server software is running. * **Date, Last-Modified, ETag:** Caching information. * **Set-Cookie:** If the server is trying to set cookies. This is invaluable for checking if a URL is actually live, if it's redirecting (`Location` header for 3xx status codes), or what kind of content to expect without incurring the overhead of a full download. ### Advanced cURL: Unleashing Its Full Potential for DevOps & Beyond Now, let's dive into some of the more powerful and frequently used features that make `curl` an absolute workhorse, especially for DevOps engineers testing APIs, automating tasks, and troubleshooting network issues. *Yahaan se asli mazaa shuru hota hai!* #### Debugging with Verbosity: `-v` When things aren't working as expected, the `-v` (verbose) flag is your best friend. It shows you the full request `curl` sends, the headers it receives, SSL handshake details, and more. ```bash curl -v example.com ``` You'll see lines starting with `*` (informational), `>` (request headers sent), and `<` (response headers received). This level of detail is critical for diagnosing connectivity issues, incorrect headers, or authentication failures. #### Specifying HTTP Request Methods: `-X` By default, `curl` performs a GET request. But for interacting with RESTful APIs, you'll often need to use POST, PUT, DELETE, etc. The `-X` flag allows you to specify the HTTP method. ```bash # Send a POST request curl -X POST https://api.example.com/resources # Send a PUT request curl -X PUT https://api.example.com/resources/123 # Send a DELETE request curl -X DELETE https://api.example.com/resources/123 ``` Without sending data, these commands just tell the server *what kind* of action you intend to perform. #### Sending Data with Requests: `-d` and `-F` Most non-GET requests involve sending data to the server. `curl` provides flexible ways to do this. 1. **Sending URL-encoded form data (`-d` or `--data`):** This is common for traditional HTML forms or simple API payloads. `curl` automatically sets the `Content-Type` header to `application/x-www-form-urlencoded`. ```bash curl -X POST -d "param1=value1¶m2=value2" https://api.example.com/submit ``` You can also provide multiple `-d` flags, or read data from a file using `@`: ```bash curl -X POST -d "@data.txt" https://api.example.com/submit ``` If `data.txt` contains `param1=value1¶m2=value2`, it will be sent as the request body. 2. **Sending JSON data (`-d` with `-H`):** For modern REST APIs, JSON is the standard. You need to explicitly set the `Content-Type` header to `application/json`. ```bash curl -X POST \ -H "Content-Type: application/json" \ -d '{"name": "Alice", "age": 30}' \ https://api.example.com/users ``` Again, you can read JSON from a file: ```bash curl -X POST \ -H "Content-Type: application/json" \ -d "@user.json" \ https://api.example.com/users ``` 3. **Uploading files (multipart/form-data) (`-F` or `--form`):** This is used for file uploads, similar to an HTML form with `enctype="multipart/form-data"`. ```bash curl -X POST \ -F "name=My Document" \ -F "document=@/path/to/my_doc.pdf" \ https://api.example.com/upload ``` Here, `name` is a regular form field, and `document` is a file upload field where `@/path/to/my_doc.pdf` specifies the file to upload. #### Handling Authentication: `-u` and `--negotiate` Many APIs and services require authentication. `curl` handles several types: 1. **Basic Authentication (`-u` or `--user`):** This is the most common form, sending a base64-encoded username:password. ```bash curl -u "admin:password123" https://secured.example.com/data ``` If you omit the password, `curl` will prompt you for it interactively, which is more secure as it doesn't store the password in your shell history. 2. **Other Authentication Schemes:** `curl` supports Digest, NTLM, Negotiate (Kerberos/SPNEGO), and more. You might need to specify `--digest`, `--ntlm`, or `--negotiate` along with `-u`. For example, for Kerberos: ```bash curl --negotiate -u ":" https://kerberos.example.com/service ``` #### Managing Cookies: `-b` and `-c` Cookies are essential for maintaining session state. `curl` lets you send and receive them. * **Sending cookies (`-b` or `--cookie`):** To send specific cookies with your request: ```bash curl -b "session_id=abc123; user_pref=dark" https://example.com/profile ``` You can also read cookies from a file (Netscape cookie format): ```bash curl -b cookies.txt https://example.com/profile ``` * **Saving cookies (`-c` or `--cookie-jar`):** To save cookies received from a server to a file: ```bash curl -c cookies.txt https://example.com/login # Then use them in a subsequent request curl -b cookies.txt https://example.com/dashboard ``` #### Working with Proxies: `-x` In corporate environments or for debugging network paths, using a proxy is common. ```bash curl -x http://proxy.example.com:8080 https://target.example.com ``` This tells `curl` to route its request through the specified proxy. You can also specify proxy authentication with `-U`. #### Following Redirects: `-L` Many web services use HTTP 3xx redirects (e.g., 301 Moved Permanently, 302 Found). By default, `curl` will stop after receiving a redirect. To follow them automatically: ```bash curl -L https://shorturl.example.com/my-link ``` This is super handy when dealing with shortened URLs or services that redirect users after login. #### Resuming File Transfers: `-C -` For large file downloads over unstable networks, `curl` can resume interrupted transfers. ```bash curl -C - -O https://example.com/bigfile.zip ``` If `bigfile.zip` was partially downloaded, `curl` will check its size and resume from where it left off. The `-C -` tells `curl` to automatically figure out the byte offset. #### Ignoring SSL Certificate Warnings (Use with Caution!): `-k` or `--insecure` Sometimes, especially in development or internal environments, you might encounter self-signed SSL certificates. `curl` is strict by default, which is good for security. But for testing, you might need to bypass certificate verification. ```bash curl -k https://dev-server.example.com/api ``` **WARNING:** Never use `-k` in production or for sensitive data transfers over untrusted networks. It opens up your connection to man-in-the-middle attacks. Only use it when you *know* what you're doing and the security implications are understood and accepted within a controlled environment. #### Handling HTTP/2 and HTTP/3 `curl` is at the forefront of adopting new HTTP standards. If your `curl` build and the server both support HTTP/2 or HTTP/3, `curl` will often negotiate to use them automatically for performance benefits. You can explicitly request HTTP/2 with `--http2` or HTTP/3 with `--http3`. ```bash curl --http2 https://example.com # Explicitly request HTTP/2 curl --http3 https://example.com # Explicitly request HTTP/3 (requires compatible build) ``` These newer protocols offer features like multiplexing and improved performance, especially over high-latency connections. ### Real-World Scenarios: Where cURL Shines in DevOps *Arre bhai*, `curl` sirf commands run karne ke liye nahi hai; yeh poora game-changer hai for DevOps. Here are some scenarios where `curl` is indispensable: 1. **API Testing and Debugging:** This is probably where I use `curl` the most. Before writing any client code, I use `curl` to test API endpoints, experiment with different payloads, and quickly verify responses. It helps me understand the API contract without relying on a full-fledged client. Whether it's a REST API, a GraphQL endpoint, or even a SOAP service (though SOAP is a bit more verbose), `curl` can send the requests and show you the raw response. * **Health Checks:** ```bash curl -s -o /dev/null -w "%{http_code}\n" https://my-service.com/health # -s: silent, -o /dev/null: discard output, -w: write out HTTP code ``` * **Interacting with Kubernetes API:** ```bash # Assuming KUBECONFIG and authentication are set up TOKEN=$(kubectl get secret -o jsonpath='{.data.token}' | base64 --decode) curl -H "Authorization: Bearer $TOKEN" https:///api/v1/pods ``` 2. **Automation and Scripting:** In CI/CD pipelines, cron jobs, or custom automation scripts, `curl` is king. * **Triggering Webhooks:** To notify other services or trigger builds. ```bash curl -X POST -H "Content-Type: application/json" -d '{"event": "deployment_success"}' https://webhook.site/abcdef ``` * **Downloading Artifacts:** Fetching build artifacts from a repository. ```bash curl -L -O https://artifactory.example.com/my-app/app-v1.0.tar.gz ``` * **Monitoring and Alerting:** Checking external service availability or submitting custom metrics. 3. **Network Troubleshooting:** Is the firewall blocking me? Is the proxy working? Is the SSL certificate valid? `curl` can help diagnose these issues. * **Checking SSL/TLS Handshake:** `curl -v https://secure.example.com` * **Testing Proxy Connectivity:** `curl -x http://myproxy:8080 -v https://google.com` 4. **Interacting with Cloud APIs:** Cloud providers like AWS, Azure, GCP expose their services via REST APIs. `curl` can be used to make direct calls, especially useful for one-off tasks or when a dedicated SDK is overkill. 5. **Data Synchronization:** Moving files between servers or pulling data from remote sources into local processing pipelines. ### Pitfalls and Best Practices While `curl` is powerful, misuse can lead to security vulnerabilities or debugging headaches. * **Security for `-k` (`--insecure`):** *Yaar*, this one is crucial. While tempting for quick tests, never use `-k` in production scripts or when dealing with sensitive data. Always ensure proper certificate validation. * **Sensitive Data in History:** When passing credentials directly in the command (`-u user:pass`), they might end up in your shell history. For sensitive scripts, consider: * Using environment variables. * Reading passwords interactively (omit password in `-u` flag). * Using a `.netrc` file. * **Error Handling in Scripts:** `curl` returns non-zero exit codes on failure. Always check `$?` in your scripts or use `curl --fail` which ensures `curl` exits with an error code on HTTP errors (like 4xx or 5xx), which is not its default behavior. * **Verbosity is Your Friend:** When debugging, always start with `-v`. It provides immense insight into what's happening. * **Configuration Files:** For complex, repetitive `curl` commands, especially in scripts, consider using a `curl` config file (`~/.curlrc` or specified with `-K`). This helps keep commands clean and manage many options. * **Know Your Protocols:** `curl` supports many protocols. Ensure you're using the correct options for the specific protocol you're interacting with (e.g., FTP specific options are different from HTTP). * **Read the Man Page:** `man curl` is a goldmine. It's extensive but incredibly detailed, covering every option and nuance. ### Key Takeaways * **Universal Tool:** `curl` is both a command-line tool and the `libcurl` library, supporting an incredible range of protocols (HTTP/S, FTP/S, SCP/SFTP, IMAP/S, SMTP/S, MQTT, and many more). * **Ubiquitous:** It's the backbone for data transfer in thousands of applications, from embedded devices to major cloud services, impacting billions daily. * **Essential for DevOps:** Critical for API testing, automation in CI/CD, network troubleshooting, and interacting with cloud platforms. * **Installation is Easy:** Pre-installed on macOS and modern Windows, and easily installed on Linux with standard package managers. * **Powerful Features:** Supports HTTP methods, data sending (forms, JSON, file uploads), various authentication schemes, cookie management, proxies, following redirects, and resuming transfers. ### Frequently Asked Questions ### What is the difference between `curl -O` and `curl -o`? `curl -O` (uppercase 'O') downloads a file and saves it in the current directory using its original filename. For example, `curl -O https://example.com/file.txt` will save the file as `file.txt`. `curl -o` (lowercase 'o') allows you to specify a custom filename or full path for the downloaded file. For instance, `curl -o custom_name.txt https://example.com/file.txt` will save it as `custom_name.txt`. ### How can I send a POST request with JSON data using `curl`? To send a POST request with JSON data, you need to specify the HTTP method as POST using `-X POST`, set the `Content-Type` header to `application/json` using `-H "Content-Type: application/json"`, and provide your JSON payload using `-d` (or `@filename` for JSON from a file). Example: `curl -X POST -H "Content-Type: application/json" -d '{"key": "value"}' https://api.example.com/resource` ### Is it safe to use `curl --insecure`? No, generally it's not safe. `curl --insecure` (or `-k`) disables SSL certificate verification, meaning `curl` will proceed with the connection even if the server's certificate is invalid, self-signed, or expired. This makes your connection vulnerable to man-in-the-middle attacks where an attacker could intercept and modify your data. It should only be used in controlled development or testing environments where you understand and accept the security risks. Never use it in production or for sensitive data transfers over untrusted networks. ### How can I check the HTTP status code of a URL without downloading its content? You can use `curl -I` or `curl --head` to send an HTTP HEAD request. This command retrieves only the HTTP headers from the server, which include the status code, without downloading the entire response body. This is efficient for checking URL availability or header information. Example: `curl -I https://example.com` will show headers like `HTTP/1.1 200 OK`. So, next time you're troubleshooting a microservice, testing an API endpoint, or scripting an automated task, remember your reliable companion, `curl`. It's a foundational skill for anyone in tech, especially in the DevOps realm. The more you use it, the more you'll appreciate its power and flexibility. For a visual walkthrough and more practical examples, don't forget to check out the original video on **@explorenystream**! They've got some great insights there. And if you found this helpful, why not subscribe to their channel for more awesome tech content?