TIL: You can make HTTP requests without curl using Bash /dev/TCP

We all know curl and wget. They are the stalwarts of command-line HTTP interaction. But what if you’re in a situation where those tools aren’t available? Or, perhaps you just want to learn a fascinating trick that demonstrates the raw power of the Bash shell? Today, we’re diving into using /dev/tcp to make HTTP requests directly from Bash – and, crucially, why this is more than just a cool hack for finance professionals dealing with APIs and data acquisition.

§What is /dev/tcp?

/dev/tcp is a special file in Linux (and other Unix-like systems) that allows you to open a TCP connection directly from the command line. It’s essentially a way to interact with a TCP socket without needing a dedicated program like netcat or, indeed, curl. It's part of the /dev/ directory, which contains device files that represent hardware or system resources. Think of it as a low-level interface to network connections.

§The general syntax is:

§```bash

/dev/tcp/<host>/<port>

This opens a TCP connection to the specified host and port. Anything you write to this file is sent to the server, and anything the server sends back is read from the file. Pretty powerful, right?

§Why Finance Professionals Should Pay Attention

Now, you might be thinking, "That’s neat, but what does this have to do with finance?" A lot, actually! Here’s where /dev/tcp can be incredibly useful:

• API Interactions: Many financial data providers (e.g., stock APIs, forex APIs) offer data through HTTP APIs. /dev/tcp allows you to interact with these APIs even if you’re working in a minimal environment, like a remote server with limited tools installed.
• Quick Data Scraping: Need to grab a quick price quote or exchange rate? /dev/tcp lets you build a simple script to fetch data directly from websites (though be mindful of terms of service and legal implications – see the disclaimer at the end!).
• Automated Reporting: Integrate data fetching into your shell scripts for automated reports and analysis.
• Troubleshooting: Diagnose connectivity issues to financial APIs without relying on complex debugging tools.
• Security (with caution): While not its primary strength, you can leverage /dev/tcp for secure HTTPS requests (more on that later). However, proceed with extreme caution – handling SSL/TLS correctly is crucial.

§Making a Simple GET Request

Let's start with a basic example: fetching the HTML source code of a website. We’ll use example.com as our target.

§```bash

cat /dev/tcp/example.com/80 <<EOF GET / HTTP/1.0 Host: example.com Connection: close

§EOF

§Let’s break down what’s happening here:

• cat /dev/tcp/example.com/80: This opens a connection to example.com on port 80 (the standard HTTP port).
• <<EOF ... EOF: This is a "here document." It allows you to pass multiple lines of text as input to the cat command.
• GET / HTTP/1.0: This is the HTTP GET request. It's requesting the root path (/) using HTTP version 1.0.
• Host: example.com: The Host header is required in HTTP/1.1 and is good practice even in HTTP/1.0. It tells the server which domain you're requesting.
• Connection: close: This tells the server to close the connection after sending the response. Without this, the connection might remain open indefinitely.

When you run this command, the HTML source code of example.com will be printed to your terminal.

§Working with Financial APIs - A Stock Price Example

Let’s look at a more practical example: fetching stock data. For this, we’ll use a hypothetical, simplified API endpoint (replace this with a real API endpoint from a provider like Alpha Vantage, IEX Cloud, or Financial Modeling Prep). Note: you will likely need an API key for these services.

§```bash

API_KEY="YOUR_API_KEY" SYMBOL="AAPL"

§response=$(cat /dev/tcp/api.example.com/443 <<EOF

GET /stock/${SYMBOL}?apikey=${API_KEY} HTTP/1.1 Host: api.example.com Connection: close

§EOF)

§echo "$response"

§Important Considerations for HTTPS:

• Port 443: We're using port 443 for HTTPS.
• SSL/TLS: /dev/tcp doesn't inherently handle SSL/TLS encryption. This is where things get tricky. You can sometimes get away with it for simple testing, but do not rely on /dev/tcp for secure production code where sensitive data is involved. Tools like openssl s_client are much better suited for handling HTTPS connections. (See resources at the end)
• Certificate Verification: Because /dev/tcp doesn't verify SSL certificates, you are vulnerable to man-in-the-middle attacks.

§Handling Responses & Parsing JSON

The response variable in the previous example will contain the entire HTTP response, including the headers and the body. You’ll often need to parse the response body to extract the data you need. For JSON responses (common with modern APIs), you can use tools like jq. If you don’t have jq installed, you can install it with your system's package manager (e.g., apt-get install jq on Debian/Ubuntu, brew install jq on macOS).

§```bash

price=$(echo "$response" | jq '.price') echo "Current stock price: $price"

This command uses jq to extract the value associated with the price key in the JSON response.

§Making POST Requests

You can also make POST requests using /dev/tcp. The key difference is including the request body in the here document.

§```bash

payload='{"name": "New User", "email": "user@example.com"}'

§response=$(cat /dev/tcp/api.example.com/443 <<EOF

POST /users HTTP/1.1 Host: api.example.com Content-Type: application/json Content-Length: $(echo -n "$payload" | wc -c)

§$payload

EOF)

§echo "$response"

• Content-Type: Specifies the format of the request body (JSON in this case).
• Content-Length: Tells the server how many bytes to expect in the request body. It's crucial to set this correctly. The echo -n "$payload" | wc -c part calculates the length of the payload.
• $payload: The actual JSON payload is included after the headers and a blank line.

§Limitations and Alternatives

§While /dev/tcp is a clever trick, it has limitations:

• No Built-in SSL/TLS Support: As mentioned earlier, handling HTTPS securely is challenging.
• Error Handling: It's difficult to handle errors gracefully. You'll need to parse the response code from the headers manually.
• Complexity: Building complex HTTP interactions can become cumbersome.
• Not Portable: /dev/tcp isn't available on all systems.

For more robust and feature-rich HTTP interaction, curl remains the preferred choice. Alternatives like wget, httpie, or even scripting languages like Python with the requests library offer better functionality and error handling. However, when you need a lightweight, dependency-free solution, /dev/tcp can be a lifesaver.

https://example.com/ A good book on Bash scripting can further enhance your ability to automate tasks like these. Consider investing in a comprehensive guide to unlock the full potential of the shell.

§Image Suggestions:

• Image: A screenshot of a Bash terminal showing a successful /dev/tcp request. **
• Image: A diagram illustrating the TCP handshake process. **
• Image: A code snippet showing the extraction of a stock price from a JSON response using jq. **
• Image: A comparison table of curl vs /dev/tcp features. **

§Disclaimer

This article is for educational purposes only. Scraping data from websites may violate their terms of service or be illegal in some jurisdictions. Always review the terms of service before scraping any website. Using /dev/tcp for HTTPS connections without proper SSL/TLS handling is strongly discouraged due to security risks. Affiliate links are included for products that may be helpful; we may receive a commission if you make a purchase through these links. We are not responsible for any damages resulting from the use of the information provided in this article.

§Resources:

• openssl s_client documentation: https://www.openssl.org/docs/manpages/s_client.html
• jq documentation: https://stedolan.github.io/jq/
• Alpha Vantage API: https://www.alphavantage.co/
• IEX Cloud API: https://iexcloud.io/
• Financial Modeling Prep API: https://financialmodelingprep.com/