How to Simulate Network Faults On Any TCP-based Traffic
This guide shows you how to use fault to simulate network faults on any TCP-oriented network traffic, even with TLS encryption.
Prerequisites
-
Install fault
If you haven’t installed fault yet, follow the installation instructions.
-
Basic Proxy Setup Be familiar with running fault run --with-[fault] commands from your terminal.
-
Understanding of TCP Proxying Explore the TCP proxy protocol reference.
Do I still need HTTP_PROXY
or HTTPS_PROXY
?
When you setup a proxy, you are effectively swapping your target address with the proxy's address in your application. You do not need to set the standard these environment variables.
What about encryption?
The traffic from the client to the proxy is in clear. From the proxy to the target host, the traffic is encrypted if the endpoint expects it to be.
A future version of fault may allow to encrypt the traffic between client and proxy as well with your own certificate.
Create a Dedicated TCP Proxy
fault can create any number of proxies that can be used as endpoints by your applications to experiment with network fault impacts.
-
Start a proxy on port
9098
- Make sure to set a host and its port. fault cannot figure it out.
You can use as many
--proxy
flags as needed. fault will start listening on port 9098 for TCP connections. Any network going to that the address 0.0.0.0:9098 will be transmitted to the endpoint, herehttps://www.google.com
. fault will apply any faults you have setup to the traffic. Please read the reference. for the supported definition of the proxy protocol. -
Make a request to the endpoint via our proxy
curl \ -4 \ # (1)! -H "Host: www.google.com" \ # (2)! -I \ -o /dev/null -s \ -w "Connected IP: %{remote_ip}\nTotal time: %{time_total}s\n" \ https://0.0.0.0:9098 # (3)!
- fault's proxy only support IPv4 for now. That my change in the future.
- Make sure the
Host
headers matches the actual target server. - Instead of connecting to
https://www.google.com
, we connect to the proxy and let it forward our HTTP request tohttps://www.google.com
on our behalf. Note that the proxy doesn't make a request, the traffic sent by curl is sent as-is (aside from the network faults) to the final target endpoint.
Simulate Network Faults on PostgreSQL Traffic
While you may benefit from learning how network faults impact your application at the API (often HTTP) level, it may also be valuable to explore effects from dependencies such as traffic between your application and its database.
-
Start a proxy on port
35432
fault run \ --proxy "35432=localhost:5432" \ # (1)! --with-latency \ --latency-mean 800 \ # (2)! --latency-per-read-write # (3)!
- Let's assume the database is local and listening on port
5432
. Change to match your system. - Let's use a fairly high latency to notice it
- The default for latency faults is to be applied only once in the
life of the connection. With
--latency-per-read-write
you tell fault to apply the fault on any read or write operation. This is useful here for our example because we will connect with psql and without this flag, the latency would be applied only once at connection time.
- Let's assume the database is local and listening on port
-
Connect with psql to the PostgreSQL server via fault's proxy
- The address of your the fault's proxy. You may use
localhost
here or a non-loopback address since the proxy is bound to all interfaces with0.0.0.0
- The proxy's port
- The username to connect to the server, adjust to your own system
- The database name, adjust to your own system
Once you are connected, any query made to the server will go through the proxy which will apply the configured network faults to it.
- The address of your the fault's proxy. You may use