I previously wrote about the technical details of the Capital One breach.
I called for improvements to the metadata service. So did Senator Warren and Senator Wyden in their letter to the FTC! Amazon responded today with some changes to the EC2 instance metadata service in direct response to the Capital One breach.
Amazon rolled out a change that they describe as using "session-based requests" in order to mitigate the risk of SSRF in the EC2 metadata service. They refer to the new offering as IMDSv2, with the previous offering used by Capital One being IMDSv1.
The IMDSv1 flow to retrieve a temporary credential looked like this
With IMDSv2, clients must first create a session before retrieving credentials from the metadata service. Clients do this by retrieving a 'session token' that is valid for a configurable amount of time.
You'll notice that everything is mostly the same, with an extra round trip to retrieve a session token from the metadata service.
$ curl -sX PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 1"Above, we create a new session token that is valid for 1 second. Sessions are created with a simple HTTP PUT request to the metadata service.
It returns a base64 encoded token with the effective security of 44 bytes, which is cryptographically secure from brute force. The leading 4 bytes of the 48 byte sequence is standard across all session tokens and is just a prefix.
$ curl -sX PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 1" | base64 -d | xxd
00000000: 0100 0000 1139 a664 12bc b9b2 7263 dad4 .....9.d....rc..
00000010: cb2a 08b1 774c 9bcb 99c8 47bc 3498 5dc2 .*..wL....G.4.].
00000020: 075b d789 6d20 22e2 .[..m ".Creating a session token requires the ability to make a PUT request. Many SSRF attacks use the HTTP GET method. Requiring HTTP PUT prevents GET based attacks (like webhooks) but this is not a complete solution to SSRF problems.
Notice, this request fails to return credentials. The metadata service returns a 405 Method Not Allowed response when using the GET method to retrieve a session token with IMDSv2.
$ curl -s "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 1" -v
* Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.58.0
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 1
>
* HTTP 1.0, assume close after body
< HTTP/1.0 405 Method Not Allowed
< Allow: OPTIONS, PUT
< Content-Length: 0
< Content-Type: text/plain
< Date: Wed, 20 Nov 2019 04:11:08 GMT
< Connection: close
< Server: EC2ws
<
* Closing connection 0This defense mechanism relies on PUT requests being hard for an attacker to forge. In many cases this is true, but in the example of a reverse proxy or firewall it is not difficult to forge.
Amazon's blog post announcing this feature says it is defense in depth for reverse proxies and firewalls specifically. This is not correct in the case where a reverse proxy or firewall proxies all HTTP methods (like PUT). L7 Firewalls (like a WAF) and reverse proxies are actually the same thing, except WAFs have additional logic to block requests.
Reverse proxies and firewalls alike are meant to forward all requests, PUT, GET, PATCH, or DELETE without discrimination, so a misconfigured WAF or reverse proxy could still retrieve instance credentials from the metadata service.
The same misconfigurations of your reverse proxies and WAFs will result in the same compromise with IMDSv2 as IMDSv1.
AWS says that the IMDSv2 metadata service will not respond to requests that include an X-Forwarded-For header. This is good defense in depth, but relying on this is a fundamentally flawed design because AWS still relies on the victim's behavior to prevent this issue. Misconfigured WAFs and Reverse Proxies can still be misconfigured to not include an X-Forwarded-For if they are acting as a transparent proxy.
Amazon cannot break compatibility with version 1 of the metadata service. They have provided a new feature for their security conscious customers. Enforcing that all requests to the metadata service must use session-based requests is manual for customers.
In order to enforce an EC2 instance use the more secure metadata service, AWS customers must configure each EC2 instance in their fleet using the optional MetadataOptions API argument.
For their security conscious customers, AWS has done the hard work to help customers find issues in their infrastructure by logging MetadataNoToken to CloudWatch for all usage of the old metadata service. However, using the new metadata service requires manually updating AWS client libraries.
Amazon has improved the metadata service but the technical merits of the change are underwhelming.
Two specific ideas come to mind that are stronger protection from SSRF attacks. First is a filesystem based approach where credentials can be read from the filesystem. Second is two-factor roles using the externalID, borrowing from the confused deputy problem protection (which was outlined in my previous blog post).
Both would be stronger protections to this problem.
This fix mostly allows AWS to respond to criticism in the wake of the Capital One breach, but it's questionable how much this really protects customers. Implementing this change in a large organization would still require a lot of work from customers, enforcing usage of the new service is opt in, and the protection achieved is not nearly comprehensive.
When comparing the protections of IMDSv2 in AWS with the metadata service protections in GCP and Azure, AWS's offering is unique. IMDSv2 protects against some attacks that GCP and Azure do not by requiring control of the HTTP method.
GCP and Azure have different protections that require attackers to have control over the HTTP headers forwarded to the metadata service.
AWS has improved their offering, but it might not be as comprehensive as their blog post is advertising, and it comes with quite a lot of complexity.
At this point, it is fair to say that AWS' IMDSv2 offering is equal in security to the offering provided by GCP and Azure, but I hope to see IMDSv3 in the future to improve AWS security further. This offering may provide defense in depth, but it is not a full solution to the problem, and adopting this at any reasonable scale is next to impossible.