Typically, an infrastructure is made up of numerous critical services which should not be exposed to everyone. In this blog by Uzziah, learn how Cloudflare Access enables you to protect internal services that you’d rather not expose to everyone.
Table of Contents:
5000-foot view of using Cloudflare Access to protect internal applications.
In this piece, I’ll present my findings on using Cloudflare to protect internal services that you’d rather not expose to everyone. The illustration above shows the 5000-foot overview of the setup and the following sections will discuss each piece of the puzzle.
The problem
Suppose you’re working on a new feature, most organizations would rather test it in an internal staging environment before publicly launching it on a production environment. I will call the collection of resources that you want to protect from the public, or even some employees, an internal app.
Traditional solutions
There are different ways to protect an internal app. One involves using a Virtual Private Network (VPN) service like Perimeter 81, and explicitly allowing the VPN IP on your internal app’s ingress. Basically, those you want to grant access will install the VPN client on their devices, connect to it, and the VPN client proxies all connections from their device using a static IP and it is this IP that you allow in your internal firewall.
Traditional VPN solutions work, but they can be expensive, provide less flexibility on how fine-grained you can manage the access. Basically you grant access by allowing the VPN IP; what about granting access based on the IAM group of the user or even the device they’re connecting from? We can do better.
Access control using a VPN and and Ingress allow list
Cloudflare Access Overview
If your organization already uses an edge compute service for caching, CDN or DNS management, chances are that you can also use that edge proxy service to gate access to your internal apps. In this article i’ll be using Cloudflare Access, a solution offered by Cloudflare.
By sitting between the user and your internal app, proxies like Cloudflare can authenticate all incoming requests and either allow or deny requests based on RBAC policies that could either be as simple as an IP Allowlist or as complex as SAML groups pulled from IDPs like Okta. Hence it is more versatile than a simple VPN client.
The illustration below captures the big picture before we dive into the details.
Setting up Cloudflare Access for various use cases.
Cloudflare does many things and Access is their solution for the kind of edge protection we desire. How you setup Access will vary depending on who you want to grant access to. In my experience, I’ve come up with the following structures based on different organizational needs.
Granting CI and third parties access.
Deploying applications using CI/CD is recommended these days. Sometimes a CI step needs to run integration tests that need access to an internal app. You can grant CI workloads access to your internal apps in one of 2 ways. First, if your CI agents have a static IP (eg TeamCity behind NAT), you could add a Bypass Rule to your Cloudflare Access application to allow those IPs access to the application. Cloudflare transparently proxies any traffic that satisfies a Bypass Rule without challenging it for credentials.
However, sometimes your CI agents do not use a known list of static IPs, as is the case with Github-hosted runners. In such cases, you can provision a Service Token in Cloudflare, and use a ServiceAuth Rule to grant that token access to the application. Then you should provide this token to your CI process (preferably as an environment variable) and add it to the headers of all the requests to the internal application. On seeing the token, Cloudflare will let the traffic through.
The same access strategy used for CI can be used for third party services: if they use a known list of static IPs, you can bypass those, otherwise, you could provision Service Tokens and configure them as custom headers in the service.
Using Cloudflare Access with third-party services and CI
Granting QA engineers access.
QA engineers and closed-beta testing groups are focused on using the app as an end user rather than fiddling with HTTP request headers or IP addresses. Furthermore, a team of testers may be geographically dispersed (each using a different IP address) and with varying technical knowledge. So we should use a strategy with minimal friction.
To grant QA engineers access, we can create a SAML group for the QA engineers and pull this into Cloudflare. Then we grant members of this group access to the application using an Allow Rule.
Any QA engineer can then visit the site on their browser and Cloudflare will automatically challenge them to authenticate with the SAML IdP (eg Okta) previously configured. If they successfully authenticate, Cloudflare will set an authorization cookie on their browser such that subsequent requests will be transparently proxied to the internal app.
Using Allow Rules to grant QA Access
Granting Administrators Access.
Administrators often need to perform certain privileged tasks like running a script on their local machine, or triggering a remote job, that deletes or moves data. Such tasks are very sensitive and only a few users should be able to run them. Furthermore, such access may need to be restricted to only a specific time period.
We can satisfy all these requirements by setting up an Allow Rule that grants the admin group access to the app. On the client side, the admin user can use a tool like cloudflared to authenticate with Cloudflare and obtain their access token, which they can then configure as a header on their favourite tool (eg Postman).
Alternatively, we could provision a service token with a short expiration and use a ServiceAuth rule to grant it access to the application. This token can then be handed over to the admin user for them to configure their tool with.
Granting Developers Access.
Developers will be accessing the internal app from their local machines on a daily basis. Sometimes this access is directly through the browser, like in the case of QA, other times, they may be running a local app (like a Next.js frontend app) that needs to access internal Staging APIs.
For these use cases, it is not scalable to provision a service token for each developer — or share one token with all developers. Neither will relying on browser-based cookie auth with Cloudflare work for local apps like Next.js. So we need a different approach.
Cloudflare provides a proxy client called WARP that can be installed locally and it will proxy all the traffic from your local computer to Cloudflare. Once configured, this simplifies the process of granting developers access to internal apps. The setup is as follows:
Setup a Gateway in Cloudflare and use a Bypass Rule to allow traffic from that Gateway to access the internal app. You can combine this Gateway Bypass Rule with an Allow Rule that requires that the traffic must also be from a user in a certain SAML group.
Install the WARP client in the developer machine and have the developer authenticate the client to Cloudflare once. Henceforth, when the WARP client is enabled, all traffic from the local machine to a Cloudflare-proxied domain, will be handled by the proxy client. Behind the scenes the proxy client decorates the request with the authentication claims of the user and sends it to Cloudflare. Cloudflare then decides to allow or deny the traffic based on the configured access rules.
Rather than requiring each developer to manually install the WARP client, an IT team can automatically push it to all developer machines using a tool like JAMF.
Proxy-based access controls like Cloudflare work by examining traffic that passes through them. So, if an attacker can route traffic around the proxy, they have effectively circumvented all access control.
This can happen if you run your internal apps in a cluster with a public load balancer IP. If the attacker can discover this public IP, they can hit the cluster directly without going through Cloudflare.
This may surprise some Cloudflare users because they know that if you manage your domains with Cloudflare and set them to proxy mode, then Cloudflare will resolve DNS queries to Cloudflare edge IPs, not your origin IPs. So, this gives a false sense of security that attackers cannot discover your origin IPs and therefore circumvent Cloudflare protection; but there are ways around that — a slight misconfiguration is all it takes.
So, in a future article, I’ll explore ways to eliminate this threat by setting up your clusters to be completely private and only accept ingress through dedicated Cloudflare-to-origin connections using Argo Tunnels.
Conclusion
In this article, I’ve presented the various challenges of granting access to internal services and how Cloudflare Access can be used to solve some of them. I also delved deeper into the various scenarios of using Cloudflare Access with automated tools, QA engineers, administrators, and developers.
I have avoided giving a tutorial style step-by-step instruction on how to setup this mechanism because they a subject to changing UI, I defer to the Cloudflare docs for that. Instead I have focused on giving the Infrastructure engineer an overview of all the various pieces of the puzzle, and trust their knowledge to source and assemble the parts they need.
Although protecting internal apps is not a trivial pursuit, services like Cloudflare can help simplify that for the Infrastructure engineer.
Squadcast is an incident management tool that’s purpose-built for SRE. Your team can get rid of unwanted alerts, receive relevant notifications, work in collaboration using the virtual incident war rooms, and use automated tools like runbooks to eliminate toil.
What you should do now
Schedule a demo with Squadcast to learn about the platform, answer your questions, and evaluate if Squadcast is the right fit for you.
Curious about how Squadcast can assist you in implementing SRE best practices? Discover the platform's capabilities through our Interactive Demo.
Typically, an infrastructure is made up of numerous critical services which should not be exposed to everyone. In this blog by Uzziah, learn how Cloudflare Access enables you to protect internal services that you’d rather not expose to everyone.
5000-foot view of using Cloudflare Access to protect internal applications.
In this piece, I’ll present my findings on using Cloudflare to protect internal services that you’d rather not expose to everyone. The illustration above shows the 5000-foot overview of the setup and the following sections will discuss each piece of the puzzle.
The problem
Suppose you’re working on a new feature, most organizations would rather test it in an internal staging environment before publicly launching it on a production environment. I will call the collection of resources that you want to protect from the public, or even some employees, an internal app.
Traditional solutions
There are different ways to protect an internal app. One involves using a Virtual Private Network (VPN) service like Perimeter 81, and explicitly allowing the VPN IP on your internal app’s ingress. Basically, those you want to grant access will install the VPN client on their devices, connect to it, and the VPN client proxies all connections from their device using a static IP and it is this IP that you allow in your internal firewall.
Traditional VPN solutions work, but they can be expensive, provide less flexibility on how fine-grained you can manage the access. Basically you grant access by allowing the VPN IP; what about granting access based on the IAM group of the user or even the device they’re connecting from? We can do better.
Access control using a VPN and and Ingress allow list
Cloudflare Access Overview
If your organization already uses an edge compute service for caching, CDN or DNS management, chances are that you can also use that edge proxy service to gate access to your internal apps. In this article i’ll be using Cloudflare Access, a solution offered by Cloudflare.
By sitting between the user and your internal app, proxies like Cloudflare can authenticate all incoming requests and either allow or deny requests based on RBAC policies that could either be as simple as an IP Allowlist or as complex as SAML groups pulled from IDPs like Okta. Hence it is more versatile than a simple VPN client.
The illustration below captures the big picture before we dive into the details.
Setting up Cloudflare Access for various use cases.
Cloudflare does many things and Access is their solution for the kind of edge protection we desire. How you setup Access will vary depending on who you want to grant access to. In my experience, I’ve come up with the following structures based on different organizational needs.
Granting CI and third parties access.
Deploying applications using CI/CD is recommended these days. Sometimes a CI step needs to run integration tests that need access to an internal app. You can grant CI workloads access to your internal apps in one of 2 ways. First, if your CI agents have a static IP (eg TeamCity behind NAT), you could add a Bypass Rule to your Cloudflare Access application to allow those IPs access to the application. Cloudflare transparently proxies any traffic that satisfies a Bypass Rule without challenging it for credentials.
However, sometimes your CI agents do not use a known list of static IPs, as is the case with Github-hosted runners. In such cases, you can provision a Service Token in Cloudflare, and use a ServiceAuth Rule to grant that token access to the application. Then you should provide this token to your CI process (preferably as an environment variable) and add it to the headers of all the requests to the internal application. On seeing the token, Cloudflare will let the traffic through.
The same access strategy used for CI can be used for third party services: if they use a known list of static IPs, you can bypass those, otherwise, you could provision Service Tokens and configure them as custom headers in the service.
Using Cloudflare Access with third-party services and CI
Granting QA engineers access.
QA engineers and closed-beta testing groups are focused on using the app as an end user rather than fiddling with HTTP request headers or IP addresses. Furthermore, a team of testers may be geographically dispersed (each using a different IP address) and with varying technical knowledge. So we should use a strategy with minimal friction.
To grant QA engineers access, we can create a SAML group for the QA engineers and pull this into Cloudflare. Then we grant members of this group access to the application using an Allow Rule.
Any QA engineer can then visit the site on their browser and Cloudflare will automatically challenge them to authenticate with the SAML IdP (eg Okta) previously configured. If they successfully authenticate, Cloudflare will set an authorization cookie on their browser such that subsequent requests will be transparently proxied to the internal app.
Using Allow Rules to grant QA Access
Granting Administrators Access.
Administrators often need to perform certain privileged tasks like running a script on their local machine, or triggering a remote job, that deletes or moves data. Such tasks are very sensitive and only a few users should be able to run them. Furthermore, such access may need to be restricted to only a specific time period.
We can satisfy all these requirements by setting up an Allow Rule that grants the admin group access to the app. On the client side, the admin user can use a tool like cloudflared to authenticate with Cloudflare and obtain their access token, which they can then configure as a header on their favourite tool (eg Postman).
Alternatively, we could provision a service token with a short expiration and use a ServiceAuth rule to grant it access to the application. This token can then be handed over to the admin user for them to configure their tool with.
Granting Developers Access.
Developers will be accessing the internal app from their local machines on a daily basis. Sometimes this access is directly through the browser, like in the case of QA, other times, they may be running a local app (like a Next.js frontend app) that needs to access internal Staging APIs.
For these use cases, it is not scalable to provision a service token for each developer — or share one token with all developers. Neither will relying on browser-based cookie auth with Cloudflare work for local apps like Next.js. So we need a different approach.
Cloudflare provides a proxy client called WARP that can be installed locally and it will proxy all the traffic from your local computer to Cloudflare. Once configured, this simplifies the process of granting developers access to internal apps. The setup is as follows:
Setup a Gateway in Cloudflare and use a Bypass Rule to allow traffic from that Gateway to access the internal app. You can combine this Gateway Bypass Rule with an Allow Rule that requires that the traffic must also be from a user in a certain SAML group.
Install the WARP client in the developer machine and have the developer authenticate the client to Cloudflare once. Henceforth, when the WARP client is enabled, all traffic from the local machine to a Cloudflare-proxied domain, will be handled by the proxy client. Behind the scenes the proxy client decorates the request with the authentication claims of the user and sends it to Cloudflare. Cloudflare then decides to allow or deny the traffic based on the configured access rules.
Rather than requiring each developer to manually install the WARP client, an IT team can automatically push it to all developer machines using a tool like JAMF.
Proxy-based access controls like Cloudflare work by examining traffic that passes through them. So, if an attacker can route traffic around the proxy, they have effectively circumvented all access control.
This can happen if you run your internal apps in a cluster with a public load balancer IP. If the attacker can discover this public IP, they can hit the cluster directly without going through Cloudflare.
This may surprise some Cloudflare users because they know that if you manage your domains with Cloudflare and set them to proxy mode, then Cloudflare will resolve DNS queries to Cloudflare edge IPs, not your origin IPs. So, this gives a false sense of security that attackers cannot discover your origin IPs and therefore circumvent Cloudflare protection; but there are ways around that — a slight misconfiguration is all it takes.
So, in a future article, I’ll explore ways to eliminate this threat by setting up your clusters to be completely private and only accept ingress through dedicated Cloudflare-to-origin connections using Argo Tunnels.
Conclusion
In this article, I’ve presented the various challenges of granting access to internal services and how Cloudflare Access can be used to solve some of them. I also delved deeper into the various scenarios of using Cloudflare Access with automated tools, QA engineers, administrators, and developers.
I have avoided giving a tutorial style step-by-step instruction on how to setup this mechanism because they a subject to changing UI, I defer to the Cloudflare docs for that. Instead I have focused on giving the Infrastructure engineer an overview of all the various pieces of the puzzle, and trust their knowledge to source and assemble the parts they need.
Although protecting internal apps is not a trivial pursuit, services like Cloudflare can help simplify that for the Infrastructure engineer.
Squadcast is an incident management tool that’s purpose-built for SRE. Your team can get rid of unwanted alerts, receive relevant notifications, work in collaboration using the virtual incident war rooms, and use automated tools like runbooks to eliminate toil.
What you should do now
Schedule a demo with Squadcast to learn about the platform, answer your questions, and evaluate if Squadcast is the right fit for you.
Curious about how Squadcast can assist you in implementing SRE best practices? Discover the platform's capabilities through our Interactive Demo.
