Hasura + Inigo: How to Prevent a Denial of Service Attack

Overview

Hasura is a great way to turn your database into a GraphQL API, and the community version of Hasura can be freely hosted on your infrastructure. Inigo now offers an integration of Inigo built directly into the Hasura community version, allowing you to observe and secure your GraphQL API running on Hasura.

In this article, you will learn how to install Inigo with Hasura and implement controls against a denial-of-service (DoS) attack against your Hasura instance. As you will see, Hasura, running on its own is vulnerable to various attack vectors that could make Hasura or your database unavailable during a DoS attack.

Getting Started with Hasura + Inigo

We have an installation guide available for Hasura + Inigo, and the simplest approach is to use Inigo’s rebuilt container image for Hasura and run it in Docker or Kubernetes. Please follow the instructions in the installation guide so you can implement the DoS attack scenario laid out in this article.

Demo Scenario Setup

To demonstrate a DoS attack scenario against Hasura, a demo scenario needs to be set up. For the demo, four tables are created in a PostgreSQL database, for which Hasura will generate a GraphQL API. The database schema setup SQL can be easily executed directly in Hasura to create the actual database tables as shown in Figure 1.

Figure 1: Database Schema Setup in Hasura Console

Setup Table/GraphQL Relationships

Now that the database tables are created, it’s possible to define the Relationships that establish the mapping of the tables for generating the GraphQL schema. These mappings make it possible to nest the author in the article, for example.

In Figures 2-4, you can see how the Relationships are set up for article, author, and article_tag for the purposes of this demo. Please note that there is a bidirectional Relationship between the article and author, as this will be important for creating a vulnerability as demonstrated in this article.

Figure 2: article relationships setup (tags, author) in Hasura Console

Figure 3: author relationship setup (articles) in Hasura Console

Figure 4: article_tag relationship setup (tag) in Hasura Console

Insert Sample Data with a GraphQL Mutation

Now that the tables and the Relationships are set up, it’s easy to populate the tables with a single GraphQL mutation that contains an article, author, and a set of tags associated with the article. The mutation can be run directly from the Hasura console in GraphiQL as shown in Figure 5.

Figure 5: Insert Sample Data in GraphiQL in Hasura Console Run Sample GraphQL Queries

In Figures 6 and 7, you can see the bidirectional nature of article and author where you can query from either direction with these sample queries. This means that for every article, you can query the author, and for every author you can query the associated articles.

Figure 6: Sample Query for article with the author and tags in GraphiQL in Hasura Console

Figure 7: Sample Query for author with the articles and tags in GraphiQL in Hasura Console

In Inigo, you can see the GraphQL query execution times under the Latency column for these two queries in Figure 8. Each query has a latency of about 200ms, respectively, on the first query run (no data is cached). Additional execution times will be lower as PostgreSQL will cache data.

Figure 8: Latencies for the two sample queries in Inigo Analytics

Now, the setup of the demo scenario is complete, and we can look at how to launch a denial of service (DoS) attack against Hasura!

DoS Attack Scenario #1: Flood with GraphQL Queries

The Hasura community version does not have rate-limiting features built into it. This means your Hasura instance will be vulnerable to bad actors flooding it with GraphQL queries, bringing Hasura and the database to a grinding halt.

To simulate this scenario, we will use the k6 load testing tool with the JavaScript code with a GraphQL query as shown in Figure 9.

Figure 9: Simple Load Test Script for k6

Using this k6 load test script, it can be run with the following command that will run with 2000 virtual users (vus):

k6 run --vus 2000 --duration 30s basic-query.js

After running this load test (ignore any connection errors from k6), you can then go to Inigo Analytics (app.inigo.io) to observe that these queries are taking over 5 seconds to execute, as shown in Figure 10. If you try to run queries from GraphiQL while the load test is running, you will see that Hasura has become nearly unresponsive as it’s flooded with GraphQL queries.

Figure 10: High latencies when Hasura is flooded with queries in Inigo Analytics

DoS Attack Scenario #2: Deeply Nested Query

While Scenario #1 slowed down Hasura to be nearly unusable, Scenario #2 made it completely unresponsive. The key is to have an extremely nested query by abusing the bidirectional relationship between article and author. This relationship allows an infinite depth to a nefarious query with a snippet shown in Figure 11.

Figure 11: Snippet of the deeply nested nefarious query

In this scenario, a nefarious query has a depth of 37 and a height of 118. A single query execution can take well over 1 second, as shown in Figure 12.

Figure 12: Single execution of the deeply nested nefarious query in Inigo Analytics

For this Scenario #2, we are going to leverage k6 again with the deeply nested query in a script, but this time with only 10 vus:

k6 run --vus 10 --duration 30s nested-query.js

When running this load test, you will observe Hasura becoming completely unresponsive, as no queries will execute from GraphiQL. Additionally, no query metrics are being sent to Inigo as Hasura cannot execute any queries, period. Once the load test ends, Hasura will eventually become operational again.

As you can see, while Scenario #1 was crippling to Hasura, Scenario #2 is even worse. Additionally, when Scenario #2 happens, it may trigger service health checks to fail and Hasura to be restarted, only to succumb to an ongoing DoS attack again.

DoS Attack Scenario #1 Mitigation: Inigo Rate Limiting

To mitigate Scenario #1, Rate Limiting can be implemented with Inigo where the Inigo middleware running inside of Hasura will apply rate-limiting controls to all incoming GraphQL query requests.

The first step is implementing an anonymous_profile with your Inigo Service as shown in Figure 13. This will be used to apply RateLimit to all unauthenticated incoming requests as shown in Figure 14.

Figure 13: Inigo Service with anonymous_profile

Figure 14: Inigo RateLimit for the anonymous profile

You can use the inigo CLI to apply these configurations:

inigo apply -f service.yml

inigo apply -f rate-limit.yml

As shown in Figure 14, a single anonymous client (tracked by IP Address) will only be able to issue 10 queries every 10 seconds. If you are using GraphiQL, it will be easy to prove the rate-limiting works by issuing 11 GraphQL query requests within 10 seconds, as shown in Figure 15.

Figure 15: Inigo RateLimit exceeded as shown in GraphiQL in Hasura Console

DoS Attack Scenario #2 Mitigation: Inigo Max Height and Depth Limits

By using Inigo’s Security configuration, you can apply max_depth and max_height restrictions to any incoming queries. This effectively prevents any nefarious attacks with deeply nested queries as demonstrated in Scenario #2. A sample Security configuration is shown in Figure 16.

Figure 16: Inigo Security configuration with max_depth and max_height

The configuration as shown in Figure 16 can be applied as such:

inigo apply -f security.yml

Using GraphiQL in Hasura, you can attempt to run a deeply nested query, which will fail as shown in Figure 17.

Figure 17: Inigo max_depth and max_height exceeded as shown in GraphiQL in Hasura Console

If we rerun the k6 load test for Scenario #2 again, you will see that there are over 2000 blocked queries that result from the load test, as shown in Figure 18. To run the load test, here is the command once again:

k6 run --vus 10 --duration 30s nested-query.js

Figure 18: Inigo Analytics showing the Blocked queries after running the k6 load test

Conclusion

The community version of Hasura is very powerful and feature rich, but is also vulnerable to DoS attacks as it does not incorporate rate limiting or query protection. While nefarious attacks are possible, even inside of your internal network, unintentional internal DoS scenarios are also possible that can make your Hasura deployment and database unavailable. In other words, you should think twice about deploying Hasura to production without rate limiting and query protection under any circumstances.

Inigo offers a drop-in replacement container image for the community version of Hasura that you can deploy with Docker or Kubernetes. All you need to do is sign up for Inigo, create a basic Inigo service, and deploy Hasura with the Inigo token. You will immediately start receiving query analytics from Inigo, and can then further explore implementing controls for query protection and rate limiting as documented in this article.

For further information, please read our Hasura documentation and examples for implementing query protection and rate limiting with Hasura.