Streamline Security Group Maintenance in Terraform with Markdown Tables

Managing security groups and their rules in AWS infrastructure can become cumbersome and error-prone as your environment grows. In this blog post, I introduce a Terraform solution that simplifies this process by using a Markdown table to define security group rules.

In a previous blog post, I presented a Python tool that automates the creation of AWS architecture diagrams using the Terraform State File. These tools can work together, allowing you to generate both the code and a diagram using a communication matrix defined in Markdown.

Security groups in AWS are crucial for managing access to resources within a VPC. As the number of security groups and their associated rules increase, maintaining them can be challenging. Manual updates can lead to human errors, resulting in misconfigurations, security vulnerabilities, or broken services. Moreover, I personally like working with communication matrices during the design phase, and this tool can efficiently translate that design into code.

The Solution

You can find the code here. Although the code is surprisingly concise, parsing the file is quite complex. It is essential to pay close attention to maintaining the file correctly, as errors can easily occur. Terraform provides limited options for validating the content of the file.


Consider the following example of a Markdown table that defines security group rules:

| Security Groups / inbound from => | frontend                                                                                      | backend                                                | database | ssm(/security_groups/external_server)                                                                   | cidr(                         |
| frontend                          |                                                                                               |                                                        |          | [{"port": 80, "description": "external server"}, {"port": 443, "description": "external server https"}] | [{"port": 443, "description": "public"}] |
| backend                           | [{"port": 80, "description": "from frontend"}, {"port": 443, "description": "from frontend"}] |                                                        |          |                                                                                                         |                                          |
| database                          |                                                                                               | [{"port": 1433, "description": "database connection"}] |          |                                                                                                         |                                          |

Each row represents a new security group, while the columns denote source security groups or CIDR ranges. The rules are defined within the cell intersecting the respective sources and targets. To allow for greater flexibility, the rules are defined as JSON lists.

Notice how security group IDs can be read from the SSM Parameter Store, which is useful when chaining together security groups not defined in the same Terraform project.

Terraform Module Call

To use this specific solution as a Terraform module, follow these simple steps:

  1. Add the following code snippet to your Terraform configuration file:
  module "security_group_matrix" {
  source             = "/path/to/module"
  vpc_id             = "vpc-xxxxxxxx"
  markdown_file_path = "./path/to/your/"
  1. Replace the vpc_id value with your VPC ID and markdown_file_path with the path to your Markdown file containing the communication matrix.

  2. Run terraform init to initialize the module, followed by terraform apply to create the security groups and rules based on the Markdown table.

  3. To visualize the result, run the tool mentioned here. Remember to run terraform refresh beforehand to ensure all needed data is in the state file.

Created Diagram


I hope you like it, and it proves useful to you! I am looking forward to hearing about it.

Similar Posts You Might Enjoy

Visualize AWS Security Groups and Rules from Terraform State

In an ever-changing AWS environment, maintaining manually created architecture diagrams can be challenging. For this reason, I decided to find an automated solution for generating diagrams based on the Terraform State File. In this blog post, I will introduce a tool that simplifies the maintenance of architecture diagrams, ensuring their accuracy and providing a clear understanding of AWS security groups and their interactions. - by Fabian Brakowski

Using AI to generate Terraform Code from actual AWS resources

The world is changing, with new AI tools emerging every day. One such tool that has been making waves recently is ChatGPT. It is only the first of many such tools hitting the market, and it urges us to think about the future of our work. I recently used it to help with a standard task that I often perform and was amazed by how well it helped me to automate it. - by Fabian Brakowski

Streamlined Kafka Schema Evolution in AWS using MSK and the Glue Schema Registry

In today’s data-driven world, effective data management is crucial for organizations aiming to make well-informed, data-driven decisions. As the importance of data continues to grow, so does the significance of robust data management practices. This includes the processes of ingesting, storing, organizing, and maintaining the data generated and collected by an organization. Within the realm of data management, schema evolution stands out as one of the most critical aspects. Businesses evolve over time, leading to changes in data and, consequently, changes in corresponding schemas. Even though a schema may be initially defined for your data, evolving business requirements inevitably demand schema modifications. Yet, modifying data structures is no straightforward task, especially when dealing with distributed systems and teams. It’s essential that downstream consumers of the data can seamlessly adapt to new schemas. Coordinating these changes becomes a critical challenge to minimize downtime and prevent production issues. Neglecting robust data management and schema evolution strategies can result in service disruptions, breaking data pipelines, and incurring significant future costs. In the context of Apache Kafka, schema evolution is managed through a schema registry. As producers share data with consumers via Kafka, the schema is stored in this registry. The Schema Registry enhances the reliability, flexibility, and scalability of systems and applications by providing a standardized approach to manage and validate schemas used by both producers and consumers. This blog post will walk you through the steps of utilizing Amazon MSK in combination with AWS Glue Schema Registry and Terraform to build a cross-account streaming pipeline for Kafka, complete with built-in schema evolution. This approach provides a comprehensive solution to address your dynamic and evolving data requirements. - by Hendrik Hagen