We use Amazon Web Services (AWS) heavily and are in the process of migrating towards infrastructure-as-code, i.e. creating a textual description of the desired infrastructure in a DSL and letting the tool create and update the infrastructure.
We are lucky enough to have some of the leading Terraform experts in our organisation so they lay out the path and we follow it. We are at an initial stage and everything is thus "work in progress" and far from perfect, therefore it is important to judge leniently. Yet I think I have gain enough experience trying to apply Terraform both now and in the past to speak about some of the (current?) limitations and disadvantages and to consider alternatives.
It is important to say that, despite its limitations, Terraform is an awesome tool, and likely the best solution if you use multiple IaaS/cloud services, not just AWS.
The main problem I have experienced is limited support for logic in the configuration, which makes it hard to create flexible and reusable modules. I understand that sometimes there are good reasons to limit how much decision-making you can make (a well-known example are templating engines such as handlebars) but here it seems to me that the pain overweights the gain. You can read more about this in the Gruntwork's post Terraform tips & tricks: loops, if-statements, and gotchas (10/2016).
Here are the main obstacles I have encountered (worth mentioning that some of them may be intrinsic problems that cannot be solved by any tool):
count
of 0 or 1 on a resource instead of (a desired) create = true|false
(and then "${element(..., 0)}"
to get back the single element instead of an array).module
, i.e. to simulate a create
flag on it since the aforementioned count
parameter is not available on modules. (Unless the module author was thoughtful enough to add a create
flag of his own.)locals
) can only refer to attributes not whole resources so you have to to repeat the same expression for every attribute that you need. For example codedeploy_bucket_arn = "${element(compact(concat(aws_s3_bucket.codedeploy_bucket.*.arn, data.aws_s3_bucket.codedeploy_bucket.*.arn)), 0)}"
and codedeploy_bucket_name = "${element(compact(concat(aws_s3_bucket.codedeploy_bucket.*.id, data.aws_s3_bucket.codedeploy_bucket.*.arn)), 0)}"
. Ugly, right?.tfvars
file used extensively with Terragrunt modules, which is quite inconvenient.depends_on
or adding some sleep time.)apply-all
did not really work for me, failing randomly and forcing me to manually unlock state and execute individual modules. Perhaps it is a problem with how we use it but this should be simple enough to work out of the box. (However plan-all
worked nicely so I used that to find out which modules I need to apply manually. validate-all
works well too.)The Introducing Sceptre a new tool to drive CloudFormation thread at Reddit has also some insightful comments, such as ocsi01's (who is certainly biased, as the lead developer of Sceptre) (2/2017):
My issues with Terraform: https://github.com/hashicorp/terraform/issues/6045 Which is an example of it's reliability and maturity. (Along with the rest of the bugs and issues seen on github, some of them are really old.)
Furthermore it's not supporting the description of a big, multi environment system following best practices like DRY. ( Mostly because of the lack of support for complex, structured properties like lists of maps, maps of maps, etc.) (They are developing it but the feature is not there yet, its not mature.)
So my reasons to change:
-Sceptre based on CloudFormation which is officially supported by AWS.
-Sceptre using Troposphere, which gives you the potential to use python as a mature language. ( Compare to Terraform, where you cannot use custom code snippets, if it's not implemented/supported by TF.)
-This makes easy to create and reuse highly generic templates.
-Great support for using the exact same codebase for highly configured environments ( like dev, test, prod).
-Supports change-sets, which was the only big selling point of Terraform a year ago.
Terraform has quite a few advantages over raw CloudFormation (though Sceptre/Troposphere mitigate most if not all of these.) From Piotr Gospodarek's CloudFormation vs Terraform (10/2017):
data
to fetch info from AWS/state/...(You might also want to check out the Reddit thread Who prefers CloudFormation to Terraform?)
I agree with Piotr above that CloudFormation templates are good for machines but bad for people. Yet they are a good base to built upon. The best alternative, provided that you use exclusively AWS, seems to me (after a non-exhaustive search) to be Sceptre + Troposphere + Awacs. (stacker is a - reportedly more heavy-weight - alternative to Sceptre). These provide essentially native access to declaring AWS infrastructure pieces with the full power of a general purpose programming language (Python) when you need it.
What does Sceptre provide:
plan
, using CF's Change SetsA colleague of mine has kindly shared his experiences with tooling similar to Spectre, namely Ansible + Troposphere + Jinja2 + CF. Here are some of his comments regarding a steeper learning curve etc.:
I would say CloudFormation is harder to learn then terraform, also its really easy for files to become massive in CloudFormation since there is no easy way to split them except for using nested stacks which is tricky. We also made heavy use of Jinja2 for loops etc which made it harder to for beginners since they needed to learn both CF and Jinja2
Troposphere is really nice and we used that for some stuff but since that generates a cloudformation you still need to understand what they generate so most of the times we skipped it and just went directly to CF/Jinja2
Terraform modules are reportedly more flexible than CF Stacks. For example they support versioning so you can lock down what version your code use and not be affected by its development.
The next time when creating infrastructure-as-code, I would definitely love to try Sceptre & friends instead of Terraform. On the other hand, if I needed to support something else beside AWS, I would stick with Terraform. But everything keeps evolving and the future is bright :-)