AWS exec: 'Our understanding of open source has started to change'

Interview AWS is wary of vendor-driven open source projects, performs business health checks on all its open source dependencies, and suffered impact on the development of Amazon Linux when CentOS as we all knew it was discontinued, The Register was told at the internet giant's re:Invent conference.

David Nalley is director of open source strategy and marketing at AWS and president of the Apache Software Foundation. “My purview is pretty much anything open source at AWS,” he said.

Is there any benefit in using Amazon Linux versus other distributions for an EC2 (Elastic Compute Cloud) virtual machine? “For a long time, Ubuntu was the primary distribution that people rolled out,” said Nalley. “We work with a number of different distributions making sure that they work well.”

FreeBSD is also an option. “I met with the FreeBSD release manager here, talking about AMI’s [Amazon Machine Images] for FreeBSD. My inkling would be that Amazon Linux does tend to be perhaps better supported, just because the people working on it are also the people inside Amazon,” he told us.

One of the snags with Amazon Linux 2023 (finally available outside AWS) is that EPEL (Extra Packages for Enterprise Linux) is not supported, whereas it was in Amazon Linux 2, the previous version. Why is that, given the importance of many EPEL packages?

“The difference is that Amazon Linux used to be based on CentOS,” said Nalley. “CentOS had a vibrant community of folks who would package lots of software. With more recent versions of Amazon Linux we rebased that, to be based on Fedora.”

This was because CentOS has been discontinued. “So that is a problem. I don’t know that it’s a huge technical blocker, because for most of those, if you have the source rpm, you can easily build it for another distribution. I used to be a Fedora packager.” Occasionally there might be issues, he said, but “for the overwhelming majority of packages that’s all that is needed.”

• You're so worried about AWS reliability, the cloud giant now lets you simulate major outages
• The AI everything show continues at AWS: Generate SQL from text, vector search, and more
• Now AWS gets a ChatGPT-style Copilot: Amazon Q to be your cloud chat assistant
• AWS unveils core-packed Graviton4 and beefier Trainium accelerators for AI

A snag is that if a security patch is needed, the process has to be repeated. And there is no alert that the package needs to be updated? “Correct. The maintenance burden, you bear,” said Nalley.

Why is there not an in-place upgrade from one Amazon Linux to another? “We’ve been focused on providing a stable platform,” said Nalley. “The nature of doing in-place upgrades tends to be destabilizing. You’re upgrading the underlying version of glibc or llvm and keeping that safe across versions is a challenge.”

It is a problem that mainly impacts smaller customers. “Our most well-architected customers are using infrastructure as code to define what all these instances look like, so it’s much easier for them to spin up a new instance, deploy that software that was running on the old instance, see if it works, then migrate the actual workload.”

Why did Linux 2023 take so long to appear – Amazon Linux 2 was quite elderly and out of date, well before its successor was released? “There’s a couple of reasons,” said Nalley.

“We did not want to release something that wasn’t ready. We were doing a fundamental rebase of the underlying distribution. Amazon Linux 2 retained good security patching and support, which is a challenge for something that’s older. There were architectural issues that led to [Linux 2023] coming out later than we desired. We pushed the ship date back a number of times.”

Fedora is a fast-moving distribution, but Amazon Linux the opposite. Does that make it a strange marriage? “It’s certainly a different experience than the one we had with CentOS,” said Nalley.

“One of the things we liked with Fedora was how fast they were driving innovation. We thought Fedora made a better platform for us to build upon, even though we were going to extend support times out well past what Fedora supports. Fedora allowed to move much faster, because by the time a new version of CentOS was out, in the old days, it was already pretty dated.”

AWS and open source

How is the relationship between AWS and the open source community evolving, bearing in mind things like the issues with Elastic, which resulted in the development of OpenSearch?

“Every company that is using open source software gets far more benefit than they give back. That is a universal truth,” said Nalley.

Every company that is using open source software gets far more benefit than they give back

“I do think that our understanding and our relationship with open source is changing. Some of that comes down to understanding how open source has evolved. In days gone by, most open source projects were true community initiatives, whether it was the Apache web server or the Linux kernel, it was people who were coming together for a common cause. For a lot of the innovative things coming out, that has shifted to a more vendor-controlled open source software project. You cannot treat the two the same, or consume those two in the same way and expect similar outcomes.

“Our understanding of open source has started to change, and realising that, we have to measure and assess risk every time we take a dependency. How do we ensure that this open source project continues to be developed? The Apache web server or Apache Tomcat is one thing, whereas other vendor-led packages might take different considerations. It’s about that community diversity and health. We’re actively looking at that as we’re taking on dependencies.”

AWS itself has many open source projects, such as SDKs, which are mainly driven by its own engineers rather than a community. “AWS uses open source for a variety of reasons, when we release software,” said Nalley.

“One of them is that open source licenses are really well understood, from an IP (Intellectual Property) perspective. If you grab something that’s under the Apache License or the MIT license, your company lawyer probably understands that and has those licenses pre-approved. There are a lot of things we release under an open source license with no expectation that a community is going to form.”

There are exceptions, such as Karpenter, an AWS project that has now been contributed to CNCF (Cloud Native Computing Foundation). “Microsoft said, we really like Karpenter,” said Nalley.

“But we don’t like Karpenter if it’s controlled by Amazon. So we had internal conversations about what that would look like, if we moved Karpenter to the CNCF. It’s an incubating project right now, and in the process of growing its community.”

Another AWS open source project that has been taken up by others in what Nalley calls “strange and delightful ways” is Bottlerocket, a lightweight Linux distribution for containers.

Does AWS worry about the poor business model of many open source projects? Nalley sighs. “It is something that every service team looks at, when they’re taking a dependency” he told us.

“Inside AWS we have the concept of strategic open source projects. We require the business owner inside a service team to report on a quarterly basis about the health of those projects. We’re doing that so we know that they’re paying attention to it. We don’t want to learn that this thing that’s really important is maintained by a guy living in a basement on public assistance. That is not acting in the best interests of our customers.”

How does AWS prevent the risk of code-tampering when consuming public open source projects? “Our builder experience team maintains an internal package repository,” said Nalley, looking at things like the source of the software and the license.

“That also allow us to respond rapidly when there’s a security issue because we can look at all the places where we’re consuming it,” he explained. ®