During the third day of QCon San Francisco, George Mao, a senior distinguished engineer at Capital One, presented on his company's journey into serverless, the best practices they found, and the lessons learned along the way. The session was part of the "Architecting for the Cloud" track.
Mao began the session by explaining why the Capital One bank started to go "all-in" with the cloud and AWS to gain more agility.
The bank chose serverless and not a traditional architecture with Virtual Machines (AWS EC2 instances), operating systems, networking, containers, application servers, and business applications. Instead, they went for serverless because of its event-based nature, less overhead managing infrastructure (removing the burden for developers), and more focus on creating business logic.
Mao stated that the benefits of serverless were:
- Lower "Run the Engine Cost," which makes developers happy
- Lower AWS Infrastructure cost, which makes the business happy
- Modern apps using a microservices-based architecture allow developers to deploy and scale independently with a small blast radius
Next, Mao started discussing what his company learned by adopting serverless. The journey involved managing many AWS accounts and Lambda functions at scale, which brought about significant challenges related to compliance controls, standards, metrics, logs, maintenance, and security vulnerabilities.
One crucial lesson learned was the importance of creating a "center of excellence (COE)" that works across the organization. This central group helps establish standards, provides technical guidance, and influences decision-making, effectively preventing siloed choices, reducing technical debt, and mitigating risks.
Additionally, Mao emphasizes the significance of a structured learning curve for Lambda development, advocating for local development environments before deploying to AWS and leveraging new tools like AWS SAM (Serverless Application Model) and CI/CD pipelines, which play an essential role in making it easier to develop and deploy serverless applications efficiently.
Furthermore, Mao touches on concurrency management, development standards, account management practices, leveraging common libraries, providing training, cost optimization through lambda tuning, and the importance of correct observability and metrics usage.
At the end of the session, Mao talks about the best practices for serverless in AWS that also apply to other cloud vendor's solutions, for example, Google Cloud Functions and Azure Functions. He discussed things not to do:
And outline the best practices that the Capitol One team learned:
- Optimize memory and use Lambda Power Tuner
- Set CloudWatch Log Retention as logs are not for free
- Use Lambda Powertools
- Use Provisioned Concurrency with Autoscaling
- Understand and monitor metrics
- Use AWS SAM - Build, Emulate, Test, and Debug all locally
- Use the right AWS SDK
- Use Snapstart (Java)
- And set standards - tags, aliases, IAM permissions, etc
Lastly, an attendee asked a question about when not to use Lambda, and Mao responded:
When event sources are not native AWS, because you have to poll, or anytime you have high idle time.