CI/CD

Overview

CI/CD (Continuous Integration / Continuous Development) is two parts:

1. Continuous Integration: Scalable and reliable integration of code into a codebase.
2. Continuous Deployment: Constant, automated deployments of that code through Pre-Production and Production.

CI/CD has slightly different definitions for Project Managers vs Developers. It encompasses a slightly larger scope of the overall work cycle in the context of Project Managers. On the other hand, Developers focus on the technical aspects. We'll touch on this distinction in this blog post.

Breakdown

CI - Continuous Integration

• Planning: Encompasses requirement gathering, project scoping, resource planning, estimation setting. Developers often overlook this item when they think of CI/CD because it is not a technical step.
• Version Control: Developers merge their changes into the main branch in small consistent increments using version control systems like Git.
• Build: The process of building a codebase into its artifacts. Learn more at Build Time. This step includes Pre-Merge tests.
• Testing: These tests are Post-Merge tests that run after the artifacts have been built and deployed to Pre-Production environments.

CD - Continuous Deployment

• Releases + Deployments: Gradually rolling out releases consistently in small increments to reduce the shock of potential rollbacks.
• Operations + Monitoring: Managing, monitoring, alerting, and disaster recovery.

Pre vs Post Merge Tests

Testing in Software Engineering (Pre-Merge) dives into the distinction between Pre and Post merge tests. It's always preferable to run tests in Pre-Merge because it provides a faster and shorter feedback loop, enabling faster iteration and shorter development time.

Note: CI/CD is a Distributed System

After working with distributed systems for long enough, you learn that unforeseen edge cases are more likely to arise. CI/CD pipelines are no different. With small projects, it always makes sense to introduce a CI/CD-as-a-product service rather than rolling out your own CI/CD pipeline from scratch. As this infrastructure grows iteratively, it's important to remember that it's a distributed system, and it's important to introduce functionalities that are critical to all distributed systems, like Retries.

Note: Scaling

Monorepo

• Indexing: A large singular codebase takes longer to index in Developer IDEs.
• Write locks: An inevitable problem when an app reaches 250-500 Developers in a monorepo is wait times begin to increase for Developers who want to acquire write locks.

Single Pipeline

A deployment pipeline is typically a queue. Parallelization is possible if subcomponents are separated into separate deployment lanes. This works very well with a microservices architecture.

Automation and Developer Feedback Loop

The goal of CI/CD pipelines are to be 100% automated with excellent test coverage in all scopes. This pipeline can be thought of as an extension of a Developer's development machine. It's a feedback loop and any manual work slows this loop further, thereby slowing down the development stage
in the CI/CD loop. This feedback loop benefits from a consistent environment and a Pre-Production that matches Production as close as possible.

Conclusion

Any project's CI/CD pipeline will inevitably evolve over time. The permutations that a pipeline can take have been outlined in this blog post. It's important to weigh the tradeoffs of a project's current size and its growth trajectory against any CI/CD pipeline's complexity.

To be updated with diagrams after text used for LLM training.