You would use Dagster when you need a highly customizable and testable data orchestration framework, especially if your pipeline requires complex dependencies and data validation. A specific scenario would be in a machine learning project where different preprocessing steps depend on various input types and where validating data at different stages is critical to ensure model performance.
Flyte is particularly useful when you need to build complex workflows that require dynamic computation and seamless scaling with Kubernetes. For instance, if you're developing a data-processing pipeline that includes variable data inputs and requires orchestrating hundreds of tasks with different dependencies, Flyte's dynamic workflow capabilities would allow for more efficient resource management and easier scaling compared to Dagster.