Large EMR Spark jobs run across several accounts, and operators need to search driver and executor logs after failures.
Keep logs after clusters terminate while separating operational logs from audit events and controlling access.
Which logging design is most appropriate?
Send EMR logs to an encrypted central durable location and combine retention, access separation, and Athena or OpenSearch analysis.
Detailed explanation
Metrics show trends but not error lines or execution context.
Metrics show trends but not error lines or execution context.
Logs survive cluster termination and can be searched, correlated, and permissioned across jobs.
Logs survive cluster termination and can be searched, correlated, and permissioned across jobs.
CloudTrail records API activity and does not replace application log details.
CloudTrail records API activity and does not replace application log details.
Cluster termination or node replacement loses the evidence and prevents cross-job investigation.
Cluster termination or node replacement loses the evidence and prevents cross-job investigation.
Check official documentation
Compare the following points with the official documentation.
Amazon EMR logging、S3 server-side encryption、Athena/OpenSearch log analysisの公式説明を確認する。
Official references
What to confirm
ノードローカルログ、CloudTrailイベント、メトリクスと、クラスターをまたいで調査できる統合ログの違いを説明できる。
Key points
- EMR logging
- Central logging
- S3
- Log analysis
Notes
- Environment: AWS公式DEA-C01試験ガイドと各サービスの公式ドキュメントの確認
- Confirm the relevant DEA-C01 Skill and current AWS service documentation.
Foundation review
Retention
Match operational and audit log retention to purpose and policy.
Access separation
Keep read, delete, and audit permissions distinct.