Senior Data Engineer

Key Responsibilities

• Provide L2 production support for AWS-based enterprise data platforms, ensuring high availability and reliability.
• Monitor and troubleshoot Big Data technologies including EMR/Spark/Hadoop, Kafka, NiFi, EKS, Aurora PostgreSQL, DocumentDB, Couchbase, and S3.
• Resolve production incidents, application failures, performance issues, and infrastructure problems within SLAs.
• Conduct root cause analysis and implement permanent fixes for recurring issues.
• Monitor and support batch and streaming data pipelines, including Oozie workflows and scheduling dependencies.
• Troubleshoot Kafka producers, consumers, topics, partitions, message processing, and consumer lag.
• Support Amazon EKS workloads by monitoring pods, nodes, deployments, and application health.
• Monitor EMR clusters, Spark jobs, resource utilization, auto-scaling, and performance.
• Perform database support for Aurora PostgreSQL, DocumentDB, and Couchbase, including query and performance analysis.
• Execute deployments, configuration changes, and release validation using Jenkins and Harness.
• Maintain monitoring dashboards, alerts, and health checks with Dynatrace and CloudWatch.
• Perform capacity planning and performance analysis while identifying opportunities for infrastructure optimization and cloud cost reduction.
• Collaborate with SRE, Infrastructure, Development, Product, and Business teams during incidents, maintenance, and releases.
• Participate in on-call support, disaster recovery exercises, production releases, and major incident management.
• Identify and implement automation to reduce manual effort and improve platform reliability.
• Follow operational procedures, security standards, compliance requirements, and change management processes.