We are seeking a highly skilled resource to design and implement high-performance, event-driven data pipelines, ensuring low-latency data processing and high availability system for the large credit card processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE, EME, Conduct>It) to build stateful services that ingest, filter, and transform data from sources like Kafka or message queues, pushing updates to dashboards or downstream databases in near-real-t

ime.

Key Responsibi

lities:

Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggre
gations.Implement event-driven data pipelines using Kafka, MQ, and file
streams.Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-once" pro
cessing.Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency perf
ormance.Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting dat
a issuesDevelop ETL pipelines for batch and real-time data ingestion and transfo
rmation.Implement and ensure data validation, data security, integrity, and compliance across big data pl
atforms.Monitor and troubleshoot performance issues in large-scale c
lusters.Collaborate with data scientists, analysts, and application teams to deliver high-quality data so
lutions.Automate workflows and improve operational efficiency using scripting and orchestratio

tools.
Required Skills & Qualif

ications:

Deep understanding of credit card proc
ess systemDeep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta>Environment), Conduct>It, and Continu
ous Flows.Understanding of Kafka, message queues, and real-time stateful
services.Proficiency in Unix/Linux shell scripting, SQL, and database technologies (e.g., Oracle,
Teradata).Experience in Java , Scala, Python or Kafk
a is plus.Familiarity with Linux/Unix environments and shell
scripting.Understanding of data security, governance, and compliance
standards.Experience with cloud-based big data
platformsExposure to containerization (Docker, Kubernetes) for big data
workloads.Knowledge of CI/CD pipelines for data engineering

projects.
Behavio

ral Skills:

Good Communic
ation skills5 days Work from Office at Berkley
Heights, N
JTeam PlayerAbility to work in a changing
environmentStrong problem solving and analy
tical skillsAbility to work independently or w
ithin a teamManage day-to-day challenges and communicate developmental risks with the te

chnical team