Engineer

Java Nifi

Technical Skills:

Strong NiFi Expertise: Deep understanding of NiFi architecture, processors, controllers, and data flow management capabilities.

Programming Languages: Proficiency in Java, Spring boot & API Microservices.

Database and SQL: Familiarity with databases, SQL, and data manipulation techniques.

Big Data Technologies: Knowledge of big data technologies like Kafka ( Event driven & streaming architecture)

Airflow – Workflow orchestration

Responsibilities :

Design and Implementation:

Creating and implementing data flow solutions using NiFi, including defining data sources, destinations, and transformation processes.

Ingestion and Transformation:

Managing data ingestion from various sources (e.g., files, databases, APIs) and transforming data into desired formats for downstream consumption.

Routing and Enrichment:

Configuring NiFi to route data to appropriate destinations and enriching data with additional information (e.g., timestamps, location).

Scalability and Performance:

Ensuring data flows are scalable and performant to handle varying data volumes and velocity.

NiFi Cluster Management:

Maintaining and improving NiFi clusters, including configuration, optimization, and troubleshooting.

Microservices Integration:

Developing and integrating microservices to support data conditioning, format validation, and transformation processes.

Troubleshooting and Issue Resolution:

Identifying and resolving issues related to data flows, NiFi configurations, and data integration processes

Airflow Expertise:

Possessing deep knowledge of Airflow's architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.

Workflow Design and Development:

Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.

Performance Optimization:

Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling

Salary Range- $100,000-$120,000 a year

#LI-SP3

#LI-VX1