Data AWS Sagemaker

Technical Skills & Expertise
SQL: Expert-level proficiency (must have).
AWS: Redshift, S3, ECS, Lambda, Glue, SQS, SNS, CloudWatch, Step Functions, CDK SQS/SNS, Athena (must have).
PySpark: Expert (must have).
Python: Strong experience with API integrations, data handling, and automation.(must have)
LLM Integration: Experience integrating LLMs via APIs (e.g., OpenAI, Claude, Bedrock) into data workflows or analytics pipelines.
LLMOps: Understanding of prompt design, prompt tuning, RAG patterns, and model evaluation in production.
Data Modeling & Query Tuning: Hands-on experience in designing optimized schemas and writing performant queries.
Big Data Ecosystem: Solid understanding of Hadoop, Hive, MapReduce.
Orchestration Tools: Airflow (Open Source), MWAA on AWS (intermediate- nice to have).
Data Migration: Experience with AWS Data Migration Service (DMS).
Analytical Skills: Strong in Exploratory Data Analysis (EDA).
ETL Design Patterns: Proficiency with window functions, reusable ETL frameworks, and scalable automation.

Preferred Knowledge
Exposure to Data Lake vs. Data Warehouse architecture.
Experience in real-time data ingestion and streaming frameworks.
Hands-on with data quality, compliance, and governance frameworks.
Familiarity with security best practices in AWS environments.
Experience with data enrichment or summarization using LLMs.
Familiarity with RAG pipelines, vector databases (e.g., OpenSearch, Pinecone, FAISS), and metadata extraction using LLMs.

Key Responsibilities
Design & Build Intelligent Data Solutions: Develop and maintain scalable, LLM-enabled data pipelines using AWS services like Glue, Redshift, Lambda, Step Functions, S3, Athena, and ECS.
Integrate LLM APIs or fine-tuned models (e.g., OpenAI, HuggingFace, Amazon Bedrock, SageMaker JumpStart) into existing AWS-based data workflows.
Enable RAG (Retrieval-Augmented Generation) pipelines using AWS services and LLMs for advanced analytics and knowledge management.
ETL Development: Build efficient, reusable ETL frameworks that can incorporate LLMs for data enrichment, summarization, classification, or metadata generation.
Use orchestration tools like Airflow/MWAA or Step Functions to manage both data and LLM workflows.
LLM & AI Services in AWS: Work with Amazon Bedrock, SageMaker, or custom containers to deploy, monitor, and scale LLM-based solutions.
Optimize LLM usage for performance and cost within the AWS ecosystem (e.g., caching responses, throttling, model selection).
Data Security & Integrity: Implement security best practices and compliance standards for handling sensitive data within LLM pipelines.
Ensure proper prompt auditing, logging, and governance for LLM interactions.
Monitoring & Troubleshooting: Continuously monitor pipelines (including LLM calls), troubleshoot failures, and optimize performance and cost.
Documentation: Maintain detailed documentation of data architectures, frameworks, and engineering processes.
Architecture & Reviews: Participate in solution architecture, code reviews, and sign-offs to ensure quality and scalability.
Project & Stakeholder Management: Apply Agile/Scrum methodology for project delivery, manage risks, and effectively communicate with both technical and non-technical stakeholders.