data engineer

data & ai engineer

data science (python) quant analyst Jobs in New York

data science (python) quant analyst

product engineer, data

python & aws data engineer Jobs in New York

python & aws data engineer

freelance data scraping engineer python

TIAA

<div>This Software Engineer will lead the development and maintenance of backend systems for the Brooklyn Investment team's equities portfolio management platform, which manages billions of dollars in client assets. The platform connects strategy generation, portfolio rebalancing, trade execution, and advisor-facing interfaces. The engineer in this role will build and maintain full stack services that integrate with multiple data sources, external APIs, and internal microservices, and will contribute to the design and implementation of distributed system components. The role requires ensuring system reliability, performance, and data integrity across the platform while collaborating with internal and client-side engineering teams and business stakeholders to deliver features that support business operations.
The Software Engineer requires strong proficiency in Python, experience designing and implementing distributed systems, and a solid understanding of API integration patterns and microservices architecture. The candidate must also possess strong interpersonal and communication skills to work effectively with clients and their engineering teams directly. 
Key Responsibilities and Duties<ul><li>
Lead the design, development, and ongoing maintenance of FastAPI-based backend services powering core platform capabilities including portfolio rebalancing, trade management, and advisor workflows</li><li>
Integrate with multiple data stores such as PostgreSQL and MongoDB/DocumentDB and build and optimize data pipelines connecting custodial data, security master databases, and real-time market data feeds</li><li>
Develop and maintain AWS infrastructure using CDK, encompassing ECS Fargate deployments and serverless components</li><li>
Collaborate with internal and client-side frontend engineers and business stakeholders to design and implement RESTful APIs consumed by the Vue/Nuxt advisor portal, with opportunities to contribute frontend code as needed</li><li>
Implement monitoring, alerting, and observability solutions using Datadog APM</li><li>
Write comprehensive unit and integration tests to ensure system reliability and resilience across the platform</li></ul>
Educational Requirements<ul><li>
University (Degree) Preferred</li></ul>
Work Experience<ul><li>
3+ Years Required; 5+ Years Preferred</li></ul> 
Career Level 
7IC
Required Qualifications<ul><li>
Minimum 3+ years of professional Python software engineering experience</li></ul>
Preferred Qualifications<ul><li>
5+ years of professional python experience</li><li>
Asset management, financial services, insurance, or fintech experience</li><li>
Data science or data modeling experience</li><li>
Experience building production REST APIs with FastAPI, Flask, or Django</li><li>
Solid understanding of relational databases (PostgreSQL)</li><li>
Familiarity with data validation libraries (Pydantic) and ORM tools (SQLAlchemy)</li><li>
Experience with Python data libraries (Pandas, NumPy) for data transformation and analysis</li><li>
Experience with observability tools (Datadog, New Relic, or similar APM solutions)
</li></ul>Anticipated Posting End Date:
2026-06-27
Base Pay Range: $143,500/yr - $195,000/yr
Actual base salary may vary based upon, but not limited to, relevant experience, time in role, base salary of internal peers, prior performance, business sector, and geographic location. In addition to base salary, the competitive compensation package may include, depending on the role, participation in an incentive program linked to performance (for example, annual discretionary incentive programs, non-annual sales incentive plans, or other non-annual incentive plans).
_____________________________________________________________________________________________________
Company Overview
Nuveen is a global investment leader, managing public and private assets for clients around the world and on behalf of TIAA, our parent company and one of the world&rsquo;s largest institutional investors. We invest in the growth of businesses, real estate, infrastructure and natural capital, providing clients with the reliability, access and foresight unique to our heritage. Our prevailing perspective on the future drives our ambition to innovate and adapt our business to the changing needs of investors - all to pursue lasting performance for our clients, our communities and our global economy. Visit www.nuveen.com to learn more about us.
Our Culture of Impact
At TIAA, we're on a mission to build on our 100+ year legacy of delivering for our clients while evolving to meet tomorrow's challenges. We equip our associates with future-focused skills and AI tools that enable us to advance our mission. Together, we are fighting to ensure a more secure financial future for all and for generations to come. We are guided by our values: Champion Our People, Be Client Obsessed, Lead with Integrity, Own It, and Win As One. They influence every decision we make and how we work together to serve our clients every day. We thrive in a collaborative in-office environment where teams work across organizational boundaries with shared purpose, accelerating innovation and delivering meaningful results. Our workplace brings together TIAA and Nuveen's entrepreneurial spirit, where we work hard and work together to create lasting impact. Here, every associate can grow through meaningful learning experiences and development pathways&mdash;because when our people succeed, our impact on clients' lives grows stronger.
Benefits and Total Rewards
The organization is committed to making financial well-being possible for its clients, and is equally committed to the well-being of our associates. That&rsquo;s why we offer a comprehensive Total Rewards package designed to make a positive difference in the lives of our associates and their loved ones. Our benefits include a superior retirement program and highly competitive health, wellness and work life offerings that can help you achieve and maintain your best possible physical, emotional and financial well-being. To learn more about your benefits, please review our Benefits Summary.
Equal Opportunity
We are an Equal Opportunity Employer. TIAA does not discriminate against any candidate or employee on the basis of age, race, color, national origin, sex, religion, veteran status, disability, sexual orientation, gender identity, or any other legally protected status.
Our full EEO &amp; Non-Discrimination statement is on our careers home page, and you can read more about your rights and view government notices here.
Accessibility Support
TIAA offers support for those who need assistance with our online application process to provide an equal employment opportunity to all job seekers, including individuals with disabilities.
If you are a U.S. applicant and desire a reasonable accommodation to complete a job application please use one of the below options to contact our accessibility support team:
Phone: (800) 842-2755
Email: accessibility.support@tiaa.org
Drug and Smoking Policy
TIAA maintains a drug-free and smoke/free workplace.</div>

Software Engineer - Models Business

Inherent Technologies

<div>
 
 Job Description
 
 
 
 Seeking a
 
 hands-on AI Engineer/Developer
 
 to rapidly prototype and deliver AI-driven solutions across investment, research, and trading functions. This role is ideal for someone who can bridge
 
 financial domain knowledge with applied AI/ML development
 
 , building proof-of-concepts that can evolve into production-grade tools supporting investment decision-making.
 
 
 
 Key Responsibilities
 
 
 
 Design and develop
 
 AI/ML prototypes
 
 using Python to support investment research, portfolio analytics, and trading insights
 
 
 Build and test
 
 LLM-based and data-driven use cases
 
 (e.g., research summarization, signal generation, document analysis, automation)
 
 
 Work closely with
 
 investment professionals, research teams, and data engineers
 
 to translate business needs into functional AI solutions
 
 
 Develop and integrate
 
 data pipelines, APIs, and analytical models
 
 to support rapid experimentation
 
 
 Leverage tools such as
 
 Jupyter, Pandas, NumPy, LangChain, and cloud AI services (AWS/Azure/OpenAI)
 
 
 
 Perform
 
 data exploration, feature engineering, and model validation
 
 on financial datasets
 
 
 Iterate quickly on prototypes, incorporating feedback to refine models and outputs
 
 
 Ensure solutions consider
 
 data security, governance, and compliance
 
 within a financial services environment
 
 
 
 Required Qualifications
 
 
 
 10+ years of experience in
 
 Python development
 
 with strong hands-on coding ability
 
 
 Experience building
 
 AI/ML or LLM-based applications and prototypes
 
 
 
 Exposure to
 
 financial services, asset management, trading, or investment research
 
 
 
 Strong experience with
 
 data analysis libraries
 
 (Pandas, NumPy) and working with structured/unstructured data
 
 
 Familiarity with
 
 APIs, microservices, and cloud platforms (AWS, Azure, or GCP)
 
 
 
 Ability to work in a
 
 fast-paced, iterative environment
 
 with direct interaction with business stakeholders
 
 
 
 Preferred Qualifications
 
 
 
 Experience with
 
 LLM frameworks
 
 (LangChain, RAG architectures, vector databases)
 
 
 Knowledge of
 
 financial data sets, market data, or portfolio analytics
 
 
 
 Experience prototyping tools for
 
 quant research, credit analysis, or trading workflows
 
 
 
 Familiarity with
 
 Snowflake, Databricks, or modern data platforms
 
 
 
 
 What Success Looks Like
 
 
 
 Rapid delivery of
 
 high-impact AI prototypes
 
 that demonstrate clear business value
 
 
 Strong collaboration with front-office teams to
 
 solve real investment and research problems
 
 
 
 Ability to evolve prototypes into scalable solutions in partnership with engineering teams
</div>

Python Developer/AI Engineer

Shields Group Search

<div>
 
 
 Data Engineer
 
 
 
 
 Hybrid in NYC
 
 
 
 
 Full-Time
 
 
 
 
 $175,000-$225,000 base + upside depending on experience
 
 
 
 
 
 
 
 About the Company
 
 
 
 Shields Group Search is partnering with an innovative private banking and wealth infrastructure company building modern tools for modern clients. Their platform is designed to help private investors access liquidity more easily while driving transparency into historically opaque markets, including fine art and commercial real estate.
 
 
 
 
 
 They are hiring a
 
 Data Engineer
 
 to help build a world-class data infrastructure that supports the modernization of private banking for independent wealth managers and family offices. This person will be responsible for designing, building, and maintaining robust data pipelines, ETL processes, and data warehousing solutions that enable analytics and drive insights across the platform.
 
 
 
 
 
 You will work closely with engineering and product teams to architect scalable, reliable data systems. The stack includes
 
 Python, SQL, cloud data warehousing platforms, and modern data engineering tooling
 
 .
 
 
 
 
 
 
 About the Role
 
 
 
 This position reports directly to the
 
 Head of Engineering
 
 . The ideal candidate will have meaningful educational and/or professional experience designing and implementing scalable data systems, building ETL pipelines, and writing clean, maintainable, well-tested code.
 
 
 
 
 
 This team is looking for someone who is genuinely excited to spend most of their day coding and building scalable systems, while also contributing architectural input. They would strongly prefer someone in
 
 NYC
 
 , and
 
 2-3 days per week in-office
 
 should be expected. They also want someone who is comfortable building with and using
 
 agentic systems
 
 . This is not the right fit for someone optimizing primarily for work/life balance.
 
 
 
 
 
 
 Responsibilities
 
 
 <ul>
 <li>
 Participate in the full software development life cycle from planning to deployment
 </li>
 <li>
 Design and build scalable ETL/ELT pipelines to ingest, transform, and load data from multiple sources
 </li>
 <li>
 Collaborate with the team to design, develop, unit test, and support new features
 </li>
 <li>
 Strive for high quality and rapid output through agile software development best practices
 </li>
 <li>
 Architect data warehouses and data lakes, optimizing for query performance and cost efficiency
 </li>
 <li>
 Perform code reviews and provide constructive feedback to peers
 </li>
 <li>
 Maintain and improve the tech stack
 </li>
 <li>
 Develop flexible and well-structured data pipelines and data warehouse schemas
 </li>
 <li>
 Implement effective security protocols, data protection measures, and storage solutions
 </li>
 <li>
 Troubleshoot, debug, and upgrade software
 </li>
 <li>
 Ensure software meets all requirements of quality, security, and configurability
 </li>
 <li>
 Contribute to documentation, tests, and automation pipelines
 </li>
 </ul>
 
 
 
 
 
 Requirements
 
 
 <ul>
 <li>
 Bachelor&rsquo;s degree in Computer Science, Software Engineering, or a related field
 </li>
 <li>
 Minimum of 3 years of experience in data engineering, ETL development, or analytics engineering, ideally within fintech or financial services
 </li>
 <li>
 Advanced proficiency in Python and SQL for data pipeline development and analysis
 </li>
 <li>
 Familiarity with CI/CD tools
 </li>
 <li>
 Experience with data warehousing platforms such as Snowflake, BigQuery, Redshift, or similar, as well as modern ETL tools
 </li>
 <li>
 Strong knowledge of the software development lifecycle
 </li>
 <li>
 Experience with testing and debugging
 </li>
 <li>
 Proficient understanding of code versioning tools such as Git
 </li>
 <li>
 Excellent problem-solving skills and attention to detail
 </li>
 <li>
 Excellent communication and teamwork skills
 </li>
 <li>
 Ability to work independently and take initiative
 </li>
 <li>
 Willingness and ability to build with agentic systems in a practical, hands-on way
 </li>
 </ul>
 
 
 
 
 
 Why This Role
 
 
 
 This is an opportunity to help architect a state-of-the-art platform and partner with an experienced leadership team during a period of rapid development. The team has additional flexibility on compensation for stronger candidates, with total base salary landing closer to
 
 $200,000-$225,000
 
 for candidates with more experience.
 
</div>

Data Engineer

Traversal

<div><h3 class="jobSectionHeader">About Traversal</h3>
Traversal is the AI Site Reliability Engineer (SRE) for the enterprise&mdash;already trusted by some of the largest companies in the world to troubleshoot, remediate, and even prevent the most complex production incidents. Our mission is to free engineers from endless firefighting and enable them to focus on creative, high-impact work.
Our roots remain deeply embedded in AI research, and we&rsquo;re channeling that scientific rigor and creativity into building the premier AI agent lab for the enterprise. Hence, what we&rsquo;re proudest of is assembling the most talented yet nicest group of individuals, including researchers from MIT, Harvard, and Berkeley, to world-class engineers from industry: Citadel Securities, Cockroach Labs, Datadog, DE Shaw, ServiceNow, Glean, Perplexity, Pinecone, and more, to take on one of the hardest problems for AI to solve. Without the entire team, none of this would be possible.<h3 class="jobSectionHeader">
The Role</h3>
As a Product Engineer at Traversal, you&rsquo;ll play a key role in designing and building core features powered by our AI site reliability engineer. From the user interface to the backend and infrastructure, you&rsquo;ll develop product components to enable real-time incident detection, root cause analysis, and automated remediation in some of the most complex production environments. This role will focus on the ability to work across the full pipeline of our product, and someone who can have a strong product sense while also being comfortable working around AI and data processing infrastructure will be the best fit.<h3 class="jobSectionHeader">
Responsibilities</h3><ul><li>
Product Impact: Translate AI-powered insights into user-friendly features that empower engineers and reduce cognitive load.</li><li>
User Experience: Innovate and drive the evolution of UX/UI design, rapidly iterating based on user feedback to improve overall usability.</li><li>
Agent Development: Build AI agents with a mind for user-experience and capabilities like long-horizon work, proactiveness, readability, and evidence citation.</li><li>
API Development: Design and implement high-performance APIs to enable seamless communication between backend systems and frontend interfaces.</li><li>
Data Management: Ensure efficient data storage, retrieval, and processing, leveraging technologies like PostgreSQL, Redis, or others.</li><li>
AI Integration: Collaborate with AI engineers/researchers to integrate AI-driven insights and solutions into backend systems, and take ownership of API authentication and authorization frameworks.</li><li>
Data Visualization: Develop intuitive interfaces for presenting and interacting with complex time-series and topological data.</li></ul><h3 class="jobSectionHeader">
Requirements</h3><ul><li>
4+ years of software engineering experience, with a strong focus on full-stack systems.</li><li>
Proven ability to lead in fast-paced startup environments, with limited resources, shifting priorities, and minimal structure.</li><li>
Strong experience with Python and web frameworks such as FastAPI.</li><li>
Hands-on experience with React, TypeScript, Next.js, and TailwindCSS to build fast, responsive, and maintainable frontends.</li><li>
Experience deploying applications on AWS, working with ECS or Kubernetes, Postgres for data storage, and S3 for large-scale object storage.</li><li>
Note: You don&rsquo;t need to meet every single requirement. If you have deep experience in a subset of these areas, we encourage you to apply.</li></ul><h3 class="jobSectionHeader">
Nice to Have</h3><ul><li>
Familiarity with Python-based ecosystems is a plus.</li><li>
Experience working with Agentic AI systems or in the observability domain.</li><li>
Background in large-scale, complex, data-driven applications.</li><li>
Familiarity with building AI or LLM-powered products.</li></ul><h3 class="jobSectionHeader">
Compensation</h3>
We offer competitive compensation, startup equity, health insurance, and additional benefits. The U.S. base salary range for this full-time, in-person role in New York is $150,000&ndash;$300,000, plus equity and benefits. Our salary ranges are based on location, level, and role. Individual compensation is determined by experience, skills, and job-related knowledge.<h3 class="jobSectionHeader">
Why You Should Join Us</h3>
We&rsquo;ll make sure you&rsquo;re fully supported with health insurance, a great tech setup, flexible time off, and plenty of in-office snacks. We offer competitive salary and equity packages, and take thoughtful consideration with every hire on our small, high-impact team.
Traversal is fully in-office, 5 days a week, based in New York near Madison Square Park. We have a collaborative, hard-working culture and are energized by building the future of AI-powered software maintenance.
Working here means owning meaningful parts of the product, having the flexibility to move fast, and learning constantly. This is a place to grow your career, make a real impact, and help define a new category of infrastructure software.</div>

Product Engineer

Garner Health

<div>
 
 Garner's mission is to transform the healthcare economy, delivering high-quality and affordable care for all.
 
 
 
 We are fundamentally reimagining how healthcare works in the U.S. by partnering with employers to redesign healthcare benefits using clear incentives and powerful, data-driven insights. Our approach guides employees to higher-quality, lower-cost care, creating a system that works better for everyone. Patients achieve better health outcomes, employers spend healthcare dollars more effectively, and physicians are rewarded for delivering exceptional care rather than performing more procedures.
 
 
 Garner is one of the fastest-growing healthcare technology companies in the country. Our products are trusted by the most sophisticated employers and providers in the industry, and we are building a team of talented, mission-driven individuals who are motivated to make a meaningful impact on healthcare at scale.
 
 
 
 About the role:
 
 
 
 We are seeking an exceptional Senior Data Engineer with a high sense of ownership who is excited about our mission, eager to learn and teach others, and can deliver across the tech stack. This role will report to our VP, Engineering. The ideal candidate should have experience with a modern data stack and implementations of data pipelines using Python and SQL in AWS.
 
 
 
 Where you will work:
 
 
 
 This role will be based in our New York City office. You must be willing to work in the office 3 days per week on Tuesday, Wednesday and Thursday.
 
 
 
 What you will do:
 
 
 
 <ul>
 <li>
 Build, optimize, and maintain data pipelines that power our business
 </li>
 <li>
 Define and build out abstracted reusable data sets to be used for Business Intelligence, Marketing, and Data Science Research
 </li>
 <li>
 Design, build, and evangelize a federated data validation frameworks to be used to monitor potential data inconsistencies
 </li>
 <li>
 Protect our users' privacy and security through best practices
 
 
 </li>
 </ul>
 
 The ideal candidate has:
 
 
 
 <ul>
 <li>
 4+ years of software/data engineering experience (distributed data processing, data warehousing, data governance, Big Data, data variance, data privacy, and data quality)
 </li>
 <li>
 Expertise in SQL, job experience, and demonstrated strength with Python
 </li>
 <li>
 Expertise in building scalable data pipelines, query optimization, data modeling, and defining reusable datasets
 </li>
 <li>
 Experience working with orchestration tools (especially Airflow), databases (especially PostgreSQL), datalakes (especially Iceberg), data warehouses (especially Snowflake)
 </li>
 <li>
 Expertise with SQL tuning, Spark development, medallion and event-driven architectures.
 </li>
 <li>
 Familiarity with healthcare or insurance
 </li>
 <li>
 Familiarity with data security and HIPAA compliance
 </li>
 <li>
 A desire to be a part of a high-performing, mission-driven team that operates with intense urgency, a strong sense of individual accountability, and a commitment to authentic feedback
 
 
 </li>
 </ul>
 
 Technologies we use:
 
 
 
 <ul>
 <li>
 Postgres/SQL, Snowflake, Python, AWS, Terraform, Argo, Airflow, Spark, DuckDB, Iceberg, Airbyte, DBT, ElasticSearch (aka OpenSearch), OpenTelemetry, Looker
 
 
 </li>
 </ul>
 This is a unique opportunity to join a fast-growing company in a transformative role, helping shape the future of healthcare.
 
 
 
 Compensation Transparency:
 
 
 
 The target salary range for this position is $220,000.00 - $245,000.00. Individual compensation for this role will depend on various factors, including qualifications, skills, and applicable laws. In addition to base compensation, this role is eligible to participate in our equity incentive and competitive benefits plans, including but not limited to: flexible PTO, Medical/Dental/Vision plan options, 401(k) with company match, flexible spending accounts, Teladoc Health and more.
 
 
 
 Fraud and Security Notice:
 
 
 
 Please be aware of recent job scam attempts. Our recruiters use getgarner.com and garnerhealth.com email domains exclusively. If you have been contacted by someone claiming to be a Garner recruiter or a hiring manager from a different domain about a potential job, please report it to law enforcement here and to candidateprotection@garnerhealth.com.
 
 
 
 Equal Employment Opportunity:
 
 
 
 Garner Health is proud to be an Equal Employment Opportunity employer and values diversity in the workplace. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or other applicable legally protected characteristics.
 
 
 Garner Health is committed to providing accommodations for qualified individuals with disabilities in our recruiting process. If you need assistance or an accommodation due to a disability, you may contact us at talent@garnerhealth.com.
</div>

Senior Data Engineer

The Carlyle Group

<div>
 
 
 Company Profile
 
 
 
 The Carlyle Group (NASDAQ: CG) is a global investment firm with $475 billion of assets under management, across 678 investment vehicles as of March 31, 2026. Founded in 1987 in Washington, DC, Carlyle has grown into one of the world's largest and most successful investment firms, with more than 2,500 professionals operating in 28 offices in North America, Europe, the Middle East, Asia and Australia.
 
 
 
 
 
 Carlyle's purpose is to connect people, ideas, and capital to fuel growth for companies and performance for investors, which range from public and private pension funds to wealthy individuals and families to sovereign wealth funds, unions and corporations. Carlyle invests across three segments - Global Private Equity, Global Credit and Carlyle AlpInvest - and has deep expertise across industries, markets, and geographies.
 
 
 
 
 
 At Carlyle, we believe that a wide spectrum of experiences and viewpoints drives performance and success. Our CEO, Harvey Schwartz, has stated that, "To build better businesses and create value for all of our stakeholders, we are focused on assembling leadership teams with the strongest insights from a range of perspectives." Reflecting this view, emphasis is placed on development, retention and inclusion through our internal processes and seven Employee Resource Groups (ERGs). We cultivate a culture where ideas are openly shared and challenged, connecting diverse expertise and perspectives to drive enduring value.
 
 
 
 
 
 
 Fund or Department Description
 
 
 
 The Data &amp; AI Engineer sits within Carlyle's Enterprise Technology &amp; Data organization and supports firm-wide data and AI initiatives spanning investment platforms, portfolio operations, investor relations, and corporate functions. The role operates within a federated data operating model, partnering with domain engineering teams to implement shared platforms and reusable patterns for data and AI under the technical direction of the Senior AI &amp; Data Architect.
 
 
 
 
 
 
 Position Summary
 
 
 
 The Data &amp; AI Engineer is an experienced, hands-on engineer who turns Carlyle's data and AI architecture into working production systems. Reporting to the Senior AI &amp; Data Architect, this role is responsible for building and operating the pipelines, semantic layers, retrieval systems, and AI-ready data products that power analytics, automation, LLMs, agents, and generative AI applications across the firm.
 
 
 
 
 
 The role requires deep, hands-on expertise across modern data engineering and applied AI engineering. The Data &amp; AI Engineer will implement retrieval-augmented generation (RAG) patterns, embedding and indexing pipelines, vector stores, and semantic models alongside core ELT, streaming, and analytical pipelines - treating LLMs, agents, and copilots as first-class consumers of the data platform.
 
 
 
 
 
 This is a senior individual-contributor engineering role that executes against architectural standards, contributes to their evolution through hands-on learning, and partners closely with data science, AI engineering, governance, and domain teams to deliver trusted, AI-consumable data at enterprise scale.
 
 
 
 
 
 
 What Success Looks Like:
 
 In the first 12 months, this role will deliver foundational AI-ready data pipelines and retrieval components defined in the target-state architecture, productionize one or more priority RAG or agent-grounding use cases, and establish reusable engineering patterns that other domain teams can adopt across the federated data platform.
 
 
 
 
 
 
 In-office requirement:
 
 4 days per week
 
 
 
 
 
 
 Primary Responsibilities:
 
 
 
 
 AI Data Pipelines &amp; Retrieval Systems (&asymp;35%)
 
 
 <ul>
 <li>
 Build and operate AI-ready data pipelines - embedding generation, chunking, indexing, and refresh workflows - that make Carlyle's enterprise data reliably retrievable by LLMs, agents, and generative AI applications.
 </li>
 <li>
 Implement retrieval-augmented generation (RAG) components, including vector store integrations, hybrid search, re-ranking, and grounding logic, against architectural patterns defined by the Senior AI &amp; Data Architect.
 </li>
 <li>
 Develop and maintain tool and function interfaces that allow agents and copilots to query and act on enterprise data safely, with appropriate guardrails, logging, and evaluation hooks.
 </li>
 <li>
 Partner with Data Science and AI Engineering teams to operationalize feature stores, evaluation datasets, and reusable AI data products.
 </li>
 <li>
 Contribute to semantic and context engineering work that powers natural-language analytics, conversational reporting, and AI-driven insights for business users.
 </li>
 </ul>
 
 
 
 
 
 Modern Data Pipeline Engineering (&asymp;30%)
 
 
 <ul>
 <li>
 Design, build, and maintain production-grade ELT, streaming, and transformation pipelines using tools such as dbt, Fivetran and Snowflake.
 </li>
 <li>
 Implement ingestion, modeling, and consumption patterns that meet enterprise standards for scalability, performance, security, resiliency, and cost efficiency.
 </li>
 <li>
 Write clean, well-tested Python and SQL; apply software engineering best practices including version control, code review, CI/CD, modular design, and automated testing.
 </li>
 <li>
 Productionize new sources and domains under the federated operating model, partnering with domain data engineers to apply shared platform capabilities consistently.
 </li>
 </ul>
 
 
 
 
 
 Semantic Layer &amp; Data Product Development (&asymp;20%)
 
 
 <ul>
 <li>
 Implement semantic models, data contracts, and analytical/dimensional models that enable trusted self-service analytics and reliable AI grounding.
 </li>
 <li>
 Build and maintain reusable data products with clear ownership, documented contracts, and contextual metadata suitable for both human and AI consumers.
 </li>
 <li>
 Collaborate with the Senior AI &amp; Data Architect to refine and extend enterprise semantic standards based on what works in production.
 </li>
 <li>
 Support discovery and consumption tooling so that analysts, applications, and agents can find and use data products with minimal friction.
 </li>
 </ul>
 
 
 
 
 
 Data Quality, Observability &amp; AI Trust (&asymp;10%)
 
 
 <ul>
 <li>
 Implement data quality checks, lineage capture, and pipeline observability across both data and AI workloads.
 </li>
 <li>
 Build logging, evaluation, and monitoring components for AI systems - including prompt and response capture, retrieval metrics, and model performance signals - in line with governance standards.
 </li>
 <li>
 Partner with Data Governance to operationalize metadata, stewardship, and access controls, ensuring AI systems consume enterprise data with the same rigor as human users.
 </li>
 <li>
 Surface issues early, propose remediations, and feed lessons learned back into architectural patterns.
 </li>
 </ul>
 
 
 
 
 
 Collaboration &amp; Engineering Craft (&asymp;5%)
 
 
 <ul>
 <li>
 Participate in architectural design reviews and contribute hands-on engineering perspective to evolving patterns and standards.
 </li>
 <li>
 Mentor junior data engineers and analysts on modern data and AI engineering practices.
 </li>
 <li>
 Document patterns, write runbooks, and share knowledge across the federated organization to accelerate adoption of reusable platform capabilities.
 </li>
 </ul>
 
 
 
 
 
 Requirements:
 
 
 
 
 Education &amp; Certifications
 
 
 <ul>
 <li>
 Bachelor's degree, required
 </li>
 <li>
 Concentration in computer science, data engineering, information systems, or a related field, preferred
 </li>
 <li>
 Masters degree, preferred
 </li>
 <li>
 Relevant certifications in cloud, data engineering, analytics, or AI/ML are preferred
 </li>
 </ul>
 
 
 
 
 
 Professional Experience
 
 
 <ul>
 <li>
 6+ years of overall relevant technical experience, required
 </li>
 <li>
 Experience in data engineering, analytics engineering, or platform engineering, with at least 1-2 years of direct, hands-on experience building generative AI or AI/ML systems in production.
 </li>
 <li>
 Proven experience implementing retrieval, grounding, and semantic components for LLM- or agent-based applications, including RAG pipelines, vector stores, embedding workflows, and structured tool use.
 </li>
 <li>
 Hands-on experience with one or more modern AI platforms and tooling categories (e.g., AWS Bedrock, Databricks ML, Snowflake Cortex, OpenAI/Anthropic APIs, LangChain/LlamaIndex or equivalents, MLflow, and vector databases such as Databricks Vector Search, pgvector, or Pinecone).
 </li>
 <li>
 Strong, demonstrable expertise in Python and SQL, with working knowledge of distributed processing frameworks (e.g., Spark).
 </li>
 <li>
 Deep, hands-on experience with modern data stacks - dbt, Fivetran, Snowflake - in AWS-based environments.
 </li>
 <li>
 Track record of building data pipelines and products whose consumers include AI systems, not only BI tools and human analysts.
 </li>
 <li>
 Palantir experience a plus.
 </li>
 <li>
 Experience operating within federated data operating models and complex, regulated enterprise environments; financial services experience preferred.
 </li>
 </ul>
 
 
 
 
 
 Competencies &amp; Attributes
 
 
 <ul>
 <li>
 Demonstrated AI-forward instinct: defaults to asking how AI changes what gets built, rather than whether AI can be added later.
 </li>
 <li>
 Fluency in current AI engineering patterns (RAG, agents, tool use, evaluations, guardrails, observability) and the practical trade-offs involved in shipping them.
 </li>
 <li>
 Strong engineering craft: clean code, automated testing, thoughtful design, and a bias toward production-quality systems over prototypes.
 </li>
 <li>
 Pragmatic, delivery-oriented mindset with strong attention to data quality, AI trust, and long-term maintainability; able to distinguish durable engineering decisions from AI hype.
 </li>
 <li>
 Collaborative partner to architects, data scientists, AI engineers, and domain teams; comfortable operating in a matrixed, federated organization and in high-visibility transformational initiatives.
 </li>
 </ul>
 
 
 
 
 
 Benefits/Compensation
 
 
 
 The compensation range for this role is specific to the applicable office location and takes into account a wide range of factors, including required and preferred skill sets; prior experience and training; and licenses and/or certifications.
 
 
 
 
 
 The anticipated base salary range for this role is $160,000 to $180,000.
 
 
 
 
 
 In addition to base salary, the hired professional will receive a comprehensive benefits package including retirement benefits, health insurance, life and disability insurance, paid time off, paid holidays, family planning benefits, and wellness programs. The hired professional may also be eligible for an annual discretionary incentive program based on individual and organizational performance.
 
</div>

Data & AI Engineer

Explore 198+ Freelance data scraping engineer python in New York this June. Find roles across top companies and apply now to boost your career.

198 freelance data scraping engineer python jobs In New York

Software Engineer - Models Business

Python Developer/AI Engineer

Data Engineer

Product Engineer

Senior Data Engineer

Data Engineer

Data & AI Engineer