data scientist

advisor - scientific data engineer Jobs in San Francisco

advisor - scientific data engineer

ai engineer, ai & data platforms (aidp) Jobs in San Francisco

ai engineer, ai & data platforms (aidp)

ai engineering lead Jobs in San Francisco

ai engineering lead

ai research engineer Jobs in San Francisco

ai research engineer

data engineer

Twitch

<div>
 
 About Us
 
 
 
 Twitch is the world&rsquo;s biggest live streaming service, with global communities built around gaming, entertainment, music, sports, cooking, and more. It is where thousands of communities come together for whatever, every day.
 
 
 We&rsquo;re about community, inside and out. You&rsquo;ll find coworkers who are eager to team up, collaborate, and smash (or elegantly solve) problems together. We&rsquo;re on a quest to empower live communities, so if this sounds good to you, see what we&rsquo;re up to on LinkedIn and X, and discover the projects we&rsquo;re solving on our Blog. Be sure to explore our Interviewing Guide to learn how to ace our interview process.
 
 
 
 About The Role
 
 
 
 As a data engineer in the Data Platform organization, you will be transforming Twitch&rsquo;s largest data sets into offerings that are usable by the entire company. You&rsquo;ll be working with other senior team members, as well as other data engineers, scientists, and stakeholders from all across the business. You will help define how our core business metrics are managed, as well as how other parts of the company define their individual metrics.
 
 
 
 You Will:
 
 
 
 
 <ul>
 <li>
 Work closely with data scientists and engineers from across the company to define and create robust data architectures and pipelines
 </li>
 <li>
 Develop and manage scalable data pipelines to extract, transform, and load data from various sources
 </li>
 <li>
 Partner with a range of stakeholders to translate the company&rsquo;s needs into data offerings
 </li>
 <li>
 Automate data processing and reporting workflows to improve efficiency and data integrity
 </li>
 <li>
 Implement data quality checks and monitor processes to maintain high data accuracy
 </li>
 <li>
 Simplify and enhance the accessibility, clarity, and usability of large and complex datasets through the development of data cubes and data sharing solutions
 
 
 
 </li>
 </ul>
 
 You Have:
 
 
 
 
 <ul>
 <li>
 3+ year of experience working in a data engineering, data science, or software engineering capacity
 </li>
 <li>
 Expertise in writing and maintaining robust data pipelines in SQL and Python
 </li>
 <li>
 A track record of turning data requirements from stakeholders into actionable plans
 </li>
 <li>
 Experience with cloud-native solutions
 </li>
 <li>
 A history of implementing and upholding best practices in data pipelines, ensuring accuracy, consistency, and reliability
 
 
 
 </li>
 </ul>
 
 Bonus Points
 
 
 
 
 <ul>
 <li>
 Bachelor's degree, or Master's degree in computer science, statistics, information systems, or a related, technical field
 </li>
 <li>
 Experience with a scripting language (e.g., Python, Java, or R)
 </li>
 <li>
 Familiarity and interest in Twitch!
 
 
 
 </li>
 </ul>
 
 Perks
 
 
 
 
 <ul>
 <li>
 Medical, Dental, Vision &amp; Disability Insurance
 </li>
 <li>
 401(k)
 </li>
 <li>
 Maternity &amp; Parental Leave
 </li>
 <li>
 Flexible PTO
 </li>
 <li>
 Amazon Employee Discount
 
 
 
 </li>
 </ul>
 
 Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.
 
 
 
 Job ID: TW9042
 
 
 The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life &amp; AD&amp;D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
 
 
 US, CA, San Francisco
 
 
 $152,000&mdash;$205,600 USD
 
 
 
 Twitch is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
 
 
 
 
 Twitch values your privacy. Please consult our Candidate Privacy Notice, for information about how we collect, use, and disclose personal information of our candidates.
 
</div>

Data Engineer

Whatnot

<div>
 
 
 🚀 Join the Future of Commerce with Whatnot!
 
 
 
 
 Whatnot is the largest livestream shopping platform in North America and Europe to buy, sell, and discover the things you love. Whether it's trading cards, fashion, electronics, or live plants, our sellers are building real businesses across hundreds of categories. We're building live commerce at a scale that's never been done in the West, and there's no playbook to copy. The people here are shaping how an entirely new industry develops.
 
 
 As a remote co-located team, we're inspired by our values and anchored in hubs across the US, UK, Ireland, Poland, Germany, and Australia. We move fast, stay close to our users, and focus on the work that drives the most impact.
 
 
 We're one of the fastest growing marketplaces and were recently named the #1 Best Startup Employer in America by Forbes. Check out the latest Whatnot updates on our news and engineering blogs and join us as we enable anyone to turn their passion into a business and bring people together through commerce.
 
 
 
 💻 Role
 
 
 
 Data is crucial to Whatnot&rsquo;s mission to bring people together through commerce.
 
 
 As our newest
 
 Data Engineer
 
 , you&rsquo;ll build and scale the systems that power data-driven decisions across the company. You&rsquo;ll work directly with stakeholders across the business &ndash; such as product, sales, marketing, finance, and trust teams &ndash; to design reliable data architectures, ship resilient pipelines, and create the foundational data products that power Whatnot&rsquo;s internal and external growth.
 
 
 
 On Any Given Day, You Will
 
 
 
 <ul>
 <li>
 Own data architecture end-to-end. Define how we capture, model, and serve critical business data&mdash;then implement it in production. You&rsquo;ll make architectural decisions around storage formats, compute patterns, and SLAs that balance cost, scalability, and consistency.
 </li>
 <li>
 Build mission-critical pipelines. Develop and operate streaming and batch data workflows that process high-volume events across multiple domains&mdash;user activity, transactions, experimentation, marketing performance, and operational telemetry&mdash;with tight guarantees for latency, completeness, and accuracy.
 </li>
 <li>
 Design and implement canonical models. Create domain-oriented data models that serve as the source of truth for analytics, ML, and real-time applications. Establish and enforce modeling standards, ownership boundaries, and data contracts across teams.
 </li>
 <li>
 Enforce data quality at scale. Build tests, lineage, monitoring, and reconciliation systems that make every dataset observable and every anomaly actionable.
 </li>
 <li>
 Automate operational workflows. Partner with business systems and platform teams to eliminate manual data handoffs and reconcile data across services, warehouses, and external systems.
 </li>
 <li>
 Enable insights and experimentation. Support analytics, ML, and product engineering teams by exposing high-quality, low-latency data through semantic layers, APIs, and real-time query systems.
 
 
 </li>
 </ul>
 Due to the cross-functional nature of this role, candidates will be matched to multiple teams based on need and experience, including product, analytics, or data platforms teams.
 
 
 
 👋 You
 
 
 
 People who do well at Whatnot tend to be comfortable figuring things out as they go, biased toward action, and genuinely curious about what they're building. They care more about outcomes than credit and stay close to the product and the people using it.
 
 
 <ul>
 <li>
 Have 3+ years of experience as a data or software engineer building data warehouses, distributed data systems, or event-driven architectures.
 </li>
 <li>
 Can design and implement data models using dimensional, Data Vault, or ledger-style techniques that support analytical and transactional workloads.
 </li>
 <li>
 Have deep hands-on expertise with modern data tooling across ingestion (e.g., Kafka, Debezium), transformation (dbt, Spark, Flink), orchestration (Dagster, Airflow), and observability (Monte Carlo, Great Expectations).
 </li>
 <li>
 Have operated cloud data warehouses such as Snowflake, BigQuery, or Redshift, including schema design, cost optimization, and workload tuning.
 </li>
 <li>
 Are comfortable writing production-grade code in Python or SQL languages, and integrating with CI/CD and infrastructure-as-code workflows.
 </li>
 <li>
 Enjoy partnering across disciplines&mdash;engineering, product, analytics&mdash;to translate messy business requirements into elegant data systems.
 </li>
 <li>
 Thrive as a self-starter in a fast-moving environment, owning both the technical design and the operational outcomes of your work.
 
 
 </li>
 </ul>
 If designing the data backbone that powers Whatnot&rsquo;s growth, insights, and product innovation excites you, we&rsquo;d love to meet you.
 
 
 
 We offer flexibility to work from home or from one of our global office hubs, and we value in-person time for planning, problem-solving, and connection. Team members in this role must live within commuting distance of our New York, Seattle, Los Angeles, and San Francisco hubs.
 
 
 
 
 💰Compensation
 
 
 
 
 For Full-Time (Salary)
 
 
 
 
 US based applicants:
 
 $180,000/year to $260,000/year + benefits + equity.
 
 
 The salary range may be inclusive of several levels that would be applicable to the position. Final salary will be based on a number of factors including, level, relevant prior experience, skills, and expertise. This range is only inclusive of base salary, not benefits (more details below) or equity.
 
 
 
 🎁 Benefits
 
 
 
 <ul>
 <li>
 Generous Holiday and Time off Policy
 </li>
 <li>
 Health Insurance options including Medical, Dental, Vision
 </li>
 <li>
 Work From Home Support
 </li>
 <ul>
 <li>
 Home office setup allowance
 </li>
 <li>
 Monthly allowance for cell phone and internet
 </li>
 </ul>
 <li>
 Care benefits
 </li>
 <ul>
 <li>
 Monthly allowance for wellness
 </li>
 <li>
 Annual allowance towards Childcare
 </li>
 <li>
 Lifetime benefit for family planning, such as adoption or fertility expenses
 </li>
 </ul>
 <li>
 Retirement; 401k offering for Traditional and Roth accounts in the US (employer match up to 4% of base salary) and Pension plans internationally
 </li>
 <li>
 Monthly allowance to dogfood the app
 </li>
 <ul>
 <li>
 All Whatnauts are expected to develop a deep understanding of our product. We're passionate about building the best user experience, and all employees are expected to use Whatnot as both a buyer and a seller as part of their job (our dogfooding budget makes this fun and easy!).
 </li>
 </ul>
 <li>
 Parental Leave
 </li>
 <ul>
 <li>
 16 weeks of paid parental leave + one month gradual return to work *company leave allowances run concurrently with country leave requirements which take precedence.
 </li>
 </ul>
 </ul>
 
 
 💛 EOE
 
 
 
 
 Whatnot is proud to be an Equal Opportunity Employer. We value diversity, and we do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, parental status, disability status, or any other status protected by local law. We believe that our work is better and our company culture is improved when we encourage, support, and respect the different skills and experiences represented within our workforce.
 
 
 Compensation Range: $180K - $260K
 
 
</div>

You.com

<div>
 
 About Us
 
 
 
 At You.com, we are building the AI Search Infrastructure that powers modern AI systems. Our goal is to create the trusted knowledge layer that agents, applications, and enterprises rely on to retrieve real-time, accurate, and citation-backed information.
 
 
 Our platform combines proprietary vertical indexes with LLM-optimized retrieval systems to power AI agents, applications, and enterprise workflows. We are solving hard problems across search, large language models, and large-scale infrastructure to make AI systems more reliable, transparent, and useful.
 
 
 Our team includes engineers, researchers, product builders, and operators who care about solving meaningful problems and delivering real-world impact. Whether you are improving core infrastructure, shaping product experiences, or helping bring new AI capabilities to market, your work will help define how modern AI finds and uses knowledge.
 
 
 
 About The Role
 
 
 
 We are looking for a hands-on Data Engineer to help build and scale our modern data platform. In this role, you will work closely with engineering, product, and analytics teams to develop reliable, high-performance data pipelines and systems.
 
 
 You'll contribute to both batch and real-time data processing using technologies like Databricks, AWS, and Kafka, while helping ensure data quality, accessibility, and usability across the organization. You'll play a key role in enabling data activation, ensuring that high-quality data flows not only into the warehouse but also outward to business tools like Salesforce, HubSpot, and Braze. Additionally, you will help power next-generation
 
 AI-driven applications
 
 , including agent-based systems and retrieval-augmented generation (RAG), by building robust data foundations and pipelines. This is a great opportunity for someone who enjoys solving data challenges end-to-end from ingestion to insights.
 
 
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Build and maintain scalable data pipelines (batch and streaming) using tools like Databricks, Spark, Kafka, and AWS services
 </li>
 <li>
 Design, develop, and optimize ETL/ELT workflows using DBT, PySpark, SQL, and tools like Fivetran
 </li>
 <li>
 Partner closely with marketing and growth teams to enable data use cases such as segmentation, campaign targeting, and lifecycle analytics
 </li>
 <li>
 Develop and maintain reverse ETL pipelines to sync data from the warehouse to tools like Salesforce, HubSpot, Braze, and other downstream systems
 </li>
 <li>
 Create and manage curated datasets to support analytics, reporting, and go-to-market initiatives
 </li>
 <li>
 Build and maintain dashboards and reporting layers to support marketing and business performance tracking
 </li>
 <li>
 Support AI/ML and agent-based applications by preparing and serving high-quality datasets for RAG pipelines and MCP (Model Context Protocol) integrations
 </li>
 <li>
 Monitor pipeline performance, troubleshoot issues, and ensure high data reliability and quality
 </li>
 <li>
 Implement data quality checks, validations, and alerting mechanisms across both ingestion and activation layers
 </li>
 <li>
 Collaborate with cross-functional teams to define data contracts and ensure consistency across systems
 
 
 </li>
 </ul>
 
 Qualifications
 
 
 
 <ul>
 <li>
 
 6+ years
 
 of experience in data engineering or a related field
 </li>
 <li>
 Strong hands-on experience with Databricks, AWS (S3, Glue, Athena, EMR, etc.), and Kafka
 </li>
 <li>
 Proficiency in Python (PySpark) and SQL for large-scale data processing
 </li>
 <li>
 Experience building and maintaining ETL/ELT pipelines (DBT/Airflow or similar experience preferred)
 </li>
 <li>
 Experience with data ingestion tools such as Fivetran (or similar)
 </li>
 <li>
 Familiarity with reverse ETL / data activation workflows and syncing data to tools like Salesforce, HubSpot, Braze
 </li>
 <li>
 Exposure to or experience with AI/ML data pipelines, including RAG architectures, vector databases, or embeddings workflows
 </li>
 <li>
 Familiarity with agent-based systems, MCP integrations, or LLM-powered applications is a strong plus
 </li>
 <li>
 Experience working with marketing, Product or growth teams on data use cases (segmentation, attribution, campaign analytics, etc.)
 </li>
 <li>
 Understanding of data modeling and working with large-scale datasets (batch and streaming)
 </li>
 <li>
 Experience creating dashboards and supporting reporting workflows (BI tools) for both internal and external audiences
 </li>
 <li>
 Strong problem-solving skills and ability to debug production data issues
 </li>
 <li>
 Strong communication skills and ability to work collaboratively across teams
 
 
 </li>
 </ul>
 Our salary bands are structured based on a combination of geographic tiers and internal leveling. Compensation is determined by multiple factors assessed during the interview process, with the final offer reflecting these considerations.
 
 
 Salary Band
 
 
 $185,000&mdash;$225,000 USD
 
 
 
 Company Perks:
 
 
 
 <ul>
 <li>
 Hubs in San Francisco and New York City offering regular in-person gatherings and co-working sessions
 </li>
 <li>
 Flexible PTO with U.S. holidays observed and a week shutdown in December to rest and recharge*
 </li>
 <li>
 A competitive health insurance plan covers 100% of the policyholder and 75% for dependents*
 </li>
 <li>
 12 weeks of paid parental leave in the US*
 </li>
 <li>
 401k program, 3% match - vested immediately!*
 </li>
 <li>
 $500 work-from-home stipend to be used up to a year of your start date*
 </li>
 <li>
 $600 technology stipend to support a portion of our hybrid/remote team's cell phone and internet expenses*
 </li>
 <li>
 $1,200 per year Health &amp; Wellness Allowance to support your personal goals*
 </li>
 <li>
 The chance to collaborate with a team at the forefront of AI research
 </li>
 <li>
 Certain perks and benefits are limited to full-time employees only
 
 
 </li>
 </ul>
 
 You.com participates in E-Verify. We will provide the Social Security Administration (SSA) and, if necessary, the Department of Homeland Security (DHS) with information from each new employee's Form I-9 to confirm work authorization. (English/Spanish: E-Verify Participation/Right to Work)
 
 
 We are also an inclusive, equitable, and accessible workplace. Please let us know if you require accommodation for any portion of the recruitment and hiring process.
 
 
 
 
 
 Beware of recruiting scams: You.com will only contact you through official @You.com email addresses and will never ask for payment or sensitive personal information during the hiring process.
 
 
</div>

Figma

<div>
 Figma is growing our team of passionate creatives and builders on a mission to make design accessible to all. Figma&rsquo;s platform helps teams bring ideas to life&mdash;whether you're brainstorming, creating a prototype, translating designs into code, or iterating with AI. From idea to product, Figma empowers teams to streamline workflows, move faster, and work together in real time from anywhere in the world. If you're excited to shape the future of design and collaboration, join us!
 
 
 We are looking for an experienced Data Engineer to partner with our Data Science and Data Infrastructure teams to own and scale our data pipelines. You&rsquo;ll also work closely with stakeholders across business teams including sales, marketing, and finance to ensure that the data they need arrives promptly and reliably. You&rsquo;ll play an integral role in building the metrics and self-serve reporting capabilities to unlock Figma&rsquo;s next phase of growth.
 
 
 This is a great role for an individual who is passionate about working with data and data systems, and who loves solving problems. You&rsquo;ll have a good sense for when it makes sense to build fast, scrappy solutions to unblock a key stakeholder vs. when to push back or bring in an outside service. The ideal candidate will be a great communicator who can help coordinate across multiple internal and external teams and takes pride in building end-to-end projects.
 
 
 This is a full time role that can be held from one of our US hubs or remotely in the United States.
 
 
 
 What you'll do at Figma
 
 
 :
 
 
 
 <ul>
 <li>
 Own, build, and maintain scalable data pipelines that connect various cloud data sources.
 </li>
 <li>
 Develop a deep understanding of Figma&rsquo;s core data models and optimize data pipelines for scale.
 </li>
 <li>
 Partner with the Data Science and Data Infrastructure teams to build new foundational data sets that are trusted, well understood, and enable self-service.
 </li>
 <li>
 Work with a wide range of cross-functional stakeholders to derive requirements and architect shared datasets; ability to document, simplify and explain complex problems to different types of audiences.
 </li>
 <li>
 Establish best practices for the development of specialized data sets for analytics and modeling.
 
 
 </li>
 </ul>
 We'd love to hear from you if you have:
 
 
 <ul>
 <li>
 4+ years in a relevant field.
 </li>
 <li>
 Fluency with both SQL and Python.
 </li>
 <li>
 Familiarity with Snowflake, dbt, Dagster, and ETL/reverse ETL tools.
 </li>
 <li>
 Excellent judgment and creative problem solving skills.
 </li>
 <li>
 A self-starting mindset along with strong communication and collaboration skills.
 
 
 </li>
 </ul>
 While not required, it&rsquo;s an added plus if you also have:
 
 
 <ul>
 <li>
 Knowledge in data modeling methodologies to design and build robust data architectures for insightful analytics.
 </li>
 <li>
 Experience with business systems such as Salesforce, Customer IO, Stripe, NetSuite is a big plus.
 
 
 </li>
 </ul>
 At Figma, one of our values is Grow as you go. We believe in hiring smart, curious people who are excited to learn and develop their skills. If you&rsquo;re excited about this role but your past experience doesn&rsquo;t align perfectly with the points outlined in the job description, we encourage you to apply anyways. You may be just the right candidate for this or other roles.
 
 
 
 Pay Transparency Disclosure
 
 
 
 If based in Figma&rsquo;s San Francisco or New York hub offices, this role has the annual base salary range stated below.
 
 
 Job level and actual compensation will be decided based on factors including, but not limited to, individual qualifications objectively assessed during the interview process (including skills and prior relevant experience, potential impact, and scope of role), market demands, and specific work location. The listed range is a guideline, and the range for this role may be modified. For roles that are available to be filled remotely, the pay range is localized according to employee work location by a factor of between 80% and 100% of range. Please discuss your specific work location with your recruiter for more information.
 
 
 Figma offers equity to employees, as well a competitive package of additional benefits, including health, dental &amp; vision, retirement with company contribution, parental leave &amp; reproductive or family planning support, mental health &amp; wellness benefits, generous PTO, company recharge days, a learning &amp; development stipend, a work from home stipend, and cell phone reimbursement. Figma also offers sales incentive pay for most sales roles and an annual bonus plan for eligible non-sales roles. Figma&rsquo;s compensation and benefits are subject to change and may be modified in the future.
 
 
 Annual Base Salary Range:
 
 
 $140,000&mdash;$348,000 USD
 
 
 At Figma we celebrate and support our differences. We know employing a team rich in diverse thoughts, experiences, and opinions allows our employees, our product and our community to flourish. Figma is an equal opportunity workplace - we are dedicated to equal employment opportunities regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity/expression, veteran status
 
 ,
 
 or any other characteristic protected by law. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements.
 
 
 We will work to ensure individuals with disabilities are provided reasonable accommodation to apply for a role, participate in the interview process, perform essential job functions, and receive other benefits and privileges of employment. If you require accommodation, please reach out to accommodations-ext@figma.com. These modifications enable an individual with a disability to have an equal opportunity not only to get a job, but successfully perform their job tasks to the same extent as people without disabilities.
 
 
 Examples of accommodations include but are not limited to:
 
 
 <ul>
 <li>
 Holding interviews in an accessible location
 </li>
 <li>
 Enabling closed captioning on video conferencing
 </li>
 <li>
 Ensuring all written communication be compatible with screen readers
 </li>
 <li>
 Changing the mode or format of interviews
 
 
 </li>
 </ul>
 To ensure the integrity of our hiring process and facilitate a more personal connection, we require all candidates keep their cameras on during video interviews. Additionally, if hired you will be required to attend in person onboarding.
 
 
 By applying for this job, the candidate acknowledges and agrees that any personal data contained in their application or supporting materials will be processed in accordance with Figma's Candidate Privacy Notice.
</div>

Octave

<div><h3 class="jobSectionHeader">About the Company:</h3>

Octave is a modern behavioral health practice creating a new standard for care delivery that's both high-quality and accessible. With in-person and virtual clinics in multiple states, the company offers evidence-based individual, couples, and family therapy, while pioneering relationships with payers to make care more affordable through insurance. By raising the bar on how care is delivered and how providers are supported, we are building a sustainable system that values equity, affordability, and effectiveness.
<h3 class="jobSectionHeader">
Job Summary:</h3>

We're looking for a Sr. Data Engineer with strong data platform experience to help evolve our modern data stack and contribute to the foundation of our emerging AI and ML platform. This role sits at the intersection of data engineering, platform architecture and machine learning enablement and will bring high-quality, scalable, and ethical AI into real-world use. You will partner closely with data scientists, analysts, and product managers to ensure our platform supports reliable data pipelines, scalable analytics, and production ready machine learning systems in addition to defining new architecture, best practices, and patterns for fellow engineers to inherit. The ideal candidate is both a systems thinker and a hands-on builder who thrives in evolving environments and is passionate about creating reliable data infrastructure that enables peers and partner teams to move faster with data.
<h3 class="jobSectionHeader">
Duties &amp; Responsibilities:</h3>
<ul><li>Design, build, and maintain scalable systems for ingestion, transformation, and storage of data, with a focus on testing and observability.</li>
<li>Implement frameworks, tooling, and automation to safely increase development velocity.</li>
<li>Develop foundational end-to-end AI/ML workflows from (1) source ingestion and preparation, (2) training and tuning, (3) experimentation and productionization, and (4) downstream systems integration (EHR modules, micro-services, dashboards).</li>
<li>Support iterative model development and production operations and observability (accuracy, drift, bias, fairness, reproducibility).</li>
<li>Contribute to a culture of continuous improvement, knowledge-sharing and mentoring of peer engineers.</li>
<li>Leverage AI tools as a core part of daily work (drafting, research, iteration) to improve efficiency, quality, and decision-making.</li>
</ul><h3 class="jobSectionHeader">Required Skills:</h3>
<ul><li>Proficiency in SQL and Python with strong familiarity towards modern data engineering frameworks, infrastructure, and tooling.</li>
<li>Proficiency with data ops best practices, monitoring, pipeline automation, and CI/CD.</li>
<li>Knowledge of modern compute and ML frameworks/libraries (i.e., Spark, TensorFlow, PyTorch, scikit-learn).</li>
<li>Ability to build production APIs and services, inclusive of MCP servers that expose internal data/services to LLMs.</li>
<li>A collaborative mindset, dependable execution, drive to reflect and improve, and humility to ask questions and learn.</li>
<li>Comfort using AI tools in day-to-day workflows, with a willingness to continuously rethink and improve how work gets done.</li>
<li>Curiosity and openness to experimenting with new tools and approaches; prior experience with AI tools is a plus.</li>
</ul><h3 class="jobSectionHeader">Education &amp; Experience:</h3>
<ul><li>Bachelor's degree (or equivalent) in Computer Science, Data Science, Statistics, Engineering or a related field.</li>
<li>5+ years of experience in data engineering, platform engineering, or ML engineering.</li>
<li>Experience working with major cloud data platforms and tools:</li>
<li>Preferred experience:</li>
<li><ul><li>Healthcare, behavioral health, EHR systems, and/or regulated industries.</li>
<li>Specific expertise with: AWS/GCP, dbt, AirflowAirbyte, Redshift/BigQuery.</li>
</ul></li></ul><h3 class="jobSectionHeader">Octave's Company Values:</h3>

The below values drive our day-to-day operations.
<ul>
<li>We're human beings first. We operate with empathy and kindness &ndash; with our clients, with our collaborators, and with ourselves.</li>
<li>People deserve better than status quo. We're willing to tackle the intractable problems, no matter how big, because someone should. We ask big questions, we craft big solutions, and we challenge ourselves and others to make it happen.</li>
<li>No bystanders. No stars. No tourists. Each person has been selected to be here, and with that comes a responsibility to bring your expertise, share your ideas, and help make this company better.</li>
<li>Partnership paves the path ahead. We don't operate in a silo, internally or externally. To transform the system, we believe in working with others to create something bigger, better, and stronger.</li>
<li>Quality is crucial at scale. Quality is core to our business, and we refuse to sacrifice it as we grow.</li>
<li>Progress is a process. In the pursuit of progress, we iterate, reflect, learn, adjust &ndash; and always leave things better than we found them.</li>
<li>There are people behind every data point. We recognize that numbers tell only one part of the story, and we also do the work to understand impacts at the individual level.</li>
</ul><h3 class="jobSectionHeader">Physical Requirements:</h3>
<ul><li>Prolonged periods sitting at a desk and working on a computer.</li>
<li>Must be able to frequently communicate with others through virtual meeting applications such as Zoom and Google Meet.</li>
<li>Must be able to observe and communicate information on company provided laptop.</li>
<li>Move up to 10 pounds on occasion.</li>
<li>Must be eligible to work in the United States without sponsorship now or in the future.</li>
</ul><h3 class="jobSectionHeader">Compensation, Equity &amp; Incentives:</h3>

Base Salary Range Octave is committed to pay equity and transparency. Our salary ranges are determined by role, level, and location/ zone. Final pay within the posted range will be decided based on a combination of job-related factors, including education, training, experience, and market demands.
<ul>
<li>Geo 1 (All other states &amp; D.C.): $142,600 - $153,100</li>
<li>Geo 2 (CO, HI, MD, RI): $156,900 - $168,400</li>
<li>Geo 3 (AK, CA, CT, MA, NJ, NY, WA): $164,000 - $176,000</li>
</ul>
Variable Compensation &amp; Equity
<ul>
<li>Performance Bonus: This role is not currently eligible for bonus incentives.</li>
<li>Equity Awards: This role is eligible to participate in Octave's equity program via Stock Options, subject to the terms of the applicable Equity Incentive Plan.</li>
<li>Remote Work Stipend: New full-time employees receive a $300 equipment reimbursement to support a home office setup.</li>
<li>Internet &amp; Phone Reimbursement: Full-time employees are eligible for a monthly reimbursement of up to $75 for internet and cell phone related expenses.</li>
</ul>
Octave is committed to pay equity and transparency. Octave will not discharge or in any other manner coerce, intimidate, threaten, discriminate or retaliate against employees or applicants because they have: (i) inquired about, discussed, compared, or disclosed their own wages, benefits, or other compensation, or the wages, benefits, or other compensation of another employee or applicant; (ii) asked Octave to provide a reason for the employee's wages or lack of opportunity for advancement; or (iii) aided or encouraged others in exercising their rights to discuss wages, compensation, or benefits, or to seek pay equity.

However, except as otherwise provided by law, employees who have access to the compensation information of other employees or applicants as part of their essential job functions cannot disclose the pay of other employees or applicants to individuals who do not otherwise have access to compensation information, unless the disclosure is (a) in response to a formal complaint or charge, (b) in furtherance of an investigation, proceeding, hearing, or action, including an investigation conducted by the employer, or (c) in compliance with the company's legal duty to furnish information under applicable law .
<h3 class="jobSectionHeader">
Comprehensive Benefits:</h3>

At Octave, we believe in supporting the &quot;human being first&quot;. Here's a snapshot of the benefits available to eligible employees:
<ul>
<li>Health &amp; Wellness: Choose what works best for you with one High Deductible Health Plan (HDHP) and two PPO medical plan options, plus comprehensive Dental and Vision coverage.</li>
<li>Medical Support: Enjoy access to a company-sponsored membership with One Medical for convenient, modern care.</li>
<li>Retirement: Plan for your future with a 401(k) that offers both traditional and Roth options.</li>
<li>Paid Time Off: Time to recharge matters. Take advantage of a generous time off policy, up to 15 paid company holidays, and accrued sick time.</li>
<li>Financial Protection: We've got you covered with company-paid Life, AD&amp;D, and Disability insurance, plus optional extras like Pet, Legal, and Worksite plans (Critical Illness, Hospital Indemnity, and Accident).</li>
<li>Parental Leave (Bonding Time): Growing your family? Full-time employees can take paid bonding leave after 6 months, whether it's a new baby, adoption, or foster placement. Time off increases with tenure.</li>
<li>Growth: Keep learning and leveling up with professional development reimbursement for role-related growth opportunities.</li>
</ul><h3 class="jobSectionHeader">How We Use Technology in Hiring:</h3>

As part of our hiring process, we may use technology tools, including AI-supported systems, to assist with reviewing applications or documenting interviews. These tools are designed to support our team, not replace human judgment, and final hiring decisions are always made by our team.

This job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee for this job. Duties, responsibilities and activities may change at any time with or without notice.<div>
Application Instructions:

Please complete the following application. Please note that the U.S. Equal Opportunity Employment Information questions below are used for the purposes of EEOC reporting and are optional to complete. Octave is unable to change these questions and we acknowledge that many of the U.S. Equal Opportunity Employment Information questions are not inclusive or affirming of all aspects of cultural identity. Octave is committed to an inclusive workplace environment, and this information will not inform how we approach hiring or employment.</div></div>

Sr. Data Engineer

Eli Lilly

<div>At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We&rsquo;re looking for people who are determined to make life better for people around the world.<h1 class="jobSectionHeader">
The Opportunity</h1>
We are building something unprecedented &mdash; an AI foundation that will push the frontier on what is possible today across drug discovery research, from target identification and disease biology through translational science.
The Applied Intelligence for Discovery (AI4D) team is a newly formed group within Lilly Research Laboratories that operates at the intersection of scientific delivery and core platform development. AI4D&rsquo;s mission is connecting scientists to petabyte-scale data through natural language interfaces, automated analysis workflows, and intelligent search &mdash; and to convert early deployments into repeatable system standards and evaluation practices that scale across therapeutic areas.
As a Scientific Data Engineer, you will close that gap. You will build the semantic layer, data harmonization infrastructure, AI-ready data products, and lakehouse architecture that bridge how data is stored and how AI systems need to consume it. You will be working at the intersection of the data infrastructure team and the generative AI engineers who build the systems scientists interact with.<h1 class="jobSectionHeader">
Responsibilities</h1><h2 class="jobSectionHeader">
Data Harmonization and Lakehouse Architecture</h2><ul><li>
Design and build the data architecture that transforms raw and processed omics data into harmonized, AI-consumable layers</li><li>
Build and optimize ETL/ELT pipelines that produce denormalized views, pre-computed aggregations, embedding-ready text representations, and feature stores optimized for AI system consumption</li><li>
Implement data quality monitoring, automated profiling, and validation checks across harmonization layers</li><li>
Create versioned, reproducible data snapshots that support model training, evaluation, and audit requirements in a regulated environment</li><li>
Partner with the teams to extend harmonization patterns as data modalities expand beyond genomics and proteomics into spatial transcriptomics, perturbational data (Perturb-Seq), single-cell, and digital pathology</li></ul><h2 class="jobSectionHeader">
Semantic Layer and Schema Engineering</h2><ul><li>
Design and maintain a semantic layer over Lilly&rsquo;s multi-omics databases that enables AI systems</li><li>
Create comprehensive schema documentation: table descriptions, column-level annotations, relationship mappings, business logic rules, and domain-specific constraints (e.g., statistical thresholds, unit conventions, experimental design metadata)</li><li>
Develop gold-standard question/SQL pairs for each major database, in collaboration with computational biologists and Generative AI Engineers, to serve as training data, few-shot examples, and evaluation benchmarks</li><li>
Build and maintain a data dictionary and ontology mapping layer that translates how scientists think and speak about data (gene names, pathway terms, assay types) into how the data is physically stored</li></ul><h2 class="jobSectionHeader">
AI-Ready Data Products</h2><ul><li>
Build and manage vector embedding pipelines for scientific documents, study metadata, and structured data descriptions to power RAG-based retrieval</li><li>
Build integration pipelines that connect heterogeneous data sources &mdash; omics databases, internal publications, electronic lab notebooks, assay results, and clinical annotations &mdash; into a unified, queryable layer</li><li>
Develop and enforce metadata standards that ensure new data sources are AI-accessible from the point of ingestion, not retroactively</li><li>
Design data products that serve multiple consumption patterns: direct SQL access for computational biologists, structured feeds for ML training pipelines, and semantic interfaces for LLM-powered tools</li></ul><h1 class="jobSectionHeader">
Qualifications</h1><ul><li>
Bachelors degree in Computer Science, Data Engineering, Bioinformatics, or a related field + 8 years data engineering experience OR Masters degree and 5 years data engineering experience</li><li>
Demonstrated expertise in building data pipelines, ETL/ELT workflows, and data products that serve downstream AI/ML systems</li></ul><h1 class="jobSectionHeader">
Additional Skills/Preferences</h1><ul><li>
Phd in data or related field</li><li>
Strong SQL skills and experience with complex relational database schemas (hundreds of tables, multi-level joins, domain-specific conventions)</li><li>
Experience with modern data platform technologies, including at least one of: Databricks, Snowflake, or equivalent lakehouse platforms</li><li>
Experience with modern data engineering tools: dbt, Spark, Airflow, or similar orchestration and transformation frameworks</li><li>
Proficiency in Python for data processing, scripting, and pipeline development</li><li>
Experience with cloud data platforms (AWS preferred: Redshift, Athena, Glue, S3, or similar)</li><li>
Familiarity with at least one of: vector databases, embedding pipelines, or semantic layer tooling</li><li>
Strong communication skills &mdash; you can work effectively with both engineers who think in schemas and scientists who think in biology</li><li>
Experience with biomedical or scientific data: omics datasets (RNA-seq, proteomics, GWAS), clinical data, or laboratory information management systems</li><li>
Experience in pharmaceutical, biotech, or life sciences environments</li></ul><ul><li>
Familiarity with biomedical ontologies and controlled vocabularies (Gene Ontology, MeSH, ChEBI, HGNC) and their application to data integration</li><li>
Experience building data products that serve AI/ML systems &mdash; feature stores, training datasets, evaluation benchmarks, or semantic annotations for text-to-SQL</li><li>
Knowledge of data governance practices in regulated industries: data lineage, access controls, versioning, and auditability</li><li>
Experience with knowledge graph technologies (Neo4j, Amazon Neptune, RDF/SPARQL) or graph-based data modeling</li><li>
Deep experience with Databricks ecosystem: Unity Catalog for data governance, Delta Lake for ACID transactions, MLflow integration, and Databricks SQL for analytics workloads</li><li>
Experience designing data architectures that bridge traditional bioinformatics workflows (Nextflow, R/Bioconductor) with modern lakehouse consumption patterns</li></ul>
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status. 
Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia Network, Black Employees at Lilly, Chinese Culture Network, Japanese International Leadership Network (JILN), Lilly India Network, Organization of Latinx at Lilly (OLA), PRIDE (LGBTQ+ Allies), Veterans Leadership Network (VLN), Women&rsquo;s Initiative for Leading at Lilly (WILL), enAble (for people with disabilities). Learn more about all of our groups.
Actual compensation will depend on a candidate&rsquo;s education, experience, skills, and geographic location. The anticipated wage for this position is
$166,500 - $266,200
Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly&rsquo;s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.
#WeAreLilly</div>

Advisor - Scientific Data Engineer

Apple

Do you have a passion for invention and self-challenge? Do you thrive on pushing the limits of what&rsquo;s considered feasible? As part of a world class modem team, you&rsquo;ll craft sophisticated leading-edge embedded firmware that deliver more performance in our products than ever before. You&rsquo;ll work across disciplines to transform improved hardware elements into a single, integrated design. 
 
Join us, and you&rsquo;ll help us innovate new wireless systems technologies that continually outperform the previous iterations. By collaborating with other product development groups across Apple, you&rsquo;ll push the industry boundaries of what wireless systems can do and improve the product experience for our customers across the world. Do you want to have an impact on every single Apple product? As a Cellular Data &amp; Automation Engineer, you will be at the center of the embedded 5G/4G/multimode cellular firmware effort within a silicon design group responsible for designing and productizing state-of-the-art cellular SoCs. 
 
Description 
 
As a Data &amp; Automation Engineer, you will be responsible for developing new features to enhance test automation system for improving test efficiency and stability. Furthermore, you will be participating in developing state-of-the-art data processing pipeline and leveraging data science algorithms &amp; machine learning models to parse massive data and logs in timely manner to automatically solve the issues or provide recommendations for the next step. In addition, you will be closely partnering with FW engineering teams in integrating and deploying models and data processing pipeline as part of entire automation system to deliver analysis results to benefit FW engineers in debugging issues. You will work with amazing team, brainstorm new ideas, and develop models and algorithms to solve complicated problems that have a substantial impact.&quot;,&quot;responsibilities&quot;:&quot;Develop new features to enhance the automation system based on the business&rsquo;s needs 
 
Develop and maintain real-time data processing pipelines to ensure processing accuracy and efficiency 
 
Collaborate closely with AI/ML engineers to integrate new AI pipelines and ML models into the production system and ensure achieving expected results and efficiency 
 
Integrate various internal tools and external tools into the existing automation system and/or data processing pipelines to enable new applications 
 
Provide technical support for users and help troubleshoot issues to unblock projects 
 
Preferred Qualifications 
 
Master&rsquo;s degree in data science or computer science or machine learning is preferred. 
 
Proficient in scripting or programming languages such as Python, Java, C/C++ with 5+ years of proven programming skills. 
 
Advanced SQL skills with the ability to write complex joins and nested queries. 
 
Proficiency in data visualization tools (such as Tableau, D3) and strong understanding of data manipulation and querying using SQL and Python together. 
 
Experience with dashboards and interactive reports. Ability to work with large datasets and conduct data cleaning and transformation. 
 
Experience with automation tools and frameworks, e.g. Jenkins, Gitlab CI/CD. 
 
Ability to leverage data science algorithms and machine-learning models to solve various kinds of issues. 
 
Experience in utilizing analytics and statistical methods to transform data into useful insights and actionable results. 
 
Ability to think creatively and identify, build, and support solutions and roadmaps focused on automation and reduction of manual processes. 
 
Strong critical thinking and communication skills with the ability and desire to learn and evaluate new technologies. 
 
Minimum Qualifications 
 
BS and a minimum of 3 years relevant industry experience. 
 
Proficient developer with 3+ years proven programming skills using Python, Pandas, Matplotlib and other key related tools. 
 
Solid knowledge and experience with relational database (e.g Postgres) and non-relational database (e.g.Mango DB). Ability to write complex queries with SQL is required. 
 
Experience with dashboards and interactive reports. Proficiency in data visualization tools (such as Tableau) and a strong understanding of data manipulation and querying using SQL and Python together. 
 
Ability to think creatively and identify, build, and support solutions and roadmaps focused on automation and reduction of manual processes. 
 
Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant . 
 
Pay &amp; Benefits 
 
At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $139,500 and $258,100, and your base pay will depend on your skills, qualifications, experience, and location. 
 
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits. 
 
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

Cellular Data & Automation Engineer

Explore 107+ Data engineer in San Francisco this May. Find roles across top companies and apply now to boost your career.

107 data engineer jobs In San Francisco

Data Engineer

Data Engineer

Data Engineer

Data Engineer

Sr. Data Engineer

Advisor - Scientific Data Engineer

Cellular Data & Automation Engineer