Business Intelligence Analyst Senior (AI Evaluation & Monitoring)
Core
Senior BI Analyst partnering with business/clinical areas to collect, clean, and interpret data sets to answer questions, solve problems, and support the evaluation of existing AI/ML projects.
Role type
Senior IC Business Intelligence Analyst (AI Evaluation & Monitoring)
Builds
Reporting solutions, analytic studies, and validation studies for Geisinger's AI portfolio
Domain
Healthcare / Clinical / Regulated Industry
Deliverable
production ML models
Required skills
SQL (large, messy clinical data warehouses), Python (pandas, numpy, scikit-learn, statsmodels), Applied Statistics (hypothesis testing, regression), Data Profiling & Validation, Project Management, Mentoring
Preferred skills
Healthcare/clinical data experience, Production monitoring design (drift detection, performance decay), LLM/generative AI evaluation (golden sets, hallucination checks)
Technologies
SQL, Python, pandas, numpy, scikit-learn, statsmodels
Responsibilities
Consult with AI stakeholders to define evaluation scope; Execute validation studies and production monitoring; Develop complex queries and reports; Present analysis to stakeholders; Mentor junior BI developers; Troubleshoot data quality issues.
Seniority
Senior, hands-on IC with mentorship responsibilities