Business Intelligence Analyst Senior (AI Evaluation & Monitoring)
Core
Senior BI Analyst partnering with business and clinical areas to evaluate and monitor AI/ML projects, ensuring safety, performance, and accountability through data analysis and validation.
Role type
Senior IC Business Intelligence Analyst (AI Evaluation & Monitoring)
Builds
AI/ML evaluation frameworks, production monitoring dashboards, and validation reports for Geisinger's AI portfolio.
Domain
Healthcare / Clinical AI / Machine Learning Operations
Deliverable
production ML models | dashboards & analysis
Required skills
SQL (large/messy data, window functions, CTEs, performance tuning), Python (pandas, numpy, scikit-learn, statsmodels), statistical methods (hypothesis testing, regression, confidence intervals), data profiling, project management
Preferred skills
Healthcare/clinical data experience, LLM/generative AI evaluation (golden sets, hallucination checks), drift detection design
Technologies
SQL, Python, pandas, numpy, scikit-learn, statsmodels, common evaluation libraries
Responsibilities
Consult with AI stakeholders to define evaluation scope; execute validation studies and production monitoring; develop complex queries and reports; present analysis to non-technical stakeholders; mentor junior developers; troubleshoot data quality issues.
Seniority
Senior, hands-on IC with mentorship responsibilities