Data Engineer (Python)

# Data Engineer (Python) ## Location Gurgaon/Hyderabad/Remote ## About the role We are expanding our Data Engineering Team and hiring passionate professionals with extensive knowledge and experience in building and managing large enterprise data and analytics platforms. We are looking for creative individuals with strong programming skills, who can understand complex business and architectural problems and develop solutions. The individual will work closely with the rest of our data engineering and data science team in implementing and managing Scalable Smart Data Lakes, Data Ingestion Platforms, Machine Learning and NLP based Analytics Platforms, Hyper-Scale Processing Clusters, Data Mining and Search Engines. ## Responsibilities - Create and manage end-to-end Data Solutions, Optimal Data Processing Pipelines and Architecture dealing with large volume, big data sets of varied data types. - Work with data science and infrastructure team members to implement practical machine learning solutions and pipelines in production. ## Requirements - 3+ years of industry experience in creating and managing end-to-end Data Solutions, Optimal Data Processing Pipelines and Architecture dealing with large volume, big data sets of varied data types. - Proficiency in Python. Knowledge of design patterns, oops concepts and strong design skills in python. - Strong knowledge of working with PySpark Dataframes, Pandas Dataframes for writing efficient pre-processing and other data manipulation tasks. - Experience with creating Restful web services and API platforms. - Experience with SQL and NoSQL databases. E.g. Postgres/MongoDB/ Elasticsearch etc. - Experience with stream-processing systems: Spark-Streaming, Kafka etc and working experience with event-driven architectures. ## Good to have - Experience with testing libraries such as pytest for writing unit-tests for the developed code. - Knowledge and experience of Dockers and Kubernetes would be good to have.