Job Description
Qualification
• Experience in service architecture, development, and high performance and scalability.
• Experience in Spark, SQL performance tuning, and optimization.
• Experience with architectural design and development of large-scale data platforms and data applications.
• Good hands-on experience in AWS
• In-depth understanding of Spark & Hive Frameworks and their internal architecture
• Strong programming background with Java / Python.
• Practical exposure to end-to-end design and implementation process of Near-Real-Time and Batch Data Pipelines.
• Strong SQL (Hive/Spark) skills and experience tuning complex queries
• Excellent understanding of AWS storage, and it's compute services. Able to effectively use of AWS managed services - Step function, EMR, Lambda, Glue and Athena.
• Hands-on experience on Data Lake and ETL pipeline development
• Expertise on designing and building new Cloud Data platform and its optimization at organization level.
• Hands-on experience in Big Data technologies – Hadoop, Sqoop, Hive and Spark including DevOps.
Must Have
Role :
• 12-15 years of big data technologies or data platform architecture experience with deep technology expertise in Hive, HDFS, Spark, Kafka, Java, or Scala,Python, Pyspark etc.
• Experience in service architecture, development, and high performance and scalability.
• Experience in Spark, SQL performance tuning, and optimization.
• Experience with architectural design and development of large-scale data platforms and data applications.
• Expertise in design and management of complex data structures and data processes like ETL/ELT.
• Expertise in managing and operating distributed big data systems, including but not limited to the Hadoop ecosystem.
• A deep understanding of issues in multiple areas such as data acquisition and processing, data management, distributed processing, and high availability is required.
• Knowledge of Teradata.
• Expertise on designing and building new Cloud Data platform and it’s optimization at organization level.
• Strong past experience on designing AWS data lake and surrounding ecosystem development.
Good to have
• Understanding of sage maker and ML algorithms
• Experience in migrating workloads from on-premise to cloud and cloud to cloud migrations
• Experience on AWS services like RDS, DynamoDB, Redshift
• Ability to drive the deployment of the customers’ workloads into AWS and provide guidance, cloud adoption model, service integrations, appropriate recommendations to overcome blockers and technical road-maps for AWS cloud implementations.
• Extensive, real-world experience designing technology components for enterprise solutions and defining solution architectures and reference architectures with a focus on cloud technologies.
• Act as a subject-matter expert OR developer around AWS and become a trusted advisor to multiple teams.
• Coach and mentor engineers to raise the technical ability of the rest of the team, and/or to become certified in required AWS technical certifications
