Data Engineer
ThoughtWorks |
Bangalore,Karnataka,India
Short Description
ThoughtWorks is hiring for Data Engineer who will be creating complex data processing pipelines, as part of diverse, high energy teams and design scalable implementations of the models developed by Data Scientists.Job Description
Responsibilities:
- Creating complex data processing pipelines, as part of diverse, high energy teams.
- Designing scalable implementations of the models developed by our Data Scientists.
- Hands-on programming based on TDD, usually in a pair programming environment.
- Deploying data pipelines in production based on Continuous Delivery practices.
- Advising clients on the usage of different distributed storage and computing technologies from the plethora of options available in the ecosystem.
Required Skills And Experience:
- Minimum of 6 years of overall industry experience.
- 3+ years of experience building and deploying large scale data processing pipelines in a production environment.
- Experience building data pipelines and data-centric applications using distributed storage platforms like HDFS, S3, NoSQL databases (HBase, Cassandra, etc) and distributed processing platforms like Hadoop, Spark, Hive, Oozie, Airflow, etc in a production setting.
- Hands on experience in MapR, Cloudera, Hortonworks and/or Cloud (AWS EMR, Azure HDInsights, Qubole etc.) based Hadoop distributions.
- Experience working with, or an interest in Agile Methodologies, such as Extreme Programming (XP) and Scrum.
- Knowledge of software best practices, like Test-Driven Development (TDD) and Continuous Integration (CI).
- Strong communication and client-facing skills with the ability to work in a consulting environment are essential.
- Senior developers (6+ years) are expected to be the Architect for small and large enterprise projects. On larger projects, you are expected to work closely with the fellow architects to come up with the architecture and take it further.
- The desire to contribute to the wider technical community through collaboration, coaching, and mentoring of other technologists.
Data Engineer
Mid-Senior-level
Information Technology
|
Technology
|
Information
Full-time
Engineering
|
Other
|
Information Technology
Data Engineer
A community of passionate individuals whose purpose is to revolutionize software design, creation and delivery, while advocating for positive social change.
We work with people and organizations who have ambitious missions - whether they are in the commercial, social or government sectors. We set up smart teams who love challenges and think disruptively to help our clients succeed. Our Agile development tools help our clients continuously improve and deliver quality software.
We are focused on helping our industry improve, and believe in sharing what we learn. We do this by writing books, blogging, running events, talking at conferences, and championing open source.
We are strong believers in the power of software and technology as tools for social change. Through our Social Impact Program, we collaborate with organizations with a humanitarian mission and broad reach, helping them use technology to make an impact.