110 Big Data Hadoop jobs in Pakistan

The Big Data Engineer is responsible for designing, developing and managing big data platforms to process and analyze large datasets. You will work with cutting-edge technologies to ensure seamless data flow, scalability and high-performance systems. Your work will be crucial in providing actionable insights for decision-making across the business. Join us to shape the future of data-driven strategies.

• Design Data Systems: Architect scalable data processing systems for business needs.
• Manage Data Pipelines: Build and maintain efficient data pipelines for real-time processing.
• Data Integration: Integrate disparate data sources for analysis and reporting.
• Ensure Data Quality: Implement methods for data cleaning and validation.
• Optimize Performance: Tune and optimize systems for speed and scalability.
• Collaborate with Teams: Work closely with data scientists and analysts to create solutions.
• Big Data Technologies: Work with Hadoop, Spark and similar big data tools.
• Security Compliance: Ensure data handling complies with privacy and security standards.

• 3+ years of experience in big data engineering or a similar role.
• Hands-on experience with big data platforms like Hadoop, Spark, or AWS.
• Previous work experience in developing and managing data pipelines.
• Strong programming skills in Python, Java, or Scala.
• Deep understanding of distributed computing systems and architectures.
• Ability to collaborate effectively across cross-functional teams.

You will be participating in exciting projects covering the end-to-end data lifecycle – from raw data integrations with primary and third-party systems, through advanced data modeling, to state-of-the-art data visualization and development of innovative data products.

You will have the opportunity to learn how to build and work with both batch and real-time data processing pipelines. You will work in a modern cloud-based data warehousing environment alongside a team of diverse, intense, and interesting co-workers. You will liaise with other departments – such as product & tech, the core business verticals, trust & safety, finance, and others – to enable them to be successful.

Your responsibilities

• Design, implement and support data warehousing;
• Raw data integrations with primary and third-party systems
• Data warehouse modeling for operational & application data layers
• Development in Amazon Redshift cluster
• SQL development as part of agile team workflow
• ETL design and implementation in Matillion ETL
• Real-time data pipelines and applications using serverless and managed AWS services such as Lambda, Kinesis, API Gateway, etc.
• Design and implementation of data products enabling data-driven features or business solutions
• Building data dashboards and advanced visualizations in Sisense for data cloud teams (formerly Periscope Data) with a focus on UX, simplicity, and usability
• Working with other departments on data products – i.e. product & technology, marketing & growth, finance, core business, advertising, and others
• Being part and contributing towards a strong team culture and ambition to be on the cutting edge of big data
• Evaluate and improve data quality by implementing test cases, alerts, and data quality safeguards
• Living the team values: Simpler. Better. Faster.
• Strong desire to learn

Required minimum experience (must)

• 1- 2 years experience in data processing, analysis, and problem-solving with large amounts of data;
• Good SQL skills across a variety of relational data warehousing technologies especially in cloud data warehousing (e.g. Amazon Redshift, Google BigQuery, Snowflake, Vertica, etc.)
• 1+ years of experience with one or more programming languages, especially Python
• Ability to communicate insights and findings to a non-technical audience
• Written and verbal proficiency in English
• Entrepreneurial spirit and ability to think creatively; highly-driven and self-motivated; strong curiosity and strive for continuous learning
• Top of class University technical degree such as computer science, engineering, math, physics.

Additional experience (strong plus)

• Experience working with customer-centric data at big data-scale, preferably in an online / e-commerce context
• Experience with modern big data ETL tools (e.g. Matillion)
• Experience with AWS data ecosystem (or other cloud providers)
• Track record in business intelligence solutions, building and scaling data warehouses, and data modeling
• Tagging, Tracking, and reporting with Google Analytics 360
• Knowledge of modern real-time data pipelines (e.g. serverless framework, lambda, kinesis, etc.)
• Experience with modern data visualization platforms such as Periscope, Looker, Tableau, Google Data Studio, etc.
• Linux, bash scripting, Javascript, HTML, XML
• Docker Containers and Kubernete

#LI-TM1

Job Title: Data Engineer

Job Title: Data Engineer

Location: Karachi, Lahore , Islamabad (Hybrid)

Experience: 5+ Years

Job Type: Full-Time

Job Overview

We are looking for a highly skilled and experienced Data Engineer with a strong foundation in Big Data, distributed computing, and cloud-based data solutions . This role demands a strong understanding of end-to-end Data pipelines, data modeling, and advanced data engineering practices across diverse data sources and environments. You will play a pivotal role in building, deploying, and optimizing data infrastructure and pipelines in a scalable cloud-based architecture.

Key Responsibilities

• Design, develop, and maintain large-scale Data pipelines using modern big data technologies and cloud-native tools.
• Build scalable and efficient distributed data processing systems using Hadoop, Spark, Hive, and Kafka.
• Work extensively with cloud platforms (preferably AWS) and services like EMR, Glue, Lambda, Athena, S3.
• Design and implement data integration solutions pulling from multiple sources into a centralized data warehouse or data lake.
• Develop pipelines using DBT (Data Build Tool) and manage workflows with Apache Airflow or Step Functions.
• Write clean, maintainable, and efficient code using Python, PySpark, or Scala for data transformation and processing.
• Build and manage relational and columnar data stores such as PostgreSQL, MySQL, Redshift, Snowflake, HBase, ClickHouse.
• Implement CI/CD pipelines using Docker, Jenkins, and other DevOps tools.
• Collaborate with data scientists, analysts, and other engineering teams to deploy data models into production.
• Drive data quality, integrity, and consistency across systems.
• Participate in Agile/Scrum ceremonies and utilize JIRA for task management.
• Provide mentorship and technical guidance to junior team members.
• Contribute to continuous improvement by making recommendations to enhance data engineering processes and architecture.
  
  

• 5+ years of hands-on experience as a Data Engineer
• Deep knowledge of Big Data technologies – Hadoop, Spark, Hive, Kafka.
• Expertise in Python, PySpark and/or Scala.
• Proficient with data modeling, SQL scripting, and working with large-scale datasets.
• Experience with distributed storage like HDFS and cloud storage (e.g., AWS S3).
• Hands-on with data orchestration tools like Apache Airflow or StepFunction.
• Experience working in AWS environments with services such as EMR, Glue, Lambda, Athena.
• Familiarity with data warehousing concepts and experience with tools like Redshift, Snowflake (preferred).
• Exposure to tools like Informatica, AbInitio, Apache Iceberg is a plus.
• Knowledge of Docker, Jenkins, and other CI/CD tools.
• Strong problem-solving skills, initiative, and a continuous learning mindset.
  
  

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• Experience with open table formats such as Apache Iceberg.
• Hands-on with AbInitio (GDE, Collect > IT) or Informatica tools.
• Knowledge of Agile methodology, working experience in JIRA.
  
  

• Self-driven, proactive, and a strong team player.
• Excellent communication and interpersonal skills.
• Passion for data and technology innovation.
• Ability to work independently and manage multiple priorities in a fast-paced environment.

• Seniority level Mid-Senior level

• Employment type Full-time

• Job function Information Technology
• Industries IT Services and IT Consulting

Referrals increase your chances of interviewing at NorthBay - Pakistan by 2x

Karachi Division, Sindh, Pakistan 5 months ago

Karachi Division, Sindh, Pakistan 9 months ago

Karachi Division, Sindh, Pakistan 7 months ago

Karachi Division, Sindh, Pakistan 1 year ago

Karachi Division, Sindh, Pakistan 2 months ago

Karachi Division, Sindh, Pakistan 1 day ago

Karachi Division, Sindh, Pakistan 4 months ago

Karachi Division, Sindh, Pakistan 3 months ago

Karachi Division, Sindh, Pakistan 1 day ago

Karachi East District, Sindh, Pakistan 5 months ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

About Burq

Burq started with an ambitious mission: how can we turn the complex process of offering delivery into a simple turnkey solution.

We started with building the largest network of delivery networks, partnering with some of the biggest delivery companies. We then made it extremely easy for businesses to plug into our network and start offering delivery to their customers. Now, we’re powering deliveries for some of the fastest-growing companies from retailers to startups.

It’s a big mission and now we want you to join us to make it even bigger!

We’re already backed by some of the Valley's leading venture capitalists, including Village Global, the fund whose investors include Bill Gates, Jeff Bezos, Mark Zuckerberg, Reid Hoffman, and Sara Blakely. We have assembled a world-class team all over the U.S.

We operate at scale, but we're still a small team relative to the opportunity. We have a staggering amount of work ahead. That means you have an unprecedented opportunity to grow while doing the most important work of your career.

We want people who are unafraid to be wrong and support decisions with numbers and narrative.

• Design, build, and maintain efficient and scalable data pipelines using tools like Airbyte, Airflow, and dbt, with a focus on integrating with Snowflake.
• Manage and optimize data warehousing solutions using Snowflake, ensuring data is organized, secure, and accessible for analysis.
• Develop and implement automations and workflows to streamline data processing and integration, ensuring seamless data flow across systems.
• Collaborate with cross-functional teams to set up and maintain data infrastructure that supports both engineering and analytical needs.
• Utilize Databricks and Spark for big data processing, ensuring data is processed, stored, and analyzed efficiently.
• Monitor data streams and processes using Kafka and Monte Cristo, ensuring data quality and integrity.
• Work closely with data analysts and other stakeholders to create, maintain, and optimize visualizations and reports using Tableau, ensuring data-driven decision-making.
• Ensure the security and compliance of data systems, implementing best practices and leveraging tools like Terraform for infrastructure management.
• Continuously evaluate and improve data processes, staying current with industry best practices and emerging technologies, with a strong emphasis on data analytics and visualization.

• Proficiency in SQL and experience with data visualization tools such as Tableau.
• Hands-on experience with data engineering tools and platforms including Snowflake, Airbyte, Airflow, and Terraform.
• Strong programming skills in Python and experience with data transformation tools like dbt.
• Familiarity with big data processing frameworks such as Databricks and Apache Spark.
• Knowledge of data streaming platforms like Kafka.
• Experience with data observability and quality tools like Monte Cristo.
• Solid understanding of data warehousing, data pipelines, and database management, with specific experience in Snowflake.
• Ability to design and implement automation, workflows, and data infrastructure.
• Strong analytical skills with the ability to translate complex data into actionable insights, particularly using Tableau.
• Excellent problem-solving abilities and attention to detail.

Investing in you

• Competitive salary
• Medical
• Educational courses

Generous Time Off

At Burq, we value diversity. We are an equal opportunity employer: we do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

Location: Karachi, Lahore , Islamabad (Hybrid)
Experience: 5+ Years
Job Type: Full-Time

We are looking for a highly skilled and experienced Data Engineer with a strong foundation in Big Data, distributed computing, and cloud-based data solutions . This role demands a strong understanding of end-to-end Data pipelines, data modeling, and advanced data engineering practices across diverse data sources and environments. You will play a pivotal role in building, deploying, and optimizing data infrastructure and pipelines in a scalable cloud-based architecture.

• Design, develop, and maintain large-scale Data pipelines using modern big data technologies and cloud-native tools.

• Build scalable and efficient distributed data processing systems using Hadoop, Spark, Hive, and Kafka .

• Work extensively with cloud platforms (preferably AWS) and services like EMR, Glue, Lambda, Athena, S3 .

• Design and implement data integration solutions pulling from multiple sources into a centralized data warehouse or data lake.

• Develop pipelines using DBT (Data Build Tool) and manage workflows with Apache Airflow or Step Functions .

• Write clean, maintainable, and efficient code using Python, PySpark, or Scala for data transformation and processing.

• Build and manage relational and columnar data stores such as PostgreSQL, MySQL, Redshift, Snowflake, HBase, ClickHouse .

• Implement CI/CD pipelines using Docker, Jenkins , and other DevOps tools.

• Collaborate with data scientists, analysts, and other engineering teams to deploy data models into production.

• Drive data quality , integrity, and consistency across systems.

• Participate in Agile/Scrum ceremonies and utilize JIRA for task management.

• Provide mentorship and technical guidance to junior team members.

• Contribute to continuous improvement by making recommendations to enhance data engineering processes and architecture.

• 5+ years of hands-on experience as a Data Engineer

• Deep knowledge of Big Data technologies – Hadoop, Spark, Hive, Kafka.

• Expertise in Python, PySpark and/or Scala .

• Proficient with data modeling , SQL scripting , and working with large-scale datasets .

• Experience with distributed storage like HDFS and cloud storage (e.g., AWS S3).

• Hands-on with data orchestration tools like Apache Airflow or StepFunction.

• Experience working in AWS environments with services such as EMR, Glue, Lambda, Athena.

• Familiarity with data warehousing concepts and experience with tools like Redshift, Snowflake (preferred).

• Exposure to tools like Informatica , AbInitio , Apache Iceberg is a plus.

• Knowledge of Docker , Jenkins , and other CI/CD tools.

• Strong problem-solving skills, initiative, and a continuous learning mindset.

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.

• Experience with open table formats such as Apache Iceberg.

• Hands-on with AbInitio (GDE, Collect > IT) or Informatica tools.

• Knowledge of Agile methodology , working experience in JIRA .

• Self-driven, proactive, and a strong team player.

• Excellent communication and interpersonal skills.

• Passion for data and technology innovation.

• Ability to work independently and manage multiple priorities in a fast-paced environment.

Data Engineer with 2+ years experience in AI, Python, SQL, Spark, Azure, Databricks. Skilled in scalable data infrastructure & pipeline design. English required.

• Bachelor’s degree in computer science, Engineering, or a related field.
• Minimum of 2 years of experience in data engineering, specifically in large-scale AI projects and production applications.
• Strong proficiency in Python, SQL, Spark, Cloud Architect, Data & Solution Architect, API, Databricks, and Azure.
• Deep understanding of designing and building scalable and robust data infrastructure.
• Experience with at least one public cloud provider.
• Strong understanding of system architecture and data pipeline construction.
• Familiarity with machine learning models and data processing requirements.
• Team player with analytical and problem-solving skills.
• Good communication skills in English.
• Optional:
• Expertise in distributed systems like Kafka and orchestration systems like Kubernetes.
• Basic knowledge in Data Lake/Data Warehousing/Big Data tools, Apache Spark, RDBMS, NoSQL, Knowledge Graph.

• Design, build, and manage data pipelines, integrating multiple data sources to support company goals.
• Develop and maintain large-scale data processing systems and machine learning pipelines, ensuring data availability and quality.
• Implement systems for data quality and consistency, collaborating with Data Science and IT teams.
• Ensure compliance with security guidelines and SDLC processes.
• Collaborate with Data Science team to provide necessary data infrastructure.
• Lead and manage large-scale AI projects, working with cloud platforms for deployment.
• Maintain and optimize databases, design schemas, and improve data flow across the organization.

“To be ILI” means traveling far to reinvigorate innovation. It symbolizes vocation, commitment, and passion. We are a dedicated taskforce focused on sensing and initiating innovation.

Like a lion in the wild, constantly hunting and alert, we stay focused and agile, unlike a lion in captivity which loses its hunting instinct due to guaranteed food supply.

We offer a steep learning curve, development opportunities, and a flexible, positive work environment.

Modern office with design furniture, rooftop terrace with city view.

Supportive, open-minded teammates. Enjoy after-work events, barbecues, excursions, and team gatherings.

Fresh fruits, professional coffee, smoothies, on-site gym, and yoga room for relaxation.