Data Engineer II

About DAT

DAT is an award-winning employer of choice and a next-generation SaaS technology company that has been at the leading edge of innovation in transportation supply chain logistics for 45 years. ders optimistically share future possibilities to inspire and motivate others toward their full potential. We expect our employees to find ways to embrace positive change, be curious and challenge the status quo, and provide solutions to unmet problems. Joining DAT means joining a culture focused on fostering development, building genuine connections, recognizing each other’s strengths and sharing in successes.

We continue to transform the industry year over year, by deploying a suite of software solutions to millions of customers every day - customers who depend on DAT for the most relevant data and most accurate insights to help them make smarter business decisions and run their companies more profitably. We operate the largest marketplace of its kind in North America, with 400 million freights posted in 2022, and a database of $150 billion of annual global shipment market transaction data. Our headquarters are in Denver, CO, with additional offices in Missouri, Oregon, and Bangalore, India. For additional information, see www.DAT.com/company.

Key Technologies

- Data Warehouse: Snowflake/Data Lake

- Transformation: dbt (data build tool)

- Orchestration: Apache Airflow

- CI/CD: Git, GitHub Actions

- Programming: Python

- AI-Assisted Development: LLM-based coding assistants (e.g., Claude, Copilot) for code generation, review, and documentation

Core Responsibilities

Data Pipeline Development & Engineering

- ELT Process Implementation: Assist senior engineers in designing, building, testing, and maintaining cloud-native ELT (Extract, Load, Transform) data pipelines, ensuring data is reliably loaded into Snowflake and S3 Iceberg layer.

- Pipeline Architecture: Contribute to the design of scalable, modular data pipelines that support incremental loads, backfills, and reprocessing with minimal manual intervention.

- Transformation with dbt: Develop and maintain data models using dbt (data build tool) for data cleaning, aggregation, and transformation and utilize the Snowflake data warehouse.

- Python Scripting: Utilize Python to build custom data extraction scripts, implement monitoring tools, and contribute to general automation efforts.

- Pipeline Reliability: Build in retry logic, alerting, and failure-handling patterns so pipelines degrade gracefully and self-recover where possible.

Workflow Orchestration and Automation

- Airflow DAGs: Learn to author, schedule, and monitor data workflows defined as Directed Acyclic Graphs (DAGs) in Apache Airflow.

- Pipeline Scheduling: Integrate and orchestrate dbt runs and other pipeline tasks within Airflow to manage dependencies and execution timing.

- Automation-First Mindset: Actively look for repetitive, manual, or error-prone steps across the data lifecycle and automate them - from data ingestion to deployment to reporting.

- Automated Testing & Deployment: Contribute to automated test suites and deployment scripts that reduce manual QA and release effort.

CI/CD & Version Control

- Git Workflow: Use Git for version control, following branching, code review, and pull request best practices.

- GitHub Actions: Build and maintain GitHub Actions workflows to automate testing, linting, dbt builds, and deployment of data pipeline code.

- Continuous Integration: Ensure new pipeline and model changes are automatically tested and validated before merging, catching issues before they reach production.

- Continuous Deployment: Support automated promotion of dbt models and pipeline code across environments (dev, staging, production).

AI-Enabled Engineering

- AI-Assisted Development: Use AI coding assistants to accelerate development, generate boilerplate, and speed up code review and refactoring.

- AI-Augmented Documentation: Leverage AI tools to help draft and maintain technical documentation for data models, DAGs, and pipeline logic.

- Applied Curiosity: Stay current on emerging AI-assisted data engineering tools and workflows, and bring suggestions for improving team efficiency.

Data Quality, Testing, and Monitoring

- Data Quality: Implement data validation and testing frameworks using features of dbt (e.g., uniqueness, non-null checks) to ensure high data quality and accuracy within Snowflake data marts.

- Troubleshooting: Monitor data pipeline health, troubleshoot failed Airflow tasks and dbt runs, and quickly resolve data flow issues.

- Documentation: Maintain clear and current technical documentation for data models, Airflow DAGs, and pipeline logic.

Qualifications & Experience

- Experience: 3-5 years of professional experience in a Data Engineering, Analytics Engineering, or similar technical role (including relevant internship experience).

- Education: Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related quantitative field.

- Technical Proficiency:

- Required: Strong proficiency in Python.

- Hands-on experience with a cloud data warehouse, preferably Snowflake.

- Familiarity with data transformation concepts and tools, dbt a bonus.

- Basic experience creating or running jobs/workflows using an orchestration tool like Apache Airflow.

- Experience with Git and CI/CD pipelines, ideally GitHub Actions.

- Comfort using AI-assisted coding tools in a professional engineering workflow.

- Soft Skills: Strong problem-solving abilities, excellent attention to detail, and a proactive, collaborative approach to teamwork.

Timings : 1.00 PM to 10.00 PM IST