Job Title: Data Engineer
Location: Wilmington , DE
Duration: 14 Months + Extension
Bill Rate: $79/hour (C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other
Job Description :
A hands-on Senior Data Engineer to build and run end-to-end data pipelines on AWS DMS and Databricks using PySpark, Python, and SQL. The position owns pipelines from source-database replication through transformation to curated, analytics-ready data, working directly with the customer. Immediate delivery on AWS is required with no ramp-up time from Azure or another cloud.
Key Responsibilities :
- Design, build, and maintain end-to-end data pipelines, from source ingestion through transformation to curated output tables.
- Configure and run AWS DMS replication tasks for full loads and ongoing Change Data Capture (CDC) from relational sources such as Oracle, SQL Server, and MariaDB.
- Land DMS output in Amazon S3 and process it in Databricks with PySpark, using Delta Lake and a bronze/silver/gold (medallion) design.
- Write complex, efficient SQL for transformations, data validation, reconciliation, and troubleshooting.
- Handle CDC logic: upserts and merges, deletes, late-arriving data, schema changes, and deduplication.
- Tune Spark jobs and DMS tasks for performance, cost, and reliability; diagnose replication lag and job failures.
- Build in data quality checks, logging, alerting, and restart/recovery so pipelines run without hand-holding.
- Orchestrate workloads with Databricks Workflows (or similar) and keep code in source control with peer review.
- Meet regularly with customer stakeholders: gather requirements, explain design choices and trade-offs, give status updates, and walk through issues and fixes.
- Write clear technical documentation for pipelines, data flows, and runbooks.
Required Qualifications (Must Have) :
- All experience below must be hands-on and on AWS (experience only on Azure without AWS does not meet the bar):
- AWS DMS: Personally set up, run, and troubleshot DMS replication (full load + CDC) in a production setting, including endpoints, replication instances, task settings, and table mappings.
- Databricks: Strong hands-on development on Databricks on AWS: notebooks and jobs, Delta Lake, MERGE/upsert patterns, cluster configuration, and performance tuning.
- PySpark and Python: Writes production-grade PySpark transformations and Python code (modular, tested, version-controlled).
- End-to-end pipelines: Has built complex pipelines from source to curated data, not only the reporting or visualization layer.
- SQL: Strong SQL, including joins across large tables, window functions, CTEs, query tuning, and data reconciliation.
- Communication: Can explain work clearly to technical and non-technical audiences, and comfortable leading customer conversations.
- Experience: Typically 6+ years in data engineering, with at least 3 years currently on AWS data services and Databricks.
Nice to Have :
- AWS Glue: Jobs, crawlers, Data Catalog
- Power BI: Enough to understand how downstream reports consume the data
- Other AWS services: S3, IAM, Lambda, Step Functions, CloudWatch, Redshift
- Databricks features: Unity Catalog, Delta Live Tables, or Auto Loader
- CI/CD for data code: Git, Databricks Asset Bundles, Terraform
- Certifications: Databricks or AWS certifications (e.g., Databricks Data Engineer Professional, AWS Data Engineer Associate)
- Sector Experience: Experience on federal or public-sector projects
Who This Role Is Not For :
- DevOps or platform engineers whose DMS/Databricks work is mostly infrastructure, provisioning, or CI/CD, not building pipelines.
- Reporting or BI developers whose main work is dashboards, Power BI, or report-layer SQL.
- Azure-first engineers who would need to learn AWS on the job.
- Glue-only engineers whose AWS pipeline experience is mainly AWS Glue, without strong DMS and Databricks.