Senior Data Engineer
We are looking for a Senior Data Engineer for our client in Rotterdam. If you’re looking for an opportunity with a scale-up growing company, have knowledge of and experience with Big Data, Apache Spark and Machine Learning, then we are looking for you!
Our client develops unique tools to make markets transparent fast. They gather and maintain data from diverse public places, which is valuable information to their clients in order to achieve their business goals. With their own tool and SaaS marketing platform, they are unique in the field of sales and marketing.
Job description Senior Data Engineer
As a Senior Data Engineer, you will be part of the “data” team that is responsible for continuously developing and deploying tools for the big data pipeline. The technology stack includes Spark, MLlib, Scala, Java, Python, Airflow, Hadoop, Parquet and different database solutions (SQL and NoSQL) running in the cloud. They use a microservices and containerization (Docker, Kubernetes) approach to develop new solutions. You will develop, maintain and improve the big data pipeline within the scope of:
- data ingestion (crawlers, external API’s, data dumps)
- stream/batch processing the newly ingested data (Kafka, Camel)
- storing the raw data (Parquet, Hadoop)
- cleaning/parsing/matching/combining the data (Spark, MLlib, SQL)
- storing the cleaned data (Elasticsearch, Cassandra, Neo4j)
- disclosing the data via secure API’s ready to use by our internal products and external partners.
- security, authentication, and authorization
- distributed systems, microservices, databases, and NoSQL technologies
Skillset Senior Data Engineer
- You have a bachelor’s or master’s degree (e.g. Computer Science, Software Engineering)
- You have good knowledge of Java, Scala and/or Python
- You have experience with Apache Spark
- You have good SQL & NoSQL skills
- You have good communication abilities and capacity for teamwork
- You have excellent skills in writing and speaking in English
- Experience with Scrum
- Experience with developing Big Data ETL pipelines
- Experience with Cloud Platforms, like GCP, AWS or Azure
- Experience with CI/CD tools
- Foundation of Machine Learning and basic algorithms.
Compensation Big Data Software Engineer
- You will work in an energetic and inspiring working environment. A brand new office in the city center of Rotterdam.
- Good salary, depending on your experience
- Office presence and homework can be combined
- Small entrepreneurial team
- Lots of technical freedom