About Skarnode

Skarnode is building the world’s first platform to quantify narrative-driven risk. We transform millions of daily news articles into real-time volatility indexes that help banks, insurers, and corporations anticipate geopolitical, regulatory, and reputational shocks before they hit markets and supply chains.

We are looking for a senior technical leader to design, build, and scale our data ingestion and real-time processing infrastructure. This is a high-impact role at the heart of our mission to become the benchmark for narrative volatility worldwide.

Event-Driven Architecture

• Expertise in Apache Kafka (or equivalent) and Apache Flink for high-performance streaming and distributed real-time processing.
• Must handle millions of events per day with fault tolerance and horizontal scalability.

Web Scraping & Data Ingestion

• Large-scale distributed scraping using headless browsers (Playwright, Puppeteer).
• Anti-bot strategies including proxy pools, IP rotation, fingerprinting, and quota management.
• High-performance orchestration and parallelization of scraping tasks.

Backend Engineering

• Expertise in Python, Django Rest Framework, Go, Rust, C++.
• Experience designing REST APIs, GraphQL, WebSockets, and microservices at scale.

Frontend

• Proficiency in JavaScript (ES6+), Vue.js, React, or equivalent frameworks.
• Building real-time dashboards and interactive client interfaces.

Databases & Storage

• Strong experience with PostgreSQL, Redis/Valkey, and S3-compatible object storage.
• Managing massive data pipelines and daily snapshots.

Cloud & Infrastructure

• Docker and Kubernetes for deployment and scaling.
• Hybrid on-premise/cloud environments, multi-region architecture, and failover strategies.

MLOps & NLP

• Experience with PyTorch, multilingual embeddings, dynamic clustering, and Vector Similarity Search (VSS).
• Building production ML pipelines with full lifecycle management.

Real-Time Data Pipeline

• Design and deploy distributed web scraping systems using Playwright and Puppeteer.
• Reverse-engineer complex APIs via HAR file analysis.
• Orchestrate data streams with Kafka + Flink + Redis.
• Implement retry logic, failure handling, and large-scale resilience.

Real-Time Skarnode Metrics

• Implement mathematical and statistical models to calculate volatility indexes in real time.
• Experience in financial mathematics is a strong plus.

Client Delivery Layer

• Build secure REST APIs, GraphQL, and WebSockets for real-time client access.
• Manage enterprise-grade authentication and permissions.

Infrastructure & Reliability

• Oversee CI/CD pipelines, monitoring, and observability tools.
• Build internal dashboards to track pipeline performance and detect anomalies.

Team Leadership

• Recruit and mentor a high-performing team across:
• Backend & frontend engineering
• Data engineering & streaming
• Machine learning & NLP
• Web scraping & parsing
• Define technical standards, enforce code reviews, and set best practices.

1. Build a fully automated, resilient ingestion system capable of monitoring hundreds of sources in parallel.
2. Ensure real-time processing scalability using Kafka, Flink, and Redis.
3. Guarantee data quality, security, and compliance with GDPR and international standards.
4. Develop internal tools for scraping management, proxy monitoring, error logs, and bot detection.
5. Deploy advanced ML models to produce Skarnode’s volatility indexes in production.

• Proven expertise in data streaming, advanced web scraping, and real-time processing.
• Deep knowledge of distributed architectures and messaging systems (Kafka, Flink).
• Strong leadership skills and experience building and scaling teams.
• Fluent English required; Chinese or Russian are a major plus.