📦 Building a Stable Data Pipeline for a Growing Logistics Ecosystem

A logistics company can process enormous amounts of information without having a predictable traffic pattern. Evrone faced exactly this situation while helping a logistics business improve the way its systems handled incoming B2B data.

Apache NiFi was responsible for transforming external information before loading it into the database. The difficulty appeared when different customers produced radically different workloads. Thousands of small events could arrive from one source, while another partner could upload a very large JSON file.

The database did not always handle these bursts smoothly. Synchronous processing could trigger background operations, slow other workloads, and even interfere with systems such as 1C.

🛠 Evrone's approach

Evrone introduced an intermediate Python service and Kafka-based data bus.

The processing chain became:

1. NiFi prepares and routes messages.
2. Kafka stores them in the appropriate topic.
3. The Python service collects messages in batches.
4. Business rules determine the processing logic.
5. The database receives the resulting data in a controlled flow.

This design helped distribute workload more evenly and created a foundation that can scale with increasing data volumes.

🚚 From Data Processing to Shipment Tracking

The collaboration continued beyond the initial architecture.

Evrone helped integrate a new customer by creating mapping scripts that transformed external data into the client's internal format. Later, the team developed a public shipment tracking application.

The tracker used Python and Django on the backend and React on the frontend. Evrone reused the existing data bus to receive shipment events from the back-office system.

The tracking service also received its own database. This separation reduced dependency on core systems and gave the new application an independent operational boundary.

The frontend included tables, expandable lists, modal windows, and automatically generated TypeScript types.

🔮 What comes next?

Evrone is preparing an authenticated tracking solution with:

• personal accounts;
• legal-entity information;
• supporting documents;
• detailed shipment history;
• timestamps;
• access for customers and their assigned managers.

The project shows how a carefully designed intermediate data layer can become useful far beyond its original purpose.

⚡ Evrone, Kafka, Python and the Challenge of Uneven ETL Workloads.