Europe is actively investing in sovereign AI infrastructure but infrastructure without accessibility is only half the story. While supercomputers hold enormous computing potential using them often requires specialist knowledge that most researchers and engineers don’t have. CSH researcher Georg Heiler built a tool to change that. Meet dagster-slurm!
Supercomputers are shared, large-scale computing facilities – usually run by national research institutions – capable of handling tasks far beyond what a laptop or standard server can manage: training complex AI models, processing massive datasets, running large-scale simulations. Across Europe, significant public investment is going into expanding this capacity, with initiatives like the EuroHPC AI Factories building GPU-equipped systems that operate under European data governance and are independent of commercial cloud providers.
But if using these systems requires deep specialist knowledge that most researchers and engineers don’t have, the investment remains underutilized.
“AI sovereignty is important, and it’s good to see Europe investing in it. But new infrastructure doesn’t appear overnight. It requires time, capital, and working through significant institutional processes. While those long-term efforts are underway, tools like dagster-slurm can already help us make better use of the systems that exist today,” says Georg Heiler, who is a researcher at the Complexity Science Hub and the Supply Chain Intelligence Institute Austria (ASCII).
In practical terms, this means improving visibility into running workloads, making it easier to connect computing environments that were previously separate, and allowing pipelines to be moved between systems with less manual intervention. The changes are incremental rather than dramatic, but they address everyday obstacles that often make large-scale computing difficult to use in practice.
THE PROBLEM: POWER WITHOUT ACCESSIBILITY
The challenge is a technical one. Modern workflows are typically developed on a laptop or local server, using tools that are flexible and easy to iterate with. Supercomputers, on the other hand, run on entirely different systems – ones that traditionally require writing custom scripts by hand, manually managing software environments, and submitting jobs to remote queues with limited observability. Once a job is running, visibility is limited: it can be difficult to see how much capacity is being used, whether processes are running as expected, or where bottlenecks arise. Transferring a working pipeline (= a sequence of automated data processing steps) from a local machine to a national supercomputer is rarely straightforward.
“This is not a minor inconvenience. It is a genuine barrier that keeps powerful computational resources in the hands of a small group of specialists, while researchers or companies who could benefit enormously from them are left working with more limited alternatives,” says Heiler.
A TOOL TO BRIDGE THE GAP
To address exactly this, Heiler, together with colleagues Hernan Picatto and Maximilian Heß, built dagster-slurm. The open-source tool connects two existing systems: Dagster, a modern data orchestration platform used widely in industry and research, and Slurm, the job scheduling system that manages computing resources on the majority of the world’s supercomputers.
Dagster-slurm is an example of what happens when research expertise meets a concrete real-world problem – the kind of work the Complexity Science Hub was built to support.
The practical effect is that users can write their pipeline once, on their laptop. When more computing power is needed, a single configuration change sends it to a national supercomputer – no code rewriting required. Dagster-slurm handles packaging, file transfer, and job submission automatically, while logs and performance metrics stream back to a unified dashboard. The exact software setup from the user’s laptop is replicated on the supercomputer, which matters for scientific reproducibility. What used to require custom scripting, manual file transfers, and high-performance computing (HPC) expertise now takes a single line of configuration.
WHO IT'S FOR
Dagster-slurm is useful for researchers who need GPU resources for AI and machine learning, research software engineers running scientific data pipelines, and companies looking to use European supercomputing infrastructure for AI applications without requiring deep expertise in cluster administration. The tool is open source, freely available, and actively supported by ASC.
EVERYTHING STARTED WITH A HACKATHON
The idea grew out of a practical necessity. Heiler was working on workflows that needed more computing power than cloud services could offer affordably – particularly GPU access, which is essential for modern AI and increasingly expensive on commercial cloud platforms. European supercomputers have substantial GPU capacity but making it practically usable for software engineers and data scientists required bridging a significant technical gap.
The idea took shape at the EuroCC AI Hackathon 2025, organized by Austrian Scientific Computing (ASC), Leibniz Supercomputing Centre, and Academic Computer Centre Cyfronet AGH, and partners including NVIDIA. The hackathon provided access to some of Europe’s most powerful supercomputers and gave Heiler’s team the time and resources to build and validate a working prototype.
