The IRMOS realtime scheduler

AFAIU, you are referring to the problem of how exactly to choose proper scheduling parameters in order to have a complex real-time application meet all of its deadlines, and what is an exact admission control scheme that ensures that all of the reserved tasks (or task groups) are granted their reserved resources (in presence of possible deadlines different from the periods, blocking times, offsets, non-preemptible sections, etc.).
These two problems need to be tackled similarly when using SCHED_DEADLINE and the IRMOS scheduler: one needs to use proper (and complex) analysis techniques in order to make a strong assessment about the capability of the system to have all real-time applications respect their timing constraints. That said, it is always possible to take simplified approaches, like requesting to the kernel an allocation with period equal to the WCET + max-blocking-time, and period equal to the minimum between the relative deadline and the minimum job inter-arrival time. More complex and precise admission control algorithms may be prohibitive for being coded in the kernel, however some user-space middleware (daemon + application library) might implement them.
The hierarchical scheduling capability of the IRMOS scheduler implies only a little difference w.r.t. the mentioned problem: if an application (or software component) is built of multiple tasks that need a scheduling guarantee as a whole, then the IRMOS scheduler allows for doing that, defining one single reservation with inside all the tasks (where the exact scheduling may still be fine-tuned by using different priorities if needed), whilst SCHED_DEADLINE would force one to use one reservation for each different task, with an increased "fragmentation" of the allocated CPU time and the little risk of having more bandwidth waste due to the need for the necessary over-provisioning. For example, in a recent modification to Jack (http://jackaudio.org/), we conveniently exploited such hierarchical feature by forcing all of the audio real-time (jack client) threads from the various multimedia processes into the same reservation, because the entire audio-processing pipeline needs to complete anyway within the next audio cycle.