Maximizing and Isolating computer resources across physical hardware systems as been around for a while.
Recently though we have seen Open Source systems having become the corner stone to operate and run distributed systems (over your operating system).
i.e. Apache Mesos & Apache Hadoop YARN
A goal is to provide a sufficient amount of functions and abstractions so that the Kafka server can launch within those and the management of the brokers, producers and consumers (including configurations) are manageable and workable by them.
Another goal is to not introduce any dependencies on brokers, producers and consumers to be able to-do this.
The work for this is being done in https://issues.apache.org/jira/browse/KAFKA-1206