If you can have a relatively stable conference staff, new people don't have to jump in and take over a large conference, they can join an existing team and get their feet wet, then move up in responsibility (letting the earlier people relax a bit or do other things they are interested in.
As for conference size, it's a lot easier to add a few rooms, or even a day to an existing conference than it is to set a new one up from scratch. You always get people saying "this was good, but if just added this one thing it would be even better"
Also, if you say you want to not have a big conference, how do you do it? tell people walking up to registration that they aren't allowed to attend?