I think this is a good area for discussion since there is more ambiguity found in the use of complex abstractions than there should be. Too often we use or accept the usage of these concepts without being sufficiently critical.
Although I agree with the point Rand is trying to make when she said, "Since there is no such entity as 'society,' since society is only a number of individual men, this [social duty] has meant, in practice, that the rulers of society were exempt from moral law..." I disagree with her wording.
In Introduction to Objectivist Epistemology, Rand acknowledges the different uses of the word "entity" and states that she uses the word the way Aristotle did, as a metaphysical category where each entity is the subject of perception and possesses attributes, characteristics and actions. (However, in other parts of the book she mentions "mental entities.")
Through out that Man's Rights article she uses the word society and clearly does so as if it is a meaningful word. So, I'm left believing that she sees "society" as not perceptual and therefore not an entity, but that it is valid as a concept of something that exists.
In the article on Man's Rights, Rand is addressing the notion of "collective rights" which allow some men to claim a right to dispose of others or their property. She says that nothing can justify such a concept, not the mysticism of "Divine rights of Kings," or the "social mystique of modern collectivists who see society as a super-organism, as some supernatural entity apart from and superior to the sum of its individual members."
It would have been better if she had NOT said that there is no such entity as "society." She could have made her point by saying, "There is no such thing as a society apart from the individuals that make it up."
Your example sentence, "When a crime occurs, society demands justice," is a good one to examine. If either of us were to hear or read that we would be very cognizant of who the author is and/or the context in which it is said, because we would know that there can be different ideas as to what 'demands' imply, to what it is proper for society (or anyone else) to demand, and that 'justice' means different things to different people. And even 'crime' is not the same from one political orientation to another. But if we make some changes to that sentence to give a tighter context, as in "When an individual initiates violence against another, American society demands the criminal be brought to justice," then we are more likely to be able to agree or disagree in a way that makes sense. The idea here is to keep increasing the precision of the context until we are able to avoid ambiguity.
Regarding "God" as a floating abstraction. I think that there can be an assertion of a metaphysical realtiy and that could keep this from being a floating abstraction. Even if anchored in the believer's mind with concrete specifics, it would be an error of fact, and the error in fact is the product of a using faith on scripture or revelation rather than the use of reason and logical evidence. From Objectivism Online we see these views of what a floating abstraction is: One poster says it means a concept that a man acquires from others but without knowing what specific units the concept denotes. Another person writes, "For instance, most people will say they are for "freedom" or "justice," but if you ask them what they mean by those words, they kinda, sorta know what they mean and they feel that they are somehow good things. THAT's what is meant by a "floating abstraction." And, there is a quote from OPAR p. 96 “A floating abstraction is not an integration of factual data; it is a memorized linguistic custom representing in the person's mind a hash made of random concretes, habits, and feelings that blend imperceptibly into other hashes which are the content of other, similarly floating abstractions”.
The bottom line seems to be that to be a floating concept isn't as much an attribute of a particular concept, as it is about how a concept is held (with the understanding that some concepts will lend themselves to this better than others, and some concepts are more frequently found as being held this way).
You said, "You are always in society, unless you leave it, like exiting a walled city, or something." Not really. You are part of a Numismatic society - that is a group of individuals that are united by their on-going participation in numismatic activities. You leave that society when you cease to have an interest in, or to participate in those activities. Some societies are more formally defined than others. Sometimes we are making statistically valid statements. For example, if I say, "American's believe that criminals should be locked up," that is a statement about 'our society' that would be considered true even if you found a single individual who was a pacificist or didn't believe in incarceration. Society can't be some 'super-organism' as Rand mentioned, but they can be a category we create that addresses significant majorities relative to the context.
Michael, all knowledge is hierarchical in the sense that every concept must fit within some hierarchy. You mentioned some different symbolic/organizational structures (trees, fishbones, clouds, entity-action, flowchart, etc.) Those all are contained within the category of visual organizational structures. And each has unique components contained within them. That is the nature of a hierarchy. There is no requirement that a entity-action diagram be a container to a flowchart, or that it be inside of the flowchart container - they can be siblings inside of a common container.