Takaki CH. 3: Question

Why is it that both whites and blacks were slaves, yet white slaves still held a higher status in society than blacks? Shouldn’t being a slave or a servant be grounds for receiving the same treatment?