It depends on the subject, I think.
In history, for example, the HL kids seem to always have a lot more work.
Math and the sciences have huge differences in material and exam difficulty.
In others, the difference is less marked. In my economics class, the HL and SL kids are combined, and the only significant difference is the math, which is pretty easy stuff. In English, the only difference at my school is that HL kids read one extra book. All the assignments are the same, but the rubric differs slightly for HL kids.