Yes. You can nest BBCodes as long as they are different types. You can have UL BBCode and inside it, you can have LI BBCodes.
And No. You can’t nest the same type of BBCodes. You can’t have UL inside another UL. Ordered or unordered list in HTML is allowed, and you can have UL, inside it several LI tags, and in those LI tags, you can have another UL. This will not work in WordPress, and BBCodes written like that will also fail to render properly. This will not change until WordPress fixes the issue.