There was a bug in Mac OS X 10.5 “Leopard” that prevented proper application of MCX settings to an Active Directory group nested inside an Open Directory group. This problem has been corrected in 10.6 “Snow Leopard,” but it’s important to note that this is a client-side issue as well that requires you to upgrade all machines to Snow Leopard in order to have the settings properly apply.
The problem evidences itself in the following way: under Apple’s Magic Triangle guidelines, the proper method for access management on the desktop is to “nest” AD users and groups within OD groups, and then to apply settings to those OD groups. This allows for user management of AD users on any Macs they log into, while avoiding the risk of extending the AD schema itself. For the most part, this worked correctly in Leopard, except on AD groups nested in OD groups when applied to Computer groups within the OD. For example, placing a user AD\joeuser into an OD group called banned_users and then denying the banned_users group login access to the LabComputers OD group would block Joe from logging in, but adding AD\alumni into the same OD group would not prevent login access.
Thankfully, this nesting behavior now works correctly in 10.6. As long as you upgrade your clients as well, you should be able to manage Computer settings just like you’d expect.