Here i have found same problem in gentoo. With kernel 4.4.6 and 4.8.0 https://www.reddit.com/r/btrfs/comments/4qz1qd/problems_with_btrfs_quota/