I have made restrictions for users based on departments (user from one department can work only on tasks that are from that department).

They can't access to tasks for which they don't have rights (tasks are from other department).
But, if users change view in Task Filter to "All Tasks" he can see all tasks that exist in the Company. If they try to access them they can't because they are in some other departments.

Could it be possible that Task Filter shows only tasks for which users have rights (to view, edit...), and that they don't see tasks for which they don't have rights.