Hi Chaitanya,
You have to remove general task and have to select the agent determination as 'General forwarding not allowed'. Then you can restrict the agents based on position/Job/Org unit/Role.
In your case Rule is also a best option. If no agents found, rule will be terminated and will set the workflow status as an error.
Thanks,
Vijay.