Taskmaster fans can track down the iconic house — but visiting it is another story.