It is currently (Alteryx Server 2018) not possible to delete Districts one created - we can only disable them. See also: https://community.alteryx.com/t5/Alteryx-Connect-Gallery/Deleting-Gallery-Districts/m-p/32684
Frankly, this is rather inconvenient if one wants to restructure a Gallery. Renaming them is an option, but there might still be a number of districts we want to delete. Would appreciate if this could be considered for a future update.