The said tax shall be paid to the collector of customs of the port or customs district to which said alien shall comes or, if there be no collector at such port or district, then to the collector nearest thereto, by the master, agent, owner, or consignee of the vessel, transportation line, or other conveyance or vehicle bringing such alien to the United States.