Scraping email addresses from remote file servers with FTP scripts

Remote file servers can contain customer exports, supplier records, event registrations, or old campaign lists. An FTP script may make those files easy to retrieve, but access alone does not create permission to collect or use the addresses inside them. In Australia, an apparently simple automation job can raise privacy, marketing, security, and contractual issues.

A responsible workflow begins with written authorisation, a clear business purpose, and controls around storage and deletion. The technical goal should be to process approved files efficiently, rather than harvest exposed directories or build unsolicited mailing lists.

Define authorised access before writing code

Confirm who owns the FTP or SFTP account, which directories may be accessed, and what file types are in scope. A signed instruction should identify the source, collection period, permitted fields, retention period, and people allowed to view the output. Publicly reachable files should still be treated as private if their contents indicate that they were intended for a restricted audience.

Avoid guessing passwords, bypassing access controls, testing random server paths, or downloading entire directories “for later review”. Those actions can turn a legitimate data task into unauthorised access or a security incident. Use a named service account with the minimum required permissions and log each connection.

Australian organisations should also check whether the Privacy Act 1988 and the Australian Privacy Principles apply. Email addresses can be personal information when they identify an individual, and overseas cloud storage may create additional disclosure obligations.

Choose secure transfer methods

Plain FTP sends credentials and data without encryption. Prefer SFTP over SSH or FTP over TLS, with certificate or host-key verification enabled. A firewall allowlist, a restricted source IP, multi-factor authentication where available, and separate credentials for testing and production reduce the damage caused by a compromised account.

A safe script should retrieve only approved filenames, stop when a size or file-count limit is reached, and write an audit record containing the timestamp, account, path, checksum, and result. It should never place usernames, passwords, or private keys directly in source code or forum posts.

The same discipline applies to staging files. Encrypt temporary storage, restrict access by role, and remove downloaded copies when processing is complete. A Melbourne agency handling a client’s event registrations, for example, should not leave a CSV in a shared downloads folder accessible to every contractor.

Inspect files without exposing the data

Before extracting addresses, validate the file type and scan it for malware. Treat CSV, XLSX, XML, JSON, and compressed archives as untrusted input. Reject unexpected extensions, decompression bombs, oversized rows, and formulas that could execute when a spreadsheet is opened.

Parse data into a controlled schema such as address, source, consent status, collection date, and suppression status. Normalise whitespace and letter case for matching, but preserve the original value in a restricted audit record when it is necessary for investigation. Do not publish sample addresses in support threads or send live records to third-party debugging services.

A useful workflow separates discovery from collection: list approved files, compare their names and checksums against the job specification, download only the required versions, then process them in an isolated workspace. This limits accidental exposure when an old backup contains more information than expected.

Method	Suitable use	Main risk	Practical control
Plain FTP	Legacy systems under controlled migration	Credentials and files can be intercepted	Replace with SFTP or TLS-protected FTP
FTP over TLS	Systems that cannot support SFTP	Weak certificate validation or legacy settings	Verify certificates and disable obsolete protocols
SFTP	Routine authorised transfers	Stolen keys or excessive account access	Use key rotation, IP restrictions, and least privilege
Manual browser download	One-off, documented retrieval	Human error and missing audit trails	Use approved accounts and record each file
Public directory scraping	Generally unsuitable	Unauthorised collection and privacy harm	Do not proceed without explicit owner permission

Remove duplicates and honour consent

Deduplication should use a carefully controlled normalised representation rather than sending addresses to an external matching service. Consider plus-addressing, Unicode characters, aliases, and shared inboxes such as accounts@ or reception@. A match is a data-quality decision, not proof that a person consented to marketing.

Maintain a suppression list containing unsubscribed addresses, complaints, invalid destinations, and records that must not be contacted. Apply it before any campaign export, including campaigns aimed at businesses in Sydney, Brisbane, Perth, or regional areas. A previous opt-out should remain effective even when a new FTP file contains the same address.

The Spam Act 2003 regulates commercial electronic messages in Australia. Commercial email generally requires consent, accurate sender identification, and a functional unsubscribe mechanism. Purchased, scraped, or inherited lists are especially risky because their origin may not demonstrate consent.

Keep collection aligned with Australian practice

Australian recipients often interact with businesses across AEST, ACST, and AWST, so scheduling should use the recipient’s likely local time rather than the server’s timezone. A list collected from a Brisbane trade event may include contacts from Adelaide or Darwin, and a single “best send time” can create poor experiences across those regions.

Local market context matters as well. A small trades business in Newcastle may use a shared office inbox, while a national retailer may operate several role-based addresses. Do not assume that a business-domain address is automatically safe to target or that a public listing grants permission for unrelated promotions.

For organisations covered by the Privacy Act, document the collection purpose, access decisions, retention period, and response process for privacy enquiries. If a downloaded file is exposed, investigate promptly and assess whether the Notifiable Data Breaches scheme is relevant.

Build defensive checks into FTP scripts

Automation should fail safely. Use allowlisted hostnames and directories, maximum download sizes, connection timeouts, retry limits, and a dry-run mode that reports intended actions without retrieving data. Alert an administrator when a directory suddenly contains an unusual number of files or a file is much larger than historical versions.

A robust process can follow this sequence:

1. Authenticate with a restricted account and verify the server identity.
2. List only the approved directory and match an allowlisted filename pattern.
3. Download to encrypted temporary storage with a strict size limit.
4. Scan, parse, validate, and quarantine malformed files.
5. Deduplicate against consent and suppression records.
6. Export only the minimum fields needed for the approved purpose.
7. Delete temporary material according to the retention policy.

These controls are useful for legitimate list maintenance, migration, and compliance audits. They should not be adapted to probe servers, evade monitoring, or collect addresses from files that the owner did not authorise.

Document provenance and accountability

Every usable address should have a recorded source, collection date, lawful or consent basis, and processing purpose. If a vendor supplied the file, retain the contract and its assurances about collection practices. If the data came from a customer relationship system, identify the system owner and the campaign or operational purpose that permits its use.

Review access logs regularly and revoke credentials when a contractor, agency, or employee no longer needs them. Keep production credentials separate from development environments, and redact addresses from screenshots, bug reports, and forum discussions.

Recommended safeguards for compliant workflows

• Use SFTP or properly configured FTP over TLS instead of plain FTP.
• Obtain written permission covering directories, fields, purposes, and retention.
• Store credentials in a secrets manager and restrict the service account.
• Apply suppression, consent, and unsubscribe records before any outreach.
• Log filenames, checksums, access times, processing results, and deletions.
• Encrypt temporary files and remove them when the approved task ends.
• Review Australian privacy and spam obligations before sending commercial messages.

Handle incidents and disputes quickly

If a script downloads an unexpected directory, encounters personal records outside scope, or exposes credentials, stop the job and preserve relevant logs. Do not continue collecting data to determine how much was available. Notify the system owner, rotate affected credentials, isolate downloaded files, and follow the organisation’s incident procedure.

A clear provenance trail helps distinguish an authorised transfer from an accidental or abusive harvest. It also supports deletion requests, client audits, and investigations into complaints. In Australia, prompt assessment is important where a data breach may cause serious harm.

Remote file servers can support reliable marketing operations when access is narrow, transfers are protected, and every address has a defensible origin. The script is only one part of that system; permission, data minimisation, consent management, and accountable handling determine whether the resulting workflow is responsible.