How to Download Large Volumes of Data Through a Proxy Without Starting Over After a Disconnect: A Step-by-Step Guide

Introduction: Why Long Downloads Almost Always Break, and Why That's Normal

If you've ever pulled a few million records or a file tens of gigabytes in size through a proxy, you know the feeling. The script ran for six hours, showed 83 percent, and then crashed with a connection error. All you have is a partial file and the realization you have to start from scratch.

The first thing to accept: a long download always breaks. Not "sometimes", not "when the network is bad", but always, if it runs long enough. There are dozens of reasons, and most are outside your control:

• The proxy changes its external IP address. With Proxeon mobile proxies this is standard behavior: rotation on a timer or on demand. The moment the IP switches, the open TCP connection drops, and the source server sees a different client.
• The source server closes the connection on its own timeout, reboots, ships an update, or just returns a 5xx error.
• Your own process restarts: system update, disk full, a code error on an unusual record, an accidental Ctrl+C.
• The auth token expires, the session times out, the pagination cursor goes stale.
• The laptop goes to sleep, Wi-Fi switches to another access point, the ISP changes the route.

Fighting each of these causes individually is pointless. The right approach is different: design the download so that a break at any moment costs you one page or one chunk of a file, not six hours. That's what this guide is about.

What You'll Get Out of It

After going through this tutorial, you'll have:

• A working function to resume a file download from the middle using the HTTP Range header, with a check that the server supports it.
• A clear checkpoint scheme for paginated exports: exactly what to save and where to store state so it doesn't die with the process.
• Idempotent result writes, so re-downloading the same page doesn't produce duplicates.
• Deduplication that doesn't eat all your RAM across millions of rows.
• A parallel downloader with a task queue, retries for failed jobs, and a concurrency limit.
• A ready-to-use skeleton of a resilient Python downloader that you'll adapt to your source in an hour.

Who This Guide Is For

For developers and analysts who already know how to make HTTP requests from Python and have lost the results of a long download at least once. Intermediate level: we explain the basics, but we don't teach programming from scratch. Advanced readers will find sections on storing hashes outside memory and on safe parallel writes to SQLite.

What You Need to Know Beforehand

• Python at the level of functions, loops, dictionaries, and exception handling.
• HTTP basics: what a method, header, status code, and body are.
• A general idea of how to set a proxy in the requests library.

One thing we'll set aside: status codes and retry strategies with exponential backoff are not covered here. That's a separate article about error 429 and retries. This guide focuses on something else: the state of the download and its resumption. Retries answer the question "when to repeat a request"; we answer the question "from what point to continue after retries are exhausted and the process is dead".

How Long It Will Take

Reading and running the examples on a test source: two to three hours. Adapting the skeleton to your real API or file server: another hour or two, depending on how non-standard the pagination is. Total: one working day with room to spare.

Preliminary Preparation and Basic Concepts

Tools and Access

1. Install Python 3.11 or newer. In 2026 the current branches are 3.12 and 3.13, and all examples are verified on them. To check the version: open a terminal and type python --version. If you see 3.11 or higher, you're good.
2. Install the requests library: pip install requests. Version 2.32 or newer is enough. The sqlite3 module is part of the Python standard library, so nothing else needs to be installed.
3. Get proxy access. Open your Proxeon dashboard, pick the channel you need, and copy four values: host, port, login, and password. They're usually combined into one string like http://USER:PASS@HOST:PORT. You'll use this string everywhere below.
4. Put the proxy string in an environment variable, not in your code. On Linux and macOS: export PROXY_URL=http://USER:PASS@HOST:PORT. On Windows PowerShell: $env:PROXY_URL='http://USER:PASS@HOST:PORT'. This way you won't accidentally commit the password to a repo.
5. Verify the proxy responds. Run in your terminal: curl -x $PROXY_URL -I https://api.example.com/, substituting your source's address. You should see a line with a status code, like HTTP/2 200. If you see a proxy auth error 407, double-check the login and password.

System Requirements

Any machine with 2 GB of free RAM and a disk with room for the download result plus 20 percent headroom for SQLite indexes. If you plan to download millions of records, disk matters more than memory: the whole approach is built on state living on disk, not in process variables.

Backups

The state file you'll create below (in the examples it's export.sqlite) becomes the most valuable artifact of the whole job. Make it a habit to copy it before any code experiments: cp export.sqlite export.sqlite.bak. Once, it'll save you a day of downloading.

Note: never edit the SQLite file by hand while the downloader is running. Even reads from a third-party program in the wrong mode can lock writes and crash the process. If you need to inspect state, stop the downloader or use WAL mode, which we'll cover in the parallelism section.

Key Terms in Plain Language

• Checkpoint — a saved marker on disk that says "everything up to this point has been downloaded and written". After a break, the downloader reads the checkpoint and continues from it.
• Cursor — an opaque string the API returns along with a page, which you must pass to get the next page. You don't build or parse the cursor yourself.
• Offset-based pagination — you ask for "page 37, 500 records each". Simple scheme, but when new records are added to the source, pages shift and you get duplicates or gaps.
• Keyset pagination — you ask for "everything with ID greater than 184203, sorted by ID". The most resilient scheme for resumption if the source supports it.
• Idempotency — a property of an operation where repeating it gives the same result as doing it once. You wrote a page twice, but it's stored once.
• Deduplication key — a value by which two records are considered the same. Ideally it's an ID from the source; if there isn't one, the key is computed as a hash of stable fields.
• Range header — a way to ask an HTTP server to send part of a file instead of the whole thing, for example bytes 1048576 onward.
• At-least-once semantics — a guarantee that every record is received at least once. Duplicates are possible, gaps are not. That's exactly what you'll get after this guide, and deduplication removes the duplicates.

The Core Principle

All seven steps below boil down to one idea: every unit of work must be atomic and repeatable. A unit of work is either a chunk of a file, a page of an API, or a task from the queue. Atomic means the result and the marker of its completion are saved together. Repeatable means that if you run the unit twice, nothing breaks. When both properties hold, a break at any point becomes safe.

Step 1: Resuming a File Download over HTTP with the Range Header

Goal of this stage: learn to download a large file through a proxy so that after a break, the download continues from the byte where it stopped, not from scratch.

How It Works

HTTP lets a client request part of a resource. To do this, you add a Range: bytes=START- header to the request, where START is a byte offset. If the server supports partial requests, it responds with 206 Partial Content and a Content-Range: bytes START-END/TOTAL header. If it doesn't, it ignores Range and sends the whole file with status 200. Your job is to tell these cases apart.

To find out in advance whether a server supports resumption, a HEAD request helps: it returns only headers, no body. Look at Accept-Ranges: bytes. A value of none or a missing header usually means no resumption, although some servers still handle Range correctly, so the final check is the status code.

Step-by-Step Instructions

1. Make a HEAD request through the proxy and save the Accept-Ranges, Content-Length, ETag, and Last-Modified headers. You'll need the ETag to tell whether the file changed on the server between your attempts.
2. Check how many bytes are already in the local file. If the file doesn't exist, treat it as zero.
3. If the local size already equals Content-Length, the file is complete, nothing to do.
4. If the local size is greater than zero and the server claims to support ranges, add a Range header with the current size. Also add an If-Range with the saved ETag: then the server will send a partial response only if the file hasn't changed, otherwise it returns the whole file with status 200.
5. Always send Accept-Encoding: identity. Without it, the server may apply on-the-fly compression, and byte offsets will no longer match your file.
6. Open the local file in append mode ab if you got 206, or overwrite mode wb if you got 200.
7. Read the body as a stream in 256 KB chunks and write to disk. Don't load the whole response into memory.
8. After finishing, compare the final size with Content-Length. If it doesn't match, the connection broke silently and you need another round.

Working Code

import os
import requests

PROXY_URL = os.environ['PROXY_URL'] # строка из личного кабинета Proxeon
PROXIES = {'http': PROXY_URL, 'https': PROXY_URL}


def probe(url):
r = requests.head(url, proxies=PROXIES, allow_redirects=True, timeout=30,
 headers={'Accept-Encoding': 'identity'})
r.raise_for_status()
return {
'ranges': r.headers.get('Accept-Ranges', 'none').lower(),
'length': int(r.headers.get('Content-Length', 0) or 0),
'etag': r.headers.get('ETag'),
}


def download_resumable(url, path):
meta = probe(url)
have = os.path.getsize(path) if os.path.exists(path) else 0
if meta['length'] and have >= meta['length']:
print('файл уже полный:', have, 'байт')
return True
headers = {'Accept-Encoding': 'identity'}
if have > 0 and meta['ranges'] == 'bytes':
headers['Range'] = 'bytes=%d-' % have
if meta['etag']:
headers['If-Range'] = meta['etag']
with requests.get(url, headers=headers, proxies=PROXIES, stream=True,
 timeout=(30, 120)) as r:
if r.status_code == 206:
expected = 'bytes %d-' % have
if not r.headers.get('Content-Range', '').startswith(expected):
raise IOError('сервер отдал не тот диапазон: ' + r.headers.get('Content-Range', ''))
mode = 'ab'
elif r.status_code == 200:
print('сервер отдаёт файл целиком, начинаем с нуля')
mode = 'wb'
have = 0
elif r.status_code == 416:
raise IOError('запрошенный диапазон вне файла, проверьте локальный размер')
else:
r.raise_for_status()
with open(path, mode) as f:
for chunk in r.iter_content(chunk_size=256 * 1024):
if chunk:
f.write(chunk)
have += len(chunk)
if meta['length'] and have != meta['length']:
print('обрыв: получено %d из %d' % (have, meta['length']))
return False
return True


def download_until_done(url, path, max_rounds=50):
for i in range(max_rounds):
try:
if download_resumable(url, path):
return
except (requests.ConnectionError, requests.Timeout, IOError) as e:
print('заход %d прерван: %s' % (i + 1, type(e).__name__))
# пауза перед следующим заходом: стратегия описана в статье про 429 и ретраи
raise RuntimeError('не удалось докачать файл за %d заходов' % max_rounds)


if __name__ == '__main__':
download_until_done('https://files.example.com/export-2026.csv.gz', 'export-2026.csv.gz')

Note the download_until_done function: it contains no wait logic between attempts. That's intentional. Drop in your pause strategy from the retries article there; here only the loop "checked size, resumed, checked again" matters.

Tip: if the file is served as an archive, don't unpack it on the fly while resuming. First get the full file, check the size, and if the server provides a checksum, verify it. Only then unpack. A partially downloaded gzip looks corrupted, and you'll waste time hunting a nonexistent error.

Expected Result

Verification: run the script on a file at least 200 MB in size, and after ten seconds interrupt it with Ctrl+C. Check the local file size, say 41 943 040 bytes. Run the script again. You should not see the "starting from scratch" line, and the file size should keep growing rather than reset. On completion, the final size should exactly match the Content-Length from the HEAD request.

Possible Problems

• The server always returns 200 instead of 206. That means resumption isn't supported. The only way out for such a source is to download the whole file in one go with a large timeout, or look for an alternative export format in parts, such as splitting by dates.
• No Content-Length header. The server sends the file in chunked mode without declaring the size. You can't verify completeness by size, and you can't resume either: Range requires known offsets. Work it out with the source or use a checksum if one is published.
• Response 416 on the very first attempt. The local file is larger than the file on the server. The file on the server changed and got shorter. Delete the local file and start over.
• Size matches but the file is corrupt. Most likely somewhere in the middle there was a break with status 200 and an overwrite from zero, then an append. Recreate the file. To prevent this, store the ETag in a separate file alongside and compare before every round.

Step 2: Checkpoints for Paginated Exports

Goal of this stage: save download state so that after any crash, the process continues from the last successfully written page.

What to Save

The minimal checkpoint depends on the source's pagination type. Let's look at three cases.

1. Cursor-based pagination. The API returns a field like next_cursor along with the data. Save exactly that. This is the simplest case: the cursor already contains everything the server needs to continue.
2. Page number or offset pagination. Save the number of the last fully written page and the page size. Remember that when records are added to the source during the download, offsets shift, so deduplication from step 4 is mandatory.
3. Keyset pagination. Save the ID of the last written record. On resume, you request everything greater than that ID. This scheme is immune to both inserts and long pauses.

Regardless of pagination type, it's worth adding service fields to the checkpoint: the last record ID (even for the cursor scheme, as a fallback anchor in case the cursor expires), page and row counters for progress tracking, the download start time, and the last update time.

Where to Store State

There are two workable options, and both are better than variables in memory.

Option A: JSON File with Atomic Replacement

Suitable if results are written to separate files rather than a database. The main trap: if you write state directly into the target file and the process crashes mid-write, you get a truncated JSON that won't parse. The solution is to write to a temporary file next to it and rename it over the main one. A rename operation within the same filesystem is atomic.

import json
import os
import tempfile


def save_state(path, state):
directory = os.path.dirname(os.path.abspath(path))
fd, tmp = tempfile.mkstemp(dir=directory, prefix='.state-')
with os.fdopen(fd, 'w', encoding='utf-8') as f:
json.dump(state, f, ensure_ascii=False)
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)


def load_state(path, default):
if not os.path.exists(path):
return dict(default)
with open(path, encoding='utf-8') as f:
return json.load(f)

Option B: A Table in SQLite Next to Your Data

The preferred option if you store records in a database. The checkpoint is updated in the same transaction as the page rows insert. Either both the data and the marker are written, or neither. There's never a gap where "data is there but the marker isn't".

import sqlite3

con = sqlite3.connect('export.sqlite')
con.executescript('''
CREATE TABLE IF NOT EXISTS records(
id TEXT PRIMARY KEY,
payload TEXT NOT NULL,
fetched_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS checkpoint(
job TEXT PRIMARY KEY,
cursor TEXT,
last_id TEXT,
pages INTEGER NOT NULL DEFAULT 0,
updated_at TEXT
);
''')


def commit_page(job, rows, next_cursor, pages, ts):
with con: # одна транзакция на страницу
con.executemany(
'INSERT OR IGNORE INTO records(id, payload, fetched_at) VALUES (?, ?, ?)',
[(str(r['id']), json.dumps(r, ensure_ascii=False), ts) for r in rows])
con.execute(
'INSERT INTO checkpoint(job, cursor, last_id, pages, updated_at) VALUES (?, ?, ?, ?, ?) '
'ON CONFLICT(job) DO UPDATE SET cursor=excluded.cursor, last_id=excluded.last_id, '
'pages=excluded.pages, updated_at=excluded.updated_at',
(job, next_cursor, str(rows[-1]['id']), pages, ts))

Order of Operations

Remember the rule: data first, checkpoint second, ideally in one transaction. If a transaction isn't available (say, data is written to files), the order is exactly that: wrote the page file, synced to disk, then updated state. If it crashes between these two actions, you'll re-download one page, which is safe thanks to step 3. The reverse order means skipping a page, and that's data loss.

Tip: store in the checkpoint the cursor of the next page that the server returned, not the current one. Then on resume you immediately request what you don't have yet, without an extra request for a page you already got.

Expected Result

Verification: start the download, wait for ten pages, and force-kill the process. Open the database with sqlite3 export.sqlite and run SELECT pages, last_id FROM checkpoint;. You should see the number 10 and an ID. Then run SELECT count(*) FROM records; and make sure the row count equals ten page sizes. Run the downloader again: the first console message should be something like "start: pages 10".

Possible Problems

• "Database is locked" error. Another process holds a connection. Close all sqlite3 windows and other tools that opened the file. For multithreaded work, enable WAL mode, see step 5.
• Cursor saved but data didn't. You updated the checkpoint outside the transaction with the data. Go back to the code above and make sure both operations are inside a single with con: block.
• The JSON state ended up empty or corrupt. You wrote directly to the file without the temporary file and replace. Use the save_state function in full.

Step 3: Idempotent Result Writes

Goal of this stage: make sure re-processing any page doesn't produce duplicates or corrupt data.

Why Repeats Are Inevitable

After step 2, you've already seen the scenario where a page gets written twice: the process crashed after inserting the data but before updating the checkpoint. On top of that, repeats come from offset pagination when the source changes, from parallel workers getting the same task after a restart, and just from a manual "just in case" restart. Fighting repeats on the request side is pointless. The right move is to make the write itself such that a repeat is harmless.

Choosing a Deduplication Key

1. There's an ID from the source. Use it. This is an id, uuid, order_number, or similar field that the source guarantees is unique. If you're pulling from multiple sources into one table, make it a composite key: source name plus ID.
2. No ID, but there's a set of fields that together identify a record. For example, for a price list row, that's SKU plus warehouse plus date. Build the key from these fields, normalizing them: lowercase strings, strip whitespace, convert dates to a single format.
3. Nothing stable at all. Then the key becomes a hash of the entire record after canonicalization. This case is covered in detail in step 4. Note that if the source updates a record (changes the price), the hash changes, and you'll get both versions. Sometimes that's exactly what you want, sometimes not.

Note: never use the row number in the response or the page number as a key. These values change with any source modification, and deduplication turns into a duplicate generator.

Idempotent Database Insert

SQLite and most relational databases have a construct that either ignores a conflict on the primary key or updates the existing row. The first variant, INSERT OR IGNORE, you saw in step 2. It suits immutable records. The second variant is needed if the source can update records and you want the fresh version:

def upsert_rows(con, rows, ts):
con.executemany(
'INSERT INTO records(id, payload, fetched_at) VALUES (?, ?, ?) '
'ON CONFLICT(id) DO UPDATE SET payload=excluded.payload, fetched_at=excluded.fetched_at',
[(str(r['id']), json.dumps(r, ensure_ascii=False), ts) for r in rows])

Idempotent File Writes

If results must land in files rather than a database, apply the same principle: one page equals one file with a deterministic name. The name depends on the page parameters, not on time or a counter. Before download, check whether the final file exists; if yes, skip the page. Write to a temporary name and rename on completion, as in the save_state function.

def page_path(base_dir, job, cursor_or_page):
safe = str(cursor_or_page).replace('/', '_').replace(':', '_')[:120]
return os.path.join(base_dir, job, 'page-%s.jsonl' % safe)


def write_page_idempotent(path, rows):
if os.path.exists(path):
return False # страница уже есть, повторная запись не нужна
os.makedirs(os.path.dirname(path), exist_ok=True)
tmp = path + '.part'
with open(tmp, 'w', encoding='utf-8') as f:
for r in rows:
f.write(json.dumps(r, ensure_ascii=False))
f.write(chr(10))
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
return True

Files with the .part extension left after a crash can safely be deleted at startup: they're incomplete by definition.

Tip: for offset pagination, don't rely solely on "file exists, so the page is done". Also check that the number of lines in the file equals the page size (except for the last page). An empty or short file under an existing name is better re-downloaded.

Expected Result

Verification: call the write function on the same page three times in a row. Then run SELECT count(*) FROM records;. The number should equal one page size, not triple. For the file variant, there should be exactly one page file in the directory and zero .part files.

Step 4: Deduplicating Results Without Blowing Up Memory

Goal of this stage: filter out duplicate records in a stream of millions of rows without keeping all keys in RAM.

Record Hash

When a record has no ID, the key becomes a hash of its contents. To make identical records produce identical hashes, the contents must be canonicalized: sort dictionary keys, strip extra whitespace, fix separators. Otherwise the same record arriving with a different field order gets a different hash.

import hashlib
import json


def record_key(rec, fields=None):
src = rec if fields is None else {k: rec.get(k) for k in fields}
canon = json.dumps(src, sort_keys=True, ensure_ascii=False, separators=(',', ':'))
return hashlib.blake2b(canon.encode('utf-8'), digest_size=16).digest()

The function returns 16 bytes. That's plenty: the chance of a random collision across a hundred million records is negligible. The fields parameter lets you hash only stable fields, excluding, say, the last-updated timestamp that changes on every request.

Why an In-Memory Set Doesn't Work at Millions of Rows

The first idea that comes to mind: create seen = set() and drop keys in there. Let's do the math. One bytes object of length 16 takes up about 49 bytes in Python plus the data itself, so roughly 65 bytes. A slot in the set, accounting for the load factor, adds about 30 more bytes. That's about 95 bytes per key. For 10 million records, that's about 950 MB; for 50 million, almost 5 GB. And the kicker: after a process restart the set is empty, and all deduplication starts from a clean slate.

Three Ways to Keep Memory Down

1. Store keys in the database itself. The simplest and most reliable path. If the key is the primary key of the records table, deduplication is already done by the INSERT OR IGNORE construct from step 3. The index lives on disk, survives restarts, and SQLite caches hot index pages on its own. For 10 million 16-byte keys, the index takes about 400-500 MB on disk, but not in memory.
2. A separate seen-keys table without rowid. Needed if you write the data itself not to SQLite but, say, to files. Then SQLite is used only as a compact set on disk.
3. A compressed in-memory set as a pre-filter. An advanced option: truncate the hash to 8 bytes and store it as an integer in a sorted array or use a Bloom filter. Memory shrinks several-fold, but a false positive probability appears. So such a pre-filter is used only to quickly cut off obviously new records, and the final check is still done against the database.

Implementing the Disk-Based Set

class DiskSeen:
def __init__(self, con):
self.con = con
con.execute('CREATE TABLE IF NOT EXISTS seen(key BLOB PRIMARY KEY) WITHOUT ROWID')

def filter_new(self, rows):
keyed = [(record_key(r), r) for r in rows]
keys = [k for k, _ in keyed]
placeholders = ','.join('?' * len(keys))
known = {row[0] for row in self.con.execute(
'SELECT key FROM seen WHERE key IN (%s)' % placeholders, keys)}
fresh = [(k, r) for k, r in keyed if k not in known]
# дедупликация внутри самой страницы
unique = {}
for k, r in fresh:
unique.setdefault(k, r)
return unique

def remember(self, keys):
self.con.executemany('INSERT OR IGNORE INTO seen(key) VALUES (?)', [(k,) for k in keys])

Call filter_new before writing a page and remember inside the same transaction as the data write and the checkpoint. Then after a crash, the seen-set, the data, and the progress marker are always consistent with one another.

Tip: a query with IN across 500 values runs against the index in milliseconds. Don't check keys one by one in a loop: that's dozens of times slower due to per-query overhead.

Expected Result

Verification: build a test page of 500 records where 100 repeat twice within the page and 100 are already in the seen table. The filter_new function should return exactly 300 records. After a process restart, the same 500 records should yield zero new ones.

Possible Problems

• Duplicates still get through. Check canonicalization: most likely the records contain a request-time field or a random element order in a list. Exclude it via the fields parameter or sort nested lists before hashing.
• Slow inserts after a few million rows. The index stopped fitting in cache. Increase the SQLite cache with PRAGMA cache_size=-200000 (that's 200 MB) and make sure inserts go in batches within one transaction per page, not one row at a time.

Step 5: Parallelism Without Loss

Goal of this stage: speed up the download with several concurrent workers through a proxy so that any worker crash doesn't lose tasks or corrupt the database.

When Parallelism Works and When It Doesn't

Cursor-based pagination is inherently sequential: the next cursor is only known after receiving the previous page. You can't parallelize it directly. But you can almost always split the download into independent shards: by day, by category, by region, by leading characters of the ID. Each shard is downloaded sequentially with its own checkpoint, and shards run in parallel. Offset pagination and keyset pagination with known boundaries parallelize directly: tasks like "pages 1 through 100" or "IDs from 0 to 100000".

Disk-Based Task Queue

An in-memory queue dies with the process. So tasks live in a table with statuses:

• pending — waiting to be done;
• running — picked up by a worker;
• done — completed and written;
• failed — attempts exhausted, needs a human.

At startup, the downloader first moves all tasks from running back to pending: if they're stuck in that status, the previous process died mid-work. Then workers pull from pending.

One Writer

SQLite allows many concurrent readers but only one writer. The simplest and safest pattern: workers only download and return data, and the main thread does all database writes. No locks in the code, no "database is locked". Additionally enable WAL mode so reading state from another process doesn't block writes.

Limiting Concurrency

Limit the number of workers by two things. First, proxy capabilities: if you have several channels in your Proxeon dashboard, it's reasonable to keep one or two workers per channel, so IP rotation on one channel doesn't tear the connections of all threads at once. Second, politeness to the source: even without formal limits, ten parallel threads against a small API will create load that gets you refusals. Start with three or four workers and raise it while watching the error rate.

Parallel Downloader Code

import json
import sqlite3
from concurrent.futures import ThreadPoolExecutor, as_completed


def init_tasks(con):
con.execute('PRAGMA journal_mode=WAL')
con.executescript('''
CREATE TABLE IF NOT EXISTS tasks(
task_id TEXT PRIMARY KEY,
params TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT
);
''')
with con:
con.execute('UPDATE tasks SET status=? WHERE status=?', ('pending', 'running'))


def enqueue(con, tasks):
with con:
con.executemany('INSERT OR IGNORE INTO tasks(task_id, params) VALUES (?, ?)',
[(t['task_id'], json.dumps(t['params'])) for t in tasks])


def claim(con, limit):
rows = con.execute('SELECT task_id, params FROM tasks WHERE status=? LIMIT ?',
 ('pending', limit)).fetchall()
with con:
con.executemany('UPDATE tasks SET status=? WHERE task_id=?',
[('running', r[0]) for r in rows])
return [(r[0], json.loads(r[1])) for r in rows]


def run_parallel(con, fetch_fn, write_fn, workers=4, max_attempts=5):
init_tasks(con)
with ThreadPoolExecutor(max_workers=workers) as pool:
while True:
batch = claim(con, workers * 2)
if not batch:
break
futures = {pool.submit(fetch_fn, params): task_id for task_id, params in batch}
for fut in as_completed(futures):
task_id = futures[fut]
try:
rows = fut.result()
except Exception as e:
with con:
con.execute(
'UPDATE tasks SET attempts=attempts+1, last_error=?, '
'status=CASE WHEN attempts+1 >= ? THEN ? ELSE ? END WHERE task_id=?',
(str(e)[:500], max_attempts, 'failed', 'pending', task_id))
continue
with con: # данные и статус задания в одной транзакции
write_fn(con, rows)
con.execute('UPDATE tasks SET status=? WHERE task_id=?', ('done', task_id))
failed = con.execute('SELECT count(*) FROM tasks WHERE status=?', ('failed',)).fetchone()[0]
print('очередь пуста, заданий с ошибкой:', failed)

The fetch_fn function makes a request through the proxy and returns a list of records. It runs in a thread and doesn't touch the database. The write_fn function is called in the main thread inside a transaction and performs the idempotent insert from step 3. A failed task is automatically returned to pending and will be picked up again in the next claim cycle; after exhausted attempts it gets the failed status, and you'll deal with it by hand.

Note: don't pass a sqlite3 connection object into workers. A connection is bound to the thread it was created in, and trying to use it from another thread will cause an error or, worse, silent data corruption. Each thread gets either its own connection, or, as in the example above, none at all.

Tip: use a separate requests.Session object per worker with its own proxy address. If you have several channels in Proxeon, distribute them round-robin across workers: worker 0 takes channel 0, worker 1 takes channel 1, and so on. Then a connection drop on one channel only affects one thread.

Expected Result

Verification: enqueue 100 tasks, start four workers, and kill the process after half a minute. Run SELECT status, count(*) FROM tasks GROUP BY status;. You'll see a few done, a few running, and the rest pending. Start it again: running should disappear at startup, and on completion all tasks should be in done, except those that genuinely failed and sit in failed with an error message in last_error.

Step 6: Resuming After a Long Pause

Goal of this stage: correctly continue a download that you stopped for a few hours or days without running into stale state.

What Goes Stale

Resuming after ten seconds and after a week are different tasks. Over a long pause, part of the saved state stops being valid.

1. Sessions and cookies. Server-side sessions usually live from a few hours to a day. Saved cookies after that lead to 401s or a redirect to the login form. Solution: at startup, do a full re-authentication rather than restoring cookies from a file.
2. Access tokens. OAuth tokens live for an hour, sometimes less. If you have a refresh token, refresh the access token before startup and on a schedule during the run, without waiting for a refusal.
3. Pagination cursors. Many APIs limit a cursor's lifetime to minutes or hours. A stale cursor returns an error 400 with a message about an invalid cursor. That's exactly why in step 2 we saved a fallback anchor: the last record ID. If the source supports a filter by ID or by modification date, build a new request from that anchor. If it doesn't, you'll have to restart the shard from the beginning, and deduplication from step 4 will cut off what you already have.
4. File contents on the server. For resumption from step 1, it's critical that the file hasn't changed. Compare the current ETag with the saved one before every round; if they differ, restart the file from scratch.
5. Proxy settings. Over a week, the port, password, or channel expiry could have changed in your Proxeon dashboard. Verify the proxy with a test request before you start chewing through the queue.
6. The dataset itself. If the download runs for a week and the source added and removed records in that time, your result will be a mix of states from different moments. For many tasks that's acceptable. If not, store the start time in the checkpoint and after completion run a separate incremental pass over records modified after that time.

Preflight Checks

Collect all checks into one function that runs at startup before any real work. It either brings state in order or stops the downloader with a clear message.

def preflight(session, state, probe_url):
# 1. прокси жив и авторизован
r = session.head(probe_url, timeout=20)
if r.status_code == 407:
raise SystemExit('прокси отверг логин или пароль, проверьте данные в кабинете Proxeon')
# 2. токен доступа свежий
refresh_access_token(session)
# 3. курсор ещё действителен
if state['cursor']:
test = session.get(API_BASE + '/orders', params={'cursor': state['cursor'], 'limit': 1}, timeout=30)
if test.status_code == 400 and 'cursor' in test.text.lower():
print('курсор протух, переключаемся на якорь по last_id =', state['last_id'])
state['cursor'] = None
state['resume_after_id'] = state['last_id']
# 4. напоминание о возрасте выгрузки
print('выгрузка стартовала', state['started_at'], 'страниц записано', state['pages'])
return state

The refresh_access_token function depends on your source: usually it's a POST request with the refresh token, after which you update the Authorization header in the session. The resume_after_id field is then used in the page-fetch function as a filter "ID greater than the given one".

Tip: store the refresh token and the proxy password not in the checkpoint but in environment variables or a separate secrets file with restricted permissions. You'll copy the checkpoint, send it to colleagues, and attach it to bug reports; secrets don't belong there.

Expected Result

Verification: manually break the cursor in the checkpoint table with UPDATE checkpoint SET cursor='broken'; and start the downloader. You should see a line about switching to the anchor by last_id, and the download should continue without crashing. The record count after completion should match a control run without the broken cursor.

Step 7: A Ready-to-Use Skeleton of a Resilient Python Downloader

Goal of this stage: assemble everything from the previous steps into one file you can run, interrupt, run again, and get a complete result without duplicates.

Skeleton Structure

• Configuration from environment variables: the Proxeon proxy address, the API address, the token, the job name.
• A Store class: SQLite with records and checkpoint tables, one transaction per page.
• A record key function for idempotency.
• A page-fetch function through the proxy.
• The main loop with checkpoint resumption and session re-creation after a break.

Full Code

# resumable_loader.py
import hashlib
import json
import os
import sqlite3
import time
from datetime import datetime, timezone

import requests

PROXY_URL = os.environ['PROXY_URL']# http://USER:PASS@HOST:PORT из кабинета Proxeon
API_BASE = os.environ.get('API_BASE', 'https://api.example.com')
API_TOKEN = os.environ.get('API_TOKEN', '')
DB_PATH = os.environ.get('DB_PATH', 'export.sqlite')
JOB = os.environ.get('JOB', 'orders-2026')
PAGE_SIZE = 500
MAX_ATTEMPTS = 8
FIELDS = ('cursor', 'last_id', 'pages', 'rows', 'started_at')


def now():
return datetime.now(timezone.utc).isoformat()


def record_key(rec):
canon = json.dumps({'id': rec['id']}, sort_keys=True, separators=(',', ':'))
return hashlib.blake2b(canon.encode('utf-8'), digest_size=16).digest()


class Store:
def __init__(self, path):
self.con = sqlite3.connect(path)
self.con.execute('PRAGMA journal_mode=WAL')
self.con.executescript('''
CREATE TABLE IF NOT EXISTS records(
key BLOB PRIMARY KEY,
payload TEXT NOT NULL,
fetched_at TEXT NOT NULL
) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS checkpoint(
job TEXT PRIMARY KEY,
cursor TEXT,
last_id TEXT,
pages INTEGER NOT NULL,
rows INTEGER NOT NULL,
started_at TEXT,
updated_at TEXT
);
''')

def load(self, job):
row = self.con.execute(
'SELECT cursor, last_id, pages, rows, started_at FROM checkpoint WHERE job=?',
(job,)).fetchone()
if row is None:
return {'cursor': None, 'last_id': None, 'pages': 0, 'rows': 0, 'started_at': now()}
return dict(zip(FIELDS, row))

def commit_page(self, job, rows, state):
ts = now()
with self.con:
self.con.executemany(
'INSERT OR IGNORE INTO records(key, payload, fetched_at) VALUES (?, ?, ?)',
[(record_key(r), json.dumps(r, ensure_ascii=False), ts) for r in rows])
self.con.execute(
'INSERT INTO checkpoint(job, cursor, last_id, pages, rows, started_at, updated_at) '
'VALUES (?, ?, ?, ?, ?, ?, ?) '
'ON CONFLICT(job) DO UPDATE SET cursor=excluded.cursor, last_id=excluded.last_id, '
'pages=excluded.pages, rows=excluded.rows, updated_at=excluded.updated_at',
(job, state['cursor'], state['last_id'], state['pages'], state['rows'],
 state['started_at'], ts))

def unique_count(self):
return self.con.execute('SELECT count(*) FROM records').fetchone()[0]


def make_session():
s = requests.Session()
s.proxies = {'http': PROXY_URL, 'https': PROXY_URL}
s.headers['User-Agent'] = 'resumable-loader/1.0'
if API_TOKEN:
s.headers['Authorization'] = 'Bearer ' + API_TOKEN
return s


def fetch_page(session, cursor):
params = {'limit': PAGE_SIZE}
if cursor:
params['cursor'] = cursor
r = session.get(API_BASE + '/orders', params=params, timeout=(15, 90))
r.raise_for_status()
body = r.json()
return body['items'], body.get('next_cursor')


def run():
store = Store(DB_PATH)
state = store.load(JOB)
print('старт: страниц %d, строк %d, уникальных в базе %d'
 % (state['pages'], state['rows'], store.unique_count()))
session = make_session()
attempts = 0
while True:
try:
items, next_cursor = fetch_page(session, state['cursor'])
attempts = 0
except (requests.ConnectionError, requests.Timeout, requests.HTTPError) as e:
attempts += 1
if attempts > MAX_ATTEMPTS:
print('попытки исчерпаны, состояние сохранено, запустите снова позже')
raise
print('обрыв (%s), попытка %d из %d' % (type(e).__name__, attempts, MAX_ATTEMPTS))
time.sleep(min(60, 2 ** attempts)) # выбор пауз описан в статье про 429 и ретраи
session = make_session() # новая сессия: соединение через прокси пересоздаётся
continue
if not items:
break
state['pages'] += 1
state['rows'] += len(items)
state['last_id'] = str(items[-1]['id'])
state['cursor'] = next_cursor
store.commit_page(JOB, items, state)
if state['pages'] % 20 == 0:
print('страниц %d, строк %d' % (state['pages'], state['rows']))
if next_cursor is None:
break
print('готово: страниц %d, строк получено %d, уникальных в базе %d'
 % (state['pages'], state['rows'], store.unique_count()))


if __name__ == '__main__':
run()

How to Adapt It to Your Source

1. Replace the /orders path and the field names items, next_cursor, id with what your API returns. These are three places in the fetch_page and record_key functions.
2. If your source uses offset pagination, replace the cursor parameter with page and compute the next value as state['pages'] + 1. Save the page number in the checkpoint instead of a cursor.
3. If it's keyset pagination, pass a parameter like after_id from state['last_id'] and drop the cursor handling.
4. If you need parallelism, move fetch_page into the fetch_fn from step 5, and use commit_page as write_fn. Split the download into shards and fill the task queue.
5. Add the preflight function from step 6 before the main loop.

Verifying the Result: A Checklist

Before you run the downloader on a real multi-hour volume, push it through this list. Each item takes a couple of minutes, and together they guarantee that your overnight download won't lose data.

1. Interruption test. Start the downloader, hit Ctrl+C after 30 seconds. Start it again. The first output line should show a nonzero page count, not "pages 0".
2. Duplicate test. Lower PAGE_SIZE to 10, interrupt the downloader five times in a row at random moments. On completion, compare the unique record count with the number of rows fetched: unique should be less than or equal to, and with clean cursor pagination and no source writes, almost equal.
3. Consistency test. After any interruption, run two queries: SELECT rows FROM checkpoint; and SELECT count(*) FROM records;. The difference between them should not exceed one page size. If it does, checkpoint and data aren't written in the same transaction.
4. Proxy test. Temporarily unset the PROXY_URL variable or use a wrong password. The downloader should fail on the first request with a clear error, not hang and not start hitting the source directly.
5. Stale cursor test. Break the cursor in the database as described in step 6 and make sure the switch to the anchor fires.
6. Disk test. Check the export.sqlite file size after a thousand pages and multiply by the expected page count. Make sure the disk has room with a 20 percent margin.

Verification: a successful run is one where, after three intentional interruptions and three restarts, the final unique record count matches what you'd get in one continuous run against the same source, and the console never once printed the starting-from-scratch line.

Extra Capabilities and Optimization

• Progress and time estimate. If the total record count is known, print the percentage and an ETA every twenty pages. That's useful for you and for telling a hang apart from slow work.
• Payload compression. For tens of millions of records, JSON in text form takes up a lot of space. Compress the payload field with zlib.compress before writing and store it as a BLOB. Savings are usually three to eight times.
• Migrating to a server database. The scheme with a checkpoint in one transaction with the data transfers to PostgreSQL almost unchanged. The ON CONFLICT construct is supported there too, and the "one writer" limitation disappears.
• Incremental downloads. Store the start time of each job, and after a full download run a separate job with a "modified after" filter. That keeps the copy up to date without a full re-download.
• A separate process per shard. Instead of threads, you can run several script instances with different JOB values and different DB_PATH files, then merge the results at the end. That's easier to debug and eliminates the concurrent-write question entirely.
• Drop metrics. Log every break with its exception type and time. After a day you'll see the breaks cluster around the IP rotation intervals on the Proxeon channel, and you'll be able to tune the rotation interval to fit your request durations.

Typical Mistakes and Solutions

Below are the situations nearly everyone hits on their first runs. Format: problem, cause, solution.

1. Problem: after a restart, the download starts from scratch every time. Cause: the checkpoint is written to memory or a file that doesn't survive a crash, or the downloader doesn't read it at startup. Solution: make sure the first action in the run function is store.load, and that state updates after every page inside a transaction.
2. Problem: the database has one and a half times more records than the source. Cause: the dedup key is unstable: it includes request time, page number, or a field with a random order. Solution: compute the key only from the source ID or from an explicit list of stable fields via the fields parameter.
3. Problem: the database has fewer records than the source, even though the download finished without errors. Cause: the checkpoint was updated before the data write, and after a crash the page was skipped. Or offset pagination shifted pages backward when records were deleted in the source. Solution: change the order to "data first, then checkpoint" in one transaction; for sources with deletions, switch to keyset pagination.
4. Problem: resuming a file yields a corrupt archive at a matching size. Cause: the server once responded with 200 instead of 206, the file was partially overwritten, then appended. Solution: store the ETag next to the file, delete the file if it changes; verify the Content-Range header against the requested offset.
5. Problem: "database is locked" error under parallel work. Cause: several threads write to SQLite at once, or a connection was passed between threads. Solution: one writer in the main thread, workers only download; WAL mode; a connection is created in the thread that uses it.
6. Problem: after an hour of work, all requests start returning 401. Cause: the access token expired. Solution: refresh the token on a schedule before expiry, and on a 401 call refresh and retry the request once, without counting it as a break.
7. Problem: RAM grows to several gigabytes. Cause: the set of seen keys or the list of all records is held in process memory. Solution: dedup via the database primary key or a seen table; write data page by page, without accumulation.
8. Problem: breaks happen on a strict few-minute schedule. Cause: they coincide with the IP rotation interval on the proxy channel. Solution: this is standard behavior, the downloader must survive it. If requests are long, tune the rotation interval in your Proxeon dashboard so it's noticeably longer than a typical single request, or use on-demand rotation between pages.

FAQ: Common Questions About Resilient Downloads

Do I have to use SQLite if I need a CSV result?

No, but it's convenient. SQLite here plays the role of reliable state storage and a set of seen keys. You export the final CSV with one command from the records table when you're done. If you really want no database, use the file variant from step 3 with one file per page and a JSON checkpoint with atomic replacement.

How often should I save the checkpoint: after every page or less often?

After every page. A single SQLite transaction with a few hundred rows runs in single-digit milliseconds, which is negligible compared to a network request through a proxy. Saving on rare checkpoints isn't worth the risk of losing dozens of pages.

What if the API gives neither a cursor nor IDs, only page numbers?

Work by page number, store it in the checkpoint, and be sure to include deduplication by record content hash. Accept that with active changes in the source some records may be skipped due to page shifts. For critical data, do a second pass through the pages in reverse order: what was skipped in the first pass is very likely to be caught in the second.

Can I resume a file with multiple threads fetching different ranges?

Yes, if the server supports Range. Split the file into 50-100 MB chunks, each chunk is a task from the step 5 queue with its own temporary file, and when all chunks are done, concatenate them in the correct order. Check each chunk by size, and the whole file by checksum if one is available.

How many workers should I run through a proxy?

Start with three or four per Proxeon channel and watch the error rate in the tasks table. If errors are under one percent, add two more. If errors grow, reduce. More than ten threads per channel rarely pays off: you hit either the channel's bandwidth or the source's patience.

Should I save session cookies between runs?

Usually no. Re-authenticating at startup takes seconds and is more reliable than restoring cookies with an unknown lifetime. Exception: the source limits the number of logins per day. Then save the cookies, but on the first 401 or login redirect, discard them and log in again.

How do I know the download finished completely and didn't break silently?

For files: the size equals Content-Length and the checksum matches. For APIs: you got a page with no next_cursor or an empty page, and the row count matches the total if the source reports one. Write an explicit completion flag to the checkpoint so a rerun doesn't start a new pass.

What should I do with tasks in the failed status?

Look at the last_error field. If it's network errors, just move the tasks back to pending with an UPDATE and run the downloader again. If they're data parsing errors, the source has records of a non-standard shape: fix the code and rerun. Never silently delete failed tasks; they're the only evidence of what's missing from your download.

Can I use this approach with an async library instead of requests?

Yes, the principles are the same: atomic unit of work, checkpoint together with data, idempotent writes, disk queue. Only the transport changes. The one nuance: keep SQLite writes synchronous and sequential, and keep parallelism at the level of network requests.

Conclusion

You've gone from a partial file and a nervous restart to a downloader that doesn't care about breaks. Let's recap what was done.

• We covered HTTP resumption: checking Accept-Ranges via HEAD, the Range header with the current file size, distinguishing statuses 206 and 200, and protecting against file swaps via ETag and If-Range.
• We built checkpoints for paginated exports: cursor, page number, or last record ID plus service counters, all in one transaction with the data.
• We made writes idempotent via the primary key and the INSERT OR IGNORE or ON CONFLICT DO UPDATE construct, and for files via deterministic names and atomic replacement.
• We organized deduplication on disk so millions of keys don't live in RAM and survive restarts.
• We added parallelism with a SQLite task queue, automatic return of failed tasks, and a single writer.
• We provided for resumption after a long pause: token refresh, re-authentication, switching from a stale cursor to an anchor by ID, verifying the Proxeon proxy before startup.
• We assembled everything into one working skeleton that adapts to a specific source by changing three or four lines.

What to Do Next

Take the skeleton from step 7 and run it on a small real volume, say ten thousand records. Go through the checklist from the verification section. Only after that run the full overnight download. In the morning you'll either see the "done" line with matching counters, or a line about exhausted attempts with saved state, and then you just run the script again.

Where to Grow

The next level is incremental downloads by modification time instead of full passes, moving state into a server database for multiple machines, and a thoughtful retry strategy that accounts for status codes, which is the subject of a separate article about 429 and retries. The combination of smart retries from that article and resilient state from this one gives you a downloader you can leave running for a week without opening a terminal.

And one last thing. A long download breaking isn't an accident, it's a normal situation you now know how to handle. Happy downloading.

About the Author

Roman Melnikov

Technical Writer and System Administrator

Work Experience: Technical writer and DevOps engineer with 9 years of experience. Created over 50 detailed guides on system configuration and administration. His instructions helped thousands of professionals successfully solve technical tasks. Popular author on Habr and YouTube.

Education: Bauman Moscow State Technical University. Information Systems and Technologies

Expertise:

Technical Documentation DevOps System Administration Linux Docker and Kubernetes CI/CD Infrastructure Automation Cloud Technologies System Monitoring Bash and Python Scripting