Skip to main content
Version: Next

ApifyRequestList

A request list that can be constructed from the standard Apify requestListSources Actor input format.

This extends the Crawlee RequestList with the ability to parse the request list sources input commonly used in Apify Actors. It supports two kinds of entries:

  • Direct URLs - entries with a url key are converted to requests directly.
  • Remote URL lists - entries with a requestsFromUrl key point to a remote resource (e.g. a plain-text file). The resource is fetched and all URLs found in the response body are extracted and converted to requests.

Both kinds of entries can optionally specify method, payload, headers, and userData fields that will be applied to every request created from that entry.

Usage

from apify import Actor
from apify.request_loaders import ApifyRequestList

async with Actor:
    actor_input = await Actor.get_input() or {}
    request_list = await ApifyRequestList.open(
        request_list_sources_input=actor_input.get('requestListSources', []),
    )

Index

Methods

Methods

open

  • Create a new ApifyRequestList from the standard Apify request list sources input.

    Each entry in request_list_sources_input is a dict with either a url key (for a direct URL) or a requestsFromUrl key (for a remote resource whose response body is scanned for URLs). Optional keys method, payload, headers, and userData are applied to every request produced from that entry.

    Parameters

    • optionalkeyword-onlyname: str | None = None

      An optional name for the request list, used for state persistence.

    • optionalkeyword-onlyrequest_list_sources_input: list[dict[str, Any]] | None = None

      A list of request source dicts in the standard Apify format. Each dict must contain either a url key or a requestsFromUrl key. If None or empty, an empty request list is returned.

    • optionalkeyword-onlyhttp_client: HttpClient | None = None

      HTTP client used to fetch remote URL lists (entries with requestsFromUrl). Defaults to ImpitHttpClient if not provided.

    Returns ApifyRequestList

    A new ApifyRequestList populated with the resolved requests.

Page Options