dap incremental
Download changes from a table within a specified time range.
Generates and downloads only the new or modified records of a table within a specified timeframe. This efficient, resource-effective approach helps keep datasets up-to-date with minimal overhead. Incremental queries enable near-real-time updates, reducing the need for frequent full dataset downloads.
Usage
Arguments
--base-url <string>
URL to the DAP API endpoint: https://api-gateway.instructure.com. Skip, if DAP_API_URL environment variable is set.
--client-id <string>
Client ID obtained from the Identity Service. Skip, if DAP_CLIENT_ID environment variable is set.
--client-secret <string>
Client Secret obtained from the Identity Service. Skip, if DAP_CLIENT_SECRET environment variable is set.
Flags
--namespace <string>
Specifies the data source (namespace). Available options: {canvas, canvas_log, catalog}.
--table <string>
Specifies the table fetch data from.
--since <datetime>
Start timestamp for the incremental query. Examples: 2024-12-01T09:30:00Z
--then <datetime>
End timestamp for the incremental query. Examples: 2024-12-01T09:30:00Z
--format <string> (default: JSONL)
Defines the output format. Available options: {CSV, JSONL, Parquet, TSV}.
--output-directory <string> (default: downloads)
Specifies the absolute or relative path to the output directory where the snapshot will be stored.
Inherited Flags
-h, --help
Displays help information for the command.
Examples
Get new or modified records of the courses table from the canvas namespace since October 1, 2024
$ dap incremental --namespace canvas --table courses --since 2024-11-01T00:00:00
Get new or modified records of the courses table from the canvas namespace between October 1, 2024, and October 31, 2024
$ dap incremental --namespace canvas --table courses --since 2024-11-01T00:00:00 --then 2024-11-01T00:00:00
Related Resources
Last updated