Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New CLI flag: after #114

Merged
merged 5 commits into from
Aug 31, 2022
Merged

Conversation

snowskeleton
Copy link
Contributor

Provide an RFC 3339 formatted date as a string, and only download items purchased after that date.
Also added the purchase date to the list of values generated by audible library export.

I added this so it's easier to programmatically download new books without worrying about what's already been downloaded.

…nd only download items purchased after that date.
@snowskeleton
Copy link
Contributor Author

Even better, seems like you can be a bit more fuzzy with the date than I thought. No specific guarantees about what will/won't work, but it seems to do a fairly good job of guessing.

@mkb79
Copy link
Owner

mkb79 commented Aug 25, 2022

Good morning and thank you for your PR. I thought about it. Here you can read about a state token when fetching the library. The state token is simply a timestamp since epoch. So you could simply convert your date to a timestamp and provide it via the state token. This way fetching the library is faster because the server only sends books, which where added after these date.

Edit:
Or described here, there are purchased_after parameter. Maybe this fit your needs?!

@snowskeleton
Copy link
Contributor Author

I like the idea of fetching fewer books from the server. Would that mean we don't have library = await Library.from_api_full_sync... unless the user specifies the --all flag?

@mkb79
Copy link
Owner

mkb79 commented Aug 25, 2022

The current code for the download command is:

# fetch the user library
    library = await Library.from_api_full_sync(
        api_client,
        image_sizes="1215, 408, 360, 882, 315, 570, 252, 558, 900, 500",
        bunch_size=bunch_size,
        response_groups=(
            "product_desc, media, product_attrs, relationships, "
            "series, customer_rights, pdf_url"
        )
    )

This have to be changed to:

# fetch the user library
    library = await Library.from_api_full_sync(
        api_client,
        image_sizes="1215, 408, 360, 882, 315, 570, 252, 558, 900, 500",
        bunch_size=bunch_size,
        response_groups=(
            "product_desc, media, product_attrs, relationships, "
            "series, customer_rights, pdf_url"
        ),
        state_token=13-digit timestamp,   # add these
        status="Active",  # add these
    )

and all should be fine.

So you have only to convert your after date to a timestamp (in milliseconds since epoch) or set timestamp to 0, if no date is provided.

@snowskeleton
Copy link
Contributor Author

Ohhhhhh I see. That's clever. I will do that.

@snowskeleton
Copy link
Contributor Author

The state token didn't end up working quite right. At least it doesn't work the way I expect it to. The purchase_date flag does work, however. Very similar to what you suggested with the Unix timestamp, I set the default to before Audible (the company) was in business, and included an example in the helper text.

Thoughts?

@mkb79
Copy link
Owner

mkb79 commented Aug 28, 2022

Can you tell me, what does not work right with the state_token? I've tested some things this week and it works.

Do you know, if Audible Plus titles get a purchase_date when added to the library?

@snowskeleton
Copy link
Contributor Author

When I pass in a state_token value of right now (date +%s, plus 000, 13 characters), the API returns all items in my library. When specifying the purchase_date, the API returns within expected parameters.

Yes, titles added to your library from the Audible Plus catalog have a purchase_date set at time of adding. And at least for me, I still get an email receipt for $0.00 from Audible whenever I add Audible Plus titles.

@mkb79
Copy link
Owner

mkb79 commented Aug 29, 2022

When I pass in a state_token value of right now (date +%s, plus 000, 13 characters), the API returns all items in my library.

Which code do you use to create your timestamp? Can you try this online converter to create your timestamp and try again. I found out that, if the timestamp is not properly formed, all items are returned from the API.

@mkb79
Copy link
Owner

mkb79 commented Aug 29, 2022

Oh I see what happens in your code. You have to convert the date to a Unix timestamp first before submit this as a state token.

Edit:
Please take care that the timestamp must be in milliseconds since epoch without decimal place.

Edit:
2022-08-29T00:00:00.000Z results in a timestamp of 1661731200000!

@snowskeleton
Copy link
Contributor Author

After making these changes

@click.option(
    "--after",
    type=str,
    default='0',
)
...
library = await Library.from_api_full_sync(
    api_client,
    image_sizes="1215, 408, 360, 882, 315, 570, 252, 558, 900, 500",
    bunch_size=bunch_size,
    response_groups=(
        "product_desc, media, product_attrs, relationships, "
        "series, customer_rights, pdf_url"
    ),
    state_token=after,
    status="Active",
)

and running
pip install --upgrade .
and
audible download --all --aax --after 1661731200000
All items in my library started downloading. I also tried putting the timestamp in single quotes and I got the same result. I also tried changing the type from str to int (both with and without quotes around the timestamp) and it still tried downloading everything.

@mkb79
Copy link
Owner

mkb79 commented Aug 29, 2022

Okay, I've found out what happened. The state_token param can't be used with the page param. And calling Library.from_api_full_sync will automatically set page=1. So all library items are fetched. Sorry for wasting your time.

@snowskeleton
Copy link
Contributor Author

That actually makes a fair bit of sense. Now we know for the future!

@mkb79
Copy link
Owner

mkb79 commented Aug 29, 2022

I will think about another feature. Storing the last state token to file. So the next download --all command will continue at this point?!

@mkb79
Copy link
Owner

mkb79 commented Aug 29, 2022

But you can use purchased_after="2022-07-01T00:00:00Z" instead of state_token. This will running with page param.

@snowskeleton
Copy link
Contributor Author

Currently I'm keeping track of the date outside of audible-cli with a simple bash script. I like the idea of integrating it all in one place like you mentioned, though.
I think the --all flag should be left more or less unchanged, so it doesn't break backwards compatibility. Maybe we could add a --new flag that does what you suggest (keep track of state token in the config file or something). I'm imagining audible download --new [--cover/--aax/--chapter/etc] would use the state token to get new items and update the state token after it's done.

@snowskeleton
Copy link
Contributor Author

But you can use purchased_after="2022-07-01T00:00:00Z" instead of state_token. This will running with page param.

yes, the purchased_after flag seems to work more or less as we'd expect.

@mkb79
Copy link
Owner

mkb79 commented Aug 30, 2022

yes, the purchased_after flag seems to work more or less as we'd expect.

What you mean with more or less? Have you any issues with the purchase date? I will implement this via the --after option. And maybe add a --before option too.

@mkb79
Copy link
Owner

mkb79 commented Aug 30, 2022

Here is my final code for a start-date and end-date:

import asyncio
import asyncio.log
import asyncio.sslproto
import json
import pathlib
import logging
from datetime import datetime

import aiofiles
import click
import httpx
import questionary
from audible.exceptions import NotFoundError
from click import echo

from ..decorators import (
    bunch_size_option,
    timeout_option,
    pass_client,
    pass_session
)
from ..exceptions import DirectoryDoesNotExists, NotDownloadableAsAAX
from ..models import Library
from ..utils import Downloader


logger = logging.getLogger("audible_cli.cmds.cmd_download")

CLIENT_HEADERS = {
    "User-Agent": "Audible/671 CFNetwork/1240.0.4 Darwin/20.6.0"
}
datetime_type = click.DateTime([
    "%Y-%m-%d",
    "%Y-%m-%dT%H:%M:%S",
    "%Y-%m-%d %H:%M:%S",
    "%Y-%m-%dT%H:%M:%S.%fZ"
])


class DownloadCounter:
    def __init__(self):
        self._aax: int = 0
        self._aaxc: int = 0
        self._annotation: int = 0
        self._chapter: int = 0
        self._cover: int = 0
        self._pdf: int = 0
        self._voucher: int = 0
        self._voucher_saved: int = 0

    @property
    def aax(self):
        return self._aax

    def count_aax(self):
        self._aax += 1
        logger.debug(f"Currently downloaded aax files: {self.aax}")

    @property
    def aaxc(self):
        return self._aaxc

    def count_aaxc(self):
        self._aaxc += 1
        logger.debug(f"Currently downloaded aaxc files: {self.aaxc}")

    @property
    def annotation(self):
        return self._annotation

    def count_annotation(self):
        self._annotation += 1
        logger.debug(f"Currently downloaded annotations: {self.annotation}")

    @property
    def chapter(self):
        return self._chapter

    def count_chapter(self):
        self._chapter += 1
        logger.debug(f"Currently downloaded chapters: {self.chapter}")

    @property
    def cover(self):
        return self._cover

    def count_cover(self):
        self._cover += 1
        logger.debug(f"Currently downloaded covers: {self.cover}")

    @property
    def pdf(self):
        return self._pdf

    def count_pdf(self):
        self._pdf += 1
        logger.debug(f"Currently downloaded PDFs: {self.pdf}")

    @property
    def voucher(self):
        return self._voucher

    def count_voucher(self):
        self._voucher += 1
        logger.debug(f"Currently downloaded voucher files: {self.voucher}")

    @property
    def voucher_saved(self):
        return self._voucher_saved

    def count_voucher_saved(self):
        self._voucher_saved += 1
        logger.debug(f"Currently saved voucher files: {self.voucher_saved}")

    def as_dict(self) -> dict:
        return {
            "aax": self.aax,
            "aaxc": self.aaxc,
            "annotation": self.annotation,
            "chapter": self.chapter,
            "cover": self.cover,
            "pdf": self.pdf,
            "voucher": self.voucher,
            "voucher_saved": self.voucher_saved
        }

    def has_downloads(self):
        for _, v in self.as_dict().items():
            if v > 0:
                return True

        return False


counter = DownloadCounter()


async def download_cover(
        client, output_dir, base_filename, item, res, overwrite_existing
):
    filename = f"{base_filename}_({str(res)}).jpg"
    filepath = output_dir / filename

    url = item.get_cover_url(res)
    if url is None:
        logger.error(
            f"No COVER found for {item.full_title} with given resolution"
        )
        return

    dl = Downloader(url, filepath, client, overwrite_existing, "image/jpeg")
    downloaded = await dl.run(stream=False, pb=False)

    if downloaded:
        counter.count_cover()


async def download_pdf(
        client, output_dir, base_filename, item, overwrite_existing
):
    url = item.get_pdf_url()
    if url is None:
        logger.info(f"No PDF found for {item.full_title}")
        return

    filename = base_filename + ".pdf"
    filepath = output_dir / filename
    dl = Downloader(
        url, filepath, client, overwrite_existing,
        ["application/octet-stream", "application/pdf"]
    )
    downloaded = await dl.run(stream=False, pb=False)

    if downloaded:
        counter.count_pdf()


async def download_chapters(
        output_dir, base_filename, item, quality, overwrite_existing
):
    if not output_dir.is_dir():
        raise DirectoryDoesNotExists(output_dir)

    filename = base_filename + "-chapters.json"
    file = output_dir / filename
    if file.exists() and not overwrite_existing:
        logger.info(
            f"File {file} already exists. Skip saving chapters"
        )
        return True

    try:
        metadata = await item.get_content_metadata(quality)
    except NotFoundError:
        logger.info(
            f"No chapters found for {item.full_title}."
        )
        return
    metadata = json.dumps(metadata, indent=4)
    async with aiofiles.open(file, "w") as f:
        await f.write(metadata)
    logger.info(f"Chapter file saved to {file}.")
    counter.count_chapter()


async def download_annotations(
        output_dir, base_filename, item, overwrite_existing
):
    if not output_dir.is_dir():
        raise DirectoryDoesNotExists(output_dir)

    filename = base_filename + "-annotations.json"
    file = output_dir / filename
    if file.exists() and not overwrite_existing:
        logger.info(
            f"File {file} already exists. Skip saving annotations"
        )
        return True

    try:
        annotation = await item.get_annotations()
    except NotFoundError:
        logger.info(
            f"No annotations found for {item.full_title}."
        )
        return
    annotation = json.dumps(annotation, indent=4)
    async with aiofiles.open(file, "w") as f:
        await f.write(annotation)
    logger.info(f"Annotation file saved to {file}.")
    counter.count_annotation()


async def download_aax(
        client, output_dir, base_filename, item, quality, overwrite_existing,
        aax_fallback
):
    # url, codec = await item.get_aax_url(quality)
    try:
        url, codec = await item.get_aax_url_old(quality)
    except NotDownloadableAsAAX:
        if aax_fallback:
            logger.info(f"Fallback to aaxc for {item.full_title}")
            return await download_aaxc(
                client=client,
                output_dir=output_dir,
                base_filename=base_filename,
                item=item,
                quality=quality,
                overwrite_existing=overwrite_existing
            )
        raise

    filename = base_filename + f"-{codec}.aax"
    filepath = output_dir / filename
    dl = Downloader(
        url, filepath, client, overwrite_existing,
        ["audio/aax", "audio/vnd.audible.aax", "audio/audible"]
    )
    downloaded = await dl.run(pb=True)

    if downloaded:
        counter.count_aax()


async def download_aaxc(
        client, output_dir, base_filename, item,
        quality, overwrite_existing
):
    lr, url, codec = None, None, None

    # https://github.com/mkb79/audible-cli/issues/60
    if not overwrite_existing:
        codec, _ = item._get_codec(quality)
        if codec is not None:
            filepath = pathlib.Path(
                output_dir) / f"{base_filename}-{codec}.aaxc"
            lr_file = filepath.with_suffix(".voucher")
        
            if lr_file.is_file():
                if filepath.is_file():
                    logger.info(
                        f"File {lr_file} already exists. Skip download."
                    )
                    logger.info(
                        f"File {filepath} already exists. Skip download."
                    )
                    return
                else:
                    logger.info(
                        f"Loading data from voucher file {lr_file}."
                    )
                    async with aiofiles.open(lr_file, "r") as f:
                        lr = await f.read()
                    lr = json.loads(lr)
                    content_metadata = lr["content_license"][
                        "content_metadata"]
                    url = httpx.URL(
                        content_metadata["content_url"]["offline_url"])
                    codec = content_metadata["content_reference"][
                        "content_format"]

    if url is None or codec is None or lr is None:
        url, codec, lr = await item.get_aaxc_url(quality)
        counter.count_voucher()

    if codec.lower() == "mpeg":
        ext = "mp3"
    else:
        ext = "aaxc"

    filepath = pathlib.Path(
        output_dir) / f"{base_filename}-{codec}.{ext}"
    lr_file = filepath.with_suffix(".voucher")

    if lr_file.is_file() and not overwrite_existing:
        logger.info(
            f"File {lr_file} already exists. Skip download."
        )
    else:
        lr = json.dumps(lr, indent=4)
        async with aiofiles.open(lr_file, "w") as f:
            await f.write(lr)
        logger.info(f"Voucher file saved to {lr_file}.")
        counter.count_voucher_saved()

    dl = Downloader(
        url,
        filepath,
        client,
        overwrite_existing,
        [
            "audio/aax", "audio/vnd.audible.aax", "audio/mpeg", "audio/x-m4a",
            "audio/audible"
        ]
    )
    downloaded = await dl.run(pb=True)

    if downloaded:
        counter.count_aaxc()


async def consume(queue):
    while True:
        item = await queue.get()
        try:
            await item
        except Exception as e:
            logger.error(e)
            raise
        finally:
            queue.task_done()


def queue_job(
        queue,
        get_cover,
        get_pdf,
        get_annotation,
        get_chapters,
        get_aax,
        get_aaxc,
        client,
        output_dir,
        filename_mode,
        item,
        cover_size,
        quality,
        overwrite_existing,
        aax_fallback
):
    base_filename = item.create_base_filename(filename_mode)

    if get_cover:
        queue.put_nowait(
            download_cover(
                client=client,
                output_dir=output_dir,
                base_filename=base_filename,
                item=item,
                res=cover_size,
                overwrite_existing=overwrite_existing
            )
        )

    if get_pdf:
        queue.put_nowait(
            download_pdf(
                client=client,
                output_dir=output_dir,
                base_filename=base_filename,
                item=item,
                overwrite_existing=overwrite_existing
            )
        )

    if get_chapters:
        queue.put_nowait(
            download_chapters(
                output_dir=output_dir,
                base_filename=base_filename,
                item=item,
                quality=quality,
                overwrite_existing=overwrite_existing
            )
        )

    if get_annotation:
        queue.put_nowait(
            download_annotations(
                output_dir=output_dir,
                base_filename=base_filename,
                item=item,
                overwrite_existing=overwrite_existing
            )
        )

    if get_aax:
        queue.put_nowait(
            download_aax(
                client=client,
                output_dir=output_dir,
                base_filename=base_filename,
                item=item,
                quality=quality,
                overwrite_existing=overwrite_existing,
                aax_fallback=aax_fallback
            )
        )

    if get_aaxc:
        queue.put_nowait(
            download_aaxc(
                client=client,
                output_dir=output_dir,
                base_filename=base_filename,
                item=item,
                quality=quality,
                overwrite_existing=overwrite_existing
            )
        )


def display_counter():
    if counter.has_downloads():
        echo("The download ended with the following result:")
        for k, v in counter.as_dict().items():
            if v == 0:
                continue

            if k == "voucher_saved":
                k = "voucher"
            elif k == "voucher":
                diff = v - counter.voucher_saved
                if diff > 0:
                    echo(f"Unsaved voucher: {diff}")
                continue
            echo(f"New {k} files: {v}")
    else:
        echo("No new files downloaded.")


@click.command("download")
@click.option(
    "--output-dir", "-o",
    type=click.Path(exists=True, dir_okay=True),
    default=pathlib.Path().cwd(),
    help="output dir, uses current working dir as default"
)
@click.option(
    "--all",
    is_flag=True,
    help="download all library items, overrides --asin and --title options"
)
@click.option(
    "--asin", "-a",
    multiple=True,
    help="asin of the audiobook"
)
@click.option(
    "--title", "-t",
    multiple=True,
    help="tile of the audiobook (partial search)"
)
@click.option(
    "--aax",
    is_flag=True,
    help="Download book in aax format"
)
@click.option(
    "--aaxc",
    is_flag=True,
    help="Download book in aaxc format incl. voucher file"
)
@click.option(
    "--aax-fallback",
    is_flag=True,
    help="Download book in aax format and fallback to aaxc, if former is not supported."
)
@click.option(
    "--quality", "-q",
    default="best",
    show_default=True,
    type=click.Choice(["best", "high", "normal"]),
    help="download quality"
)
@click.option(
    "--pdf",
    is_flag=True,
    help="downloads the pdf in addition to the audiobook"
)
@click.option(
    "--cover",
    is_flag=True,
    help="downloads the cover in addition to the audiobook"
)
@click.option(
    "--cover-size",
    type=click.Choice(["252", "315", "360", "408", "500", "558", "570", "882",
                       "900", "1215"]),
    default="500",
    help="the cover pixel size"
)
@click.option(
    "--chapter",
    is_flag=True,
    help="saves chapter metadata as JSON file"
)
@click.option(
    "--annotation",
    is_flag=True,
    help="saves the annotations (e.g. bookmarks, notes) as JSON file"
)
@click.option(
    "--start-date",
    type=datetime_type,
    default="1900-1-1",
    help=(
        "Only considers books added to library before or on this UTC date."
        "Following formats are supported: '%Y-%m-%d', '%Y-%m-%dT%H:%M:%S', "
        "'%Y-%m-%d %H:%M:%S', '%Y-%m-%dT%H:%M:%S.%fZ'"
    )
)
@click.option(
    "--end-date",
    type=datetime_type,
    default=datetime.utcnow(),
    help=(
        "Only considers books added to library on or after this UTC date."
        "Following formats are supported: '%Y-%m-%d', '%Y-%m-%dT%H:%M:%S', "
        "'%Y-%m-%d %H:%M:%S', '%Y-%m-%dT%H:%M:%S.%fZ'"
    )
)
@click.option(
    "--no-confirm", "-y",
    is_flag=True,
    help="start without confirm"
)
@click.option(
    "--overwrite",
    is_flag=True,
    help="rename existing files"
)
@click.option(
    "--ignore-errors",
    is_flag=True,
    help="ignore errors and continue with the rest"
)
@click.option(
    "--jobs", "-j",
    type=int,
    default=3,
    show_default=True,
    help="number of simultaneous downloads"
)
@click.option(
    "--filename-mode", "-f",
    type=click.Choice(
        ["config", "ascii", "asin_ascii", "unicode", "asin_unicode"]
    ),
    default="config",
    help="Filename mode to use. [default: config]"
)
@timeout_option
@click.option(
    "--resolve-podcasts",
    is_flag=True,
    help="Resolve podcasts to download a single episode via asin or title"
)
@click.option(
    "--ignore-podcasts",
    is_flag=True,
    help="Ignore a podcast if it have episodes"
)
@bunch_size_option
@pass_session
@pass_client(headers=CLIENT_HEADERS)
async def cli(session, api_client, **params):
    """download audiobook(s) from library"""
    client = api_client.session
    output_dir = pathlib.Path(params.get("output_dir")).resolve()

    # which item(s) to download
    get_all = params.get("all") is True
    asins = params.get("asin")
    titles = params.get("title")
    if get_all and (asins or titles):
        logger.error(f"Do not mix *asin* or *title* option with *all* option.")
        click.Abort()

    # what to download
    get_aax = params.get("aax")
    get_aaxc = params.get("aaxc")
    aax_fallback = params.get("aax_fallback")
    if aax_fallback:
        if get_aax:
            logger.info("Using --aax is redundant and can be left when using --aax-fallback")
        get_aax = True
        if get_aaxc:
            logger.warning("Do not mix --aaxc with --aax-fallback option.")
    get_annotation = params.get("annotation")
    get_chapters = params.get("chapter")
    get_cover = params.get("cover")
    get_pdf = params.get("pdf")
    if not any(
        [get_aax, get_aaxc, get_annotation, get_chapters, get_cover, get_pdf]
    ):
        logger.error("Please select an option what you want download.")
        raise click.Abort()

    # additional options
    sim_jobs = params.get("jobs")
    quality = params.get("quality")
    cover_size = params.get("cover_size")
    overwrite_existing = params.get("overwrite")
    ignore_errors = params.get("ignore_errors")
    no_confirm = params.get("no_confirm")
    resolve_podcats = params.get("resolve_podcasts")
    ignore_podcasts = params.get("ignore_podcasts")
    bunch_size = session.params.get("bunch_size")

    start_date = params.get("start_date")
    purchased_after = start_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ')
    # TODO: implement before
    end_date = params.get("end_date")

    filename_mode = params.get("filename_mode")
    if filename_mode == "config":
        filename_mode = session.config.get_profile_option(
            session.selected_profile, "filename_mode") or "ascii"

    # fetch the user library
    library = await Library.from_api_full_sync(
        api_client,
        image_sizes="1215, 408, 360, 882, 315, 570, 252, 558, 900, 500",
        bunch_size=bunch_size,
        response_groups=(
            "product_desc, media, product_attrs, relationships, "
            "series, customer_rights, pdf_url"
        ),
        purchased_after=purchased_after,
        status="Active",
    )

    if resolve_podcats:
        await library.resolve_podcats()

    # collect jobs
    jobs = []

    if get_all:
        asins = []
        titles = []
        for i in library:
            jobs.append(i.asin)

    for asin in asins:
        if library.has_asin(asin):
            jobs.append(asin)
        else:
            if not ignore_errors:
                logger.error(f"Asin {asin} not found in library.")
                click.Abort()
            logger.error(
                f"Skip asin {asin}: Not found in library"
            )

    for title in titles:
        match = library.search_item_by_title(title)
        full_match = [i for i in match if i[1] == 100]

        if match:
            if no_confirm:
                [jobs.append(i[0].asin) for i in full_match or match]
            else:
                choices = []
                for i in full_match or match:
                    a = i[0].asin
                    t = i[0].full_title
                    c = questionary.Choice(title=f"{a} # {t}", value=a)
                    choices.append(c)

                answer = await questionary.checkbox(
                    f"Found the following matches for '{title}'. Which you want to download?",
                    choices=choices
                ).unsafe_ask_async()
                if answer is not None:
                    [jobs.append(i) for i in answer]
                
        else:
            logger.error(
                f"Skip title {title}: Not found in library"
            )

    queue = asyncio.Queue()
    for job in jobs:
        item = library.get_item_by_asin(job)
        items = [item]
        odir = pathlib.Path(output_dir)

        if not ignore_podcasts and item.is_parent_podcast():
            items.remove(item)
            if item._children is None:
                await item.get_child_items()

            for i in item._children:
                if i.asin not in jobs:
                    items.append(i)

            podcast_dir = item.create_base_filename(filename_mode)
            odir = output_dir / podcast_dir
            if not odir.is_dir():
                odir.mkdir(parents=True)

        for item in items:
            purchase_date = datetime_type.convert(item.purchase_date, None, None)
            if not start_date <= purchase_date <= end_date:
                continue

            queue_job(
                queue=queue,
                get_cover=get_cover,
                get_pdf=get_pdf,
                get_annotation=get_annotation,
                get_chapters=get_chapters,
                get_aax=get_aax,
                get_aaxc=get_aaxc,
                client=client,
                output_dir=odir,
                filename_mode=filename_mode,
                item=item,
                cover_size=cover_size,
                quality=quality,
                overwrite_existing=overwrite_existing,
                aax_fallback=aax_fallback
            )

    try:
        # schedule the consumer
        consumers = [
            asyncio.ensure_future(consume(queue)) for _ in range(sim_jobs)
        ]
        # wait until the consumer has processed all items
        await queue.join()

    finally:
        # the consumer is still awaiting an item, cancel it
        for consumer in consumers:
            consumer.cancel()
    
        await asyncio.gather(*consumers, return_exceptions=True)
        display_counter()

@mkb79
Copy link
Owner

mkb79 commented Aug 30, 2022

Be careful with the end date. If no hour/minute/second is provided it will take 00:00:00 (the beginning of the day)! In that case you have to take the next day as end date.

@mkb79
Copy link
Owner

mkb79 commented Aug 31, 2022

What do you think about the current status? I would like to merge your pull request if all is fine!

@snowskeleton
Copy link
Contributor Author

This looks great. In principle, it would be nice to have the logic for purchased_after and purchased_before in the same spot, rather than letting Audible handle one and doing the other in code. But ultimately I don't think it'll end up making much difference.

One note.

...
    "--start-date",
    type=datetime_type,
    default="1900-1-1",
...

In my testing, any date earlier than the Unix Epoch (1970-01-01) causes the Audible API to return an error. Have you tried this 1900 date to see if Audible accepts it as valid?

@mkb79
Copy link
Owner

mkb79 commented Aug 31, 2022

In my testing, any date earlier than the Unix Epoch (1970-01-01) causes the Audible API to return an error.

It runs fine for me. But you/we can get safe and you can change the date to 1970-01-01.

In principle, it would be nice to have the logic for purchased_after and purchased_before in the same spot, rather than letting Audible handle one and doing the other in code. But ultimately I don't think it'll end up making much difference.

I‘m thinking about this too! But things going confuse, if we name it after and before. Because the specified date counts, too. But hey, we can name it as you suggest.

@snowskeleton
Copy link
Contributor Author

I like your naming better.
If you manually provide only "2022-08-18" then yeah, it can't tell when in the day you want. What I do to compensate for that is keep the timestamp of the last book I downloaded, which goes down to the second. Then I provide that same timestamp for the purchased_after timestamp, so unless two books are purchased at the exact same second, then there shouldn't be any possibility for overlap.

Overall, this looks good. I say it's ready.

@mkb79
Copy link
Owner

mkb79 commented Aug 31, 2022

With my implementation you can specify a start date of 2021-1-1 and a end date of 2022-1-1 to download all books added in 2021 ;)!

@mkb79 mkb79 merged commit c53e4d2 into mkb79:master Aug 31, 2022
@snowskeleton snowskeleton deleted the time-based-downloading branch August 31, 2022 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants