Add unpacking of extra deps

[philderbeast]

• Any changes that could be relevant to users have been recorded in ChangeLog.md.
• The documentation has been updated, if necessary

I haven't added any tests. I tested by adding to extra-deps in the stack project itself and then running:

$ stack repl
ghci> :main unpack PACKAGE

I have a tool that I'd like to distribute as a package (rather than as a git submodule) and this would allow me to do that as the tool needs to be located in its own folder in the root of a project.

[mpilgrem]

@philderbeast, what is meant by a 'source repository package'? Do you mean that stack unpack should accept a URL to a *.tar.gz file in addition to package names or package versions?

[philderbeast]

I use the term source repository package (borrowed from cabal-speak) to mean things like:

extra-deps:
- git: https://github.com/example-user/my-repo
  commit: 08c9b4cdf977d5bcd1baba046a007940c1940758

[mpilgrem]

For cloning a Git repository and setting it to a specific commit, is that not somewhat duplicative of what the git tool does?

EDIT: In contrast, the equivalent of:

extra-deps:
- github: https://github.com/example-user/my-repo
  commit: 08c9b4cdf977d5bcd1baba046a007940c1940758

( see https://docs.haskellstack.org/en/stable/pantry/#github-archive-files) would be doing something that git does not do, as that references the archive file produced by GitHub's archive API. However, that feature is specific to GitHub repositories.

[develop7]

@philderbeast so it doesn't actually unpack dependencies mentioned in extra-deps after all, it unpacks an arbitrary repo/tarball (bit unclear here) as if it was a dependency?

[philderbeast]

Another more usual motivation: if I'm having trouble with a dependency, I can unpack it, add it as a local package and add tracing to see what's up with it. We have this for packages on stackage and for package-version extra-deps. This pull request gives us the same thing for git dependencies.

[philderbeast]

@philderbeast so it doesn't actually unpack dependencies mentioned in extra-deps after all, it unpacks an arbitrary repo/tarball (bit unclear here) as if it was a dependency?

For my testing, I added it git style dependency to extra-deps.

[philderbeast]

This is a lower ceremony way of debugging a dependency than git submodule add ....

[mpilgrem]

The CI is failing because something in MSYS2 now changes the way that haskeline behaves. If you rebase against the master branch, that should go away.

[mpilgrem]

@philderbeast, I still do not follow what you are proposing for the stack unpack command. Its current functionality is described here: https://docs.haskellstack.org/en/stable/unpack_command/. Are you suggesting the possibility of commands like:

stack unpack <url_to_an_archive_file>

or (making use of the GitHub archive API):

stack unpack <url_to_a_GitHub_repository> --commit <commit>

or (effectively cloning a Git repository and setting it to a specific commit)

stack unpack <url_to_a_Git_repository> --commit <commit>

or are you suggesting a new flag like --[no-]use-extra-dep where a package name would be understood to be a reference not to the most recent version in the package index, but to the package version (if it exists) specified as an extra dep in the project-level configuration file (stack.yaml)? Something like:

stack unpack <package_name> --use-extra-dep

EDIT: If the latter, why restrict it to extra deps? Why not extend the concept to the project's 'dependencies' (its snapshot as extended by extra deps and truncated by drop-packages)? So, you would have something like:

stack unpack <package_name> --use-dep

[develop7]

Why not extend the concept to the project's 'dependencies' (its snapshot as extended by extra deps and truncated by drop-packages)?

For some reason I've always thought stack unpack works the way you described. I've even red it's source with such assumption in mind; no wonder I couldn't make sense of it back then :)

[mpilgrem]

@develop7, I could be convinced that --use-dep should be the default, but I think a flag is needed so that people who want to retain the current functionality could do so by passing --no-use-dep.

Currently, if you use (say) stack unpack aeson outside of a project directory, it will go and fetch aeson-2.2.0.0 from Hackage and put it in new subdirectory aeson-2.2.0.0.

If Stack was to rely on dependencies, it would - instead - (outside of a project directory) look up the project-level YAML configuration in the 'global project'.

Given the quite different behaviour, I think you would need a flag.

You might also need another flag to opt between putting what is fetched info directory aeson-<version> or just aeson (in my example). I suspect that most local package directories would not specify the version in the directory name, but stack unpack always uses the package version as the subdirectory name.

EDIT: There is also the question of what should happen if a package is not a dependency. For example (assuming that acme-missiles is not an extra dep):

stack unpack acme-missiles --use-dep

Should that report that acme-missiles is not a dependency and fail. Or should it, in the alternative, behave as stack unpack acme-missiles --no-use-dep (especially if --use-dep is the default)?

[develop7]

@develop7, I could be convinced that --use-dep should be the default, but I think a flag is needed so that people who want to retain the current functionality could do so by passing --no-use-dep.

It's kind of the way I expected it should work, you know? We've got the project, the project depends on libraries of certain versions, hence it makes perfect sense stack unpack considers project dependencies.

Currently, if you use (say) stack unpack aeson outside of a project directory, it will go and fetch aeson-2.2.0.0 from Hackage and put it in new subdirectory aeson-2.2.0.0.

If Stack was to rely on dependencies, it would - instead - (outside of a project directory) look up the project-level YAML configuration in the 'global project'.

In order to improve backward compatibility I'd make --use-dep default in a project, but not in a global project. And deprecate such behavior few years later.

Given the quite different behaviour, I think you would need a flag.

You might also need another flag to opt between putting what is fetched info directory aeson-<version> or just aeson (in my example). I suspect that most local package directories would not specify the version in the directory name, but stack unpack always uses the package version as the subdirectory name.

uh-huh, something along the lines of --dirname (bit confusing anyway since the --to is present too). Or, maybe, a flag that would interpret the --to directory as complete path to unpack the source.

EDIT: There is also the question of what should happen if a package is not a dependency. For example (assuming that acme-missiles is not an extra dep):

stack unpack acme-missiles --use-dep

Should that report that acme-missiles is not a dependency and fail. Or should it, in the alternative, behave as stack unpack acme-missiles --no-use-dep (especially if --use-dep is the default)?

Print a warning, and fallback to the --no-use-dep?

@philderbeast what do you think BTW?

[mpilgrem]

@philderbeast, do you have a view on the UI discussion above? I am thinking of releasing a new version of Stack soon. Letting you know in case you want to see some version of this idea reflected in it.

[philderbeast]

Thanks for the alert. I've settled on using make and downloading a tar archive from hackage for the package containing the tooling I want to bring into a project. This feels better than adding a dependency to a project that is only a build tool, something like a project-level build-tool-depends.

@mpilgrem and @develop7 thanks for the discussion so far.

@mpilgrem please go ahead with the next stack release without this half-baked feature. I need to review how stack and cabal currently handle unpacking. They both preserve the package-x.y.z version in the name of the directory the package is unpacked to but this pull request drops the version (commit hash). I'm wary of introducing different command behaviour depending on scope like how cabal update sometimes needs an --ignore-project option.

[mpilgrem]

I have converted the status of this pull request to 'draft', given the above.