POP-FE a tool to automate the process converting PSX disk images and install
them on various platforms, including:
* PSP/VITA
* PSIO
* PS2
* PS3
* Retroarch

This is all LGPLv2.1

TL;DR
For Linux and windows.
Connect PSP to your Linux/Windows box via USB, then run:

  $ ./pop-fe.py --psp-dir=auto /psx/Grandia1of2.img /psx/Grandia2of2.img

Or create a PS3 PKG:

  $ ./pop-fe.py --ps3-pkg=Grandia.pkg /psx/Grandia1of2.img /psx/Grandia2of2.img

or ...


Enjoy.



HOW TO RUN POP-FE:
==================
Pop-fe needs to create a bunch of temporary files while it is processing the
data. So run it in a shell from the same directory that pop-fe and its
dependencies are installed.
You do not need to copy the game images to this directory.
Pop-fe can read and process the images from remote directories.


POP-FE:
=======
A utility to automate building and installing PSX games onto different
systems.
The current directory where you run this utility from needs to be writable
so that we can use it to store temporary files during the conversing process.
It supports and can convert games stored in the following types of file
formats:

.cue   : CUE file. The preferred option. The actual image file is extracted
         from the content of the cue file. If the file-name found inside the
         cue is a relative path it is assumed that the bin/img file is stored
         in the same directory as the cue file.
.bin   : BIN/IMG files. In this case a temporary .cue file will be created
.img     in the local directory and used for the conversion. This cue file
         will assume that the bin/img file is just one single track of type
         MODE2/2352
.zip   : ZIP file. The ZIP file will be extracted into the local directory
         and if a .cue file is found it will be used.
.*     : Various memory card image formats.
         Memory cards are detected by file size so the extensions does not
	 matter.

The utility supports building and installing the games on various different
target platforms.
Command line arguments are:
-v                     : Verbose. Print additional information about what
		         operations are performed during the game installation.
--title                : Force the title to use for this game.
		         By default pop-fe will automatically discover the
			 game title from the image file provided. This argument
			 can be used to override the title if the autodetection
			 is wrong or fails.
--game_id              : Force the game id. By default pop-fe will extract
		         the SYSTEM.CNF file off the game iso and use it to
			 detect the game-id. This argument can be used to
			 override the id.
--resolution	       : By default resolution is set to '2' (640x512)
		         for PAL games. I.e. games with a GameId that starts
			 with SCES or SLES.
			 And it is set to '1' (640x480) for all other games,
			 which are assumed to all be NTSC.
			 This argument can be used to force the resolution
			 to a specific value.
		         See https://www.psdevwiki.com/ps3/PARAM.SFO#RESOLUTION
			 for a list of possible values.
--psio-dir <path>      : This specifies the path to where a PSIO sd card has
                         been mounted. The games will be be installed as
                         <path>/<game-title>/<game-title>.bin
                         If you list more than one file it is assumed these
                         are all the disks part of a multidisc game and
                         MULTIDISC.LST will be created accordingly.
                         PSIO uses CU2 files so must have cue2cu2.py installed
			 in the same directory as pop-fe.py.
--psp-dir <path>       : This specifies the path to where a PSP sd card has
                         been mounted. The games will be converted into
                         an EBOOT.PBP and will be installed as
                         <path>/PSP/GAME/<game-id>/EBOOT.PBP
                         The EBOOT.PBP will have a cover icon as well as a
                         background image embedded.
			 If you specify multiple images then a single multi-disk
			 EBOOT.PBP will be created.
			 If you specify path as 'auto' then pop-fe will
			 scan all mounted media and pick the first one that
			 contains a PSP or VITA memory card.
--psp-install-memory-card : Used to install memory card images on the PSP.
			 See memory card section below for usage.
--ps2-dir             : This is the directory that holds the USB stick to
		        install VCD files for running under POPS on a PS2.
--ps3-pkg             : This creates a PS3 PKG
			This requires CU2 files so make sure this script is
			installed.
--ps3-libcrypt        : Apply libcrypt patches also when building PS3 Packages.
		        This should normally not be needed as we automatically
			inject the magic word to the emulator running on the
			PS3.
--snd0                : Provide an audio file to be played when this game
		        is focused on the XMB. See the SND0.AT3 section below.
--auto-libcrypt       : Try to automatically create and apply a patch for
                        libcrypt. This can be used if there is no proper PPF
			file available. This is not 100% reliable.
--retroarch-bin-dir  : The directory where retroarch game bins are to be
                        installed, inside a subfolder. Includes an m3u file 
                        for correctly loading multi-dics games.
--retroarch-cue-dir  : The directory where retroarch game cues (as .CD files)
                        are to be installed with their respective bins, inside 
                        a subfolder. Includes an m3u file for correctly loading
                        multi-dics games.
                        the different discs inputted.
--retroarch-pbp-dir  : The directory where retroarch game pbp images are to be
                        installed.
--retroarch-thumbnail-dir : Where the coverimage for retroarch should go.

#
Examples:
Assume I have connected my PSP with USB and it is shows up as :
   /run/media/sahlberg/disk
   
$ ./pop-fe.py --psp-dir=/run/media/sahlberg/disk /psx/Metal\ Gear\ Solid\ VR\ Missions.cue

Or you can just let pop-fe try to discover it automagically for you. This
works for both Linux and Windows:

$ ./pop-fe.py --psp-dir=auto /psx/Metal\ Gear\ Solid\ VR\ Missions.cue

If you specify more than one cue file then it is assumed that this is
a multidisc game and we will use the first cue file to determine the
game id and title to be used for the whole set.

Example:
 # ./pop-fe.py --psp-dir=auto /psx/Grandia1of2.img /psx/Grandia2of2.img

During the conversion process the img/bin file will be temporarily converted
into an ISO file as NORMAL01.iso in the current directory.
This is in order to open the iso9660 image and read the system.cnf file
to extract the game id.
You can avoid/skip this step by forcing the game-id from the command line,
using (example only):
    --game-id=SLUS00957
    
Game art and images are fetched from https://psxdatacenter.com/
If a file ICON0.PNG is found in the game directory it assumed to be the
cover image and thus we skip pulling it from psxdatacenter.
Similarly, if PIC1.PNG is found then it is assumed to be a screenshot
or background image and again we skip pulling this file from the site.


Image files for PS3 packages
============================
PS3 packages uses 3 image files: ICON0, PIC0 and PIC1
ICON0 is the game cover and shown in the game list on the XMB.
It is square and it will be resized to 176x176 pixels.
If you provide your own custom image here, create it as 176x176 as then there
will be no blurring due to rescaling.

PIC0 is the foreground image when the game is selected in the XMB. It is
1000x560 pixels in size.

PIC1 is the background image when the game is selected in the XMB. It is
1920x1080 pixels in size.


Installing Memory Card images on PSP
====================================
Most memory card image formats are supported.
To provide a memory card image to install along with the the game
you just add the filename to the list of images.
The file extension does not matter as we detect the memory cards by file size.
Some platforms, such as PSP, require a two step process to install the
memory cards so follow the onscreen instructions carefully.

Example:
   ./pop-fe.py --psp-dir=auto Crash.cue Crash.srm


You can also install a memory card image directly for an already installed
game. But before this works, you must have ran the game at least once
so that the PSP creates the required metadata files in the SAVEDATA
directory. Since you are not providing a disk image to determine the game-id
from you must specify game-id explicitely.

Example:
   ./pop-fe.py --psp-dir=auto --game_id=SCUS94900 --psp-install-memory-card Crash.srm


VITA SUPPORT
============
Vita is supported and you can install EBOOTs for its memory card also.
The only difference is that when installing on a PSP you specify the path to the
root of the SD card. If you want to install to a SD card for a Vita then you
specify the path to the pspemu directory on the SD card:

Example:  
$ ./pop-fe.py --psp-dir=/run/media/sahlberg/disk/pspemu/ /psx/Metal\ Gear\ Solid\ VR\ Missions.cue

or just use the auto feature:

$ ./pop-fe.py --psp-dir=auto /psx/Metal\ Gear\ Solid\ VR\ Missions.cue


--fetch-metadata
================
This argument will download and install ICON0.PNG, PIC1.PNG as well as the
game id and title in the directory where the game is stored.
This requires that the game directory is writeable.

VMP.PY
======
vmp.py is a utility to convert between signed VMP memory card images
(as used by PSP) and raw memory card images as created for example by
Memory Card Annihilator on the PS2.


SND0.AT3
========
For PS3/PSP this is the audio that will be played when this games icon is
focused on the XMB. It can either be a 16bit/44100/Stereo WAVE file or a
youtube link.
If you specify --snd0='auto' then pop-fe will search youtube for the name of
the game and the string 'ps1 ost' and use the first search result.
YMMV

The game database in gamedb.py contains curated youtube links for some games.
IF a game has a curated link then it will be automatically used even if the
--snd0 argument is not provided.
Please send patches to populate these fields in gamedb.py with more entries.


Examples:

  $ ./pop-fe.py --ps3-pkg=alundra.pkg Alundra.img --snd0='https://www.youtube.com/watch?v=wKmZTw7bmI0&list=PL2S7vVonTF8UZrKK17J8FBksvYbnqyHz-&index=1'
  
  $ ./pop-fe.py --ps3-pkg=alundra.pkg Alundra.img --snd0=Alundra.WAV
  
  $ ./pop-fe.py --ps3-pkg=alundra.pkg Alundra.img --snd0=auto

If specified as a file it must be WAVE/16bit/stereo/44100Hz.

This file/link will be automatically converted into ATRAC3 format using the
atracdenc tool.


A good way to generate these WAV files is to find a good OST music file for
the game at youtube and download it in 3GP format. Then convert it to
44100Hz/Stereo/16bit WAV like so:

$ ffmpeg -i Azure\ Dreams\ OST\ 3.\ Overture.3gp  -ar 44100 -ac 2 Azure\ Dreams\ OST\ 3.\ Overture.wav


(See:
  how to create these type of files by hand:
    https://atunnic.gitbook.io/tnpsh-wiki/big-stinky-brew/aesthetic/snd0-atrac3
  relation between EA3 and WAV files for ATRAC3 data:
    https://github.com/GrapheneCt/ElevenMPV-A/blob/04885b62dcd7dfb318986cf0b553483975b745a6/ElevenMPV-A/source/audio/at3.cpp#L264
    and
    https://github.com/GrapheneCt/ElevenMPV-A/blob/04885b62dcd7dfb318986cf0b553483975b745a6/ElevenMPV-A/source/audio/at3.cpp#L132
  discussions at psx-place:
    https://www.psx-place.com/threads/pop-fe-a-utility-to-create-psx-classics-packages-for-ps3.37426/
)


PSIO
====
PSIO uses a varient of CUE files called CU2.
To generate a CU2 file when installing PSIO games you need to have the
CUE2CU2.PY tool installed in the same directory as POP-FE.PY.
This tool can be downloaded from :

  https://github.com/NRGDEAD/Cue2cu2

Just copy cue2cu2.py to the same directory as pop-fe.py and make it executable :

  chmod +x cue2cu2.py


PS2 Support
===========
Pop-fe can convert games into VCD format and install it on a USB memory stick
that has been prepared to be used with POPStarter and OPL to play PS1 games.
The USB stick must have a POPS subdirectory where the VCD file and memory cards
will be installed and an ART subdirectory for cover and background images.

PS3 Support
===========
Pop-fe can convert games into a PS3 PKG that can be installed on PS3 systems
with CFW or HEN.

RetroArch Support
=================
The PSX emulator cores in retroarch support a variety of formats, with m3u and
pbp files being the most convenient for loading multi-disc games. Pop-fe can
prepare these formats/schemes for retroarch to load.

Preparations
------------
Install ecdsa python module:

$ pip3 install ecdsa

$ cd <directory where pop-fe is installed>

Install PSL1GHT as a subdirectory in the pop-fe directory
and compile the pgcrypt module:

$ git clone https://github.com/sahlberg/PSL1GHT
$ cd PSL1GHT/tools/ps3py
$ git checkout origin/use-python3
$ make
$ cd ../../..

Then to use it just run it as

$ ./pop-fe.py --ps3-pkg=Grandia.pkg Grandia_d1.img Grandia_d2.img


LIBCRYPT
========
Libcrypt is a copy protection used on a small number of PAL releases.
https://red-j.github.io/Libcrypt-PS1-Protection-bible/index.htm

The protection is based on careful corruption of the subchannel data of the
disk and may prevent the converted games from running on PSP/PS3.

PSP/VITA
--------
When converting to PSP/VITA pop-fe will try to locate and apply a PPF patch
file to disable the libcrypt protection, making it possible to run the
game on PSP/VITA.
This is provided through a curated list of URLs for where to download
the fix. This list is very incomplete and needs to be extended, patches
welcome.  See gamedb.py/libcrypt

PS3
---
Libcrypt protected games are automatically patched by injecting the MagicWord
into the package file.
(The MagicWord is a compact representation of the subchannel corruption)
This should be sufficient to allow any games to work when converted to
a PS3 package.
If not, you can use the --ps3-libcrypt argument to force pop-fe to try to find
and apply PPF patches for this game.

ADDITONAL DEPENDENCIES:
=======================
Iso-parser
----------
You need python support to read ISO9660 images. This is needed to be able to
read system.cnf to discover the game-id.
I support two different such python extensions since the are not all available
on all flavors of linux. Pycdio and Pycdlib.
As long as you have one of them installed things should work.

pip3 install pycdio
or
pip3 install pycdlib

cue2cu2
-------
This is a small program that parses a CUE and reformats it into CU2 format.
It is used by PSIO but also to create the TOC structure we need for PSP
EBOOTs when a CCD file is not present.
It is also required when creating PS3 PKGs.

Please install this python script in the same directory as pop-fe.
It can be downloaded from   https://github.com/NRGDEAD/Cue2cu2
Make it executable by calling

chmod +x cue2cu2.py

binmerge
--------
This is a small python program to merge multi-bin files into a single bin.
This is required if you intend to use pop-fe with multi-bin disks.

This tool can be downloaded from
https://github.com/putnam/binmerge
and should be placed in the same directory as pop-fe.py.

Make it executable with
chmod +x binmerge

requests_cache
--------------
This is to reduce the amount of data we fetch from the internet.
Install this with :
pip3 install requests_cache

CDDA SUPPORT
------------
CDDA is supported for PSP/VITA/PS3 but requires an external tool to convert the
audio tracks into ATRAC3 LP2 format.
If you do not have this tool installed then pop-fe will still create the
EBOOT.PBP image but without the audio tracks.

The instructions to set this tool up are part of the Linux/Windows install
instructions below. (atracdenc)


INSTALLATION
============
Pop-fe has a few external dependencies. It requires python3 and some additional
bits and pieces. Here are some examples on how to install pop-fe and its
dependencies on different OSes:

For installations on Linux/Posix/ you need to have a build environment.
Make sure you have 'git', 'gcc', 'g++', 'make', 'cmake' installed.


Fedora36 Automatic Install:
-------------------------
sudo dnf install python-is-python3
sudo dnf install pip3
sudo dnf install python3-devel
sudo dnf install libsndfile-devel
git clone https://github.com/sahlberg/pop-fe.git
cd pop-fe
./pop-fe.py --install


Debian/Mint Automatic Install:
------------------------------
sudo apt install python-is-python3
sudo apt install python3-pip
sudo apt install libsndfile-dev
sudo apt install git
git clone https://github.com/sahlberg/pop-fe.git
cd pop-fe
./pop-fe.py --install


Debian/Mint Manual Install:
------------------------
sudo apt install python-is-python3
sudo apt install python3-pip
sudo apt install libsndfile-dev
sudo apt install git
git clone https://github.com/sahlberg/pop-fe.git
git clone https://github.com/NRGDEAD/Cue2cu2.git
git clone https://github.com/putnam/binmerge.git
git clone https://github.com/dcherednik/atracdenc.git
pip3 install git+https://github.com/nficano/pytube
pip3 install requests_cache
pip3 install pycdlib  (or pip3 install pycdio)
pip3 install ecdsa
cd pop-fe
cp ../Cue2cu2/cue2cu2.py .
chmod +x cue2cu2.py
cp ../binmerge/binmerge .
chmod +x binmerge
git clone http://github.com/sahlberg/PSL1GHT
cd PSL1GHT/tools/ps3py
git checkout origin/use-python3
make
cd ../../..
cd atracdenc/src/
cmake .
make
cd ../..

Windows (x86_64)
----------------
Install Mingw64/MSYS2
Open Mingw64 Prompt and type

export PATH=$PATH:/mingw64/bin
pacman -S mingw-w64-x86_64-gcc
pacman -S make
pacman -S mingw-w64-x86_64-python
pacman -S git
pacman -S unzip
pacman -S mingw-w64-x86_64-python-pip
pacman -S mingw-w64-x86_64-python-pillow
pip3 uninstall pillow
pip3 install pillow
pip3 install requests_cache
pip3 install pycdlib
pip3 install pycryptodome
pip3 install ecdsa
git clone https://github.com/sahlberg/pop-fe.git
git clone https://github.com/NRGDEAD/Cue2cu2.git
git clone https://github.com/putnam/binmerge.git
cd pop-fe
cp ../Cue2cu2/cue2cu2.py .
chmod +x cue2cu2.py
cp ../binmerge/binmerge .
chmod +x binmerge
git clone http://github.com/sahlberg/PSL1GHT
cd PSL1GHT/tools/ps3py
git checkout origin/use-python3
make
cd ../../..
wget https://github.com/dcherednik/atracdenc/releases/download/0.0.3/atracdenc-x86_0.0.3.zip
unzip -j atracdenc-x86_0.0.3 -x README.TXT
pip install pyinstaller
pip install pygubu
pyinstaller PSL1GHT/tools/ps3py/pkg.py
pyinstaller cue2cu2.py
pyinstaller binmerge
pyinstaller sign3.py
pyinstaller --add-data "PS3LOGO.DAT;." pop-fe.py
pyinstaller --add-data "PS3LOGO.DAT;." --add-data "pop-fe-ps3.ui;." pop-fe-ps3.py --hidden-import pop-fe --hidden-import pygubu.builder.tkstdwidgets --hidden-import pygubu.builder.ttkstdwidgets --hidden-import pygubu.builder.widgets.pathchooserinput
mkdir dist/pop-fe-ps3/atracdenc
mkdir dist/pop-fe-ps3/atracdenc/src
cp dist/binmerge/binmerge.exe dist/pop-fe-ps3/.
cp dist/cue2cu2/cue2cu2.exe dist/pop-fe-ps3/.
cp dist/pkg/pkg.exe dist/pop-fe-ps3/.
cp dist/pkg/pkgcrypt.cp39-mingw_x86_64.pyd dist/pop-fe-ps3/.
cp dist/pop-fe/pop-fe.exe dist/pop-fe-ps3/.
cp dist/sign3/sign3.exe dist/pop-fe-ps3/.
cp atracdenc.exe dist/pop-fe-ps3/atracdenc/src/.


On windows, always run pop-fe from git-bash and from within the pop-fe
directory.
The directory names for the E: drive-letter is /e etc in the git-bash shell
so use --psp-dir=/e or --psp-dir=auto

You may need to run this command every time you open the MSYS2 shell:
export PATH=$PATH:/mingw64/bin


POP-FE-PS3
==========
pop-fe-ps3.py is a graphical frontend to pop-fe:create_ps3() and can be used to
create PS3 packages from a GUI instead of using the command line.

Docker
----------------
# You can also run pop-fe inside a docker container.
# Assuming Docker is already installed on your preferred distro, run:
git clone https://github.com/sahlberg/pop-fe.git
cd pop-fe
docker build -t pop-fe .

# You can then execute pop-fe directly:
docker run -it pop-fe --help
# or get a shell and run pop-fe interactively:
docker run -it --entrypoint /bin/bash pop-fe:latest
./pop-fe.py

# Here is another Docker example where we mount the current host directory into the container at "/psp".
# Directory structure:
# .
# ├── game.bin
# ├── game.cue
# └── pspemu/
#     └── PSP/
#         └── GAME/
docker run -it --volume "$(pwd):/psp" pop-fe --psp-dir=/psp/pspemu /psp/game.cue