Tool [Alpha 0.4.0] StarFuse :: .pak utility for Linux/OS X

Discussion in 'Starbound Modding' started by Qix, May 16, 2016.

  1. Qix

    Qix Space Hobo

    starfuse_demo.gif

    StarFuse
    Mount .pak files as regular directories under Linux and OS X

    This tool allows developers to mount StarBound .pak (and .modpak) files as regular directories.

    This tool is currently in alpha stage. It should work as a read-only filesystem. The Beta release will include writing to existing Pakfiles, and the first major release will include creating new Pakfiles altogether.



    Installation
    Installation requires that the FUSE libraries be installed on your system. They do require administrator access (usually not a problem) but are otherwise easy to install.

    Linux:
    (You may need to update these commands to make sense for your particular distribution)

    Install pip (python package manager) if you don't already have it:
    Code:
    $ sudo apt-get install python-pip python-dev build-essential
    $ sudo pip install --upgrade pip
    $ sudo pip install --upgrade virtualenv
    
    Install FUSE:
    Code:
    $ sudo apt-get install libfuse-dev
    
    Install StarFuse:
    Code:
    $ pip install starfuse
    
    Mac OS X
    Install pip (python package manager) if you don't already have it:
    Code:
    $ sudo easy_install pip
    
    Install OSXFUSE via this link (or by visiting https://osxfuse.github.io/ and choosing the appropriate OSXFUSE installation for you).

    Alternatively, you may use Homebrew to install OSXFUSE, though I have never really had the best luck with that.

    Finally, install StarFuse:
    Code:
    $ pip install starfuse
    


    Using StarFuse
    Once installed, it's fairly easy to use StarFuse. Most users will want to use the following:

    Code:
    $ mkdir ~/pak
    $ starfuse /path/to/packed.pak ~/pak
    
    Right now, there is no daemon mode (fairly small change; I just haven't added it yet) so it will stay there until you unmount it:

    Code:
    $ unmount ~/pak
    
    Make sure no programs have any files open (including any terminals cd'ed into it) prior to unmounting!

    For completeness, here is the full help documentation:

    Code:
    $ starfuse --help
    usage: starfuse [-h] [-v] [-w] [--pages PAGES] pakfile mount_dir
    
    Mounts StarBound .pak files as FUSE filesystems
    
    positional arguments:
      pakfile        the .pak file on which to operate
      mount_dir      the directory on which to mount the PAK file
    
    optional arguments:
      -h, --help     show this help message and exit
      -v, --verbose  be noisy
      -w, --write    allow modifications to the .pak file
      --pages PAGES  map this number of pages at a time (default: 256)
    


    FAQ

    Why would I use this?
    Right now, to browse the contents of a .pak or a .modpak file under linux/OS X, you must use Chucklefish's provided asset unpacker utility, which is not only slow but requires that you unpack all of the modpack's contents before you can really do anything with them.

    Not only does this process take a while, it also duplicates the files on your system (meaning not only do you have a large .pak file to begin with, you now have a whole directory with exactly the same contents).

    Luckily, because Chucklefish wrote their filesystem format correctly (seriously, well done), this leaves .pak files very similar to regular ol' hard disk formats, and thus easy to emulate using FUSE.

    This means, without unpacking every file from a .pak file, you can browse the contents of a .pak file as if it were already unpacked into a directory. This includes using image editors, text editors, regular filesystem commands (mv, cp, tar, etc.) on the contents - without ever unpacking them.

    Does this work with modpak files?
    Short answer: most likely.

    Longer answer: I haven't tested it (lack of time). If you have tried it out, let me know if it works/doesn't work in a reply on this thread and I'll update accordingly!

    What's the catch?
    Right now, you can only read from the pak files. Why? Because writing to them requires some lengthy code that has not yet been written (that's the next step, I promise!)

    As well, you have to install FUSE libraries for your system, which requires administrator privileges. Don't worry, they won't interfere with anything and are completely harmless. If you have already used FUSE filesystems before, chances are you don't need to install anything other than Starfuse itself. Neat!

    When will I be able to edit Pakfiles?
    In Beta :)

    When will I be able to create new Pakfiles?
    Short answer: at the first major release.

    Slightly longer answer: When modification functionality is added in beta, you could (theoretically) take an existing pakfile, copy it, mount it, and just remove all of the files in it already to create an empty pakfile.

    Why release before having modification functionality?
    For a few reasons, two of which being to find bugs, as well as to gauge what's more important to the community in terms of functionality. If there is something I can deliver quickly that I didn't think of, I'd rather catch it now than later.

    I do have a full time job (and by full time, I mean take-over-my-life job) so the times I can work on Starfuse are pretty limited. Getting people to test it out really helps.

    I'm getting a Too many open files error!
    Run Starfuse with the --pages flag set to something around 1024.

    Code:
    $ starfuse --pages=1024 /path/to/pakfile.pak ~/pak
    
    If it still occurs, try setting it even higher - 2048 should be high enough. Anything higher and it's definitely a bug. Most users shouldn't run into this problem unless they're on some crazy bizarre operating system (or, if you're working with really, really big Pakfiles).

    I have a problem, where do I post about it?
    As of right now, protocol should be to check this thread first to see if it's been brought up. If not, make a reply to this thread about it (mods are welcome to police this however they please).



    Meta
    The source code is available as open source: https://github.com/qix-/starfuse
    Documentation is available at CodeDocs: https://codedocs.xyz/Qix-/starfuse/annotated.html

    Major props to Blixt for some solid work in this area. The BTreeDB4 implementation as it stands right now (version 0.4.0) is a moderately refactored version of his code. The SBON implementation is, for the most part, completely unmodified (originally by Blixt).

    As well, thanks to The Suit for his blessing on this project.
     
    Last edited: May 16, 2016
    The | Suit likes this.
  2. Qix

    Qix Space Hobo

    [reserved]
     
  3. The | Suit

    The | Suit Agent S. Forum Moderator

    Hah you really didn't need my Blessing - but always a fan of more tools!
    Great work by the way - I am sure it would make advanced modders on Linux and OSX lives easier.
     
  4. RenoFezzedOne

    RenoFezzedOne Void-Bound Voyager

    I'm having issues with it:

    upload_2018-6-23_14-53-9.png

    For reference, I'm running Python 3.6 on Arch.
     

Share This Page