close
Skip to content

polymonster/diig

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

661 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

diig

scrape release

A music platform for record collectors. Available on the web and as a native iOS/Android app. Fast audio player, infinite scroll, offline caching. Swipe through new releases and charts from your favourite stores, tap through to buy.

Check out the website.

Image

Beta App

Available now on iOS and Android. Contact to join:

Image

Contributing

Contributions welcome. Got a store you want scraped? Raise an issue or better yet, implement it yourself and open a PR.

If you're finding diig useful and want to help keep it running, sponsorship helps cover developer fees and hosting costs.

Scrapers

Scrapers run nightly to pull releases from record stores. Each store needs a schema defining its structure and a parser to extract release data from HTML.

Running Locally

pip install google-auth google-auth-oauthlib google-auth-httplib2
cd scrape
python3 dig.py -urls -store juno -verbose -local-only

Results are written to local registry files only — no Firebase credentials needed. Run python3 dig.py -help for full options.

Schemas

Release Schema

{
    "id": "store-specific release id",
    "link": "url to release page",
    "artist": "artist name",
    "title": "release title",
    "label": "label name",
    "label_link": "url to label page",
    "cat": "catalogue number",
    "artworks": ["small", "medium", "large"],
    "track_names": ["track 1", "track 2"],
    "track_urls": ["url to mp3 snippet", "..."],
    "store": "store name",
    "store_tags": {
        "out_of_stock": false,
        "preorder": false
    }
}

Store Schema

{
    "sections": ["techno", "house", "electro"],
    "section_display_names": ["Techno", "House", "Electro"],
    "views": {
        "weekly_chart": {
            "url": "https://store.com/${{section}}/charts/weekly",
            "page_count": 1
        },
        "new_releases": {
            "url": "https://store.com/${{section}}/new/page/${{page}}",
            "page_count": 10
        }
    }
}

Discogs

Discogs links are fetched or attempted to be fetched for all releases. This algrithm is work in progress and it is tricky to find the exact match in some cases. You can add items directly to you Discogs wantlist from in the app. In time deeper integration with discogs will be implemented.

Building the Web App

Built with Nuxt and deployed to Firebase Hosting.

Dev server

cd web
pnpm install
pnpm dev

Generate & deploy

Requires the Firebase CLI (npm install -g firebase-tools).

cd web
pnpm generate
firebase deploy

Building the App

Built with C++, Objective-C and ImGui on top of pmtech.

git submodule update --init --recursive
cd app
pmtech/pmbuild ios    # or: mac, android

Xcode/Android Studio project lands in app/build/<platform>. Build from IDE or command line:

pmtech/pmbuild make ios diig