Created
July 30, 2023 15:26
-
-
Save ClaireCJS/1ed2666704ebec88ec3ffb3c2fa6d204 to your computer and use it in GitHub Desktop.
audio workflow description
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ********* HOW TO PROCESS A BATCH OF AUDIO **************** | |
| ********* HOW TO PROCESS A BATCH OF AUDIO **************** | |
| ********* HOW TO PROCESS A BATCH OF AUDIO **************** | |
| ********* HOW TO PROCESS A BATCH OF AUDIO **************** | |
| ********* HOW TO PROCESS A BATCH OF AUDIO **************** | |
| ********* HOW TO PROCESS A BATCH OF AUDIO **************** | |
| - create/sequester new batch into it's own folder | |
| - make a folder for the new batch, i.e. c:\new\music\BATCH.YYYYMM {good to give the batch a dated name} | |
| - find new music in %NEWCL%\music\ {or wherever else it is downloaded to} | |
| - move some music to the new batch folder | |
| - you can go into the batch folder and type "counts" to see how many files are in it so far | |
| - batch sizes are usually 200 - 2000 songs | |
| - generally, and repeatedly, if the batch folder moves around, run "bthis.bat" from the root of the batch, so that "c:\b" points to our new batch location. | |
| - This makes life easier! And "c:\b" is referred to in our notes a lot | |
| - Everything from here on out is referring to our batch ("c:\b") and not to anything else! | |
| - within the batch, run: c:\new\music\prepreprocess.bat | |
| - do all the renamings/pre-research of albums | |
| - albums should be FLAC, MP3, or WAV -- any other formats need to be converted to wav (extract-all-wavs.bat) and the original audio files trashed or archived | |
| - DO NOT look at/deal with cover art, because later we will be downloading more, so this would end up making us do it twice!! | |
| - ******** "ap" for "autoprocesss" -- does amazing things!!!! ******** | |
| - run: check-for-filenames-that-may-be-questions.bat - look for filenames that don't have an underscore representing a question mark | |
| - OBSOLETE/NO LONGER DONE: once upon a time (1995-2022), this is where we would extract-all-wavs for every damn (non-wav) adio file by doing "sweep extract-all-wavs" | |
| - but now (2023), we allow mp3/flac/wav files to exist at this point in the process, because we no longer re-encode EVERY file, because we now have faith in ReplayGain | |
| - run: from inside the batch folder, from a Windows 7 machine: sweep naw - normalize all wavs - yes even in 2023 we still do this! - still applicable post-flac because some stuff we get is WAV-only format - yes we rely on replaygain tags these days but if the WAV is there, we'd still like to make the adjustment | |
| - run: check-for-invalid-audio-files.bat - see if all the files here are in the correct format now | |
| - MOVE BATCH: into processing (mp3\_PROCESSING\processing) - run "bthis.bat" within the bath folder if you want c:\b\ to point to it easily | |
| - 202303 note: honestly, if you "bthis.bat" so that it's just c:\b\, it kind of doesn't matter where this folder really lives | |
| - "CoolEdit" / audio edit phase - fix up anything we need to fix up | |
| - open EVERY file just to visually check for anamolies / hidden tracks / annoying shit | |
| - check for TODOs ("ldg todo") afterward because filenames may indicate some stuff we were supposed to do in edit | |
| - Encode! Any edited WAV files need encoding to mp3. | |
| * These days, most edits are mp3/flac inline so no need to re-encode everything like before. | |
| * But there are still things that we get in WAV format sometimes (extracted youtube videos, CD rips), so encoding needs may still exist! | |
| * from the root of our batch: | |
| * CLOSE Adobe Audition! Seriously! | |
| (It doesn't always unlock file handles of every WAV file and the next step can fail to delete a WAV if you don't!) | |
| * run: c:\bat\audio-batch-processing---encode-everything-and-remove-redundant-WAVs.bat | |
| * just for reference, what that command does [as of 20030407] is: | |
| sweep if exist *.wav call allfiles wav2mp3 -------- encodes any WAVs into MP3 using wav2mp3.bat | |
| sweep call del-WAVs-if-mp3s-or-flacs-exist -------- deletes any WAVs that are sitting next to MP3/FLAC equivalents | |
| sweep del *.xmp -------- deletes these crap files Adobe Audition leaves everywhere | |
| - OPTIONAL: MOVE BATCH: into tagging (mp3\_PROCESSING\READY-FOR-TAGGING-1-(id3v2 only)) | |
| - we're kinda phasing this out | |
| - if you do, run "bthis.bat" within the batch to make "c:\b" point to our batch folder again | |
| - Inside the batch, run: preprocess-audio-tagging-batch | |
| - this deletes some junk files | |
| - Now, we are ready to go to tagging!!!! | |
| - TAGGING: STEP 1: INITIAL: MusicBrainz Picard pre-tag | |
| - first time setup: * have it save cover art of every type, but only embed one front image [there's a checkbox for that] | |
| - EVERY time, before starting: * make sure Options->Rename files is off!!!!!!!!!!!!!! | |
| - why do we use this? It adds a LOT of extra/useful/neat tags, genre, year all kinds of stuff Tag & Rename doesn't do. | |
| Finding what year MISC tracks came out in is super useful and time-saving. | |
| - the general process: | |
| * Note that when we say "left" and "right" pane, we are talking about the two large, main panes in the middle. | |
| NOT the folder-browser/file-picker leftmost narrow "pane" with all the directories listed | |
| 1) remove any files listed in the 2 main panes | |
| 2) use the "Add Folder" button to add the current batch (c:\b\ if you 'bthis' every time you move a batch to a new location) | |
| - it may take a long time downloading cover art - as much as even an hour for every 1000 songs! | |
| - to tell: look at the timer in the bottom-right section of the status bar next to the hourglass, except it's not that accurate | |
| 3) left pane: select all, then click 'cluster' | |
| - should be pretty quick, almost instant. | |
| - there's an opportunity to manually move unclustered files from the left pane into clusters on the right pain, but it's generally not worth bothering. | |
| 4) left pane: select all, then click 'lookup' | |
| - sometimes the button isn't enabled? like? nothing can be looked up? huh? | |
| - this can take a bit - like 3 minutes per 1000 songs? - again, watch timer in the bottom-right corner, except it's not that accurate | |
| 5) left pane: select all, then click 'scan' | |
| - this will take awhile - 10min per 1000 songs? - again: watch timer in the bottom-right corner for somewhat f a prediction | |
| 6) at this point, select all the files in the right pane, right-click, select 'save'. this should save the initial MusicBrainz tags | |
| 7) We're done! MusicBrainz can be closed! | |
| - TAGGING: STEP 2: ALBUM ART: Claire's tools | |
| - If it's a folder of MISC songs (ONEDIR, TRIBUTES, MISC, COVERS, anything like that) by differnet acts | |
| - 1) run CoverDonwloader.py to run Claire's cover-downloader script. It will take maybe 30ish seconds per song. | |
| CoverDownloader.py is very well documented within the script, if you are looking for more tips/info. | |
| - 2) run get-art.bat, which is generated in step #1 | |
| - 3) manually review the JPGs downloaded, deleting the invalid ones (so one is left per song), and ***cropping all the ones that need cropping*** | |
| - 4) run wedding-party.py to sort the married files into another folder | |
| - 5) Optionally repeat steps #1-4 again. Failing to find out in one go doesn't guarantee it won't find it in the next. | |
| Sometimes this may involve renaming the files a bit first? | |
| - 6) Eventually, decide we aren't going to repeat steps 1-5! We're done downloading and reivewing art automatically and will do the rest manually! | |
| - 7) run CoverEmbedder.py to run Claire's cover-embedder script | |
| - 8) run embed-art.bat, which is generated by step #7 | |
| - 9) no JPGs shoudl be left after this, as they are all embedded and the script should have deleted them after successful embedding! | |
| Now all that's left is the manual part; but hopefully this part reduced the work of the manual part, as album art is one of the most labor intensive parts of this process. | |
| - TAGGING: STEP 3: ALBUM ART: Mp3Tag - embed album art and review any embeddings done in "TAGGING: STEP 2: ALBUM ART: Claire's tools" | |
| - embed album art using the Mp3Tag program - note that on first run after installation you will need to go to Options->Tags->Mpeg and uncheck 'APE' under 'Read' and 'APEv2' under 'Write' and 'APE' under 'Remove' - basically, turn off APE support to properly "see" art embedded with my scripts | |
| - if missing art, first try to locally add whatever cover.jpg/png is in the folder already | |
| - we like to have >500x500 res if possible, but really, we love having >1000x1000 res. | |
| - the one we add locally isn't always >1000x1000 | |
| - we care more for stuff we actually listen to than stuff we just collect | |
| - searching can be automated using Mp3Tag's "tools": | |
| - you can use the tool (right click, but also there are some keystrokes like Ctrl-3 through Ctrl-7) to automate google image searches for better cover art | |
| - TODO: setting up the tools is tedious - finish this list | |
| #1/Ctrl-1: name: winamp.bat enqueue path: c:\tcmd\tcc.exe parameter: C:\bat\winamp-add.bat "%_path%" // check 'for all selected files': Yes | |
| #2/Ctrl-2: name: enqueueue path: c:\tcmd\tcc.exe parameter: c:\bat\enqueue.bat "%_path%" // check 'for all selected files': Yes | |
| #3/Ctrl-3: name: album-art-tool-for-mp3Tag [using filename] path: c:\tcmd\tcc.exe parameter: c:\bat\album-art-tool-for-mp3Tag.btm "%_filename%" // check 'for all selected files': Yes | |
| #4/Ctrl-4: name: album-art-tool-for-mp3Tag-beforeDash path: C:\tcmd\tcc.exe parameter: c:\bat\album-art-tool-for-mp3Tag-beforedash.btm "%_filename%" // check 'for all selected files': Yes | |
| #5/Ctrl-5: name: album-art-tool-for-mp3Tag ARTIST+ALBUM path: c:\tcmd\tcc.exe parameter: c:\bat\album-art-tool-for-mp3Tag.btm "%artist%" "%album%" // check 'for all selected files': Yes | |
| #6/Ctrl-6: name: album-art-tool-for-mp3Tag ALBUM ONLY path: c:\tcmd\tcc.exe parameter: c:\bat\album-art-tool-for-mp3Tag.btm "%album%" // check 'for all selected files': Yes | |
| #7/Ctrl-7: name: album-art-tool-for-mp3Tag artist + filename path: c:\tcmd\tcc.exe parameter: c:\bat\album-art-tool-for-mp3Tag.btm "%artist%" "%_filename%" // check 'for all selected files': Yes | |
| - for ONEDIR/MISC folders from many different years, this can actually be a good time to edit the year tag | |
| - [yes, tags can be edited in mp3tag too! it's just not the preferred tool we usually use for that] | |
| - Ctrl-3 / tool#3 / "using filename" is often the best tool for ONEDIR stuff that has the artist name in the filename | |
| - and oh: only embed ONE album art image -- the cover -- per file ... forget multi-image embedding | |
| - sure, it's possible to embed back.jpgs officially as the back, pictures of the disc.jpgs as pictures of the disc, etc etc, but: | |
| - multiple ones cause weird behavior in MiniLyrics | |
| - WinAmp can't always view the extra covers so what's the point | |
| - DEVELOPING A NEW STEP HERE - THIS IS PROBABLY WHERE WE WANT IT | |
| - BACKGROUND: Often times, when we embedded album art into our mp3|flac files, we found ones on the internet that are much higher resolution than the cover|folder.jpg|gif|png files sitting in the folder | |
| - so we need to make these superior higher resolution versions in the mp3|flac files ALSO be the one sitting in the folder as a separate file | |
| - "eaa" is the command - extract album art - this exracts the cover art and overwrites the existing one ONLY IF it's a larger file | |
| - but when should we run it? | |
| - always makes sense to run in a folder that already has a cover|folder.jpg|gif|png because it may find a higher quality one | |
| - always makes sense to run in album folders (easy to detect as they almost always have filenames that start with a number) | |
| - never makes sense to run in MISC/ONEDIR type folders | |
| - WHAT TO ACTUALLY DO: | |
| "run-eaa-in-appropriate-subfolders" - this will sweep through and run "eaa" if it's an album/appropriate place to do so. Involves over 20 checks. | |
| - finally, go through a full directory-crawl reviewing/deleting/renaming the downloaded album art from the previous step so that everything is perfect | |
| - TAGGING: STEP 3: THE ACTUAL TAGGING: - Tag&Rename | |
| - get all fields of all file tags right | |
| - finally giving up on id3v1 tags in 2023! | |
| - TAGGING: STEP 4: add ReplayGain tags - "sweep argt" - to all (flac/mp3) files | |
| - may not work with songs (esp flac) that have a percent in their name! May have to temp-rename (esp flacs) to work at command line! | |
| - won't find this out until the review later, just something being mentioned | |
| - manually review ReplayGain tags | |
| - WE DO IT AUTOMATICALLY NOW!!!!!!!!!!!!!!!!!!! | |
| * Step 1: run: audit-music-files.py ----------- requires working python with a lot of installed packages tho | |
| * Step 2: if RG_fix.bat is generated, run RG_fix.bat and repeat Step 1 | |
| If repeating leaves some stragglers (RG_no.dat is nonzero in size) you may have to refer to "SOME SONGS WILL HAVE FAILED" in the deprecated instructions below | |
| - DEPRECATED BUT KEEP AROUND: | |
| - walk the folder tree manually, typing "lr" [list-replaygain-tags] at command line in each folder | |
| - use "nd" to go to next folder for ease | |
| - use "nd" only after being done with any subfolders -- don't forget "cheese", or "ORIGINAL-VERSIONS", or *any* subfolders | |
| - SOME SONGS WILL HAVE FAILED! That's why we're doing the review! | |
| - be vigilant about keeping an eye out for songs that have a % symbol in them, they (esp flac) may have failed | |
| - the command line tool only works with 2-channel songs, so all 5.1 songs will fail | |
| - occasionally some files will fail randomly because metaflac/metamp3 isn't perfect at reading/writing the tags | |
| - *** for all of these various failures, you can open the file in Foobar media player, and rightclick the files to add ReplayGain tags with Foobar *** | |
| - when you notice one that's missing, use "cthis.bat" to make c:\c\ point to the folder you are in | |
| - open Foobar | |
| - file->add file->browse to "c:\c" if it's not there already->add the file to foobar | |
| - right click the file(s)s -> ReplayGain -> scan as invidual track/album (depending on which is appropriate) -> update file tags | |
| - then do "lr" again to verify that it worked | |
| - sometimes EVEN AFTER THAT, they still won't be viewable at the command line ("lr" will fail). | |
| - it's extremely rare | |
| - open them in WinAmp to verify the ReplayGain tags were successfully added | |
| - if they still aren't viewable, WE HAVE A PROBLEM. It may behoove us to manually normalize the audio in Adobe Audition if we notice problems. | |
| - FINAL CHECKS: | |
| - run "sweep dep-mp3s-if-flacs-exist" to deprecate mp3s if a companion flac exists | |
| - we deprecate the file rather than delete, in case something goes wrong, it's nice to have the m3ps to fall back on. | |
| - A few years from now (2022) when we are more comfortable with the validity of flac files, we can delete the deprecated companion mp3s, but for now (2022), we're going to keep them around | |
| - run "ldg \.wav$" to check for WAV files -- they really shouldn't be here at this point in the workflow, at all!!!!!!!!! | |
| - at this point, any WAV files were already supposed to be mp3s that have embedded art and are tagged properly, so wtf! | |
| - might have to kick it back for a future batch, delete it, or figure it out | |
| - run "ldg TODO" to check for any TODOs that may need to be done | |
| - THE FINAL FINAL CHECK: | |
| run "dir /s /p" to just do a basic visual review that everything looks ***FUCKING BEAUTIFUL AND AS PERFECT AS CAN BE*** | |
| - MOVE BATCH: final assimilation of new batch into collection repository - generally one folder at a time | |
| - SUMMARY: now is time to move files from our batch to the main, official mp3 collection! | |
| - move the entire batch into \mp3\_processing\testing\BATCHNAME [we could get away with skipping this though] | |
| - it may behoove us to make a playlist of the music here and let it run for awhile to actually test it, but life goes by too fast to always allow this | |
| - there is a preprocess.bat inside \mp3\_processing\testing\ that may need to be run now! | |
| - TIP: we should [have already] ran "bthis.bat" in our batch, so that the batch is just c:\b\ | |
| - TIP: for best efficiency, one screen (left?) should be at c:\mp3\ at the command line, and another should be at c:\b\ (our batch) | |
| - do album/song upgrade-replacements first, as these are the most complicated to do, and don't affect the base attrib lst | |
| - when replacing an album: | |
| - takes some critical thought | |
| - deprecate original album folder at begining of process to prevent confusion | |
| - grab it's shit! | |
| - copy txt & lrc files - check that they match the new album version's naming | |
| - move attrib.lst from old to new - if the names of the songs changed (i.e. a 1-disc album is now a 2-disc album) attrib.lst will need updating | |
| - move artwork/image files | |
| - other stuff will require judgment, it may or may not apply | |
| - put.bat old version to \mp3\ready-to-delete\{bandname} -- you may need to create the subfolder | |
| - then do ONEDIR folders - each song in the DIR must be dealt with ONE at a time, case-by-case | |
| - next, process ALL non-album songs | |
| - any MISC folders (even ones within other folders--type "tree|highlight misc" to look for those) should be gone | |
| - there should be no "loose" songs whatsoever anywhere, for example often a \TRIBUTES\ folder will have non-album/loose songs | |
| - Now we are at a point where the only remaining stuff are new *ALBUMS* by *BANDS* (right? maybe type "tree" to be sure?) | |
| - the new albums need to be added to our base attrib lst!, which is done in regex | |
| - type "al" to open up base attrib lst | |
| - search for "NEWEST ASSIMILATED MUSIC GOES HERE" which has arrows pointing upward to the place new albums are supposed to go into | |
| - from inside of your BATCH folder: | |
| c:\mp3\_PROCESSING\testing\print-album-data-for-BaseAttribLst.bat >clip: | |
| ...and paste the results into the base attrib lst | |
| - the tricky part here is hand-editing the entries. | |
| You want them to be specific enough to ONLY hit the new files. | |
| But not so specific that they don't match if things change slightly. | |
| It's an art. | |
| You can use always mchk to check if you edited it right -- but only AFTER running the hour-plus index-mp3 process. | |
| - things to take out... | |
| - For example "incomplete" and "todo" references should be taken out, they may be tempmorary | |
| - "the.*" in bandmames should usually go, because we often invert things like "The Church" into "Church, The" and best to just not use the unless necessary | |
| - often times the year should be taken out if it's at the very end | |
| - 'MISC' folders and such -- better to treat those in the folder (creating MISC\changer, MISC\changerrecent, moving files one at a time into those) than to ever try addressing those in the base attribute file | |
| - things like this are onedir folders, the fact that the folder is called 'changerrecent' is what controls things, it should not be in here except as an informative comment that such folder exists: | |
| #TV.*CARTOONS.*1980s.*changer.*cheese:changer | |
| ^see how it's commented out? because there's no need to put this folder in changer, the fact that 'changer' is part of the folder's name does it already | |
| #Metallica.*MISC:changer | |
| ^see how it's commented out? because MISC should have 'changer', 'changerrecent', 'tolerable', etc folders and THAT's how those get put in changer/changerrecent/tolerable/etc | |
| - make sure the new entries are only the new albums! | |
| - if it's an album that is demos - the filename may be ambiguous with the studio album. | |
| - You may need to rename the original album foldr to disambiguate for example: "God Ween Satan (demos)" and "God Ween Satan (the album)" | |
| - and then update the entries for *both* versions in attrib.lst so that the demo-version and non-demo-version are both distinct | |
| - make sure to comment out those albums that aren't really albums ("UNPROCESSED/ARCHIVED-ORIGINALS" folders often end up here) | |
| - make sure to make decisions about which ones are going into changerrecent, or just changer, or just Christmas, or just preferred | |
| - by default things get added like this: {note how changerrecent also has a year and decade version of changerrecent} | |
| - example: BabyMetal.*2.*2014.*Metal.*Resistance:changer,changerrecent,changerrecent2022,changerrecent2020s | |
| - example of something we'll listen to later and put in changer but not changerrecent: Bis.*Some album:changer,to learn | |
| - "to learn" is kind of a "let's put this in changerrecent at a later date" note-to-self | |
| - when making a new folder for a brand new band that we've never had albums before | |
| - keep in mind that there may be songs by that band in the collection already that maybe should live in this new folder instead of where they were. Maybe. | |
| - and if they don't live in the new folder, nice to make a >"__ zero byte note talking about this other track elsewhere __" just to link things together | |
| - "STEP GREEN" - time to start moving folders into the official collection! | |
| - do non-music | |
| - do TV/cartoon/movie/novelty stuff | |
| - eventually it's possible that we will get to a point where everything in our current batch precisely matches the structure of where it needs to go and can all be moved in with one command | |
| - might not be a good idea for Carolyn to ever do that tho | |
| - run backup-mp3s to back up everything to all backup locations, reflecting the new changes | |
| *************************************** ALL DONE!!! *************************************** | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment