Writing Traktor data to ALAC / iTunes files

Discussion in 'Feature Suggestions' started by ibsh, Mar 23, 2013.

  1. ibsh

    ibsh NI Product Owner

    Messages:
    75
    Hey all,

    My collection is almost exclusively ALAC (too many iDevices to go FLAC, sadly) and each time I upgrade to a new version of Traktor I have to reanalyse/beatgrid/cue point several files (up to 10% of my collection) because the NML database migration* loses data from some of my songs. This is most tiresome.

    This doesn't appear to happen with file formats where the grid/cues etc are stored directly in the file metadata (eg MP3), and I was just wondering why Traktor doesn't store its data in ALACs. Obviously the iTunes metadata standard is different to, say, ID3, but clearly it can be read and written by Traktor (e.g. for BPM) so I don't see why the core collection data isn't stored in a custom atom of the file data.

    Has anyone else had any experience with this?

    * it has occurred to me to manually (or with a Ruby script or similar) copy the entries from the old NML file, but I don't want to corrupt the new one and haven't found any documentation of the NML structure.
     
  2. malzfreund

    malzfreund NI Product Owner

    Messages:
    2,495
    When starting a new Traktor version for the first time, it asks you whether it can copy over your settings from the previous version. You let it do that, right?

    And the ALAC files are part of your Traktor collection, right?

    (I know I'm asking pretty basic stuff. It's just for clarification.)
     
  3. ibsh

    ibsh NI Product Owner

    Messages:
    75
    Yes and yes. And no problem =)
     
  4. malzfreund

    malzfreund NI Product Owner

    Messages:
    2,495
    I'm not using ALAC myself. Hope someone else will chime in. But this sounds like a bug to me.

    It's weird, though. ALAC is also MPEG-4/Part 14, just like AAC. Both use the same metadata format. Both can be decoded using the QT framework.
     
  5. NReek

    NReek NI Product Owner

    Messages:
    1,227
    i'm in the same boat, mostly alac collection.

    i have not observed the issue you describe in a massive manner, however i'm prone to (grid) lock all files once the grid has been (manually) reviewed/validated.

    there's always room for surprises when re-importing itunes collection files if these are not locked.

    almost 2 years ago, i opened a feature request and sent several emails to support asking for *official* alac support, including 24bit (currently not supported at all). of course, nothing in the future features roadmap is ever confirmed or denied, and expecting to retrieve an ETA is a chimera.

    as stated in the post, i don't understand the point of marketing as "itunes compatible" when there's no *official* support for apple lossless.
     
  6. ibsh

    ibsh NI Product Owner

    Messages:
    75
    Yeah, I should have mentioned that the tracks in question had their grids locked too. It doesn't seem to be relevant to whether they have tags saved.

    NReek, would you be willing to check some of your files with kid3 to determine whether Traktor is saving the cues etc to ALAC files in your case?
     
  7. NReek

    NReek NI Product Owner

    Messages:
    1,227
    yes of course, i'll check some of them this afternoon at home and let you know what i find.

    what do you expect me to find, that files have no traktor tags at all? that some of them do and some not?
     
  8. malzfreund

    malzfreund NI Product Owner

    Messages:
    2,495
    I had some free time on my hands and did a quick test.

    I generated an 16/44.1 ALAC using QuickTime. Tagged it with Mp3tag using Discogs metadata as the source. Imported into Traktor and analyzed. Added a few hotcues and locked grid.

    In Mp3tag, I can then see that Traktor adds a field "INITIALKEY". I do not see any other field being added.

    Then I transferred the file to another computer. Imported the file to Traktor on that other box. Beatgrid is right and locked and all hotcues are there.

    So, on my Windows boxes, I don't see any obvious problem.
     
  9. NReek

    NReek NI Product Owner

    Messages:
    1,227
    now that is a weird finding!

    where is that stripe, cue and beatgrid info stored, if it's not in the tag?

    as i have just one computer, one collection, it's been a while since i cared about if it was being stored in the tag or in the collection as long as it was showing properly, which it did.

    after malzfreund input, i'm curious.
     
  10. malzfreund

    malzfreund NI Product Owner

    Messages:
    2,495
    i'm curious as well. obviously, i transferred nothing but the audio file to my second traktor installation so the data MUST be in the file. but NI has started hiding it somewhere, I cannot see a field being added (other than the key-related field) in a tag editor.

    btw, the same applies to .m4a files (AAC in MP4 container) as well. I cannot find Traktor-specific tags in those files, and I've looked at them with Mp3tag, Kid3, and puddletag.
     
  11. NReek

    NReek NI Product Owner

    Messages:
    1,227
    i guessed you were doing it properly, i mean just moving the file to a different collection to see what was embedded, but thanks for confirming it.

    i believed that whenever the info could not be embedded for whatever the reason, it was stored in the collection database and 'linked' to that file by its filename.

    may they be attaching an additional proprietary tag outside id3 and/or whatever m4a may have?

    as long as it is bombproof, i'd be happy to find that's how it's being done currently. it would also clear any doubt about traktor messing with id3, for those that have concerns.

    is this just happening for m4a audio, or mp3 as well?

    i guess for mp3's the test should be done with a 'virgin' one, freshly compressed, getting in a traktor collection for the first time. (sorry for indirectly giving you more homework malzfreund :))
     
  12. NReek

    NReek NI Product Owner

    Messages:
    1,227
    update:

    i've picked 2 alac tracks and checked their tags before being added to traktor for the first time.

    i've checked them with both kid3 as suggested and musicbrainz picard.

    i've added them to the traktor collection, corrected the grid, added a couple of cues and locked them.

    i've checked them back with both apps, and no tag has been added, not even initial key.

    however, i've found a weird issue in picard. look at the screenshots before and after. look at the changes in "original value" column. although the data still shows properly in itunes, traktor and kid3, picard no longer shows it.

    i don't know how to interpret this. i need to understand what is happening, starting by what are the differences between mp3's id3 and whatever the container it is for tags in m4a files.
     

    Attached Files:

  13. ibsh

    ibsh NI Product Owner

    Messages:
    75
    Thanks for looking into this, fellas.
    malzfreund's findings are genuinely weird. I'm gonna have to take a look at that myself; if a tag inspector can't see the difference then a hex editor will. Will try and find some time over the weekend.

    And NReek, you can check out some information on the iTunes metadata standard here, but it's a bit technical as you might imagine.
     
  14. malzfreund

    malzfreund NI Product Owner

    Messages:
    2,495
    i think similar things are happening with mp3. traktor doesn't write a tag visible in a tag editor, probably one would need to search for it using a hex editor.

    i think the behavior for mp3 changed some time ago. i'm def curious enough to do some testing at some point. actually, what i also wanna do is grab an old backup of my audio files and check whether there was a visible tag in the past (i believe there was).

    i believe INITIALKEY is added only when doing a full analysis (including key analysis) for the first time.

    neither do i.

    please share what you find.

    yeah, i came across this as well. "iTunes-style metadata isn't defined in any publicly available document - its format is determined by the types of files that iTunes & the iTunes Music Store produce & provide."

    it's a shame that there's no tightly-defined and universally-accepted tagging standard. and it's not just a problem with mpeg-4 part 14. id3 and vorbiscomment aren't without problems, either.
     
  15. ibsh

    ibsh NI Product Owner

    Messages:
    75
    OK. Quick experiment complete. I lined up three formats of the same track, each with minimal tags:

    WAV, ID3 v2.2 tag
    MP3, ID3 v2.4 tag
    ALAC, MP4 tag

    Analysed them all in Traktor (just for BPM, not key or gain), and made sure the data persisted between two Traktor sessions. Then deleted the tracks from the collection and reanalysed.

    The WAV file had no changes to its ID3 tags but it had about 33kB added to it in a LIST chunk at the very end of the file. It appears to contain the Traktor information; when I re-added it to the collection the grid and cues were still there.

    The MP3 file had a BPM and a TRAKTOR4 frame added to its ID3 tag; these were visible in kid3. Again, when I re-added it to the collection the grid and cues were still there.

    The ALAC file had only a BPM field added to its MP4 tag. It had something like 40k added in size, in a block after the tag but before the audio; and it once again retained the Traktor information.

    So I guess some of my assumptions were wrong. But I'm left wondering why the hell my tracks can lose information between Traktor versions if it's all stored in the file.
     
  16. malzfreund

    malzfreund NI Product Owner

    Messages:
    2,495
    you're right. i missed that. they're not visible in MP3tag as they're PRIV frames which you cannot edit.

    My guess: NI may modify what's stored in TRAKTOR4 between versions. And a lack of backwards compatibility is why you don't see information when going back to an older traktor release.

    Admittedly, that's nothing more than pure speculation. I did some googling. It seems a guy wrote a tool that displays the contents of TRAKTOR4 based on some reverse-engineering. (Unfortunately, the tool is a few years old and based on 1.2.7.) For me personally, it's not worth digging deeper into that. But I just thought I mention it considering you're a coder and may be interested in that.
     
  17. ibsh

    ibsh NI Product Owner

    Messages:
    75
    Will take a look, nice one.