Latest post Mon, May 20 2019 9:56 AM by Phil Norris. 15 replies.
Page 1 of 2 (16 items) 1 2 Next >
Sort Posts: Previous Next
  • Wed, Feb 24 2016 8:22 PM

    .avb ID structure

    Can anyone share the way .avb ID is structured? I have isolated the byte sequence in .avb files that make Avid bins unique. I can create new bins with unique IDs outside of Media Composer by writing random values to these bytes. But I have no clue how this ID is structured, if it's trully random or if there is some order to it.

    To make it a bit more clear--each Avid bin has a unique ID. This is what prevents you from opening the attic bin and the parent bin at the same time. The bins may have different file names but if the ID is the same they are considered to be the same bin by Media Composer. This logic works well for attic retreaval in a way. But I'm writing automation tools and want to avoid creating invalid IDs.

    Igor Ridanovic

    www.HDhead.com

    Filed under:
  • Thu, Feb 25 2016 1:48 PM In reply to

    • Marianna
    • Top 10 Contributor
    • Joined on Thu, Oct 13 2005
    • Avid
    • Posts 10,674
    • Points 224,020
    • Avid Beta Moderators
      Avid Customer Advocate
      Avid Developer Moderator
      BlogAuthor
      SystemAdministrator

    Re: .avb ID structure

    Hey Igor....

    I reached out to a few folks in engineering who I think can help.... might take a day to get  response back as I dont know their schedule but stand by....

    I will email you once I know.

    Hope all is well on the west coast...

    Marianna

    Director of Online Communities and Forums/Customer Advocate [view my complete system specs]

    marianna.montague@avid.com

    mobile 813-493-6800

    Twitter:  avidmarianna

    Facebook: Marianna Montague

    www.avidcustomerassociation.com   |  Connect 2019 | April 6-7 | Aria, Las Vegas, NV

    WWLD

  • Thu, Feb 25 2016 5:04 PM In reply to

    Re: .avb ID structure

    Thanks Marianna.

    Igor Ridanovic

    www.HDhead.com

  • Tue, Feb 7 2017 3:10 PM In reply to

    Re: .avb ID structure

    I'm also automating with a django module in python - and I can imagine a lot that can be done if I know for certain how the bin ID fits in and is generated, especially if any of the Interplay modules are dependent on it - which might cause problems later (creating archives, moving, nearline etc.)

    As far as I can tell Interplay doesn't totally depend on the bin ID - as you can check in a bin with the same ID into the same project in Interplay (even though you can't open them at the same time in MC) - I tested deleting one set of referenced assets - and Interplay leaves the media alone as it is protected by the other (ie. they seem to be unique instances in Interplay - as you might expect for an asset manager!) but as far as transfer, nearline archive etc. I'm not at all sure what might happen with those modules and if they start to depend on binaries from MC.

    I am also interested in finding out if the bin ID generated on the project level by MC is totally unique - or unique only on a project level. eg.

    Opening the same bin in the MC project twice generates an error because of the clash of both ID's being registered (probably) in the projects binary *.avp file, but I can't be certain that a project might also generate a new bin with an ID that is the same as an ID generated by some other project - this would lead to clashes later at some point if people are moving non-unique bins from project to project.

    As far as I'm concerned, I only want to create templates - so the bins would be generated and added (windows copied) on the creation of a new project - and the IDs will get registered in the project automatically before any other bins are created.

    Would these template bin ID's then become not available for assignment as new bin ID's in this or in any other project?

     

     

  • Thu, Feb 9 2017 2:35 AM In reply to

    Re: .avb ID structure

    Phil Norris:

    ... interested in finding out if the bin ID generated on the project level by MC is totally unique - or unique only on a project level. eg.

    Opening the same bin in the MC project twice generates an error because of the clash of both ID's being registered (probably) in the projects binary *.avp file

     

    I don't think bin IDs are registered in the .avp. You could test that by comparing hex dumps of a .avb and a corresponding .avp file. I could easily be wrong but my feeling is that MC simply won't open two bins with the same ID.

    We use a project creation script I wrote pretty much daily. It generates a MC project of desired frame rate and color sampling and populates it with a complex standard bin structure we use (some bins are empty and some hold commonly used assetts).

    For the non-empty bins, the script simply copies existing .avb files and renames them. For the empty bins, I use a short binary of an empty .avb file, replace the ID with a random ID and save as a new .avb file.

    Igor Ridanovic

    www.HDhead.com

  • Thu, Feb 9 2017 2:39 AM In reply to

    Re: .avb ID structure

    I just read my original post. After about a year of randomly generating bin IDs, I'm confident that there is no harm to doing it externally. The projects have been 100% stable on both Mac and Windows. I never heard back from anyone in the engineering, but you know how it's with these old pieces of code. It's possible that no dev has looked at how bins are generated for years. It's just works and they have other things to worry about.

    Igor Ridanovic

    www.HDhead.com

  • Mon, Feb 13 2017 9:21 AM In reply to

    Re: .avb ID structure

    Hi Igor - thanks for your responses.

    I agree with you. In this case MC is just reading bin ID's directly from the project folder. Some other NLE's keep track of bins directly in the project file, and some helpfully store the project file as a readable XML like format and not as a binary (GV Edius and Premiere, also Final Cut) - which makes injecting stuff into the project much easier.

    Igor Ridanovic:

     

    For the non-empty bins, the script simply copies existing .avb files and renames them. For the empty bins, I use a short binary of an empty .avb file, replace the ID with a random ID and save as a new .avb file.

    If you have many editors sharing the same projects, injecting populated bins into the project like this could eventually cause a clash of ID, as you can't really control what these folks do Hmm - and I'm not at all sure it's safe to do this if you are using Interplay.

    With the exception of empty bins where you changed the ID maybe?

    Question - If you know the address of the ID, then why not change ID in bins that contain content too?

     

    Edit:

    Ok I 've been looking at a few empty bins and it looks as if quite a lot changes across the file.

    0000   ◦◦Domain
    0008   DJBO◦◦AO
    0010   bjDoc◦◦◦
    0018   2017/02/
    0020   13 11:26
    0028   :35◦◦◦◦◦
    0030   ◦◦◦IIII◦
    0038   ◦◦X1_◦◦b
    0040   oTAevTA◦
    0048   ◦Media C
    0050   omposer
    0058   8.2.11

     

    I guess the bin ID might be 8 bytes at hex 0030 - last byte 56th after IIII & right before the string 'boTAevTA'

    - but quite a bit changes also at the end of the file too in the last 64 bytes or so...

     

    Would be nice to know what you are doing Igor?

     

  • Mon, Feb 13 2017 10:27 PM In reply to

    Re: .avb ID structure

    I'll need to dig up the documentation I've left behind when hacking .avb, if any. I'm trying to remember why I decided not to alter the bin IDs of non-empty bins. It's possible that the ID wasn't clearly delineated. My recollection is that it was located somewhere towards the end of the binary and it shifted place based on the dynamic changes to the bin contents. Therefore, the location was never the same.

    But either way, in the normal work day we may have 4-5 editors working off the same project sharing these bins. I have not seen any ill effects aside the fact if I have a let's say 01_StandardSlates_Proj1.avb open I can't open 01_StandardSlates_Proj2.avb from a different project at the same time. They have different names but are sharing the ID. The behavior is the same as what you get when trying to open Avid Attic bins.

    Ok, I don't see any documentation-to-self on my laptop, but here are comment lines and Python code from the project generator that may shed some light:

    # Bin ID consists of 2 byte ID, 2 byte delimiter and 3 byte ID, so I believe.

    # We are only replacing the 3 byte ID which gives us 2^24 permutations.

    # "CD56' delineator is not consistent from project to project. Must use a knwown avb.

     

    hxlist = hx.split('cd56') # split around 'CD56' delimiter

    hxlist[0] = hxlist[0][:48] + newTimestamp + hxlist[0][86:] # replace timestamp

    hxlist[2] = newID + hxlist[2][6:] # replace 3 byte bin ID in the 2nd element

    hxmod = 'cd56'.join(hxlist) # put 'CD56' delimiter back in place

     

    # Back to hex values

    hxdec = hxmod.decode('hex')

     

    # Write to new avb file

    avbPath = os.path.join(dirPath, avbName) + '.avb'

    newavb = open(avbPath, 'wb')

    newavb.write(hxdec)

    newavb.close()

    Igor Ridanovic

    www.HDhead.com

  • Mon, Feb 13 2017 10:44 PM In reply to

    Re: .avb ID structure

    Hmm,


    Rereading my commnet lines, I don't see how the 0xCD56 could be a delimiter if it changed from project to project. It would probably be more correct to say that the bin ID is 2+2+3 bytes long and that the 0xCD56 is a part of it. I probably used these two bytes because they were unique in my template .avb and made it easy to locate and modify some bytes od the ID. 

    But, I could have done this in a couple of other ways too. The problem is that I don't understand the underlining structure of the .avb binary. Without spending many hours hacking it, this was a brute force method that worked. It is dependent on the template .avb as you can see and that's not good.

    ...and yes, hacking XML project files is infinitely easier.

    Igor Ridanovic

    www.HDhead.com

  • Tue, Feb 14 2017 11:47 AM In reply to

    Re: .avb ID structure

    Hi Igor,

    Thanks for sharing...

    I had a bit of time to look at some binaries....

    Not much is changing in the header except the date and some data in the 8 byte (decimal) range 55-62.

    There is data in the centre & footer changing also - at 206, 624-32, 6016 and 6088.

    It's not in the header, after quite a bit of messing about.

     

     

  • Fri, Feb 17 2017 1:11 PM In reply to

    Re: .avb ID structure

    ...but it is towards the footer in all the bins I looked at (and I looked at a LOT) - I don't like doing it this way, but I'm looking for a 5 byte hex starting with an end of text char. The ID is very close to that and you can change up to 8 bytes. I can successfully find & change it in any type of bin now, even though the position changes constantly.

    All copied content is instances of media - Interplay seems to handle these bins fine, allthough I haven't tested everything.

    Chances for a clash with 8 bytes of ID is about 1 in 100 Trillion for a thousand generated bins - even less for smaller numbers of bins (2^64)

  • Mon, Dec 25 2017 10:15 PM In reply to

    • yale
    • Not Ranked
    • Joined on Thu, May 15 2008
    • West of Hollywood, but east of Santa Monica
    • Posts 192
    • Points 2,325

    Re: .avb ID structure

    Sorry to raise a thread that is almost a year old, but I just wanted to add some information to get it out there. .avb files are composed of a series of atoms that all follow a defined format. The bin ID is an 8-byte character string that is part of the "ABIN" atom. Actually, at least on .avb files created on a Mac, all atom names are stored backwards, so you'll see it as the "NIBA" atom (I believe they are stored forwards for bins created on a PC but I only ever work with Mac files so I'm not sure).

    Atoms consist of a four character string identifying the type, a four byte integer indicating the length of the payload, and then the payload itself. The bin ID is bytes 6-9 (counting from 0) of the payload. So if all you want to do is modify the bin ID, you can find the offset of the "NIBA" (or possibly "ABIN") string, add 14 (4 for the name, 4 for the length, 6 for the offset into the payload), and set the next 4 bytes to something random.

    Example from a random .avb file:

    4E494241 84000000 020E3600 0000B20E 0954

    N I B A  (length) . . . .  . . (Bin id )

    (sorry for the formatting but the forum doesn't seem to want to display a monospace font correctly)

    Set the bytes in bold to a random value and you should be fine. I wrote a script to do this to get files out of the attic to open at the same time as the original files, and have had no problems with doing so for years. The only weirdness is if you open a bin with the same sequence as a bin that is already open, Avid will effectively duplicate the sequence in the second bin and make a copy, renaming it in the process (i.e., by adding "Copy.01"). But I haven't noticed any problems with that.

    I've been working on a project off and on to decode more of the .avb format for some time. Happy to share any knowledge anyone has about this. The holy grail would be the ability to read and decode sequence and clip information directly from .avb files without needing to generate an EDL first. If there were some way to do that, it would enable some extremely powerful and time-saving workflows for tracking cuts, visual effects, and all sorts of other interesting things...

    Ability to extract clip and sequence information programmatically and real scripting/automation support would be my two biggest requests for Media Composer… if anyone is listening. Big Smile

  • Tue, Dec 26 2017 1:51 AM In reply to

    • jwrl
    • Top 25 Contributor
    • Joined on Thu, Oct 13 2005
    • Melbourne, Australia
    • Posts 8,393
    • Points 97,070

    Re: .avb ID structure

    For what it's worth, the atom is stored backwards cross platform.  I would have thought that it would have to be in any case to support reliable cross-platform project interchange.  From a random Windows bin file that I just checked here in the sunny land down under:

    4E494241 E9040000

    So it appears that the information in your post applies to any Avid bin on any supported platform.

    MC 7.0.4 - Asus P6T Deluxe V2 mobo - Intel i7 920 2.66GHz - Windows 7 Ult64 SP1 - nVidia Quadro FX 1800 - 16 Gbyte low latency DDR3 RAM - Internal 8 Tb... [view my complete system specs]
  • Thu, May 2 2019 12:48 AM In reply to

    • Zach Fine
    • Not Ranked
    • Joined on Thu, Mar 31 2011
    • Posts 37
    • Points 435

    Re: .avb ID structure

    I found my way to this thread because I happened upon the 'Kaitai Struct' page, about a specification for documenting and set of utilities for manipulating file structures, and was curious whether the .avb format had been released and/or figured out. For my part, I'd like to be able to put together a list of all source media clips referenced within a bin -- akin to whatever Automatic Duck Media Copy must do internally.

    Media Composer 2018.4.0 OS X 10.13.3 Late 2013 MacPro 3GHz 8-Core 128Gb RAM D700 GPU [view my complete system specs]
  • Sat, May 4 2019 1:24 PM In reply to

    • luca.mg
    • Top 25 Contributor
    • Joined on Thu, Oct 13 2005
    • Roma - Italy
    • Posts 5,611
    • Points 65,840

    Re: .avb ID structure

    By opening the bin with a text editor You should be able to read the bin's content.

    Symphony 2018.12.3, BM Intensity Pro 4k, Windows 10, i7-5930K, 32 GB ram, Quadro K620 [view my complete system specs]

    peace luca

Page 1 of 2 (16 items) 1 2 Next >

© Copyright 2011 Avid Technology, Inc.  Terms of Use |  Privacy Policy |  Site Map |  Find a Reseller