Latest post Wed, Dec 5 2012 10:46 PM by cable. 10 replies.
Page 1 of 1 (11 items)
Sort Posts: Previous Next
  • Wed, Nov 28 2012 11:31 PM

    • cable
    • Top 500 Contributor
    • Joined on Sun, May 27 2007
    • Pacific NW
    • Posts 395
    • Points 5,070

    hitting max folder and file count

    Does anyone know what happens, if anything, when I exceed the Media Networks limit of 500K files and 10K folder?  I am just about there.  Just curious if I should be worried about this.

     

    Thanks.

    Cablet

    20 x Media Composer 8.6.5 Mac Pros 32MB RAM OSX 10.11.6 AJA and Nitris DX hardware Avid Nexis Enterprise [view my complete system specs]
  • Thu, Nov 29 2012 12:17 AM In reply to

    • Chris Hocking
    • Not Ranked
    • Joined on Thu, Mar 13 2008
    • Melbourne, Australia
    • Posts 179
    • Points 2,070

    Re: hitting max folder and file count

    On our old LANSHARE's as soon as we hit the file/folder limit the system just grinded to a halt. It still worked - but it was unusably slow, and constantly threw up errors in the Media Composer suites.

    Basically, I think once you hit the limit, all the RAM is used up on the Unity - and you start running into problems.

    Avoid it at all costs is my advice...

  • Thu, Nov 29 2012 1:00 AM In reply to

    • Brendon
    • Not Ranked
    • Joined on Mon, May 21 2007
    • New Zealand
    • Posts 32
    • Points 340

    Re: hitting max folder and file count

    Hi there I agree with the last post, system performance is significantly impacted when you start go over those limits .... especially the folder count. We had 2 LanShares and 1 Unity and at various times went over that limit. It was never a good day when that happend.

    1 Site with Interplay and ISIS with 80 Assist Workstations, 25 Newscutters, 25 Mediacomposers, Avid Capture Manager and Countdown using Airspeed classics... [view my complete system specs]
  • Thu, Nov 29 2012 5:22 AM In reply to

    Re: hitting max folder and file count

    If you actually hit it, you'll get errors that the disk is full.  Are you hitting both the 500,000 and 10,000?  Just prior to v2.0, we were hitting that limit, and I talked to an Avid engineer and requested a trade-off between files & folders.  Soon after that, Avid introduced "Dynamic Allocation".  Every time you cross a 10,000 folder limit, you lose 32,000 files, e.g., 468,000 left, and get another 10,000 folders.  But if you're really hitting both, you have no choice but to clean off the system ... or, of course, upgrade to ISIS.

    Symphony 8.8.x w/Avid Nitris DX, HP z420, Windows 10, QT v7.7.x, Hexa Core 3.2GHz, 16GB RAM, nVidia Quadro Series 2000 w/353.82. 2TB Boot, 2 x 3TB internal... [view my complete system specs]

    "There are few technological barriers.  You can fix almost anything if you throw enough money at it."
    *******************************
    Randall L. Rike, ACI, ACSR Mac*Win*Unity*ISIS*DS
    Systems Engineer @ BET Networks [a Viacom company]
    (wwld)

  • Thu, Nov 29 2012 5:48 AM In reply to

    • cable
    • Top 500 Contributor
    • Joined on Sun, May 27 2007
    • Pacific NW
    • Posts 395
    • Points 5,070

    Re: hitting max folder and file count

    Hi,

    We are at 9000 folders and 489K files and our project is just really getting started. Just this last week I upgraded all of our systems to 6.0.3.2 from 5.5.3 and now we are having very sluggish performance. But to rule out the Unity being an issue I am currently moving a project (with 200K+files) to a local drive (it's already archived to tape and offsite in case I lose that fw drive) for the only person that will need it. I'll see if that helps.

    December 2013 we upgrade to Isis.

    Thanks for the tips.

     

    -Cablet

    20 x Media Composer 8.6.5 Mac Pros 32MB RAM OSX 10.11.6 AJA and Nitris DX hardware Avid Nexis Enterprise [view my complete system specs]
  • Thu, Nov 29 2012 6:01 AM In reply to

    Re: hitting max folder and file count

    ISIS rocks!

    Symphony 8.8.x w/Avid Nitris DX, HP z420, Windows 10, QT v7.7.x, Hexa Core 3.2GHz, 16GB RAM, nVidia Quadro Series 2000 w/353.82. 2TB Boot, 2 x 3TB internal... [view my complete system specs]

    "There are few technological barriers.  You can fix almost anything if you throw enough money at it."
    *******************************
    Randall L. Rike, ACI, ACSR Mac*Win*Unity*ISIS*DS
    Systems Engineer @ BET Networks [a Viacom company]
    (wwld)

  • Sun, Dec 2 2012 11:43 AM In reply to

    Re: hitting max folder and file count

    Randall L Rike:
    ISIS rocks!

    Yes it does, but 4G fiber unity is still ok if you ask me.

     

    From the old Apple Quadro 950 to HP Z8xx. My current own system: 1x Z420 E5 1650 16GB memory, 128GB SSD, 1x XW8600, 2x 3.0Ghz Quadcore, 12GB memory... [view my complete system specs]

    Jeroen van Eekeres 

    Technical director, Broadcast support engineer, Avid ACSR.

     

    Always have a backup of your projects....Always!!!! Yes Always!!!!

    A.V.I.D....... Another Version In Development

    www.mediaoffline.com

     

     

     

  • Sun, Dec 2 2012 2:01 PM In reply to

    Re: hitting max folder and file count

    You will receive constant errors as bins will fail to save, media databases cannot be updated and captures are aborted. It will be unusable.

     

    Buy yourself some time buy backing up your Attic and deleting it entirely. You may also be needed to do the same process with other non Avid files in your Unity (text, QT clips, user settings, etc., etc.).

     

    MC 8.6.1, Windows 10, ASROCK X99M, Intel 5930K, Zotac GTX970, 32GB RAM, Transcend 512GB M.2 SSD & Dell U2713H. MBP 2012 Thunderbolt, OSX 10.10.5, MC... [view my complete system specs]
  • Tue, Dec 4 2012 3:31 PM In reply to

    • Steve Kistner
    • Not Ranked
    • Joined on Thu, Jul 19 2012
    • New York City
    • Posts 33
    • Points 385

    Re: hitting max folder and file count

    In the mitigation category - look at the root directory of all of your workspaces - you should have:

    1) an Avid MediaFiles folder for MXF media

    2) an OMFI MediaFiles folder if anyone is using OMF

    3) a projects folder (possibly on one workspace, hopefully not everywhere) & Unity Attic... 

    You're up against the wall with your folder/file count, and as others have indicated the results are not pretty when writes to shared storage are denied due to inability for Unity to track additional files/folders.  If you have ~any~ other folders present at the root of your workspaces, you should look at migrating that content to commodity storage. 

    An older edit system may not be stout enough for production, but it's got Gig-E & multiple sata ports.  Give one of those boxes an IP address & system name known to the group, & setup shared folders that you can prove are mapped to the clients as a first step.

    In my experience, operations is a lot more accomodating of a solution planned & implemented & proven functional in advance than approaching them asking "please can I move this content".  Your Medianetwork is expensive storage, & hardware capped to a certain drive size, while 2 TB drives are around $100 & a handful can buy you considerable breathing room. Add a second system if warranted, or a raid card to add capacity - odds are good you've got a box or two lying around somewhere or can source cheaply, & this resource can persist past your ISIS upgrade as a location for commodity files to live.  The other benefit is that Unity drive replacements will go faster when you're managing a smaller number of large files rather than a large number of smaller files. 

    You'll likely find a graphics or sound effects library that can be put elsewhere.  Other pain points tend to be software installers & Interplay (if in use) database backups pushing to Unity (Early 1x Interplay was particularly bad about this - later versions compressed the property store to a series of compressed files, rather than duplicating thousands of individual files).  Likewise, the "Quarantine files" folder may be hiding a bunch of files that you're not aware of. 

    Bottom line, if it's not media or projects, it's time to find a different home.  Randall, I didn't recall you being involved in the dynamic folder change (way back in the day, what was that - 3.2x Unity?) doesn't suprise me though - good to see you on the forums.

    Hope this helps,

    - Steve Kistner

    -- Steve

  • Tue, Dec 4 2012 4:36 PM In reply to

    Re: hitting max folder and file count

    Steve .... It was around 2000 or 2001.  I was at The Post Group/Fox Sports Net in West Los Angeles.  We did 500 spots per week.  And, we were using a Workspace to store backups.  We had the system up for just a few weeks when we hit the limit.  It might have been v2.0-ish?  I opened and closed the case with Frontline Support.  A few days later, Francis called to inquire about our situation.  I explained we had nearly 10,000 directories, but only 120,000 files, out of 250,000 max.  Almost as a joke, I offered to trade in unused file-count if I could get more directories.  A few weeks later, Francis called back to let me know it would be implemented in a future release.  As a side note, .csv reports were my idea as well.  I made a mock-up in Excel and gave it to Becki Phelps when she and a team from Avid came out to visit us at TPG.

    Symphony 8.8.x w/Avid Nitris DX, HP z420, Windows 10, QT v7.7.x, Hexa Core 3.2GHz, 16GB RAM, nVidia Quadro Series 2000 w/353.82. 2TB Boot, 2 x 3TB internal... [view my complete system specs]

    "There are few technological barriers.  You can fix almost anything if you throw enough money at it."
    *******************************
    Randall L. Rike, ACI, ACSR Mac*Win*Unity*ISIS*DS
    Systems Engineer @ BET Networks [a Viacom company]
    (wwld)

  • Wed, Dec 5 2012 10:46 PM In reply to

    • cable
    • Top 500 Contributor
    • Joined on Sun, May 27 2007
    • Pacific NW
    • Posts 395
    • Points 5,070

    Re: hitting max folder and file count

    Great idea on the Attic. I'll do that though I believe this Friday I get to wipe 200K+ files.

     

    Thanks.

    20 x Media Composer 8.6.5 Mac Pros 32MB RAM OSX 10.11.6 AJA and Nitris DX hardware Avid Nexis Enterprise [view my complete system specs]
Page 1 of 1 (11 items)

© Copyright 2011 Avid Technology, Inc.  Terms of Use |  Privacy Policy |  Site Map |  Find a Reseller