Latest post Thu, May 17 2018 6:59 AM by Jeroen van Eekeres. 6 replies.
Page 1 of 1 (7 items)
Sort Posts: Previous Next
  • Tue, May 1 2018 2:03 PM

    Quarantining media on ISIS/Nexis. Right or wrong?

    Dear Avid, ACSR's and Avid shared storage users.

    I'm going to describe a scenario I have just encountered that makes me wonder if I'm a massive idiot or if all these years 'we' missed a huge flaw in Avid's media indexer functionality.

    First the simplified facts:

    - ISIS 7000 suffers from an electonic component problem that can cause the ISB microcontrollers to fail booting if power is removed from an ISIS 7000 chassis. Avid will replace these controllers free of charge regardless of support contract status as this is considered a manufacturing fault. Kudos Avid! (I do however wonder if this policy to replace this these boards should be altered from 'when failed' to 'known to fail' but I can't estimate the costs)

    - As probably everybody knows, the integrated media indexer on all media composer systems, standalone, using shared storage and in an interplay environment, continuously monitors the file count of local storage AvidMediaFiles folder and scans the storage if a mismatch is detected (or pmr & mdb files are not present) and if during the scan they detect a corrupt mxf/omf file they will move that file into the quarantine folder. This scanning process is a visible process.

    - The integrated media indexer on Media composer systems with shared storage will do the same for the Avid MediaFiles\MXF\machinename.number folders in the ISIS/Nexis workspaces if no interplay components are installed. This is also a visible process. 

    - In an Interplay environent the central media indexer takes care of all workspace indexing. It does not initiate an automatic re-index or re-scan when changes occur on workspaces. However it has a feature called the Media Indexer Full Resync Time which by default is set at 01:00 at night.  This process is only visible on the media indexer web page.

    Media Indexer relies on ISIS/Nexis file system notifications in detection of new files being created on the Avid shared storage. These notifications comes in three flavours, CREATED, UPDATED, DELETED. When a file is being created or updated, Media Indexer starts an index process for the file, providing the file is in a non-excluded folder. Media Indexer by default will ignore files in folders with names "temp", "creating" and "quarantined files", files created in these folders will be detected, but ignored.

    The scenario:

    An unlucky customer encounters an issue with his UPS and power fails at around 00:00 at night. PANIC!!! UPS techs, electricians and broadcast techs are called and those still on site are running around. At 00:15 slowly slowly power is restored to all equipment. Many will recognize this situation. PSU's not powered down for years failing, smoking capacitors, clocks needing setting, drives failing and raid sets (attempting) rebuilding and... ISIS 7x00 ISB microcontrollers failing. Disaster...

    Before 01:00 all Avid servers are powered up and the system directors present the file system to all clients including the media indexers. But due to the multiple ISB failure most media is missing file parts. But no panic, the Avid storage support team is here to help 24/7 to get those failed ISB up and running a.s.a.p. However...

    At 01:00 the media indexers perform the automated re-index and quarantine all media it considers corrupt. So it moves the files in their corrupt state. When after 24-48h the new ISB microcontroller boards arrive all ISB's are brought back up, 7 ISB's however remain in the unknown state. All of these 7 ISB's report one or both drives missing. 6 of these ISB's funtion normally after reseat, the 1 ISB does not play ball. The power cut has killed on of its drives but no panic hot Spare ISB(s) are available and the system was 'only' 90% full. Of course the ISB's perform their rectifying files when reinserted and a repairing mirrors because of the failed ISB's. To replace the failed ISB the spare is added and a redistribution starts. In the morning the redistribution finishes but 90% of the media is in quarantine and indeed trying several mxf's show they are corrupt.

    Could this be prevented? Yes of course but is this procedure documented? Was it the quarantine action that killed the system directors ability to recover from this catastrophy, was it the repairing mirrors or the redistribution? All projects and other files outside of the Avid MediaFiles folders are 100% ok.

    But this made me think (yes I know that's a bad idea). Why does the media indexer need to move corrupt media which is the last thing you want on any storage facing problems. For the local media indexer... let's skip that chapter for now.

     In an interplay environment, why can't the media indexer just flag the mxf's as corrupt without touching them? Why not allow a manual retry after a user was able to restore his ISIS to 100%?

    The automatic re-index is only documented in the Best practices documentation. An ability to disable it is not available only to change the time it happens.

    As an ISIS and Interplay ACSR I believe I should have known the above and how to avoid it or I should have known how to deduct the prevention of such a catastrophy by putting the functional knowledge of the individual parts together. But even if I did, if the power restore to the system directors and media indexers happens 5-10 minutes before the automatic re-index is initiated... is that a case of bad luck?

     

     

    Most of the MC/NC, Interplay, Nexis, ISIS, Unity stuff. [view my complete system specs]

    Avid reseller ACSR at Telmaco

    http://www.telmaco.gr/en/

  • Tue, May 1 2018 5:41 PM In reply to

    • Marianna
    • Top 25 Contributor
    • Joined on Thu, Oct 13 2005
    • Avid
    • Posts 9,170
    • Points 205,055
    • Avid Beta Moderators
      Avid Customer Advocate
      Avid Developer Moderator
      BlogAuthor
      SystemAdministrator

    Re: Quarantining media on ISIS/Nexis. Right or wrong?

    Hey my friend....

    This is more a question about Media Composer and Media Indexer functionality scanning NEXIS (than NEXIS itself) IMO.

    From what Ed has told me, quarantined files are not permanently excluded, they can be restored – but again this is an Interplay / MediaCentral Q.

    Also the premise at the beginning of your post about replacing all controllers even if customers are not on support, I don’t believe that to be correct, customers need to be on support to get the controller card update.

    Also, Roger of the interplay team responded as well....  that he would agree that the Media Indexer should suspend re-indexing operations if the underlying storage system was not in a fully-operational state. This is something that should be considered for implementation.  What he didnt know was if API calls are available for the MI to detect that the storage system is in good or bad state, that would be required for this to work.

    I'll keep you posted as more chime in to the email exchange.

    Marianna

     

     

    Director of Online Communities and Forums/Customer Advocate [view my complete system specs]

    marianna.montague@avid.com

    mobile 813-493-6800

    AOL IM:  avidmarianna

    Twitter:  avidmarianna

    Skype: mariannamontague

    WWLD

  • Tue, May 1 2018 7:03 PM In reply to

    Re: Quarantining media on ISIS/Nexis. Right or wrong?

    Marianna:
    he would agree that the Media Indexer should suspend re-indexing operations if the underlying storage system was not in a fully-operational state. This is something that should be considered for implementation.

    Hi Marianna,

    Thanks for following up. Avid has provided the boards for 3 sites where I encountered this of which 2 were out of support.

    Happy to hear Roger agrees re-indexing suspention needs to be looked at in these conditions for Nexis and ISIS.

    Maybe including the guys/girls that write the ACSR training docs should also be pinged to include this scenario in the ACSR training program.

    Just saying.

    Most of the MC/NC, Interplay, Nexis, ISIS, Unity stuff. [view my complete system specs]

    Avid reseller ACSR at Telmaco

    http://www.telmaco.gr/en/

  • Thu, May 3 2018 6:40 PM In reply to

    Re: Quarantining media on ISIS/Nexis. Right or wrong?

    And guess what?

    Avid had already realized the above. Support in Madrid just mailed me that since Media indexer 3.8.x the media indexer does no longer move the files to the quarantine folders.

    It is described in the latest best practices document.

    Quarantined Files MI v3.8 and later defaults to a new approach for handling corrupted files or files that the MI is unable to work with. MI no longer moves files into quarantine folders. Now the MI internally marks the files as to-be-ignored. Ignored files will be reconsidered for processing when the file modification time changes. This new approach maintains separate to-be-ignored lists for each indexing approach; one for classic and one for format independence. Now a file that is not supported by one indexing approach may still be supported by the other approach.

    I do not remember seeing this in a readme or the emea tech seminars... 

    Most of the MC/NC, Interplay, Nexis, ISIS, Unity stuff. [view my complete system specs]

    Avid reseller ACSR at Telmaco

    http://www.telmaco.gr/en/

  • Fri, May 4 2018 5:29 AM In reply to

    • Chris Aust
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Munich
    • Posts 435
    • Points 4,750
    • Avid Certified Instructors - Video
      Avid Certified User
      Moderator: Avid Certified Instructors - Video

    Re: Quarantining media on ISIS/Nexis. Right or wrong?

    Kalimera Jeroen,

    This is documented in the "Whats New in Interplay 3.8.pdf"

    We have had this discussion some time ago. This seems to be a matter of pros and cons.... There are some reports where media was quarantined but it actually shouldn't.

    There are some files that could temporarily be broken, for instance while they are written/copied or there is a connectivity issue to the storage. If those are moved to quarantined folder they will not be visible to the system. Now since they stay they will automatically recover and made available to the system. Apparently this behaviour currently has higher priority.

    However, this isn't what customers are used to since years. Think about Dark Matter...

    I am awaiting some more information about this.

    Thanks

    Chris

     

    MC/Symphony 8.7.2 | VMware Workstation 14 virtual Interplay on Ubuntu Linux 16.04 4.15.03| Dell XPS 15 |||| obsolete MacBookPro 17", 2.5GHz Quad Core... [view my complete system specs]

    Chris Aust

    Munich, Germany

    Avid Workflow Consultant | ACI | Trainer

    Splash!Zone

     

  • Fri, May 4 2018 1:16 PM In reply to

    • Chris Aust
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Munich
    • Posts 435
    • Points 4,750
    • Avid Certified Instructors - Video
      Avid Certified User
      Moderator: Avid Certified Instructors - Video

    Re: Quarantining media on ISIS/Nexis. Right or wrong?

    Jeroen,

    the new behaviour can be turned off in the jini.config file:

    com.avid.ummd.quarantine.QuarantineManager

    {

       static isIgnoreFailedItemsApproachEnabled = Boolean.FALSE;

    }

     

    Chris

    MC/Symphony 8.7.2 | VMware Workstation 14 virtual Interplay on Ubuntu Linux 16.04 4.15.03| Dell XPS 15 |||| obsolete MacBookPro 17", 2.5GHz Quad Core... [view my complete system specs]

    Chris Aust

    Munich, Germany

    Avid Workflow Consultant | ACI | Trainer

    Splash!Zone

     

  • Thu, May 17 2018 6:59 AM In reply to

    Re: Quarantining media on ISIS/Nexis. Right or wrong?

    Hi Chris,

    Sorry for the delay but I'm overloaded with setting up a new interplay, mediacentral orad, inews, one archive... the full set of integration stuff (and accomodating problems).

    While the issue has technically been resolved in interplay 3.8, what worries me is that the storage support team does not warn the customer/reseller in case of an isis 7000 multiple ISB failure in interplay pre 3.8 systems that it is crucial for them to stop (or shutdown) the media indexers to prevent an unrecoverable quarantine of almost all media. Due to the nature of the controller board failure after a power outage I would expect this has occured on other sites as well.

    An unhappy customer...

    We'll be upgrading this customers media indexers to 3.8 or 2017.2 to prevent another quarantine disaster but I assume the new microcontrollers will have updated components that do not fail within 3 years. 

    Most of the MC/NC, Interplay, Nexis, ISIS, Unity stuff. [view my complete system specs]

    Avid reseller ACSR at Telmaco

    http://www.telmaco.gr/en/

Page 1 of 1 (7 items)

© Copyright 2011 Avid Technology, Inc.  Terms of Use |  Privacy Policy |  Site Map |  Find a Reseller