Notes from Megameeting 11th January 2010

Attendees

Summary

There was further discussion of the appropriate cycle for auditing and re-auditing of TDRs, and the relation to ISO 17021's requirements. The American National Accreditation Body and other accreditation bodies were introduced as possibly relevant.

Actions

• All to comment on David's document at http://wiki.digitalrepositoryauditandcertification.org/pub/Main/ReqtsForAuditors/AuditorGuidelines-CCSDS-format-1-without-boxes.doc
• HelenTibbo to contact CRL for their experience of initial audits using TRACbodies
• All to research ANAB (American National Accreditation Body)

Transcript of chat

BruceAmbacher >> (All): When we get a few more can we discuss the audit cycle 
Mark posted from the American Assoc of Museums?  It seems a more realistic cycle 
for most digital repositories.
Mark Conrad >> (All): Bruce, You are further than me. I am on page 20 of 49.
BruceAmbacher >> (All): I am doing a quick once-through then I will go back and 
review in detail.
Mark Conrad >> (All): Did anyone have a chance to look at the AAM document on 
their audit and certification process?
Mark Conrad >> (All): It is at: http://www.aam-us.org/museumresources/accred/upload/Steps%20in%20Process%20w-guides&charts.pdf
JohnGarrett >> (All): I looked at the summary  a bit.
Mark Conrad >> (All): Did anyone besides Bruce review the document without the 
boxes that David posted?
Helen Tibbo >> (All): No, I ws out straight with all my meetings
BruceAmbacher >> (All): I think the more extended time frame is more realistic 
given the size and staff of many digital repositories.  Adding a similar time 
chart would assist us in "selling" and in understanding out audit process.
JohnGarrett >> (All): I also looked tht over
Mark Conrad >> (All): So what do we want to do today?
BruceAmbacher >> (All): Does anyone know if we must stick to the 3 year cycle in 
the parent standard we are piggy-backing on? Or, can we extend the cycle to a 
longer time fram and still be within that standard?
JohnGarrett >> (All): We can't conform to the standard we are piggy-backing on 
if we change the cycle.
Helen Tibbo >> (All): What I like about the AAM (and what happens in LIS 
schools) is that the self study part shifts much/most of the work on the 
institution and off of the reviewers. This keeps auditing costs "relatively" 
low. 
Mark Conrad >> (All): Bruce, Our document says that our requirements are "in 
addition to" the requirements in ISO 17021. That doesn't leave much wiggle room.
JohnGarrett >> (All): However, if we want to change the cycle that would be OK, 
we could still mostly piggy back on it and just note the exception
Mark Conrad >> (All): John, I don't see how we could do that.
SimonLambert >> (All): I would think there has to be some flexibility to change 
the cycle - because the parent standard was written first and could not 
anticipate all possible standards that might follow from it.
Helen Tibbo >> (All): If the repositories collect most of the data then the 
auditors validate and check this implies a large role for self-audit before the 
auditors come and before any materials are sent to the audit team.
TerryLongstreth >> (All): We could submit a RID to the parent standard to loosen 
up the cycle
BruceAmbacher >> (All): John, isn't that what we are doing in other parts of the 
standard when we specify criteria for the auditors, and other places where we 
say 'in addition, the following are also required  . . ."
Helen Tibbo >> (All): This also implies educational efforts to conduct these 
self audits and how to access one's likelihood of certification.
JohnGarrett >> (All): Mark, rather than make a blanket statement at the start 
that we are fully compliant, we just say in each section that we inherit the 
specific things (except the 3 year cycle)
Mark Conrad >> (All): Terry, Can you submit a RID to an approved standard?
TerryLongstreth >> (All): If the committee is still active
BruceAmbacher >> (All): Couldn't a RID also be placed during the 5 year update 
cycle?  I don't know what the cycle is for this standard.
JohnGarrett >> (All): Yes, we could submit a RID which would be addressed in the 
next RID cycle.  I do not think they would agree to the change.
Mark Conrad >> (All): So how do we want to proceed?
TerryLongstreth >> (All): It really a matter of how well accepted the standard 
is question, is
BruceAmbacher >> (All): Do all/most/few/none agree that the 3 year cycle is too 
onerous?
JohnGarrett >> (All): The committee is still active.
Mark Conrad >> (All): Bruce, I do.
Helen Tibbo >> (All): Without some sort of professional mandate a 3-yr cycle 
seems unrealistic to me.
Mark Conrad >> (All): I still don't understand the value of piggy backing onto 
ISO 17021. We are auditing digital repositories - not management systems per se.
TerryLongstreth >> (All): Do we know if the 3-year cycle is actually being 
followed for incumbent 17021 systems?
Mark Conrad >> (All): Is there a certification body for 17021?
BruceAmbacher >> (All): Would ISO allow us to establish a standard for a
TDR certification process that does not conform to the existing standard?
JohnGarrett >> (All): Yes, the 3 year cycle is being used by at least some very 
visible systems including ISO 9000 and the ISO security management system 
standards
TerryLongstreth >> (All): I mean, target environments that are being audited
JohnGarrett >> (All): Yes, I'm sure we could establish a standard with a 
different time scale.  Actually you can do about anything in ISO if you can get 
past the review and comment cycle.  
BruceAmbacher >> (All): John, Is there a threshhold size lower limit for ISO 
9000?
JohnGarrett >> (All): I think we can certainly make a case for a longer review 
cycle for TDRs based on economics.
JohnGarrett >> (All): We may get comments from the 17021 folks.  We would need 
to respond to their comments, but we don't have to get their approval.
TerryLongstreth >> (All): I still say, the 'next review" determination should be 
at the discretion of the audit committee, and the standard should be a 
guideline, not  a normative timeline
BruceAmbacher >> (All): The amount of time required to self-assess, gather 
documents, and undergo audit will not vary all that much based on size of 
organization and the real impact will be greater on the smaller organizations.
JohnGarrett >> (All): Bruce, there is no stated lower size for ISO 9000.  But 
smaller size does result I think in lower costs down to some minimum level.  And 
again it only matters if you choose to have an external audit that you pay for 
and that is registered so you can claim ISO 9000 compliance, usually as a 
business advantage.
BruceAmbacher >> (All): Terry, the AAM document does just that by allowing the 
audit team to say how long until the next audit (between 5-10 years)
TerryLongstreth >> (All): Bruce- I agree.  The only problem I see with AAM is 
that it is oriented to physical objects. 
Helen Tibbo >> (All): ALA gives 3 types of accreditation - provisional that 
needs to be revisted in a year or 2; then a five year term, and the best places 
get 7 years between reviews.
Helen Tibbo >> (All): This is for LIS schools.
Helen Tibbo >> (All): I would think the nature of the content held and the file 
formats, their ages, migration path and likelihood would all play into the need 
for the next review cycle with TDRs
BruceAmbacher >> (All): Terry, True.  But OAIS, which the audit ultimately is 
based on,  also includes physical objects.  
BruceAmbacher >> (All): Helen, I like that reasoning as the basis for 
flexibility by the audit team in assigning when the next audit will occur.
BruceAmbacher >> (All): John,
TerryLongstreth >> (All): I wasn't excluding the physical, but we should try to 
account for the special needs of digital artifacts
Mark Conrad >> (All): So do we stick with 17021? Develop our own standard? 
Something else?
BruceAmbacher >> (All): John, how much weaker would the audit standard be if it 
rremained just a CCSDS atandard for the first 5 years or so?
JohnGarrett >> (All): Bruce, I don't know.  What do all of you think?  My guess 
is that ISO would be much better recognized than CCSDS by most people.
BruceAmbacher >> (All): A TDR that serves just its parent organization should 
have a much simplier task that a TDR that serves multiple producers with 
multiple formats and object types.
RobertDowns >> (All): It seems reasonable for the rigor of continuing review to 
reflect the complexity of the TDR.
TerryLongstreth >> (All): Re: Mark's verbal - do the 17021 accreditors have any 
discussions of the cycle?
JohnGarrett >> (All): I think the hardest thing is preparing for the first 
audit.  Following that if the audits are not too far in the future would be a 
much easier task as you will reuse and update most of the audit proofs.
Mark Conrad >> (All): Terry, Not that I have found so far.
TerryLongstreth >> (All): Mark - can you post a URL here?
BruceAmbacher >> (All): JOhn, That is true but a TDR reading the standard and 
seeing the 3 year cycle will see an ongoing, constant overhead cost and not 
understand how much can be reused in future audits,
Mark Conrad >> (All): http://www.orion4value.com/Pub/AboutORI/NewsBankDocs/17021/ANAB%20and%20RvA%20Accredit%20Orion%20Registrar%20to%20ISO%2017021.pdf
Mark Conrad >> (All): http://www.tuv.com/web/media_get.php?mediaid=23651&fileid=55660&sprachid=2
JohnGarrett >> (All): Yes, Bruce, I agree.  So we need to pick some period (or 
range of periods) and decide what will be acceptable for our community.
BruceAmbacher >> (All): This sense of varying audit cycles based on the 
complexity of the data, the volume, the number of different data streams, etc. 
could be written into 9.4 Recertification.
RobertDowns >> (All): Allowing for a self-reporting component as part of the 
process of continuing certification may enable TDRs to limit costs.
Mark Conrad >> (All): Here is a particularly interesting document: http://www.anab.org/HTMLFiles/docs/Aps/CL3000-17021matrix.pdf
TerryLongstreth >> (All): Checklists are good
Helen Tibbo >> (All): I agree that a one size fits all probably won't work here 
or at least will discourage a range of repositories.
Helen Tibbo >> (All): Checklists are very good. That's the part of TRAC that is 
most used.
RobertDowns >> (All): The continuing certification also could provide the TDRs 
with an opportunity to report on progress to address any concerns identified 
during the initial certification process.
BruceAmbacher >> (All): Does someone need to have a conversation with the 
American National Accreditation Body (ANAB)?
Helen Tibbo >> (All): I think that is a great idea.
BruceAmbacher >> (All): Robert, NARA used to do that with its inspections of 
agency records management activities - create a workplan focusing on the 
deficiencies and timelines to clear all deficiencies.
BruceAmbacher >> (All): Mark, Has your unit had any contacts with ANAB?
Mark Conrad >> (All): No. I never heard of them until a few minutes ago.
Mark Conrad >> (All): There is also the International Accreditation Forum, Inc.
BruceAmbacher >> (All): Wouldn't aour audit team(s) in the US fall within the 
ANAB sphere?
Mark Conrad >> (All): The more I look. The more confused I get. There is from 
ISO 17021, 17011, Guides 62 and 66.
BruceAmbacher >> (All): Simon, do you know if there is a parallel body in the UK 
or EU?
JohnGarrett >> (All): There is a list somewhere for ISO of accrediting bodies 
that certifies audit organizations (that comply to ISO 17021)
JohnGarrett >> (All): I think in general each country tends to have a body that 
does that.
JohnGarrett >> (All): Yes auditing in ISO is a big deal.  The committee 
responsible for 17021 has a large number of standards and guides (maybe 30 to 
40) that cover various aspects of auditing.
SimonLambert >> (All): I don'tknow of any international bodies - there are of 
course national bodies like BSI, DIN, ...
Mark Conrad >> (All): So if there are so many audit standards, why can't we have 
our own?
JohnGarrett >> (All): Back to the certifiers of auditors.   An example I see is 
Quality Austria and they have a list of organizations they have accredited as 
auditors available at: 
BruceAmbacher >> (All): Attaching out audit teams to those bodies could be a 
benefit but it also could be a real burden.  And what validity would our audits 
have if we don't align with such bodies?  And would our desire for a longer and 
variable audit cycle cause issues?  We really do need to get a handle on this
Mark Conrad >> (All): Bruce, How do we do that?
JohnGarrett >> (All): http://www.qualityaustria.com/index.php?id=798&L=1
BruceAmbacher >> (All): Someone needs to get in touch with one of the national 
bodies to begin a dialogue.
RobertDowns >> (All): Awarding different types of certification might be a way 
to have different types of continuing review, based on the type of certification 
received.
BruceAmbacher >> (All): Robert, I like that possibility.  We may be seeing a 
path forward.
BruceAmbacher >> (All): but would such bodies consider preservation systems just 
another type of management system?
TerryLongstreth >> (All): That's an important issue to me.  The Austrian web 
site has no mention of resource or asset management related systems (banks, 
libraries, museums...
Helen Tibbo >> (All): We really need some sort of funding to do some surveying, 
rigorous test audits, etc.
Mark Conrad >> (All): Everything I am reading on 17021 refers to quality 
management systems (ISO 9000) and environmental management systems.
Helen Tibbo >> (All): What has CRL done with testing TRAC?
TerryLongstreth >> (All): of course, we're just skimming through these urls
Mark Conrad >> (All): Terry, Agreed, but its like trying to untangle a bowl of 
spaghetti.
JohnGarrett >> (All): That's right.  The national bodies tend to register 
auditors (in general) that might the base auditor certification.   Then to audit 
any particular type of system, you would need to meet the extra stuff added for 
each type of auditing.
JohnGarrett >> (All): I think our interest is more in the particular 
requirements for auditing a TDR.
BruceAmbacher >> (All): For our initial audits should we include such a 
certified auditor on the team?
JohnGarrett >> (All): I think we are less interested in if the auditing 
organizations for us are able to meet ISO 17021 than if they are competent to 
audit a TDR.
JohnGarrett >> (All): I don't think we care if the auditor can meet 17021.
JohnGarrett >> (All): So I think we can make review periods longer than 17021 
and get auditing of archives started.
BruceAmbacher >> (All): John, but what would that say about the quality of the 
audit and the confidence the public should place in the result?
Mark Conrad >> (All): Bruce, 17021 requires a member of the audit team who is 
competetnt in auditing.
JohnGarrett >> (All): Any auditing we get started and accepted is an improvement 
on what we have now and will be a service to our community.
TerryLongstreth >> (All): Mark, would an Certified public accountant qualify?
Mark Conrad >> (All): John, Are you suggesting we abandon 17021?
BruceAmbacher >> (All): Is that 17021 competent or could that be competent in 
the TDR metrics only?
JohnGarrett >> (All): It appears that the museum audits are not ISO compliant, 
but they are still well accepted.
JohnGarrett >> (All): Who does the certifying is really an issue of what will be 
accepted in your community.
Helen Tibbo >> (All): Again, how has CRL done their initial audits using TRAC??? 
I believe they have audited JStor and the HathiTrust.
BruceAmbacher >> (All): John, agreed but we don't want to get a spitting contest 
with ISO over audit standards.  We would be downwind in that contest.
JohnGarrett >> (All): I would suggest that we inherit as much as possible from 
17021, but if there are individual things that make it unfeasible for the 
archives community, then we drop those particular requirements.
BruceAmbacher >> (All): Helen, what is the resukting product? a report, a 
certification, a recognized, valid whatever?
BruceAmbacher >> (All): For next week can we all post RIDs for the document 
David posted last week?  Can someone volunteer to research ANAB or a parallel 
national body?
JohnGarrett >> (All): If what I saw some time back (kept pretty close at 
requests of the organizations audited) was the initial audits from TRAC, they 
were reports
Helen Tibbo >> (All): I'll try to touch base with either Bernie or Marie at CRL 
and see what they have done.
JohnGarrett >> (All): There was  a range of detail in the reports that I saw and 
not much consistency from audit to audit.
Mark Conrad >> (All): Terry, In terms of the qualifications of the auditor see 
17021, clause 7.2.5.
BruceAmbacher >> (All): Were they considered "test" audits?  Or were any "real" 
audits suggesting corrective actions?
TerryLongstreth >> (All): Thanks Mark
Mark Conrad >> (All): Bruce, How do we post RIDS for David's document? It is a 
Word document that you download.
BruceAmbacher >> (All): Mark, I was going to send an email to the group and 
leave it to David to amend document after comments are accepted, modified, 
rejected.
Mark Conrad >> (All): Ok. I am using highlighting, Track Changes, and comments 
in the Word document. I will just e-mail the document to David when I am done.
Mark Conrad >> (All): Next Monday is a Federal Holiday and I will be on the 
road.
TerryLongstreth >> (All): To answer my own question, it looks like a CPA could 
be retrained.
JohnGarrett >> (All): I believe what I saw were test audits I think to test how 
well the TRAC worked.  Bruce were you involved in any of those or did TRAC group 
use the results of those audits to refine their work?
TerryLongstreth >> (All): 17021 7.2.5 says  the team includes "auditors and 
audit team leaders possessing generic auditing skills"
BruceAmbacher >> (All): I did not participate in any test audits.  Robin Dale 
did some under a grant to CRL.  David did some and others
Mark Conrad >> (All): So what are the action items?
BruceAmbacher >> (All): 1. ALL - comment on document 2. Helen - cotact CRL
3. Research ANAB 
Mark Conrad >> (All): Ok. See you in two weeks. Bye.
JohnGarrett >> (All): OK Bye.  Thanks everybody.
SimonLambert >> (All): Bye all

-- SimonLambert - 11 Jan 2010