Notes from Megameeting 5th October 2009

Attendees

Summary

Discussion began on aspects of the "Requirements for Bodies Providing Audit and Certification", based on the extracts that particularly seem to need attention. There was some concern that too much familiarity with the particular repository is required, and that the process appears adversarial. It was agreed:

• to add some words in an introductory sentence explaining that the auditor is like a "critical friend"
• to clarify the educational level expected of the audit team.

To look at next time:

• Section 7.2.1.1 d, g amd h
• Section 9.2

Transcript of chat

BruceAmbacher >> (All): Are we starting with 7.2 in Auditors Handbook?  If so I 
am concerned about the stress on technical knowledge and on the auditor knowing 
the specific IT systems of the TDR being audited.  How specific is that intended 
to be?
SimonLambert >> (All): There is a version with some revisions (by David?) at 
http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/AuditorGuidelinesIssues
SimonLambert >> (All): Is that what we are looking at?
JohnGarrett >> (All): Yes, I think that would be a good place to start unless 
there is some other suggestion.
SimonLambert >> (All): I extracted those areas of the requirements that seemed 
like they might need revision in the TDR context
SimonLambert >> (All): and that is what is on that page.
BruceAmbacher >> (All): Can any external auditor who has not worked in the TDR 
being audited have the level of competence suggested in 7.1?
SimonLambert >> (All): Bruce - what is in that section that causes that concern?
JohnGarrett >> (All): I think most of the requirements seem pretty general and 
would be required.  I can see the need to look at d, f and h
BruceAmbacher >> (All): In 7.1.1.1 are we examining the specific clauses of a 
contract or understanding the general purpose and the main issues being 
addressed?
BruceAmbacher >> (All): My general concern is that no one (person or team) 
outside the TDR being examined can have the full competence
JohnGarrett >> (All): So we're looking at 7.1.1.1.  I think it is looking at 
ensuring the competence of the auditors to do an audit.   
David Giaretta >> (All): Hi soory I'm late
JohnGarrett >> (All): Of course the TDR should be competent, but how much should 
auditing organization know before they are allowed to audit?
BruceAmbacher >> (All): I can't say any specific clause in 7.1 is incorrect.  
The emphasis makes the audit more adverserial than I would hope it would be.  The
team is relying on the TDR to have conducetd a self audit and ammassed the 
information and documents required for the audit.  I see more of a partnership 
between TDR and audit team.
David Giaretta >> (All): Most of the wording is fairly standing in the auditing 
world - so we probably should not change that
JohnGarrett >> (All): Yes, I don't want it to be adverserial either.   It should 
be a team.  I don't think we should require a self audit first.  Although 
everyone will probably do so.
David Giaretta >> (All): However we could add some wording which makes the 
partnershiop idea clearer
JohnGarrett >> (All): I like the idea of the partnership 
David Giaretta >> (All): I think the "normal" wording in the parent standards 
from which this inherits makes it clear the aim is to help improve whatever is 
being audited
BruceAmbacher >> (All): The TDR will amass the evidence needed for the audit and 
will have the standard to know what will be audited si they could/should do a 
self-audit and discover areas of weakness.
David Giaretta >> (All): On the other hand the auditors should not just accept 
what he/she is told
SimonLambert >> (All): Is there an issue what is meant by "understand the areas 
of activity of the client organization and the associated business risks"?
David Giaretta >> (All): Perhaps the auditor is a "critical friend"
SimonLambert >> (All): How deep does that understanding have to be?
JohnGarrett >> (All): Yes, I agree that they should do a self audit and they 
probably will.  Are they 'required' to share the results of the self audit.  I 
don't think so.  But by doing a self audit, they will make the documentation 
easier to find.
David Giaretta >> (All): Simon - that is a critical question
BruceAmbacher >> (All): Simon has raised one of my concerns - how knowledgeable 
about specific systems must the auditors be?
Robert Downs >> (All): Completing a self-audit should help the TDR to prepare 
for the audit.
Robert Downs >> (All): Some systems could be created in-house, which would make 
it difficult for an auditor to gain such expertise.
David Giaretta >> (All): Simon - my guiding principle is that all types of audit 
organisations face the same problem so we should be able to solve it in the same 
way they do
JohnGarrett >> (All): I don't think the auditors need to be knowledgeable about 
the specific system in use at a specific archives, but they should have 
competence in knowing about archives systems in general and possibly about the 
specific domain where the particular archives operates.
David Giaretta >> (All): ...in other words the body doing the audit does not 
have to be an expert in the repository's business but has to have a general 
understanding of the area (interpretaed very broadly)
BruceAmbacher >> (All): We are a small community worldwide.  It is inevitable 
that some member(s) of the audit team will have "impressions"of how successful 
the TDR is in fulfilling its mission and can test that impression through the 
audit process.
David Giaretta >> (All): Bruce - a large community if one includes everyone with 
a repository
BruceAmbacher >> (All): Now how/where does this approach outlined by John and 
David get incorporated into the handbook?
David Giaretta >> (All): My impression is that we can leave the general wording
David Giaretta >> (All): ..but we can add some extra words that the auditor is a 
critical friend 
BruceAmbacher >> (All): Can we insert more in the training auditors materials?
David Giaretta >> (All): Bruce - we can insert whatever we want but it has to be 
sufficiently general to give us wiggle room
BruceAmbacher >> (All): Agreed.
JohnGarrett >> (All): Any suggestions for added (or subtracted) wording?
BruceAmbacher >> (All): No, As I said earlier it is not any specific words just 
the overall tone.
BruceAmbacher >> (All): In 7.2.1.3.1 d and g need some clarification for me, 
especially d.  Does any other audit process spell out the continuing education 
requirements?
David Giaretta >> (All): Perhaps we can add some introductory material saying 
that the audit should be a partnership which aims at helping the repository to 
improve
Robert Downs >> (All): That seems like a good suggestion, David.
BruceAmbacher >> (All): Are we assuming that the vast majority of TDRs seeking 
certification will pass?  That they will use the requirements to conduct the 
self-assessment and delay audit until they can pass?
JohnGarrett >> (All): I like David's suggestion.
David Giaretta >> (All): I guess my assumption is that none will pass everything 
perfectly and all will need some improvement
David Giaretta >> (All): ...but that they will almost certainly have done a 
self-audit to make sure they have covered the basics
JohnGarrett >> (All): Bruce, yes, I think we hope that most will pass.  Probably 
with several suggestions for improvement in some areas.
David Giaretta >> (All): We need to figure out what the certification would 
actually say - i.e. what do other audit certificates say?
Robert Downs >> (All): David - are you referring to a pass-fail versus some 
graduated certification?
BruceAmbacher >> (All): What if the suggestions exceed the resources available 
to the TDR?  Can the audit tean provide some provisional certification?
David Giaretta >> (All): For example we could say "Does security and bit 
preservation OK but needs to improve its understandability stuff"
Robert Downs >> (All): Provisional certification might be a way to address 
deficiencies and offer an opportunity and guidance for improvement.
BruceAmbacher >> (All): Ah, but you are not in their designated community it may 
seem like mush but to the DC it is fully transparent.
David Giaretta >> (All): I assume the purpose is that users will be able to see 
the certificate and say "OK - these guys do bit preservation and they are 
working on their understandability thigs so I can talk to them about whether 
what they do fits my needs"
David Giaretta >> (All): Bruce - the repsoitory needs to produce evidence of 
what they do about understandility and how they test it - neither we nor they 
need to understand the details of the material
Robert Downs >> (All): So, bit preservation would be one level of certification?
David Giaretta >> (All): Robert - that is certainly one possibility - that is an 
area we need to work on but I don't know if it needs to be in this docuemnt 
BruceAmbacher >> (All): Shouldn't a TDR also be competent in long-term access 
and usability, as well as bit level preservation?
JohnGarrett >> (All): I think we are currently at a point where we are going to 
have a pass/fail.  I don't think we have a metrics document that will support 
having different levels of trust for the metrics.  The metrics are each more or 
less pass/fail. 
David Giaretta >> (All): Bruce the question is always "how competent"
BruceAmbacher >> (All): Can someone work through 7.2.1.3.1 for me.?  I see 
internal contradictions in the requirement - to be an auditor your must first 
participate in 4 audits, am I reading that correctly?
David Giaretta >> (All): John - I don't think pass fail is what other audits do 
in general
BruceAmbacher >> (All): I was referring to 7.2.1.3.1.d
David Giaretta >> (All): Bruce - that's why I say that we need a primary 
committee - one that starts the whole process
David Giaretta >> (All): ...so that committee does not have the experience of 4 
audits but the members are internationally accepted as knowing what they are 
doing
JohnGarrett >> (All): Yes, that is what it says.  I think we need to decide how 
much experience is needed before you can be an auditor.   I think we need to 
relax that at the start.  We can update and make more restrictive later if we 
think it is worthwhile to do so.
BruceAmbacher >> (All): David, understood.  But d says in the future the only 
acceptable way to become an auditor is to do 4 audits (I think)
Robert Downs >> (All): The term, "secondary level" might have different meanings 
within different cultures.
Robert Downs >> (All): In 7.2.1.3.1 a
David Giaretta >> (All): Bruce - I guess we inherit that from other standards - 
we could change it - how many would you say?
BruceAmbacher >> (All): David, what is the understanding of secondary education 
in Europe?  here it is through high school but not college 
David Giaretta >> (All): In Europe I would say high school but not 
university/college
David Giaretta >> (All): ..which seems a bit low!
Robert Downs >> (All): I agree
David Giaretta >> (All): We could specity "masters" or " university degree"
BruceAmbacher >> (All): But possibly not for an IT type who worked up through 
the ranks.
David Giaretta >> (All): ***I have to leave in a couple of minutes to get to my 
conference
Robert Downs >> (All): Specifying a masters degree would be more specific than 
just specifying university degree.
David Giaretta >> (All): Bruce - I would say that we do need to allow someone 
who has worked up the ranks - as long as that it not the case with everyone in 
the audit team
David Giaretta >> (All): My recollection is that the audit team member 
competence is rather low and general but the audit team has to have an 
appropriate combination of skills
Robert Downs >> (All): We could state that the official auditors have the 
qualifications. 
JohnGarrett >> (All): I would probably say a university degree in some related 
field or some university work and additional experience would be acceptable.
BruceAmbacher >> (All): Setting a master's may be too high, especially a subject 
masters.  When some of us old fogies went through none of this was taught in 
organized programs.
David Giaretta >> (All): ...at least that would seem to make sense
David Giaretta >> (All): We could say "In the team at least one persone should 
have a masters" for example
BruceAmbacher >> (All): I do find the language vague enough to allow the audit 
team to select on the basis of competence.
JohnGarrett >> (All): Yes emphasis on the team having the experience needed.  No 
need for each individual auditor on the team to have the whole experience.
David Giaretta >> (All): Yes
BruceAmbacher >> (All): I think I am seeing a consensus that the current 
language in 7.2 can remain as it is.  Am I correct?
David Giaretta >> (All): Looks like it
David Giaretta >> (All): I guess lots of groups like ours in other areas have 
been through the same discussions!
David Giaretta >> (All): Must go - bye
BruceAmbacher >> (All): Should we address 9.2 or hold that for next week?
JohnGarrett >> (All): I think much of 7.2 is fine, but may want to drop or 
modify a couple of them d, g, h for example
JohnGarrett >> (All): I think we said that we would have hour sessions, so maybe 
9.2 should wait till next week.
BruceAmbacher >> (All): We also should address John's issue of modifying d, g, h 
next week.

-- SimonLambert - 05 Oct 2009