Notes from Megameeting 2nd November 2009

Attendees

Summary

DavidGiaretta had put a proposed definition and role of the lead audit committee on the wiki. This was discussed, along with requirements for training and issues in the general audit procedure.

It was agreed to avoid referring to a "TDR" when it is still under audit, and instead to use terms like "the organisation's repository" or "the repository being audited".

Transcript of chat

Mark Conrad >> (All): Has anyone had a chance to look at David's suggested changes at: 
http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/AuditorGuidelinesIssues
BruceAmbacher >> (All): I looked at the first part but did not compare to 
previous language
Mark Conrad >> (All): Unfortunately I had two meetings this morning and have not 
had a chance to review David's changes, yet.
SimonLambert >> (All): David has proposed some wording for the lead audit 
committee and its role.
RobertDowns >> (All): It looks like at least some of the new changes are 
underlined.
BruceAmbacher >> (All): I see some underlining, strikethroughs and 
question/hints for expansion triangles.
Mark Conrad >> (All): So how should we proceed?
SimonLambert >> (All): All the underlines and strikethroughs should be relative 
to the original version; the questions/hints are points from our discussions 
that will need attention.
SimonLambert >> (All): I guess we could look at the proposed definition and use 
of the lead audit cttee and see if that's acceptable.
RobertDowns >> (All): It refers to the authors of the RAC document. Will the RAC 
document contain a list of the authors? 
BruceAmbacher >> (All): Simon, in prefatory  notes there is a refecence to 
"surveillance - implications on frequency " Is there a special meaning to 
surveillance or is this just referring to how often the audit should take place? 
 I do not remember any discussion of ongoing interaction with TDR between audits
David Giaretta >> (All): Sorry I'm late
BruceAmbacher >> (All): David, the last two postings are for you to answer
Mark Conrad >> (All): Simon, Are you referring to the definintion listed under 
"Points to check"?
SimonLambert >> (All): Mark - yes.
SimonLambert >> (All): "definition of "initial audit committee": * The initial 
audit committee will consist of internationally recognised experts in digital 
preservation, the membership building on members of the authors of the <ISO 
XXXXX - RAC Document "
Mark Conrad >> (All): Ok, thanks.
TerryLongstreth >> (All): The section looks much better, but I'm still 
ambivalent about the training requirements (or requirements for training).  
7.2.1.1 needs to be clarified, perhaps with an introductory paragraph like 
"Audit teams shall be trained to operate together, and individuals shall all be 
trained in specific skills."
David Giaretta >> (All): Robert - the authors of the RAC doc - I don't think we 
need to name them - not sure how other ISO cttees work
David Giaretta >> (All): Bruce - surveillance - is used throughout the various 
ISO docs so I guess I would opt for follolwing what everyone else does
BruceAmbacher >> (All): David - what does it mean in this context?
David Giaretta >> (All): Terry - not sure how to train people to operate 
together - although I see what you mean. IN particular I wonder how this applies 
to the initial audit committee
BruceAmbacher >> (All): To me it implies and ongoing interaction between the 
audit team and the TDR
David Giaretta >> (All): ...but clearly we (assuming it is us) need to see eye 
to eye about things - which suggests the need for some test audits together 
(virtually)
David Giaretta >> (All): Bruce - I'd need to check back on what is implied.
Mark Conrad >> (All): David, After our last web meeting, were you able to find 
out any more about how other initial audit committees were set up. I know you 
said that you were not going to be able to meet with that one person you hoped 
to meet with, but did you run into anyone else who could shed some light on the 
topic?
David Giaretta >> (All): Mark I have not had a chance to yet - I'll be trying to 
make an appointment to see or talk to someone later this week
David Giaretta >> (All): ....in particular I need to talk to the British 
Standards Institute people
TerryLongstreth >> (All): David - I think 7.2.1.1 currently contains an 
indiscriminate mix of team training and individual training requirements.  Team 
Training is a recognized discipline (military's been doing it for millenia). I'm 
not sure that we want to mandate it, but separating the concerns in 7.2.1.1 is 
still needed..
Mark Conrad >> (All): David, OK. I was just curious as to whether or not you had 
run into someone else at the meeting in the Netherlands who might have some 
insight on this topic.
Mark Conrad >> (All): I agree with Terry. The training requirements should be 
explicit about who should have what training and for what purposes.
David Giaretta >> (All): Mark - no - no-one last week
TerryLongstreth >> (All): The sentence that set me off reads: "The certification 
body shall have criteria for the training of audit teams that ensures"
David Giaretta >> (All): Mar/Terry - but where do we specify it - the way I read 
it the doc is very unspecific - I tried to add something that allows the initial 
audit cttee to be more proscriptive
David Giaretta >> (All): ...so we need to have procedures but we don't have to 
say what they are in this doc
TerryLongstreth >> (All): Easiest to avoid team training by amending the 
sentence: "....training of audit team MEMBERS to ensure..."
David Giaretta >> (All): Ahhh - I see what you mean
BruceAmbacher >> (All): Terry, are you looking for more "objective" criteria 
than those set up in a, b, c?
TerryLongstreth >> (All): Perhaps a checklist, in an appendix
BruceAmbacher >> (All): Sorry, I was in 7.2.1.1.1
TerryLongstreth >> (All): I don't want to overburden 7.2  but we should provide 
better guidance somewhere
David Giaretta >> (All): Terry - if we add an appendix then that would take a 
long time - we jusat need procedures in a separate (not part of the standard) 
document 
Mark Conrad >> (All): What is the relationship between 7.1.1.1. and 7.2.1.1.1?
BruceAmbacher >> (All): But the requirements/guidance for a science TDR could be 
very different from that for a social science/humanities TDR based on the nature 
of the collections and the complexity of the preservation system
SimonLambert >> (All): 7.1.1.1 is about the organisation, 7.2.1.1.1 about the 
individuals - I think.
David Giaretta >> (All): That looks right
BruceAmbacher >> (All): Organization being the certification body and 
individuals being the specific team sent to a specific TDR?
David Giaretta >> (All): Yes
TerryLongstreth >> (All): I do get confused between audit team and TDR staff 
requirements.  If the TDR staff has a media engineer (responsible for managing 
to avoid deleterious effects of media degradation), does the audit team need one 
too?
David Giaretta >> (All): Terry - I don't think there is a 1-to-1 match.
TerryLongstreth >> (All): not in individuals, but in skills, traiining
BruceAmbacher >> (All): The team needs someone who knoes enough to not have the 
wool pulled over his/her eyes but not necessarily enough to actually run that 
process
David Giaretta >> (All): ....this doc just specifies things at a high level and 
also says the audit team can bring in technical experts
Mark Conrad >> (All): Terry, When you say TDR staff, do you mean staff at the 
repository or staff of the certifying organization?
TerryLongstreth >> (All): So the checklist has an entry "team includes expertise 
in x, y, z, technologies
BruceAmbacher >> (All): as appropriate to the TDR being audited
David Giaretta >> (All): Terry - the wiki draft says "understanding of risk 
assessment and risk management of digitally encoded information"
RobertDowns >> (All): It might be impossible to have an audit team that has 
expertise in every single aspect of a repository's activities.
TerryLongstreth >> (All): access to expertise, then
David Giaretta >> (All): ...and "technical knowledge of the digital preservation 
aspects which apply to the  activity to be audited"
Mark Conrad >> (All): In many places throughout this document it refers to the 
client organization's repository as a TDR before it is even audited. Do we 
really want to say that it is a TDR before it has been certified?
TerryLongstreth >> (All): We could say "candidate TDR"
David Giaretta >> (All): .......so the Body offering audit and certification 
would choose a team and arrange expert advice that covers what is necessary 
about preservation. Some knowledge of media i.e. physical storage - in general 
terms - would be needed in every case
RobertDowns >> (All): Mark - I do not think that we want to call a repository a 
TDR until it has been certified.
David Giaretta >> (All): Good point about "candidate TDR"!!
TerryLongstreth >> (All): or .....CTDR (>))
BruceAmbacher >> (All): Yes, but can a repository call itself a TDR if it has 
conducted a self audit and can demonstrate that it meets all critreria?
David Giaretta >> (All): Not wrt this docuemnt
BruceAmbacher >> (All): I agree but good luck enforcing that
TerryLongstreth >> (All): From the perspective of the auditors, it's always a 
candidate, even if it's been approved before
David Giaretta >> (All): ...sorry should have said - we should not call it such 
in this doc - use "candidate TDR" instead
Mark Conrad >> (All): Bruce, If they can, what is the point to having a 
certification process?
TerryLongstreth >> (All): If it's been approved before, does it lose 
certification if it fails this time?
BruceAmbacher >> (All): Agreed but we know how "poor" and "cheap" many 
repositories are.  OThers may be very comfortable stopping after self assessment
David Giaretta >> (All): maybe we just say "organisation's repository" or "the 
repository being audited"
Mark Conrad >> (All): David, That is probably best.
RobertDowns >> (All): I agree with your last suggestion, David
Mark Conrad >> (All): Bruce, The cheap repositories should not be able to say 
ISO XXX-certified.
David Giaretta >> (All): Bruce - I guess the point is that anyone can call 
oneself anything - it's proving it that is difficult. Not sure who would sue.
David Giaretta >> (All): That's why we need to talk to out national standards 
bodies
BruceAmbacher >> (All): I know but they could say they "comply with ISOxxx" 
without actually being certified.
TerryLongstreth >> (All): I agree with Mark.  And if an organization claims to 
be a TDR without being audited, they are probably liable for damages from their 
customers' perspective
Mark Conrad >> (All): Bruce, So we push the brand for ISO XXX-certified 
repositories.
BruceAmbacher >> (All): But what if the repository's "Board" is satisfied with a 
self assessment?  I don't see us suing a national archives that answers to its 
congress/parliament or a large data center that gets a green light from its 
board.
Mark Conrad >> (All): Terry, Does that mean the certifying authority would be 
liable for damages for certified TDRs?
Mark Conrad >> (All): Bruce, We just have to prominently publish the list of 
repositories that have actually been certified - sell the brand.
BruceAmbacher >> (All): Are these issues to be raised and addressed in this 
auditor's guide or are these issues for the audit board to work out?
BruceAmbacher >> (All): Mark, perhaps ISO has some guidanec on this, or the 
group that developed the auditor's ISO.
Mark Conrad >> (All): Let's hope so.
David Giaretta >> (All): Mark - that's why I suggested that we "need for funding 
to obtain legal advice " - BUT I don't think that ISO 27000 people get sues if a 
respoitory turns out to be in-secure
David Giaretta >> (All): (my typing in terrible!)
RobertDowns >> (All): Perhaps the wording on the certificate can address the 
issue of potential liability.
TerryLongstreth >> (All): Only if it can be shown that they lied, just like a 
financial audit
RobertDowns >> (All): Each audit will need to document its findings as evidence 
for its decisions.
Mark Conrad >> (All): Bruce, With the exception of what we call a repository 
that has not been audited/certified in this document, the rest of the issues we 
are raising now are probably best left for the Board.
TerryLongstreth >> (All): ... and to show a good faith effort
BruceAmbacher >> (All): The first few audits will have to be done carefully, and 
carefully worded
David Giaretta >> (All): Returning to the doc - the wording I added was to make 
it clear that the initial audit cttee has some control over the training and 
qualifications of any auditor
BruceAmbacher >> (All): So should we be making assignments to address the "need 
to be done" areas?
David Giaretta >> (All): Bruce - need to be done for this document and also 
finding out some backgroyund info
David Giaretta >> (All): perhaps people here could comemnt on the wording I 
suggested - I hoped it would also cut through some of the other discussions we 
have had e.g. about the length of the training.
Mark Conrad >> (All): I think we need to know about how other groups have kicked 
off the audit and certification process before we make changes to this document.
RobertDowns >> (All): Looking at how it has worked elsewhere would be 
informative.
David Giaretta >> (All): Mark - certainly an action on us all to speak to people 
about that
Mark Conrad >> (All): The only process I am vaguely familiar with is how the 
Academy of Certified Archivists was formed. I do not think there are too many 
lessons there we can draw from.
TerryLongstreth >> (All): Certainly, the longest history would be in Finance.  
Anyone know a good bank auditor?
BruceAmbacher >> (All): Who are the people "in the know" and who knows them to 
approach them?  Or is this a question to ask ISO?
David Giaretta >> (All): Mark - perhsps the ANSI people or 27000. As I said, I 
plan to speak to BSI
BruceAmbacher >> (All): Mark, What about NIST?
Mark Conrad >> (All): I am not aware of NIST auditing or certifiying anything.
David Giaretta >> (All): Also I assume that the wording of our "baseline" doc 
i.e. the one we took a lot of the text from - is basically OK to support a 
certification process and we are just making changes at the edges.
BruceAmbacher >> (All): David, when did/does the TDR audit & certification begin 
draft review?
David Giaretta >> (All): Bruce - the CCSDS review has started and I hope the ISO 
review will start very shortly - still awaiting word on the date
BruceAmbacher >> (All): and is it different time frame for CCSDS and for ISO?
David Giaretta >> (All): ...CCSDS review started a few weeks ago
David Giaretta >> (All): ...... CCSDS comemnts due by 12/16/2009
BruceAmbacher >> (All): coordinated end dates?  coordinated review of comments 
by a single team?
RobertDowns >> (All): Will the ISO review begin when the CCSDS comments have 
been received?
RobertDowns >> (All): Or will revisions need to be completed first?
David Giaretta >> (All): Bruce - yes ISO will take a bit longer - the two can be 
in parallel. What we need is to be able to resolve CCSDS and ISO comemnts 
simultaneously
David Giaretta >> (All): Robert - just to be clear - the CCSDS and ISO reviews 
can be in parallel - they are NOT sequential
David Giaretta >> (All): .....we need to be able to resolve CCSDS and ISO 
comemnts simultaneously
David Giaretta >> (All): ........otherwise the CCSDS and ISO versions will be 
different
David Giaretta >> (All): ....which would mean we would not be able to give free 
access to the ISO doc via CCSDS - which is what happens now for OAIS
BruceAmbacher >> (All): David, why does that necessarily follow?  Can't the 
CCSDS red book benefit from the iSO comments and make the same changes and still 
post it as a final version?
David Giaretta >> (All): But it would have to go through CCSDS review again
BruceAmbacher >> (All): We all know how "expensive" ISO standards are.  Many 
repositories will not even play in the arena if it costs them to even get the 
rules.
David Giaretta >> (All): ....so best to resolve issues simultaneously and end up 
with the same doc for CCSDS as well as ISO
David Giaretta >> (All): Bruce - exactly - that is one reason OAIS was 
successful - poeple could get it for free!
BruceAmbacher >> (All): I was not aware the end product would not be available 
in the same fashion as OAIS
BruceAmbacher >> (All): Maybe funding x number of copies should be part of a 
grant proposal for an experts meeting, promotion of standard and 
training/funding first audit teams
David Giaretta >> (All): Bruce - it WILL be available free if we keep CCSDS and 
ISO in step
BruceAmbacher >> (All): David, ok I misunderstood what you were writing.
Mark Conrad >> (All): If you want to see some impressive marketing/branding of 
auditing/certification services do a Google search on SAS 70 audits.
David Giaretta >> (All): ....it should be OK - it's just that we need to be 
aware of some potential pitfalls
David Giaretta >> (All): Mark - yes there are lots of possibilities....
David Giaretta >> (All): ....but we need to finish the doc
David Giaretta >> (All): What actions?
BruceAmbacher >> (All): Are there any assignments that can move this guidealong?
TerryLongstreth >> (All): List of "here there be tygers" items to be assigned?
David Giaretta >> (All): Actions: (1) review additions to the Issues page
David Giaretta >> (All): ACtion (2) contact standards audit people if possible 
to ask about audit setup
David Giaretta >> (All): action (3) draft 2 pager to put to funders etc as 
suggested in http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/CcsdsMeeting20091028
David Giaretta >> (All): Not sure about "tygers"
BruceAmbacher >> (All): What are tygers?
TerryLongstreth >> (All): David - Where are the Issues?
David Giaretta >> (All): .,.. they should be quiet as long as we finish the doc 
fairly soon and make sure the ISO review starts ASAP
David Giaretta >> (All): Bruce - I assume tygers are things to beware of
David Giaretta >> (All): Terry - the issues are the non-technical things - in 
CCSDS and ISO - that need to be taken care of to get these tweo docs through to 
standards
David Giaretta >> (All): ......and set up the accreditation and certification 
process
TerryLongstreth >> (All): tygers - places in the document where someone has 
indicated a need for change or expansion.
David Giaretta >> (All): Ok I was wrong
BruceAmbacher >> (All): David, looking at the minutes/actions from the CCSDS 
meeting points for the two page document - who are the funding agencies and why 
is there a belief they run or have responsibility for repositories?
David Giaretta >> (All): ...but I was hoping a lot of those tygers would be 
taken care of by introducing the "initial audit cttee"
David Giaretta >> (All): Bruce - they do in Europe
TerryLongstreth >> (All): ...from an old joke about mapmakers. If they couldn't 
supply data for an area they wrote "HTBT" (or dragons, or monsters..._
David Giaretta >> (All): ....I believe they fund repositories in the USA also
David Giaretta >> (All): .......i.e. any repository must have a funder
BruceAmbacher >> (All): I think of funding bodies as NSF, IMLS, NEH, etc. not as 
the parent for a repository such as a business, organization, university
TerryLongstreth >> (All): funder --  First item in the checklist!
David Giaretta >> (All): Bruce - the money may go through an intermediary but 
NSF etc do fund repositories I think - if not then the organisation which host 
them could be considered as their funder
David Giaretta >> (All): ....but if one could talk to NSF then there are 
potentially lots of repsoitories
David Giaretta >> (All): .......anyway - just a suggestion
Mark Conrad >> (All): NSF issues policies on how an organization should set up a 
repository for data derived from NSF supported research.
BruceAmbacher >> (All): That can be the hook - get such funders to "strongly 
urge" an audit for such repositories.  But what if it is a new venture just 
establishing a repository?
Mark Conrad >> (All): See the NSF call for Datanet proposals.
David Giaretta >> (All): Bruce - probably could not be audited - but could seek 
advice
BruceAmbacher >> (All): David, I have to leave.  Will anyone be sending out 
assignments or actions for next week?
David Giaretta >> (All): ....i.e.they could seek our advice
David Giaretta >> (All): I suggested a list of actions above - Simon and I can 
formalise them in the minutes.
Mark Conrad >> (All): Ok. See you all on the Internet next week.
TerryLongstreth >> (All): I take it the current baseline document is the one 
John Garrett sent out 2? weeks ago, as amended by the Issues web page
David Giaretta >> (All): Yes 
TerryLongstreth >> (All): Good.  I'll see you next week

-- SimonLambert - 02 Nov 2009