Notes from Megameeting 4th May 2009


BruceAmbacher UM
DavidGiaretta STFC
HelenTibbo UNC
JohnGarrett GSFC
KatiaThomaz INPE
RobertDowns CIESIN, Columbia University
MarieWaltz CRL
MarkConrad NARA


There was discussion of thw Word document put on the WIki - some sections were still to be reviewed - B1, B4, C1, C2. The questionnaire will be completed based on the metrics - but will include the Supporting Text The planning for the test audits may have some problems e.g. confidentiality. It was stressed that the test audits would by default be confidential; in the extreme case only the auditors need know the details - as long as they could feed back the information about the suitability of the metrics.

Actions (see transcript for details):

  • ALL : find errors in the metrics or report that there are none
  • ALL : organise the test audits ASAP to make sure there are enough
  • ALL : point out any other errors in the text e.g. references
  • DG : complete the questionnaire
  • DG : put up WIki page about the test audits

Transcript of chat

Mark Conrad >> (All): test
BruceAmbacher >> (All): Good morning Mark
Mark Conrad >> (All): Good morning Bruce
Mark Conrad >> (All): Good morning Robert
RobertDowns >> (All): Good morning Mark and Bruce
BruceAmbacher >> (All): Robert, did you get the idea we are supposed to meet today?
Mark Conrad >> (All): We were just discussing whether or not anyone will show up and whether or not there is anything to discuss.
RobertDowns >> (All): I have it as an ongoing meeting in my calendar and have not cancelled it for today.
Helen Tibbo >> (All): Good morning everyone! As it turns out, Monday mornings for the summer are not such a good time for me.
Marie Waltz >> (All): hi all
Helen Tibbo >> (All): Actually, anyother time in the week would be good after 1 on Mondays.
Mark Conrad >> (All): According to David's message of yesterday, he has not yet received input from everyone who was supposed to send it in.
JohnGarrett >> (All): Good morning, all.
Mark Conrad >> (All): Good morning.
Mark Conrad >> (All): Does anyone happen to know if today is a bank holiday in the UK?
Mark Conrad >> (All): I am pretty sure that it is in Ireland.
JohnGarrett >> (All): Monday mornings are a good time for me at least for now.
Marie Waltz >> (All): I can do anytime, as long as its consistent
JohnGarrett >> (All): I believe David said it was a bank holiday, but he indicated that he still planned to attend.
Mark Conrad >> (All): It is a Bank Holiday. I just checked.
Mark Conrad >> (All): That may be why no one from the other side of the pond has joined us.
KatiaThomaz >> (All): hi all
Marie Waltz >> (All): Hi Katia
BruceAmbacher >> (All): Marie, Please send me your telephone number or email.  I want to discuss something with you
David Giaretta >> (All): Hi sorry I'm late
Marie Waltz >> (All): 773-95504545 ext267
Marie Waltz >> (All): that 0 is a -
David Giaretta >> (All): It s a bank holiday here in the UK
David Giaretta >> (All): I wondered if people had had a chance to look at the document I put on the WIki
BruceAmbacher >> (All): David, You nited you did not get all of the sections.  What is still not updated?
David Giaretta >> (All): I got info from Bob, Bruce, Marie and...
David Giaretta >> (All): ....I'll have to check
BruceAmbacher >> (All): I have a style question on - there is a reference in the text to a DRAMBORA section is that appropriate for this document?  DRAMBORA is not an ISO standard.  Just wondering.
David Giaretta >> (All): Not sure - I think we can reference anything that is commonly available
David Giaretta >> (All): I also got updates from Barbara and SImon
Mark Conrad >> (All): David, I read your e-mail this morning and noted that you indicated the document was still incomplete. Given your advice from the last meeting to wait until you had a complete draft to provide comments, I did not look at the document.
BruceAmbacher >> (All): Does DRAMBORA now have a "fixed" text and underlying governmental authority?  I would not like to reference a document that is still in development and possible change.
JohnGarrett >> (All): Yes, I think we can reference anything we want, even if it is not an ISO standard.
David Giaretta >> (All): I hope that the incompletenes is only in terms of some references and maybe some small corrections in the supporting text.
RobertDowns >> (All): It looks like DRAMBORA also is referenced in 5.2.1
David Giaretta >> (All): Mark - so I'm sorry I mislead people. I think that unless people see errors in the metrics it should be good to go...
David Giaretta >> (All): ...with the proviso that we will expect some small updates from the test audits
David Giaretta >> (All): So perhaps people can respond to the list if they see errors/typos...
BruceAmbacher >> (All): Who has responsibility for adjudicating and incorporating the comments that have not been incorporated?
David Giaretta >> (All): Bruce - did I miss things?
David Giaretta >> (All): This group has to agree on things surely
BruceAmbacher >> (All): There are track change comments that are not adjudicated.  See 3.5.2 for example.
BruceAmbacher >> (All): 4.1.6 still awaits discussion.
David Giaretta >> (All): Good point, but I guess I was hoping we could close the discussion of the metrics themselves and then put in the small corrections to the informative text during the test audits
BruceAmbacher >> (All): I'm just raising the issue.  Do we have a schedule and/or candidates for the test audits?
David Giaretta >> (All): There is another small issue - we are about to release the OAIS update for public examination - today I hope. There may be some terminology there that may be of use that may lead us to update the informative text
David Giaretta >> (All): Bruce - I'm always an optimist so I was hoping we would be able to do these in the next month i.e. by mid-June
BruceAmbacher >> (All): Is that a job for David and John since they were involved with revising OAIS?
David Giaretta >> (All): Bruce ... I think John and I and Don have been looking at things from that point of view but I think it's for this group to come to a collective decision after looking at the proposals
David Giaretta >> (All): In terms of the test audits we had some candates from the Face to face meeting and also the questionnaire to help capture comemnts from the test audits
David Giaretta >> (All): ...I received some comemnts on the questionnaire from Barbara
BruceAmbacher >> (All): I am assuming the changes to OAIS are not major and it might be easier for those involved to show the rest how the changes affect what we have developed.
David Giaretta >> (All): If we agree on the metrics I can complete the questionnaire 
David Giaretta >> (All): Bruce - yes, I think it should be fairly easy - it's all backwardly compatibe - just a few new concepts
David Giaretta >> (All): ...and lots of small corrections and clarifications.
David Giaretta >> (All): So unless people think it's a waste of time I plan to complete the questionnaire - putting in all the metrics.
BruceAmbacher >> (All): I say go for it!
Mark Conrad >> (All): Does that include the sub- and the sub-sub- metrics?
David Giaretta >> (All): ...and set up a wiki page starting with the list of test archives from the face to face meeting
Marie Waltz >> (All): Bruce there is some information in the meeting notes on who is doing which audits,
David Giaretta >> (All): Mark - I was propsing to group things i.e. include the sub and sub-sub metrics in with their main metric
David Giaretta >> (All): ..otherwise it's too long
David Giaretta >> (All): ...I'll put in some navigation also
David Giaretta >> (All): one can jump to the metric of interest i.e. no need to do all the metrics at one sitting
Mark Conrad >> (All): I am just trying to understand what it is you are calling the metrics and what you are calling the informative text.
David Giaretta >> (All): Mark - metrics are the sub-headings and the things under "Supporting Text/Examples../Discussion" are what I mean by infomrative text
David Giaretta >> (All): ...metrics also the sub-sub-headings and sub-sub-sub headings
BruceAmbacher >> (All): That is my definition also.
David Giaretta >> (All): The metrics are where we needed "shall" - but that was not necessary elsewhere
BruceAmbacher >> (All): On the test audits I assume the insider will contact the outsider to establish protocols and times.
Mark Conrad >> (All): Isn't the supporting text supposed to be mandatory as well?
David Giaretta >> (All): Bruce - yes
David Giaretta >> (All): Mark - no
Mark Conrad >> (All): When was that change made?
David Giaretta >> (All): Mark - at least that is my understanding since the wording is in terms of examples etc
David Giaretta >> (All): Mark - I thought that was always the way - even with TRAC (although none of that was mandatory)
David Giaretta >> (All): I'm just using the term "informative" because that is the distinction which ISO and CCSDS make
Mark Conrad >> (All): Supporting text reads the "The repository must..." This is necessary because..."
David Giaretta >> (All): In 
David Giaretta >> (All): Mark - true - most of the time. We could make those pieces of text mandatory
BruceAmbacher >> (All): Mark, the metrics appendix  in the first  version of TRAC was just the metrics and that led to the type of inconsistencies you are hinting at.  But to include all supporting text would be virtually half the document and that would defeat the purpose of a "checklist"  So what is a practical approach?
David Giaretta >> (All): just pulling them away from the "This is necessary ..."
Mark Conrad >> (All): In all of our past discussions Supporting text was mandatory Evidence and Discussion were not mandatory. When did this change?
David Giaretta >> (All): But not all the supporting text is so clear
David Giaretta >> (All): I can go with "The repository must.." as mandatory but the text "This is necessary..." etc is clearly explanatory
BruceAmbacher >> (All): I would say it has not changed but what is reasonable?  Could we place a note before the metrics saying each one is supported by mandatory supporting text and no auditor should examine a repository without reading and understanding all of the supporting text.
JohnGarrett >> (All): How do you determine what is mandatory and what is not if only some of the supporting text is mandatory?
David Giaretta >> (All): Bruce - I think as long as it is clear - and stated in our Conventions section, we could make things mandatory - that's why I said we could put that "The repositroy must.." away from the rest of the supporting test
David Giaretta >> (All): ..text
David Giaretta >> (All): But in that case what is the difference from the metric?
Mark Conrad >> (All): John, Everything under the heading "Supporting Text" is supposed to be mandatory. We have been operating under that convention for months.
David Giaretta >> (All): The terminology about the difference between normative and infomrative text comes from CCSDS and ISO - the whole document is the standard
KatiaThomaz >> (All): the right use of "shall, should, may, and is" will do the job
JohnGarrett >> (All): But we just said that "This is necessary because ..." is not mandatory.
Mark Conrad >> (All): John, No we did not say that.
BruceAmbacher >> (All): Mark, just to be clear - you do not support an annex with just the netrics statements.  You would only support an annex that had metrics and supporting text.
Mark Conrad >> (All): I thought we were talking about the questionaire to be used by the auditors.
David Giaretta >> (All): Oooopppss!!
Mark Conrad >> (All): What kind of annex are you talking about, Bruce.
David Giaretta >> (All): Ok - I can paste in as much text as people want in the questionnaire
David Giaretta >> (All): Obviously we were talking at cross purposes!!
David Giaretta >> (All): Or do understand correct;y - MArk you were asking about the questionanire I was putting together - putting metrics and sub-,metrics together... and I can put the supporting text in there also
BruceAmbacher >> (All): Mark,  see TRAC beginning at p. 50
BruceAmbacher >> (All): That is an annex of just the bold metrics.
BruceAmbacher >> (All): That is what I was envisioning as an annex to the ISO standard.
David Giaretta >> (All): ... and Bruce is talking about an annex in the document taht makes it easier for people doing a self-audit to write in
David Giaretta >> (All): ...although we can also supply spreadsheet etc
BruceAmbacher >> (All): I was conflating David's questionnaire and such an annex in my mind.
David Giaretta >> (All): Panic over!
BruceAmbacher >> (All): But the issue remains of whether we want to create such an annex.
Mark Conrad >> (All): Buit the annex is something different from the questionaire?
David Giaretta >> (All): Yes
David Giaretta >> (All): Bruce is talking about an annex to our document which would go to ISO
David Giaretta >> (All): I'm not sure I've seen that with other ISo standards
Mark Conrad >> (All): The questionaire is a separate document for the auditor? We wouldn't want to send this to ISO?
BruceAmbacher >> (All): and the questionnaire has to lasting legality in relation to the draft standard.  It is a working tool for the test audits to help inform us and to help us find any major defects.
David Giaretta >> (All): ..and we can supply spreadsheets and other things
Mark Conrad >> (All): Ok.
David Giaretta >> (All): The questionnaire is a tool for the test audit only
Mark Conrad >> (All): Ok. Thanks for that clarification.
David Giaretta >> (All): We can make a separate one for real audits if necessary
BruceAmbacher >> (All): The auditors also will have the draft document, right?
Helen Tibbo >> (All): When do we envision having everything done so as to be able to do the test audits?
David Giaretta >> (All): SO the ACTION remains on (1) finding errors in the metrics (2) organising the test audits and (3) fixing any other erros in the text e.g. references etc
David Giaretta >> (All): Helen - I was hoping the document we have is good enough to do test audits
Mark Conrad >> (All): David, Do you have a list of the metrics that you have not yet received the updates for?
David Giaretta >> (All): Mark - yesy the people doing the  test audits would work from the current doc - assuming people agree with the edits I made
Helen Tibbo >> (All): OK, but what about the questionnaire, etc. we will use for the audits? Do we consider that done as well?
David Giaretta >> (All): Mark - I received responses from Bob, Bruce, Marie, SImon and Barbara
David Giaretta >> (All): I'd like confirmation taht I put those in right
David Giaretta >> (All): Helen - I'll complete the questionanire by the end of the week
BruceAmbacher >> (All): Were there any sections that did not need editing?
David Giaretta >> (All): Bob said he was OK with what was there
Mark Conrad >> (All): Who are you still awaiting responses from? More importantly what metrics have you not received updates for?
RobertDowns >> (All): I believe that B5 did not require additional editing.
David Giaretta >> (All): Let's see I don't think I received responses about B1, B4, C1, C2
Helen Tibbo >> (All): There were no indications that anything had to be done with what was section c.2 so I did not make any changes. 
David Giaretta >> (All): Helen - OK that's fine
JohnGarrett >> (All): I think B4 is OK, but I'll take another look
David Giaretta >> (All): I guess it would be good to have a positive repsponse about each section rather than just ask form errors to be reported
David Giaretta >> (All): I'm a little worried about exceeding my remit sometimes - I can get acrried away editing!
David Giaretta >> (All): You should be able to see the most recent edits because I have changed my reviewer name
David Giaretta >> (All): something like DG20090501
David Giaretta >> (All): you can just look at those changes - or do a comparison between different Word versions on the Wiki
BruceAmbacher >> (All): I reread B2 quickly and had no comments.  Iwill spend more time with it today. 
David Giaretta >> (All): Great
Marie Waltz >> (All): I'll read through A and email the group if it looks like anything is wonky.
BruceAmbacher >> (All): Can we set an assignment for all to review the 
David Giaretta >> (All): So do I understand the ACTION are for people to feedback positive and negatives about their sections - any any others
BruceAmbacher >> (All): still need to be discussed" sections for next week?
David Giaretta >> (All): also ACTION to start organising test audits
David Giaretta >> (All): Bruce - yes we will need to discuss those next week
BruceAmbacher >> (All): Can someone take responsibility to find all or those and list them for us?
David Giaretta >> (All): ACTIOn on me to complete the questionnaire and to put up a WIki page so people can record plans for the test audits
David Giaretta >> (All): I can ask SImon to pull things out from the comemnts if that would help
BruceAmbacher >> (All): John, When can you be ready for the test audit?
Marie Waltz >> (All): Is there a need to keep information about who is being audited and related information private?
David Giaretta >> (All): I included something in the questionnaire about confidentiality
David Giaretta >> (All): I assume that we (doing the test audits and actively involved) are all allowed to see everything
David Giaretta >> (All): I'll circulate to us (not the whole maining list) the URL to see the questionnaire results - password protected
Marie Waltz >> (All): Yes, the information they give is confidential but I'm wondering if the information as to who is being audited and by whom can be made public
David Giaretta >> (All): Marie - good point - we can leave that off the WIki
David Giaretta >> (All): We just need to know the number of test audits and who is doing them - although we don't even have to put everything on the WIki
JohnGarrett >> (All): As I said at the face-to-face, I'm not sure if I can get approval.  Almost certainly wouldn't be before our Senior Review in June now.
BruceAmbacher >> (All): Keeping the names of the audit team also should be kept off the wiki as that will give a clue to the site.  Does that mean we need to modify the Face-to-face minutes?
Helen Tibbo >> (All): Since we have not done Institutional Review Board approvals for the test audits I am assuming there is not plan to publish any of this data, even anonymously. We in the US associated with universities cannot do so.
BruceAmbacher >> (All): John, that is way too late to be useful for this phase so it should be dropped.
JohnGarrett >> (All): But I'll bring up the subject again, because there is also interest in auditing and a workshop on auditiing NASA archives being proposed for October time frame
David Giaretta >> (All): If people want confidentiality we can arrange that ....
David Giaretta >> (All): ... I think the test audits should be viewed as something to inform us about the docuemnt's shortcomings - we don't have to publish anything
David Giaretta >> (All): ..about them - and certainly not the questionnaire results
Helen Tibbo >> (All): Good. We can do internal stuff to our heart's content but we just can't published based on anything for which we did not seek and receive IRB approval.
David Giaretta >> (All): SImilarly \any table on the Wiki is just so we keep up momentum - we don't need to put things there as long as they get done!
David Giaretta >> (All): Clearly if some repositories are OK with it then things can be published about them but it would be rather a strange report
David Giaretta >> (All): ...since it will reflect as much on the document as on the repository
David Giaretta >> (All): .......So - what else?
David Giaretta >> (All): I noted the ACTIONS above - are they OK?
Mark Conrad >> (All): SO the ACTION remains on (1) finding errors in the metrics (2) organising the test audits and (3) fixing any other erros in the text e.g. references etc?
BruceAmbacher >> (All): ok
Helen Tibbo >> (All): Any data collected by Bruce, Reagan or I needs to go through IRB approval (no matter what the repositories agree to) if it is to be published. If you want to publish this stuff then we shouldn't be doing these at this point as it may take up to a month to get approval and it is a very time consuming process.
David Giaretta >> (All): ...and also on me to complete the questionnaire and WIki page about the test audits
BruceAmbacher >> (All): Helen, can the other party publish it?  This does not involve human subjects.
David Giaretta >> (All): Helen - certainly no publication of anything unless it's all agreed. The main thing is for us toi understand the shortcomings of the metrics etc
David Giaretta >> (All): I'm clearly anxious to start the test audits for real
RobertDowns >> (All): Which questions elicit information about individuals?
David Giaretta >> (All): If we discover we don't have enough test audits we'll have to re-plan
Helen Tibbo >> (All): Bruce, talking with humans always involves human subjects review. 
David Giaretta >> (All): ...but we won't know that until we start for real
Helen Tibbo >> (All): It's not that we are eliciting information about individuals, its that we are TALKING with people.
David Giaretta >> (All): So no publication of anything - aprt from this small group
BruceAmbacher >> (All): IRB process is so picky!
David Giaretta >> (All): Otherwise we will not start
Helen Tibbo >> (All): All you non university folks are good to go. I would have, and I am not kidding a 24 page form with extensive narrative to fill out. I am not up to that now. And then it goes through a very strick review here.
BruceAmbacher >> (All): The only audit in question mentioned today was John's, right?
Marie Waltz >> (All): There is no need for a publication, we're testing the metrics
Helen Tibbo >> (All): However, if we did an audit here and none of the data was included in any publications that would be OK.
David Giaretta >> (All): That's why I'd like to start for real - then we discover what can really be done and whether or not we need to re-plan
Helen Tibbo >> (All): I just want to make all this very clear right now. I know it seems excessive but it is what we live with in university life.
BruceAmbacher >> (All): Geez, Helen, my IRB form is onoy 4 pages plus narrative
David Giaretta >> (All): Sure - let's just agree this is not for publication 
Helen Tibbo >> (All): Bruce, I'll send you the most recent one I have done (not approved yet) that was 23 pages. 
David Giaretta >> (All): ...can you even share things with us?
David Giaretta >> (All): If not then you can be the cut-out and simply tell us your conclusions
David Giaretta >> (All): ...since we are trying to learn about the metrics not the archive
BruceAmbacher >> (All): David, the important issue is "informed consent" and undestanding how the results will be used.
BruceAmbacher >> (All): I have to go now.  I will be here next week and I will provide feedback before
Helen Tibbo >> (All): All the stuff Beth Yakel and Wendy Duff and I have done for the Archival Metrics project has only lo0oked at making the metrics and it has taken extensive IRB review.
David Giaretta >> (All): I guess that's fo us each to negotiate with our archives but the most confidential archive should surely be happy with th leveel of isolation provided by a cut-out and not passing any info on about the archive?
David Giaretta >> (All): If not then we should rule any such test audit out for the moment
David Giaretta >> (All): Which is fine unless we end up with almost no test audits!
David Giaretta >> (All): Then we will have to re-plan
David Giaretta >> (All): I msu go - time for another telecon
Marie Waltz >> (All): bye 
David Giaretta >> (All): Bye all
KatiaThomaz >> (All): have a nice week. bye all.
JohnGarrett >> (All): Bye
Mark Conrad >> (All): Bye!
RobertDowns >> (All): Bye
-- DavidGiaretta - 05 May 2009
Topic revision: r1 - 2009-05-05 - DavidGiaretta
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback