http://www.lib.ua.edu/wiki/digcoll/index.php?title=Special:NewPages&feed=atom&hidebots=1&hideredirs=1&limit=50&offset=&namespace=0&username=&tagfilter=&size-mode=max&size=0UA Libraries Digital Services Planning and Documentation - New pages [en]2019-09-15T10:11:02ZFrom UA Libraries Digital Services Planning and DocumentationMediaWiki 1.31.3http://www.lib.ua.edu/wiki/digcoll/index.php/Database_Relations_in_AcumenDatabase Relations in Acumen2017-07-05T17:59:34Z<p>Jlderidder: </p>
<hr />
<div><br />
Acumen has numerous tables, some of which I'm not even sure are in use. First I'll list the tables, and then I'll talk about the ones that I've worked with to sort out problems. Here's the tables (remember, this is from all 3 acumen databases -- one for live, one for staging, and one for dev). The ones in bold I will describe further:<br />
<br />
| aauth_group_to_group | (empty)<br />
| aauth_groups | (3 entries; clearly Will or Tonio was hoping to set up multiple levels of access)<br />
| aauth_login_attempts | (empty)<br />
| aauth_perm_to_group | (empty)<br />
| aauth_perm_to_user | (empty)<br />
| aauth_perms | (empty)<br />
| aauth_pms | (empty)<br />
| aauth_user_to_group | (2 entries)<br />
| aauth_user_variables | (empty)<br />
| aauth_users | (1 entry)<br />
| '''asset''' |<br />
| asset_orphan | (empty)<br />
| '''asset_type''' |<br />
| '''authority''' |<br />
| '''authority_type''' |<br />
| crowd_status | (to be used if anyone bothered to review tags/transcripts and approve them)<br />
| featured | (to be used with what Will was setting up in dev, to showcase collections)<br />
| '''file''' |<br />
| '''file_type''' |<br />
| note | (empty)<br />
| note_type | (empty)<br />
| privilege | only one entry<br />
| privilege_type | only one entry<br />
| scan_task | (empty)<br />
| scratch | (not used often, and not sure how/why it's used)<br />
| search_category | (empty)<br />
| '''status_type''' |<br />
| '''tags''' |<br />
| '''tags_assets''' |<br />
| '''transcripts''' |<br />
| user | (two test entries)<br />
| user_action | (empty)<br />
| user_action_type | (empty)<br />
<br />
==Tables of Interest==<br />
*asset<br />
*asset_type<br />
*authority<br />
*authority_type<br />
*file<br />
*file_type<br />
*status_type<br />
*tags<br />
*tags_assets<br />
*transcripts<br />
<br />
===asset_type table===<br />
This table lists the types of digital content (apart from metadata) that we currently support in Acumen, with the extensions to be expected for the delivery file and the extensions to be expected for the thumbnails. Each type of asset has an assigned ID:<br />
<br />
| id | type | asset_tails | thumb_tails | parent_folder |<br />
+----+----------+---------------------------+---------------------------------+---------------+<br />
| 1 | image | .tif,.tiff,_2048.jpg | _512.jpg,_128.jpg,.txt,.ocr.txt | NULL |<br />
| 3 | video | _512kb.mp4,.mp4 | _128v.jpg,_800s.jpg | NULL |<br />
| 4 | audio | _64kb.mp3,_256kb.mp3,.mp3 | .txt,.ocr.txt | NULL |<br />
| 2 | document | .pdf | _128c.jpg | NULL |<br />
| 6 | ignore | .icon.jpg | | NULL |<br />
<br />
If you need Acumen to support new types of files, add them here. If they will exist in a subdirectory (such as SupplementalFiles), I think that would go into the parent_folder field.<br />
<br />
===asset table===<br />
Here's the columns in the asset table. This table tracks every single image, audio, video and text file, each of which is considered an &quot;asset&quot; to a metadata record somewhere, in some capacity:<br />
<br />
| Field | Type | Null | Key | Default | Extra |<br />
+--------------------+----------------------+------+-----+---------+----------------+<br />
| id | bigint(20) | NO | PRI | NULL | auto_increment | This is referred to in other tables as file_id<br />
| asset_type_id | int(11) | NO | | NULL | | See asset_type ID values in asset_type table above<br />
| name | varchar(255) | NO | UNI | NULL | | This is of the form: u0003_0000581_0000001<br />
| orig_path | varchar(255) | NO | | NULL | | This is the acumen path for web delivery, including file name<br />
| thumb_path | varchar(255) | NO | | NULL | | This is the acumen path for web delivery of thumb (no file name)<br />
| file_id | bigint(20) | YES | MUL | NULL | | See file_type ID values in file_type table below<br />
| file_size | bigint(20) | NO | | NULL | | size in bytes<br />
| file_last_modified | bigint(20) | NO | | NULL | | this is in seconds since the Epoch<br />
| status_type_id | int(11) | NO | MUL | NULL | | See status_type ID values in status_type table below<br />
| found | smallint(5) unsigned | NO | MUL | NULL | | If &quot;1&quot; the system found the file; if &quot;0&quot; there's a problem.<br />
<br />
<br />
As you can see, fields in some tables refer to fields in others. Notably, the different types of IDs are used to pull information together. That's why this is called a relational database.<br />
<br />
<br />
===authority===<br />
The authority table brings together ID values from the file table, the asset table, the authority_type table, and matches them to a value.<br />
So if you wanted a list of all titles, you could query: &quot;select value from authority where authority_type_id = 1;&quot; since in authority_type, title is assigned ID number 1. (Normally, though, we want to find a value for a particular file; for that, you'll need to look up the file_id in the file table.)<br />
<br />
| Field | Type | Null | Key | Default | Extra |<br />
+-------------------+------------+------+-----+---------+----------------+<br />
| id | bigint(20) | NO | PRI | NULL | auto_increment | this is the authority_id used elsewhere<br />
| file_id | bigint(20) | NO | MUL | NULL | | this is the ID from the file table to specify which file exactly<br />
| asset_id | bigint(20) | YES | MUL | NULL | | this is the ID from the asset table to specify which asset exactly<br />
| authority_type_id | int(11) | NO | MUL | NULL | | this is the ID from the authority_type table below, for which field it is<br />
| value | text | NO | | NULL | | this is the actual text value of the field in question<br />
<br />
So if I've already looked in the file table and found that the file_id for u0003_0000581_0000001.mods.xml is 150791, and I want to look at the title for that record (titles are authority_type_id of 1 from the authority_type table above, I can query:<br />
<br />
MariaDB [acumen]&gt; select * from authority where file_id=150791 and authority_type_id=1;<br />
+----------+---------+----------+-------------------+-------------------------------------------------------------+<br />
| id | file_id | asset_id | authority_type_id | value |<br />
+----------+---------+----------+-------------------+-------------------------------------------------------------+<br />
| 23391915 | 150791 | -1 | 1 | Letter from Henry L. Abbot to William C. Gorgas, 1904-02-11 |<br />
+----------+---------+----------+-------------------+-------------------------------------------------------------+<br />
<br />
As you can see, the asset_id value is -1 because this value does not apply to an asset in this case, but instead to a file.<br />
<br />
<br />
===authority_type===<br />
In authority_type, each field (or type) is assigned an id number:<br />
# title<br />
# creator<br />
# archivist<br />
# subject<br />
# date <br />
... and so on. The ID number assigned here is used to refer to the value stored in the authority table.<br />
<br />
===file===<br />
<br />
The file table and the asset table are the heart of how content is managed in Acumen. The entries in each one have to relate to the other entries (including the ones in the other table) properly. If they do not, Acumen gets very confused, and delivers garbage online. Check first to see if files are out of order in the web directories. If they are not, then the database has become corrupt. You'll have to find the cause, and fix it. When I found it corrupted in mid 2016, many assets had the wrong file_id entry, and many file_name entries had the wrong parent_id.<br />
<br />
| Field | Type | Null | Key | Default | Extra |<br />
+--------------------+----------------------+------+-----+---------+----------------+<br />
| id | bigint(11) | NO | PRI | NULL | auto_increment | this is the file_id used elsewhere<br />
| parent_id | bigint(11) | YES | MUL | NULL | | the file_id of the file that hierarchically is above this one<br />
| file_type_id | int(11) | NO | | NULL | | the file_type id from the file_type table<br />
| title | varchar(255) | NO | MUL | NULL | | the title again; a duplication from the authority table (why??)<br />
| file_name | varchar(255) | NO | MUL | NULL | | the file name WITH EXTENSION<br />
| file_path | varchar(255) | NO | | NULL | | the SERVER full path to the file, including file name<br />
| file_size | int(11) | NO | | NULL | | size in bytes<br />
| file_last_modified | bigint(11) | NO | | NULL | | in seconds from the epoch<br />
| status_type_id | int(11) | NO | MUL | NULL | | see status_type for ID value; 1 is good! others are not<br />
| found | smallint(5) unsigned | NO | MUL | NULL | | Whether Acumen found this file the last time it indexed<br />
<br />
Parent_Id for a page level MODS should correspond to the item-level MODS. Parent_id for the item_level MODS should correspond to the EAD (or if there isn't one, the collection xml file). Parent_id for the EAD (or collection xml file) should be the holder level, such as u0003 or w0001.<br />
<br />
<br />
'''One way to force a reindex of metadata when things are badly screwed up, is to set all the file_last_modified fields = 0; then reindex.''' That way, the indexer will check each and every metadata record, and not assume anything. Will Jones taught me this one.<br />
<br />
<br />
<br />
===file_type===<br />
<br />
* The file_type table is interesting for a number of reasons. Here is where you would need to add information for new types of metadata.<br />
* The ID value is the file_type_id used elsewhere; the priority indicates what is retrieved first in a search.<br />
* The extension defines the file, and a bit of namespace is in the &quot;signature&quot; field (missed this on collection xml). <br />
* This table tells Acumen which XSL file to use to generate the &quot;summary&quot; for search result lists, and which XSL file to use for display (rendering), and also tells the software at what point to insert the stylesheet into the webpage.<br />
<br />
MariaDB [acumen]&gt; select * from file_type;<br />
+----+----------+-----------+------------+----------------------------+------------------------+----------------+----------------------------+<br />
| id | priority | extension | type | signature | summary_xsl | render_xsl | stylesheet_insertion_point |<br />
+----+----------+-----------+------------+----------------------------+------------------------+----------------+----------------------------+<br />
| 1 | 2 | .mets.xml | mets | http://www.loc.gov/METS/ | mets_summary.xsl | mets.xsl | &lt;mets:mets |<br />
| 2 | 3 | .ead.xml | ead | urn:isbn:1-931666-22-9 | ead_summary.xsl | ead.xsl | &lt;ead |<br />
| 3 | 1 | .mods.xml | mods | http://www.loc.gov/mods/v3 | mods_summary.xsl | mods.xsl | &lt;mods |<br />
| 4 | 4 | .xml | collection | &lt;collInfo&gt; | collection_summary.xsl | collection.xsl | &lt;collInfo |<br />
<br />
<br />
===status_type===<br />
<br />
Here's the status_type IDs. &quot;Release&quot; means it's good, it's online. <br />
<br />
| id | type |<br />
+----+----------------+<br />
| 1 | release |<br />
| 2 | development |<br />
| 3 | obsolete |<br />
| 4 | missing |<br />
| 5 | child_metadata |<br />
| 6 | broken |<br />
<br />
===tags===<br />
This is for user-generated tags.<br />
A simple table with 2 fields: id and value. Each tag is entered only once, and referred to by its assigne tag_id ever afterwards, in the next table.<br />
<br />
===tags_assets===<br />
Again, this is for user-generated tags. It has 3 fields: tag_id, asset_name and status.<br />
Clearly the assumption is that users tag assets, not metadata files. Any asset may have multiple tags. Obviously, many assets could be tagged with the same value, but since they're matched up using the tag_id, the tag value only gets stored once, in the tags table.<br />
Status right now = 1 for every tag. I think if you want to make it not approved, you change it to zero. This was never set up in an interface (yet).<br />
<br />
Here's an example of use:<br />
MariaDB [acumen]&gt; select * from tags_assets where tag_id=909;<br />
+--------+----------------------------+--------+<br />
| tag_id | asset_name | status |<br />
+--------+----------------------------+--------+<br />
| 909 | u0003_0000900_0000382_0089 | 1 |<br />
+--------+----------------------------+--------+<br />
<br />
MariaDB [acumen]&gt; select value from tags where id = 909;<br />
+---------+<br />
| value |<br />
+---------+<br />
| cholera |<br />
+---------+<br />
<br />
So we can see that u0003_0000900_0000382_0089 was tagged with the word &quot;cholera&quot;. <br />
<br />
<br />
<br />
<br />
===transcripts===<br />
Here's the transcripts table:<br />
<br />
| Field | Type | Null | Key | Default | Extra |<br />
+------------+------------------+------+-----+-------------------+-----------------------------+<br />
| id | int(10) unsigned | NO | PRI | NULL | auto_increment | a transcript_id not used anywhere, I don't think<br />
| asset_name | varchar(255) | NO | MUL | NULL | | the asset_name<br />
| status | tinyint(3) | NO | MUL | NULL | | (see below) <br />
| value | blob | NO | | NULL | | the transcript itself!<br />
| timestamp | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP | the date it was entered.<br />
<br />
Status in transcripts: <br />
* 1 is transcript<br />
* 0 is OCR<br />
* -1 is old (previous versions)<br />
* -2 means it has one or more words that have been flagged as inappropriate (such as when &quot;p&quot; was left off the word &quot;pass&quot;)</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Acumen_Database_AuthoritiesAcumen Database Authorities2017-07-05T16:29:39Z<p>Jlderidder: </p>
<hr />
<div>There are three tables in the acumen/acumen_staging/acumen_dev databases that deal with fields such as titles, subjects, dates and such:<br />
*authority_type<br />
*authority<br />
*file<br />
<br />
In authority_type, each field is assigned an id number:<br />
# title<br />
# creator<br />
# archivist<br />
# subject<br />
# date <br />
... and so on. The ID number assigned here is used to refer to the value stored in the authority table.<br />
<br />
<br />
The authority table brings together ID values from the file table, the asset table, the authority_type table, and matches them to a value.<br />
So if you wanted a list of all titles, you could query: &quot;select value from authority where authority_type_id = 1;&quot; since in authority_type, title is assigned ID number 1. Normally, though, we want to find a value for a particular file.<br />
<br />
The file table has multiple fields, some of which you will need if troublshooting Acumen. Of interest in this discussion are the file_name, such as &quot;u0003_0000581_0000001.mods.xml&quot; and the title field, which oddly enough echoes what's already stored in the authority table.<br />
<br />
<br />
<br />
Let's say I want to see the title that is stored for u0003_0000581_0000001. I need to check two different places, and be sure to use the full file name.<br />
First: `select id, title from file where file_name like &quot;u0003_0000581_0000001.mods.xml&quot;`;<br />
That gives me the following result:<br />
<br />
<br />
| 150791 | Letter from Henry L. Abbot to William C. Gorgas, 1904-02-11 |<br />
<br />
<br />
Now, I already know that title has ID of 1. So my next query is: `select value from authority where file_id=150791 and authority_type_id=1;`<br />
<br />
<br />
| Letter from Henry L. Abbot to William C. Gorgas, 1904-02-11 |<br />
<br />
<br />
As you can see, the two values are the same. But should you have to modify them directly in the database, be sure to modify them both.<br />
<br />
Now if I want ALL the authority fields for this record, I can use the ID from querying the file table:<br />
<br />
`select authority_type_id, value from authority where file_id=150791;`<br />
<br />
This gives me a result of 43 rows of information, but with an authority_type_id for each field so I can check the authority_type database to see how each field was indexed.<br />
<br />
For more info, see [[Database_Relations_in_Acumen]].</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Changing_what_gets_indexedChanging what gets indexed2017-07-05T16:00:37Z<p>Jlderidder: </p>
<hr />
<div>I don't recommend messing with this until you have a programmer comfortable with SOLR.<br />
<br />
The indexing depends on 3 parts and a couple of processes.<br />
<br />
In the XSL directories (such as /srv/www/htdocs/acumen-old/staging/assets/xsl/ ), there are xsl files that specify which fields get indexed as what from each type of metadata. This allows all titles to be indexed together, all subjects to be indexed together, and so forth:<br />
* collection_to_solr.xsl for collection xml<br />
* ead_to_solr.xsl for EADs<br />
* mods_to_solr.xsl for MODS<br />
* the mets one is not in use.<br />
<br />
Each metadata type may have one or more fields mapped to a field for indexing. For example: subtitle, title, &amp; series title may all be mapped to &quot;title&quot;. Sender, recipient, donor, author, creator, etcetera may all be mapped to &quot;names&quot;.<br />
<br />
Each Acumen &quot;field&quot; to which these have been mapped needs to be assigned to SOLR fields, for SOLR indexing -- and needs to be represented in the database in the authority tables. Here &quot;title&quot; is mapped to &quot;title_s&quot; since it's a string type of content.<br />
<br />
The SOLR mapping takes place in /srv/solr/acumen_whichever/conf/schema.xml -- this file tells SOLR how to deal with the data mapped to that field.<br />
<br />
The database tables that have to match up are authority and authority_type in all three databases: acumen, acumen_staging, and acumen_dev.<br />
<br />
See, told you not to go there. :-)</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/OAI_deliveryOAI delivery2017-07-05T15:38:01Z<p>Jlderidder: </p>
<hr />
<div><br />
[http://www.openarchives.org/OAI/openarchivesprotocol.html OAI (Open Archives Initiative)] is the protocol which allows us to vend our metadata out to metadata harvesters, so that users can search for our content in AlabamaMosaic, in Scout, in WorldCat, in american-south.org and in other portals.<br />
<br />
OAI has 5 verbs, which can be modified with date ranges, metadata types and more.<br />
<br />
**A testing interface for [http://www.openarchives.org/OAI/openarchivesprotocol.html OAI] can be found here: [http://re.cs.uct.ac.za/]. <br />
** On the server, the configuration files for OAI are in '''/srv/www/htdocs/acumen-old/legacy/mv/vendors''' <br />
*** You'll see here XSL for MODS, EAD, METS, Collection and tags; only MODS is in use. I do not know if the SOLR ones are in use.<br />
*** The MODS xsl has 3 versions: one for Ebsco Discovery Services (EDS, or Scout), one for the ASERL project (more on these below), and then the primary one.<br />
*** These files are named (in the same order as above) mods_oai_eds.xsl, mods_oai_aserl.xsl and mods_oai.xsl<br />
*** Changes to these will change how the MODS is translated into unqualified Dublin Core metadata, which is required by OAI.<br />
** And the software that makes it happen is in '''/srv/www/htdocs/acumen-old/legacy/mv/plugins'''<br />
** And here are our OAI URLs:<br />
*** The base URL for OAI is '''http://acumen.lib.ua.edu/legacy/oai'''?<br />
*** For the ASERL project (american-south.org) it was '''http://acumen.lib.ua.edu/legacy/oai_aserl'''? which only provided content with the project link in the MODS.<br />
*** For Scout (the Ebsco discovery interface) it is '''http://acumen.lib.ua.edu/legacy/oai_eds'''? which leaves out the mass-digitized collections.<br />
<br />
Our implementation does not current support sets.<br />
<br />
Currently our implementation returns 200 records at a time.<br />
<br />
The contact person is set in /srv/www/htdocs/acumen-old/legacy/mv/plugis/oai.php<br />
<br />
Here are some test URLs:<br />
* Identify: [http://acumen.lib.ua.edu/legacy/oai?verb=Identify http://acumen.lib.ua.edu/legacy/oai?verb=Identify]<br />
* To fetch a single record, prefix the identifier with &quot;oai:acumen.lib.ua.edu&quot; and use it with the GetRecord verb. This is the link for u0003_0000581_0000001: [http://acumen.lib.ua.edu/legacy/oai?verb=GetRecord&amp;metadataPrefix=oai_dc&amp;identifier=oai:acumen.lib.ua.edu:u0003_0000581_0000001 http://acumen.lib.ua.edu/legacy/oai?verb=GetRecord&amp;metadataPrefix=oai_dc&amp;identifier=oai:acumen.lib.ua.edu:u0003_0000581_0000001]<br />
* To fetch the first 200 records: [http://acumen.lib.ua.edu/legacy/oai?verb=ListRecords&amp;metadataPrefix=oai_dc http://acumen.lib.ua.edu/legacy/oai?verb=ListRecords&amp;metadataPrefix=oai_dc] -- at the end you will see a resumptionToken which allows you to get the next 200 records.<br />
* To fetch the next 200 records: [http://acumen.lib.ua.edu/legacy/oai?verb=ListRecords&amp;resumptionToken=oai_dc;200 http://acumen.lib.ua.edu/legacy/oai?verb=ListRecords&amp;resumptionToken=oai_dc;200] and so on.<br />
<br />
<br />
For more information, read the [http://www.openarchives.org/OAI/openarchivesprotocol.html documentation] and try the [http://re.cs.uct.ac.za/ public testing interface].</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/XSLT_displayXSLT display2017-07-05T15:30:48Z<p>Jlderidder: Created page with &quot;Each type of XML we use in Acumen must have XSLT files for two different things: display and indexing. Here we talk about display. There's a different XSL file for each type...&quot;</p>
<hr />
<div>Each type of XML we use in Acumen must have XSLT files for two different things: display and indexing. Here we talk about display.<br />
<br />
There's a different XSL file for each type of metadata:<br />
* mods.xsl for MODS<br />
* ead.xsl for EAD<br />
* collection.xsl for collection xml files<br />
If you add a new type of metadata, you'll need to provide an xsl for display.<br />
<br />
The XSLT that transforms our XML metadata into a nice display resides in the assets/xsl subdirectories within acumen-old (live), acumen-old/staging, and acumen-old/dev in /srv/www/htdocs/.<br />
<br />
# '''When you need to modify something, first test on your desktop with XMLspy.'''<br />
# THEN copy to the staging directory, after backing up the previous version of the file (/srv/www/htdocs/acumen-old/staging/assets/xsl/)<br />
# View it in staging: [http://acumen.lib.ua.edu/staging/ http://acumen.lib.ua.edu/staging/] and modify if necessary<br />
# Once it looks good, copy the new file to live: /srv/www/htdocs/acumen-old/assets/xsl/ <br />
# Check the live version to make sure it's good: [http://acumen.lib.ua.edu/ http://acumen.lib.ua.edu/]<br />
<br />
NOTE: the oai XSL in these directories are not currently in use. For OAI, look in the LEGACY directory: /srv/www/htdocs/acumen-old/legacy/mv/vendors/.<br />
There is a separate XSL file for each type of metadata and for each target audience there. The mets and collection and tags and ead xsl are not in use at this time.</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Live,_Staging_and_DevLive, Staging and Dev2017-07-05T14:56:31Z<p>Jlderidder: </p>
<hr />
<div>== The Four Faces of Acumen ==<br />
<br />
To keep Acumen running while indexing is going on requires it to have a second face, that people see only briefly. This second face comes in handly also for testing new display options. Any real heavy development, however, needs to take place in yet another face of Acumen. And then there's the legacy face, which is mostly disabled now. <br />
<br />
All three of the primary faces use the content located in '''/srv/www/htdocs/content/''' for delivery, and all three use indexing software in '''/home/acumen/'''.<br />
<br />
Each of the three has their own database.<br />
<br />
Here's the scoop:<br />
<br />
===LIVE===<br />
<br />
This is what you see 98% of the time, when you use Acumen. <br />
* Online visible at '''[http://acumen.lib.ua.edu http://acumen.lib.ua.edu]'''<br />
* The web delivery software is in '''/srv/www/htdocs/acumen-old/'''<br />
* The SOLR index is in '''/srv/solr/acumen_live'''<br />
* The database used is acumen on lib-mariadb-prod01.ua.edu<br />
<br />
===STAGING===<br />
<br />
This is what you see when the live version is indexing; this is also where we test XSLT display before setting it live.<br />
* Online visible at '''[http://acumen.lib.ua.edu/staging http://acumen.lib.ua.edu/staging]'''<br />
* The web delivery software is in '''/srv/www/htdocs/acumen-old/staging/'''<br />
* The SOLR index is in '''/srv/solr/acumen_staging'''<br />
* The database used is acumen_staging on lib-mariadb-prod01.ua.edu<br />
<br />
===DEV===<br />
<br />
This is where Will Jones did his development work, and tried new things.<br />
* Online visible at '''[http://acumen.lib.ua.edu/dev http://acumen.lib.ua.edu/dev]'''<br />
* The web delivery software is in '''/srv/www/htdocs/acumen-old/dev/'''<br />
* The SOLR index is in '''/srv/solr/acumen_dev'''<br />
* The database used is acumen_dev on lib-mariadb-prod01.ua.edu<br />
<br />
===LEGACY===<br />
<br />
This is what remains of Tonio's version of Acumen. We depend on it for [http://www.openarchives.org/OAI/openarchivesprotocol.html OAI (Open Archives Initiative)] delivery, which allows us to search for our content in AlabamaMosaic, in Scout, in WorldCat, in american-south.org and in other portals.<br />
* Online visible at '''http://acumen.lib.ua.edu/legacy'''<br />
* The web delivery software is in '''/srv/www/htdocs/acumen-old/legacy/'''<br />
* This has no SOLR index; this no longer works with the current acumen database<br />
* OAI:<br />
**A testing interface for [http://www.openarchives.org/OAI/openarchivesprotocol.html OAI] can be found here: [http://re.cs.uct.ac.za/]. <br />
** On the server, the configuration files for OAI are in '''/srv/www/htdocs/acumen-old/legacy/mv/vendors''' <br />
** And the software that makes it happen is in '''/srv/www/htdocs/acumen-old/legacy/mv/plugins'''<br />
** And here are our OAI URLs:<br />
*** The base URL for OAI is '''http://acumen.lib.ua.edu/legacy/oai'''?<br />
*** For the ASERL project (american-south.org) it was '''http://acumen.lib.ua.edu/legacy/oai_aserl'''? which only provided content with the project link in the MODS.<br />
*** For Scout (the Ebsco discovery interface) it is '''http://acumen.lib.ua.edu/legacy/oai_eds'''? which leaves out the mass-digitized collections.</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Acumen_DatabasesAcumen Databases2017-06-30T20:14:47Z<p>Jlderidder: </p>
<hr />
<div><br />
== Database Access ==<br />
<br />
* Configuration for connection of database, including username, password, host, character set, etcetera, is found in '''/srv/www/htdocs/acumen-old/application/config/database.php''' and the corresponding database.php files in the staging and dev directories.<br />
* The database is NOT on the same server as the content and delivery software.<br />
<br />
* There are 3 databases:<br />
** acumen for the live version<br />
** acumen_staging for the staging<br />
** acumen_dev for the development area<br />
<br />
See [[Live,_Staging_and_Dev]] for more information. acumen and acumen_staging need to be kept in sync.<br />
<br />
<br />
For table information, see: [[Database_Relations_in_Acumen]].</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Acumen_Front_PageAcumen Front Page2017-06-30T20:11:11Z<p>Jlderidder: </p>
<hr />
<div><br />
== Remember to test all modifications first in the staging area. ==<br />
* '''Staging can be viewed online from [http://acumen.lib.ua.edu/staging http://acumen.lib.ua.edu/staging]'''<br />
* '''Staging area is in /srv/www/htdocs/acumen-old/staging/'''<br />
<br />
<br />
=== Header Links and Banner ===<br />
# Navigate to /srv/www/htdocs/acumen-old/staging/application/views/<br />
# Make a backup copy of index_template.php<br />
# Modify carefully<br />
# View in staging interface online to see if this is correct, and if it causes any problems<br />
# If good, then copy the new version live: cp index_template.php /srv/www/htdocs/acumen-old/application/views/.<br />
<br />
<br />
NOTE: The Decade Chart linked on the front page is located here: http://libcontent.lib.ua.edu/decade_chart/ /srv/www/htdocs/decade_chart</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Acumen_TroubleshootingAcumen Troubleshooting2017-06-30T19:53:52Z<p>Jlderidder: </p>
<hr />
<div>{| class=&quot;wikitable&quot; border=&quot;1&quot; cellpadding=&quot;2&quot; cellspacing=&quot;1&quot;<br />
|- <br />
! Symptoms: <br />
! Other symptoms:<br />
! Likely Cause(s):<br />
! What to do:<br />
! If that doesn't work:<br />
|-<br />
||'''Search doesn't return anything'''<br />
||Item numbers added to the base URL do not display<br />
||Apache down; database access interrupted; or indexer broken<br />
||Restart apache; check database access on commandline (if down contact OIT); manually run indexer and look for errors<br />
||Look at content in web directory to see if anomalies in file/directory names and organization broke something; fix and [[Acumen_Indexing | reindex]].<br />
|-<br />
||'''Display is confused'''<br />
||Items are in wrong collection, have too many parts, are on collection level<br />
||Misnaming, misorganization on back end; or database confusion/corruption<br />
||Look at content in web directory to see if anomalies in file/directory names and organization broke something; fix and reindex.<br />
||see [[Database Relations in Acumen]]; correct what's causing the problem, and then clean up the database by reindexing.<br />
|-<br />
||'''Words don't display correctly from metadata record'''<br />
||The underlying XML file is correct (if not, fix this and reindex)<br />
||XSLT needs modification; diacritic problem; or database issue<br />
||see [[XSLT_display]]<br />
||If it's a diacritic problem, the XML is likely not in Unicode. Correct that instead. If it's the title, see [[Acumen Database Authorities]].<br />
|-<br />
|}<br />
<br />
===Resources===<br />
<br />
* [http://libcontent1.lib.ua.edu:8080/solr/admin.html Online SOLR access including logs]<br />
* Logs on server: /var/log/apache2/ and /var/log/tomcat6/<br />
* [https://wiki.apache.org/solr/ Solr Wiki]<br />
* [[Live,_Staging_and_Dev]]<br />
* Tomcat configurations are set in /etc/sysconfig/tomcat6 and /etc/tomcat6/tomcat6.conf<br />
* OIT support person is Allan Beddingfield; cc all messages to itsd@ua.edu</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Acumen_IndexingAcumen Indexing2017-06-30T19:24:52Z<p>Jlderidder: </p>
<hr />
<div>Acumen indexing is automated by a cron job scheduled under the &quot;acumen&quot; username, which runs a script in /home/acumen/bin called acumen.pl<br />
<br />
==This script does the following things:==<br />
# Backs up the primary acumen database<br />
# Switches out the index.php scripts in the primary web directory (/srv/www/htdocs/acumen-old/ ) so that the staging subdirectory becomes what is live online instead, and the staging database is the one referenced for delivery<br />
# Performs a solr index on the acumen database (java -Xms1g -Xmx2g -jar /home/acumen/acumen.jar acumen &gt;&gt; /home/acumen/acumen.log) – (NOTE: I recently raised the amount of RAM allotted in this call, so you shouldn’t have to modify it again unless the quantity of content grows a great deal before you switch systems.)<br />
# Dumps copies of the primary tables out of the newly indexed database<br />
# Re-switches out the index.php scripts so that now the acumen-old directory is again what is live online<br />
# Pulls copies of the primary tables into the staging database<br />
# Imports that data into the staging version of solr<br />
# And removes older versions of the backed up databases and tables (you'll see these in /home/acumen/sql/ -- should you need older ones, look in /srv/backups/ ). <br />
# It also logs what it’s doing. (You can see this in /home/acumen/acumen.log and acumen_cron.log)<br />
<br />
It’s been working fine without modifications for almost a year now, and hasn’t needed any attention.<br />
<br />
What’s more likely to go wrong is that content will be mucked up that gets put into the web directories – and that has to be corrected. Sometimes that means correcting what’s in the database as well as the web directories, and then reindexing. <br />
<br />
==You can always reindex by:==<br />
<br />
# Change directories to /home/acumen/bin<br />
# As root, become the acumen user. Type in: `su acumen`<br />
# Run the script acumen.pl <br />
<br />
This uses a lot of CPU, so if possible, do this when few people are using the system.</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Acumen_MaintenanceAcumen Maintenance2017-06-30T19:22:54Z<p>Jlderidder: </p>
<hr />
<div>1) '''[[Acumen Indexing | Indexing]] ''' occurs every night @ 8. Content removed or added will not be reflected in the interface until the next index occurs.<br />
<br />
2) [[Acumen Front Page |Modifying front page, headers and footers]]<br />
<br />
3) How to restart apache: As root, run `/usr/sbin/rcapache2 graceful`<br />
<br />
4) [[Acumen Databases | Databases and database access]]<br />
<br />
5) [[Live, Staging and Dev | Live, Staging, and Development areas]]<br />
<br />
6) [[XSLT display | Updating display of metadata (XSLT changes)]]<br />
<br />
7) [[OAI delivery]]<br />
<br />
8) [[Changing what gets indexed]]<br />
<br />
9) '''[[Acumen Troubleshooting]]'''</div>Jlderidderhttp://www.lib.ua.edu/wiki/digcoll/index.php/Setting_Up_Metadata_Librarian_Work_AreaSetting Up Metadata Librarian Work Area2017-06-30T14:36:00Z<p>Jlderidder: /* On the etd server */</p>
<hr />
<div>==On the share drive==<br />
<br />
* Navigate to S:\Metadata\Staging\Staging_Area_Setup and copy the &quot;Copy_This&quot; directory to S:\Metadata\Staging\<br />
* Rename the new version of Copy_This with the metadata librarian's name.<br />
<br />
<br />
==On the libcontent server==<br />
<br />
1) As root, create a new user and home directory with the correct permissions: <br />
<br />
useradd -m -g www newUserName<br />
<br />
2) Assign a temporary password:<br />
<br />
passwd newUserName<br />
{enter password}<br />
{enter password}<br />
<br />
3) Give the username and password to the metadata librarian; tell them to use SSH ([https://www.bitvise.com/ bitvise] is good) and log into libcontent.lib.ua.edu<br />
<br />
4) Copy the contents of the /home/metadataLibrarian directory to the new home area: <br />
<br />
cp /home/metadataLibrarian/* /home/&lt;font color=&quot;red&quot;&gt;newUserName&lt;/font&gt;/.<br />
<br />
5) Change into that directory, and navigate to the ModsEditor folder. Open the getMods script and replace &quot;metadataLibrarian&quot; with the name used for the new directory you just created on the share drive in the staging area.<br />
<br />
6) Change the mount point in that script and in all the others in this home area, to point to the mount point you are assigning to this employee.<br />
grep cifs-mount * in each directory to find them all. Each script needs to be modified by hand, and the mount point must be correct for the <br />
scripts to work properly.<br />
<br />
7) Set up the mount point. Navigate to root (cd / ) to see what's in use. If you ls a mount point and see nothing, it's not in use or has come unmounted. For safety's sake, select a number for which there is no directory yet. The form of the directory name will be &quot;cifs-mount9&quot;, so we'll use that as an example:<br />
* mkdir cifs-mount9<br />
* mount -t cifs -o uid=&lt;font color=&quot;red&quot;&gt;UALIB username here&lt;/font&gt;,gid=www,username=&lt;font color=&quot;red&quot;&gt;YOUR UALIB username here&lt;/font&gt;,domain=lib //libfs1.lib.ua-net.ua.edu/share/Metadata/Staging/&lt;font color=&quot;red&quot;&gt;New staging area directory name here&lt;/font&gt;/MODSedits/ /cifs-mount9<br />
* enter YOUR UALIB password to complete the mount point. &lt;font color=&quot;red&quot;&gt;When your password changes, you'll have to remount all these again!&lt;/font&gt;<br />
<br />
8) Test. ls your mount point to see if the staging directories are there: ls /cifs-mount9<br />
<br />
==On the etd server ==<br />
<br />
1) As root, create a new user and home directory with the correct permissions: <br />
<br />
useradd -m -g users newUserName<br />
<br />
2) Assign a temporary password:<br />
<br />
passwd newUserName<br />
{enter password}<br />
{enter password}<br />
<br />
3) Give the username and password to the metadata librarian; tell them to use SSH ([https://www.bitvise.com/ bitvise] is good) and log into etd.lib.ua.edu<br />
<br />
4) Copy the contents of the /home/metadataLibrarian directory to the new home area: <br />
<br />
cp /home/metadataLibrarian/* /home/&lt;font color=&quot;red&quot;&gt;newUserName&lt;/font&gt;/.</div>Jlderidder