From SemanticLab

Contents 1 Introduction 1.1 Key features 1.2 Architecture 2 Usage 2.1 bibtex 2.2 bstreader 2.3 bibreader 2.4 Import userdefined functions 2.4.1 Python functions 2.4.2 SQL functions 3 Sources 4 Installation 4.1 Database 4.1.1 Create database objects 4.1.2 Insert metata data 4.2 Configuration 4.3 Dependencies 5 Documentation 5.1 Database 5.1.1 ER-diagram 5.2 Schema objects for Oracle DB 5.3 Python Sources 6 Release notes

Introduction

BibSQL is used as a replacement for BibTeX. Bibliography styles (.bst files) and bibliography data (.bib files) are stored directly in the database. The usage of BibSQL is simular to BibTeX. The .aux file generated from LaTeX compiler is used as input. After running BibSQL (.aux file as parameter) the the .bbl file is saved to the working directory. The third & fourth steps (LaTeX compilation) will be executed as usual.

original BibTeX workflow:

BibSQL workflow:

Key features

• Import .bib files into database
• Translate .bst files and imports style definiton to the DB repository
• Generate .bbl file

Architecture

• Module bstreader

This module is subdiveded into th parser and the compiler. The parser opens a bst file and creates a record in the table S_BIBSTYLE. The file content will be split into separate elements (entryfields, variables, functions, macros) and saved in temporary tables, which categorize the functions into format, type, sort and label functions. (Step p1). Merged with metadata, the content from the temporary table will be compiled into SQL functions and stored in the BibSQL repository. In step 2, user defined functions will be transfered into the repository. Similarly in step 3, the entry fields will be tranferred from temporary tables (merged with metadata) into the repository.

• Module bibreader

A bibfile will be read and transformed to a generic, normalized data model.

• Module bibtex

The module bibtex import a LaTeX (.aux)-file that include the information which bib-database and style is used. Furthermore it includes the list of bibliography entries. All generated fields for user defined functions will be stored into the table TEMP_B_FIELDS. In a next step a SQL query will be generated xampl.sql and executed. The output of this query will be fetched into the (.bbl) file.

Usage

 bibdb.py [Optionen] file

 Options:
   -h, --help     show this help message and exit
   -b, --bibtex   bib operation
   -r, --readbib  read bib file
   -s, --readbst  read bst file

bibtex

This module reads a LaTeX aux-file and produce a bbl-file.

Example:

\bibsqlhome\bibsql.py -b xampl(.aux)

The file parameter describes the used (.aux) file. The style (.bst) and the bibliography (.bib) is included in the aux-file and must be already loaded into the database.

stdout -b

[[Generated SQL statement]r

bstreader

Import a BibTeX style file and translate it to SQL. In the first step, several elements from the bst file will be stored into the DB (Temp). As final step, the the translated function will be imported to the bibsql repository.

These are individual steps and can be executed independently.

Example:

 \bibsql\bibsql -s Mybst (.bst)

 Rewrite Temp for this style? Y/N :y

type "n" to skip this step

 Continue process with compiling? Y/N :y

stdout

NB: The translation of the standard styles "plain","alpha","unsrt" and "abbrv" are not included in the published version.

For example: to import the style "plain" enter:

 \bibsql\bibsql -s \bibtexhome\plain.bst

bibreader

Import bib database file

 \bibsql\bibsql -r Mybib (.bib)

Import userdefined functions

A utility is available to import userdefined functions into the database. When a bst file is imported, the compiler will skip the translation of those functions. The user defined functions for compiling "plain","alpha","unsrt" and "abbrv" are already included in the standard installation (insert metadata)!

Python functions

User defined Python functions are stored in the table META_FUNCTIONS (column FUNCTION_PYTHON).

To import functions run:

impPySqlFunctions.py

Configuration: Open the file impPySqlFunctions.py and edit line 5:

bst = 'abbrv'

Type the name of the connected style e.g. 'abbrv' or "" for common use. The program will open the file pySqlFnc_"+bst+".py, e.g. pySqlFnc_abbrv.py.

Each function requires a header:

NAME                     ='format.names'
FUNCTION_BST             ='v format.names'
FUNCTION_SQL             ='v1_FORMAT_NAMES' 

Examples:

pySqlFnc_.py

pySqlFnc_plain.py

pySqlFnc_alpha.py

pySqlFnc_abbrv.py

pySqlFnc_unsrt.py

The nature of these functions are that the value will pre-calculate and temporarily store in a pseudo column e.g. 'editor_FORMAT_NAMES'. The entry field will generated by runtime. To learn more about user defined functions, compare the content of the files with the table META_FUNCTIONS.

SQL functions

User defined SQL functions are stored in the tables META_TYPEFUNCTIONS and META_FUNCTIONITEMS.

Defined SQL functions will skip the translation of the corresponding bst function.

The following functions are distributed:

• format.vol.num.pages
• calc.label
• format.tr.number
• calc.label only for "alpha"
• presort only for unsrt
• presort

To import functions run:

impSqlFunction.py

Configuration: Open file impPySqlFunctions.py and edit line 4:

bst = 'alpha'

Examples:

sqlFnc_alpha.sql

Sources

You can checkout BibSQL from subversion:

svn checkout http://svn.semanticlab.net/svn/oss/thesis/bibtex-management/thesis/bibsql

Installation

Steps:

1. Copy the source (including all subdirectories) into a working directory.
2. Create database
3. Import Metadata
4. Compile the style

Database

• Option 1

Create all database objects and insert metadata with SQL scripts. Download SQL scripts

• Option 2

If You are using an Oracle 11.1 or higher, You can use the utility IMP to import the dumpfile directly. EXPDAT_bibdb.DMP

Create database objects

SQLPLUS script: BIBDB_BUILD_ALL.sql

Alternative:

Run each SQL script in correct order (see BIBDB_BUILD_ALL.sql) with any other SQL tool.

• Tablespaces

Note: It's not recomended to use the "USER" tablespace. If You create a own tablespace, You have to change the tablespace definition in all Table- and Index-scripts.

• Users

Create an Oracle User

Note: It's recomended but not obligatory to use the BIBDB user. If you are using a differing user You must change the all scripts. Take care that this user have the "RESOURCE" and the "CONNECT" Role and grants to the tablespace.

• Sequences
• Tables
• Indexes
• Views
• Constraints

Insert metata data

Run insert script in following order:

1. OWNERS
2. META_FUNCTIONS
3. META_ENTRY_FIELDS
4. META_TYPEFUNCTIONS
5. META_FUNCTIONITEMS

optional:

• INFO_BUILDIN

Configuration

Edit the config.py file for basic settings:

       self.__BIBSQL_HOME="c:\\bibsql\\sqllib\\"
       self.__DEFAULT_NLS= 'GERMAN'
       self.__OWNER =   "2"     #'DEFAULT'                    # do not use  META ("1") --> Table OWNER
       self.__VERBOSE = True    # verbose mode True or False

Edit the database.py file for database settings:

       self.__dbapi = 'cx_Oracle'            # oracle
       #self.__dbapi = 'pysqlite2'           # sqlite3
       #self.__dbapi = 'postgress'           # postgress
       
       if self.__dbapi == 'cx_Oracle':
           self.__cnx = cx_Oracle.Connection("BIBdb/bibdb@orcl")
       elif self.__dbapi == 'pysqlite2':
           self.__cnx =sqlite3.connect("bibdb.db")
       elif self.__dbapi == 'postgress': 
           self.__cnx =psycopg2.connect('host=localhost user=bibnew dbname=bibnew')

Dependencies

• Python 2.5 or higher
• LaTeX
• ANSI-SQL Database (currend Version is tested on Oracle 11.1)

Documentation

Database

ER-diagram

Schema objects for Oracle DB

Schema BIBDB in ORACLE Database

Python Sources

bibsql.py

Release notes

BibSQL Version 0.90

Open topics:

1. The render function (finally step: setting periods , commas and new blocks) is not implemented correct
2. Macros are not supported
3. @PREAMBLE is not supported
4. Functional tests only on Oracle 11g
5. No comparative test with BibTeX

Feedback: mailto:bibsql@froehlich.info