Abstract

LanguageDCAT-AP caters for the description of language data, language models and language technology services.

These are also referred to as "language resources" or "language resources and technologies". As defined in the META-SHARE ontology, a "language resource" is "a resource composed of linguistic material used in the construction, improvement and/or evaluation of language processing applications, but also, in a broader sense, in language and language-mediated research studies and applications; the term is used with a broader meaning, encompassing (a) data sets (textual, multimodal/multimedia and lexical data, grammars, language models, etc.) in machine readable form, and (b) tools/technologies/services used for their processing and management."

Work on LanguageDCAT-AP was initiated in the Common European Language Data Space (LDS), but it is based on a long history of activities carried out in the context of infrastructures intended for the sharing of language resources. LanguageDCAT-AP is, thus, designed with a wider scope covering broader description needs in the Language Data, Language Technology and Language-centric Artificial Intelligence domain.

Introduction

LanguageDCAT-AP is conceived and implemented as an extension of (a) the DCAT-AP application profile for Catalogues containing Datasets and Data Services in Europe [[DCAT-AP]] and (b) the MLDCAT-AP profile for machine learning models and their datasets. It also builds upon the META-SHARE ontology, and the vocabularies associated with it, such as the OMTD-SHARE ontology, especially for domain-specific concepts and properties. To cover language data specificities, it adopts classes and properties from these vocabularies, introduces new ones, where needed, and proposes domain-specific controlled vocabularies.

Τhe current version (v0.9.3) caters for all resource types (i.e., datasets, and more specifically, corpora and lexical/conceptual resources (see Section on Terminology), and language models), as well as for language technology (Natural Language Processing and Language-centric AI) services. All resource types are accompanied with a minimal set of attributes that help consumers discover them, while the full set of attributes are an ongoing work and will be added in future versions.

Licence

All material in this repository is published under the licence CC-BY 4.0, unless explicitly otherwise mentioned.

Context

One of the main objectives of LanguageDCAT-AP is to cover the requirements of exchanging language data and models in the context of the Common European Language Data Space and, as such, takes into account the building principles and specifications of data spaces. The most notable ones are the Data Space Protocol (DSP) and the IDS Reference Architecture Model (RAM, version 4 at the time of writing). Other data spaces initiatives, especially Gaia-X and its specifications (cf., Gaia-X architecture document), and the Data Spaces Support Centre (DSSC), with its Blueprint (version 3.0 at the time of writing), bear a strong influence on the design of LanguageDCAT-AP.

For semantic interoperability purposes, data spaces initiatives recommend the use of DCAT [[vocab-dcat-3]] for the description of catalogues, datasets and data services, and [[ODRL]] for the representation of policies. Specific data spaces have further adopted and extend [[DCAT-AP]] to their domains needs (cf., for instance, mobilityDCAT-AP and HealthDCAT-AP). LDS carries on along these steps, and to this end, closely collaborates with the SEMIC team.

The language technology community has specific needs and demands that lead to the adaptation and extension of DCAT-AP and, for models, MLDCAT-AP. Compatibility with the aforemetioned specifications and guidelines, as well as implementation requirements for data spaces impose further constraints on the way these two models are deployed in LanguageDCAT-AP. Differences are manifested in the data type used for properties, the cardinality of properties, the recommended controlled vocabularies as well as the introduction of new classes and properties. These differences are documented in the descriptions of classes and properties in the following sections (see introduction in the Main entities section for the symbols used). A subset of such differences is presented here for illustration purposes:

• The "language" property is of utmost importance for the description of language data. Therefore, it is mandatory for all resources. It is also distinguished in various types of (sub)properties: referring to the language used for the contents of a dataset, the source/target language of bi-/multilingual datasets, the metalanguage used in lexica and grammars to describe a language, the language(s) of the dataset that a service can process, etc.; these properties can be used, together with the mandatory property, to provide a more detailed account of the resource that is described. Moreover, although the use of the vocabulary recommended in DCAT-AP (EU authority language vocabulary) for the assignment of language values is crucial for interoperability purposes, it does not capture the desired precision for the language community. Thus, in addition to the mandatory "dct:language" property, that takes values from this vocabulary, and fulfills the interoperability needs, the "ms:language" property, that implements the BCP 47 standard [[rfc5646]] is obligatorily used, together with the optional "ms:languageVarietyName" property that encodes language variants, dialects, jargons, etc. in free text.
• Language equality is obviously a foundational principle for the language community, hence the requisite to ensure that metadata descriptions of language resources can be offered in all languages with the proper encoding. LanguageDCAT-AP supports multilinguality by dictating the use of "rdf:langString" for all free text properties instead of the more general "rdfs:Literal" recommended in DCAT-AP and MLDCAT-AP.
• The use of community-specific vocabularies, as, for instance, for size units that can be used to measure the size of datasets (e.g., words, sentences for datasets, terms for terminological glossaries, concepts for ontologies, etc.), for the functions of language models (e.g., annotation, sentiment analysis, machine translation, etc.) is combined with community-specific properties.
• It is important to stress that LT services are represented with a distinct class from that of Data services as their description needs impose a richer and more complex set of properties. The fact that they are not linked to specific datasets but can be used to process different types of data and enrich them with new information, or retrieve from them information and/or generate new resources reinforces the arguments for their separate treatment.

Terminology

An Application Profile is a data specification aimed to facilitate the data exchange in a well-defined application context. It re-uses concepts from one or more semantic data specifications, while adding more specificity, by identifying mandatory, recommended, and optional elements, addressing particular application needs, and providing recommendations for controlled vocabularies to be used. More information can be found on the SEMIC Style Guide. [Definition from MLDCAT-AP]

A dataset represents a collection of data, published or curated by a single agent or identifiable community. [Definition from [[vocab-dcat-3]]]

A language resource is a resource composed of linguistic material used in the construction, improvement and/or evaluation of language processing applications, but also, in a broader sense, in language and language-mediated research studies and applications; the term is used with a broader meaning, encompassing (a) data sets (textual, multimodal/multimedia and lexical data, grammars, language models, etc.) in machine readable form, and (b) tools/technologies/services used for their processing and management. [Definition from META-SHARE]

Language resources are further distinguished into:

• corpora: structured collections of pieces of data (textual, audio, video, multimodal/multimedia, etc.) typically of considerable size and selected according to criteria external to the data (e.g., size, type of language, type of text producers or expected audience, etc.) to represent as comprehensively as possible the object of study;
• lexical/conceptual resources: datasets organised on the basis of lexical or conceptual entries (lexical items, terms, concepts, etc.) with their supplementary information (e.g., grammatical, semantic, statistical information, etc.)
• machine learning models: model artifacts created through a training process involving a Machine Learning (ML) algorithm (that is, the learning algorithm) and the training data to learn from;
• language technology tools/services: tools/services/any piece of software that performs language processing and/or any Language Technology related operation;
• computational grammars and other language descriptions, i.e., resources that formally describe a language.

Initially, models were considered under the class of "language descriptions", but given the way they have evolved, especially with the advent of Large Language Models (LLMs), they have been recognised as a distinct category.

Used prefixes

LanguageDCAT-AP is a semantic specification, where each class and property has an URI. The prefixed form of the URI, can be found in part in the class diagram and in the quick reference table at the bottom, while, for ease of reading, full URI can be found in the hyperlink of each class and property. This specification uses the following prefixes to shorten the URIs for readability:

Overview

Main entities

The main entities are those that form the core of the Application Profile.

The properties and their associated constraints that apply in the context of this profile are listed in a tabular form. Each row corresponds to one property. In addition to the constraints also cross-references are provided to DCAT-AP, and MLDCAT-AP. To save space, the following abbreviations are used:

• DCAT-AP [A]: reused as-is defined and expressed in DCAT-AP
• DCAT-AP [E]: reused with additional usage notes or additional restrictions compared to DCAT-AP
• MLDCAT-AP [A]: reused as-is defined and expressed in MLDCAT-AP
• MLDCAT-AP [E]: reused with additional usage notes or additional restrictions compared to MLDCAT-AP
• P: LanguageDCAT-AP profile specific extension where DCAT-AP and MLDCAT-AP had no requirements earlier

This reuse qualification assessment is with respect to a specific version of the two profiles. Therefore it may vary over time when new versions thereof are created.

It should also be noted that the cross-references are also used in the case of subclasses or equivalent classes of relevant entities; for instance, properties assigned to Corpus and Lexical/Conceptual Resource (subclasses of "Dataset") include links to the DCAT-AP properties assigned to "Dataset", when re-used from it.

Supportive entities

The supportive entities are supporting the main entities in the Application Profile. They are included in the Application Profile because they form the range of properties.

Datatypes

The following datatypes are used within this specification.

Controlled Vocabularies

Requirements for controlled vocabularies

LanguageDCAT-AP adopts the requirements set in DCAT-AP for the recommendation of controlled vocabularies. According to these, controlled vocabularies SHOULD:

• Be published under an open licence.
• Be operated and/or maintained by an institution of the European Union, by a recognised standards organisation or another trusted organisation.
• Be properly documented.
• Have terms that are identified by URIs with each URI resolving to documentation about the term.

• Have associated persistence and versioning policies.

The following two requirements identified for DCAT-AP are considered of less importance for LanguageDCAT-AP:

• Have labels in multiple languages, ideally in all official languages of the European Union.
• Contain a relatively small number of terms (e.g. 10-25) that are general enough to enable a wide range of resources to be classified.

Controlled vocabularies to be used

LanguageDCAT-AP specifies a number of controlled vocabularies that MUST be used for specific properties in order to increase semantic interoperability. These vocabularies are selected from among those recommended in DCAT-AP and MLDCAT-AP, and those used in the language data/technologies and neighbouring communities. In the first case, they are controlled by an EU institution, while in the latter case, they are published and maintained by the LDS metadata working group. Requests for changes and addition to the LDS-controlled vocabularies can be made by adding a new issue at https://github.com/LanguageDCAT-AP/LanguageDCAT-AP/issues.

The following table lists the properties that take values from controlled vocabularies, as well as the classes for which these properties are used. The range of these properties is defined as a class (cf. column "Range") as well as a skos:concept belonging to the specified controlled vocabulary.