The Directory of Open Access Journals (DOAJ) is increasingly being used as a benchmark to determine whether a journal is fully OA, most notably as part of both the original and recently revised Plan S guidelines. This month we take a look at the DOAJ and consider how it compares to other sources for evaluating fully OA status.

Background

The draft Plan S Implementation guidelines suggested that a listing in the DOAJ is mandatory if a journal is to be considered fully OA. This led to commentators raising concerns that many DOAJ Journals do not meet other Plan S criteria, and are therefore not compliant.

In an open letter on its blog, the DOAJ team addressed concerns about its coverage, journal quality, promotion of APC-charging journals, and Plan S compliance. The DOAJ is not (at the time of writing) a formal partner of Plan S. The team do not claim it offers exhaustive coverage and remain neutral to whether journals charge APCs. In 2014 it tightened its acceptance criteria and made all of its 10,000+ indexed journals reapply to remain indexed, which led to a shrinkage in its coverage. The team is now taking active steps to ensure that lost ground in arts & social sciences coverage (in particular) is made up and they have ambitions to keep it updated at least annually.

Whatever its shortcomings, the DOAJ is used as a benchmark by academic studies as it’s a convenient data set to analyze. ImpactStory uses a journal’s presence in the DOAJ as a key indicator of its fully OA (gold) status,* which means that products which consume ImpactStory data (such as Web of Science and Dimensions) do so as well. Following its recent consultation exercise, the latest implementation guidelines from cOAlition S restate the DOAJ as being the de facto index of compliant fully OA journals. cOAlition S also hints at further work to be done, stating that it “will work with the…DOAJ…and other potential partners to establish mechanisms for identifying and signalling whether journals…fulfil [its] requirements.”

Coverage

Since 2016, the team at Bielefeld University’s Institute for Interdisciplinary Studies of Science (I2SoS) have analyzed the DOAJ’s coverage of fully OA journals. Part of their latest update is shown in Table 1.

Source: Bruns, A., et al., 2019. ISSN-Matching of Gold OA Journals (ISSN-GOLD-OA) 3.0, Bielefeld University.Data readings taken 9th & 10th Jan, 2019.

• The Bielefeld team combined lists from multiple sources: The DOAJ, ROAD (issn.org’s fully OA journals database), PMC (PubMed Central) and the Open APC project.
• Some journals appeared in more than one source. The team de-duplicated these using ISSN information to arrive at a total sample size. The totals of the individual indexes will add up to more than this as some journals appear in more than one index
• Around 33%, or 12,582, of the 37,755 fully OA journals the team identified are covered in DOAJ.
• For comparison purposes, the table shows overlaps with Web of Science (WoS) and Scopus indexes.
• If restricted to the fully OA journals that appear in Web of Science, DOAJ covers ~75% of them. Likewise, it covers ~65% of those in Scopus.

So, if using the DOAJ alone to indicate fully OA status, we could – in theory – miss significant coverage. For example, Scopus lists just under 6,000 fully OA journals, almost one third of which (2,077) are not in the DOAJ.

However, it is important to note that this study does not account for the DOAJ’s inclusion criteria. Many journals may legitimately not be in the DOAJ because they either fail the DOAJ’s inclusion criteria, or because their publishers have not submitted (or updated) their information. The team at the DOAJ have run their own studies to confirm these factors are indeed at work.

“Accuracy”

The Bielefeld study analyzes presence or absence of journal ISSN(s) in various sources. A different approach is to compare the data against publishers’ own lists and put this in the context of the number of journals that are not fully OA. Figure 1 shows the results.

Sources: Publishers’ Price Lists, DOAJ, Delta Think Analysis.

The chart looks at whether fully OA journals (as stated by their publishers) are included in the DOAJ. Those that are not in the DOAJ but should be (“false omissions”) are shown in red, and those that are not fully OA but are in the DOAJ (“false inclusions”) are in green. Our data span around 14,000 journals from the 30 largest publishers. For the purposes of comparison, in the cases of omission we assume that a journal in a “top 30” publisher’s list would be of sufficient quality to meet the DOAJ’s inclusion criteria if the publisher applied.

• Our Price List data suggest that, of our sample of large publishers, around 4.9% of their fully OA journals are not listed in the DOAJ. 0.6% of their journals which are not fully OA are listed in the DOAJ, suggesting a total “mis-match” rate of 5.5%.
• These numbers are varying year by year. In 2016 and 2018 around 5% of publishers’ fully OA journals were not in the DOAJ; in 2017 that number leapt to around 10% for all but one publisher in our sample. The 2017 number is an outlier, largely due to the DOAJ team working through the backlog of its resubmission exercise.

Conclusions

The DOAJ is now being used as an ipso facto source which feeds a number of services and is a core part in OA compliance policies (even allowing for caveats). However, if the DOAJ is being used as a sole indicator of journals’ fully OA status, then their coverage will be incomplete.

Much of this discrepancy may be due to omitted journals not meeting the DOAJ inclusion criteria. It ultimately remains the responsibility of publishers to make sure that their (eligible) DOAJ data are up to date. If publishers fail to do this, then their journals may not appear – or not appear correctly – in some widely-used discovery services.

Meanwhile, if concerns exist about the quality of the DOAJ’s coverage, the question is then: What’s the alternative? Other sources (such as PMC or OpenAPC) may offer some help in supplementing the DOAJ, but their coverage is relatively limited and they (along with other sources) may not offer consistently sufficient quality of coverage. The ISSN International Centre (CIEPS) offers another view of fully OA journals via its ROAD database, as does Ulrich’s. But their more permissive criteria raise questions of journals of dubious quality confounding the coverage and, indeed, what the industry standard for inclusion should be.

Almost 20 years after the BOAI declaration, it seems that our industry still lacks a definitive means of certifying basic OA journal activity. The DOAJ is now providing an industry standard by default. It may not be perfect, but absent an alternative, as an industry we need to ensure that it is good enough.

* Note on definition. Open Access is technically the property of an article. We refer to journals that publish only OA articles as “fully OA” journals, but many sources call them “gold journals.”