Crowdsourced ranking of journals in Management; any news?

Vote for management journals

Management researchers love journal rankings. A new way to get the judgement of thousands of voters was launched recently by Teppo Felin, Brigham Young University. After a few weeks already 80.000 votes were submitted to rank more than 100 journals in the field of management. Voters can submit a vote as many times as they like.  After filtering out about 20.000 automated robot votes, an impressive amount of 60.000 votes remained.

There are already some blogposts on this ranking where peers discuss the pro’s and cons of performing the ranking this way.

The crowd initiative is here @ All Our Ideas: http://www.allourideas.org/management

Besides the obvious trouble to compare all these journals (a nearly impossible task), one can ask what the ‘wisdom of the crowds’ adds to already existing lists of the top journals in the field.

A lot of normal suspects turn up in the top 25: (as of today 22/01/2011, with 80.000 votes, the results before the cleaning).

A score of 100 means a journal is expected to win all the time when compared with another journal in the contest and a score of zero means the journal is expected to lose all the time. There are no zero scores in this ranking (yet).

 

  1. JOURNAL OF APPLIED PSYCHOLOGY (score: 81)
  2. ADMINISTRATIVE SCIENCE QUARTERLY (76)
  3. ACADEMY OF MANAGEMENT REVIEW (74)
  4. ORGANIZATION SCIENCE (73)
  5. ACADEMY OF MANAGEMENT JOURNAL (72)
  6. JOURNAL OF MANAGEMENT (71)
  7. MIS QUARTERLY (70)
  8. STRATEGIC MANAGEMENT JOURNAL (68)
  9. ORGANIZATIONAL BEHAVIOR AND HUMAN DECISION PROCESSES (67)
  10. MANAGEMENT SCIENCE (66)
  11. DECISION SCIENCES (65)
  12. ORGANIZATIONAL RESEARCH METHODS (64)
  13. JOURNAL OF ORGANIZATIONAL BEHAVIOR (63)
  14. JOURNAL OF MANAGEMENT STUDIES (63)
  15. PERSONNEL PSYCHOLOGY (63)
  16. JOURNAL OF INTERNATIONAL BUSINESS STUDIES (60)
  17. RESEARCH IN ORGANIZATIONAL BEHAVIOR (60)
  18. ACADEMY OF MANAGEMENT PERSPECTIVES (60)
  19. HUMAN RELATIONS (60)
  20. LEADERSHIP QUARTERLY (59)
  21. ORGANISATION STUDIES (59)
  22. JOURNAL OF OPERATIONS MANAGEMENT (59)
  23. HARVARD BUSINESS REVIEW (58)
  24. MIT SLOAN MANAGE REVIEW (57)
  25. HUMAN RESOURCE MANAGEMENT-US (57)

The initiator of the survey reported today (29-jan-2011) on the results:

So, what can a crowdsourced ranking tell us?  Nothing definitive is my guess, though I’m sure other rankings don’t necessarily give us a definitive signal either.  Though I do think that aggregated perceptions perhaps give us another data point when evaluating and comparing journals (along with impact and influence factors and other, more “objective” measures).  These rankings can of course mirror extant rankings (raising causal questions). But they might also capture more up-to-date anticipations of future performance. For example, the UC Irvine Law School (established in 2008) has not graduated a single student, though the school is already well within the top 50 in the crowdsourced ranking.

Lots of other questions can be raised, specifically related to a management journal ranking like this.  For example, should micro and macro journals be lumped together like this?  And certainly disciplinary journals play a large role in management – should they be included (sociology, psychology, economics)?

Strategic “gaming” of the results of course can also occur.  For example, I ended up having to delete some 25,000+ automatically generated votes (it looked like a computer script was created to throw the ranking off), votes that were explicitly cast to sabotage the effort (the African Journal of Management beat all the top journals according to this mega, robo-voter).  Though, it is interesting to see how the “crowd” essentially plays a role in averaging bias and in putting a check on strategic voting.

Ironically, I’m actually not one to necessarily really care about journal rankings like this.  I wonder whether article-effects trump journal-effects?  (I believe Joel Baum has a provocative paper on this.)  Of course I read and submit to “top” journals, but there are many “lesser” (ahem)  journals that are just as much on my radar screen, for example Industrial and Corporate Change, Managerial and Decision Economics andStrategic Organization. Obsessions with journal standing can detract from ideas.

Pragmatically, yes, journal rankings matter: promotions indirectly depend on it, as do resource flows etc.  So, perhaps a “democratic,” crowdsourced ranking like this can provide additional information for decision-makers and scholars in the field.

 

The top 15 list after filtering out the 20.000 automated robot-votes: the picture is now even more ‘traditional’.

Management Journals %
1 Administrative Science Quarterly 90.43%
2 Academy of Management Journal 90.41%
3 Academy of Management Review 88.94%
4 Organization Science 88.31%
5 Strategic Management Journal 84.42%
6 Journal of Applied Psychology 84.00%
7 Management Science 82.56%
8 Journal of Management 82.46%
9 OBHDP 78.93%
10 Organizational Research Methods 74.06%
11 Journal of Organizational Behavior 72.79%
12 Personnel Psychology 71.93%
13 Journal of Management Studies 71.10%
14 Research in OB 70.37%
15 Organization Studies 69.68%

 

Where do all these votes come from?

The whole picture is US en EU dominated. For example: only 16 votes from Shanghai, China are recorded.

overall crowd votes worldmap 20110122

The complete ranking is here as PDF file of the ranking website (22/01/2011): https://rmimr.files.wordpress.com/2011/01/all-our-ideas-crwodsourcing-management-journals-20110122.pdf

Science-metrix develops new journals classification system (and interactive webtool) using both the Web of Science (Thomson Reuters) and Scopus (Elsevier).

sciencemetrix explorerA new set of tools recently released (first public release: 2010-12-01 (v1.00)) by Science-Metrix Inc. seeks to improve the way we talk about and understand science. The US/Canada-based research evaluation firm has developed a new, multi-lingual (18 languages!) classification of scientific journals, which is accompanied by an interactive web tool.  The journal classification, which covers 15,000 peer-reviewed scientific journals, is translated by more than 22 international experts who volunteered their time and expertise, making the tools available to a worldwidScience-metrix logoe audience.  The complete journals list is available for download as excel sheet.

The interactive ‘Scientific Journals Ontology Explorer’ allows users to visualise the links between 175 scientific specialties (subject categories) in 18 languages, from Arabic to Swedish.

The visualization contains 3 different views: a circular “Subfield Citation Wheel” (representing both citations as well as references), a “Field Citation Wheel” (showing the links between distinct scientific disciplines) and a network “Map of Science” (revealing similarities between disciplines by relative distance). The goal of this visualization is to show people how science spans a broad universe and how interlinked scientific research actually is.

How is the field of Business & Management covered in the journals list?

The field of  Economics & Business (as part of the domain Economic & Social Sciences) contains 822 journals (this is 5,4% of he total Science metrix list)  in 12 subfields, where Business & Management is included with 222 journals (27%):

Every journal is classified into only one category.

Subfields of ‘Economics & Business’ N %
Accounting 32 3,9%
Agricultural Economics & Policy 27 3,3%
Business & Management 222 27,0%
Development Studies 42 5,1%
Econometrics 13 1,6%
Economic Theory 13 1,6%
Economics 244 29,7%
Finance 63 7,7%
Industrial Relations 21 2,6%
Logistics & Transportation 49 6,0%
Marketing 61 7,4%
Sport, Leisure & Tourism 35 4,3%
Total Journals 822 100%

I will compare these with our ERIM Journals list, the ISI quartiles and the SJR (scopus) quartiles scores to see how the list is structured in terms of quality layers.

(I wil add these details later this week.)

In the ontology browser, you can create a map of science and learn how the field of business and management is connected to other subject categories. I have selected the closest fields in the screenshot below.

business and management in science metrix

About Science-Metrix:

Science-Metrix is the world’s largest independent firm dedicated to scientometrics, technometrics, and science and technology (S&T) evaluation. The firm’s core business involves supporting evidence-based decision-making with strong empirical data and sound theoretical approaches. This contract research organization combines qualitative and quantitative techniques to deliver high quality program evaluations, performance and outcome assessments, and evaluation frameworks. Every Science-Metrix report is produced by a team of dedicated high-calibre experts and relies on the world-class data found in the Scopus, Web of Science and Questel databases.

How are the stocks (top journals) performing over a period of 28 years?

Sciencewatch has published the latest Journal Performance Indicators (JPI’s) for the field of business.  This chart shows which journals perform in the top over decades.  It shows that the very top of the field is very stable over a longer period.  Publishing in these top journals can be seen as a stable investment for researchers; it increases the possibility of a good citation yield.

The table below compares the citation impact of journals in the field of business as measured over three different time spans. The left-hand column ranks journals based on their 2008 “impact factor,” as enumerated in the current edition of Journal Citations Report. The 2008 impact factor is calculated by taking the number of all current citations to source items published in a journal over the previous two years and dividing by the number of articles published in the journal during the same period–in other words, a ratio between citations and recent citable items published. The rankings in the next two columns show impact over longer time spans, based on figures from Journal Performance Indicators. In these columns, total citations to a journal’s published papers are divided by the total number of papers that the journal published, producing a citations-per-paper impact score over a five-year period (middle column) and a 28-year period (right-hand column).

Journals Ranked by Impact: Business
Rank 2008 Impact Factor Impact 2004-08 Impact 1981-2008
1 Academy of Management Review
(6.13)
Administrative Science Quarterly
(9.83)
Administrative Science Quarterly
(89.77)
2 Academy of Management Journal
(6.08)
Academy of Management Journal
(9.79)
Academy of Management Review
(66.56)
3 Journal of Retailing
(4.10)
Academy of Management Review
(8.91)
Journal of Marketing
(53.84)
4 Journal of Marketing
(3.60)
Journal of Marketing
(8.72)
Academy of Management Journal
(50.48)
5 Strategic Management Journal
(3.34)
Strategic Management Journal
(7.07)
Strategic Management Journal
(46.01)
6 Marketing Science
(3.31)
Journal of Management
(6.35)
Journal of Consumer Research
(39.30)
7 Journal of Management
(3.08)
Marketing Science
(6.10 )
Journal of Marketing Research
(36.34)
8 Journal of International Business Studies
(2.99)
Journal of Consumer Psychology
(5.30)
Journal of Management
(28.14)
9 Administrative Science Quarterly
(2.85)
Journal of International Business Studies
(5.05)
Sloan Management Review
(20.38)
10 Journal of Consumer Psychology
(2.84)
Journal of Organizational Behavior
(4.85)
Marketing Science
(20.36)

Author Affiliation Index (AAI); the pattern of authorship/coauthorship across journals

Although this recent study by Chen en Huan (Journal of Corporate Finance 2007) is focused on the field of Finance, the concept of AAI is valuable for all fields of management research. The AAI is calculated as the ratio of articles authored by faculty at the world’s top 80 finance programs divided by the total number of articles by all authors. It provides provides academics with a credible alternative measurement of journal quality, in ddition to the traditional survey-based and citation-based journal ratings.

Abstract:

In this paper we use a new method to rank finance journals and study the pattern of authorship/coauthorship
across journals. Defined as the ratio of articles authored by faculty at the world’s top 80 finance
programs to the total number of articles by all authors, the Author Affiliation Index is a cost-effective and
intuitively easy-to-understand approach to journal rankings. Forty-one finance journals are ranked
according to this index. If properly constructed, the Author Affiliation Index provides an easy and credible
way to supplement the existing journal ranking methods. Our ranking system reveals the journal–researcher
clientele, and we find that collaboration (co-authoring) between faculty within elite programs exists only in
top-tier and near-top-tier journals. Publications in lower-tier journals by researchers of elite programs are
driven by their co-authors. Collaboration between faculty in elite and non-elite programs, however, is more
prevalent than that within elite programs across all tiers of journals. Co-authorship among top 80 programs,
nevertheless, is more common in top-tier journals, while co-authorship between top 80 and other programs
is more dominant in lower-ranked journals.

Measures of Value of a Journal Beyond the Impact Factor

Anita Coleman gives a nice overview of possible multi-dimensional measure of the value of academic journals next to the traditional impact factor. To measure the value of a journals Coleman first selects three measures, namely journal attraction power, author associativity, and journal consumption power; she redefines two of them as journal measures of affinity (the proportion of foreign authors) and associativity (the amount of collaboration), and calculate these as objective indicators of journal value. To illustrate the multi-dimensional identity of the value of a journal I selected the following list of possible measures from Coleman’s article (in alfabetical order):

  1. Acceptance and Rejection rates
  2. Adjusted Impact Factor,
  3. Article Quality,
  4. Author Reputation Score,
  5. Average Ranking Position,
  6. Circulation size,
  7. Citation Rate,
  8. Citing Half-Life,
  9. Degree of specialization,
  10. Disciplinary Impact Factor,
  11. Editorial board,
  12. Editorial standards,
  13. Immediacy Index,
  14. Impact Factor,
  15. Importance Index
  16. Influence Weight,
  17. Journal Age,
  18. Journal origin and orientation,
  19. Mean Response Time,
  20. Popularity Factor
  21. Readership
  22. References per Paper,
  23. Reprint distribution,
  24. School Reputation Score,
  25. Self-citedness,
  26. Standing,
  27. Type of research covered,
  28. Uncitedness,

RSS Business School News

  • An error has occurred; the feed is probably down. Try again later.
Locations of Site Visitors
%d bloggers like this: