Proposal

Use the same artificially intelligent attorneys that are gaining popularity as a way to save law firms money to do the initial grading for scores for the California State Bar Exam, thereby significantly shortening the wait time for results (obviously, this could be done in other states as well).

How Things Work Right Now

I understand that grading thousands of applicants’ exams must be no easy task. To read about the process, see this article: The California Bar Exam Grading Process. The main point: it takes four months for July applicants to learn their scores, and three months for February applicants to learn their scores. Here’s the thing, according to the State Bar of California Website, Bar examination graders must only, at minimum:

1. Be active members in good standing of The State Bar of California;
2. Have been admitted to The State Bar of California  not less than one year prior to the examination for which the grader is to be selected, and:
3. Have passed the California Bar Examination on the first or second attempt.

Rosie the Robot Will Represent You Now

Companies are already using artificially intelligent robot lawyers to do legal work. In a post just last week week, Above the Law wrote “[artificial intelligence] is paving the way for early stage start-ups like Lawgeex, Beagle, and ROSS (just to name a few), as well as [. . .] Seal Software.” Both Ross and Thompson Retuers (the folks behind Westlaw) are using IBM’s Watson computer to provide artificially intelligent legal research services. In this podcast, a representative from Thompson Reuters discusses a concept product that would be always on and listen in on a firm’s or company’s meetings, and, like Amazon’s Alexa, the product would always be ready to give real-time answers. Except the concept product would give answers to legal questions.

Could Ross, or any of the other artificially intelligent lawyers currently on the market, satisfy the minimum requirements to be a California State Bar Examination grader?

1. Be active members in good standing of The State Bar of California;

The main components are:

• 5 years of education (2 of approved pre-legal education and 3 of approved legal education),
• A background check (known as a “Moral Character determination”), and
• Passing three examinations:

The First-year Law Students’ Examination:
Most lawyers are exempt from this exam, but for those who are not, it includes essay and multiple choice components

The Multistate Professional Responsibility Examination:
The “Ethics” exam is all multiple choice.

The California State Bar Exam:
Most states have a Bar examination, but the California Bar includes essay, multiple choice, and performance test components.

So, could an artificially intelligent lawyer satisfy those requirements?

1. Active Members in Good Standing

• Legal education teaches four things: issue spotting, rule memorization, applying rules to the facts at hand, and concluding. Programming the artificially intelligent lawyer would serve as its legal education. There could easily be a set database of law that all artificially intelligent lawyers are expected to use.
• A log of all of an artificially intelligent lawyer’s previous conclusions could serve as a background check.
• The artificially intelligent lawyer could earn admission to the Bar simply by the programmer loading each of the exams. While humans are typically limited by their ability to access the rules (that is, they have to memorize the rules and there are a lot of them), programs have their own limitations in producing written answers.

Producing Written Answers

Dan Rubins at Legal Robot was kind enough to meet with me to discuss the reality of what turns out to be the hardest part of this problem: producing written answers. I asked Dan whether the currently available technology would be able to process a fact pattern about a few people who go into a building, and proceed to do a bunch of things which just so happen to line up precisely with the elements of burglary, robbery, and battery. Dan proposed that the legal research technology already on the market might be able to process the fact pattern if it were really precisely written, but the current technology’s output would look much more like lists of crimes and statistical likelihood that the fact pattern reflected those crimes having been committed.

He proposed that technology like Narrative Science in combination with adversarial machine learning could potentially produce written answers. Narrative Science writes stories based on data. Adversarial machine learning takes two artificially intelligent programs and tells them to battle each other to produce the most optimal answer. He explained that Narrative Science might be able to take that statistical likelihood data I just mentioned and turn it into something that more resembles an essay response, and that adversarial machine learning could be used to optimize the artificially intelligent lawyer’s solution to the fact pattern. In sum, Dan thinks it’s possible, but the artificially intelligent lawyer isn’t quite available for purchase yet. It would still need to be made.

2. The Artificially Intelligent Lawyer Can Wait Forever

Waiting for one year would give the artificially intelligent lawyer the same advantage as a human lawyer: time to build its database of data. Translated: by waiting a year, the program gets even more opportunities for the programmer to train the program to recognize what a passing answer should look like.

3. I Don’t Like This Requirement

Honestly, I don’t like the last requirement. Why does a grader have to have passed the Bar on the first or second try? Anyway, there are ways around this with regards to an artificially intelligent lawyer. It’s much harder to define who/what is taking the Bar when that who/what is a program.

The Grading

Essay prompts and the performance tests on the Bar are so carefully crafted that there is really no invitation for alternative answers. A simple search for certain key words (and their legal synonyms) would determine whether the applicant identified the correct issues. The Bar grading system would also need a tool designed to grade writing ability in order to prevent applicants from passing the exam merely by typing in keywords. As it is, programs already exist to grade essay writing. ETS’s e-rater can, “automatically detect responses that are off-topic or otherwise anomalous and, therefore, should not be scored.” The e-rater can also “detect presence of a thesis statement, main points, and source use.” Ultimately, by merely combining a program trained to search for a particular set of words, and a program like ETS’s e-rater, the Committee of Bar Examiners could create an artificially intelligent grader for the writing portions of the Bar Examination.

Dan also told me that grading could be another good place to apply adversarial machine learning. Artificially intelligent programs have the potential to learn through programmers purposefully trying to trick the programs. If people purposefully try to write answers that will register as correct answers but are actually very poorly written answers with great keywords, those answers simply give the artificially intelligent grader more opportunities to learn and build its knowledge base.

The Human Touch

I get it, there are people (lots, I’m willing to bet) that want a human to grade their exam. They want a human to grade their exam even though a human could have a glass of wine and be more lenient, or have a rough day at work and be the reason you fail the Bar exam by a handful of points. So here’s my compromise: the program has to pass every single Bar exam it grades, and the program will only provide the first set of grades. Just like it is now, if you score above a 1390, a second grader will step in and review your essay and performance test scores (this time, a human grader).

Artificially Intelligent Lawyers Would Improve the Admission Process For the Humans

With any luck at all, this would cut down on the wait time, and programs could actually help to remove the psychological torture element of the Bar exam: the multiple month wait.

 

—

A special thanks to my University of San Francisco School of Law Class of 2015 classmates for inspiring this post. They voted me “most likely to invent a program to take law school exams for you.” Instead, I decided it would be cooler to invent a program to grade the exams 🙂

Update: I realized I was remiss in forgetting to thank Dan Rubins, CEO and co-founder of Legal Robot, for taking the time to meet with me. Legal Robot is currently going through a funding round and can be reached at hello@legalrobot.com. For only $15, his App lets you take a picture of a contract and the App will tell you in plain language what each section of the contract says. Legal Robot is special because of its commitment to privacy: all of the contract analysis happens on device, and the database the App has used to build its Contract Law knowledge is a public database. Translation: when you sign into your Legal Robot account, the Legal Robot you are working with has the capability to learn from each new document you introduce to it, but it is not sharing that new found knowledge with any other customer’s Legal Robot.