I’ve been testing Hotseat - an AI-based Q&A for legal - with a small set of lawyers to see whether AI can produce high-caliber answers to legal questions. Our prelimenary results suggest you can suprisingly good answers, but you have to ask AI the right questions.
The initial feedback I received was that the Hotseat answers “to the side” of a question. A careful analysis revealed that the question people asked the AI didn’t have all the correct details, and the AI went ahead and implicitly guessed them and answered the question with these hidden assumptions. Yet the guesses weren’t correct.
I was perplexed since I was working with lawyers - whose core job function is questioning and answering, and language precision is the name of the game. Upon some soul-searching, I realized that if a lawyer had a perfect recollection of a 150+ pages-long regulation in their head to align a question with the intricacies of the law perfectly, they wouldn’t need an AI to help find an answer.
So, we have the stage set: we have a technology prone to silent assumptions and a information gap between the user and the AI that opens up a space for making these silent assumptions. What do we do?
Question refinement
We decided to make the illegible process of hidden assumptions legible. We ask AI to rate the clarity of the user’s legal question in the context of a legal document, and if the question has too many gaps, we give feedback to the user instead of answering “to the side.”
Here’s an excerpt from an example of a feedback email a user (not a lawyer) received upon asking a question about the EU’s Digital Services Act (165 pages long):
Your original question was:
I’m running an ambitious bootstrapped company called Justjoin.it - a job board for programmers. I used Facebook and my personal to talk about company building, and our successes. Facebook blocked my profile claiming I’m using it for advertising by posting from my personal profile. Can Digital Services Act help me unblock my profile?
Followup Questions and Example Answers:
- Have you received any formal communication or notice from Facebook outlining the specific reasons for the block and your rights to contest the decision?
- 📝 Example: Facebook provided a general notice stating the block is due to using the personal profile for advertising.
- What content did you share on your personal profile related to your company?
- 📝 Example: I shared updates about the company’s milestones, products, services, and events.
- Is there a distinction between your personal posts and those that are strictly promotional or advertising for your company?
- 📝 Example: Most posts were narrative and personal updates about the company, but some could be construed as promotional.
- Have you attempted to use Facebook’s internal process for appealing the block on your account?
- 📝 Example: I have tried appealing through Facebook’s internal process but have not received a resolution.
- Do you operate within the European Union where the DSA would be applicable?
- 📝 Example: Yes, my company and I are based in the European Union.
Revised Question with details from example answers incorporated:
Based on these potential details, your refined question might be:
As the owner of Justjoin.it, a tech startup based in the European Union, I have been using my personal Facebook profile for sharing both personal reflections and promotional content about my company. Facebook has blocked my profile, alleging that the activity constitutes advertising. Despite attempting to appeal the decision through Facebook’s internal process, my profile remains blocked. Does the Digital Services Act offer any recourse for me to have my profile unblocked?
(the bolded parts are answers to example questions)
Please look over this rephrased question and adjust it to fit your specific circumstances. With this information, we can offer you the most accurate and pertinent legal advice for your particular situation.
Hotseatai - legal brainstorming and answers at warp-speed
This is a long email and a few things are happening here, so let me dissect it. We quote the original question, which asks whether the EU’s Digital Services Act can be used as leverage against Facebook. The original question was asked by an entrepreneur who had just a hunch.
We provide a list of follow-up questions whose answers would fill in the missing details. The follow-up questions are generated by analyzing the user question and contrasting it against the text of the Digital Services Act. We also provide example answers to follow-up questions to help the user see what we need. Notice that we point out the obvious but hidden assumption: the Digital Services Act applies just to the EU but the original question didn’t mention where the company is based.
Finally, we rewrite the original user question as it had the details we seek filled in with example answers. We bold the parts of the revised question corresponding to example answers to make the revision easy to glance at. The user will need to take this illustration and adapt it to its actual circumstances. The feedback email aims not to leave the user staring at a blank question box and guide the user towards success - a satisfying answer that fits his (her) situation perfectly.
When I showed this to our beta testers, the feedback I got was that this is immensely powerful. They pointed out that the AI-guided question refinement encapsulates a very time-consuming and rather unsatisfying process of learning about the details of a lawyer client’s case, i.e., bridging the information gap between a person representing a business and the lawyer. The state-of-the-art processes are either video calls or long email chains.
Closing
You can check out the refined version of the example question and its fully AI-made answer here. As a reminder, it’s based on a complex analysis of a 165 pages long document.
Armed with questions’ refinement, we’re busy testing whether the frontier AI has the reasoning abilities to deliver high-caliber legal analysis. If you’d like to try out Hotseat yourself, get your answers in minutes, and have your legal research kept private, you can sign up here. Since we’re at larval stage of product development, we offer money-back guarantee in case you’re not happy with the results.
We currently support a few tech regulations that are coming into force and affect large number of internet businesses with the EU presence: Digital Services Act, Digital Markets Act, and AI Act. If you have pressing legal questions in other areas and would love to see other regulations covered, contact me at greg@hotseatai.com.