Teach Mode [Updates, Discussion & Feedback Thread]

[PaulBacon]

It would be very useful if teach mode can record and replay the use of the TAB key, like it currently does with the Enter key. That wouid make teaching navigation for a lot of forms on pages a lot more reliable imo.

[FingerTaps]

Looking forward to it… been bet testing… update in on my r1…

[msdin]

Hi, when using teach mode via the r1, can we have an option to not speak every step? It’s ok to display each step on screen but I don’t need to hear every step repeated every time. I just want to hear the result.

Thanks!

[msdin]

I created a lesson that navigates to a web site, searches for a specific product and reads the current price. I had to fiddle with it a bit though because it seems that by default it replaces the product name with what is provided in the prompt. What I really wanted was just to have it use the specific product I taught it with every time.

For example, say I want to be able to check the price of a specific microwave oven at a specific web site so I can tell when it goes on sale. So I use teach mode to teach it how to find the specific model I am looking for and read the price. On the r1 I want to just be able to say “use teach mode to check the price of the microwave”. However, it seems like it will instead replace the keyword I used to search with while teaching (e.g. the model number) with what was said in the prompt (e.g. “microwave”). And then it fails.

I managed to get it to work by adding an annotation at the enter keyword step that says to always use the model number for the search (specifying it exactly).

[simon]

I brought this up before, and it seemed like other people appreciated it speaking out the steps! So we may need to run a wider poll on that at some point soon!

[nzwaneveld]

Partially failed attempt to create a lesson that adds a task on Google.
The lesson works properly when I use a real date, like “November 23, 2024”, but it has issues when I tell it that the date is “tomorrow”.

• Base URL: “https://calendar.google.com/”
• Name of the lesson: “Add Task to Google Calendar”
• Task description: “Add [title] to my tasks for [time] on [date]”

https://hole.rabbit.tech/teach-mode/lessons/e1915e09d09369e5639a3839/replay

Additional note: I don’t log in as part of the lesson. I am using the Google cookies that I added using the cookie jar.

[nzwaneveld]

Why not make speaking out each step an option instead of default? Right now it speaks out each step by default. Let it speak out steps when you add it to the prompt.

I tried adding “do not speak out each step” to the prompt, but that doesn’t work.

[nzwaneveld]

Failed attempt to summarize headlines from a Dutch news website in the Dutch language using Teach mode.

• Base URL: “https://nos.nl/nieuws/laatste”
• Name of the lesson: “Summarize latest headlines on NOS news”
• Task description: Check the latest news on [news source] and summarize the news in the [language] language.

Lesson URL: https://hole.rabbit.tech/teach-mode/lessons/69d88f7899d716a558100eac/replay

[clel]

@simon Replay fails at the cookie banner: https://hole.rabbit.tech/teach-mode/lessons/d2eb608034b5dfaa901d5dc0/replay.

I had several problems with cookie banners in general (LAM playground as well).

Apart from that: I am happy that cookie jar is released, as now it is finally possible to see all signed in services and have some control over it (as well as the sharing between LAM and teach mode, which is a needed feature for use in production).

[clel]

Next one with cookie banner problems: https://hole.rabbit.tech/teach-mode/lessons/e5dfdcb746550aaa8637205c/replay.

This one works on replay. However, the cookie banner is never closed. This is likely due to the action never being recognized in the log when recording. The website (the-decoder.de) also has the cookie banner problem in LAM playground.

[Trev]

Loving what’s happening so no complaints but very curious if teach mode will extend to Windows functions and pc and phone apps?
Example, I am on the same ZOOM call every week, I’d love to be able to tell rabbit to Start my zoom call, and it open Zoom, select the ID, enter my password and join the meeting. That’s just one of the many things I’d love to see my Rabbit help me do on a daily basis.

[Trev]

I just created a teach mode task for Rabbit to read me the list of the latest obituaries from my local newspaper, and IT WORKED perfectly, although a bit slow beginning the task. But I would like for Rabbit to show the picture of the deceased on the Rabbit screen. Is this possible? If so, how can I teach Rabbit to do this?

[PaulBacon]

What often works is if you prompt “always add photo’s with your future responses”

And then proceed with your query.

[leahcim]

Hey everyone, I have summarized some ideas here that could certainly open up endless possibilities:

Update and edit of learned things in Teach Mode

It would be really great if you had the option to edit a lesson created in Teach Mode later. That would be really helpful, especially for more complex things, so that you don’t always have to do everything again. I imagine that you should confirm step by step whether an action is still current and if not, you would have the option to simply change it. In principle, you can click through each step and if something is no longer correct, just change it and that’s it, save it. In t…

Targeted image description in teach mode

It would be wonderful if you could use targeted image description in Teach mode. What is that? Very simple… If you could select an image on a website in Teach mode and instruct Teach mode (via prompt) to look for something specific in the image and describe it or use it for something else. For example, fill in an input field or whatever… The possibilities would be, as always, almost unlimited.

Output what r1 see in Teach Mode

We need a way to use the output of what r1 sees in Teach Mode. This would open up completely new possibilities, for example r1 could then also make essential purchases if you show r1 the contents of a refrigerator. Ideally this would be possible in combination with the contents of the favorites list of an online shop, but for this to happen there would also have to be the possibility in Teach Mode for r1 to compare what it has recognized in the picture with the favorites in the list and then s…

Alternative Actions in Teach Mode

For some websites it is necessary that the teach mode should also be able to perform alternative actions (clicks on alternative buttons). This means that if, after an action, e.g. entering a location, the location is present multiple times or in a different way than taught, the teach mode is then also able to execute a learned alternative fallback action so that the task can still be completed successfully.

We need the Rabbit-Eye in Teach Mode

I’ve noticed that there are unfortunately a lot of hurdles on the internet for 1r, I would love to teach my r1 how to solve Captcha, because that’s really a problem, at least currently. @simon [Adam Sandler Comedy GIF by Netflix Is a Joke] I have an idea, but it would take a lot of time and it would be good to develop it as a general solution, so it’s not necessarily a community job. But you can still do so many great things with it! [Jan Böhmermann Reaction GIF by ZDF Magazin Royale]

[simon]

Logged this one with the team.

[simon]

For future reference!

When posting details of a failed lesson, please try to use this format:

Name of Lesson:
Task Description:
Lesson URL:

This will help me to log them faster!

[Siriap]

@simon, is it possible to add some rules in the log regarding the readback on the R1 device?

IE the lesson I created, I don’t need the R1 to talk me through the step by step. Just the final confirmation is fine.

[j.ursoleo]

Hey, so after trying a bunch of times, you were right. I’m only having these issues when the restaurant is closed.

Thanks for looking into it, Rabbit team!

[blanc]

I agree with this.
Also, if Rabbit could cross learn and combine similar lessons shared by each and every one of us within the community list, it might accelerate the rate of success and minimize the chance of failure for a particular action/request.
Moreover, the model could then learn from our failures or inputs and in return teach us back on how to write better annotations or improve our actions for each step.
This would be awesome and could introduce a more engaged learning environment.

Think of it like a Claude Project for LAM collaborations:

Anthropic Wants Its Claude AI to Be Your New Project Manager

[AggressiveBull]

Teach URL - https://hole.rabbit.tech/teach-mode/lessons/5bd5a8d46c2acf3a468f442a/replay

I am trying to get a summary of a pdf document on a specific URL. The result keeps saying that the HTML header is empty. Summary - go to a specific website, click a link to open a PDF, read the PDF and provide a summary.