Teach Mode [Updates, Discussion & Feedback Thread]

[leahcim]

Hey everyone, I have summarized some ideas here that could certainly open up endless possibilities:

Update and edit of learned things in Teach Mode

It would be really great if you had the option to edit a lesson created in Teach Mode later. That would be really helpful, especially for more complex things, so that you don’t always have to do everything again. I imagine that you should confirm step by step whether an action is still current and if not, you would have the option to simply change it. In principle, you can click through each step and if something is no longer correct, just change it and that’s it, save it. In t…

Targeted image description in teach mode

It would be wonderful if you could use targeted image description in Teach mode. What is that? Very simple… If you could select an image on a website in Teach mode and instruct Teach mode (via prompt) to look for something specific in the image and describe it or use it for something else. For example, fill in an input field or whatever… The possibilities would be, as always, almost unlimited.

Output what r1 see in Teach Mode

We need a way to use the output of what r1 sees in Teach Mode. This would open up completely new possibilities, for example r1 could then also make essential purchases if you show r1 the contents of a refrigerator. Ideally this would be possible in combination with the contents of the favorites list of an online shop, but for this to happen there would also have to be the possibility in Teach Mode for r1 to compare what it has recognized in the picture with the favorites in the list and then s…

Alternative Actions in Teach Mode

For some websites it is necessary that the teach mode should also be able to perform alternative actions (clicks on alternative buttons). This means that if, after an action, e.g. entering a location, the location is present multiple times or in a different way than taught, the teach mode is then also able to execute a learned alternative fallback action so that the task can still be completed successfully.

We need the Rabbit-Eye in Teach Mode

I’ve noticed that there are unfortunately a lot of hurdles on the internet for 1r, I would love to teach my r1 how to solve Captcha, because that’s really a problem, at least currently. @simon [Adam Sandler Comedy GIF by Netflix Is a Joke] I have an idea, but it would take a lot of time and it would be good to develop it as a general solution, so it’s not necessarily a community job. But you can still do so many great things with it! [Jan Böhmermann Reaction GIF by ZDF Magazin Royale]