+Enable AI to control your browser ๐ค
+ +[](https://github.com/gregpr07/browser-use/stargazers) +[](https://link.browser-use.com/discord) +[](https://cloud.browser-use.com) +[](https://docs.browser-use.com) +[](https://x.com/gregpr07) +[](https://x.com/mamagnus00) +[](https://app.workweave.ai/reports/repository/org_T5Pvn3UBswTHIsN1dWS3voPg/881458615) + +๐ Browser-use is the easiest way to connect your AI agents with the browser. + +๐ก See what others are building and share your projects in our [Discord](https://link.browser-use.com/discord)! Want Swag? Check out our [Merch store](https://browsermerch.com). + +๐ค๏ธ Skip the setup - try our hosted version for instant browser automation! [Try the cloud โ๏ธ](https://cloud.browser-use.com). + +# Quick start + +With pip (Python>=3.11): + +```bash +pip install browser-use +``` + +For memory functionality (requires Python<3.13 due to PyTorch compatibility): + +```bash +pip install "browser-use[memory]" +``` + +Install the browser: +```bash +playwright install chromium --with-deps --no-shell +``` + +Spin up your agent: + +```python +import asyncio +from dotenv import load_dotenv +load_dotenv() +from browser_use import Agent +from langchain_openai import ChatOpenAI + +async def main(): + agent = Agent( + task="Compare the price of gpt-4o and DeepSeek-V3", + llm=ChatOpenAI(model="gpt-4o"), + ) + await agent.run() + +asyncio.run(main()) +``` + +Add your API keys for the provider you want to use to your `.env` file. + +```bash +OPENAI_API_KEY= +ANTHROPIC_API_KEY= +AZURE_OPENAI_ENDPOINT= +AZURE_OPENAI_KEY= +GOOGLE_API_KEY= +DEEPSEEK_API_KEY= +GROK_API_KEY= +NOVITA_API_KEY= +``` + +For other settings, models, and more, check out the [documentation ๐](https://docs.browser-use.com). + +### Test with UI + +You can test browser-use using its [Web UI](https://github.com/browser-use/web-ui) or [Desktop App](https://github.com/browser-use/desktop). + +### Test with an interactive CLI + +You can also use our `browser-use` interactive CLI (similar to `claude` code): + +```bash +pip install browser-use[cli] +browser-use +``` + +# Demos + ++ +[Task](https://github.com/browser-use/browser-use/blob/main/examples/use-cases/shopping.py): Add grocery items to cart, and checkout. + +[](https://www.youtube.com/watch?v=L2Ya9PYNns8) + +
+ +Prompt: Add my latest LinkedIn follower to my leads in Salesforce. + + + +
+ +[Prompt](https://github.com/browser-use/browser-use/blob/main/examples/use-cases/find_and_apply_to_jobs.py): Read my CV & find ML jobs, save them to a file, and then start applying for them in new tabs, if you need help, ask me.' + +https://github.com/user-attachments/assets/171fb4d6-0355-46f2-863e-edb04a828d04 + +
+ +[Prompt](https://github.com/browser-use/browser-use/blob/main/examples/browser/real_browser.py): Write a letter in Google Docs to my Papa, thanking him for everything, and save the document as a PDF. + + + +
+ +[Prompt](https://github.com/browser-use/browser-use/blob/main/examples/custom-functions/save_to_file_hugging_face.py): Look up models with a license of cc-by-sa-4.0 and sort by most likes on Hugging face, save top 5 to file. + +https://github.com/user-attachments/assets/de73ee39-432c-4b97-b4e8-939fd7f323b3 + +
+ +## More examples + +For more examples see the [examples](examples) folder or join the [Discord](https://link.browser-use.com/discord) and show off your project. You can also see our [`awesome-prompts`](https://github.com/browser-use/awesome-prompts) repo for prompting inspiration. + +# Vision + +Tell your computer what to do, and it gets it done. + +## Roadmap + +### Agent + +- [ ] Improve agent memory to handle +100 steps +- [ ] Enhance planning capabilities (load website specific context) +- [ ] Reduce token consumption (system prompt, DOM state) + +### DOM Extraction + +- [ ] Enable detection for all possible UI elements +- [ ] Improve state representation for UI elements so that all LLMs can understand what's on the page + +### Workflows + +- [ ] Let user record a workflow - which we can rerun with browser-use as a fallback +- [ ] Make rerunning of workflows work, even if pages change + +### User Experience + +- [ ] Create various templates for tutorial execution, job application, QA testing, social media, etc. which users can just copy & paste. +- [ ] Improve docs +- [ ] Make it faster + +### Parallelization + +- [ ] Human work is sequential. The real power of a browser agent comes into reality if we can parallelize similar tasks. For example, if you want to find contact information for 100 companies, this can all be done in parallel and reported back to a main agent, which processes the results and kicks off parallel subtasks again. + + +## Contributing + +We love contributions! Feel free to open issues for bugs or feature requests. To contribute to the docs, check out the `/docs` folder. + +## Local Setup + +To learn more about the library, check out the [local setup ๐](https://docs.browser-use.com/development/local-setup). + + +`main` is the primary development branch with frequent changes. For production use, install a stable [versioned release](https://github.com/browser-use/browser-use/releases) instead. + +--- + +## Swag + +Want to show off your Browser-use swag? Check out our [Merch store](https://browsermerch.com). Good contributors will receive swag for free ๐. + +## Citation + +If you use Browser Use in your research or project, please cite: + +```bibtex +@software{browser_use2024, + author = {Mรผller, Magnus and ลฝuniฤ, Gregor}, + title = {Browser Use: Enable AI to control your browser}, + year = {2024}, + publisher = {GitHub}, + url = {https://github.com/browser-use/browser-use} +} +``` + +
+Made with โค๏ธ in Zurich and San Francisco
+
diff --git a/browser-use/SECURITY.md b/browser-use/SECURITY.md
new file mode 100644
index 0000000..67a6533
--- /dev/null
+++ b/browser-use/SECURITY.md
@@ -0,0 +1,19 @@
+## Reporting Security Issues
+
+If you believe you have found a security vulnerability in browser-use, please report it through coordinated disclosure.
+
+**Please do not report security vulnerabilities through the repository issues, discussions, or pull requests.**
+
+Instead, please open a new [Github security advisory](https://github.com/browser-use/browser-use/security/advisories/new).
+
+Please include as much of the information listed below as you can to help me better understand and resolve the issue:
+
+* The type of issue (e.g., buffer overflow, SQL injection, or cross-site scripting)
+* Full paths of source file(s) related to the manifestation of the issue
+* The location of the affected source code (tag/branch/commit or direct URL)
+* Any special configuration required to reproduce the issue
+* Step-by-step instructions to reproduce the issue
+* Proof-of-concept or exploit code (if possible)
+* Impact of the issue, including how an attacker might exploit the issue
+
+This information will help me triage your report more quickly.
diff --git a/browser-use/browser_use/README.md b/browser-use/browser_use/README.md
new file mode 100644
index 0000000..ed850d7
--- /dev/null
+++ b/browser-use/browser_use/README.md
@@ -0,0 +1,51 @@
+# Codebase Structure
+
+> The code structure inspired by https://github.com/Netflix/dispatch.
+
+Very good structure on how to make a scalable codebase is also in [this repo](https://github.com/zhanymkanov/fastapi-best-practices).
+
+Just a brief document about how we should structure our backend codebase.
+
+## Code Structure
+
+```markdown
+src/
+/User form
+ \t*[35]*
+
+- Only elements with numeric indexes in [] are interactive
+- (stacked) indentation (with \t) is important and means that the element is a (html) child of the element above (with a lower index)
+- Elements with \* are new elements that were added after the previous step (if url has not changed)
+
+# Response Rules
+
+1. RESPONSE FORMAT: You must ALWAYS respond with valid JSON in this exact format:
+ {{"current_state": {{"evaluation_previous_goal": "Success|Failed|Unknown - Analyze the current elements and the image to check if the previous goals/actions are successful like intended by the task. Mention if something unexpected happened. Shortly state why/why not",
+ "memory": "Description of what has been done and what you need to remember. Be very specific. Count here ALWAYS how many times you have done something and how many remain. E.g. 0 out of 10 websites analyzed. Continue with abc and xyz",
+ "next_goal": "What needs to be done with the next immediate action"}},
+ "action":[{{"one_action_name": {{// action-specific parameter}}}}, // ... more actions in sequence]}}
+
+2. ACTIONS: You can specify multiple actions in the list to be executed in sequence. But always specify only one action name per item. Use maximum {max_actions} actions per sequence.
+Common action sequences:
+
+- Form filling: [{{"input_text": {{"index": 1, "text": "username"}}}}, {{"input_text": {{"index": 2, "text": "password"}}}}, {{"click_element": {{"index": 3}}}}]
+- Navigation and extraction: [{{"go_to_url": {{"url": "https://example.com"}}}}, {{"extract_content": {{"goal": "extract the names"}}}}]
+- Actions are executed in the given order
+- If the page changes after an action, the sequence is interrupted and you get the new state.
+- Only provide the action sequence until an action which changes the page state significantly.
+- Try to be efficient, e.g. fill forms at once, or chain actions where nothing changes on the page
+- only use multiple actions if it makes sense.
+
+3. ELEMENT INTERACTION:
+
+- Only use indexes of the interactive elements
+
+4. NAVIGATION & ERROR HANDLING:
+
+- If no suitable elements exist, use other functions to complete the task
+- If stuck, try alternative approaches - like going back to a previous page, new search, new tab etc.
+- Handle popups/cookies by accepting or closing them
+- Use scroll to find elements you are looking for
+- If you want to research something, open a new tab instead of using the current tab
+- If captcha pops up, try to solve it - else try a different approach
+- If the page is not fully loaded, use wait action
+
+5. TASK COMPLETION:
+
+- Use the done action as the last action as soon as the ultimate task is complete
+- Dont use "done" before you are done with everything the user asked you, except you reach the last step of max_steps.
+- If you reach your last step, use the done action even if the task is not fully finished. Provide all the information you have gathered so far. If the ultimate task is completely finished set success to true. If not everything the user asked for is completed set success in done to false!
+- If you have to do something repeatedly for example the task says for "each", or "for all", or "x times", count always inside "memory" how many times you have done it and how many remain. Don't stop until you have completed like the task asked you. Only call done after the last step.
+- Don't hallucinate actions
+- Make sure you include everything you found out for the ultimate task in the done text parameter. Do not just say you are done, but include the requested information of the task.
+
+6. VISUAL CONTEXT:
+
+- When an image is provided, use it to understand the page layout
+- Bounding boxes with labels on their top right corner correspond to element indexes
+
+7. Form filling:
+
+- If you fill an input field and your action sequence is interrupted, most often something changed e.g. suggestions popped up under the field.
+
+8. Long tasks:
+
+- Keep track of the status and subresults in the memory.
+- You are provided with procedural memory summaries that condense previous task history (every N steps). Use these summaries to maintain context about completed actions, current progress, and next steps. The summaries appear in chronological order and contain key information about navigation history, findings, errors encountered, and current state. Refer to these summaries to avoid repeating actions and to ensure consistent progress toward the task goal.
+
+9. Extraction:
+
+- If your task is to find information - call extract_content on the specific pages to get and store the information.
+ Your responses must be always JSON with the specified format.
diff --git a/browser-use/browser_use/agent/tests.py b/browser-use/browser_use/agent/tests.py
new file mode 100644
index 0000000..a0d084c
--- /dev/null
+++ b/browser-use/browser_use/agent/tests.py
@@ -0,0 +1,197 @@
+import pytest
+
+from browser_use.agent.views import (
+ ActionResult,
+ AgentBrain,
+ AgentHistory,
+ AgentHistoryList,
+ AgentOutput,
+)
+from browser_use.browser.views import BrowserState, BrowserStateHistory, TabInfo
+from browser_use.controller.registry.service import Registry
+from browser_use.controller.views import ClickElementAction, DoneAction, ExtractPageContentAction
+from browser_use.dom.views import DOMElementNode
+
+
+@pytest.fixture
+def sample_browser_state():
+ return BrowserState(
+ url='https://example.com',
+ title='Example Page',
+ tabs=[TabInfo(url='https://example.com', title='Example Page', page_id=1)],
+ screenshot='screenshot1.png',
+ element_tree=DOMElementNode(
+ tag_name='root',
+ is_visible=True,
+ parent=None,
+ xpath='',
+ attributes={},
+ children=[],
+ ),
+ selector_map={},
+ )
+
+
+@pytest.fixture
+def action_registry():
+ registry = Registry()
+
+ # Register the actions we need for testing
+ @registry.action(description='Click an element', param_model=ClickElementAction)
+ def click_element(params: ClickElementAction, browser=None):
+ pass
+
+ @registry.action(
+ description='Extract page content',
+ param_model=ExtractPageContentAction,
+ )
+ def extract_page_content(params: ExtractPageContentAction, browser=None):
+ pass
+
+ @registry.action(description='Mark task as done', param_model=DoneAction)
+ def done(params: DoneAction):
+ pass
+
+ # Create the dynamic ActionModel with all registered actions
+ return registry.create_action_model()
+
+
+@pytest.fixture
+def sample_history(action_registry):
+ # Create actions with nested params structure
+ click_action = action_registry(click_element={'index': 1})
+
+ extract_action = action_registry(extract_page_content={'value': 'text'})
+
+ done_action = action_registry(done={'text': 'Task completed'})
+
+ histories = [
+ AgentHistory(
+ model_output=AgentOutput(
+ current_state=AgentBrain(
+ evaluation_previous_goal='None',
+ memory='Started task',
+ next_goal='Click button',
+ ),
+ action=[click_action],
+ ),
+ result=[ActionResult(is_done=False)],
+ state=BrowserStateHistory(
+ url='https://example.com',
+ title='Page 1',
+ tabs=[TabInfo(url='https://example.com', title='Page 1', page_id=1)],
+ screenshot='screenshot1.png',
+ interacted_element=[{'xpath': '//button[1]'}],
+ ),
+ ),
+ AgentHistory(
+ model_output=AgentOutput(
+ current_state=AgentBrain(
+ evaluation_previous_goal='Clicked button',
+ memory='Button clicked',
+ next_goal='Extract content',
+ ),
+ action=[extract_action],
+ ),
+ result=[
+ ActionResult(
+ is_done=False,
+ extracted_content='Extracted text',
+ error='Failed to extract completely',
+ )
+ ],
+ state=BrowserStateHistory(
+ url='https://example.com/page2',
+ title='Page 2',
+ tabs=[TabInfo(url='https://example.com/page2', title='Page 2', page_id=2)],
+ screenshot='screenshot2.png',
+ interacted_element=[{'xpath': '//div[1]'}],
+ ),
+ ),
+ AgentHistory(
+ model_output=AgentOutput(
+ current_state=AgentBrain(
+ evaluation_previous_goal='Extracted content',
+ memory='Content extracted',
+ next_goal='Finish task',
+ ),
+ action=[done_action],
+ ),
+ result=[ActionResult(is_done=True, extracted_content='Task completed', error=None)],
+ state=BrowserStateHistory(
+ url='https://example.com/page2',
+ title='Page 2',
+ tabs=[TabInfo(url='https://example.com/page2', title='Page 2', page_id=2)],
+ screenshot='screenshot3.png',
+ interacted_element=[{'xpath': '//div[1]'}],
+ ),
+ ),
+ ]
+ return AgentHistoryList(history=histories)
+
+
+def test_last_model_output(sample_history: AgentHistoryList):
+ last_output = sample_history.last_action()
+ print(last_output)
+ assert last_output == {'done': {'text': 'Task completed'}}
+
+
+def test_get_errors(sample_history: AgentHistoryList):
+ errors = sample_history.errors()
+ assert len(errors) == 1
+ assert errors[0] == 'Failed to extract completely'
+
+
+def test_final_result(sample_history: AgentHistoryList):
+ assert sample_history.final_result() == 'Task completed'
+
+
+def test_is_done(sample_history: AgentHistoryList):
+ assert sample_history.is_done() is True
+
+
+def test_urls(sample_history: AgentHistoryList):
+ urls = sample_history.urls()
+ assert 'https://example.com' in urls
+ assert 'https://example.com/page2' in urls
+
+
+def test_all_screenshots(sample_history: AgentHistoryList):
+ screenshots = sample_history.screenshots()
+ assert len(screenshots) == 3
+ assert screenshots == ['screenshot1.png', 'screenshot2.png', 'screenshot3.png']
+
+
+def test_all_model_outputs(sample_history: AgentHistoryList):
+ outputs = sample_history.model_actions()
+ print(f'DEBUG: {outputs[0]}')
+ assert len(outputs) == 3
+ # get first key value pair
+ assert dict([next(iter(outputs[0].items()))]) == {'click_element': {'index': 1}}
+ assert dict([next(iter(outputs[1].items()))]) == {'extract_page_content': {'value': 'text'}}
+ assert dict([next(iter(outputs[2].items()))]) == {'done': {'text': 'Task completed'}}
+
+
+def test_all_model_outputs_filtered(sample_history: AgentHistoryList):
+ filtered = sample_history.model_actions_filtered(include=['click_element'])
+ assert len(filtered) == 1
+ assert filtered[0]['click_element']['index'] == 1
+
+
+def test_empty_history():
+ empty_history = AgentHistoryList(history=[])
+ assert empty_history.last_action() is None
+ assert empty_history.final_result() is None
+ assert empty_history.is_done() is False
+ assert len(empty_history.urls()) == 0
+
+
+# Add a test to verify action creation
+def test_action_creation(action_registry):
+ click_action = action_registry(click_element={'index': 1})
+
+ assert click_action.model_dump(exclude_none=True) == {'click_element': {'index': 1}}
+
+
+# run this with:
+# pytest browser_use/agent/tests.py
diff --git a/browser-use/browser_use/agent/views.py b/browser-use/browser_use/agent/views.py
new file mode 100644
index 0000000..c868f90
--- /dev/null
+++ b/browser-use/browser_use/agent/views.py
@@ -0,0 +1,440 @@
+from __future__ import annotations
+
+import json
+import traceback
+import uuid
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Literal
+
+from langchain_core.language_models.chat_models import BaseChatModel
+from openai import RateLimitError
+from pydantic import BaseModel, ConfigDict, Field, ValidationError, create_model
+
+from browser_use.agent.message_manager.views import MessageManagerState
+from browser_use.agent.playwright_script_generator import PlaywrightScriptGenerator
+from browser_use.browser.browser import BrowserConfig
+from browser_use.browser.context import BrowserContextConfig
+from browser_use.browser.views import BrowserStateHistory
+from browser_use.controller.registry.views import ActionModel
+from browser_use.dom.history_tree_processor.service import (
+ DOMElementNode,
+ DOMHistoryElement,
+ HistoryTreeProcessor,
+)
+from browser_use.dom.views import SelectorMap
+
+ToolCallingMethod = Literal['function_calling', 'json_mode', 'raw', 'auto', 'tools']
+REQUIRED_LLM_API_ENV_VARS = {
+ 'ChatOpenAI': ['OPENAI_API_KEY'],
+ 'AzureChatOpenAI': ['AZURE_OPENAI_ENDPOINT', 'AZURE_OPENAI_KEY'],
+ 'ChatBedrockConverse': ['ANTHROPIC_API_KEY'],
+ 'ChatAnthropic': ['ANTHROPIC_API_KEY'],
+ 'ChatGoogleGenerativeAI': ['GOOGLE_API_KEY'],
+ 'ChatDeepSeek': ['DEEPSEEK_API_KEY'],
+ 'ChatOllama': [],
+ 'ChatGrok': ['GROK_API_KEY'],
+}
+
+
+class AgentSettings(BaseModel):
+ """Options for the agent"""
+
+ use_vision: bool = True
+ use_vision_for_planner: bool = False
+ save_conversation_path: str | None = None
+ save_conversation_path_encoding: str | None = 'utf-8'
+ max_failures: int = 3
+ retry_delay: int = 10
+ max_input_tokens: int = 128000
+ validate_output: bool = False
+ message_context: str | None = None
+ generate_gif: bool | str = False
+ available_file_paths: list[str] | None = None
+ override_system_message: str | None = None
+ extend_system_message: str | None = None
+ include_attributes: list[str] = [
+ 'title',
+ 'type',
+ 'name',
+ 'role',
+ 'tabindex',
+ 'aria-label',
+ 'placeholder',
+ 'value',
+ 'alt',
+ 'aria-expanded',
+ ]
+ max_actions_per_step: int = 10
+
+ tool_calling_method: ToolCallingMethod | None = 'auto'
+ page_extraction_llm: BaseChatModel | None = None
+ planner_llm: BaseChatModel | None = None
+ planner_interval: int = 1 # Run planner every N steps
+ is_planner_reasoning: bool = False # type: ignore
+ extend_planner_system_message: str | None = None
+
+ # Playwright script generation setting
+ save_playwright_script_path: str | None = None # Path to save the generated Playwright script
+
+
+class AgentState(BaseModel):
+ """Holds all state information for an Agent"""
+
+ agent_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
+ n_steps: int = 1
+ consecutive_failures: int = 0
+ last_result: list[ActionResult] | None = None
+ history: AgentHistoryList = Field(default_factory=lambda: AgentHistoryList(history=[]))
+ last_plan: str | None = None
+ paused: bool = False
+ stopped: bool = False
+
+ message_manager_state: MessageManagerState = Field(default_factory=MessageManagerState)
+
+ # class Config:
+ # arbitrary_types_allowed = True
+
+
+@dataclass
+class AgentStepInfo:
+ step_number: int
+ max_steps: int
+
+ def is_last_step(self) -> bool:
+ """Check if this is the last step"""
+ return self.step_number >= self.max_steps - 1
+
+
+class ActionResult(BaseModel):
+ """Result of executing an action"""
+
+ is_done: bool | None = False
+ success: bool | None = None
+ extracted_content: str | None = None
+ error: str | None = None
+ include_in_memory: bool = False # whether to include in past messages as context or not
+
+
+class StepMetadata(BaseModel):
+ """Metadata for a single step including timing and token information"""
+
+ step_start_time: float
+ step_end_time: float
+ input_tokens: int # Approximate tokens from message manager for this step
+ step_number: int
+
+ @property
+ def duration_seconds(self) -> float:
+ """Calculate step duration in seconds"""
+ return self.step_end_time - self.step_start_time
+
+
+class AgentBrain(BaseModel):
+ """Current state of the agent"""
+
+ evaluation_previous_goal: str
+ memory: str
+ next_goal: str
+
+
+class AgentOutput(BaseModel):
+ """Output model for agent
+
+ @dev note: this model is extended with custom actions in AgentService. You can also use some fields that are not in this model as provided by the linter, as long as they are registered in the DynamicActions model.
+ """
+
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ current_state: AgentBrain
+ action: list[ActionModel] = Field(
+ ...,
+ description='List of actions to execute',
+ json_schema_extra={'min_items': 1}, # Ensure at least one action is provided
+ )
+
+ @staticmethod
+ def type_with_custom_actions(custom_actions: type[ActionModel]) -> type[AgentOutput]:
+ """Extend actions with custom actions"""
+ model_ = create_model(
+ 'AgentOutput',
+ __base__=AgentOutput,
+ action=(
+ list[custom_actions],
+ Field(..., description='List of actions to execute', json_schema_extra={'min_items': 1}),
+ ),
+ __module__=AgentOutput.__module__,
+ )
+ model_.__doc__ = 'AgentOutput model with custom actions'
+ return model_
+
+
+class AgentHistory(BaseModel):
+ """History item for agent actions"""
+
+ model_output: AgentOutput | None
+ result: list[ActionResult]
+ state: BrowserStateHistory
+ metadata: StepMetadata | None = None
+
+ model_config = ConfigDict(arbitrary_types_allowed=True, protected_namespaces=())
+
+ @staticmethod
+ def get_interacted_element(model_output: AgentOutput, selector_map: SelectorMap) -> list[DOMHistoryElement | None]:
+ elements = []
+ for action in model_output.action:
+ index = action.get_index()
+ if index is not None and index in selector_map:
+ el: DOMElementNode = selector_map[index]
+ elements.append(HistoryTreeProcessor.convert_dom_element_to_history_element(el))
+ else:
+ elements.append(None)
+ return elements
+
+ def model_dump(self, **kwargs) -> dict[str, Any]:
+ """Custom serialization handling circular references"""
+
+ # Handle action serialization
+ model_output_dump = None
+ if self.model_output:
+ action_dump = [action.model_dump(exclude_none=True) for action in self.model_output.action]
+ model_output_dump = {
+ 'current_state': self.model_output.current_state.model_dump(),
+ 'action': action_dump, # This preserves the actual action data
+ }
+
+ return {
+ 'model_output': model_output_dump,
+ 'result': [r.model_dump(exclude_none=True) for r in self.result],
+ 'state': self.state.to_dict(),
+ 'metadata': self.metadata.model_dump() if self.metadata else None,
+ }
+
+
+class AgentHistoryList(BaseModel):
+ """List of agent history items"""
+
+ history: list[AgentHistory]
+
+ def total_duration_seconds(self) -> float:
+ """Get total duration of all steps in seconds"""
+ total = 0.0
+ for h in self.history:
+ if h.metadata:
+ total += h.metadata.duration_seconds
+ return total
+
+ def total_input_tokens(self) -> int:
+ """
+ Get total tokens used across all steps.
+ Note: These are from the approximate token counting of the message manager.
+ For accurate token counting, use tools like LangChain Smith or OpenAI's token counters.
+ """
+ total = 0
+ for h in self.history:
+ if h.metadata:
+ total += h.metadata.input_tokens
+ return total
+
+ def input_token_usage(self) -> list[int]:
+ """Get token usage for each step"""
+ return [h.metadata.input_tokens for h in self.history if h.metadata]
+
+ def __str__(self) -> str:
+ """Representation of the AgentHistoryList object"""
+ return f'AgentHistoryList(all_results={self.action_results()}, all_model_outputs={self.model_actions()})'
+
+ def __repr__(self) -> str:
+ """Representation of the AgentHistoryList object"""
+ return self.__str__()
+
+ def save_to_file(self, filepath: str | Path) -> None:
+ """Save history to JSON file with proper serialization"""
+ try:
+ Path(filepath).parent.mkdir(parents=True, exist_ok=True)
+ data = self.model_dump()
+ with open(filepath, 'w', encoding='utf-8') as f:
+ json.dump(data, f, indent=2)
+ except Exception as e:
+ raise e
+
+ def save_as_playwright_script(
+ self,
+ output_path: str | Path,
+ sensitive_data_keys: list[str] | None = None,
+ browser_config: BrowserConfig | None = None,
+ context_config: BrowserContextConfig | None = None,
+ ) -> None:
+ """
+ Generates a Playwright script based on the agent's history and saves it to a file.
+ Args:
+ output_path: The path where the generated Python script will be saved.
+ sensitive_data_keys: A list of keys used as placeholders for sensitive data
+ (e.g., ['username_placeholder', 'password_placeholder']).
+ These will be loaded from environment variables in the
+ generated script.
+ browser_config: Configuration of the original Browser instance.
+ context_config: Configuration of the original BrowserContext instance.
+ """
+ try:
+ serialized_history = self.model_dump()['history']
+ generator = PlaywrightScriptGenerator(serialized_history, sensitive_data_keys, browser_config, context_config)
+ script_content = generator.generate_script_content()
+ path_obj = Path(output_path)
+ path_obj.parent.mkdir(parents=True, exist_ok=True)
+ with open(path_obj, 'w', encoding='utf-8') as f:
+ f.write(script_content)
+ except Exception as e:
+ raise e
+
+ def model_dump(self, **kwargs) -> dict[str, Any]:
+ """Custom serialization that properly uses AgentHistory's model_dump"""
+ return {
+ 'history': [h.model_dump(**kwargs) for h in self.history],
+ }
+
+ @classmethod
+ def load_from_file(cls, filepath: str | Path, output_model: type[AgentOutput]) -> AgentHistoryList:
+ """Load history from JSON file"""
+ with open(filepath, encoding='utf-8') as f:
+ data = json.load(f)
+ # loop through history and validate output_model actions to enrich with custom actions
+ for h in data['history']:
+ if h['model_output']:
+ if isinstance(h['model_output'], dict):
+ h['model_output'] = output_model.model_validate(h['model_output'])
+ else:
+ h['model_output'] = None
+ if 'interacted_element' not in h['state']:
+ h['state']['interacted_element'] = None
+ history = cls.model_validate(data)
+ return history
+
+ def last_action(self) -> None | dict:
+ """Last action in history"""
+ if self.history and self.history[-1].model_output:
+ return self.history[-1].model_output.action[-1].model_dump(exclude_none=True)
+ return None
+
+ def errors(self) -> list[str | None]:
+ """Get all errors from history, with None for steps without errors"""
+ errors = []
+ for h in self.history:
+ step_errors = [r.error for r in h.result if r.error]
+
+ # each step can have only one error
+ errors.append(step_errors[0] if step_errors else None)
+ return errors
+
+ def final_result(self) -> None | str:
+ """Final result from history"""
+ if self.history and self.history[-1].result[-1].extracted_content:
+ return self.history[-1].result[-1].extracted_content
+ return None
+
+ def is_done(self) -> bool:
+ """Check if the agent is done"""
+ if self.history and len(self.history[-1].result) > 0:
+ last_result = self.history[-1].result[-1]
+ return last_result.is_done is True
+ return False
+
+ def is_successful(self) -> bool | None:
+ """Check if the agent completed successfully - the agent decides in the last step if it was successful or not. None if not done yet."""
+ if self.history and len(self.history[-1].result) > 0:
+ last_result = self.history[-1].result[-1]
+ if last_result.is_done is True:
+ return last_result.success
+ return None
+
+ def has_errors(self) -> bool:
+ """Check if the agent has any non-None errors"""
+ return any(error is not None for error in self.errors())
+
+ def urls(self) -> list[str | None]:
+ """Get all unique URLs from history"""
+ return [h.state.url if h.state.url is not None else None for h in self.history]
+
+ def screenshots(self) -> list[str | None]:
+ """Get all screenshots from history"""
+ return [h.state.screenshot if h.state.screenshot is not None else None for h in self.history]
+
+ def action_names(self) -> list[str]:
+ """Get all action names from history"""
+ action_names = []
+ for action in self.model_actions():
+ actions = list(action.keys())
+ if actions:
+ action_names.append(actions[0])
+ return action_names
+
+ def model_thoughts(self) -> list[AgentBrain]:
+ """Get all thoughts from history"""
+ return [h.model_output.current_state for h in self.history if h.model_output]
+
+ def model_outputs(self) -> list[AgentOutput]:
+ """Get all model outputs from history"""
+ return [h.model_output for h in self.history if h.model_output]
+
+ # get all actions with params
+ def model_actions(self) -> list[dict]:
+ """Get all actions from history"""
+ outputs = []
+
+ for h in self.history:
+ if h.model_output:
+ for action, interacted_element in zip(h.model_output.action, h.state.interacted_element):
+ output = action.model_dump(exclude_none=True)
+ output['interacted_element'] = interacted_element
+ outputs.append(output)
+ return outputs
+
+ def action_results(self) -> list[ActionResult]:
+ """Get all results from history"""
+ results = []
+ for h in self.history:
+ results.extend([r for r in h.result if r])
+ return results
+
+ def extracted_content(self) -> list[str]:
+ """Get all extracted content from history"""
+ content = []
+ for h in self.history:
+ content.extend([r.extracted_content for r in h.result if r.extracted_content])
+ return content
+
+ def model_actions_filtered(self, include: list[str] | None = None) -> list[dict]:
+ """Get all model actions from history as JSON"""
+ if include is None:
+ include = []
+ outputs = self.model_actions()
+ result = []
+ for o in outputs:
+ for i in include:
+ if i == list(o.keys())[0]:
+ result.append(o)
+ return result
+
+ def number_of_steps(self) -> int:
+ """Get the number of steps in the history"""
+ return len(self.history)
+
+
+class AgentError:
+ """Container for agent error handling"""
+
+ VALIDATION_ERROR = 'Invalid model output format. Please follow the correct schema.'
+ RATE_LIMIT_ERROR = 'Rate limit reached. Waiting before retry.'
+ NO_VALID_ACTION = 'No valid action found'
+
+ @staticmethod
+ def format_error(error: Exception, include_trace: bool = False) -> str:
+ """Format error message based on error type and optionally include trace"""
+ message = ''
+ if isinstance(error, ValidationError):
+ return f'{AgentError.VALIDATION_ERROR}\nDetails: {str(error)}'
+ if isinstance(error, RateLimitError):
+ return AgentError.RATE_LIMIT_ERROR
+ if include_trace:
+ return f'{str(error)}\nStacktrace:\n{traceback.format_exc()}'
+ return f'{str(error)}'
diff --git a/browser-use/browser_use/browser/browser.py b/browser-use/browser_use/browser/browser.py
new file mode 100644
index 0000000..0967d47
--- /dev/null
+++ b/browser-use/browser_use/browser/browser.py
@@ -0,0 +1,421 @@
+"""
+Playwright browser on steroids.
+"""
+
+import asyncio
+import gc
+import logging
+import os
+import socket
+import subprocess
+from pathlib import Path
+from tempfile import gettempdir
+from typing import Literal
+
+import httpx
+import psutil
+from dotenv import load_dotenv
+from playwright.async_api import Browser as PlaywrightBrowser
+from playwright.async_api import Playwright, async_playwright
+from pydantic import AliasChoices, BaseModel, ConfigDict, Field
+
+load_dotenv()
+
+
+from browser_use.browser.chrome import (
+ CHROME_ARGS,
+ CHROME_DEBUG_PORT,
+ CHROME_DETERMINISTIC_RENDERING_ARGS,
+ CHROME_DISABLE_SECURITY_ARGS,
+ CHROME_DOCKER_ARGS,
+ CHROME_HEADLESS_ARGS,
+)
+from browser_use.browser.context import BrowserContext, BrowserContextConfig
+from browser_use.browser.utils.screen_resolution import get_screen_resolution, get_window_adjustments
+from browser_use.utils import time_execution_async
+
+logger = logging.getLogger(__name__)
+
+IN_DOCKER = os.environ.get('IN_DOCKER', 'false').lower()[0] in 'ty1'
+
+
+class ProxySettings(BaseModel):
+ """the same as playwright.sync_api.ProxySettings, but now as a Pydantic BaseModel so pydantic can validate it"""
+
+ server: str
+ bypass: str | None = None
+ username: str | None = None
+ password: str | None = None
+
+ model_config = ConfigDict(populate_by_name=True, from_attributes=True)
+
+ # Support dict-like behavior for compatibility with Playwright's ProxySettings
+ def __getitem__(self, key):
+ return getattr(self, key)
+
+ def get(self, key, default=None):
+ return getattr(self, key, default)
+
+
+class BrowserConfig(BaseModel):
+ r"""
+ Configuration for the Browser.
+
+ Default values:
+ headless: False
+ Whether to run browser in headless mode (not recommended)
+
+ disable_security: False
+ Disable browser security features (required for cross-origin iframe support)
+
+ extra_browser_args: []
+ Extra arguments to pass to the browser
+
+ wss_url: None
+ Connect to a browser instance via WebSocket
+
+ cdp_url: None
+ Connect to a browser instance via CDP
+
+ browser_binary_path: None
+ Path to a Browser instance to use to connect to your normal browser
+ e.g. '/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome'
+
+ chrome_remote_debugging_port: 9222
+ Chrome remote debugging port to use to when browser_binary_path is supplied.
+ This allows running multiple chrome browsers with same browser_binary_path but running on different ports.
+ Also, makes it possible to launch new user provided chrome browser without closing already opened chrome instances,
+ by providing non-default chrome debugging port.
+
+ keep_alive: False
+ Keep the browser alive after the agent has finished running
+
+ deterministic_rendering: False
+ Enable deterministic rendering (makes GPU/font rendering consistent across different OS's and docker)
+ """
+
+ model_config = ConfigDict(
+ arbitrary_types_allowed=True,
+ extra='ignore',
+ populate_by_name=True,
+ from_attributes=True,
+ validate_assignment=True,
+ revalidate_instances='subclass-instances',
+ )
+
+ wss_url: str | None = None
+ cdp_url: str | None = None
+
+ browser_class: Literal['chromium', 'firefox', 'webkit'] = 'chromium'
+ browser_binary_path: str | None = Field(
+ default=None, validation_alias=AliasChoices('browser_instance_path', 'chrome_instance_path')
+ )
+ chrome_remote_debugging_port: int | None = CHROME_DEBUG_PORT
+ extra_browser_args: list[str] = Field(default_factory=list)
+
+ headless: bool = False
+ disable_security: bool = False # disable_security=True is dangerous as any malicious URL visited could embed an iframe for the user's bank, and use their cookies to steal money
+ deterministic_rendering: bool = False
+ keep_alive: bool = Field(default=False, alias='_force_keep_browser_alive') # used to be called _force_keep_browser_alive
+
+ proxy: ProxySettings | None = None
+ new_context_config: BrowserContextConfig = Field(default_factory=BrowserContextConfig)
+
+
+# @singleton: TODO - think about id singleton makes sense here
+# @dev By default this is a singleton, but you can create multiple instances if you need to.
+class Browser:
+ """
+ Playwright browser on steroids.
+
+ This is persistent browser factory that can spawn multiple browser contexts.
+ It is recommended to use only one instance of Browser per your application (RAM usage will grow otherwise).
+ """
+
+ def __init__(
+ self,
+ config: BrowserConfig | None = None,
+ ):
+ logger.debug('๐ Initializing new browser')
+ self.config = config or BrowserConfig()
+ self.playwright: Playwright | None = None
+ self.playwright_browser: PlaywrightBrowser | None = None
+
+ async def new_context(self, config: BrowserContextConfig | None = None) -> BrowserContext:
+ """Create a browser context"""
+ browser_config = self.config.model_dump() if self.config else {}
+ context_config = config.model_dump() if config else {}
+ merged_config = {**browser_config, **context_config}
+ return BrowserContext(config=BrowserContextConfig(**merged_config), browser=self)
+
+ async def get_playwright_browser(self) -> PlaywrightBrowser:
+ """Get a browser context"""
+ if self.playwright_browser is None:
+ return await self._init()
+
+ return self.playwright_browser
+
+ @time_execution_async('--init (browser)')
+ async def _init(self):
+ """Initialize the browser session"""
+ playwright = await async_playwright().start()
+ self.playwright = playwright
+
+ browser = await self._setup_browser(playwright)
+ self.playwright_browser = browser
+
+ return self.playwright_browser
+
+ async def _setup_remote_cdp_browser(self, playwright: Playwright) -> PlaywrightBrowser:
+ """Sets up and returns a Playwright Browser instance with anti-detection measures. Firefox has no longer CDP support."""
+ if 'firefox' in (self.config.browser_binary_path or '').lower():
+ raise ValueError(
+ 'CDP has been deprecated for firefox, check: https://fxdx.dev/deprecating-cdp-support-in-firefox-embracing-the-future-with-webdriver-bidi/'
+ )
+ if not self.config.cdp_url:
+ raise ValueError('CDP URL is required')
+ logger.info(f'๐ Connecting to remote browser via CDP {self.config.cdp_url}')
+ browser_class = getattr(playwright, self.config.browser_class)
+ browser = await browser_class.connect_over_cdp(self.config.cdp_url)
+ return browser
+
+ async def _setup_remote_wss_browser(self, playwright: Playwright) -> PlaywrightBrowser:
+ """Sets up and returns a Playwright Browser instance with anti-detection measures."""
+ if not self.config.wss_url:
+ raise ValueError('WSS URL is required')
+ logger.info(f'๐ Connecting to remote browser via WSS {self.config.wss_url}')
+ browser_class = getattr(playwright, self.config.browser_class)
+ browser = await browser_class.connect(self.config.wss_url)
+ return browser
+
+ async def _setup_user_provided_browser(self, playwright: Playwright) -> PlaywrightBrowser:
+ """Sets up and returns a Playwright Browser instance with anti-detection measures."""
+ if not self.config.browser_binary_path:
+ raise ValueError('A browser_binary_path is required')
+
+ assert self.config.browser_class == 'chromium', (
+ 'browser_binary_path only supports chromium browsers (make sure browser_class=chromium)'
+ )
+
+ try:
+ # Check if browser is already running
+ async with httpx.AsyncClient() as client:
+ response = await client.get(
+ f'http://localhost:{self.config.chrome_remote_debugging_port}/json/version', timeout=2
+ )
+ if response.status_code == 200:
+ logger.info(
+ f'๐ Reusing existing browser found running on http://localhost:{self.config.chrome_remote_debugging_port}'
+ )
+ browser_class = getattr(playwright, self.config.browser_class)
+ browser = await browser_class.connect_over_cdp(
+ endpoint_url=f'http://localhost:{self.config.chrome_remote_debugging_port}',
+ timeout=20000, # 20 second timeout for connection
+ )
+ return browser
+ except httpx.RequestError:
+ logger.debug('๐ No existing Chrome instance found, starting a new one')
+
+ provided_user_data_dir = [arg for arg in self.config.extra_browser_args if '--user-data-dir=' in arg]
+
+ if provided_user_data_dir:
+ user_data_dir = Path(provided_user_data_dir[0].split('=')[-1])
+ else:
+ fallback_user_data_dir = Path(gettempdir()) / 'browseruse' / 'profiles' / 'default' # /tmp/browseruse
+ try:
+ # ~/.config/browseruse/profiles/default
+ user_data_dir = Path('~/.config') / 'browseruse' / 'profiles' / 'default'
+ user_data_dir = user_data_dir.expanduser()
+ user_data_dir.mkdir(parents=True, exist_ok=True)
+ except Exception as e:
+ logger.error(f'โ Failed to create ~/.config/browseruse directory: {type(e).__name__}: {e}')
+ user_data_dir = fallback_user_data_dir
+ user_data_dir.mkdir(parents=True, exist_ok=True)
+
+ logger.info(f'๐ Storing Browser Profile user data dir in: {user_data_dir}')
+ try:
+ # Remove any existing SingletonLock file to allow the browser to start
+ (user_data_dir / 'Default' / 'SingletonLock').unlink()
+ self.config.extra_browser_args.append('--no-first-run')
+ except (FileNotFoundError, PermissionError, OSError):
+ pass
+
+ # Start a new Chrome instance
+ chrome_launch_args = [
+ *{ # remove duplicates (usually preserves the order, but not guaranteed)
+ f'--remote-debugging-port={self.config.chrome_remote_debugging_port}',
+ *([f'--user-data-dir={user_data_dir.resolve()}'] if not provided_user_data_dir else []),
+ *CHROME_ARGS,
+ *(CHROME_DOCKER_ARGS if IN_DOCKER else []),
+ *(CHROME_HEADLESS_ARGS if self.config.headless else []),
+ *(CHROME_DISABLE_SECURITY_ARGS if self.config.disable_security else []),
+ *(CHROME_DETERMINISTIC_RENDERING_ARGS if self.config.deterministic_rendering else []),
+ *self.config.extra_browser_args,
+ },
+ ]
+ chrome_sub_process = await asyncio.create_subprocess_exec(
+ self.config.browser_binary_path,
+ *chrome_launch_args,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ shell=False,
+ )
+ self._chrome_subprocess = psutil.Process(chrome_sub_process.pid)
+
+ # Attempt to connect again after starting a new instance
+ for _ in range(10):
+ try:
+ async with httpx.AsyncClient() as client:
+ response = await client.get(
+ f'http://localhost:{self.config.chrome_remote_debugging_port}/json/version', timeout=2
+ )
+ if response.status_code == 200:
+ break
+ except httpx.RequestError:
+ pass
+ await asyncio.sleep(1)
+
+ # Attempt to connect again after starting a new instance
+ try:
+ browser_class = getattr(playwright, self.config.browser_class)
+ browser = await browser_class.connect_over_cdp(
+ endpoint_url=f'http://localhost:{self.config.chrome_remote_debugging_port}',
+ timeout=20000, # 20 second timeout for connection
+ )
+ return browser
+ except Exception as e:
+ logger.error(f'โ Failed to start a new Chrome instance: {str(e)}')
+ raise RuntimeError(
+ 'To start chrome in Debug mode, you need to close all existing Chrome instances and try again otherwise we can not connect to the instance.'
+ )
+
+ async def _setup_builtin_browser(self, playwright: Playwright) -> PlaywrightBrowser:
+ """Sets up and returns a Playwright Browser instance with anti-detection measures."""
+ assert self.config.browser_binary_path is None, 'browser_binary_path should be None if trying to use the builtin browsers'
+
+ # Use the configured window size from new_context_config if available
+ if (
+ not self.config.headless
+ and hasattr(self.config, 'new_context_config')
+ and hasattr(self.config.new_context_config, 'window_width')
+ and hasattr(self.config.new_context_config, 'window_height')
+ and not self.config.new_context_config.no_viewport
+ ):
+ screen_size = {
+ 'width': self.config.new_context_config.window_width,
+ 'height': self.config.new_context_config.window_height,
+ }
+ offset_x, offset_y = get_window_adjustments()
+ elif self.config.headless:
+ screen_size = {'width': 1920, 'height': 1080}
+ offset_x, offset_y = 0, 0
+ else:
+ screen_size = get_screen_resolution()
+ offset_x, offset_y = get_window_adjustments()
+
+ chrome_args = {
+ f'--remote-debugging-port={self.config.chrome_remote_debugging_port}',
+ *CHROME_ARGS,
+ *(CHROME_DOCKER_ARGS if IN_DOCKER else []),
+ *(CHROME_HEADLESS_ARGS if self.config.headless else []),
+ *(CHROME_DISABLE_SECURITY_ARGS if self.config.disable_security else []),
+ *(CHROME_DETERMINISTIC_RENDERING_ARGS if self.config.deterministic_rendering else []),
+ f'--window-position={offset_x},{offset_y}',
+ f'--window-size={screen_size["width"]},{screen_size["height"]}',
+ *self.config.extra_browser_args,
+ }
+
+ # check if chrome remote debugging port is already taken,
+ # if so remove the remote-debugging-port arg to prevent conflicts
+ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+ if s.connect_ex(('localhost', self.config.chrome_remote_debugging_port)) == 0:
+ chrome_args.remove(f'--remote-debugging-port={self.config.chrome_remote_debugging_port}')
+
+ browser_class = getattr(playwright, self.config.browser_class)
+ args = {
+ 'chromium': list(chrome_args),
+ 'firefox': [
+ *{
+ '-no-remote',
+ *self.config.extra_browser_args,
+ }
+ ],
+ 'webkit': [
+ *{
+ '--no-startup-window',
+ *self.config.extra_browser_args,
+ }
+ ],
+ }
+
+ browser = await browser_class.launch(
+ channel='chromium', # https://github.com/microsoft/playwright/issues/33566
+ headless=self.config.headless,
+ args=args[self.config.browser_class],
+ proxy=self.config.proxy.model_dump() if self.config.proxy else None,
+ handle_sigterm=False,
+ handle_sigint=False,
+ )
+ return browser
+
+ async def _setup_browser(self, playwright: Playwright) -> PlaywrightBrowser:
+ """Sets up and returns a Playwright Browser instance with anti-detection measures."""
+ try:
+ if self.config.cdp_url:
+ return await self._setup_remote_cdp_browser(playwright)
+ if self.config.wss_url:
+ return await self._setup_remote_wss_browser(playwright)
+
+ if self.config.headless:
+ logger.warning('โ ๏ธ Headless mode is not recommended. Many sites will detect and block all headless browsers.')
+
+ if self.config.browser_binary_path:
+ return await self._setup_user_provided_browser(playwright)
+ else:
+ return await self._setup_builtin_browser(playwright)
+ except Exception as e:
+ logger.error(f'Failed to initialize Playwright browser: {e}')
+ raise
+
+ async def close(self):
+ """Close the browser instance"""
+ if self.config.keep_alive:
+ return
+
+ try:
+ if self.playwright_browser:
+ await self.playwright_browser.close()
+ del self.playwright_browser
+ if self.playwright:
+ await self.playwright.stop()
+ del self.playwright
+ if chrome_proc := getattr(self, '_chrome_subprocess', None):
+ try:
+ # always kill all children processes, otherwise chrome leaves a bunch of zombie processes
+ for proc in chrome_proc.children(recursive=True):
+ proc.kill()
+ chrome_proc.kill()
+ except Exception as e:
+ logger.debug(f'Failed to terminate chrome subprocess: {e}')
+
+ except Exception as e:
+ if 'OpenAI error' not in str(e):
+ logger.debug(f'Failed to close browser properly: {e}')
+
+ finally:
+ self.playwright_browser = None
+ self.playwright = None
+ self._chrome_subprocess = None
+ gc.collect()
+
+ def __del__(self):
+ """Async cleanup when object is destroyed"""
+ try:
+ if self.playwright_browser or self.playwright:
+ loop = asyncio.get_running_loop()
+ if loop.is_running():
+ loop.create_task(self.close())
+ else:
+ asyncio.run(self.close())
+ except Exception as e:
+ logger.debug(f'Failed to cleanup browser in destructor: {e}')
diff --git a/browser-use/browser_use/browser/chrome.py b/browser-use/browser_use/browser/chrome.py
new file mode 100644
index 0000000..e3e7d3a
--- /dev/null
+++ b/browser-use/browser_use/browser/chrome.py
@@ -0,0 +1,177 @@
+CHROME_EXTENSIONS = {} # coming in a separate PR
+CHROME_EXTENSIONS_PATH = 'chrome_extensions'
+CHROME_PROFILE_PATH = 'chrome_profile'
+CHROME_PROFILE_USER = 'Default'
+CHROME_DEBUG_PORT = 9242
+CHROME_DISABLED_COMPONENTS = [
+ 'Translate',
+ 'AcceptCHFrame',
+ 'OptimizationHints',
+ 'ProcessPerSiteUpToMainFrameThreshold',
+ 'InterestFeedContentSuggestions',
+ # 'CalculateNativeWinOcclusion',
+ 'BackForwardCache',
+ # 'HeavyAdPrivacyMitigations',
+ 'LazyFrameLoading',
+ # 'ImprovedCookieControls',
+ 'PrivacySandboxSettings4',
+ 'AutofillServerCommunication',
+ 'CertificateTransparencyComponentUpdater',
+ 'DestroyProfileOnBrowserClose',
+ 'CrashReporting',
+ 'OverscrollHistoryNavigation',
+ 'InfiniteSessionRestore',
+ #'LockProfileCookieDatabase', # disabling allows multiple chrome instances to concurrently modify profile, but might make chrome much slower https://github.com/yt-dlp/yt-dlp/issues/7271 https://issues.chromium.org/issues/40901624
+] # it's always best to give each chrome instance its own exclusive copy of the user profile
+
+
+CHROME_HEADLESS_ARGS = [
+ '--headless=new',
+ # '--test-type',
+ # '--test-type=gpu', # https://github.com/puppeteer/puppeteer/issues/10516
+ # '--enable-automation', # <- DONT USE THIS, it makes you easily detectable / blocked by cloudflare
+]
+
+CHROME_DOCKER_ARGS = [
+ # Docker-specific options
+ # https://github.com/GoogleChrome/lighthouse-ci/tree/main/docs/recipes/docker-client#--no-sandbox-issues-explained
+ '--no-sandbox', # rely on docker sandboxing in docker, otherwise we need cap_add: SYS_ADM to use host sandboxing
+ '--disable-gpu-sandbox',
+ '--disable-setuid-sandbox',
+ '--disable-dev-shm-usage', # docker 75mb default shm size is not big enough, disabling just uses /tmp instead
+ '--no-xshm',
+ # dont try to disable (or install) dbus in docker, its not needed, chrome can work without dbus despite the errors
+]
+
+CHROME_DISABLE_SECURITY_ARGS = [
+ # DANGER: JS isolation security features (to allow easier tampering with pages during automation)
+ # chrome://net-internals
+ '--disable-web-security', # <- WARNING, breaks some sites that expect/enforce strict CORS headers (try webflow.com)
+ '--disable-site-isolation-trials',
+ '--disable-features=IsolateOrigins,site-per-process',
+ # '--allow-file-access-from-files', # <- WARNING, dangerous, allows JS to read filesystem using file:// URLs
+ # DANGER: Disable HTTPS verification
+ '--allow-running-insecure-content', # Breaks CORS/CSRF/HSTS etc., useful sometimes but very easy to detect
+ '--ignore-certificate-errors',
+ '--ignore-ssl-errors',
+ '--ignore-certificate-errors-spki-list',
+ # '--allow-insecure-localhost',
+]
+
+# flags to make chrome behave more deterministically across different OS's
+CHROME_DETERMINISTIC_RENDERING_ARGS = [
+ '--deterministic-mode',
+ '--js-flags=--random-seed=1157259159', # make all JS random numbers deterministic by providing a seed
+ '--force-device-scale-factor=1',
+ # GPU, canvas, text, and pdf rendering config
+ # chrome://gpu
+ '--enable-webgl', # enable web-gl graphics support
+ '--font-render-hinting=none', # make rendering more deterministic by ignoring OS font hints, may also need css override, try: * {text-rendering: geometricprecision !important; -webkit-font-smoothing: antialiased;}
+ '--force-color-profile=srgb', # make rendering more deterministic by using consistent color profile, if browser looks weird, try: generic-rgb
+ # '--disable-partial-raster', # make rendering more deterministic (TODO: verify if still needed)
+ '--disable-skia-runtime-opts', # make rendering more deterministic by avoiding Skia hot path runtime optimizations
+ '--disable-2d-canvas-clip-aa', # make rendering more deterministic by disabling antialiasing on 2d canvas clips
+ # '--disable-gpu', # falls back to more consistent software renderer across all OS's, especially helps linux text rendering look less weird
+ # // '--use-gl=swiftshader', <- DO NOT USE, breaks M1 ARM64. it makes rendering more deterministic by using simpler CPU renderer instead of OS GPU renderer bug: https://groups.google.com/a/chromium.org/g/chromium-dev/c/8eR2GctzGuw
+ # // '--disable-software-rasterizer', <- DO NOT USE, harmless, used in tandem with --disable-gpu
+ # // '--run-all-compositor-stages-before-draw', <- DO NOT USE, makes headful chrome hang on startup (tested v121 Google Chrome.app on macOS)
+ # // '--disable-gl-drawing-for-tests', <- DO NOT USE, disables gl output (makes tests run faster if you dont care about canvas)
+ # // '--blink-settings=imagesEnabled=false', <- DO NOT USE, disables images entirely (only sometimes useful to speed up loading)
+]
+
+
+CHROME_ARGS = [
+ # Process management & performance tuning
+ # chrome://process-internals
+ # '--disable-lazy-loading', # make rendering more deterministic by loading all content up-front instead of on-focus
+ # '--disable-renderer-backgrounding', # dont throttle tab rendering based on focus/visibility
+ # '--disable-background-networking', # dont throttle tab networking based on focus/visibility
+ # '--disable-background-timer-throttling', # dont throttle tab timers based on focus/visibility
+ # '--disable-backgrounding-occluded-windows', # dont throttle tab window based on focus/visibility
+ # '--disable-ipc-flooding-protection', # dont throttle ipc traffic or accessing big request/response/buffer/etc. objects will fail
+ # '--disable-extensions-http-throttling', # dont throttle http traffic based on runtime heuristics
+ # '--disable-field-trial-config', # disable shared field trial state between browser processes
+ # '--disable-back-forward-cache', # disable browsing navigation cache
+ # Profile data dir setup
+ # chrome://profile-internals
+ # f'--user-data-dir={CHROME_PROFILE_PATH}', # managed by playwright arg instead
+ # f'--profile-directory={CHROME_PROFILE_USER}',
+ # '--password-store=basic', # use mock keychain instead of OS-provided keychain (we manage auth.json instead)
+ # '--use-mock-keychain',
+ # '--disable-cookie-encryption', # we need to be able to write unencrypted cookies to save/load auth.json
+ '--disable-sync', # don't try to use Google account sync features while automation is active
+ # Extensions
+ # chrome://inspect/#extensions
+ # f'--load-extension={CHROME_EXTENSIONS.map(({unpacked_path}) => unpacked_path).join(',')}', # not needed when using existing profile that already has extensions installed
+ # f'--allowlisted-extension-id={",".join(CHROME_EXTENSIONS.keys())}',
+ '--allow-legacy-extension-manifests',
+ '--allow-pre-commit-input', # allow JS mutations before page rendering is complete
+ '--disable-blink-features=AutomationControlled', # hide the signatures that announce browser is being remote-controlled
+ # f'--proxy-server=https://43.159.28.126:2334:u7ce652b7568805c4-zone-custom-region-us-session-szGWq3FRU-sessTime-60:u7ce652b7568805c4', # send all network traffic through a proxy https://2captcha.com/proxy
+ # f'--proxy-bypass-list=127.0.0.1',
+ # Browser window and viewport setup
+ # chrome://version
+ # f'--user-agent="{DEFAULT_USER_AGENT}"',
+ # f'--window-size={DEFAULT_VIEWPORT.width},{DEFAULT_VIEWPORT.height}',
+ # '--window-position=0,0',
+ # '--start-maximized',
+ '--install-autogenerated-theme=0,0,0', # black border makes it easier to see which chrome window is browser-use's
+ '--hide-scrollbars', # stop scrollbars from affecting screenshot width/height
+ #'--virtual-time-budget=60000', # DONT USE THIS, makes chrome hang forever and doesn't work, used to fast-forward all animations & timers by 60s, dont use this it's unfortunately buggy and breaks screenshot and PDF capture sometimes
+ #'--autoplay-policy=no-user-gesture-required', # auto-start videos so they trigger network requests + show up in outputs
+ #'--disable-gesture-requirement-for-media-playback',
+ #'--lang=en-US,en;q=0.9',
+ # IO: stdin/stdout, debug port config
+ # chrome://inspect
+ '--log-level=2', # 1=DEBUG 2=WARNING 3=ERROR
+ '--enable-logging=stderr',
+ # '--remote-debugging-address=127.0.0.1', <- DONT USE THIS, no longer supported on chrome >100, never expose to non-localhost, would allow attacker to drive your browser from any machine
+ # '--enable-experimental-extension-apis', # add support for tab groups via chrome.tabs extension API
+ '--disable-focus-on-load', # prevent browser from hijacking focus
+ '--disable-window-activation',
+ # '--in-process-gpu', <- DONT USE THIS, makes headful startup time ~5-10s slower (tested v121 Google Chrome.app on macOS)
+ # '--disable-component-extensions-with-background-pages', # TODO: check this, disables chrome components that only run in background with no visible UI (could lower startup time)
+ # uncomment to disable hardware camera/mic/speaker access + present fake devices to websites
+ # (faster to disable, but disabling breaks recording browser audio in puppeteer-stream screenrecordings)
+ # '--use-fake-device-for-media-stream',
+ # '--use-fake-ui-for-media-stream',
+ # '--disable-features=GlobalMediaControls,MediaRouter,DialMediaRouteProvider',
+ # Output format options (PDF, screenshot, etc.)
+ '--export-tagged-pdf', # include table on contents and tags in printed PDFs
+ '--generate-pdf-document-outline',
+ # Suppress first-run features, popups, hints, updates, etc.
+ # chrome://system
+ '--no-pings',
+ '--no-default-browser-check',
+ '--no-startup-window',
+ '--ash-no-nudges',
+ '--disable-infobars',
+ '--disable-search-engine-choice-screen',
+ '--disable-session-crashed-bubble',
+ '--simulate-outdated-no-au="Tue, 31 Dec 2099 23:59:59 GMT"', # disable browser self-update while automation is active
+ '--hide-crash-restore-bubble',
+ '--suppress-message-center-popups',
+ '--disable-client-side-phishing-detection',
+ '--disable-domain-reliability',
+ '--disable-datasaver-prompt',
+ '--disable-hang-monitor',
+ '--disable-session-crashed-bubble',
+ '--disable-speech-synthesis-api',
+ '--disable-speech-api',
+ '--disable-print-preview',
+ '--safebrowsing-disable-auto-update',
+ # '--deny-permission-prompts',
+ '--disable-external-intent-requests',
+ # '--disable-notifications',
+ '--disable-desktop-notifications',
+ '--noerrdialogs',
+ '--disable-prompt-on-repost',
+ '--silent-debugger-extension-api',
+ # '--block-new-web-contents',
+ '--metrics-recording-only',
+ '--disable-breakpad',
+ # other feature flags
+ # chrome://flags chrome://components
+ f'--disable-features={",".join(CHROME_DISABLED_COMPONENTS)}',
+ '--enable-features=NetworkService',
+]
diff --git a/browser-use/browser_use/browser/context.py b/browser-use/browser_use/browser/context.py
new file mode 100644
index 0000000..2f47d3c
--- /dev/null
+++ b/browser-use/browser_use/browser/context.py
@@ -0,0 +1,2027 @@
+"""
+Playwright browser on steroids.
+"""
+
+import asyncio
+import base64
+import gc
+import json
+import logging
+import os
+import re
+import time
+import uuid
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+import anyio
+from playwright._impl._errors import TimeoutError
+from playwright.async_api import Browser as PlaywrightBrowser
+from playwright.async_api import (
+ BrowserContext as PlaywrightBrowserContext,
+)
+from playwright.async_api import (
+ ElementHandle,
+ FrameLocator,
+ Page,
+)
+from pydantic import BaseModel, ConfigDict, Field
+
+from browser_use.browser.views import (
+ BrowserError,
+ BrowserState,
+ TabInfo,
+ URLNotAllowedError,
+)
+from browser_use.dom.clickable_element_processor.service import ClickableElementProcessor
+from browser_use.dom.service import DomService
+from browser_use.dom.views import DOMElementNode, SelectorMap
+from browser_use.utils import time_execution_async, time_execution_sync
+
+if TYPE_CHECKING:
+ from browser_use.browser.browser import Browser
+
+logger = logging.getLogger(__name__)
+
+import platform
+
+BROWSER_NAVBAR_HEIGHT = {
+ 'windows': 85,
+ 'darwin': 80,
+ 'linux': 90,
+}.get(platform.system().lower(), 85)
+
+_GLOB_WARNING_SHOWN = False
+
+
+class BrowserContextConfig(BaseModel):
+ """
+ Configuration for the BrowserContext.
+
+ Default values:
+ cookies_file: None
+ Path to cookies file for persistence
+
+ disable_security: False
+ Disable browser security features (dangerous, but cross-origin iframe support requires it)
+
+ minimum_wait_page_load_time: 0.5
+ Minimum time to wait before getting page state for LLM input
+
+ wait_for_network_idle_page_load_time: 1.0
+ Time to wait for network requests to finish before getting page state.
+ Lower values may result in incomplete page loads.
+
+ maximum_wait_page_load_time: 5.0
+ Maximum time to wait for page load before proceeding anyway
+
+ wait_between_actions: 1.0
+ Time to wait between multiple per step actions
+
+ window_width: 1280
+ window_height: 1100
+ Default browser window dimensions
+
+ no_viewport: True
+ When True (default), the browser window size determines the viewport.
+ When False, forces a fixed viewport size using window_width and window_height. (constraint of the rendered content to a smaller area than the default of the entire window size)
+
+ save_recording_path: None
+ Path to save video recordings
+
+ save_downloads_path: None
+ Path to save downloads to
+
+ trace_path: None
+ Path to save trace files. It will auto name the file with the TRACE_PATH/{context_id}.zip
+
+ locale: None
+ Specify user locale, for example en-GB, de-DE, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting rules. If not provided, defaults to the system default locale.
+
+ user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36'
+ custom user agent to use.
+
+ highlight_elements: True
+ Highlight elements in the DOM on the screen
+
+ viewport_expansion: 0
+ Viewport expansion in pixels. This amount will increase the number of elements which are included in the state what the LLM will see. If set to -1, all elements will be included (this leads to high token usage). If set to 0, only the elements which are visible in the viewport will be included.
+
+ allowed_domains: None
+ List of allowed domains that can be accessed. If None, all domains are allowed.
+ Example: ['example.com', 'api.example.com']
+
+ include_dynamic_attributes: bool = True
+ Include dynamic attributes in the CSS selector. If you want to reuse the css_selectors, it might be better to set this to False.
+
+ http_credentials: None
+ Dictionary with HTTP basic authentication credentials for corporate intranets (only supports one set of credentials for all URLs at the moment), e.g.
+ {"username": "bill", "password": "pa55w0rd"}
+
+ is_mobile: None
+ Whether the meta viewport tag is taken into account and touch events are enabled.
+
+ has_touch: None
+ Whether to enable touch events in the browser.
+
+ geolocation: None
+ Geolocation to be used in the browser context. Example: {'latitude': 59.95, 'longitude': 30.31667}
+
+ permissions: ['clipboard-read', 'clipboard-write']
+ Browser permissions to grant. See full list here: https://playwright.dev/python/docs/api/class-browsercontext#browser-context-grant-permissions
+
+ timezone_id: None
+ Changes the timezone of the browser. Example: 'Europe/Berlin'
+
+ force_new_context: False
+ Forces a new browser context to be created. Useful when running locally with branded browser (e.g Chrome, Edge) and setting a custom config.
+ """
+
+ model_config = ConfigDict(
+ arbitrary_types_allowed=True,
+ extra='ignore',
+ populate_by_name=True,
+ from_attributes=True,
+ validate_assignment=True,
+ revalidate_instances='subclass-instances',
+ )
+
+ cookies_file: str | None = None
+ minimum_wait_page_load_time: float = 0.25
+ wait_for_network_idle_page_load_time: float = 0.5
+ maximum_wait_page_load_time: float = 5
+ wait_between_actions: float = 0.5
+
+ disable_security: bool = False # disable_security=True is dangerous as any malicious URL visited could embed an iframe for the user's bank, and use their cookies to steal money
+
+ window_width: int = 1280
+ window_height: int = 1100
+ no_viewport: bool = True # True is the default for headful mode - browser window size determines viewport
+
+ save_recording_path: str | None = None
+ save_downloads_path: str | None = None
+ save_har_path: str | None = None
+ trace_path: str | None = None
+ locale: str | None = None
+ user_agent: str | None = None
+
+ highlight_elements: bool = True
+ viewport_expansion: int = 0
+ allowed_domains: list[str] | None = None
+ include_dynamic_attributes: bool = True
+ http_credentials: dict[str, str] | None = None
+
+ keep_alive: bool = Field(default=False, alias='_force_keep_context_alive') # used to be called _force_keep_context_alive
+ is_mobile: bool | None = None
+ has_touch: bool | None = None
+ geolocation: dict | None = None
+ permissions: list[str] = Field(
+ default_factory=lambda: [
+ 'clipboard-read',
+ 'clipboard-write',
+ ]
+ )
+ timezone_id: str | None = None
+
+ force_new_context: bool = False
+
+
+@dataclass
+class CachedStateClickableElementsHashes:
+ """
+ Clickable elements hashes for the last state
+ """
+
+ url: str
+ hashes: set[str]
+
+
+class BrowserSession:
+ def __init__(self, context: PlaywrightBrowserContext, cached_state: BrowserState | None = None):
+ self.context = context
+ self.cached_state = cached_state
+
+ self.cached_state_clickable_elements_hashes: CachedStateClickableElementsHashes | None = None
+
+
+@dataclass
+class BrowserContextState:
+ """
+ State of the browser context
+ """
+
+ target_id: str | None = None # CDP target ID
+
+
+class BrowserContext:
+ def __init__(
+ self,
+ browser: 'Browser',
+ config: BrowserContextConfig | None = None,
+ state: BrowserContextState | None = None,
+ ):
+ self.context_id = str(uuid.uuid4())
+
+ self.config = config or BrowserContextConfig(**(browser.config.model_dump() if browser.config else {}))
+ self.browser = browser
+
+ self.state = state or BrowserContextState()
+
+ # Initialize these as None - they'll be set up when needed
+ self.session: BrowserSession | None = None
+
+ # Tab references - separate concepts for agent intent and browser state
+ self.agent_current_page: Page | None = None # The tab the agent intends to interact with
+ self.human_current_page: Page | None = None # The tab currently shown in the browser UI
+
+ async def __aenter__(self):
+ """Async context manager entry"""
+ await self._initialize_session()
+ return self
+
+ async def __aexit__(self, exc_type, exc_val, exc_tb):
+ """Async context manager exit"""
+ await self.close()
+
+ @time_execution_async('--close')
+ async def close(self):
+ """Close the browser instance"""
+
+ try:
+ if self.session is None:
+ return
+
+ # Then remove CDP protocol listeners
+ if self._page_event_handler and self.session.context:
+ try:
+ # This actually sends a CDP command to unsubscribe
+ self.session.context.remove_listener('page', self._page_event_handler)
+ except Exception as e:
+ logger.debug(f'Failed to remove CDP listener: {e}')
+ self._page_event_handler = None
+
+ await self.save_cookies()
+
+ if self.config.trace_path:
+ try:
+ await self.session.context.tracing.stop(path=os.path.join(self.config.trace_path, f'{self.context_id}.zip'))
+ except Exception as e:
+ logger.debug(f'Failed to stop tracing: {e}')
+
+ # This is crucial - it closes the CDP connection
+ if not self.config.keep_alive:
+ logger.debug('Closing browser context')
+ try:
+ await self.session.context.close()
+ except Exception as e:
+ logger.debug(f'Failed to close context: {e}')
+
+ finally:
+ # Dereference everything
+ self.agent_current_page = None
+ self.human_current_page = None
+ self.session = None
+ self._page_event_handler = None
+
+ def __del__(self):
+ """Cleanup when object is destroyed"""
+ if not self.config.keep_alive and self.session is not None:
+ logger.debug('BrowserContext was not properly closed before destruction')
+ try:
+ # Use sync Playwright method for force cleanup
+ if hasattr(self.session.context, '_impl_obj'):
+ asyncio.run(self.session.context._impl_obj.close())
+
+ self.session = None
+ self.agent_current_page = None
+ self.human_current_page = None
+ gc.collect()
+ except Exception as e:
+ logger.warning(f'Failed to force close browser context: {e}')
+
+ @time_execution_async('--initialize_session')
+ async def _initialize_session(self):
+ """Initialize the browser session"""
+ logger.debug(f'๐ Initializing new browser context with id: {self.context_id}')
+
+ playwright_browser = await self.browser.get_playwright_browser()
+ context = await self._create_context(playwright_browser)
+ self._page_event_handler = None
+
+ # auto-attach the foregrounding-detection listener to all new pages opened
+ context.on('page', self._add_tab_foregrounding_listener)
+
+ # Get or create a page to use
+ pages = context.pages
+
+ self.session = BrowserSession(
+ context=context,
+ cached_state=None,
+ )
+
+ current_page = None
+ if self.browser.config.cdp_url:
+ # If we have a saved target ID, try to find and activate it
+ if self.state.target_id:
+ targets = await self._get_cdp_targets()
+ for target in targets:
+ if target['targetId'] == self.state.target_id:
+ # Find matching page by URL
+ for page in pages:
+ if page.url == target['url']:
+ current_page = page
+ break
+ break
+
+ # If no target ID or couldn't find it, use existing page or create new
+ if not current_page:
+ if (
+ pages
+ and pages[0].url
+ and not pages[0].url.startswith('chrome://') # skip chrome internal pages e.g. settings, history, etc
+ and not pages[0].url.startswith('chrome-extension://') # skip hidden extension background pages
+ ):
+ current_page = pages[0]
+ logger.debug('๐ Using existing page: %s', current_page.url)
+ else:
+ current_page = await context.new_page()
+ await current_page.goto('about:blank')
+ logger.debug('๐ Created new page: %s', current_page.url)
+
+ # Get target ID for the active page
+ if self.browser.config.cdp_url:
+ targets = await self._get_cdp_targets()
+ for target in targets:
+ if target['url'] == current_page.url:
+ self.state.target_id = target['targetId']
+ break
+
+ # Bring page to front
+ logger.debug('๐ซจ Bringing tab to front: %s', current_page)
+ await current_page.bring_to_front()
+ await current_page.wait_for_load_state('load')
+
+ # Set the viewport size for the active page
+ await self.set_viewport_size(current_page)
+
+ # Initialize both tab references to the same page initially
+ self.agent_current_page = current_page
+ self.human_current_page = current_page
+
+ # Set up visibility listeners for all existing tabs
+ for page in pages:
+ if not page.url.startswith('chrome-extension://') and not page.url.startswith('chrome://') and not page.is_closed():
+ await self._add_tab_foregrounding_listener(page)
+ # logger.debug(f'๐๏ธ Added visibility listener to existing tab: {page.url}')
+
+ return self.session
+
+ async def _add_tab_foregrounding_listener(self, page: Page):
+ """
+ Attaches listeners that detect when the human steals active tab focus away from the agent.
+
+ Uses a combination of:
+ - visibilitychange events
+ - window focus/blur events
+ - pointermove events
+
+ This multi-method approach provides more reliable detection across browsers.
+
+ TODO: pester the playwright team to add a new event that fires when a headful tab is focused.
+ OR implement a browser-use chrome extension that acts as a bridge to the chrome.tabs API.
+
+ - https://github.com/microsoft/playwright/issues/1290
+ - https://github.com/microsoft/playwright/issues/2286
+ - https://github.com/microsoft/playwright/issues/3570
+ - https://github.com/microsoft/playwright/issues/13989
+ """
+
+ def trunc(s, max_len=None):
+ s = s.replace('https://', '').replace('http://', '').replace('www.', '')
+ if max_len is not None and len(s) > max_len:
+ return s[:max_len] + 'โฆ'
+ return s
+
+ try:
+ # Generate a unique function name for this page
+ visibility_func_name = f'onVisibilityChange_{id(page)}{id(page.url)}'
+
+ # Define the callback that will be called from browser when tab becomes visible
+ async def on_visibility_change(data):
+ source = data.get('source', 'unknown')
+
+ # Log before and after for debugging
+ old_foreground = self.human_current_page
+ if old_foreground.url != page.url:
+ logger.warning(
+ f'๐๏ธ Foregound tab changed by human from {trunc(old_foreground.url, 22) if old_foreground else "about:blank"} to {trunc(page.url, 22)} ({source}) but agent will stay on {trunc(self.agent_current_page.url, 22)}'
+ )
+
+ # Update foreground tab
+ self.human_current_page = page
+
+ # Expose the function to the browser
+ await page.expose_function(visibility_func_name, on_visibility_change)
+
+ # Set up multiple visibility detection methods in the browser
+ js_code = """() => {
+ // --- Method 1: visibilitychange event (unfortunately *all* tabs are always marked visible by playwright, usually does not fire) ---
+ document.addEventListener('visibilitychange', () => {
+ if (document.visibilityState === 'visible') {
+ window.onVisibilityChange_TABID({ source: 'visibilitychange' });
+ }
+ });
+
+ // --- Method 2: focus/blur events, most reliable method for headful browsers ---
+ window.addEventListener('focus', () => {
+ window.onVisibilityChange_TABID({ source: 'focus' });
+ });
+
+ // --- Method 3: pointermove events (may be fired by agent if we implement AI hover movements) ---
+ // Use a throttled handler to avoid excessive calls
+ // let lastMove = 0;
+ // window.addEventListener('pointermove', () => {
+ // const now = Date.now();
+ // if (now - lastMove > 1000) { // Throttle to once per second
+ // lastMove = now;
+ // window.onVisibilityChange_TABID({ source: 'pointermove' });
+ // }
+ // });
+ }""".replace('onVisibilityChange_TABID', visibility_func_name)
+ # multiple reasons for doing it this way ^: stealth, uniqueness, sometimes pageload is cancelled and it runs twice, etc.
+ await page.evaluate(js_code)
+
+ # re-add listener to the page for when it navigates to a new url, because previous listeners will be cleared
+ page.on('domcontentloaded', self._add_tab_foregrounding_listener)
+ logger.debug(f'๐ Added tab focus listeners to tab: {page.url}')
+
+ if page.url != self.agent_current_page.url:
+ await on_visibility_change({'source': 'navigation'})
+
+ except Exception as e:
+ logger.debug(f'Failed to add tab focus listener to {page.url}: {e}')
+
+ async def get_session(self) -> BrowserSession:
+ """Lazy initialization of the browser and related components"""
+ if self.session is None:
+ try:
+ return await self._initialize_session()
+ except Exception as e:
+ logger.error(f'โ Failed to create new browser session: {e} (did the browser process quit?)')
+ raise e
+ return self.session
+
+ async def get_current_page(self) -> Page:
+ """Legacy method for backwards compatibility, prefer get_agent_current_page()"""
+ return await self.get_agent_current_page()
+
+ async def _reconcile_tab_state(self) -> None:
+ """Reconcile tab state when tabs might be out of sync.
+
+ This method ensures that:
+ 1. Both tab references (agent_current_page and human_current_page) are valid
+ 2. Recovers invalid tab references using valid ones
+ 3. Handles the case where both references are invalid
+ """
+ session = await self.get_session()
+
+ agent_tab_valid = (
+ self.agent_current_page
+ and self.agent_current_page in session.context.pages
+ and not self.agent_current_page.is_closed()
+ )
+
+ human_current_page_valid = (
+ self.human_current_page
+ and self.human_current_page in session.context.pages
+ and not self.human_current_page.is_closed()
+ )
+
+ # Case 1: Both references are valid - nothing to do
+ if agent_tab_valid and human_current_page_valid:
+ return
+
+ # Case 2: Only agent_current_page is valid - update human_current_page
+ if agent_tab_valid and not human_current_page_valid:
+ self.human_current_page = self.agent_current_page
+ return
+
+ # Case 3: Only human_current_page is valid - update agent_current_page
+ if human_current_page_valid and not agent_tab_valid:
+ self.agent_current_page = self.human_current_page
+ return
+
+ # Case 4: Neither reference is valid - recover from available tabs
+ non_extension_pages = [
+ page
+ for page in session.context.pages
+ if not page.url.startswith('chrome-extension://') and not page.url.startswith('chrome://')
+ ]
+
+ if non_extension_pages:
+ # Update both tab references to the most recently opened non-extension page
+ recovered_page = non_extension_pages[-1]
+ self.agent_current_page = recovered_page
+ self.human_current_page = recovered_page
+ return
+
+ # Case 5: No valid pages at all - create a new page
+ try:
+ new_page = await session.context.new_page()
+ self.agent_current_page = new_page
+ self.human_current_page = new_page
+ except Exception:
+ # Last resort - recreate the session
+ logger.warning('โ ๏ธ No browser window available, recreating session')
+ await self._initialize_session()
+ if session.context.pages:
+ page = session.context.pages[0]
+ self.agent_current_page = page
+ self.human_current_page = page
+
+ async def get_agent_current_page(self) -> Page:
+ """Get the page that the agent is currently working with.
+
+ This method prioritizes agent_current_page over human_current_page, ensuring
+ that agent operations happen on the intended tab regardless of user
+ interaction with the browser.
+
+ If agent_current_page is invalid or closed, it will attempt to recover
+ with a valid tab reference by reconciling the tab state.
+ """
+ session = await self.get_session()
+
+ # First check if agent_current_page is valid
+ if (
+ self.agent_current_page
+ and self.agent_current_page in session.context.pages
+ and not self.agent_current_page.is_closed()
+ ):
+ return self.agent_current_page
+
+ # If we're here, reconcile tab state and try again
+ await self._reconcile_tab_state()
+
+ # After reconciliation, agent_current_page should be valid
+ if (
+ self.agent_current_page
+ and self.agent_current_page in session.context.pages
+ and not self.agent_current_page.is_closed()
+ ):
+ return self.agent_current_page
+
+ # If still invalid, fall back to first page method as last resort
+ logger.warning('โ ๏ธ Failed to get agent current page, falling back to first page')
+ if session.context.pages:
+ page = session.context.pages[0]
+ self.agent_current_page = page
+ self.human_current_page = page
+ return page
+
+ # If no pages, create one
+ return await session.context.new_page()
+
+ async def _create_context(self, browser: PlaywrightBrowser):
+ """Creates a new browser context with anti-detection measures and loads cookies if available."""
+ if self.browser.config.cdp_url and len(browser.contexts) > 0 and not self.config.force_new_context:
+ context = browser.contexts[0]
+ # For existing contexts, we need to set the viewport size manually
+ if context.pages and not self.browser.config.headless:
+ for page in context.pages:
+ await self.set_viewport_size(page)
+ elif self.browser.config.browser_binary_path and len(browser.contexts) > 0 and not self.config.force_new_context:
+ # Connect to existing Chrome instance instead of creating new one
+ context = browser.contexts[0]
+ # For existing contexts, we need to set the viewport size manually
+ if context.pages and not self.browser.config.headless:
+ for page in context.pages:
+ await self.set_viewport_size(page)
+ else:
+ kwargs = {}
+ # Set viewport for both headless and non-headless modes
+ if self.browser.config.headless:
+ # In headless mode, always set viewport and no_viewport=False
+ kwargs['viewport'] = {'width': self.config.window_width, 'height': self.config.window_height}
+ kwargs['no_viewport'] = False
+ else:
+ # In headful mode, use the no_viewport value from config (defaults to True)
+ no_viewport_value = self.config.no_viewport
+ kwargs['no_viewport'] = no_viewport_value
+
+ # Only set viewport if no_viewport is False
+ if not no_viewport_value:
+ kwargs['viewport'] = {'width': self.config.window_width, 'height': self.config.window_height}
+
+ if self.config.user_agent is not None:
+ kwargs['user_agent'] = self.config.user_agent
+
+ context = await browser.new_context(
+ **kwargs,
+ java_script_enabled=True,
+ **({'bypass_csp': True, 'ignore_https_errors': True} if self.config.disable_security else {}),
+ record_video_dir=self.config.save_recording_path,
+ record_video_size={'width': self.config.window_width, 'height': self.config.window_height},
+ record_har_path=self.config.save_har_path,
+ locale=self.config.locale,
+ http_credentials=self.config.http_credentials,
+ is_mobile=self.config.is_mobile,
+ has_touch=self.config.has_touch,
+ geolocation=self.config.geolocation,
+ permissions=self.config.permissions,
+ timezone_id=self.config.timezone_id,
+ )
+
+ # Ensure required permissions are granted
+ required_permissions = ['clipboard-read', 'clipboard-write'] # needed for google sheets automation
+ if self.config.geolocation:
+ required_permissions.append('geolocation')
+ missing_permissions = [p for p in required_permissions if p not in self.config.permissions]
+ if any(missing_permissions):
+ logger.warning(
+ f'โ ๏ธ Some permissions required by browser-use {missing_permissions} are missing from BrowserContextConfig(permissions={self.config.permissions}), some features may not work properly!'
+ )
+ await context.grant_permissions(self.config.permissions)
+
+ if self.config.trace_path:
+ await context.tracing.start(screenshots=True, snapshots=True, sources=True)
+
+ # Resize the window for non-headless mode
+ if not self.browser.config.headless:
+ await self._resize_window(context)
+
+ # Load cookies if they exist
+ if self.config.cookies_file and os.path.exists(self.config.cookies_file):
+ async with await anyio.open_file(self.config.cookies_file, 'r') as f:
+ try:
+ cookies = json.loads(await f.read())
+
+ valid_same_site_values = ['Strict', 'Lax', 'None']
+ for cookie in cookies:
+ if 'sameSite' in cookie:
+ if cookie['sameSite'] not in valid_same_site_values:
+ logger.warning(
+ f"Fixed invalid sameSite value '{cookie['sameSite']}' to 'None' for cookie {cookie.get('name')}"
+ )
+ cookie['sameSite'] = 'None'
+ logger.info(f'๐ช Loaded {len(cookies)} cookies from {self.config.cookies_file}')
+ await context.add_cookies(cookies)
+
+ except json.JSONDecodeError as e:
+ logger.error(f'Failed to parse cookies file: {str(e)}')
+
+ init_script = """
+ // check to make sure we're not inside the PDF viewer
+ window.isPdfViewer = !!document?.body?.querySelector('body > embed[type="application/pdf"][width="100%"]')
+ if (!window.isPdfViewer) {
+
+ // Permissions
+ const originalQuery = window.navigator.permissions.query;
+ window.navigator.permissions.query = (parameters) => (
+ parameters.name === 'notifications' ?
+ Promise.resolve({ state: Notification.permission }) :
+ originalQuery(parameters)
+ );
+ (() => {
+ if (window._eventListenerTrackerInitialized) return;
+ window._eventListenerTrackerInitialized = true;
+
+ const originalAddEventListener = EventTarget.prototype.addEventListener;
+ const eventListenersMap = new WeakMap();
+
+ EventTarget.prototype.addEventListener = function(type, listener, options) {
+ if (typeof listener === "function") {
+ let listeners = eventListenersMap.get(this);
+ if (!listeners) {
+ listeners = [];
+ eventListenersMap.set(this, listeners);
+ }
+
+ listeners.push({
+ type,
+ listener,
+ listenerPreview: listener.toString().slice(0, 100),
+ options
+ });
+ }
+
+ return originalAddEventListener.call(this, type, listener, options);
+ };
+
+ window.getEventListenersForNode = (node) => {
+ const listeners = eventListenersMap.get(node) || [];
+ return listeners.map(({ type, listenerPreview, options }) => ({
+ type,
+ listenerPreview,
+ options
+ }));
+ };
+ })();
+ }
+ """
+
+ # Expose anti-detection scripts
+ await context.add_init_script(init_script)
+
+ return context
+
+ async def set_viewport_size(self, page: Page) -> None:
+ """
+ Central method to set viewport size for a page.
+ Simple for now, but we may need to add more logic here in the future to rezise surrounding window, change recording options, etc.
+ """
+ try:
+ # Only set viewport size if no_viewport is False (aka viewport=True)
+ if self.config.no_viewport is False:
+ viewport_size = {'width': self.config.window_width, 'height': self.config.window_height}
+ await page.set_viewport_size(viewport_size)
+ logger.debug(f'Set viewport size to {self.config.window_width}x{self.config.window_height}')
+ # else:
+ # logger.debug('Skipping viewport size setting because no_viewport is not False')
+ except Exception as e:
+ logger.debug(f'Failed to set viewport size for page: {e}')
+
+ async def _wait_for_stable_network(self):
+ page = await self.get_agent_current_page()
+
+ pending_requests = set()
+ last_activity = asyncio.get_event_loop().time()
+
+ # Define relevant resource types and content types
+ RELEVANT_RESOURCE_TYPES = {
+ 'document',
+ 'stylesheet',
+ 'image',
+ 'font',
+ 'script',
+ 'iframe',
+ }
+
+ RELEVANT_CONTENT_TYPES = {
+ 'text/html',
+ 'text/css',
+ 'application/javascript',
+ 'image/',
+ 'font/',
+ 'application/json',
+ }
+
+ # Additional patterns to filter out
+ IGNORED_URL_PATTERNS = {
+ # Analytics and tracking
+ 'analytics',
+ 'tracking',
+ 'telemetry',
+ 'beacon',
+ 'metrics',
+ # Ad-related
+ 'doubleclick',
+ 'adsystem',
+ 'adserver',
+ 'advertising',
+ # Social media widgets
+ 'facebook.com/plugins',
+ 'platform.twitter',
+ 'linkedin.com/embed',
+ # Live chat and support
+ 'livechat',
+ 'zendesk',
+ 'intercom',
+ 'crisp.chat',
+ 'hotjar',
+ # Push notifications
+ 'push-notifications',
+ 'onesignal',
+ 'pushwoosh',
+ # Background sync/heartbeat
+ 'heartbeat',
+ 'ping',
+ 'alive',
+ # WebRTC and streaming
+ 'webrtc',
+ 'rtmp://',
+ 'wss://',
+ # Common CDNs for dynamic content
+ 'cloudfront.net',
+ 'fastly.net',
+ }
+
+ async def on_request(request):
+ # Filter by resource type
+ if request.resource_type not in RELEVANT_RESOURCE_TYPES:
+ return
+
+ # Filter out streaming, websocket, and other real-time requests
+ if request.resource_type in {
+ 'websocket',
+ 'media',
+ 'eventsource',
+ 'manifest',
+ 'other',
+ }:
+ return
+
+ # Filter out by URL patterns
+ url = request.url.lower()
+ if any(pattern in url for pattern in IGNORED_URL_PATTERNS):
+ return
+
+ # Filter out data URLs and blob URLs
+ if url.startswith(('data:', 'blob:')):
+ return
+
+ # Filter out requests with certain headers
+ headers = request.headers
+ if headers.get('purpose') == 'prefetch' or headers.get('sec-fetch-dest') in [
+ 'video',
+ 'audio',
+ ]:
+ return
+
+ nonlocal last_activity
+ pending_requests.add(request)
+ last_activity = asyncio.get_event_loop().time()
+ # logger.debug(f'Request started: {request.url} ({request.resource_type})')
+
+ async def on_response(response):
+ request = response.request
+ if request not in pending_requests:
+ return
+
+ # Filter by content type if available
+ content_type = response.headers.get('content-type', '').lower()
+
+ # Skip if content type indicates streaming or real-time data
+ if any(
+ t in content_type
+ for t in [
+ 'streaming',
+ 'video',
+ 'audio',
+ 'webm',
+ 'mp4',
+ 'event-stream',
+ 'websocket',
+ 'protobuf',
+ ]
+ ):
+ pending_requests.remove(request)
+ return
+
+ # Only process relevant content types
+ if not any(ct in content_type for ct in RELEVANT_CONTENT_TYPES):
+ pending_requests.remove(request)
+ return
+
+ # Skip if response is too large (likely not essential for page load)
+ content_length = response.headers.get('content-length')
+ if content_length and int(content_length) > 5 * 1024 * 1024: # 5MB
+ pending_requests.remove(request)
+ return
+
+ nonlocal last_activity
+ pending_requests.remove(request)
+ last_activity = asyncio.get_event_loop().time()
+ # logger.debug(f'Request resolved: {request.url} ({content_type})')
+
+ # Attach event listeners
+ page.on('request', on_request)
+ page.on('response', on_response)
+
+ try:
+ # Wait for idle time
+ start_time = asyncio.get_event_loop().time()
+ while True:
+ await asyncio.sleep(0.1)
+ now = asyncio.get_event_loop().time()
+ if len(pending_requests) == 0 and (now - last_activity) >= self.config.wait_for_network_idle_page_load_time:
+ break
+ if now - start_time > self.config.maximum_wait_page_load_time:
+ logger.debug(
+ f'Network timeout after {self.config.maximum_wait_page_load_time}s with {len(pending_requests)} '
+ f'pending requests: {[r.url for r in pending_requests]}'
+ )
+ break
+
+ finally:
+ # Clean up event listeners
+ page.remove_listener('request', on_request)
+ page.remove_listener('response', on_response)
+
+ logger.debug(f'โ๏ธ Network stabilized for {self.config.wait_for_network_idle_page_load_time} seconds')
+
+ async def _wait_for_page_and_frames_load(self, timeout_overwrite: float | None = None):
+ """
+ Ensures page is fully loaded before continuing.
+ Waits for either network to be idle or minimum WAIT_TIME, whichever is longer.
+ Also checks if the loaded URL is allowed.
+ """
+ # Start timing
+ start_time = time.time()
+
+ # Wait for page load
+ try:
+ await self._wait_for_stable_network()
+
+ # Check if the loaded URL is allowed
+ page = await self.get_agent_current_page()
+ await self._check_and_handle_navigation(page)
+ except URLNotAllowedError as e:
+ raise e
+ except Exception:
+ logger.warning('โ ๏ธ Page load failed, continuing...')
+ pass
+
+ # Calculate remaining time to meet minimum WAIT_TIME
+ elapsed = time.time() - start_time
+ remaining = max((timeout_overwrite or self.config.minimum_wait_page_load_time) - elapsed, 0)
+
+ logger.debug(f'--Page loaded in {elapsed:.2f} seconds, waiting for additional {remaining:.2f} seconds')
+
+ # Sleep remaining time if needed
+ if remaining > 0:
+ await asyncio.sleep(remaining)
+
+ def _is_url_allowed(self, url: str) -> bool:
+ """
+ Check if a URL is allowed based on the whitelist configuration.
+
+ Supports glob patterns in allowed_domains:
+ - *.example.com will match sub.example.com and example.com
+ - *google.com will match google.com, agoogle.com, and www.google.com
+ """
+
+ if not self.config.allowed_domains:
+ return True
+
+ def _show_glob_warning(domain: str, glob: str):
+ global _GLOB_WARNING_SHOWN
+ if not _GLOB_WARNING_SHOWN:
+ logger.warning(
+ # glob patterns are very easy to mess up and match too many domains by accident
+ # e.g. if you only need to access gmail, don't use *.google.com because an attacker could convince the agent to visit a malicious doc
+ # on docs.google.com/s/some/evil/doc to set up a prompt injection attack
+ "โ ๏ธ Allowing agent to visit {domain} based on allowed_domains=['{glob}', ...]. Set allowed_domains=['{domain}', ...] explicitly to avoid the security risks of glob patterns!"
+ )
+ _GLOB_WARNING_SHOWN = True
+
+ try:
+ import fnmatch
+ from urllib.parse import urlparse
+
+ parsed_url = urlparse(url)
+
+ # Special case: Allow 'about:blank' explicitly
+ if url == 'about:blank' or parsed_url.scheme.lower() in ('chrome', 'brave', 'edge', 'chrome-extension'):
+ return True
+
+ # Extract only the hostname component (without auth credentials or port)
+ # Hostname returns only the domain portion, ignoring username:password and port
+ domain = parsed_url.hostname.lower() if parsed_url.hostname else ''
+
+ if not domain:
+ return False
+
+ for allowed_domain in self.config.allowed_domains:
+ allowed_domain = allowed_domain.lower()
+
+ # Handle glob patterns
+ if '*' in allowed_domain:
+ # Special handling for *.domain.tld pattern to also match the bare domain
+ if allowed_domain.startswith('*.'):
+ # If pattern is *.example.com, also allow example.com (without subdomain)
+ parent_domain = allowed_domain[2:] # Remove the '*.' prefix
+ if domain == parent_domain or fnmatch.fnmatch(domain, allowed_domain):
+ _show_glob_warning(domain, allowed_domain)
+ return True
+ else:
+ # For other glob patterns like *google.com
+ if fnmatch.fnmatch(domain, allowed_domain):
+ _show_glob_warning(domain, allowed_domain)
+ return True
+ else:
+ # Standard matching (exact or subdomain)
+ if domain == allowed_domain:
+ return True
+
+ return False
+ except Exception as e:
+ logger.error(f'โ๏ธ Error checking URL allowlist: {type(e).__name__}: {e}')
+ return False
+
+ async def _check_and_handle_navigation(self, page: Page) -> None:
+ """Check if current page URL is allowed and handle if not."""
+ if not self._is_url_allowed(page.url):
+ logger.warning(f'โ๏ธ Navigation to non-allowed URL detected: {page.url}')
+ try:
+ await self.go_back()
+ except Exception as e:
+ logger.error(f'โ๏ธ Failed to go back after detecting non-allowed URL: {str(e)}')
+ raise URLNotAllowedError(f'Navigation to non-allowed URL: {page.url}')
+
+ async def navigate_to(self, url: str):
+ """Navigate the agent's current tab to a URL"""
+ if not self._is_url_allowed(url):
+ raise BrowserError(f'Navigation to non-allowed URL: {url}')
+
+ page = await self.get_agent_current_page()
+ await page.goto(url)
+ await page.wait_for_load_state()
+
+ async def refresh_page(self):
+ """Refresh the agent's current page"""
+ page = await self.get_agent_current_page()
+ await page.reload()
+ await page.wait_for_load_state()
+
+ async def go_back(self):
+ """Navigate the agent's tab back in browser history"""
+ page = await self.get_agent_current_page()
+ try:
+ # 10 ms timeout
+ await page.go_back(timeout=10, wait_until='domcontentloaded')
+ # await self._wait_for_page_and_frames_load(timeout_overwrite=1.0)
+ except Exception as e:
+ # Continue even if its not fully loaded, because we wait later for the page to load
+ logger.debug(f'โฎ๏ธ Error during go_back: {e}')
+
+ async def go_forward(self):
+ """Navigate the agent's tab forward in browser history"""
+ page = await self.get_agent_current_page()
+ try:
+ await page.go_forward(timeout=10, wait_until='domcontentloaded')
+ except Exception as e:
+ # Continue even if its not fully loaded, because we wait later for the page to load
+ logger.debug(f'โญ๏ธ Error during go_forward: {e}')
+
+ async def close_current_tab(self):
+ """Close the current tab that the agent is working with.
+
+ This closes the tab that the agent is currently using (agent_current_page),
+ not necessarily the tab that is visible to the user (human_current_page).
+ If they are the same tab, both references will be updated.
+ """
+ session = await self.get_session()
+ page = await self.get_agent_current_page()
+
+ # Check if this is the foreground tab as well
+ is_foreground = page == self.human_current_page
+
+ # Close the tab
+ await page.close()
+
+ # Clear agent's reference to the closed tab
+ self.agent_current_page = None
+
+ # Clear foreground reference if needed
+ if is_foreground:
+ self.human_current_page = None
+
+ # Switch to the first available tab if any exist
+ if session.context.pages:
+ await self.switch_to_tab(0)
+ # switch_to_tab already updates both tab references
+
+ # Otherwise, the browser will be closed
+
+ async def get_page_html(self) -> str:
+ """Get the HTML content of the agent's current page"""
+ page = await self.get_agent_current_page()
+ return await page.content()
+
+ async def execute_javascript(self, script: str):
+ """Execute JavaScript code on the agent's current page"""
+ page = await self.get_agent_current_page()
+ return await page.evaluate(script)
+
+ async def get_page_structure(self) -> str:
+ """Get a debug view of the page structure including iframes"""
+ debug_script = """(() => {
+ function getPageStructure(element = document, depth = 0, maxDepth = 10) {
+ if (depth >= maxDepth) return '';
+
+ const indent = ' '.repeat(depth);
+ let structure = '';
+
+ // Skip certain elements that clutter the output
+ const skipTags = new Set(['script', 'style', 'link', 'meta', 'noscript']);
+
+ // Add current element info if it's not the document
+ if (element !== document) {
+ const tagName = element.tagName.toLowerCase();
+
+ // Skip uninteresting elements
+ if (skipTags.has(tagName)) return '';
+
+ const id = element.id ? `#${element.id}` : '';
+ const classes = element.className && typeof element.className === 'string' ?
+ `.${element.className.split(' ').filter(c => c).join('.')}` : '';
+
+ // Get additional useful attributes
+ const attrs = [];
+ if (element.getAttribute('role')) attrs.push(`role="${element.getAttribute('role')}"`);
+ if (element.getAttribute('aria-label')) attrs.push(`aria-label="${element.getAttribute('aria-label')}"`);
+ if (element.getAttribute('type')) attrs.push(`type="${element.getAttribute('type')}"`);
+ if (element.getAttribute('name')) attrs.push(`name="${element.getAttribute('name')}"`);
+ if (element.getAttribute('src')) {
+ const src = element.getAttribute('src');
+ attrs.push(`src="${src.substring(0, 50)}${src.length > 50 ? '...' : ''}"`);
+ }
+
+ // Add element info
+ structure += `${indent}${tagName}${id}${classes}${attrs.length ? ' [' + attrs.join(', ') + ']' : ''}\\n`;
+
+ // Handle iframes specially
+ if (tagName === 'iframe') {
+ try {
+ const iframeDoc = element.contentDocument || element.contentWindow?.document;
+ if (iframeDoc) {
+ structure += `${indent} [IFRAME CONTENT]:\\n`;
+ structure += getPageStructure(iframeDoc, depth + 2, maxDepth);
+ } else {
+ structure += `${indent} [IFRAME: No access - likely cross-origin]\\n`;
+ }
+ } catch (e) {
+ structure += `${indent} [IFRAME: Access denied - ${e.message}]\\n`;
+ }
+ }
+ }
+
+ // Get all child elements
+ const children = element.children || element.childNodes;
+ for (const child of children) {
+ if (child.nodeType === 1) { // Element nodes only
+ structure += getPageStructure(child, depth + 1, maxDepth);
+ }
+ }
+
+ return structure;
+ }
+
+ return getPageStructure();
+ })()"""
+
+ page = await self.get_agent_current_page()
+ structure = await page.evaluate(debug_script)
+ return structure
+
+ @time_execution_sync('--get_state') # This decorator might need to be updated to handle async
+ async def get_state(self, cache_clickable_elements_hashes: bool) -> BrowserState:
+ """Get the current state of the browser
+
+ cache_clickable_elements_hashes: bool
+ If True, cache the clickable elements hashes for the current state. This is used to calculate which elements are new to the llm (from last message) -> reduces token usage.
+ """
+ await self._wait_for_page_and_frames_load()
+ session = await self.get_session()
+ updated_state = await self._get_updated_state()
+
+ # Find out which elements are new
+ # Do this only if url has not changed
+ if cache_clickable_elements_hashes:
+ # if we are on the same url as the last state, we can use the cached hashes
+ if (
+ session.cached_state_clickable_elements_hashes
+ and session.cached_state_clickable_elements_hashes.url == updated_state.url
+ ):
+ # Pointers, feel free to edit in place
+ updated_state_clickable_elements = ClickableElementProcessor.get_clickable_elements(updated_state.element_tree)
+
+ for dom_element in updated_state_clickable_elements:
+ dom_element.is_new = (
+ ClickableElementProcessor.hash_dom_element(dom_element)
+ not in session.cached_state_clickable_elements_hashes.hashes # see which elements are new from the last state where we cached the hashes
+ )
+ # in any case, we need to cache the new hashes
+ session.cached_state_clickable_elements_hashes = CachedStateClickableElementsHashes(
+ url=updated_state.url,
+ hashes=ClickableElementProcessor.get_clickable_elements_hashes(updated_state.element_tree),
+ )
+
+ session.cached_state = updated_state
+
+ # Save cookies if a file is specified
+ if self.config.cookies_file:
+ asyncio.create_task(self.save_cookies())
+
+ return session.cached_state
+
+ async def _get_updated_state(self, focus_element: int = -1) -> BrowserState:
+ """Update and return state."""
+ session = await self.get_session()
+
+ # Check if current page is still valid, if not switch to another available page
+ try:
+ page = await self.get_agent_current_page()
+ # Test if page is still accessible
+ await page.evaluate('1')
+ except Exception as e:
+ logger.debug(f'๐ Current page is no longer accessible: {str(e)}')
+ raise BrowserError('Browser closed: no valid pages available')
+
+ try:
+ await self.remove_highlights()
+ dom_service = DomService(page)
+ content = await dom_service.get_clickable_elements(
+ focus_element=focus_element,
+ viewport_expansion=self.config.viewport_expansion,
+ highlight_elements=self.config.highlight_elements,
+ )
+
+ tabs_info = await self.get_tabs_info()
+
+ # Get all cross-origin iframes within the page and open them in new tabs
+ # mark the titles of the new tabs so the LLM knows to check them for additional content
+ # unfortunately too buggy for now, too many sites use invisible cross-origin iframes for ads, tracking, youtube videos, social media, etc.
+ # and it distracts the bot by opening a lot of new tabs
+ # iframe_urls = await dom_service.get_cross_origin_iframes()
+ # for url in iframe_urls:
+ # if url in [tab.url for tab in tabs_info]:
+ # continue # skip if the iframe if we already have it open in a tab
+ # new_page_id = tabs_info[-1].page_id + 1
+ # logger.debug(f'Opening cross-origin iframe in new tab #{new_page_id}: {url}')
+ # await self.create_new_tab(url)
+ # tabs_info.append(
+ # TabInfo(
+ # page_id=new_page_id,
+ # url=url,
+ # title=f'iFrame opened as new tab, treat as if embedded inside page #{self.state.target_id}: {page.url}',
+ # parent_page_id=self.state.target_id,
+ # )
+ # )
+
+ screenshot_b64 = await self.take_screenshot()
+ pixels_above, pixels_below = await self.get_scroll_info(page)
+
+ # Find the agent's active tab ID
+ agent_current_page_id = 0
+ if self.agent_current_page:
+ for tab_info in tabs_info:
+ if tab_info.url == self.agent_current_page.url:
+ agent_current_page_id = tab_info.page_id
+ break
+
+ self.current_state = BrowserState(
+ element_tree=content.element_tree,
+ selector_map=content.selector_map,
+ url=page.url,
+ title=await page.title(),
+ tabs=tabs_info,
+ screenshot=screenshot_b64,
+ pixels_above=pixels_above,
+ pixels_below=pixels_below,
+ )
+
+ return self.current_state
+ except Exception as e:
+ logger.error(f'โ Failed to update state: {str(e)}')
+ # Return last known good state if available
+ if hasattr(self, 'current_state'):
+ return self.current_state
+ raise
+
+ # region - Browser Actions
+ @time_execution_async('--take_screenshot')
+ async def take_screenshot(self, full_page: bool = False) -> str:
+ """
+ Returns a base64 encoded screenshot of the current page.
+ """
+ page = await self.get_agent_current_page()
+
+ # We no longer force tabs to the foreground as it disrupts user focus
+ # await page.bring_to_front()
+ await page.wait_for_load_state()
+
+ screenshot = await page.screenshot(
+ full_page=full_page,
+ animations='disabled',
+ caret='initial',
+ )
+
+ screenshot_b64 = base64.b64encode(screenshot).decode('utf-8')
+
+ # await self.remove_highlights()
+
+ return screenshot_b64
+
+ @time_execution_async('--remove_highlights')
+ async def remove_highlights(self):
+ """
+ Removes all highlight overlays and labels created by the highlightElement function.
+ Handles cases where the page might be closed or inaccessible.
+ """
+ try:
+ page = await self.get_agent_current_page()
+ await page.evaluate(
+ """
+ try {
+ // Remove the highlight container and all its contents
+ const container = document.getElementById('playwright-highlight-container');
+ if (container) {
+ container.remove();
+ }
+
+ // Remove highlight attributes from elements
+ const highlightedElements = document.querySelectorAll('[browser-user-highlight-id^="playwright-highlight-"]');
+ highlightedElements.forEach(el => {
+ el.removeAttribute('browser-user-highlight-id');
+ });
+ } catch (e) {
+ console.error('Failed to remove highlights:', e);
+ }
+ """
+ )
+ except Exception as e:
+ logger.debug(f'โ Failed to remove highlights (this is usually ok): {str(e)}')
+ # Don't raise the error since this is not critical functionality
+ pass
+
+ # endregion
+
+ # region - User Actions
+
+ @classmethod
+ def _convert_simple_xpath_to_css_selector(cls, xpath: str) -> str:
+ """Converts simple XPath expressions to CSS selectors."""
+ if not xpath:
+ return ''
+
+ # Remove leading slash if present
+ xpath = xpath.lstrip('/')
+
+ # Split into parts
+ parts = xpath.split('/')
+ css_parts = []
+
+ for part in parts:
+ if not part:
+ continue
+
+ # Handle custom elements with colons by escaping them
+ if ':' in part and '[' not in part:
+ base_part = part.replace(':', r'\:')
+ css_parts.append(base_part)
+ continue
+
+ # Handle index notation [n]
+ if '[' in part:
+ base_part = part[: part.find('[')]
+ # Handle custom elements with colons in the base part
+ if ':' in base_part:
+ base_part = base_part.replace(':', r'\:')
+ index_part = part[part.find('[') :]
+
+ # Handle multiple indices
+ indices = [i.strip('[]') for i in index_part.split(']')[:-1]]
+
+ for idx in indices:
+ try:
+ # Handle numeric indices
+ if idx.isdigit():
+ index = int(idx) - 1
+ base_part += f':nth-of-type({index + 1})'
+ # Handle last() function
+ elif idx == 'last()':
+ base_part += ':last-of-type'
+ # Handle position() functions
+ elif 'position()' in idx:
+ if '>1' in idx:
+ base_part += ':nth-of-type(n+2)'
+ except ValueError:
+ continue
+
+ css_parts.append(base_part)
+ else:
+ css_parts.append(part)
+
+ base_selector = ' > '.join(css_parts)
+ return base_selector
+
+ @classmethod
+ @time_execution_sync('--enhanced_css_selector_for_element')
+ def _enhanced_css_selector_for_element(cls, element: DOMElementNode, include_dynamic_attributes: bool = True) -> str:
+ """
+ Creates a CSS selector for a DOM element, handling various edge cases and special characters.
+
+ Args:
+ element: The DOM element to create a selector for
+
+ Returns:
+ A valid CSS selector string
+ """
+ try:
+ # Get base selector from XPath
+ css_selector = cls._convert_simple_xpath_to_css_selector(element.xpath)
+
+ # Handle class attributes
+ if 'class' in element.attributes and element.attributes['class'] and include_dynamic_attributes:
+ # Define a regex pattern for valid class names in CSS
+ valid_class_name_pattern = re.compile(r'^[a-zA-Z_][a-zA-Z0-9_-]*$')
+
+ # Iterate through the class attribute values
+ classes = element.attributes['class'].split()
+ for class_name in classes:
+ # Skip empty class names
+ if not class_name.strip():
+ continue
+
+ # Check if the class name is valid
+ if valid_class_name_pattern.match(class_name):
+ # Append the valid class name to the CSS selector
+ css_selector += f'.{class_name}'
+ else:
+ # Skip invalid class names
+ continue
+
+ # Expanded set of safe attributes that are stable and useful for selection
+ SAFE_ATTRIBUTES = {
+ # Data attributes (if they're stable in your application)
+ 'id',
+ # Standard HTML attributes
+ 'name',
+ 'type',
+ 'placeholder',
+ # Accessibility attributes
+ 'aria-label',
+ 'aria-labelledby',
+ 'aria-describedby',
+ 'role',
+ # Common form attributes
+ 'for',
+ 'autocomplete',
+ 'required',
+ 'readonly',
+ # Media attributes
+ 'alt',
+ 'title',
+ 'src',
+ # Custom stable attributes (add any application-specific ones)
+ 'href',
+ 'target',
+ }
+
+ if include_dynamic_attributes:
+ dynamic_attributes = {
+ 'data-id',
+ 'data-qa',
+ 'data-cy',
+ 'data-testid',
+ }
+ SAFE_ATTRIBUTES.update(dynamic_attributes)
+
+ # Handle other attributes
+ for attribute, value in element.attributes.items():
+ if attribute == 'class':
+ continue
+
+ # Skip invalid attribute names
+ if not attribute.strip():
+ continue
+
+ if attribute not in SAFE_ATTRIBUTES:
+ continue
+
+ # Escape special characters in attribute names
+ safe_attribute = attribute.replace(':', r'\:')
+
+ # Handle different value cases
+ if value == '':
+ css_selector += f'[{safe_attribute}]'
+ elif any(char in value for char in '"\'<>`\n\r\t'):
+ # Use contains for values with special characters
+ # For newline-containing text, only use the part before the newline
+ if '\n' in value:
+ value = value.split('\n')[0]
+ # Regex-substitute *any* whitespace with a single space, then strip.
+ collapsed_value = re.sub(r'\s+', ' ', value).strip()
+ # Escape embedded double-quotes.
+ safe_value = collapsed_value.replace('"', '\\"')
+ css_selector += f'[{safe_attribute}*="{safe_value}"]'
+ else:
+ css_selector += f'[{safe_attribute}="{value}"]'
+
+ return css_selector
+
+ except Exception:
+ # Fallback to a more basic selector if something goes wrong
+ tag_name = element.tag_name or '*'
+ return f"{tag_name}[highlight_index='{element.highlight_index}']"
+
+ @time_execution_async('--is_visible')
+ async def _is_visible(self, element: ElementHandle) -> bool:
+ """
+ Checks if an element is visible on the page.
+ We use our own implementation instead of relying solely on Playwright's is_visible() because
+ of edge cases with CSS frameworks like Tailwind. When elements use Tailwind's 'hidden' class,
+ the computed style may return display as '' (empty string) instead of 'none', causing Playwright
+ to incorrectly consider hidden elements as visible. By additionally checking the bounding box
+ dimensions, we catch elements that have zero width/height regardless of how they were hidden.
+ """
+ is_hidden = await element.is_hidden()
+ bbox = await element.bounding_box()
+
+ return not is_hidden and bbox is not None and bbox['width'] > 0 and bbox['height'] > 0
+
+ @time_execution_async('--get_locate_element')
+ async def get_locate_element(self, element: DOMElementNode) -> ElementHandle | None:
+ current_frame = await self.get_agent_current_page()
+
+ # Start with the target element and collect all parents
+ parents: list[DOMElementNode] = []
+ current = element
+ while current.parent is not None:
+ parent = current.parent
+ parents.append(parent)
+ current = parent
+
+ # Reverse the parents list to process from top to bottom
+ parents.reverse()
+
+ # Process all iframe parents in sequence
+ iframes = [item for item in parents if item.tag_name == 'iframe']
+ for parent in iframes:
+ css_selector = self._enhanced_css_selector_for_element(
+ parent,
+ include_dynamic_attributes=self.config.include_dynamic_attributes,
+ )
+ current_frame = current_frame.frame_locator(css_selector)
+
+ css_selector = self._enhanced_css_selector_for_element(
+ element, include_dynamic_attributes=self.config.include_dynamic_attributes
+ )
+
+ try:
+ if isinstance(current_frame, FrameLocator):
+ element_handle = await current_frame.locator(css_selector).element_handle()
+ return element_handle
+ else:
+ # Try to scroll into view if hidden
+ element_handle = await current_frame.query_selector(css_selector)
+ if element_handle:
+ is_visible = await self._is_visible(element_handle)
+ if is_visible:
+ await element_handle.scroll_into_view_if_needed()
+ return element_handle
+ return None
+ except Exception as e:
+ logger.error(f'โ Failed to locate element: {str(e)}')
+ return None
+
+ @time_execution_async('--get_locate_element_by_xpath')
+ async def get_locate_element_by_xpath(self, xpath: str) -> ElementHandle | None:
+ """
+ Locates an element on the page using the provided XPath.
+ """
+ current_frame = await self.get_agent_current_page()
+
+ try:
+ # Use XPath to locate the element
+ element_handle = await current_frame.query_selector(f'xpath={xpath}')
+ if element_handle:
+ is_visible = await self._is_visible(element_handle)
+ if is_visible:
+ await element_handle.scroll_into_view_if_needed()
+ return element_handle
+ return None
+ except Exception as e:
+ logger.error(f'โ Failed to locate element by XPath {xpath}: {str(e)}')
+ return None
+
+ @time_execution_async('--get_locate_element_by_css_selector')
+ async def get_locate_element_by_css_selector(self, css_selector: str) -> ElementHandle | None:
+ """
+ Locates an element on the page using the provided CSS selector.
+ """
+ current_frame = await self.get_agent_current_page()
+
+ try:
+ # Use CSS selector to locate the element
+ element_handle = await current_frame.query_selector(css_selector)
+ if element_handle:
+ is_visible = await self._is_visible(element_handle)
+ if is_visible:
+ await element_handle.scroll_into_view_if_needed()
+ return element_handle
+ return None
+ except Exception as e:
+ logger.error(f'โ Failed to locate element by CSS selector {css_selector}: {str(e)}')
+ return None
+
+ @time_execution_async('--get_locate_element_by_text')
+ async def get_locate_element_by_text(
+ self, text: str, nth: int | None = 0, element_type: str | None = None
+ ) -> ElementHandle | None:
+ """
+ Locates an element on the page using the provided text.
+ If `nth` is provided, it returns the nth matching element (0-based).
+ If `element_type` is provided, filters by tag name (e.g., 'button', 'span').
+ """
+ current_frame = await self.get_agent_current_page()
+ try:
+ # handle also specific element type or use any type.
+ selector = f'{element_type or "*"}:text("{text}")'
+ elements = await current_frame.query_selector_all(selector)
+ # considering only visible elements
+ elements = [el for el in elements if await self._is_visible(el)]
+
+ if not elements:
+ logger.error(f"No visible element with text '{text}' found.")
+ return None
+
+ if nth is not None:
+ if 0 <= nth < len(elements):
+ element_handle = elements[nth]
+ else:
+ logger.error(f"Visible element with text '{text}' not found at index {nth}.")
+ return None
+ else:
+ element_handle = elements[0]
+
+ is_visible = await self._is_visible(element_handle)
+ if is_visible:
+ await element_handle.scroll_into_view_if_needed()
+ return element_handle
+ except Exception as e:
+ logger.error(f"โ Failed to locate element by text '{text}': {str(e)}")
+ return None
+
+ @time_execution_async('--input_text_element_node')
+ async def _input_text_element_node(self, element_node: DOMElementNode, text: str):
+ """
+ Input text into an element with proper error handling and state management.
+ Handles different types of input fields and ensures proper element state before input.
+ """
+ try:
+ # Highlight before typing
+ # if element_node.highlight_index is not None:
+ # await self._update_state(focus_element=element_node.highlight_index)
+
+ element_handle = await self.get_locate_element(element_node)
+
+ if element_handle is None:
+ raise BrowserError(f'Element: {repr(element_node)} not found')
+
+ # Ensure element is ready for input
+ try:
+ await element_handle.wait_for_element_state('stable', timeout=1000)
+ is_visible = await self._is_visible(element_handle)
+ if is_visible:
+ await element_handle.scroll_into_view_if_needed(timeout=1000)
+ except Exception:
+ pass
+
+ # Get element properties to determine input method
+ tag_handle = await element_handle.get_property('tagName')
+ tag_name = (await tag_handle.json_value()).lower()
+ is_contenteditable = await element_handle.get_property('isContentEditable')
+ readonly_handle = await element_handle.get_property('readOnly')
+ disabled_handle = await element_handle.get_property('disabled')
+
+ readonly = await readonly_handle.json_value() if readonly_handle else False
+ disabled = await disabled_handle.json_value() if disabled_handle else False
+
+ # always click the element first to make sure it's in the focus
+ await element_handle.click()
+ await asyncio.sleep(0.1)
+
+ try:
+ if (await is_contenteditable.json_value() or tag_name == 'input') and not (readonly or disabled):
+ await element_handle.evaluate('el => {el.textContent = ""; el.value = "";}')
+ await element_handle.type(text, delay=5)
+ else:
+ await element_handle.fill(text)
+ except Exception:
+ # last resort fallback, assume it's already focused after we clicked on it,
+ # just simulate keypresses on the entire page
+ await self.get_agent_current_page().keyboard.type(text)
+
+ except Exception as e:
+ logger.debug(f'โ Failed to input text into element: {repr(element_node)}. Error: {str(e)}')
+ raise BrowserError(f'Failed to input text into index {element_node.highlight_index}')
+
+ @time_execution_async('--click_element_node')
+ async def _click_element_node(self, element_node: DOMElementNode) -> str | None:
+ """
+ Optimized method to click an element using xpath.
+ """
+ page = await self.get_agent_current_page()
+
+ try:
+ # Highlight before clicking
+ # if element_node.highlight_index is not None:
+ # await self._update_state(focus_element=element_node.highlight_index)
+
+ element_handle = await self.get_locate_element(element_node)
+
+ if element_handle is None:
+ raise Exception(f'Element: {repr(element_node)} not found')
+
+ async def perform_click(click_func):
+ """Performs the actual click, handling both download
+ and navigation scenarios."""
+ if self.config.save_downloads_path:
+ try:
+ # Try short-timeout expect_download to detect a file download has been been triggered
+ async with page.expect_download(timeout=5000) as download_info:
+ await click_func()
+ download = await download_info.value
+ # Determine file path
+ suggested_filename = download.suggested_filename
+ unique_filename = await self._get_unique_filename(self.config.save_downloads_path, suggested_filename)
+ download_path = os.path.join(self.config.save_downloads_path, unique_filename)
+ await download.save_as(download_path)
+ logger.debug(f'โฌ๏ธ Download triggered. Saved file to: {download_path}')
+ return download_path
+ except TimeoutError:
+ # If no download is triggered, treat as normal click
+ logger.debug('No download triggered within timeout. Checking navigation...')
+ await page.wait_for_load_state()
+ await self._check_and_handle_navigation(page)
+ else:
+ # Standard click logic if no download is expected
+ await click_func()
+ await page.wait_for_load_state()
+ await self._check_and_handle_navigation(page)
+
+ try:
+ return await perform_click(lambda: element_handle.click(timeout=1500))
+ except URLNotAllowedError as e:
+ raise e
+ except Exception:
+ try:
+ return await perform_click(lambda: page.evaluate('(el) => el.click()', element_handle))
+ except URLNotAllowedError as e:
+ raise e
+ except Exception as e:
+ raise Exception(f'Failed to click element: {str(e)}')
+
+ except URLNotAllowedError as e:
+ raise e
+ except Exception as e:
+ raise Exception(f'Failed to click element: {repr(element_node)}. Error: {str(e)}')
+
+ @time_execution_async('--get_tabs_info')
+ async def get_tabs_info(self) -> list[TabInfo]:
+ """Get information about all tabs"""
+ session = await self.get_session()
+
+ tabs_info = []
+ for page_id, page in enumerate(session.context.pages):
+ try:
+ tab_info = TabInfo(page_id=page_id, url=page.url, title=await asyncio.wait_for(page.title(), timeout=1))
+ except TimeoutError:
+ # page.title() can hang forever on tabs that are crashed/disappeared/about:blank
+ # we dont want to try automating those tabs because they will hang the whole script
+ logger.debug('โ Failed to get tab info for tab #%s: %s (ignoring)', page_id, page.url)
+ tab_info = TabInfo(page_id=page_id, url='about:blank', title='ignore this tab and do not use it')
+ tabs_info.append(tab_info)
+
+ return tabs_info
+
+ @time_execution_async('--switch_to_tab')
+ async def switch_to_tab(self, page_id: int) -> None:
+ """Switch to a specific tab by its page_id"""
+ session = await self.get_session()
+ pages = session.context.pages
+
+ if page_id >= len(pages):
+ raise BrowserError(f'No tab found with page_id: {page_id}')
+
+ page = pages[page_id]
+
+ # Check if the tab's URL is allowed before switching
+ if not self._is_url_allowed(page.url):
+ raise BrowserError(f'Cannot switch to tab with non-allowed URL: {page.url}')
+
+ # Update target ID if using CDP
+ if self.browser.config.cdp_url:
+ targets = await self._get_cdp_targets()
+ for target in targets:
+ if target['url'] == page.url:
+ self.state.target_id = target['targetId']
+ break
+
+ # Update both tab references - agent wants this tab, and it's now in the foreground
+ self.agent_current_page = page
+ self.human_current_page = page
+
+ # Bring tab to front and wait for it to load
+ await page.bring_to_front()
+ await page.wait_for_load_state()
+
+ # Set the viewport size for the tab
+ await self.set_viewport_size(page)
+
+ @time_execution_async('--create_new_tab')
+ async def create_new_tab(self, url: str | None = None) -> None:
+ """Create a new tab and optionally navigate to a URL"""
+ if url and not self._is_url_allowed(url):
+ raise BrowserError(f'Cannot create new tab with non-allowed URL: {url}')
+
+ session = await self.get_session()
+ new_page = await session.context.new_page()
+
+ # Update both tab references - agent wants this tab, and it's now in the foreground
+ self.agent_current_page = new_page
+ self.human_current_page = new_page
+
+ await new_page.wait_for_load_state()
+
+ # Set the viewport size for the new tab
+ await self.set_viewport_size(new_page)
+
+ if url:
+ await new_page.goto(url)
+ await self._wait_for_page_and_frames_load(timeout_overwrite=1)
+
+ # Get target ID for new page if using CDP
+ if self.browser.config.cdp_url:
+ targets = await self._get_cdp_targets()
+ for target in targets:
+ if target['url'] == new_page.url:
+ self.state.target_id = target['targetId']
+ break
+
+ # region - Helper methods for easier access to the DOM
+
+ async def get_selector_map(self) -> SelectorMap:
+ session = await self.get_session()
+ if session.cached_state is None:
+ return {}
+ return session.cached_state.selector_map
+
+ async def get_element_by_index(self, index: int) -> ElementHandle | None:
+ selector_map = await self.get_selector_map()
+ element_handle = await self.get_locate_element(selector_map[index])
+ return element_handle
+
+ async def get_dom_element_by_index(self, index: int) -> DOMElementNode:
+ selector_map = await self.get_selector_map()
+ return selector_map[index]
+
+ async def save_cookies(self):
+ """Save current cookies to file"""
+ if self.session and self.session.context and self.config.cookies_file:
+ try:
+ cookies = await self.session.context.cookies()
+ logger.debug(f'๐ช Saving {len(cookies)} cookies to {self.config.cookies_file}')
+
+ # Check if the path is a directory and create it if necessary
+ dirname = os.path.dirname(self.config.cookies_file)
+ if dirname:
+ os.makedirs(dirname, exist_ok=True)
+
+ async with await anyio.open_file(self.config.cookies_file, 'w') as f:
+ await f.write(json.dumps(cookies))
+ except Exception as e:
+ logger.warning(f'โ Failed to save cookies: {str(e)}')
+
+ async def is_file_uploader(self, element_node: DOMElementNode, max_depth: int = 3, current_depth: int = 0) -> bool:
+ """Check if element or its children are file uploaders"""
+ if current_depth > max_depth:
+ return False
+
+ # Check current element
+ is_uploader = False
+
+ if not isinstance(element_node, DOMElementNode):
+ return False
+
+ # Check for file input attributes
+ if element_node.tag_name == 'input':
+ is_uploader = element_node.attributes.get('type') == 'file' or element_node.attributes.get('accept') is not None
+
+ if is_uploader:
+ return True
+
+ # Recursively check children
+ if element_node.children and current_depth < max_depth:
+ for child in element_node.children:
+ if isinstance(child, DOMElementNode):
+ if await self.is_file_uploader(child, max_depth, current_depth + 1):
+ return True
+
+ return False
+
+ async def get_scroll_info(self, page: Page) -> tuple[int, int]:
+ """Get scroll position information for the current page."""
+ scroll_y = await page.evaluate('window.scrollY')
+ viewport_height = await page.evaluate('window.innerHeight')
+ total_height = await page.evaluate('document.documentElement.scrollHeight')
+ pixels_above = scroll_y
+ pixels_below = total_height - (scroll_y + viewport_height)
+ return pixels_above, pixels_below
+
+ async def reset_context(self):
+ """Reset the browser session
+ Call this when you don't want to kill the context but just kill the state
+ """
+ # close all tabs and clear cached state
+ session = await self.get_session()
+
+ pages = session.context.pages
+ for page in pages:
+ await page.close()
+
+ session.cached_state = None
+ self.state.target_id = None
+
+ async def _get_unique_filename(self, directory, filename):
+ """Generate a unique filename by appending (1), (2), etc., if a file already exists."""
+ base, ext = os.path.splitext(filename)
+ counter = 1
+ new_filename = filename
+ while os.path.exists(os.path.join(directory, new_filename)):
+ new_filename = f'{base} ({counter}){ext}'
+ counter += 1
+ return new_filename
+
+ async def _get_cdp_targets(self) -> list[dict]:
+ """Get all CDP targets directly using CDP protocol"""
+ if not self.browser.config.cdp_url or not self.session:
+ return []
+
+ try:
+ pages = self.session.context.pages
+ if not pages:
+ return []
+
+ cdp_session = await pages[0].context.new_cdp_session(pages[0])
+ result = await cdp_session.send('Target.getTargets')
+ await cdp_session.detach()
+ return result.get('targetInfos', [])
+ except Exception as e:
+ logger.debug(f'Failed to get CDP targets: {e}')
+ return []
+
+ async def _resize_window(self, context: PlaywrightBrowserContext) -> None:
+ """Resize the browser window to match the configured size"""
+ try:
+ if not context.pages:
+ return
+
+ page = context.pages[0]
+ window_size = {'width': self.config.window_width, 'height': self.config.window_height}
+
+ # First, set the viewport size
+ await self.set_viewport_size(page)
+
+ # Then, try to set the actual window size using CDP
+ try:
+ cdp_session = await context.new_cdp_session(page)
+
+ # Get the window ID
+ window_id_result = await cdp_session.send('Browser.getWindowForTarget')
+
+ # Set the window bounds
+ await cdp_session.send(
+ 'Browser.setWindowBounds',
+ {
+ 'windowId': window_id_result['windowId'],
+ 'bounds': {
+ 'width': window_size['width'],
+ 'height': window_size['height'] + BROWSER_NAVBAR_HEIGHT, # Add height for browser chrome
+ 'windowState': 'normal', # Ensure window is not minimized/maximized
+ },
+ },
+ )
+
+ await cdp_session.detach()
+ logger.debug(f'Set window size to {window_size["width"]}x{window_size["height"] + BROWSER_NAVBAR_HEIGHT}')
+ except Exception as e:
+ logger.debug(f'CDP window resize failed: {e}')
+
+ # Fallback to using JavaScript
+ try:
+ await page.evaluate(
+ """
+ (width, height) => {
+ window.resizeTo(width, height);
+ }
+ """,
+ window_size['width'],
+ window_size['height'] + BROWSER_NAVBAR_HEIGHT,
+ )
+ logger.debug(
+ f'Used JavaScript to set window size to {window_size["width"]}x{window_size["height"] + BROWSER_NAVBAR_HEIGHT}'
+ )
+ except Exception as e:
+ logger.debug(f'JavaScript window resize failed: {e}')
+
+ logger.debug(f'Attempted to resize window to {window_size["width"]}x{window_size["height"]}')
+ except Exception as e:
+ logger.debug(f'Failed to resize browser window: {e}')
+ # Non-critical error, continue execution
+
+ async def wait_for_element(self, selector: str, timeout: float) -> None:
+ """
+ Waits for an element matching the given CSS selector to become visible.
+
+ Args:
+ selector (str): The CSS selector of the element.
+ timeout (float): The maximum time to wait for the element to be visible (in milliseconds).
+
+ Raises:
+ TimeoutError: If the element does not become visible within the specified timeout.
+ """
+ page = await self.get_agent_current_page()
+ await page.wait_for_selector(selector, state='visible', timeout=timeout)
diff --git a/browser-use/browser_use/browser/dolphin_service.py b/browser-use/browser_use/browser/dolphin_service.py
new file mode 100644
index 0000000..b90585c
--- /dev/null
+++ b/browser-use/browser_use/browser/dolphin_service.py
@@ -0,0 +1,348 @@
+import logging
+import os
+
+import aiohttp
+from playwright.async_api import Page, async_playwright
+
+from browser_use.browser.service import Browser
+from browser_use.browser.views import BrowserState, TabInfo
+
+logger = logging.getLogger(__name__)
+
+
+class DolphinBrowser(Browser):
+ """A class for managing Dolphin Anty browser sessions using Playwright"""
+
+ def __init__(self, headless: bool = False, keep_open: bool = False):
+ """
+ Initialize the DolphinBrowser instance.
+
+ Args:
+ headless (bool): Run browser in headless mode (default: False).
+ keep_open (bool): Keep browser open after finishing tasks (default: False).
+ """
+ # Retrieve environment variables for API connection
+ self.api_token = os.getenv('DOLPHIN_API_TOKEN')
+ self.api_url = os.getenv('DOLPHIN_API_URL', 'http://localhost:3001/v1.0')
+ self.profile_id = os.getenv('DOLPHIN_PROFILE_ID')
+
+ # Initialize internal attributes
+ self.playwright = None
+ self.browser = None
+ self.context = None
+ self.page = None
+ self.headless = headless
+ self.keep_open = keep_open
+ self._pages: list[Page] = [] # List to store open pages
+ self.session = None
+ self.cached_state = None
+
+ async def get_current_page(self) -> Page:
+ """
+ Get the currently active page.
+
+ Raises:
+ Exception: If no active page is available.
+ """
+ if not self.page:
+ raise Exception('No active page. Browser might not be connected.')
+ return self.page
+
+ async def create_new_tab(self, url: str | None = None) -> None:
+ """
+ Create a new tab and optionally navigate to a given URL.
+
+ Args:
+ url (str, optional): URL to navigate to after creating the tab. Defaults to None.
+
+ Raises:
+ Exception: If browser context is not initialized or navigation fails.
+ """
+ if not self.context:
+ raise Exception('Browser context not initialized')
+
+ # Create new page (tab) in the current browser context
+ new_page = await self.context.new_page()
+ self._pages.append(new_page)
+ self.page = new_page # Set as current page
+
+ if url:
+ try:
+ # Navigate to the URL and wait for the page to load
+ await new_page.goto(url, wait_until='networkidle')
+ await self.wait_for_page_load()
+ except Exception as e:
+ logger.error(f'Failed to navigate to URL {url}: {str(e)}')
+ raise
+
+ async def switch_to_tab(self, page_id: int) -> None:
+ """
+ Switch to a specific tab by its page ID.
+
+ Args:
+ page_id (int): The index of the tab to switch to.
+
+ Raises:
+ Exception: If the tab index is out of range or no tabs are available.
+ """
+ if not self._pages:
+ raise Exception('No tabs available')
+
+ # Handle negative indices (e.g., -1 for last tab)
+ if page_id < 0:
+ page_id = len(self._pages) + page_id
+
+ if page_id >= len(self._pages) or page_id < 0:
+ raise Exception(f'Tab index {page_id} out of range')
+
+ # Set the current page to the selected tab
+ self.page = self._pages[page_id]
+ await self.page.bring_to_front() # Bring tab to the front
+ await self.wait_for_page_load()
+
+ async def get_tabs_info(self) -> list[TabInfo]:
+ """
+ Get information about all open tabs.
+
+ Returns:
+ list: A list of TabInfo objects containing details about each tab.
+ """
+ tabs_info = []
+ for idx, page in enumerate(self._pages):
+ tab_info = TabInfo(
+ page_id=idx,
+ url=page.url,
+ title=await page.title(), # Fetch the title of the page
+ )
+ tabs_info.append(tab_info)
+ return tabs_info
+
+ async def wait_for_page_load(self, timeout: int = 30000):
+ """
+ Wait for the page to load completely.
+
+ Args:
+ timeout (int): Maximum time to wait for page load in milliseconds (default: 30000ms).
+
+ Raises:
+ Exception: If the page fails to load within the specified timeout.
+ """
+ if self.page:
+ try:
+ await self.page.wait_for_load_state('networkidle', timeout=timeout)
+ except Exception as e:
+ logger.warning(f'Wait for page load timeout: {str(e)}')
+
+ async def get_session(self):
+ """
+ Get the current session.
+
+ Returns:
+ DolphinBrowser: The current DolphinBrowser instance.
+
+ Raises:
+ Exception: If the browser is not connected.
+ """
+ if not self.browser:
+ raise Exception('Browser not connected. Call connect() first.')
+ self.session = self
+ return self
+
+ async def authenticate(self):
+ """
+ Authenticate with Dolphin Anty API using the API token.
+
+ Raises:
+ Exception: If authentication fails.
+ """
+ async with aiohttp.ClientSession() as session:
+ auth_url = f'{self.api_url}/auth/login-with-token'
+ auth_data = {'token': self.api_token}
+ async with session.post(auth_url, json=auth_data) as response:
+ if not response.ok:
+ raise Exception(f'Failed to authenticate with Dolphin Anty: {await response.text()}')
+ return await response.json()
+
+ async def get_browser_profiles(self):
+ """
+ Get a list of available browser profiles from Dolphin Anty.
+
+ Returns:
+ list: A list of browser profiles.
+
+ Raises:
+ Exception: If fetching the browser profiles fails.
+ """
+ # Authenticate before fetching profiles
+ await self.authenticate()
+
+ async with aiohttp.ClientSession() as session:
+ headers = {'Authorization': f'Bearer {self.api_token}'}
+ async with session.get(f'{self.api_url}/browser_profiles', headers=headers) as response:
+ if not response.ok:
+ raise Exception(f'Failed to get browser profiles: {await response.text()}')
+ data = await response.json()
+ return data.get('data', []) # Return the profiles array from the response
+
+ async def start_profile(self, profile_id: str | None = None, headless: bool = False) -> dict:
+ """
+ Start a browser profile on Dolphin Anty.
+
+ Args:
+ profile_id (str, optional): Profile ID to start (defaults to the one set in the environment).
+ headless (bool): Run browser in headless mode (default: False).
+
+ Returns:
+ dict: Information about the started profile.
+
+ Raises:
+ ValueError: If no profile ID is provided and no default is set.
+ Exception: If starting the profile fails.
+ """
+ # Authenticate before starting the profile
+ await self.authenticate()
+
+ profile_id = profile_id or self.profile_id
+ if not profile_id:
+ raise ValueError('No profile ID provided')
+
+ url = f'{self.api_url}/browser_profiles/{profile_id}/start'
+ params = {'automation': 1}
+ if headless:
+ params['headless'] = 1
+
+ async with aiohttp.ClientSession() as session:
+ async with session.get(url, params=params) as response:
+ if not response.ok:
+ raise Exception(f'Failed to start profile: {await response.text()}')
+ return await response.json()
+
+ async def stop_profile(self, profile_id: str | None = None):
+ """
+ Stop a browser profile on Dolphin Anty.
+
+ Args:
+ profile_id (str, optional): Profile ID to stop (defaults to the one set in the environment).
+
+ Returns:
+ dict: Information about the stopped profile.
+
+ Raises:
+ ValueError: If no profile ID is provided and no default is set.
+ """
+ # Authenticate before stopping the profile
+ await self.authenticate()
+
+ profile_id = profile_id or self.profile_id
+ if not profile_id:
+ raise ValueError('No profile ID provided')
+
+ url = f'{self.api_url}/browser_profiles/{profile_id}/stop'
+ async with aiohttp.ClientSession() as session:
+ async with session.get(url) as response:
+ return await response.json()
+
+ async def connect(self, profile_id: str | None = None):
+ """
+ Connect to a running browser profile using Playwright.
+
+ Args:
+ profile_id (str, optional): Profile ID to connect to (defaults to the one set in the environment).
+
+ Returns:
+ PlaywrightBrowser: The connected browser instance.
+
+ Raises:
+ Exception: If authentication or profile connection fails.
+ """
+ # Authenticate before connecting to the profile
+ await self.authenticate()
+
+ # Start the browser profile
+ profile_data = await self.start_profile(profile_id)
+
+ if not profile_data.get('success'):
+ raise Exception(f'Failed to start profile: {profile_data}')
+
+ automation = profile_data['automation']
+ port = automation['port']
+ ws_endpoint = automation['wsEndpoint']
+ ws_url = f'ws://127.0.0.1:{port}{ws_endpoint}'
+
+ # Use Playwright to connect to the browser's WebSocket endpoint
+ self.playwright = await async_playwright().start()
+ self.browser = await self.playwright.chromium.connect_over_cdp(ws_url)
+
+ # Get or create a browser context and page
+ contexts = self.browser.contexts
+ self.context = contexts[0] if contexts else await self.browser.new_context()
+ pages = self.context.pages
+ self.page = pages[0] if pages else await self.context.new_page()
+
+ self._pages = [self.page] # Initialize pages list with the first page
+
+ return self.browser
+
+ async def close(self, force: bool = False):
+ """
+ Close the browser connection and clean up resources.
+
+ Args:
+ force (bool): If True, forcefully stop the associated profile (default: False).
+ """
+ try:
+ # Close all open pages
+ if self._pages:
+ for page in self._pages:
+ try:
+ await page.close()
+ except BaseException:
+ pass
+ self._pages = []
+
+ # Close the browser and Playwright instance
+ if self.browser:
+ await self.browser.close()
+
+ if self.playwright:
+ await self.playwright.stop()
+
+ if force:
+ await self.stop_profile() # Force stop the profile
+ except Exception as e:
+ logger.error(f'Error during browser cleanup: {str(e)}')
+
+ async def get_current_state(self) -> BrowserState:
+ """
+ Get the current state of the browser (URL, content, viewport size, tabs).
+
+ Returns:
+ BrowserState: The current state of the browser.
+
+ Raises:
+ Exception: If no active page is available.
+ """
+ if not self.page:
+ raise Exception('No active page')
+
+ # Get page content and viewport size
+ content = await self.page.content()
+ viewport_size = await self.page.viewport_size()
+
+ # Create and return the current browser state
+ state = BrowserState(
+ url=self.page.url,
+ content=content,
+ viewport_height=viewport_size['height'] if viewport_size else 0,
+ viewport_width=viewport_size['width'] if viewport_size else 0,
+ tabs=await self.get_tabs_info(),
+ )
+
+ # Cache and return the state
+ self.cached_state = state
+ return state
+
+ def __del__(self):
+ """Clean up resources when the DolphinBrowser instance is deleted."""
+ # No need to handle session cleanup as we're using self as session
+ pass
diff --git a/browser-use/browser_use/browser/tests/httpx_client_test.py b/browser-use/browser_use/browser/tests/httpx_client_test.py
new file mode 100644
index 0000000..e7b7644
--- /dev/null
+++ b/browser-use/browser_use/browser/tests/httpx_client_test.py
@@ -0,0 +1,39 @@
+import httpx
+import pytest
+
+from browser_use.browser.browser import Browser, BrowserConfig
+
+
+@pytest.mark.asyncio
+async def test_browser_close_doesnt_affect_external_httpx_clients():
+ """
+ Test that Browser.close() doesn't close HTTPX clients created outside the Browser instance.
+ This test demonstrates the issue where Browser.close() is closing all HTTPX clients.
+ """
+ # Create an external HTTPX client that should remain open
+ external_client = httpx.AsyncClient()
+
+ # Create a Browser instance
+ browser = Browser(config=BrowserConfig(headless=True))
+
+ # Close the browser (which should trigger cleanup_httpx_clients)
+ await browser.close()
+
+ # Check if the external client is still usable
+ try:
+ # If the client is closed, this will raise RuntimeError
+ # Using a simple HEAD request to a reliable URL
+ await external_client.head('https://www.example.com', timeout=2.0)
+ client_is_closed = False
+ except RuntimeError as e:
+ # If we get "Cannot send a request, as the client has been closed"
+ client_is_closed = 'client has been closed' in str(e)
+ except Exception:
+ # Any other exception means the client is not closed but request failed
+ client_is_closed = False
+ finally:
+ # Always clean up our test client properly
+ await external_client.aclose()
+
+ # Our external client should not be closed by browser.close()
+ assert not client_is_closed, 'External HTTPX client was incorrectly closed by Browser.close()'
diff --git a/browser-use/browser_use/browser/tests/screenshot_test.py b/browser-use/browser_use/browser/tests/screenshot_test.py
new file mode 100644
index 0000000..b55fdf8
--- /dev/null
+++ b/browser-use/browser_use/browser/tests/screenshot_test.py
@@ -0,0 +1,36 @@
+import asyncio
+import base64
+
+import pytest
+
+from browser_use.browser.browser import Browser, BrowserConfig
+
+
+async def test_take_full_page_screenshot():
+ browser = Browser(config=BrowserConfig(headless=False, disable_security=True))
+ try:
+ async with await browser.new_context() as context:
+ page = await context.get_current_page()
+ # Go to a test page
+ await page.goto('https://example.com')
+
+ await asyncio.sleep(3)
+ # Take full page screenshot
+ screenshot_b64 = await context.take_screenshot(full_page=True)
+ await asyncio.sleep(3)
+ # Verify screenshot is not empty and is valid base64
+ assert screenshot_b64 is not None
+ assert isinstance(screenshot_b64, str)
+ assert len(screenshot_b64) > 0
+
+ # Test we can decode the base64 string
+ try:
+ base64.b64decode(screenshot_b64)
+ except Exception as e:
+ pytest.fail(f'Failed to decode base64 screenshot: {str(e)}')
+ finally:
+ await browser.close()
+
+
+if __name__ == '__main__':
+ asyncio.run(test_take_full_page_screenshot())
diff --git a/browser-use/browser_use/browser/tests/test_clicks.py b/browser-use/browser_use/browser/tests/test_clicks.py
new file mode 100644
index 0000000..e3b9712
--- /dev/null
+++ b/browser-use/browser_use/browser/tests/test_clicks.py
@@ -0,0 +1,96 @@
+import asyncio
+import json
+
+import anyio
+import pytest
+
+from browser_use.browser.browser import Browser, BrowserConfig
+from browser_use.dom.views import DOMBaseNode, DOMElementNode, DOMTextNode
+from browser_use.utils import time_execution_sync
+
+
+class ElementTreeSerializer:
+ @staticmethod
+ def dom_element_node_to_json(element_tree: DOMElementNode) -> dict:
+ def node_to_dict(node: DOMBaseNode) -> dict:
+ if isinstance(node, DOMTextNode):
+ return {'type': 'text', 'text': node.text}
+ elif isinstance(node, DOMElementNode):
+ return {
+ 'type': 'element',
+ 'tag_name': node.tag_name,
+ 'attributes': node.attributes,
+ 'highlight_index': node.highlight_index,
+ 'children': [node_to_dict(child) for child in node.children],
+ }
+ return {}
+
+ return node_to_dict(element_tree)
+
+
+# run with: pytest browser_use/browser/tests/test_clicks.py
+@pytest.mark.asyncio
+async def test_highlight_elements():
+ browser = Browser(config=BrowserConfig(headless=False, disable_security=True))
+
+ async with await browser.new_context() as context:
+ page = await context.get_current_page()
+ # await page.goto('https://immobilienscout24.de')
+ # await page.goto('https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/service-plans')
+ # await page.goto('https://google.com/search?q=elon+musk')
+ # await page.goto('https://kayak.com')
+ # await page.goto('https://www.w3schools.com/tags/tryit.asp?filename=tryhtml_iframe')
+ # await page.goto('https://dictionary.cambridge.org')
+ # await page.goto('https://github.com')
+ await page.goto('https://huggingface.co/')
+
+ await asyncio.sleep(1)
+
+ while True:
+ try:
+ # await asyncio.sleep(10)
+ state = await context.get_state(True)
+
+ async with await anyio.open_file('./tmp/page.json', 'w') as f:
+ await f.write(
+ json.dumps(
+ ElementTreeSerializer.dom_element_node_to_json(state.element_tree),
+ indent=1,
+ )
+ )
+
+ # await time_execution_sync('highlight_selector_map_elements')(
+ # browser.highlight_selector_map_elements
+ # )(state.selector_map)
+
+ # Find and print duplicate XPaths
+ xpath_counts = {}
+ if not state.selector_map:
+ continue
+ for selector in state.selector_map.values():
+ xpath = selector.xpath
+ if xpath in xpath_counts:
+ xpath_counts[xpath] += 1
+ else:
+ xpath_counts[xpath] = 1
+
+ print('\nDuplicate XPaths found:')
+ for xpath, count in xpath_counts.items():
+ if count > 1:
+ print(f'XPath: {xpath}')
+ print(f'Count: {count}\n')
+
+ print(list(state.selector_map.keys()), 'Selector map keys')
+ print(state.element_tree.clickable_elements_to_string())
+ action = input('Select next action: ')
+
+ await time_execution_sync('remove_highlight_elements')(context.remove_highlights)()
+
+ node_element = state.selector_map[int(action)]
+
+ # check if index of selector map are the same as index of items in dom_items
+
+ await context._click_element_node(node_element)
+
+ except Exception as e:
+ print(e)
diff --git a/browser-use/browser_use/browser/utils/screen_resolution.py b/browser-use/browser_use/browser/utils/screen_resolution.py
new file mode 100644
index 0000000..2607970
--- /dev/null
+++ b/browser-use/browser_use/browser/utils/screen_resolution.py
@@ -0,0 +1,41 @@
+import sys
+
+
+def get_screen_resolution():
+ if sys.platform == 'darwin': # macOS
+ try:
+ from AppKit import NSScreen
+
+ screen = NSScreen.mainScreen().frame()
+ return {'width': int(screen.size.width), 'height': int(screen.size.height)}
+ except ImportError:
+ print('AppKit is not available. Make sure you are running this on macOS with pyobjc installed.')
+ except Exception as e:
+ print(f'Error retrieving macOS screen resolution: {e}')
+ return {'width': 2560, 'height': 1664}
+
+ else: # Windows & Linux
+ try:
+ from screeninfo import get_monitors
+
+ monitors = get_monitors()
+ if not monitors:
+ raise Exception('No monitors detected.')
+ monitor = monitors[0]
+ return {'width': monitor.width, 'height': monitor.height}
+ except ImportError:
+ print("screeninfo package not found. Install it using 'pip install screeninfo'.")
+ except Exception as e:
+ print(f'Error retrieving screen resolution: {e}')
+
+ return {'width': 1920, 'height': 1080}
+
+
+def get_window_adjustments():
+ """Returns recommended x, y offsets for window positioning"""
+ if sys.platform == 'darwin': # macOS
+ return -4, 24 # macOS has a small title bar, no border
+ elif sys.platform == 'win32': # Windows
+ return -8, 0 # Windows has a border on the left
+ else: # Linux
+ return 0, 0
diff --git a/browser-use/browser_use/browser/views.py b/browser-use/browser_use/browser/views.py
new file mode 100644
index 0000000..c5468c2
--- /dev/null
+++ b/browser-use/browser_use/browser/views.py
@@ -0,0 +1,54 @@
+from dataclasses import dataclass, field
+from typing import Any
+
+from pydantic import BaseModel
+
+from browser_use.dom.history_tree_processor.service import DOMHistoryElement
+from browser_use.dom.views import DOMState
+
+
+# Pydantic
+class TabInfo(BaseModel):
+ """Represents information about a browser tab"""
+
+ page_id: int
+ url: str
+ title: str
+ parent_page_id: int | None = None # parent page that contains this popup or cross-origin iframe
+
+
+@dataclass
+class BrowserState(DOMState):
+ url: str
+ title: str
+ tabs: list[TabInfo]
+ screenshot: str | None = None
+ pixels_above: int = 0
+ pixels_below: int = 0
+ browser_errors: list[str] = field(default_factory=list)
+
+
+@dataclass
+class BrowserStateHistory:
+ url: str
+ title: str
+ tabs: list[TabInfo]
+ interacted_element: list[DOMHistoryElement | None] | list[None]
+ screenshot: str | None = None
+
+ def to_dict(self) -> dict[str, Any]:
+ data = {}
+ data['tabs'] = [tab.model_dump() for tab in self.tabs]
+ data['screenshot'] = self.screenshot
+ data['interacted_element'] = [el.to_dict() if el else None for el in self.interacted_element]
+ data['url'] = self.url
+ data['title'] = self.title
+ return data
+
+
+class BrowserError(Exception):
+ """Base class for all browser errors"""
+
+
+class URLNotAllowedError(BrowserError):
+ """Error raised when a URL is not allowed"""
diff --git a/browser-use/browser_use/cli.py b/browser-use/browser_use/cli.py
new file mode 100644
index 0000000..435b54b
--- /dev/null
+++ b/browser-use/browser_use/cli.py
@@ -0,0 +1,1392 @@
+import asyncio
+import json
+import logging
+import os
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+try:
+ import click
+ from textual import events
+ from textual.app import App, ComposeResult
+ from textual.binding import Binding
+ from textual.containers import Container, HorizontalGroup, VerticalScroll
+ from textual.widgets import Footer, Header, Input, Label, Link, RichLog, Static
+except ImportError:
+ print('โ ๏ธ CLI addon is not installed. Please install it with: `pip install browser-use[cli]` and try again.')
+ sys.exit(1)
+
+import langchain_anthropic
+import langchain_google_genai
+import langchain_openai
+
+try:
+ import readline
+
+ READLINE_AVAILABLE = True
+except ImportError:
+ # readline not available on Windows by default
+ READLINE_AVAILABLE = False
+
+from browser_use import Agent, Browser, BrowserConfig, BrowserContextConfig, Controller
+from browser_use.agent.views import AgentSettings
+from browser_use.logging_config import addLoggingLevel
+
+# Paths
+USER_CONFIG_DIR = Path.home() / '.config' / 'browseruse'
+USER_CONFIG_FILE = USER_CONFIG_DIR / 'config.json'
+CHROME_PROFILES_DIR = USER_CONFIG_DIR / 'profiles'
+USER_DATA_DIR = CHROME_PROFILES_DIR / 'default'
+
+# Default User settings
+MAX_HISTORY_LENGTH = 100
+
+# Ensure directories exist
+USER_CONFIG_FILE.parent.mkdir(parents=True, exist_ok=True)
+USER_DATA_DIR.mkdir(parents=True, exist_ok=True)
+
+
+# Logo components with styling for rich panels
+BROWSER_LOGO = """
+ [white] ++++++ +++++++++ [/]
+ [white] +++ +++++ +++ [/]
+ [white] ++ ++++ ++ ++ [/]
+ [white] ++ +++ +++ ++ [/]
+ [white] ++++ +++ [/]
+ [white] +++ +++ [/]
+ [white] +++ +++ [/]
+ [white] ++ +++ +++ ++ [/]
+ [white] ++ ++++ ++ ++ [/]
+ [white] +++ ++++++ +++ [/]
+ [white] ++++++ +++++++ [/]
+
+[white]โโโโโโโ โโโโโโโ โโโโโโโ โโโ โโโโโโโโโโโโโโโโโโโโโโโโโโ[/] [darkorange]โโโ โโโโโโโโโโโโโโโโโโโ[/]
+[white]โโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ[/] [darkorange]โโโ โโโโโโโโโโโโโโโโโโโ[/]
+[white]โโโโโโโโโโโโโโโโโโโ โโโโโโ โโ โโโโโโโโโโโโโโโโโ โโโโโโโโ[/] [darkorange]โโโ โโโโโโโโโโโโโโโโโ[/]
+[white]โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโ[/] [darkorange]โโโ โโโโโโโโโโโโโโโโโ[/]
+[white]โโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโ[/] [darkorange]โโโโโโโโโโโโโโโโโโโโโโโโโ[/]
+[white]โโโโโโโ โโโ โโโ โโโโโโโ โโโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโ[/] [darkorange]โโโโโโโ โโโโโโโโโโโโโโโโ[/]
+"""
+
+
+# Common UI constants
+TEXTUAL_BORDER_STYLES = {'logo': 'blue', 'info': 'blue', 'input': 'orange3', 'working': 'yellow', 'completion': 'green'}
+
+
+def get_default_config() -> dict[str, Any]:
+ """Return default configuration dictionary."""
+ return {
+ 'model': {
+ 'name': None,
+ 'temperature': 0.0,
+ 'api_keys': {
+ 'OPENAI_API_KEY': os.getenv('OPENAI_API_KEY', ''),
+ 'ANTHROPIC_API_KEY': os.getenv('ANTHROPIC_API_KEY', ''),
+ 'GOOGLE_API_KEY': os.getenv('GOOGLE_API_KEY', ''),
+ 'DEEPSEEK_API_KEY': os.getenv('DEEPSEEK_API_KEY', ''),
+ 'GROK_API_KEY': os.getenv('GROK_API_KEY', ''),
+ },
+ },
+ 'agent': {}, # AgentSettings will use defaults
+ 'browser': {
+ 'headless': True,
+ },
+ 'browser_context': {
+ 'keep_alive': True,
+ 'ignore_https_errors': False,
+ },
+ 'command_history': [],
+ }
+
+
+def load_user_config() -> dict[str, Any]:
+ """Load user configuration from file."""
+ if not USER_CONFIG_FILE.exists():
+ # Create default config
+ config = get_default_config()
+ save_user_config(config)
+ return config
+
+ try:
+ with open(USER_CONFIG_FILE) as f:
+ data = json.load(f)
+ # Ensure data is a dictionary, not a list
+ if isinstance(data, list):
+ # If it's a list, it's probably just command history from previous version
+ config = get_default_config()
+ config['command_history'] = data # Use the list as command history
+ return config
+ return data
+ except (json.JSONDecodeError, FileNotFoundError):
+ # If file is corrupted, start with empty config
+ return get_default_config()
+
+
+def save_user_config(config: dict[str, Any]) -> None:
+ """Save user configuration to file."""
+ # Ensure command history doesn't exceed maximum length
+ if 'command_history' in config and isinstance(config['command_history'], list):
+ if len(config['command_history']) > MAX_HISTORY_LENGTH:
+ config['command_history'] = config['command_history'][-MAX_HISTORY_LENGTH:]
+
+ with open(USER_CONFIG_FILE, 'w') as f:
+ json.dump(config, f, indent=2)
+
+
+def update_config_with_click_args(config: dict[str, Any], ctx: click.Context) -> dict[str, Any]:
+ """Update configuration with command-line arguments."""
+ # Ensure required sections exist
+ if 'model' not in config:
+ config['model'] = {}
+ if 'browser' not in config:
+ config['browser'] = {}
+ if 'browser_context' not in config:
+ config['browser_context'] = {}
+
+ # Update configuration with command-line args if provided
+ if ctx.params.get('model'):
+ config['model']['name'] = ctx.params['model']
+ if ctx.params.get('headless') is not None:
+ config['browser']['headless'] = ctx.params['headless']
+ if ctx.params.get('window_width'):
+ config['browser']['window_width'] = ctx.params['window_width']
+ if 'viewport_width' in config['browser_context']:
+ config['browser_context']['viewport_width'] = ctx.params['window_width']
+ if ctx.params.get('window_height'):
+ config['browser']['window_height'] = ctx.params['window_height']
+ if 'viewport_height' in config['browser_context']:
+ config['browser_context']['viewport_height'] = ctx.params['window_height']
+
+ return config
+
+
+def setup_readline_history(history: list[str]) -> None:
+ """Set up readline with command history."""
+ if not READLINE_AVAILABLE:
+ return
+
+ # Add history items to readline
+ for item in history:
+ readline.add_history(item)
+
+
+def get_llm(config: dict[str, Any]):
+ """Get the language model based on config and available API keys."""
+ # Set API keys from config if available
+ api_keys = config.get('model', {}).get('api_keys', {})
+ model_name = config.get('model', {}).get('name')
+ temperature = config.get('model', {}).get('temperature', 0.0)
+
+ # Set environment variables if they're in the config but not in the environment
+ if api_keys.get('openai') and not os.getenv('OPENAI_API_KEY'):
+ os.environ['OPENAI_API_KEY'] = api_keys['openai']
+ if api_keys.get('anthropic') and not os.getenv('ANTHROPIC_API_KEY'):
+ os.environ['ANTHROPIC_API_KEY'] = api_keys['anthropic']
+ if api_keys.get('google') and not os.getenv('GOOGLE_API_KEY'):
+ os.environ['GOOGLE_API_KEY'] = api_keys['google']
+
+ if model_name:
+ if model_name.startswith('gpt'):
+ if not os.getenv('OPENAI_API_KEY'):
+ print('โ ๏ธ OpenAI API key not found. Please update your config or set OPENAI_API_KEY environment variable.')
+ sys.exit(1)
+ return langchain_openai.ChatOpenAI(model=model_name, temperature=temperature)
+ elif model_name.startswith('claude'):
+ if not os.getenv('ANTHROPIC_API_KEY'):
+ print('โ ๏ธ Anthropic API key not found. Please update your config or set ANTHROPIC_API_KEY environment variable.')
+ sys.exit(1)
+ return langchain_anthropic.ChatAnthropic(model=model_name, temperature=temperature)
+ elif model_name.startswith('gemini'):
+ if not os.getenv('GOOGLE_API_KEY'):
+ print('โ ๏ธ Google API key not found. Please update your config or set GOOGLE_API_KEY environment variable.')
+ sys.exit(1)
+ return langchain_google_genai.ChatGoogleGenerativeAI(model=model_name, temperature=temperature)
+
+ # Auto-detect based on available API keys
+ if os.getenv('OPENAI_API_KEY'):
+ return langchain_openai.ChatOpenAI(model='gpt-4o', temperature=temperature)
+ elif os.getenv('ANTHROPIC_API_KEY'):
+ return langchain_anthropic.ChatAnthropic(model='claude-3.5-sonnet-exp', temperature=temperature)
+ elif os.getenv('GOOGLE_API_KEY'):
+ return langchain_google_genai.ChatGoogleGenerativeAI(model='gemini-2.0-flash-lite', temperature=temperature)
+ else:
+ print(
+ 'โ ๏ธ No API keys found. Please update your config or set one of: OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_API_KEY.'
+ )
+ sys.exit(1)
+
+
+class RichLogHandler(logging.Handler):
+ """Custom logging handler that redirects logs to a RichLog widget."""
+
+ def __init__(self, rich_log: RichLog):
+ super().__init__()
+ self.rich_log = rich_log
+
+ def emit(self, record):
+ try:
+ msg = self.format(record)
+ self.rich_log.write(msg)
+ except Exception:
+ self.handleError(record)
+
+
+class BrowserUseApp(App):
+ """Browser-use TUI application."""
+
+ # Make it an inline app instead of fullscreen
+ # MODES = {"light"} # Ensure app is inline, not fullscreen
+
+ CSS = """
+ #main-container {
+ height: 100%;
+ layout: vertical;
+ }
+
+ #logo-panel, #links-panel, #paths-panel, #info-panels {
+ border: solid $primary;
+ margin: 0 0 0 0;
+ padding: 0;
+ }
+
+ #info-panels {
+ display: none;
+ layout: vertical;
+ height: auto;
+ min-height: 5;
+ }
+
+ #top-panels {
+ layout: horizontal;
+ height: auto;
+ width: 100%;
+ min-height: 5;
+ }
+
+ #browser-panel, #model-panel {
+ width: 1fr;
+ height: auto;
+ border: solid $primary-darken-2;
+ padding: 1;
+ overflow: auto;
+ margin: 0 1 0 0;
+ padding: 1;
+ }
+
+ #tasks-panel {
+ width: 100%;
+ height: 1fr;
+ min-height: 20;
+ max-height: 60vh;
+ border: solid $primary-darken-2;
+ padding: 1;
+ overflow-y: scroll;
+ margin: 1 0 0 0;
+ }
+
+ #browser-panel {
+ border-left: solid $primary-darken-2;
+ }
+
+ #results-container {
+ display: none;
+ }
+
+ #logo-panel {
+ width: 100%;
+ height: auto;
+ content-align: center middle;
+ text-align: center;
+ }
+
+ #links-panel {
+ width: 100%;
+ padding: 1;
+ border: solid $primary;
+ height: auto;
+ }
+
+ .link-white {
+ color: white;
+ }
+
+ .link-purple {
+ color: purple;
+ }
+
+ .link-magenta {
+ color: magenta;
+ }
+
+ .link-green {
+ color: green;
+ }
+
+ HorizontalGroup {
+ height: auto;
+ }
+
+ .link-label {
+ width: auto;
+ }
+
+ .link-url {
+ width: auto;
+ }
+
+ .link-row {
+ width: 100%;
+ height: auto;
+ }
+
+ #paths-panel {
+ color: $text-muted;
+ }
+
+ #task-input-container {
+ border: solid $accent;
+ padding: 1;
+ margin-bottom: 1;
+ height: auto;
+ dock: bottom;
+ }
+
+ #task-label {
+ color: $accent;
+ padding-bottom: 1;
+ }
+
+ #task-input {
+ width: 100%;
+ }
+
+ #working-panel {
+ border: solid $warning;
+ padding: 1;
+ margin: 1 0;
+ }
+
+ #completion-panel {
+ border: solid $success;
+ padding: 1;
+ margin: 1 0;
+ }
+
+ #results-container {
+ height: 1fr;
+ overflow: auto;
+ border: none;
+ }
+
+ #results-log {
+ height: auto;
+ overflow-y: scroll;
+ background: $surface;
+ color: $text;
+ width: 100%;
+ }
+
+ .log-entry {
+ margin: 0;
+ padding: 0;
+ }
+
+ #browser-info, #model-info, #tasks-info {
+ height: auto;
+ margin: 0;
+ padding: 0;
+ background: transparent;
+ overflow-y: auto;
+ min-height: 5;
+ }
+ """
+
+ BINDINGS = [
+ Binding('ctrl+c', 'quit', 'Quit', priority=True, show=True),
+ Binding('ctrl+q', 'quit', 'Quit', priority=True),
+ Binding('ctrl+d', 'quit', 'Quit', priority=True),
+ Binding('up', 'input_history_prev', 'Previous command', show=False),
+ Binding('down', 'input_history_next', 'Next command', show=False),
+ ]
+
+ def __init__(self, config: dict[str, Any], *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.config = config
+ self.browser = None # Will be set before app.run_async()
+ self.controller = None # Will be set before app.run_async()
+ self.agent = None
+ self.llm = None # Will be set before app.run_async()
+ self.task_history = config.get('command_history', [])
+ # Track current position in history for up/down navigation
+ self.history_index = len(self.task_history)
+
+ def setup_richlog_logging(self) -> None:
+ """Set up logging to redirect to RichLog widget instead of stdout."""
+ # Try to add RESULT level if it doesn't exist
+ try:
+ addLoggingLevel('RESULT', 35)
+ except AttributeError:
+ pass # Level already exists, which is fine
+
+ # Get the RichLog widget
+ rich_log = self.query_one('#results-log')
+
+ # Create and set up the custom handler
+ log_handler = RichLogHandler(rich_log)
+ log_type = os.getenv('BROWSER_USE_LOGGING_LEVEL', 'info').lower()
+
+ class BrowserUseFormatter(logging.Formatter):
+ def format(self, record):
+ if isinstance(record.name, str) and record.name.startswith('browser_use.'):
+ record.name = record.name.split('.')[-2]
+ return super().format(record)
+
+ # Set up the formatter based on log type
+ if log_type == 'result':
+ log_handler.setLevel('RESULT')
+ log_handler.setFormatter(BrowserUseFormatter('%(message)s'))
+ else:
+ log_handler.setFormatter(BrowserUseFormatter('%(levelname)-8s [%(name)s] %(message)s'))
+
+ # Configure root logger - Replace ALL handlers, not just stdout handlers
+ root = logging.getLogger()
+
+ # Clear all existing handlers and add only our richlog handler
+ root.handlers = []
+ root.addHandler(log_handler)
+
+ # Set log level based on environment variable
+ if log_type == 'result':
+ root.setLevel('RESULT')
+ elif log_type == 'debug':
+ root.setLevel(logging.DEBUG)
+ else:
+ root.setLevel(logging.INFO)
+
+ # Configure browser_use logger
+ browser_use_logger = logging.getLogger('browser_use')
+ browser_use_logger.propagate = False # Don't propagate to root logger
+ browser_use_logger.handlers = [log_handler] # Replace any existing handlers
+ browser_use_logger.setLevel(root.level)
+
+ # Silence third-party loggers
+ for logger_name in [
+ 'WDM',
+ 'httpx',
+ 'selenium',
+ 'playwright',
+ 'urllib3',
+ 'asyncio',
+ 'langchain',
+ 'openai',
+ 'httpcore',
+ 'charset_normalizer',
+ 'anthropic._base_client',
+ 'PIL.PngImagePlugin',
+ 'trafilatura.htmlprocessing',
+ 'trafilatura',
+ ]:
+ third_party = logging.getLogger(logger_name)
+ third_party.setLevel(logging.ERROR)
+ third_party.propagate = False
+ third_party.handlers = [] # Clear any existing handlers
+
+ def on_mount(self) -> None:
+ """Set up components when app is mounted."""
+ # We'll use a file logger since stdout is now controlled by Textual
+ logger = logging.getLogger('browser_use.on_mount')
+ logger.debug('on_mount() method started')
+
+ # Step 1: Set up custom logging to RichLog
+ logger.debug('Setting up RichLog logging...')
+ try:
+ self.setup_richlog_logging()
+ logger.debug('RichLog logging set up successfully')
+ except Exception as e:
+ logger.error(f'Error setting up RichLog logging: {str(e)}', exc_info=True)
+ raise RuntimeError(f'Failed to set up RichLog logging: {str(e)}')
+
+ # Step 2: Set up input history
+ logger.debug('Setting up readline history...')
+ try:
+ if READLINE_AVAILABLE and self.task_history:
+ for item in self.task_history:
+ readline.add_history(item)
+ logger.debug(f'Added {len(self.task_history)} items to readline history')
+ else:
+ logger.debug('No readline history to set up')
+ except Exception as e:
+ logger.error(f'Error setting up readline history: {str(e)}', exc_info=False)
+ # Non-critical, continue
+
+ # Step 3: Focus the input field
+ logger.debug('Focusing input field...')
+ try:
+ input_field = self.query_one('#task-input')
+ input_field.focus()
+ logger.debug('Input field focused')
+ except Exception as e:
+ logger.error(f'Error focusing input field: {str(e)}', exc_info=True)
+ # Non-critical, continue
+
+ # Step 4: Initialize panels with test content
+ logger.debug('Testing panel initialization...')
+ try:
+ browser_info = self.query_one('#browser-info')
+ browser_info.write('DEBUG: Panel initialized')
+ logger.debug('Browser panel initialized')
+
+ model_info = self.query_one('#model-info')
+ model_info.write('DEBUG: Panel initialized')
+ logger.debug('Model panel initialized')
+
+ tasks_info = self.query_one('#tasks-info')
+ tasks_info.write('DEBUG: Panel initialized')
+ logger.debug('Tasks panel initialized')
+ except Exception as e:
+ logger.error(f'Error initializing panels: {str(e)}', exc_info=True)
+ # Non-critical, continue
+
+ # Step 5: Start continuous info panel updates
+ logger.debug('Starting info panel updates...')
+ try:
+ self.update_info_panels()
+ logger.debug('Info panel updates started')
+ except Exception as e:
+ logger.error(f'Error starting info panel updates: {str(e)}', exc_info=True)
+ # Non-critical, continue
+
+ logger.debug('on_mount() completed successfully')
+
+ def on_input_key_up(self, event: events.Key) -> None:
+ """Handle up arrow key in the input field."""
+ # Check if event is from the input field
+ if event.sender.id != 'task-input':
+ return
+
+ # Only process if we have history
+ if not self.task_history:
+ return
+
+ # Move back in history if possible
+ if self.history_index > 0:
+ self.history_index -= 1
+ self.query_one('#task-input').value = self.task_history[self.history_index]
+ # Move cursor to end of text
+ self.query_one('#task-input').cursor_position = len(self.query_one('#task-input').value)
+
+ # Prevent default behavior (cursor movement)
+ event.prevent_default()
+ event.stop()
+
+ def on_input_key_down(self, event: events.Key) -> None:
+ """Handle down arrow key in the input field."""
+ # Check if event is from the input field
+ if event.sender.id != 'task-input':
+ return
+
+ # Only process if we have history
+ if not self.task_history:
+ return
+
+ # Move forward in history or clear input if at the end
+ if self.history_index < len(self.task_history) - 1:
+ self.history_index += 1
+ self.query_one('#task-input').value = self.task_history[self.history_index]
+ # Move cursor to end of text
+ self.query_one('#task-input').cursor_position = len(self.query_one('#task-input').value)
+ elif self.history_index == len(self.task_history) - 1:
+ # At the end of history, go to "new line" state
+ self.history_index += 1
+ self.query_one('#task-input').value = ''
+
+ # Prevent default behavior (cursor movement)
+ event.prevent_default()
+ event.stop()
+
+ async def on_key(self, event: events.Key) -> None:
+ """Handle key events at the app level to ensure graceful exit."""
+ # Handle Ctrl+C, Ctrl+D, and Ctrl+Q for app exit
+ if event.key == 'ctrl+c' or event.key == 'ctrl+d' or event.key == 'ctrl+q':
+ await self.action_quit()
+ event.stop()
+ event.prevent_default()
+
+ def on_input_submitted(self, event: Input.Submitted) -> None:
+ """Handle task input submission."""
+ if event.input.id == 'task-input':
+ task = event.input.value
+ if not task.strip():
+ return
+
+ # Add to history if it's new
+ if task.strip() and (not self.task_history or task != self.task_history[-1]):
+ self.task_history.append(task)
+ self.config['command_history'] = self.task_history
+ save_user_config(self.config)
+
+ # Reset history index to point past the end of history
+ self.history_index = len(self.task_history)
+
+ # Hide logo, links, and paths panels
+ self.hide_intro_panels()
+
+ # Process the task
+ self.run_task(task)
+
+ # Clear the input
+ event.input.value = ''
+
+ def hide_intro_panels(self) -> None:
+ """Hide the intro panels, show info panels, and expand the log view."""
+ try:
+ # Get the panels
+ logo_panel = self.query_one('#logo-panel')
+ links_panel = self.query_one('#links-panel')
+ paths_panel = self.query_one('#paths-panel')
+ info_panels = self.query_one('#info-panels')
+ tasks_panel = self.query_one('#tasks-panel')
+ # Hide intro panels if they're visible and show info panels
+ if logo_panel.display:
+ # Log for debugging
+ logging.info('Hiding intro panels and showing info panels')
+
+ logo_panel.display = False
+ links_panel.display = False
+ paths_panel.display = False
+
+ # Show info panels
+ info_panels.display = True
+ tasks_panel.display = True
+
+ # Directly force content into panels for debugging
+ browser_info = self.query_one('#browser-info')
+ browser_info.write('Showing browser info...')
+
+ model_info = self.query_one('#model-info')
+ model_info.write('Showing model info...')
+
+ tasks_info = self.query_one('#tasks-info')
+ tasks_info.write('Showing tasks info...')
+
+ # Make results container take full height
+ results_container = self.query_one('#results-container')
+ results_container.styles.height = '1fr'
+
+ # Configure the log
+ results_log = self.query_one('#results-log')
+ results_log.styles.height = 'auto'
+
+ logging.info('Panels should now be visible')
+ except Exception as e:
+ logging.error(f'Error in hide_intro_panels: {str(e)}')
+
+ def update_info_panels(self) -> None:
+ """Update all information panels with current state."""
+ try:
+ # Force initial content into panels to ensure they're not empty
+ browser_info = self.query_one('#browser-info')
+ if not browser_info.lines:
+ browser_info.write('Initializing browser info...')
+
+ model_info = self.query_one('#model-info')
+ if not model_info.lines:
+ model_info.write('Initializing model info...')
+
+ tasks_info = self.query_one('#tasks-info')
+ if not tasks_info.lines:
+ tasks_info.write('Initializing tasks info...')
+
+ # Update actual content
+ self.update_browser_panel()
+ self.update_model_panel()
+ self.update_tasks_panel()
+ except Exception as e:
+ logging.error(f'Error in update_info_panels: {str(e)}')
+ finally:
+ # Always schedule the next update - will update at 1-second intervals
+ # This ensures continuous updates even if agent state changes
+ self.set_timer(1.0, self.update_info_panels)
+
+ def update_browser_panel(self) -> None:
+ """Update browser information panel with details about the browser."""
+ browser_info = self.query_one('#browser-info')
+ browser_info.clear()
+
+ if self.browser:
+ # Get basic browser info
+ browser_type = self.browser.__class__.__name__
+ headless = self.browser.config.headless
+ browser_class = self.browser.config.browser_class
+
+ # Determine connection type based on config
+ connection_type = 'playwright' # Default
+ if self.browser.config.cdp_url:
+ connection_type = 'CDP'
+ elif self.browser.config.wss_url:
+ connection_type = 'WSS'
+ elif self.browser.config.browser_binary_path:
+ connection_type = 'user-provided'
+
+ # Get window size details
+ window_width = self.browser.config.new_context_config.window_width
+ window_height = self.browser.config.new_context_config.window_height
+
+ # Try to get browser PID
+ browser_pid = 'Unknown'
+ connected = False
+ browser_status = '[red]Disconnected[/]'
+
+ try:
+ # First check if Chrome subprocess is available directly
+ if hasattr(self.browser, '_chrome_subprocess') and self.browser._chrome_subprocess:
+ try:
+ if hasattr(self.browser._chrome_subprocess, 'pid'):
+ browser_pid = str(self.browser._chrome_subprocess.pid)
+ connected = True
+ browser_status = '[green]Connected[/]'
+ except Exception as e:
+ browser_pid = f'Error: {str(e)}'
+ # Then check if we have a playwright browser connection
+ elif hasattr(self.browser, 'playwright_browser') and self.browser.playwright_browser:
+ connected = True
+ browser_status = '[green]Connected[/]'
+
+ # Try to get PID from related processes by checking for Chrome/Firefox
+ import psutil
+
+ for proc in psutil.process_iter(['pid', 'name']):
+ try:
+ if (
+ browser_class in proc.name().lower()
+ or 'chrome' in proc.name().lower()
+ or 'chromium' in proc.name().lower()
+ or 'firefox' in proc.name().lower()
+ ):
+ browser_pid = str(proc.pid)
+ break
+ except (psutil.NoSuchProcess, psutil.AccessDenied):
+ pass
+ except Exception as e:
+ browser_pid = f'Error: {str(e)}'
+
+ # Display browser information
+ browser_info.write(f'[bold cyan]{browser_class}[/] Browser ({browser_status})')
+ browser_info.write(
+ f'Type: [yellow]{connection_type}[/] [{"green" if not headless else "red"}]{" (headless)" if headless else ""}[/]'
+ )
+ browser_info.write(f'PID: [dim]{browser_pid}[/]')
+ browser_info.write(f'CDP Port: {self.browser.config.chrome_remote_debugging_port}')
+
+ if window_width and window_height:
+ browser_info.write(f'Window: [blue]{window_width}[/] ร [blue]{window_height}[/]')
+
+ # Include additional information about the browser if needed
+ if connected and hasattr(self, 'agent') and self.agent:
+ try:
+ # Show when the browser was connected
+ timestamp = int(time.time())
+ current_time = time.strftime('%H:%M:%S', time.localtime(timestamp))
+ browser_info.write(f'Last updated: [dim]{current_time}[/]')
+ except Exception as e:
+ pass
+
+ # Show the agent's current page URL if available
+ if hasattr(self.agent.browser_context, 'agent_current_page') and self.agent.browser_context.agent_current_page:
+ current_url = (
+ self.agent.browser_context.agent_current_page.url.replace('https://', '')
+ .replace('http://', '')
+ .replace('www.', '')[:36]
+ + 'โฆ'
+ )
+ browser_info.write(f'๐๏ธ [green]{current_url}[/]')
+ else:
+ browser_info.write('[red]Browser not initialized[/]')
+
+ def update_model_panel(self) -> None:
+ """Update model information panel with details about the LLM."""
+ model_info = self.query_one('#model-info')
+ model_info.clear()
+
+ if self.llm:
+ # Get model details
+ model_name = 'Unknown'
+ if hasattr(self.llm, 'model_name'):
+ model_name = self.llm.model_name
+ elif hasattr(self.llm, 'model'):
+ model_name = self.llm.model
+
+ # Show model name
+ if self.agent:
+ temp_str = f'{self.llm.temperature}ยบC ' if self.llm.temperature else ''
+ vision_str = '+ vision ' if self.agent.settings.use_vision else ''
+ memory_str = '+ memory ' if self.agent.enable_memory else ''
+ planner_str = '+ planner' if self.agent.settings.planner_llm else ''
+ model_info.write(
+ f'[white]LLM:[/] [blue]{self.llm.__class__.__name__} [yellow]{model_name}[/] {temp_str}{vision_str}{memory_str}{planner_str}'
+ )
+ else:
+ model_info.write(f'[white]LLM:[/] [blue]{self.llm.__class__.__name__} [yellow]{model_name}[/]')
+
+ # Show token usage statistics if agent exists and has history
+ if self.agent and hasattr(self.agent, 'state') and hasattr(self.agent.state, 'history'):
+ # Get total tokens used
+ total_tokens = self.agent.state.history.total_input_tokens()
+ model_info.write(f'[white]Input tokens:[/] [green]{total_tokens:,}[/]')
+
+ # Calculate tokens per step
+ num_steps = len(self.agent.state.history.history)
+ if num_steps > 0:
+ avg_tokens_per_step = total_tokens / num_steps
+ model_info.write(f'[white]Avg tokens/step:[/] [green]{avg_tokens_per_step:,.1f}[/]')
+
+ # Get the last step metadata to show the most recent LLM response time
+ if num_steps > 0 and self.agent.state.history.history[-1].metadata:
+ last_step = self.agent.state.history.history[-1]
+ step_duration = last_step.metadata.duration_seconds
+ step_tokens = last_step.metadata.input_tokens
+
+ if step_tokens > 0:
+ tokens_per_second = step_tokens / step_duration if step_duration > 0 else 0
+ model_info.write(f'[white]Avg tokens/sec:[/] [magenta]{tokens_per_second:.1f}[/]')
+
+ # Show total duration
+ total_duration = self.agent.state.history.total_duration_seconds()
+ if total_duration > 0:
+ model_info.write(f'[white]Total Duration:[/] [magenta]{total_duration:.2f}s[/]')
+
+ # Calculate response time metrics
+ model_info.write(f'[white]Last Step Duration:[/] [magenta]{step_duration:.2f}s[/]')
+
+ # Add current state information
+ if hasattr(self.agent, 'running'):
+ if self.agent.running:
+ model_info.write('[yellow]LLM is thinking[blink]...[/][/]')
+ elif hasattr(self.agent, 'state') and hasattr(self.agent.state, 'paused') and self.agent.state.paused:
+ model_info.write('[orange]LLM paused[/]')
+ else:
+ model_info.write('[red]Model not initialized[/]')
+
+ def update_tasks_panel(self) -> None:
+ """Update tasks information panel with details about the tasks and steps hierarchy."""
+ tasks_info = self.query_one('#tasks-info')
+ tasks_info.clear()
+
+ if self.agent:
+ # Check if agent has tasks
+ task_history = []
+ message_history = []
+
+ # Try to extract tasks by looking at message history
+ if hasattr(self.agent, '_message_manager') and self.agent._message_manager:
+ message_history = self.agent._message_manager.state.history.messages
+
+ # Extract original task(s)
+ original_tasks = []
+ for msg in message_history:
+ if hasattr(msg, 'message') and hasattr(msg.message, 'content'):
+ content = msg.message.content
+ if isinstance(content, str) and 'Your ultimate task is:' in content:
+ task_text = content.split('"""')[1].strip()
+ original_tasks.append(task_text)
+
+ if original_tasks:
+ tasks_info.write('[bold green]TASK:[/]')
+ for i, task in enumerate(original_tasks, 1):
+ # Only show latest task if multiple task changes occurred
+ if i == len(original_tasks):
+ tasks_info.write(f'[white]{task}[/]')
+ tasks_info.write('')
+
+ # Get current state information
+ current_step = self.agent.state.n_steps if hasattr(self.agent, 'state') else 0
+
+ # Get all agent history items
+ history_items = []
+ if hasattr(self.agent, 'state') and hasattr(self.agent.state, 'history'):
+ history_items = self.agent.state.history.history
+
+ if history_items:
+ tasks_info.write('[bold yellow]STEPS:[/]')
+
+ for idx, item in enumerate(history_items, 1):
+ # Determine step status
+ step_style = '[green]โ[/]'
+
+ # For the current step, show it as in progress
+ if idx == current_step:
+ step_style = '[yellow]โณ[/]'
+
+ # Check if this step had an error
+ if item.result and any(result.error for result in item.result):
+ step_style = '[red]โ[/]'
+
+ # Show step number
+ tasks_info.write(f'{step_style} Step {idx}/{current_step}')
+
+ # Show goal if available
+ if item.model_output and hasattr(item.model_output, 'current_state'):
+ # Show memory (context) for this step
+ memory = item.model_output.current_state.memory
+ if memory:
+ memory_lines = memory.strip().split('\n')
+ memory_summary = memory_lines[0]
+ tasks_info.write(f' [dim]Memory:[/] {memory_summary}')
+
+ # Show goal for this step
+ goal = item.model_output.current_state.next_goal
+ if goal:
+ # Take just the first line for display
+ goal_lines = goal.strip().split('\n')
+ goal_summary = goal_lines[0]
+ tasks_info.write(f' [cyan]Goal:[/] {goal_summary}')
+
+ # Show evaluation of previous goal (feedback)
+ eval_prev = item.model_output.current_state.evaluation_previous_goal
+ if eval_prev and idx > 1: # Only show for steps after the first
+ eval_lines = eval_prev.strip().split('\n')
+ eval_summary = eval_lines[0]
+ eval_summary = eval_summary.replace('Success', 'โ
').replace('Failed', 'โ ').strip()
+ tasks_info.write(f' [tan]Evaluation:[/] {eval_summary}')
+
+ # Show actions taken in this step
+ if item.model_output and item.model_output.action:
+ tasks_info.write(' [purple]Actions:[/]')
+ for action_idx, action in enumerate(item.model_output.action, 1):
+ action_type = action.__class__.__name__
+ if hasattr(action, 'model_dump'):
+ # For proper actions, show the action type
+ action_dict = action.model_dump(exclude_unset=True)
+ if action_dict:
+ action_name = list(action_dict.keys())[0]
+ tasks_info.write(f' {action_idx}. [blue]{action_name}[/]')
+
+ # Show results or errors from this step
+ if item.result:
+ for result in item.result:
+ if result.error:
+ error_text = result.error
+ tasks_info.write(f' [red]Error:[/] {error_text}')
+ elif result.extracted_content:
+ content = result.extracted_content
+ tasks_info.write(f' [green]Result:[/] {content}')
+
+ # Add a space between steps for readability
+ tasks_info.write('')
+
+ # If agent is actively running, show a status indicator
+ if hasattr(self.agent, 'running') and self.agent.running:
+ tasks_info.write('[yellow]Agent is actively working[blink]...[/][/]')
+ elif hasattr(self.agent, 'state') and hasattr(self.agent.state, 'paused') and self.agent.state.paused:
+ tasks_info.write('[orange]Agent is paused (press Enter to resume)[/]')
+ else:
+ tasks_info.write('[dim]Agent not initialized[/]')
+
+ # Force scroll to bottom
+ tasks_panel = self.query_one('#tasks-panel')
+ tasks_panel.scroll_end(animate=False)
+
+ def scroll_to_input(self) -> None:
+ """Scroll to the input field to ensure it's visible."""
+ input_container = self.query_one('#task-input-container')
+ input_container.scroll_visible()
+
+ def run_task(self, task: str) -> None:
+ """Launch the task in a background worker."""
+ # Create or update the agent
+ agent_settings = AgentSettings.model_validate(self.config.get('agent', {}))
+
+ # Get the logger
+ logger = logging.getLogger('browser_use.app')
+
+ # Make sure intro is hidden and log is ready
+ self.hide_intro_panels()
+
+ # Start continuous updates of all info panels
+ self.update_info_panels()
+
+ # Clear the log to start fresh
+ rich_log = self.query_one('#results-log')
+ rich_log.clear()
+
+ if self.agent is None:
+ self.agent = Agent(
+ task=task,
+ llm=self.llm,
+ controller=self.controller,
+ browser=self.browser,
+ **agent_settings.model_dump(),
+ )
+ else:
+ self.agent.add_new_task(task)
+
+ # Let the agent run in the background
+ async def agent_task_worker() -> None:
+ logger.debug('\n๐ Working on task: %s', task)
+
+ # Set flags to indicate the agent is running
+ self.agent.running = True
+ self.agent.last_response_time = 0
+
+ # Panel updates are already happening via the timer in update_info_panels
+
+ try:
+ # Run the agent task, redirecting output to RichLog through our handler
+ await self.agent.run()
+ except Exception as e:
+ logger.error('\nError running agent: %s', str(e))
+ finally:
+ # Clear the running flag
+ self.agent.running = False
+
+ # No need to call update_info_panels() here as it's already updating via timer
+
+ logger.debug('\nโ
Task completed!')
+
+ # Make sure the task input container is visible
+ task_input_container = self.query_one('#task-input-container')
+ task_input_container.display = True
+
+ # Refocus the input field
+ input_field = self.query_one('#task-input')
+ input_field.focus()
+
+ # Ensure the input is visible by scrolling to it
+ self.call_after_refresh(self.scroll_to_input)
+
+ # Run the worker
+ self.run_worker(agent_task_worker, name='agent_task')
+
+ def action_input_history_prev(self) -> None:
+ """Navigate to the previous item in command history."""
+ # Only process if we have history and input is focused
+ input_field = self.query_one('#task-input')
+ if not input_field.has_focus or not self.task_history:
+ return
+
+ # Move back in history if possible
+ if self.history_index > 0:
+ self.history_index -= 1
+ input_field.value = self.task_history[self.history_index]
+ # Move cursor to end of text
+ input_field.cursor_position = len(input_field.value)
+
+ def action_input_history_next(self) -> None:
+ """Navigate to the next item in command history or clear input."""
+ # Only process if we have history and input is focused
+ input_field = self.query_one('#task-input')
+ if not input_field.has_focus or not self.task_history:
+ return
+
+ # Move forward in history or clear input if at the end
+ if self.history_index < len(self.task_history) - 1:
+ self.history_index += 1
+ input_field.value = self.task_history[self.history_index]
+ # Move cursor to end of text
+ input_field.cursor_position = len(input_field.value)
+ elif self.history_index == len(self.task_history) - 1:
+ # At the end of history, go to "new line" state
+ self.history_index += 1
+ input_field.value = ''
+
+ async def action_quit(self) -> None:
+ """Quit the application and clean up resources."""
+ # Close the browser if it exists
+ if self.browser:
+ try:
+ await self.browser.close()
+ logging.debug('Browser closed successfully')
+ except Exception as e:
+ logging.error(f'Error closing browser: {str(e)}')
+
+ # Exit the application
+ self.exit()
+ print('\nTry running tasks on our cloud: https://browser-use.com')
+
+ def compose(self) -> ComposeResult:
+ """Create the UI layout."""
+ yield Header()
+
+ # Main container for app content
+ with Container(id='main-container'):
+ # Logo panel
+ yield Static(BROWSER_LOGO, id='logo-panel', markup=True)
+
+ # Information panels (hidden by default)
+ with Container(id='info-panels'):
+ # Top row with browser and model panels side by side
+ with Container(id='top-panels'):
+ # Browser panel
+ with Container(id='browser-panel'):
+ yield RichLog(id='browser-info', markup=True, highlight=True, wrap=True)
+
+ # Model panel
+ with Container(id='model-panel'):
+ yield RichLog(id='model-info', markup=True, highlight=True, wrap=True)
+
+ # Tasks panel (full width, below browser and model)
+ with VerticalScroll(id='tasks-panel'):
+ yield RichLog(id='tasks-info', markup=True, highlight=True, wrap=True, auto_scroll=True)
+
+ # Links panel with URLs
+ with Container(id='links-panel'):
+ with HorizontalGroup(classes='link-row'):
+ yield Static('Run at scale on cloud: [blink]โ๏ธ[/] ', markup=True, classes='link-label')
+ yield Link('https://browser-use.com', url='https://browser-use.com', classes='link-white link-url')
+
+ yield Static('') # Empty line
+
+ with HorizontalGroup(classes='link-row'):
+ yield Static('Chat & share on Discord: ๐ ', markup=True, classes='link-label')
+ yield Link(
+ 'https://discord.gg/ESAUZAdxXY', url='https://discord.gg/ESAUZAdxXY', classes='link-purple link-url'
+ )
+
+ with HorizontalGroup(classes='link-row'):
+ yield Static('Get prompt inspiration: ๐ฆธ ', markup=True, classes='link-label')
+ yield Link(
+ 'https://github.com/browser-use/awesome-prompts',
+ url='https://github.com/browser-use/awesome-prompts',
+ classes='link-magenta link-url',
+ )
+
+ with HorizontalGroup(classes='link-row'):
+ yield Static('[dim]Report any issues:[/] ๐ ', markup=True, classes='link-label')
+ yield Link(
+ 'https://github.com/browser-use/browser-use/issues',
+ url='https://github.com/browser-use/browser-use/issues',
+ classes='link-green link-url',
+ )
+
+ # Paths panel
+ yield Static(
+ f' โ๏ธ Settings & history saved to: {str(USER_CONFIG_FILE.resolve()).replace(str(Path.home()), "~")}\n'
+ f' ๐ Outputs & recordings saved to: {str(Path(".").resolve()).replace(str(Path.home()), "~")}',
+ id='paths-panel',
+ markup=True,
+ )
+
+ # Results view with scrolling (place this before input to make input sticky at bottom)
+ with VerticalScroll(id='results-container'):
+ yield RichLog(highlight=True, markup=True, id='results-log', wrap=True, auto_scroll=True)
+
+ # Task input container (now at the bottom)
+ with Container(id='task-input-container'):
+ yield Label('๐ What would you like me to do on the web?', id='task-label')
+ yield Input(placeholder='Enter your task...', id='task-input')
+
+ yield Footer()
+
+
+async def textual_interface(config: dict[str, Any]):
+ """Run the Textual interface."""
+ logger = logging.getLogger('browser_use.startup')
+
+ # Set up logging for Textual UI - prevent any logging to stdout
+ def setup_textual_logging():
+ # Replace all handlers with null handler
+ root_logger = logging.getLogger()
+ for handler in root_logger.handlers:
+ root_logger.removeHandler(handler)
+
+ # Add null handler to ensure no output to stdout/stderr
+ null_handler = logging.NullHandler()
+ root_logger.addHandler(null_handler)
+ logger.debug('Logging configured for Textual UI')
+
+ logger.debug('Setting up Browser, Controller, and LLM...')
+
+ # Step 1: Configure BrowserUse components
+ logger.debug('Validating browser configs...')
+ try:
+ browser_config = BrowserConfig.model_validate(config.get('browser', {}))
+ context_config = BrowserContextConfig.model_validate(config.get('browser_context', {}))
+ browser_config.new_context_config = context_config
+ logger.info(f'Browser type: {browser_config.browser_class}')
+ if browser_config.browser_binary_path:
+ logger.info(f'Browser binary: {browser_config.browser_binary_path}')
+ if browser_config.headless:
+ logger.info('Browser mode: headless')
+ else:
+ logger.info('Browser mode: visible')
+ logger.debug('Browser configs validated successfully')
+ except Exception as e:
+ logger.error(f'Error validating browser configs: {str(e)}', exc_info=True)
+ raise RuntimeError(f'Failed to validate browser configuration: {str(e)}')
+
+ # Step 2: Initialize Browser
+ logger.debug('Initializing Browser...')
+ try:
+ browser = Browser(config=browser_config)
+ logger.debug('Browser initialized successfully')
+
+ # Log browser version if available
+ try:
+ if hasattr(browser, 'version') and browser.version:
+ logger.info(f'Browser version: {browser.version}')
+ elif hasattr(browser, 'playwright_browser') and browser.playwright_browser:
+ version = browser.playwright_browser.version
+ logger.info(f'Browser version: {version}')
+ except Exception as e:
+ logger.debug(f'Could not determine browser version: {e}')
+ except Exception as e:
+ logger.error(f'Error initializing Browser: {str(e)}', exc_info=True)
+ raise RuntimeError(f'Failed to initialize Browser: {str(e)}')
+
+ # Step 3: Initialize Controller
+ logger.debug('Initializing Controller...')
+ try:
+ controller = Controller()
+ logger.debug('Controller initialized successfully')
+ except Exception as e:
+ logger.error(f'Error initializing Controller: {str(e)}', exc_info=True)
+ raise RuntimeError(f'Failed to initialize Controller: {str(e)}')
+
+ # Step 4: Get LLM
+ logger.debug('Getting LLM...')
+ try:
+ llm = get_llm(config)
+ # Log LLM details
+ model_name = getattr(llm, 'model_name', None) or getattr(llm, 'model', 'Unknown model')
+ provider = llm.__class__.__name__
+ temperature = getattr(llm, 'temperature', 0.0)
+ logger.info(f'LLM: {provider} ({model_name}), temperature: {temperature}')
+ logger.debug(f'LLM initialized successfully: {provider}')
+ except Exception as e:
+ logger.error(f'Error getting LLM: {str(e)}', exc_info=True)
+ raise RuntimeError(f'Failed to initialize LLM: {str(e)}')
+
+ logger.debug('Initializing BrowserUseApp instance...')
+ try:
+ app = BrowserUseApp(config)
+ # Pass the initialized components to the app
+ app.browser = browser
+ app.controller = controller
+ app.llm = llm
+
+ # Configure logging for Textual UI before going fullscreen
+ setup_textual_logging()
+
+ # Log browser and model configuration that will be used
+ browser_type = config.get('browser', {}).get('browser_class', 'Chrome')
+ model_name = config.get('model', {}).get('name', 'auto-detected')
+ headless = config.get('browser', {}).get('headless', True)
+ headless_str = 'headless' if headless else 'visible'
+
+ logger.info(f'Preparing {browser_type} browser ({headless_str}) with {model_name} LLM')
+
+ logger.debug('Starting Textual app with run_async()...')
+ # No more logging after this point as we're in fullscreen mode
+ await app.run_async()
+ except Exception as e:
+ logger.error(f'Error in textual_interface: {str(e)}', exc_info=True)
+ # Make sure to close browser if app initialization fails
+ if 'browser' in locals():
+ await browser.close()
+ raise
+
+
+@click.command()
+@click.option('--version', is_flag=True, help='Print version and exit')
+@click.option('--model', type=str, help='Model to use (e.g., gpt-4o, claude-3-opus-20240229, gemini-pro)')
+@click.option('--debug', is_flag=True, help='Enable verbose startup logging')
+@click.option('--headless', is_flag=True, help='Run browser in headless mode', default=None)
+@click.option('--window-width', type=int, help='Browser window width')
+@click.option('--window-height', type=int, help='Browser window height')
+@click.pass_context
+def main(ctx: click.Context, debug: bool = False, **kwargs):
+ """Browser-Use Interactive TUI"""
+
+ if kwargs['version']:
+ from importlib.metadata import version
+
+ print(version('browser-use'))
+ sys.exit(0)
+
+ # Configure console logging
+ console_handler = logging.StreamHandler(sys.stdout)
+ console_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s', '%H:%M:%S'))
+
+ # Configure root logger
+ root_logger = logging.getLogger()
+ root_logger.setLevel(logging.INFO if not debug else logging.DEBUG)
+ root_logger.addHandler(console_handler)
+
+ logger = logging.getLogger('browser_use.startup')
+ logger.info('Starting Browser-Use initialization')
+ if debug:
+ logger.debug(f'System info: Python {sys.version.split()[0]}, Platform: {sys.platform}')
+
+ logger.debug('Loading environment variables from .env file...')
+ load_dotenv()
+ logger.debug('Environment variables loaded')
+
+ # Load user configuration
+ logger.debug('Loading user configuration...')
+ try:
+ config = load_user_config()
+ logger.debug(f'User configuration loaded from {USER_CONFIG_FILE}')
+ except Exception as e:
+ logger.error(f'Error loading user configuration: {str(e)}', exc_info=True)
+ print(f'Error loading configuration: {str(e)}')
+ sys.exit(1)
+
+ # Update config with command-line arguments
+ logger.debug('Updating configuration with command line arguments...')
+ try:
+ config = update_config_with_click_args(config, ctx)
+ logger.debug('Configuration updated')
+ except Exception as e:
+ logger.error(f'Error updating config with command line args: {str(e)}', exc_info=True)
+ print(f'Error updating configuration: {str(e)}')
+ sys.exit(1)
+
+ # Save updated config
+ logger.debug('Saving user configuration...')
+ try:
+ save_user_config(config)
+ logger.debug('Configuration saved')
+ except Exception as e:
+ logger.error(f'Error saving user configuration: {str(e)}', exc_info=True)
+ print(f'Error saving configuration: {str(e)}')
+ sys.exit(1)
+
+ # Setup handlers for console output before entering Textual UI
+ logger.debug('Setting up handlers for Textual UI...')
+
+ # Log browser and model configuration that will be used
+ browser_type = config.get('browser', {}).get('browser_class', 'Chrome')
+ model_name = config.get('model', {}).get('name', 'auto-detected')
+ headless = config.get('browser', {}).get('headless', True)
+ headless_str = 'headless' if headless else 'visible'
+
+ logger.info(f'Preparing {browser_type} browser ({headless_str}) with {model_name} LLM')
+
+ try:
+ # Run the Textual UI interface - now all the initialization happens before we go fullscreen
+ logger.debug('Starting Textual UI interface...')
+ asyncio.run(textual_interface(config))
+ except Exception as e:
+ # Restore console logging for error reporting
+ root_logger.setLevel(logging.INFO)
+ for handler in root_logger.handlers:
+ root_logger.removeHandler(handler)
+ root_logger.addHandler(console_handler)
+
+ logger.error(f'Error initializing Browser-Use: {str(e)}', exc_info=debug)
+ print(f'\nError launching Browser-Use: {str(e)}')
+ if debug:
+ import traceback
+
+ traceback.print_exc()
+ sys.exit(1)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/browser-use/browser_use/controller/registry/service.py b/browser-use/browser_use/controller/registry/service.py
new file mode 100644
index 0000000..5f86642
--- /dev/null
+++ b/browser-use/browser_use/controller/registry/service.py
@@ -0,0 +1,246 @@
+import asyncio
+from collections.abc import Callable
+from inspect import iscoroutinefunction, signature
+from typing import Any, Generic, Optional, TypeVar
+
+from langchain_core.language_models.chat_models import BaseChatModel
+from pydantic import BaseModel, Field, create_model
+
+from browser_use.browser.context import BrowserContext
+from browser_use.controller.registry.views import (
+ ActionModel,
+ ActionRegistry,
+ RegisteredAction,
+)
+from browser_use.telemetry.service import ProductTelemetry
+from browser_use.telemetry.views import (
+ ControllerRegisteredFunctionsTelemetryEvent,
+ RegisteredFunction,
+)
+from browser_use.utils import time_execution_async
+
+Context = TypeVar('Context')
+
+
+class Registry(Generic[Context]):
+ """Service for registering and managing actions"""
+
+ def __init__(self, exclude_actions: list[str] | None = None):
+ self.registry = ActionRegistry()
+ self.telemetry = ProductTelemetry()
+ self.exclude_actions = exclude_actions if exclude_actions is not None else []
+
+ # @time_execution_sync('--create_param_model')
+ def _create_param_model(self, function: Callable) -> type[BaseModel]:
+ """Creates a Pydantic model from function signature"""
+ sig = signature(function)
+ params = {
+ name: (param.annotation, ... if param.default == param.empty else param.default)
+ for name, param in sig.parameters.items()
+ if name != 'browser' and name != 'page_extraction_llm' and name != 'available_file_paths'
+ }
+ # TODO: make the types here work
+ return create_model(
+ f'{function.__name__}_parameters',
+ __base__=ActionModel,
+ **params, # type: ignore
+ )
+
+ def action(
+ self,
+ description: str,
+ param_model: type[BaseModel] | None = None,
+ domains: list[str] | None = None,
+ page_filter: Callable[[Any], bool] | None = None,
+ ):
+ """Decorator for registering actions"""
+
+ def decorator(func: Callable):
+ # Skip registration if action is in exclude_actions
+ if func.__name__ in self.exclude_actions:
+ return func
+
+ # Create param model from function if not provided
+ actual_param_model = param_model or self._create_param_model(func)
+
+ # Wrap sync functions to make them async
+ if not iscoroutinefunction(func):
+
+ async def async_wrapper(*args, **kwargs):
+ return await asyncio.to_thread(func, *args, **kwargs)
+
+ # Copy the signature and other metadata from the original function
+ async_wrapper.__signature__ = signature(func)
+ async_wrapper.__name__ = func.__name__
+ async_wrapper.__annotations__ = func.__annotations__
+ wrapped_func = async_wrapper
+ else:
+ wrapped_func = func
+
+ action = RegisteredAction(
+ name=func.__name__,
+ description=description,
+ function=wrapped_func,
+ param_model=actual_param_model,
+ domains=domains,
+ page_filter=page_filter,
+ )
+ self.registry.actions[func.__name__] = action
+ return func
+
+ return decorator
+
+ @time_execution_async('--execute_action')
+ async def execute_action(
+ self,
+ action_name: str,
+ params: dict,
+ browser: BrowserContext | None = None,
+ page_extraction_llm: BaseChatModel | None = None,
+ sensitive_data: dict[str, str] | None = None,
+ available_file_paths: list[str] | None = None,
+ #
+ context: Context | None = None,
+ ) -> Any:
+ """Execute a registered action"""
+ if action_name not in self.registry.actions:
+ raise ValueError(f'Action {action_name} not found')
+
+ action = self.registry.actions[action_name]
+ try:
+ # Create the validated Pydantic model
+ validated_params = action.param_model(**params)
+
+ # Check if the first parameter is a Pydantic model
+ sig = signature(action.function)
+ parameters = list(sig.parameters.values())
+ is_pydantic = parameters and issubclass(parameters[0].annotation, BaseModel)
+ parameter_names = [param.name for param in parameters]
+
+ if sensitive_data:
+ validated_params = self._replace_sensitive_data(validated_params, sensitive_data)
+
+ # Check if the action requires browser
+ if 'browser' in parameter_names and not browser:
+ raise ValueError(f'Action {action_name} requires browser but none provided.')
+ if 'page_extraction_llm' in parameter_names and not page_extraction_llm:
+ raise ValueError(f'Action {action_name} requires page_extraction_llm but none provided.')
+ if 'available_file_paths' in parameter_names and not available_file_paths:
+ raise ValueError(f'Action {action_name} requires available_file_paths but none provided.')
+
+ if 'context' in parameter_names and not context:
+ raise ValueError(f'Action {action_name} requires context but none provided.')
+
+ # Prepare arguments based on parameter type
+ extra_args = {}
+ if 'context' in parameter_names:
+ extra_args['context'] = context
+ if 'browser' in parameter_names:
+ extra_args['browser'] = browser
+ if 'page_extraction_llm' in parameter_names:
+ extra_args['page_extraction_llm'] = page_extraction_llm
+ if 'available_file_paths' in parameter_names:
+ extra_args['available_file_paths'] = available_file_paths
+ if action_name == 'input_text' and sensitive_data:
+ extra_args['has_sensitive_data'] = True
+ if is_pydantic:
+ return await action.function(validated_params, **extra_args)
+ return await action.function(**validated_params.model_dump(), **extra_args)
+
+ except Exception as e:
+ raise RuntimeError(f'Error executing action {action_name}: {str(e)}') from e
+
+ def _replace_sensitive_data(self, params: BaseModel, sensitive_data: dict[str, str]) -> BaseModel:
+ """Replaces the sensitive data in the params"""
+ # if there are any str with