AGENTS.md — PyGWalker contributor & agent guide

This is the fast-start map of the PyGWalker repo for both human contributors and coding agents. It explains how the project is put together, how to run it in dev mode with live frontend reload, and where all the logs go. Read this first — it is written to save you from re-deriving the architecture by grepping.

Deeper references: docs/ARCHITECTURE.md (how it is built), docs/DEVELOPMENT.md (dev workflow + troubleshooting), docs/CONTRIBUTING.md (validation & CI).

---

1. What PyGWalker is (30-second model)

PyGWalker turns a pandas / polars / pyarrow dataframe into an interactive Graphic Walker UI inside notebooks, Streamlit, and plain web servers. It has two halves that ship together:

• Python package (pygwalker/) — public API (walk, render, table, Walker, to_html), data parsing, and the transports that talk to the UI.
• Frontend app (app/, React + Vite) — the UI. It is compiled into JavaScript bundles that are checked into the wheel under pygwalker/templates/dist/ and loaded by the Python side at render time.

The Python side never renders charts itself; it hands built JS + serialized data to a notebook/browser and then answers data/spec requests over a message channel.

---

2. Repo map

Path	What lives here
pygwalker/api/	Public entry points. adapter.py picks jupyter vs webserver; jupyter.py = notebook dispatch; walker.py = the reusable Walker; pygwalker.py = the core PygWalker.
pygwalker/services/	Rendering + display. anywidget_widget.py (default transport), render.py + templates/*.html (iframe transport), global_var.py (runtime globals), jupyter_display.py.
pygwalker/communications/	Kernel⇄frontend transports: anywidget_comm.py (default), hacker_comm.py (iframe), streamlit_comm.py, gradio_comm.py, reflex_comm.py. protocol.py is the shared message schema.
pygwalker/data_parsers/	Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…).
pygwalker/templates/dist/	Build output (git-ignored). The JS bundles the Python side loads.
pygwalker/utils/	Helpers: frontend_assets.py (locate/load bundles), log.py (logging), encoders.
app/src/	Frontend source. index.tsx = entry; utils/communication.tsx = transports; dataSource/ = data ingest; interfaces/comm.generated.ts = generated protocol types; store/ = MobX state.
scripts/	dev.py (dev orchestrator), compile.sh (build frontend), local_ci.py (mirror CI), generate_comm_protocol_ts.py (regenerate protocol types).
tests/	Python tests + *.ipynb notebooks run by nbmake. app/tests/ holds Playwright smoke tests.

---

3. How the two halves fit together (build & load model)

app/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it

Frontend build variants (app/vite.config.ts, output to pygwalker/templates/dist/):

Bundle	Built from	Loaded by
pygwalker-app.es.js	src/index.tsx	anywidget transport (the default pyg.walk path)
pygwalker-app.iife.js	src/index.tsx	iframe transport / static to_html()
dsl-to-workflow.umd.js	src/lib/dslToWorkflow.ts	kernel-side DSL→workflow conversion
vega-to-dsl.umd.js	src/lib/vegaToDsl.ts	kernel-side Vega→DSL conversion

yarn build builds all four (+ typecheck). yarn build:app builds only the two app bundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.

The message protocol is generated, not hand-written. Python Pydantic models in pygwalker/communications/protocol.py are the source of truth. Running python scripts/generate_comm_protocol_ts.py regenerates app/src/interfaces/comm.generated.ts. If you change protocol.py, regenerate and rebuild the frontend. Never edit comm.generated.ts by hand.

Transports. The default notebook transport is anywidget (env='JupyterAnywidget'). env='Jupyter' / env='JupyterWidget' are deprecated aliases that are coerced to anywidget and slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.

---

4. First-time setup

Requires Python 3.10+, Node.js 22.x, Yarn 1.x.

# Python (editable install with dev extras)
python -m venv venv && source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -e ".[dev]"

# Frontend deps + one full build so pygwalker/templates/dist/ is populated
cd app && yarn install && yarn build && cd ..

---

5. Dev mode: edit the frontend and see it live (anywidget HMR)

The default pyg.walk(df) uses the anywidget transport, which loads pygwalker-app.es.js from disk. In dev mode we (a) rebuild that bundle on every source change and (b) let anywidget hot-reload it into open widgets. One command starts everything and captures all logs:

source venv/bin/activate
python scripts/dev.py

This launches, and tees the output of, two long-running processes:

• frontend — cd app && yarn dev:build (vite build --watch) rebuilds the bundles into pygwalker/templates/dist/ on every edit under app/src/. → logs/frontend.log
• jupyter — jupyter lab started with PYGWALKER_DEV=1 and ANYWIDGET_HMR=1 in its environment (inherited by kernels). → logs/jupyter.log

Then, in a notebook cell, no special setup is needed — just use PyGWalker normally:

import pandas as pd, pygwalker as pyg
pyg.walk(pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]}))

Edit a file under app/src/, wait for the rebuild to finish in logs/frontend.log (built in …), and the widget hot-reloads in place — usually without re-running the cell. For a heavy change you can always re-run the cell.

Why it works: with ANYWIDGET_HMR=1 (or PYGWALKER_DEV=1), WalkerAnyWidget._esm is set to a pathlib.Path pointing at the built pygwalker-app.es.js instead of an embedded string. anywidget then reads that file and watches it (via watchfiles), pushing new code to the frontend when vite build --watch rewrites it. With the flags off (normal installs), the bundle is embedded as a string exactly as before — production behavior is unchanged.

Useful flags: --no-jupyter (only rebuild the frontend), --no-frontend (only Jupyter), --jupyter-port N, --no-browser (auto-enabled when output is not a TTY, e.g. agent runs), --log-dir DIR. Stop everything with Ctrl+C (services are torn down cleanly).

Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is documented in docs/DEVELOPMENT.md. Prefer the anywidget HMR path above.

---

6. Logs: one place to look

scripts/dev.py writes everything under logs/ (git-ignored) at the repo root:

File	Contents
logs/frontend.log	Vite build/watch output — watch for built in … (rebuild done) and TypeScript errors.
logs/jupyter.log	JupyterLab server output — the URL + token to open the notebook is here.
logs/pygwalker.log	Kernel-side PyGWalker Python logs (set via PYGWALKER_LOG_FILE).

Python log controls (honored by pygwalker/utils/log.py, independent of the orchestrator):

• PYGWALKER_LOG_FILE=/path/to/file.log — also append Python logs to a file.
• PYGWALKER_LOG_LEVEL=DEBUG — raise/lower verbosity (default INFO).

Frontend runtime logs (what happens in the browser, e.g. comm errors) appear in the browser devtools console, not in logs/. The frontend surfaces user-facing errors as in-app toast notifications rather than console.log.

Agent tip: tail logs/jupyter.log for the server URL, logs/frontend.log to know when a rebuild finished, and logs/pygwalker.log for kernel-side errors.

---

7. Everyday commands

# Frontend (run from app/)
yarn build            # full build: typecheck + all 4 bundles (CI-equivalent)
yarn build:app        # fast: only the two app bundles, no typecheck
yarn dev:build        # rebuild-on-change (what scripts/dev.py runs)
yarn typecheck        # tsc --noEmit
yarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)

# Python (run from repo root, venv active)
python -m ruff check pygwalker tests scripts bin pygwalker_tools
python -m ruff format --check pygwalker tests scripts bin pygwalker_tools
python -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests
python -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests

# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py
python scripts/generate_comm_protocol_ts.py

# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)
python scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow

---

8. Rules & gotchas

• Do not commit pygwalker/templates/dist/ — it is generated and git-ignored. The wheel build (and CI) rebuilds it via the Hatch jupyter-builder hook.
• Regenerate + rebuild after protocol changes. Editing communications/protocol.py without running scripts/generate_comm_protocol_ts.py and rebuilding the frontend leaves Python and JS out of sync.
• yarn build:app skips typecheck and the DSL bundles. Run full yarn build (or yarn typecheck) before pushing frontend changes.
• anywidget is the transport of record. Don't reach for the deprecated env='Jupyter' / iframe path for new work; it is removed in 0.7.0.
• Dev-mode flags are opt-in. PYGWALKER_DEV / ANYWIDGET_HMR only affect a dev session. Never rely on them being set at runtime for end users.
• First run needs a build. In dev mode the widget loads dist/pygwalker-app.es.js from disk; if it is missing you'll get a clear "Missing PyGWalker frontend asset" error — run the frontend build (or wait for scripts/dev.py's first build to finish).

---

9. Where to look for X

Question	Start here
How does pyg.walk() decide what to display?	pygwalker/api/adapter.py → api/jupyter.py (env_display_map)
How is the frontend bundle loaded / dev-swapped?	pygwalker/services/anywidget_widget.py, utils/frontend_assets.py
What messages can the frontend send the kernel?	pygwalker/communications/protocol.py ⇄ app/src/interfaces/comm.generated.ts
How is data sent to the browser?	pygwalker/services/data_communication.py, app/src/dataSource/
How is a chart exported to PNG/SVG/code?	pygwalker/services/chart_export.py, app/src/tools/
How does the iframe/static-HTML path render?	pygwalker/services/render.py + pygwalker/templates/*.html