Project Generation

The Challenge#

I’ve been working on my hobby project the Agent’s Playground for over a year. The basic idea is it is a simulation engine that allows rapidly prototyping 2D simulations for autonomous agents.

During this time, when I wanted to created a new simulation I did that inside the engine’s source code directly. It was a rather tedious process and isn’t conducive to other people using the app. It went kinda like this.

Me: “I want to create a new simulation to try something out.”
Playground: “Cool, follow these 23 easy steps.”
Me: “Never mind, I want to play videos games.”

So, I decided the time was right to finally add support for project files. This post documents how I solved both creating and loading simulation projects.

What’s a Project?#

A simulation is a combination of a scene declared in a TOML file and logic that is distributed across renderers, interactive entities, and coroutines.

Because simulation logic is defined in Python code it can take advantage of system libraries and 3rd party packages. Simulations can get complicated. In an attempt to tame the complexity I strive to write tests for all the Python code.

All these things add up to the conclusion that a simulation project is really a small Python project. A new project follows this structure.

my_sim
├── my_sim 
│   ├── __init__.py
│   ├── scene.py
│   └── scene.toml
├── tests
│   └── scene_test.py
├── dev
│   └── shell.nix
├── libs
│   └── agents_playground-0.1.0-py3-none-any.whl
├── README.md
├── Makefile
├── pyproject.toml
└── requirements.txt

Here is the breakdown of the various files.

A project follows the same pattern for managing installing and bootstrapping that the Playground engine itself does. A Makefile is used to orchestrate the following.

1. The Nix package manager creates a shell and installs Python and Pip.
2. Pip uses the requirements.txt file to install Poetry.
3. Poetry then uses the pyproject.toml file to install all of the project’s dependencies.

The Solution#

After landing on how a project should be structured and used by developers there remained three core challenges.

1. Enable project’s to extend the Playgrounds capabilities with project specific renders, entities, and coroutines.
2. Actually running an external project with the Playground.
3. Provide a way to easily setup a new project.

I tackled figuring out how I want projects to extend the engine first.

Plugins and Decorators#

Because a simulation project is a Python project itself I decided to go with the plug-in pattern. This allows a project to register functions with the engine that a simulation can then leverage. This sounds a bit complicated but it’s actually pretty simple to do.

The first step is to have a place to collect all of the registered extensions. I creatively called this the SimulationExtensions class. It’s just a collection of dictionaries. Each extension type has its own dedicated dictionary.

class SimulationExtensions:
  def __init__(self) -> None:
    self._entity_extensions: Dict[str, Callable] = {}
    self._renderer_extensions: Dict[str, Callable] = {}
    self._task_extensions: Dict[str, Callable] = {}

  # Ignoring all the properties and utility methods

  def reset(self) -> None
    """Enable clearing all the registered extensions."""
    self._entity_extensions.clear()
    self._renderer_extensions.clear()
    self._task_extensions.clear()

There is only on instance of the SimulationExtensions in the engine. It is accessed with the simulation_extensions() function.

I use decorators to register methods as an extension. Below is the decorator that handles registering a renderer. The label is the name the function is accessed by in the project’s scene.toml file.

The same pattern is followed for registering entities and tasks (i.e. coroutines).

def register_renderer(label: str) -> Callable:
  def decorator_register_renderer(func: Callable) -> Callable:
    simulation_extensions().register_renderer(label, func)
    return func
  return decorator_register_renderer

A developer can use the decorator to register any function in their project as a renderer.

@register_renderer(label='draw_stuff')
def my_awesome_rendering_code(*args, **kwargs) -> None:
  # rendering logic.

When the simulation is being loaded it merges the system defined items with the user defined functions and then builds the scene. When a simulation is closed all user defined extensions are removed.

class Simulation
  # skipping so much stuff...

  def launch(self) -> None:
    """Opens the Simulation Window"""
    self._load_scene()
    # additional setup steps...

  def _load_scene(self) -> None:
    """Load the scene data from a TOML file."""
    # skipping lots stuff...
    scene_builder: SceneBuilder = self._init_scene_builder()
    self._context.scene = scene_builder.build(scene_data)

  def _init_scene_builder(self) -> SceneBuilder:
    se: SimulationExtensions = simulation_extensions()
    return SceneBuilder(
      id_generator      = dpg.generate_uuid, 
      task_scheduler    = self._task_scheduler, 
      pre_sim_scheduler = self._pre_sim_task_scheduler,
      render_map        = RENDERERS_REGISTRY | se.renderer_extensions, 
      task_map          = TASKS_REGISTRY | se.task_extensions,
      entities_map      = ENTITIES_REGISTRY | se.entity_extensions
    )

  def _handle_close(self) -> None
    # skipping even more stuff...
    # Purge any extensions defined by the Simulation's Project
    simulation_extensions().reset()

Project Loading#

Now we have the ability to add and remove extensions when we want. However, we still have the problem of how do we load a project’s Python code to an already running Playground instance?

I encapsulate the project loading logic in the ProjectLoader class. Before I dive into that, this is how the class is used.

pl = ProjectLoader()
try:
  # Raises an exception if something isn't right 
  # with the project structure.   
  pl.validate(module_name, project_path) 
  pl.load_or_reload(module_name, project_path)
except ProjectLoaderError as e:
  #Handle the exception...

The ProjectLoader class farms out the responsibility of validating a project file to a series of rule validator classes. These are collected in a list and are run sequentially by the validate method.

If the project passes the validation steps then the ProjectLoader attempts to load it. If the project has been previously run then it attempts to reload it.

from importlib.machinery import SourceFileLoader
from importlib.util import spec_from_loader, module_from_spec

class ProjectLoader:
  """
  Responsible for loading a Simulation Project.
  """
  def __init__(self) -> None:
    # Setup all the validators for creating a new project.
    self._validators = [
      ValidModuleName(),
      DirectoryExists(),
      InitFileExist(),
      SceneFileExist()
    ]

  def validate(
    self, 
    module_name: str, 
    project_path: str) -> None:
    """
    Run all the validation rules to ensure a project 
    is setup correctly.
    """
    [
      rule.validate(module_name, project_path) 
      for rule in self._validators
    ]
      
  def load_or_reload(
    self, 
    module_name: str, 
    project_path: str) -> None:
    """Load or reload a module."""
    if module_name in sys.modules:
      # More on this in a minute.
      self._reload_project(module_name) 
    else:
      self._load_project(module_name, project_path)

  def _load_project(self, 
    module_name: str, 
    project_path: str) -> None:
    """Compile and run a python module."""
    init_path: str = f'{project_path}/__init__.py'

    # 1.  Build a file loader
    loader = SourceFileLoader(
      fullname = module_name, 
      path = init_path)

    # 2. Try to build a module spec. 
    #    Otherwise raise an exception.
    spec = get_or_raise(
      spec_from_loader(loader.name, loader), 
      Exception(SPEC_FAILED_ERROR_MSG))

    # 3. Try to get the module from the spec. 
    #    Otherwise raise an exception.
    module = get_or_raise(
      module_from_spec(spec), 
      Exception(MOD_FAILED_FROM_SPEC_ERROR_MSG))

    # 4 Register the module with sys.modules.
    sys.modules[module_name] = module

    #5. Finally, build a spec loader and compile 
    #   module into byte code and run it.
    spec_loader = get_or_raise(spec.loader, Exception())
    spec_loader.exec_module(module)

All of the magic in the above code snippet is in the _load_project method. The method uses modules defined in importlib to dynamically import the project code. The process is:

1. Get a module specification for the project’s __init__.py file.
2. Create a module from the specification and add it to sys.modules.
3. Compile the module into byte code and run it.

Reloading#

The first time a project is opened the Playground engine will go through the multiple steps of loading all of the Python modules in the project. Things get more complicated if the user closes the active project and then reopens it.

It turns out that Python doesn’t have a mechanism for unloading a module. There are hacks that folks do like deleting from sys.modules but those can lead to unexpected behavior.

Python 3.4 introduced the importlib.reload(module) that can reload a single Python file. Currently it only works with a module name. Not a string or path.

When I found this, my experience was:

Me: “Well this is easy. I’ll just reload __init__.py and all the dependencies will take care of themselves.”
Python: “Haha haha. Stupid human.”

Sadly, imports are not rerun when a module is reloaded.

I spent way too much time chasing down various strategies for reloading everything in a project. I finally settled on flipping the script. Rather than the engine finding all the modules in a project to reload it’s a lot easier to have the project be responsible for doing its own reload.

# Module: agents_playground.project.rules.project_loader
class ProjectLoader:
  def _reload_project(self, module_name) -> None:
    project_module: ModuleType = sys.modules[module_name]    
    project_module.reload()

You can see the reload function that is automatically added to new projects in the below snippet.

# Example File: demos/a_star_navigation/__init__.py
import importlib

import a_star_navigation.renderers
import a_star_navigation.entities
import a_star_navigation.generate_agents
import a_star_navigation.agent_movement

def reload():
  [ 
    importlib.reload(sys.modules[module_name]) 
    for module_name in list( 
      filter( 
        lambda mod_key: mod_key.startswith('${a_star_navigation}.'), 
        sys.modules.keys()
      )
    ) 
  ]

In the Playground engine the ProjectLoader class can simply find the project’s init module and call the reload function. Seems really easy but it took me longer than I care to admit to come up with that solution.

There is one big caveat to this approach. Everything that needs to be reloaded must be in the same directory as the __init__.py file.

Circular Dependencies#

A complexity with having project defined engine extensions (i.e. plug-ins) is there are circular dependencies between the host application and the guest project. I address this by building the Agent Playground app as a Python Wheel.

The project can add the wheel as a dependency and access engine modules by importing agent_playground.

This is all handled automatically when a new project is created.

Project Generation#

After I landed on a project structure and figured out how to dynamically load that structure into the running app I flushed out auto-magically creating new projects.

This is all handled by the ProjectBuilder class and the use of a few Python modules provided by the standard library.

The heart of the ProjectBuilder class is the build method. It is responsible for orchestrating the new project process. I won’t go into all the details because they’re pretty boring frankly. However, I will highlight the interesting parts.

# Module: agents_playground.project.project_builder

class ProjectBuilder:  
  def build(
    self, 
    project_options: ProjectTemplateOptions, 
    input_processors: List[InputProcessor]) -> None:
    self._process_form_inputs(project_options, input_processors)
    self._validate_form_inputs(input_processors)
    self._prevent_over_writing_existing_directories(project_options)
    self._copy_template_directory(project_options)
    self._rename_pkg_directory(project_options)
    self._generate_project_files(project_options)
    self._copy_wheel(project_options)

Creating a new project is basically two steps. Copy a template directory with all of the stock files and then dynamically populate templates with project specific data. The method _copy_template_directory is responsible for setting up the project’s directory structure. The most interesting thing about this method is that is uses the shutil.copytree module to copy the directory structure and static files.

# Module: agents_playground.project.project_builder

import os
from pathlib import Path
import shutil

class ProjectBuilder:
  def _copy_template_directory(
    self, 
    project_options: ProjectTemplateOptions) -> None:
    """
    Responsible for copying the default project structure 
    to the new project's root directory.
    """
    new_project_dir = os.path.join(
      project_options.project_parent_directory, 
      project_options.project_name)
    
    template_dir = os.path.join(
      Path.cwd(), 
      'agents_playground/templates/new_project')
    
    shutil.copytree(template_dir, new_project_dir)

After the new project’s directory has been created the _generate_project_files method creates the project specific files by dynamically populating a series of template files. This is done using the standard string.Template class.

It is tempting to reach for a robust 3rd party template package but honestly, I don’t need one right now. Better to err on the side of simplicity. The below snippet shows my current template strategy.

# Module: agents_playground.project.project_builder

from string import Template

def populate_str(str: str, inputs: dict[str, Any]) -> str:
  return Template(str).substitute(inputs)

def populate_template(
  path_to_template: Path, 
  path_to_target: Path, 
  template_inputs: dict[str, Any]) -> None:
  """
  Helper function for populating a template 
  and writing it to disk.
  """
  scene_template: str = path_to_template.read_text()
  scene_file = Template(scene_template).substitute(template_inputs)
  path_to_target.write_text(scene_file)

class TemplateFile(NamedTuple):
  """Simple tuple to represent a template to process."""
  template_name: str
  template_location: str
  target_location: str

# All of the templates that need to be populated.
TEMPLATES = [
  TemplateFile('scene.toml',    '$default_template_path', '$project_pkg'),
  TemplateFile('__init__.py',   '$default_template_path', '$project_pkg'),
  TemplateFile('scene.py',      '$default_template_path', '$project_pkg'),
  TemplateFile('scene_test.py', '$default_template_path', 'tests')
]

class ProjectBuilder:
  def _generate_project_files(self, project_options: ProjectTemplateOptions) -> None:
    """Generates project specific files from engine templates."""
    template_inputs                          = vars(project_options)
    template_inputs['project_pkg']           = project_options.project_name
    template_inputs['default_template_path'] = 'agents_playground/templates/base_files'
    
    new_project_dir = os.path.join(
      project_options.project_parent_directory, 
      project_options.project_name)

    for template in TEMPLATES:
      hydrated_template_location = populate_str(
        template.template_location, 
        template_inputs)

      hydrated_template_name = populate_str(
        template.template_name,
        template_inputs)

      hydrated_target = populate_str(
        template.target_location, 
        template_inputs)

      template_path = Path(
        os.path.join(
          Path.cwd(), 
          hydrated_template_location, hydrated_template_name
        )
      )

      target_path = Path(
        os.path.join(
          new_project_dir, 
          hydrated_target, hydrated_template_name
        )
      )

      populate_template(template_path, target_path, template_inputs)

All Together Now#

The GIF below demonstrates the new features. In the recording I first create a new project and then open it with the Agent Playground app.

Future Work#

At the moment I’m pretty happy with how the New Project/Load Project functionality has turned out. It’s pretty powerful and easy to extend.

My only concern is related to the UI framework. Going through this process required me to add a GUI form, a few directory picker modules, and some error/success message popups. That was all easy to do but it highlighted the limitations of the DearPyGUI framework the app uses for the UI layer.

As the app becomes more feature rich the limitations of DearPyGUI are getting harder to ignore. It tries to straddle the spectrum of being both a UI framework and a mid-level 2D API. I’m finding though that it’s not close enough to either end of the spectrum. The UI capabilities are pretty limited and don’t provide the richness of a native app. The graphics API is also very limited.

It may be time to pivot to a different set of frameworks. However, that’s all problems for a different day.

Until next time…

-Sam