Fluexy LayerD

Turn a flat raster design into a layered motion video. Fluexy LayerD combines local image decomposition, vision-based semantic grouping, deterministic Remotion presets, and in-browser MP4 rendering.

PNG, JPEG, or WebP → LayerD → semantic groups → animated SVG → H.264 MP4

Demo • Features • How it works • Architecture • Getting started • Usage • Design decisions

Demo

Watch the product walkthrough on YouTube.

The walkthrough covers the product rationale, end-to-end workflow, relevant code, runtime logs, and persisted Neon records.

Example outputs

The animated previews play directly on GitHub. Select one to open the full MP4.

Input designs • Output previews • MP4 files

Input	Output

Features

• Reconstructs a flat image as independently positioned SVG image layers with LayerD.
• Uses a vision model to group related text, illustrations, decorations, products, logos, and calls to action into semantic units.
• Provides ten deterministic motion presets with immediate Remotion previews.
• Renders a five-second, 30 fps H.264 MP4 entirely in a compatible browser.
• Saves self-contained SVG projects to Neon for authenticated users to reopen, reanimate, or delete.
• Keeps rendered videos local to the browser as temporary object URLs.

How it works

Fluexy LayerD is inspired by MG-Gen and its central idea of controlling video motion through explicit, editable structure. Instead of running the paper's large original model, LayerD recovers movable visual elements from a flat design, a vision-model workflow organizes them into semantic units, and Remotion applies reproducible motion controls before rendering the result as video.

[!NOTE] Fluexy LayerD starts from a user-supplied design. Its scope is the motion-control pipeline: converting a flat visual into editable semantic layers and producing a controlled animation from them.

Architecture

The project separates expensive image understanding from predictable animation execution:

1. The authenticated Next.js route forwards the uploaded image to the local FastAPI service.
2. LayerD decomposes it into transparent raster fragments with positions and dimensions.
3. A vision model sees the source image, a numbered layer contact sheet, and layer geometry, then returns a validated semantic grouping plan.
4. The plan is embedded in the layered SVG as metadata and the self-contained SVG is saved to Neon.
5. Remotion animates each semantic group with a selected preset and renders the final MP4 through WebCodecs.

Browser
  └─ Next.js + Clerk
       ├─ FastAPI + LayerD          layer extraction
       ├─ OpenAI-compatible model  semantic grouping
       ├─ Neon + Drizzle           SVG project history
       └─ Remotion + WebCodecs     preview and MP4 render

The direct FastAPI response is ungrouped JSON containing the LayerD SVG, canvas size, and transparent layer images. Semantic grouping happens in the authenticated Next.js /api/convert route.

Prerequisites

• Node.js and npm
• Python 3.11 or 3.12
• uv
• Clerk and Neon projects
• An OpenAI-compatible vision model endpoint with structured output support
• A browser with WebCodecs and H.264 encoding support

[!NOTE] The first API startup downloads the LayerD BiRefNet and LaMa model weights and can take several minutes.

Getting started

Install the JavaScript and Python dependencies from the repository root:

npm install
uv sync --project apps/api

Create apps/web/.env:

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=...
CLERK_SECRET_KEY=...
DATABASE_URL=...

OPENAI_API_KEY=...
OPENAI_BASE_URL=https://your-openai-compatible-endpoint/v1
OPENAI_MODEL=your-vision-model

# Optional local API overrides
LAYERD_API_URL=http://127.0.0.1:8000
LAYERD_API_KEY=local-layerd-key

Create the database table:

npm run db:push

Start the API and web app in separate terminals:

npm run dev:api
npm run dev:web

Open http://localhost:3001. FastAPI runs at http://127.0.0.1:8000, with interactive API documentation at http://127.0.0.1:8000/docs.

[!IMPORTANT] LAYERD_API_KEY is shared by Next.js and FastAPI. If you change it in apps/web/.env, start the API with the same value: LAYERD_API_KEY=replace-me npm run dev:api.

Usage

1. Sign in and drop a PNG, JPEG, or WebP image into the studio.
2. Select one of the ten motion presets.
3. Choose Extract and group to run LayerD, create semantic groups, and save the SVG project.
4. Preview the animation and switch presets without repeating extraction.
5. Choose Render MP4 to render and preview the video in the browser.

Open History to revisit or delete saved SVG projects. Reopening a project does not rerun LayerD or the vision model. Rendered MP4s are not persisted.

H.264 requires even dimensions, so odd source dimensions are rounded down by at most one pixel for export.

API

Call LayerD directly by sending an image in the image multipart field:

curl -X POST http://127.0.0.1:8000/convert \
  -H 'X-API-Key: local-layerd-key' \
  -F image=@apps/web/public/image.png \
  --output layers.json

The response has this shape:

{
  "svg": "<svg>...</svg>",
  "canvas_size": [1080, 1080],
  "elements": [
    {
      "id": 0,
      "type": "image",
      "box": {
        "x_min": 0,
        "y_min": 0,
        "x_max": 1080,
        "y_max": 1080
      },
      "image": "data:image/png;base64,..."
    }
  ]
}

Design decisions and trade-offs

Semantic planning, deterministic execution

A flat bitmap has pixels but no concepts such as “headline,” “background,” or “CTA.” LayerD restores manipulable fragments, and the vision model supplies the missing semantic grouping. Its output is constrained to known roles, exactly one group per layer, and a separate full-canvas background.

Animation remains a pure frame-based function:

visual state = f(group, preset, frame, canvas size)

This hybrid keeps the model where visual judgment is useful and keeps transforms, timing, and final-frame reconstruction reproducible. Arbitrary generated animation code would offer more variety, but validated presets are safer, easier to inspect and test, and guaranteed to settle on the reconstructed artwork. A model-based render evaluator is intentionally omitted because deterministic invariants can check the render path without adding another variable, latency-heavy step.

Self-contained SVG persistence

Each saved SVG embeds its transparent layer images and semantic group metadata. That makes a project portable and lets users reopen it without rerunning either ML stage. The cost is a larger database record and duplicated raster data; the project API therefore rejects SVGs over 20 MB.

Browser-side rendering

Rendering through WebCodecs avoids a render server and keeps MP4 data on the user's device. The trade-off is browser and hardware dependence: H.264 encoding must be available, large compositions consume client resources, and completed videos disappear when their temporary browser URL is released.

Persistent model worker

FastAPI loads the LayerD pipeline once during application startup and reuses it across requests. This avoids paying model initialization cost for every conversion, but the API has a slow first start and retains the model's memory footprint while running.

Project structure

apps/
├── api/        FastAPI wrapper around the LayerD pipeline
└── web/        Authenticated Next.js studio, grouping route, and renderer
packages/
├── config/     Shared TypeScript configuration
├── db/         Drizzle schema and Neon client
├── env/        Server and browser environment validation
└── ui/         Shared UI components and styles

Useful commands

Troubleshooting

LayerD fails on Apple silicon

The API selects MPS when available. If PyTorch reports an unsupported MPS operation, run LayerD on CPU:

LAYERD_DEVICE=cpu npm run dev:api

The browser cannot render the MP4

Use a browser with WebCodecs and H.264 encoding support. The studio checks compatibility before rendering and displays the browser-reported issues.

Next.js cannot reach FastAPI

Confirm FastAPI is running at the LAYERD_API_URL value and that both services use the same LAYERD_API_KEY. The browser talks only to the authenticated Next.js proxy; it does not call FastAPI directly.