Data Flow Architecture

This document explains how data flows through the Next.js applications in the monorepo, specifically examining the architecture used in www.schwab.com which integrates multiple packages for data fetching, transformation, and rendering.

Overview

The data flow architecture follows a clean separation of concerns across multiple packages:

1. Fetching: @schwab/fetch - Retrieves data from external APIs (primarily Drupal CMS)
2. Schema Validation: @schwab/schema - Provides Zod schemas for type safety and validation
3. Transformation: @schwab/transformer - Converts external data formats to UI-compatible schemas
4. UI Rendering: @schwab/ui - React components that render transformed data

Data Flow Diagram

Data Flow Pipeline

1. External Data Sources

The applications primarily fetch data from:

• Drupal CMS: Content management system accessed via JSON:API
• Feature Flags: LaunchDarkly for dynamic configuration
• Analytics: Various tracking and measurement services

2. Data Fetching Layer (@schwab/fetch)

The fetch package provides typed functions for retrieving data from external sources:

// Example: Fetching a story from Drupal
import { getStory } from '@schwab/fetch/getStory';
import { EDrupalSource } from '@schwab/schema/native-enums/EDrupalSource';

const story = await getStory(EDrupalSource.Education, '/story/market-insights');

Key fetch functions discovered:

• getStory() - Retrieves individual story/article content
• getRelatedContent() - Fetches related articles and content
• getProgrammaticCta() - Gets call-to-action content
• Multiple Drupal API integrations through drupalFetch()

3. Schema Validation (@schwab/schema)

All external data is validated using Zod schemas to ensure type safety:

// Fetched data schema (from external API)
export const FetchedStorySchema = z.object({
  uuid: z.string(),
  title: z.string(),
  subtitle: z.string().optional(),
  summary: z.string(),
  components: z.array(z.unknown()),
  // ... more properties
});

// UI-ready schema (after transformation)
export const StorySchema = z.object({
  role: z.nativeEnum(ERole),
  identifier: z.string(),
  heading: z.string(),
  subheading: z.string().optional(),
  components: z.array(ComponentSchema),
  // ... transformed properties
});

4. Data Transformation (@schwab/transformer)

The transformer package converts external data formats into UI-compatible schemas:

import { transformStory } from '@schwab/transformer/transform/cmap-api/transformStory';

// Transform fetched data to UI schema
export function transformStory(data: TFetchedStory): TStory {
  const storyHeroImage = transformStoryHeroImage(data);
  const authors = transformStoryAuthor(data.authors);
  const components = transformStoryComponents(data.components, data.urlAlias);
  
  return mapComponent<TStory>({
    role: ERole.Story,
    heading: data?.title,
    subheading: data.subtitle,
    body: data.summary,
    image: storyHeroImage,
    authors,
    components,
    // ... more transformed fields
  });
}

Transformation capabilities:

• Component Transformation: Individual story components (text, images, videos, etc.)
• Media Processing: Image optimization, aspect ratio handling
• Content Structuring: Converting CMS structure to React component props
• URL Processing: Link prefixing and path resolution

5. UI Components (@schwab/ui)

The UI package provides React components that consume transformed data:

import StoryLoader from '@schwab/ui/StoryLoader';
import Story from '@schwab/ui/Story';

// Server component that orchestrates data loading
export default async function StoryLoader({ urlAlias }: TStoryLoader) {
  const story = await getStory(EDrupalSource.Education, urlAlias);
  
  return (
    <Story
      heading={story.data?.heading}
      subheading={story.data?.subheading}
      components={story.data?.components}
      // ... other props
    />
  );
}

Next.js Integration Pattern

App Router Implementation

The applications use Next.js App Router with server components for optimal performance:

// apps/www.schwab.com/src/app/[code]/learn/story/[...alias]/page.tsx
export default async function Page(props: { params: Promise<TStoryStaticParams> }) {
  const params = await props.params;
  const urlDetails = parseUrl(params.alias);

  return (
    <StoryLoader
      urlAlias={`/story/${urlDetails.urlAlias}`}
      pageNumber={urlDetails.pageNumber}
      pageParams={params}
      allFlags={allFlags}
    />
  );
}

Metadata Generation

Pages generate SEO metadata using the same data flow:

export async function generateMetadata(props): Promise<Metadata> {
  const params = await props.params;
  const urlDetails = parseUrl(params.alias);
  
  const story = await getStory(EDrupalSource.Education, `/story/${urlDetails.urlAlias}`);
  
  return {
    title: story.data?.pageTitle,
    description: story.data?.pageDescription,
    openGraph: {
      title: story.data?.heading,
      description: story.data?.body,
      images: [story.data?.image?.src],
    },
  };
}

Package Integration Points

1. Path Mapping Configuration

The monorepo uses TypeScript path mapping for clean imports:

// tsconfig.json paths
{
  "#components/*": ["./src/components/*"],
  "#functions/*": ["./src/functions/*"],
  "@schwab/fetch/*": ["../../../packages/fetch/src/*"],
  "@schwab/transformer/*": ["../../../packages/transformer/src/*"],
  "@schwab/ui/*": ["../../../packages/ui/src/*"]
}

2. Server-Only Operations

Data fetching and transformation occur server-side only:

import 'server-only'; // Ensures code doesn't run in browser

// Server component with data fetching
export default async function ServerComponent() {
  const data = await getStory(); // Server-only operation
  return <ClientComponent data={data} />;
}

Data Flow Examples

Story Page Flow

1. Request: User visits /learn/story/investment-basics
2. Parsing: parseUrl() extracts story alias from URL
3. Fetching: getStory() retrieves data from Drupal API
4. Validation: FetchedStorySchema validates raw API response
5. Transformation: transformStory() converts to UI schema
6. Validation: StorySchema validates transformed data
7. Rendering: Story component renders final UI

Component-Level Flow

// 1. Fetch raw data
const rawStory = await drupalFetch('/api/story/123');

// 2. Validate fetched data
const fetchedStory = FetchedStorySchema.parse(rawStory);

// 3. Transform to UI schema
const uiStory = transformStory(fetchedStory);

// 4. Validate transformed data
const validatedStory = StorySchema.parse(uiStory);

// 5. Render component
return <Story {...validatedStory} />;

Error Handling

The data flow includes comprehensive error handling:

// Fetch with error handling
const story = await getStory(source, urlAlias);

if (!story?.isOk) {
  notFound(); // Next.js 404 handling
}

// Zod validation with error handling
try {
  const validStory = StorySchema.parse(transformedData);
} catch (error) {
  return <ZodErrorOutput error={error} />;
}

Performance Optimizations

Static Generation

Pages use static generation where possible:

export const dynamic = 'force-static';

export async function generateStaticParams() {
  // Return static paths for build-time generation
}

Parallel Data Loading

Multiple data sources are loaded in parallel:

const [story, relatedContent, programmaticCta] = await Promise.all([
  getStory(source, urlAlias),
  getRelatedContent(source, urlAlias),
  getProgrammaticCta(source, urlAlias),
]);

Package Dependencies

The data flow relies on these key packages:

• @schwab/fetch: Data retrieval functions
• @schwab/schema: Zod schemas for validation
• @schwab/transformer: Data transformation functions
• @schwab/ui: React components
• @schwab/utilities: Helper functions (parseUrl, stripHTML, etc.)

Summary

The data flow architecture provides:

• Type Safety: End-to-end TypeScript with Zod validation
• Performance: Server-side data fetching with static generation
• Maintainability: Clear separation of concerns across packages
• Scalability: Modular architecture supporting multiple applications
• Developer Experience: Path mapping and consistent patterns across the monorepo

This architecture enables the team to build performant, type-safe web applications while maintaining clean code organization and developer productivity.