Skip to content
Tallyfy

Extract tasks from meetings

Converting meeting recordings into tasks

The meeting productivity problem

Your team finishes an hour-long meeting. Six people agreed to do twelve things. By next week, three get done, five are forgotten, and four were misunderstood.

Meetings generate lots of work, but without tracking, action items disappear. You’ve seen the cycle - repeated meetings where people ask “wait, who was supposed to do that?” because nobody captured tasks from the last one.

How Tallyfy will solve this

We’re building an integration that connects your meeting platform (Zoom, Teams, Google Meet) to Tallyfy’s task system. Record your meeting as usual - Tallyfy handles the rest.

The system will transcribe your meeting using AI that works with multiple languages, then analyze the conversation to identify who committed to what and when.

Identity recognition

“John, can you send the proposal to the client by Friday?”

Simple for humans. Surprisingly hard for AI.

The planned system will identify both the task creator (the person asking) and the assignee (John) using SSO integration. When your team logs into both the meeting platform and Tallyfy with the same SSO provider, voices get matched to user profiles automatically. John gets assigned the task. The requester becomes the creator.

If someone joins as a guest or SSO isn’t configured, the system will create placeholder assignments you can map to real users during review.

Rich task context, not just titles

A task titled “Send proposal” tells you nothing. The system will capture full context:

Task Title: Send Q4 budget proposal to ACME Corp Description: During the October 15 planning meeting, John committed to sending the revised Q4 budget proposal to ACME Corp’s CFO. Meeting Context:

  • Quoted from recording at 23:45: “John, can you send the updated proposal with the new pricing structure to their CFO? They need it for their board meeting.”
  • AI Rationale: Identified as a task due to specific assignee (John), clear deliverable (proposal), explicit deadline (Friday), and confirmation response (“Sure, I’ll get that over by Thursday”). Deadline: October 18, 2024 at 5:00 PM Assignee: John Smith Creator: Sarah Johnson

Every extracted task will include the actual quote from the meeting and the AI’s reasoning for flagging it as a task.

Conversation thread tracking

Conversations aren’t linear. Someone proposes an idea at minute 5. It gets modified at minute 15. By minute 30, the team decides something different. Naive extraction would create three conflicting tasks.

The AI will track conversation threads throughout the meeting and understand when action items evolve, get canceled, or change - delivering only the final agreed-upon version of each task.

The planned Draft state

AI isn’t perfect. That’s why we plan to introduce a new task state: Draft.

Diagram

Draft tasks would work like a moderation queue - all extracted tasks appear grouped together for review. You’d edit assignments, adjust deadlines, delete irrelevant items, and convert good ones to active tasks.

This human-in-the-loop approach means AI handles extraction and organization while you provide the final quality check.

Technical challenges

Accuracy at scale

Current systems face several hurdles:

  • Background noise and cross-talk reduces transcription accuracy
  • Technical jargon causes misinterpretation
  • Multiple speakers at once creates attribution errors
  • Long meetings exceed AI context windows

The hallucination problem

Language models sometimes generate tasks that were never discussed - invented from context clues rather than explicit conversation.

The planned approach uses multiple validation layers: requiring explicit transcript evidence for every task, confidence scoring with warning flags on low-confidence items, and human review before tasks go live.

Meeting evolution patterns

Conversations follow common patterns that affect extraction:

  • Progressive refinement: Ideas evolve through discussion before becoming tasks
  • Implicit delegation: “Someone should…” becomes “John will…” over time
  • Conditional commitments: “If X happens, then I’ll do Y” requires tracking conditions
  • Retroactive cancellation: “Actually, let’s not do that” negates earlier commitments

Planned platform support

Phase 1 - Major video conferencing

  • Zoom, Microsoft Teams, Google Meet, Cisco Webex

Phase 2 - Specialized platforms

  • Slack Huddles, Discord, GoToMeeting, BlueJeans

Phase 3 - Recording tools

  • Otter.ai, Fireflies.ai, Sembly AI, Jamie

Phase 4 - Async collaboration

  • Loom, Vidyard, BombBomb, Soapbox

Current workaround - build it yourself

You can build your own meeting-to-tasks pipeline today using Tallyfy’s API. This requires technical expertise and ongoing maintenance.

What you’ll need

You’re responsible for:

  • Capturing and storing meeting recordings
  • Transcribing audio to text (using OpenAI Whisper, Google Speech-to-Text, etc.)
  • Extracting action items from transcripts (using GPT-4 or Claude)
  • Mapping speakers to Tallyfy users
  • Creating tasks via Tallyfy’s API
  • Handling errors and retries

Pipeline architecture

  1. Capture meeting recording

    // Option A: Webhook from Zoom/Teams
    app.post('/webhook/meeting-ended', async (req, res) => {
      const recordingUrl = req.body.download_url;
      await processRecording(recordingUrl);
    });
    

    // Option B: Poll cloud storage
    const recordings = await checkGoogleDrive('/meeting-recordings');

  2. Transcribe audio

    import openai
    

    def transcribe_meeting(audio_file):
        with open(audio_file, "rb") as f:
            transcript = openai.Audio.transcribe(
                model="whisper-1",
                file=f,
                response_format="verbose_json"  # Includes timestamps
            )
        return transcript

  3. Extract action items with AI

    def extract_tasks(transcript):
        prompt = """
        Extract action items from this transcript.
        Return JSON array with: task_name, assignee, deadline, context
        Only include items with clear owners and deadlines.
        Include quote from transcript as evidence.
        """
    

        response = openai.ChatCompletion.create(
            model="gpt-4",
            messages=[
                {"role": "system", "content": prompt},
                {"role": "user", "content": transcript}
            ],
            temperature=0.3  # Lower temperature for consistency
        )
    

        return json.loads(response.choices[0].message.content)

  4. Map to Tallyfy users

    const userMapping = {
      "John Smith": "user_abc123",
      "Sarah Johnson": "user_def456",
      "john@company.com": "user_abc123"
    };
    

    function resolveAssignee(extractedName) {
      if (userMapping[extractedName]) {
        return userMapping[extractedName];
      }
    

      const emailMatch = Object.keys(userMapping).find(key =>
        key.includes('@') && key.toLowerCase().includes(extractedName.toLowerCase())
      );
    

      return emailMatch ? userMapping[emailMatch] : null;
    }

  5. Create tasks in Tallyfy

    async function createTallyfyTasks(tasks, meetingTitle) {
      const firstTask = tasks[0];
      const response = await fetch(
        `https://go.tallyfy.com/api/organizations/${orgId}/tasks`,
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${accessToken}`,
            'X-Tallyfy-Client': 'APIClient',
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({
            title: `Tasks from: ${meetingTitle}`,
            task_type: 'task',
            owners: { users: [firstTask.assigneeId] },
            separate_task_for_each_assignee: true,
            deadline: firstTask.deadline
          })
        }
      );
    

      const process = await response.json();
      const processId = process.data.run.id;
    

      // Add remaining tasks to same process
      for (let i = 1; i < tasks.length; i++) {
        await createTask(tasks[i], processId);
      }
    }

Extraction patterns that work

Explicit action patterns:

  • “[Name] will/to [action] by [date]”
  • “Action item: [description]”
  • “Next steps: [list]”
  • ”@[name] please [action]”

Temporal markers:

  • “by end of day” - today at 5 PM
  • “next Friday” - calculate from meeting date
  • “ASAP” - 2 business days default
  • “before the board meeting” - check calendar API

Assignment indicators:

  • Direct: “John, can you…”
  • Confirmation: “I’ll handle…”
  • Delegation: “Let’s have Sarah…”
  • Volunteering: “I can take that”

Handling edge cases

# Multiple assignees
if "and" in assignee_text or "," in assignee_text:
    assignees = parse_multiple_assignees(assignee_text)
    create_task_for_each(assignees)


# Conditional tasks
if "if" in task_text or "assuming" in task_text:
    task.add_note("Conditional: " + extract_condition(task_text))


# Rejected tasks
if check_for_cancellation(transcript, task_timestamp):
    skip_task(task)  # Don't create if later rejected


# Vague deadlines
deadline = parse_deadline(deadline_text)
if not deadline:
    deadline = meeting_date + timedelta(days=7)  # Default 1 week

Production considerations

Rate limiting: Use exponential backoff for API calls

async function retryWithBackoff(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429) {
        await sleep(Math.pow(2, i) * 1000);  // 1s, 2s, 4s
      } else {
        throw error;
      }
    }
  }
}

Idempotency: Prevent duplicate task creation

def generate_task_fingerprint(task):
    content = f"{meeting_id}:{task['assignee']}:{task['description']}"
    return hashlib.md5(content.encode()).hexdigest()


if not task_exists(fingerprint):
    create_task(task)

Privacy and compliance:

  • Store recordings in compliant storage (HIPAA/GDPR)
  • Redact sensitive information from transcripts
  • Control access to task creation
  • Maintain audit logs linking tasks to source recordings

Testing your pipeline

test_cases = [
    {
        "transcript": "John, please send the proposal by Friday",
        "expected": {
            "assignee": "John",
            "task": "Send the proposal",
            "deadline": "Friday"
        }
    },
    {
        "transcript": "Someone should update the dashboard. Sarah: I'll do it tomorrow",
        "expected": {
            "assignee": "Sarah",
            "task": "Update the dashboard",
            "deadline": "tomorrow"
        }
    },
    {
        "transcript": "Let's review the budget next week. Actually, cancel that.",
        "expected": None  # Should not create task
    }
]

Success metrics

Track these to measure pipeline effectiveness:

  • Extraction accuracy: Percentage of real action items captured
  • False positive rate: Tasks created that weren’t real action items
  • Assignment accuracy: Correctly identified task owners
  • Deadline accuracy: Properly parsed due dates
  • Processing time: Meeting end to tasks created

What’s next

When the official feature launches, you’ll get:

  • Zero-configuration setup
  • Built-in transcription
  • Conversation thread tracking
  • The Draft state review workflow
  • Native integration with major platforms
  • Automatic user mapping via SSO
  • No coding or maintenance required

The initial release will target Zoom, Teams, and Google Meet, with more platforms to follow.

Want early access? Contact our support team to join the beta program.

Vendors > Microsoft Teams

Microsoft Teams handles communication but lacks process management capabilities like request tracking and accountability so Tallyfy fills this gap by converting Teams messages into structured workflows with assigned owners and deadlines through Power Automate or custom integrations.

Pro > Integrations

Tallyfy offers twelve distinct ways to connect with your existing business software — ranging from a full REST API and webhooks for developers to no-code middleware platforms like Zapier and Make for non-technical users — along with email integration and chat tools like Slack and upcoming BYO AI capabilities so every team can automate data sharing and eliminate manual copy-pasting between systems regardless of technical skill level.

How To > Improve accountability

Tallyfy improves task accountability through transparent workflows, automatic reminders, manual nudges, single-person ownership via Take Over, comment-based discussions, blocker documentation, task expiration policies, escalation paths, and balanced recognition systems that prevent task debt.

Vendors > Slack

Connect Slack with Tallyfy to turn conversations into tracked workflows with clear ownership, deadlines, and accountability - so chat requests don’t get buried.
Get in touch
About Tallyfy