AdaDocsAda AI for Azure DevOpsTry the demo

Organization administrator configuration

Ada does not include a hosted AI service. After installing the extension, an Azure DevOps organization administrator connects it to a compatible backend that the organization operates (see DEPLOY.md). Until then, users see "AI features are not configured for this Azure DevOps organization" and the non-AI features keep working.

Configure

  1. In Azure DevOps, open Organization settings > Ada AI, tab Backend settings (or select Ada settings > Admin settings at the right end of Ada's bar). Azure DevOps decides who can open Organization settings; Ada adds no check of its own.
  2. Enter:
    • Backend URL: https:// only. It must not contain credentials, a query string or a fragment; trailing slashes are removed.
    • Display name.
    • Expected API version: default 1.0. Backends at 1.1 or later of the same major version also pass; features added in 1.1 appear when the backend offers them.
    • Trusted hostnames (optional): an allow-list policy such as *.your-organization.example.
    • Enable AI features.
  3. Select Test connection. Ada calls GET /api/metadata without following redirects. It checks the response status, content type, service identifier, API version, required capabilities and that authentication is required.
  4. Select Save configuration.
    • A backend is enabled only after a successful test.
    • Unreachable backend: a backend that cannot be reached can be saved only by ticking the override, and then stays disabled.
    • Replacing a backend: needs an explicit confirmation. Every user must then confirm the new backend before data is sent to it.

Only non-secret settings are stored. Never enter API keys, client secrets, tokens or Key Vault values here; these fields don't take them, and the stored document rejects unknown fields. Provider keys belong in your backend's secret store, or, on a backend someone runs for you, in the card below.

Your AI provider's key (a backend run for you)

Where someone runs your backend for you, such as Valor Consulting with Ada as a managed service, the backend can take your AI provider's key from Ada, so the key never has to be sent to anyone. Backend settings then shows Your AI provider's key under the backend's address.

  1. Paste the key, and the one-time setup code you received from whoever runs the backend.
  2. Select Test and save. The backend tries the key with your provider first and keeps it only if it works: encrypted, never shown again and not stored in Azure DevOps. Ada then shows when it was set and its last four characters.

Each change records who made it and when. The last 20 changes are listed under Change history.

Where settings are stored

Data Storage Scope
Backend URL, display name, enabled, API version, trusted hostnames, change history Azure DevOps extension data, collection scope This organization only
"I confirm this backend" acknowledgement Azure DevOps extension data, user scope Per user
Backend session token The browser tab's sessionStorage (memory only where the browser blocks it) Until it expires (SESSION_TTL_MINUTES), the tab closes or the backend rejects it
Project templates Azure DevOps extension data, collection scope, one document per project That project; changes need review and approval in Project Templates
Pull-request check settings (repositories, Wiki scope, reviewers) The organization's backend This organization only
AI provider key entered in Ada (a backend run for you) The organization's backend, encrypted (AES-256-GCM); never returned That backend only

Nothing else is stored in the browser: no localStorage and no cookies.

Who can change the configuration: platform limitation

Azure DevOps extension data offers no access control: any user who can run the extension can technically write collection-scoped extension data with their own credentials. The page lives in Organization settings, which Azure DevOps shows according to its own permissions, but that does not restrict the stored document either.

Ada protects users with checks that do not depend on who wrote the configuration:

  1. User confirmation. Before any content is sent, each user sees the backend's name and hostname, plus who configured it and when. They must confirm it the first time and after every change.
  2. The backend authenticates and authorizes every request.
    • Without a separate sign-in (AUTH_MODE=azure-devops, the default), Azure DevOps itself confirms each user in the organization when a session starts; the backend accepts only the configured Azure DevOps organizations.
    • With a Microsoft sign-in (AUTH_MODE=entra), users sign in with the organization's Microsoft Entra ID, and the backend accepts only its own tenant, the configured Azure DevOps organizations, and the configured users, groups or roles. It verifies organization membership with Azure DevOps at sign-in.
    • A rogue URL can't impersonate your backend's sign-in. Restricting user consent to unverified applications in Entra ID blocks look-alike sign-ins.
  3. Trusted hostname policy, stored URL validation and HTTPS are rechecked on every load.
  4. Tamper detection. A malformed or tampered configuration document is rejected.

Recommended: restrict user consent in Entra ID, use a trusted hostname policy, and review the change history regularly.

Setup

Organization settings > Ada AI > Setup shows what your organization's backend offers:

Nothing is changed from this page. The settings are the backend's own: whoever runs it sets them where it runs (in Azure: App Service > Settings > Environment variables) and restarts it. Everyone who can open Ada's organization settings sees this page; it shows no setting's value.

Pull-request documentation checks

Shown under Organization settings > Ada AI > Pull-request documentation checks when the connected backend offers them; its operator switches them on (see DEPLOY.md). Changing them requires membership in Project Collection Administrators (when the backend has its own Azure DevOps access) or the backend's administrator app role (Ada.Admin by default); everyone else sees them read-only.

Proposals and the pull-request metadata behind them are visible to everyone in the organization who uses Ada. Enable only repositories and Wikis whose documentation impact may be shared that widely.

Sprint documentation

Shown under Organization settings > Ada AI > Sprint documentation when the backend has its own Azure DevOps access. Who may change it is decided as for the pull-request documentation checks above. Nothing runs until you add a team.

Azure DevOps keeps no sprint goal, so the documents mark it TODO. A sprint counts as over when its last day has ended in UTC; set the days to wait if your team closes work later than that. Each team needs its own Wiki folder, because sprints of different teams can have the same names.

Search and diagnosis

Shown under Organization settings > Ada AI > Search and diagnosis when the backend has Azure AI Search (DEPLOY.md). Changing it requires the same administrator rights as the pull-request checks.

Ada's agent

Shown under Organization settings > Ada AI > Ada's agent when the backend has Ada's agent in the chat. Only organization administrators see and change what is here.

Skills

A skill is your organization's own guidance for a kind of work: how you write release notes, what a bug report needs, how you name branches. Skills are files in one of your repositories, in the open Agent Skills format that coding tools read too.

  1. In a repository, make a folder for the skills, for example /skills. Each skill is a folder in it that holds a SKILL.md:

    ---
    name: release-notes
    description: How we write release notes. Use for release notes and changelogs.
    ---
    
    Start with what changed for users. One line per change, newest first.
    

    The name takes lower-case letters, digits and hyphens; where it is missing or isn't one, Ada takes the folder's name. The description says when to use the skill: Ada decides by it.

  2. Under Skills, enter the project, the repository and the folder, and choose Check the folder. Ada reads it with your own access and lists the skills, and the folders that aren't one with the reason (no SKILL.md, no description).

  3. Save. Stop using skills removes the folder from Ada's settings; the files stay.

What to know:

Routines

Shown when the backend offers them: they need Claude as the AI provider, the backend's own Azure DevOps access and storage that survives restarts. Where they are missing, the Setup tab says which.

A routine is work Ada's agent does on its own, on a schedule or when a pull request merges: a weekly list of stale bugs, a note on what each merged change means for the people who run the service.

  1. Add a routine. Give it a name, the project it works in, and the task, in words, as you would ask it in the chat. Nobody is there to answer a question back, so say what matters: the area, the period, the repository.
  2. Choose what starts it:
    • A schedule: every day, Monday to Friday or once a week, and the hour it starts, in UTC. A run starts within a minute or two of its hour; if the backend was stopped then, it runs once when the backend is back.
    • A merged pull request: the routine runs once for each pull request that completes in its project. You can limit it to one repository and to merges into one branch, for example main. Ada is told which pull request it was and reads it (title, description, changed files, commits, linked work items); the task says what she should do about it. Pull requests that merge while a run is under way wait their turn, up to 20; more are left out and noted in the backend's log.
  3. Choose who sees its reports: administrators only, or everyone who can see the routine's project. A routine that a merged pull request starts can also post the report as a comment on the pull request (ticked for a new routine).
  4. Save routine. Run now starts it at once, also while it is switched off; a routine that waits for pull requests then takes the one that merged last. The list shows when it runs next or how many pull requests wait, its last report or why the last run failed, and under Earlier reports the ones before it.

Routine Reports. Each routine's history, run by run, is on the Routine Reports page on the Others tab of Ada: the routines of the project Ada is open in, what started each run (with a link to the pull request), whether the report was posted there, and the report. Administrators see every routine; everyone else sees the routines shown to the project.

A merged pull request can start a routine only where Azure DevOps tells the backend about pull requests: whoever runs the backend sets PR_DOCS_HOOK_SECRET and adds the service hook in the project (DEPLOY.md). It is the hook the pull-request documentation checks and Repository Documentation use; where it is missing, the form says so and offers schedules only.

What to know:

Connections

Shown under Organization settings > Ada AI > Connections when the backend has Ask with search and an AI provider that works with tools (Claude). A connection is an outside system that offers a Model Context Protocol (MCP) server: monitoring, logs, or any other. Ada may ask it while she answers in Ask Ada and Diagnose a Ticket. Only organization administrators see and change connections.

  1. Add a ready-made kind, or Another MCP server.
    • Azure Managed Grafana: enter your Grafana's address followed by /api/azure-mcp. Ada signs in with the backend's own identity: give that identity the Grafana Viewer role on your Grafana. Azure public cloud only, and not through a private endpoint.
    • Another MCP server: its HTTPS address, and how Ada signs in: without a sign-in, or with a secret in a header you name (sent as a bearer token in Authorization, as it is in any other header).
  2. Test and list the tools. Ada asks the server for its tools and shows each with what it does: Reads (its server says so), Its server doesn't say whether it only reads, or Changes something: not offered.
  3. Tick the tools Ada may use, at most 20 per connection. Every ticked tool is described to the AI model with each question, so tick what is needed and no more.
  4. Choose where it is used (Ask Ada, Diagnose a Ticket, and, where the backend offers Ada to coding tools, Coding tools) and Save connection.

What to know before you tick:

What goes where: DATA-HANDLING.md.

Code

Shown to organization administrators under Organization settings > Ada AI > Code when the backend offers Code with Ada (DEPLOY.md).

Repository documentation

Shown under Organization settings > Ada AI > Repository documentation when the backend offers Repository Documentation (DEPLOY.md). Everyone sees the settings; changing them requires the same administrator rights as the pull-request checks. Until they are saved once, no repository is enabled.

Without the hook nothing updates by itself; a merge that was missed is taken up by the next one, or by a run someone starts, because an update starts from the commit the documents follow. Updates are shown as started by Ada (automatic), and only administrators can stop, discard or retry them. They run on what the backend's identity can read, not a person's rights, like the rest: enable only repositories that everyone who uses Ada may read.

The backend reads the repositories with its own identity. It checks that the user who starts a run may read the chosen repositories, and the checks of findings against connected services read only enabled repositories that user may read; quotes from them can appear in the documentation of the repository being documented. Everyone with the Code tab sees every run: repository names, pull-request links and counts of findings by severity (limit that with Access). Only who started a run and administrators can stop, discard or retry it or download its files. Enable only repositories that everyone who uses Ada may read. The backend's identity needs Read, Contribute, Create branch and Contribute to pull requests on them, and if your repositories restrict branches to folders, in the ada/docs folder (ada/dev-docs for dev documents). A run asks Azure DevOps about this first, before anything is read or any AI request is made: a repository where something is missing ends as Failed, saying what, and the others go on; grant it and select Retry. If Azure DevOps still refuses the push or the pull request at the end, the repository ends as Needs permission, and its files can be downloaded instead.

Ratings

Organization settings > Ada AI > Monitoring shows, below the runs per quick action, what people said of Ada's results. Under an answer in Ask Ada and under a ticket's diagnosis, everyone can choose Right, Partly right or Wrong and add a note.

Access

Shown under Organization settings > Ada AI > Access when the backend has its own Azure DevOps access (ADO_AUTH_MODE) and, in production, STATE_STORE=blob. Only administrators (Project Collection Administrators or the backend's admin role) see and change it.

Outbound connections

Under Organization settings > Ada AI > Monitoring (backend 1.64.0), administrators see every outside address the backend called in the last 30 days: how many requests, on how many days, and the last day. Only addresses and counts are kept, never what was sent. localhost is a container beside the backend, such as web search; the search engines it asks in turn aren't counted. AdaDocs' Security and trust page lists what to expect.

Disable AI

Administrator responsibilities