yaml-pipeline-transfer

Skill 22: YAML 流水线转换指南

概述

YAML 流水线转换是 BK-CI 的核心功能之一，支持 Pipeline as Code（PAC）模式。本 Skill 详细介绍 YAML 与 Model 之间的双向转换机制、模板系统、触发器配置等关键技术。

触发条件

当用户需要实现以下功能时，使用此 Skill：

• YAML 流水线解析与转换
• Model 转换为 YAML
• PAC（Pipeline as Code）实现
• 流水线模板引用
• YAML 语法校验
• 自定义 YAML 处理逻辑

---

核心组件架构

1. TransferMapper（转换映射器）

位置：common-pipeline-yaml/src/main/kotlin/.../transfer/TransferMapper.kt

职责：YAML 与对象之间的序列化/反序列化核心引擎

关键方法

object TransferMapper {
    // YAML 字符串转对象
    fun <T> to(str: String): T
    
    // 对象转 YAML 字符串
    fun toYaml(bean: Any): String
    
    // 任意对象转换
    fun <T> anyTo(any: Any?): T
    
    // 格式化 YAML
    fun formatYaml(yaml: String): String
    
    // 合并 YAML（保留注释和锚点）
    fun mergeYaml(old: String, new: String): String
    
    // 获取 YAML 第一层级的坐标定位
    fun getYamlLevelOneIndex(yaml: String): Map<String, TransferMark>
    
    // YAML 节点索引
    fun indexYaml(yaml: String, line: Int, column: Int): NodeIndex?
    
    // 标记 YAML 节点位置
    fun markYaml(index: NodeIndex, yaml: String): TransferMark?
    
    // 获取 YAML 工厂
    fun getYamlFactory(): Yaml
    
    // 获取 ObjectMapper
    fun getObjectMapper(): ObjectMapper
}

自定义特性

1. 自定义字符串引号检查器

解决 YAML on 关键字的特殊用法：

class CustomStringQuotingChecker : StringQuotingChecker() {
    override fun needToQuoteName(name: String): Boolean {
        // 自定义逻辑：on 关键字不加引号
        if (name == "on") return false
        return reservedKeyword(name) || looksLikeYAMLNumber(name)
    }
    
    // 检测十六进制数字（0x开头需要加引号）
    private fun looksLikeHexNumber(value: String): Boolean {
        if (value.length < 3) return false
        return value.startsWith("0x", ignoreCase = true)
    }
}

2. 自定义 YAML 生成器

去除换行符前的尾随空格，支持 YAML Block 输出：

override fun writeString(text: String) {
    super.writeString(removeTrailingSpaces(text))
}

private fun removeTrailingSpaces(text: String): String {
    val result = StringBuilder(text.length)
    // 逐行处理，移除每行末尾的空格
    // ...
    return result.toString()
}

3. 锚点（Anchor）管理

// 收集 YAML 中的所有锚点
private fun anchorNode(node: Node, anchors: MutableMap<String, Node>)

// 替换相同节点为锚点引用
private fun replaceAnchor(node: Node, anchors: Map<String, Node>)

// 自定义锚点生成器（保持原命名）
class CustomAnchorGenerator : AnchorGenerator {
    override fun nextAnchor(node: Node): String {
        return node.anchor  // 不重命名
    }
}

4. mergeYaml 功能

智能合并两个 YAML，保留注释和锚点：

fun mergeYaml(old: String, new: String): String {
    if (old.isBlank()) return new
    
    val oldE = getYamlFactory().parse(old.reader()).toList()
    val newE = getYamlFactory().parse(new.reader()).toMutableList()
    
    // 使用 Myers Diff 算法计算差异
    val patch = DiffUtils.diff(oldE, newE, MeyersDiffWithLinearSpace.factory().create())
    
    // 处理注释和锚点
    for (delta in patch.deltas) {
        when (delta.type) {
            DeltaType.DELETE -> {
                // 保留源文件的注释
                val sourceComment = checkCommentEvent(delta.source.lines)
                if (sourceComment.isNotEmpty()) {
                    newE.addAll(delta.target.position, sourceComment)
                }
            }
            DeltaType.INSERT -> {
                // 保留锚点信息
                anchorChecker[delta.source.position]?.let { checker ->
                    // 恢复锚点
                }
            }
        }
    }
    
    // 重建节点，恢复锚点引用
    val newNode = eventsComposer(newE).singleNode
    replaceAnchor(newNode, anchorNodes)
    
    return getYamlFactory().serialize(newNode)
}

---

2. ModelTransfer（Model 转换器）

位置：common-pipeline-yaml/src/main/kotlin/.../transfer/ModelTransfer.kt

职责：YAML ↔ Pipeline Model 的核心转换逻辑

关键方法

@Component
class ModelTransfer @Autowired constructor(
    val client: Client,
    val modelStage: StageTransfer,
    val elementTransfer: ElementTransfer,
    val variableTransfer: VariableTransfer,
    val transferCache: TransferCacheService
) {
    // YAML 转 Model
    fun yaml2Model(yamlInput: YamlTransferInput): Model
    
    // YAML 转 Setting
    fun yaml2Setting(yamlInput: YamlTransferInput): PipelineSetting
    
    // YAML 转 Labels
    fun yaml2Labels(yamlInput: YamlTransferInput): List<String>
    
    // Model 转 YAML
    fun model2Yaml(input: ModelTransferInput): PreTemplateScriptBuildYamlParser
}

yaml2Model 实现流程

fun yaml2Model(yamlInput: YamlTransferInput): Model {
    // 1. 前置切面处理
    yamlInput.aspectWrapper.setYaml4Yaml(yamlInput.yaml, BEFORE)
    
    // 2. 构建 Model 基础结构
    val stageList = mutableListOf<Stage>()
    val model = Model(
        name = yamlInput.yaml.name ?: yamlInput.pipelineInfo?.pipelineName ?: "",
        desc = yamlInput.yaml.desc ?: yamlInput.pipelineInfo?.pipelineDesc ?: "",
        stages = stageList,
        labels = emptyList(),
        instanceFromTemplate = false,
        pipelineCreator = yamlInput.pipelineInfo?.creator ?: yamlInput.userId
    )
    
    val stageIndex = AtomicInteger(0)
    
    // 3. 构建 Trigger Stage
    if (!yamlInput.yaml.checkForTemplateUse()) {
        stageList.add(modelStage.yaml2TriggerStage(yamlInput, stageIndex.incrementAndGet()))
    }
    
    // 4. 构建普通 Stage
    formatStage(yamlInput, stageList, stageIndex)
    
    // 5. 构建 Finally Stage
    formatFinally(yamlInput, stageList, stageIndex.incrementAndGet())
    
    // 6. 处理模板引用
    formatTemplate(yamlInput, model)
    
    // 7. 后置切面处理
    yamlInput.aspectWrapper.setModel4Model(model, AFTER)
    
    return model
}

yaml2Setting 实现

fun yaml2Setting(yamlInput: YamlTransferInput): PipelineSetting {
    val yaml = yamlInput.yaml
    return PipelineSetting(
        projectId = yamlInput.pipelineInfo?.projectId ?: "",
        pipelineId = yamlInput.pipelineInfo?.pipelineId ?: "",
        buildNumRule = yaml.customBuildNum,
        pipelineName = yaml.name ?: yamlInput.yamlFileName ?: yamlInput.pipelineInfo?.pipelineName ?: "",
        desc = yaml.desc ?: yamlInput.pipelineInfo?.pipelineDesc ?: "",
        
        // 并发控制
        concurrencyGroup = yaml.concurrency?.group ?: PIPELINE_SETTING_CONCURRENCY_GROUP_DEFAULT,
        concurrencyCancelInProgress = yaml.concurrency?.cancelInProgress ?: false,
        runLockType = when {
            yaml.disablePipeline == true -> PipelineRunLockType.LOCK
            yaml.concurrency?.group != null -> PipelineRunLockType.GROUP_LOCK
            else -> PipelineRunLockType.MULTIPLE
        },
        waitQueueTimeMinute = yaml.concurrency?.queueTimeoutMinutes ?: DEFAULT_WAIT_QUEUE_TIME_MINUTE,
        maxQueueSize = yaml.concurrency?.queueLength ?: DEFAULT_PIPELINE_SETTING_MAX_QUEUE_SIZE,
        maxConRunningQueueSize = yaml.concurrency?.maxParallel ?: PIPELINE_SETTING_MAX_CON_QUEUE_SIZE_MAX,
        
        // 标签和方言
        labels = yaml2Labels(yamlInput),
        pipelineAsCodeSettings = yamlSyntaxDialect2Setting(yaml.syntaxDialect),
        
        // 通知订阅
        successSubscriptionList = yamlNotice2Setting(
            projectId = yamlInput.projectCode,
            notices = yaml.notices?.filter { it.checkNotifyForSuccess() }
        ),
        failSubscriptionList = yamlNotice2Setting(
            projectId = yamlInput.projectCode,
            notices = yaml.notices?.filter { it.checkNotifyForFail() }
        ),
        
        // 其他配置
        failIfVariableInvalid = yaml.failIfVariableInvalid.nullIfDefault(false),
        buildCancelPolicy = BuildCancelPolicy.codeParse(yaml.cancelPolicy)
    )
}

---

3. ElementTransfer（元素转换器）

位置：common-pipeline-yaml/src/main/kotlin/.../transfer/ElementTransfer.kt

职责：YAML Step ↔ Model Element 转换

元素类型映射

YAML Step	Model Element	说明
uses: checkout@v2	GitCheckoutElement	代码检出
uses: <atomCode>@v*	MarketBuildAtomElement	研发商店插件
run: <script>	LinuxScriptElement / WindowsScriptElement	脚本执行
uses: manual-review@v*	ManualReviewUserTaskElement	人工审核
template: <path>	StepTemplateElement	步骤模板

关键方法

@Component
class ElementTransfer @Autowired constructor(
    val client: Client,
    val creator: TransferCreator,
    val transferCache: TransferCacheService,
    val triggerTransfer: TriggerTransfer
) {
    // YAML 转触发器
    fun yaml2Triggers(yamlInput: YamlTransferInput, elements: MutableList<Element>)
    
    // 触发器转 YAML
    fun baseTriggers2yaml(elements: List<Element>, aspectWrapper: PipelineTransferAspectWrapper): TriggerOn?
    
    // SCM 触发器转 YAML
    fun scmTriggers2Yaml(elements: List<Element>, projectId: String, aspectWrapper: PipelineTransferAspectWrapper): Map<ScmType, List<TriggerOn>>
    
    // YAML Step 转 Element
    fun yaml2Step(step: Step, job: Job, yamlInput: YamlTransferInput): Element
    
    // Element 转 YAML Step
    fun element2YamlStep(element: Element, projectId: String): PreStep
}

yaml2Step 实现示例

fun yaml2Step(step: Step, job: Job, yamlInput: YamlTransferInput): Element {
    return when {
        // checkout 步骤
        step is PreCheckoutStep -> {
            GitCheckoutElement(
                name = step.name ?: "Checkout",
                repositoryHashId = step.with?.get("repository") as? String,
                branchName = step.with?.get("ref") as? String,
                // ... 其他参数
            )
        }
        
        // uses: 插件步骤
        step.uses != null -> {
            val (atomCode, version) = parseAtomCodeAndVersion(step.uses!!)
            MarketBuildAtomElement(
                name = step.name ?: atomCode,
                atomCode = atomCode,
                version = version,
                data = step.with ?: emptyMap()
            )
        }
        
        // run: 脚本步骤
        step.run != null -> {
            when (job.runsOn) {
                JobRunsOnType.WINDOWS -> WindowsScriptElement(
                    name = step.name ?: "Script",
                    script = step.run!!,
                    scriptType 

---

*Content truncated.*