Jul 12 19:20:00.441498 ip-10-0-129-169 systemd[1]: Starting Kubernetes Kubelet... Jul 12 19:20:00.759244 ip-10-0-129-169 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:20:00.759244 ip-10-0-129-169 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 12 19:20:00.759244 ip-10-0-129-169 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:20:00.759244 ip-10-0-129-169 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 12 19:20:00.759244 ip-10-0-129-169 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:20:00.760132 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.760058 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 12 19:20:00.765333 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765316 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:20:00.765333 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765329 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:20:00.765333 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765333 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765336 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765339 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765342 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765346 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765348 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765351 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765353 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765356 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765358 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765361 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765363 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765366 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765368 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765371 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765373 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765376 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765385 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765388 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765390 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:20:00.765415 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765393 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765395 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765398 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765400 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765402 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765405 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765407 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765410 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765412 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765414 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765416 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765419 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765422 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765425 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765427 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765430 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765434 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765438 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765441 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:20:00.765860 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765445 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765448 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765450 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765453 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765455 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765458 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765460 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765462 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765465 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765467 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765469 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765472 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765474 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765476 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765479 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765482 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765485 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765487 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765490 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:20:00.766358 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765493 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765495 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765497 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765501 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765503 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765505 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765508 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765510 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765512 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765515 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765517 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765521 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765524 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765526 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765529 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765531 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765534 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765536 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765538 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765541 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:20:00.766797 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765543 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765545 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765548 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765550 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765552 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765555 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765557 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765560 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765562 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765565 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765568 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765570 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765572 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765575 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765579 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765582 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765585 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765588 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.765591 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:20:00.767275 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766300 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766307 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766310 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766313 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766316 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766319 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766323 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766325 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766328 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766330 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766333 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766335 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766338 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766340 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766342 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766345 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766347 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766350 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766352 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766355 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:20:00.767712 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766357 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766360 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766363 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766365 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766368 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766370 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766373 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766375 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766378 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766380 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766382 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766384 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766387 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766389 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766391 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766394 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766396 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766399 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766402 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766404 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:20:00.768186 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766406 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766409 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766412 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766414 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766416 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766419 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766421 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766423 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766426 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766429 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766431 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766434 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766436 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766438 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766441 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766443 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766446 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766449 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766451 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:20:00.768674 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766454 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766458 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766461 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766465 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766469 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766471 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766474 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766477 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766479 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766482 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766485 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766488 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766491 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766494 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766496 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766498 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766501 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766503 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:20:00.769131 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766505 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766507 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766511 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766513 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766517 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766519 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766522 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766524 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766527 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766529 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766532 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766534 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766537 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766540 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766542 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766544 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766546 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766549 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766552 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766556 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:20:00.769567 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766558 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766561 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766903 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766908 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766911 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766913 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766920 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766923 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766925 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766927 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766930 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766932 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766935 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766938 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766940 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766942 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766945 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766948 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766953 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766955 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:20:00.770054 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766959 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766963 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766966 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766969 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766971 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766974 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766977 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766980 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766982 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766984 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766987 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766989 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766992 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766994 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766996 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.766999 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767001 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767003 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767006 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767008 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:20:00.770538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767011 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767014 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767016 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767018 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767021 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767023 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767025 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767028 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767031 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767033 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767036 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767039 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767041 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767044 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767046 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767048 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767051 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767054 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767056 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:20:00.771008 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767058 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767060 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767063 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767065 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767067 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767070 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767072 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767074 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767077 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767080 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767082 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767084 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767086 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767089 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767092 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767095 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767097 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767099 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767101 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:20:00.771510 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767103 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767107 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767110 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767113 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767118 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767120 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767123 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767125 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767127 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767130 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767132 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767135 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767137 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767139 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767142 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767144 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767147 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767149 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767152 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:20:00.771945 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767154 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767156 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767158 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767161 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767241 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767248 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767254 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767258 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767262 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767266 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767269 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767278 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767281 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767284 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767287 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767291 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767293 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767297 2565 flags.go:64] FLAG: --cgroup-root="" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767300 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767303 2565 flags.go:64] FLAG: --client-ca-file="" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767305 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767308 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767312 2565 flags.go:64] FLAG: --cluster-domain="" Jul 12 19:20:00.772402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767315 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767318 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767320 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767323 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767327 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767329 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767332 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767335 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767338 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767340 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767343 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767346 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767348 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767352 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767355 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767357 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767360 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767362 2565 flags.go:64] FLAG: --enable-server="true" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767365 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767368 2565 flags.go:64] FLAG: --event-burst="100" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767372 2565 flags.go:64] FLAG: --event-qps="50" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767374 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767377 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767379 2565 flags.go:64] FLAG: --eviction-hard="" Jul 12 19:20:00.773042 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767383 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767386 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767390 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767392 2565 flags.go:64] FLAG: --eviction-soft="" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767395 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767398 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767400 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767403 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767406 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767408 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767411 2565 flags.go:64] FLAG: --feature-gates="" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767414 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767417 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767420 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767422 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767425 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767428 2565 flags.go:64] FLAG: --help="false" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767430 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-169.ec2.internal" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767433 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767435 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767438 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767441 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767444 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767446 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 12 19:20:00.773637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767449 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767451 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767454 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767457 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767460 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767463 2565 flags.go:64] FLAG: --kube-reserved="" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767467 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767469 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767472 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767474 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767478 2565 flags.go:64] FLAG: --lock-file="" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767481 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767483 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767486 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767491 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767493 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767496 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767498 2565 flags.go:64] FLAG: --logging-format="text" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767501 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767504 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767507 2565 flags.go:64] FLAG: --manifest-url="" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767509 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767513 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767515 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767519 2565 flags.go:64] FLAG: --max-pods="110" Jul 12 19:20:00.774247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767522 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767524 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767527 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767529 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767532 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767535 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767538 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767544 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767546 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767549 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767552 2565 flags.go:64] FLAG: --pod-cidr="" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767555 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767560 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767562 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767566 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767569 2565 flags.go:64] FLAG: --port="10250" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767572 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767575 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00ea33f1af3938443" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767578 2565 flags.go:64] FLAG: --qos-reserved="" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767581 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767583 2565 flags.go:64] FLAG: --register-node="true" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767586 2565 flags.go:64] FLAG: --register-with-taints="" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767589 2565 flags.go:64] FLAG: --registry-burst="10" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767592 2565 flags.go:64] FLAG: --registry-qps="5" Jul 12 19:20:00.774851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767594 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767597 2565 flags.go:64] FLAG: --reserved-memory="" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767600 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767603 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767605 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767608 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767611 2565 flags.go:64] FLAG: --runonce="false" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767613 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767616 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767619 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767621 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767624 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767626 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767629 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767632 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767634 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767637 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767639 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767642 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767644 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767647 2565 flags.go:64] FLAG: --system-cgroups="" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767650 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767656 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767658 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767661 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 12 19:20:00.775455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767667 2565 flags.go:64] FLAG: --tls-min-version="" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767669 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767672 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767675 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767677 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767680 2565 flags.go:64] FLAG: --v="2" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767683 2565 flags.go:64] FLAG: --version="false" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767687 2565 flags.go:64] FLAG: --vmodule="" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767690 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.767693 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767809 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767813 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767816 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767818 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767821 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767824 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767828 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767831 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767833 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767835 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767838 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767840 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:20:00.776029 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767843 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767845 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767847 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767850 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767852 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767854 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767862 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767866 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767868 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767871 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767874 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767876 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767879 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767881 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767884 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767886 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767888 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767891 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767894 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:20:00.776594 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767896 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767898 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767901 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767903 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767905 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767908 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767910 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767912 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767915 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767917 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767920 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767922 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767924 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767927 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767929 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767931 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767934 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767936 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767939 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767941 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:20:00.777044 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767946 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767949 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767951 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767954 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767956 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767958 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767961 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767963 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767965 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767967 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767970 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767972 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767975 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767977 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767979 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767981 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767984 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767986 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767988 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767991 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:20:00.777534 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767993 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767995 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.767998 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768000 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768002 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768005 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768007 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768009 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768012 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768014 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768017 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768019 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768024 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768027 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768030 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768033 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768035 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768037 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768040 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:20:00.778011 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768043 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768046 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768049 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768051 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768054 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768056 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768059 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768061 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.768064 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.768068 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.774399 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.774413 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.774994 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.775001 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 12 19:20:00.778477 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775053 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775058 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775062 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775066 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775070 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775073 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775076 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775078 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775081 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775083 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775086 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775088 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775091 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775093 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775096 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775098 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775100 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775103 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775106 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:20:00.778823 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775108 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775111 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775113 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775116 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775118 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775120 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775123 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775125 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775127 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775130 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775133 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775136 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775138 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775140 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775143 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775145 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775148 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775150 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775152 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775154 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:20:00.779287 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775157 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775160 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775162 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775164 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775166 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775169 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775171 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775173 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775176 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775178 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775180 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775183 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775185 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775187 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775190 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775192 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775195 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775197 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775199 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775202 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:20:00.779754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775205 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775209 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775212 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775214 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775217 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775219 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775222 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775224 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775242 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775246 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775250 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775254 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775257 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775261 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775265 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775270 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775273 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775275 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775278 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775280 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:20:00.780250 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775282 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775285 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775288 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775290 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775294 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775297 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775300 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775303 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775305 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775308 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775311 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775313 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775316 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775319 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775321 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775324 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775327 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775330 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775332 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:20:00.780736 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775334 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.775339 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775444 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775449 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775452 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775454 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775457 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775459 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775462 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775465 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775467 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775470 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775472 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775474 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775477 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:20:00.781184 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775479 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775481 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775484 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775487 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775489 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775492 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775494 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775496 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775498 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775501 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775503 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775506 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775508 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775510 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775513 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775516 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775519 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775521 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775523 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775525 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:20:00.781576 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775528 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775530 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775532 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775535 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775537 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775539 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775542 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775544 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775546 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775548 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775551 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775553 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775555 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775558 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775560 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775563 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775565 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775568 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775570 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775572 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:20:00.782065 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775575 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775577 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775579 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775582 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775584 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775586 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775589 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775591 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775594 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775597 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775599 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775601 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775604 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775606 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775608 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775610 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775613 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775615 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775617 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775619 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:20:00.782818 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775622 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775625 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775628 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775631 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775634 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775636 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775639 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775642 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775645 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775647 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775651 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775654 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775656 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775659 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775662 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775664 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775667 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775669 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775671 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:20:00.783445 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775674 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775676 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775679 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775681 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775685 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775688 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:00.775691 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.775695 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.775790 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.777795 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.779082 2565 server.go:1019] "Starting client certificate rotation" Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.779179 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.779208 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:20:00.783890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.781465 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:20:00.785359 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.785339 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:20:00.793522 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.793506 2565 log.go:25] "Validated CRI v1 runtime API" Jul 12 19:20:00.795442 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.795430 2565 log.go:25] "Validated CRI v1 image API" Jul 12 19:20:00.796214 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.796202 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 12 19:20:00.799352 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.799335 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c19f61dc-e21a-43dc-88b4-a9010a4a69b9:/dev/nvme0n1p4 c61693d0-714d-40f2-bc1c-e78f7c53720e:/dev/nvme0n1p3] Jul 12 19:20:00.799402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.799352 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 12 19:20:00.803903 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.803800 2565 manager.go:216] Machine: {Timestamp:2026-07-12 19:20:00.80274102 +0000 UTC m=+0.265218998 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3194883 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29551d018dcf9c86ba031cd85b9b3e SystemUUID:ec29551d-018d-cf9c-86ba-031cd85b9b3e BootID:0ad8d3c8-efea-41e7-b25c-30888e0f24ef Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6d:af:62:8b:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6d:af:62:8b:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:a3:e7:85:f3:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 12 19:20:00.803903 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.803896 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 12 19:20:00.804015 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.803966 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 12 19:20:00.806134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.806116 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:20:00.806637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.806605 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 12 19:20:00.806769 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.806638 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-169.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 12 19:20:00.807303 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.807294 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 12 19:20:00.807339 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.807307 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 12 19:20:00.807339 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.807314 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:20:00.807908 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.807878 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:20:00.808794 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.808784 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 12 19:20:00.812735 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.812724 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:20:00.812864 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.812855 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 12 19:20:00.812895 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.812884 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 12 19:20:00.812921 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.812904 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 12 19:20:00.812950 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.812923 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 12 19:20:00.812950 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.812934 2565 kubelet.go:398] "Adding apiserver pod source" Jul 12 19:20:00.812950 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.812943 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 12 19:20:00.813279 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.813264 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:20:00.813324 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.813289 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:20:00.816243 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.816214 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 12 19:20:00.817548 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.817535 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 12 19:20:00.817593 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.817564 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 12 19:20:00.818141 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818129 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818144 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818150 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818156 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818164 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818169 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818176 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818183 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818194 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818201 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818210 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 12 19:20:00.818211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.818215 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 12 19:20:00.819085 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.819063 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 12 19:20:00.819147 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.819088 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 12 19:20:00.821607 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.821583 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-169.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 12 19:20:00.822088 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.822065 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 12 19:20:00.822133 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.822066 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 12 19:20:00.823286 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.823261 2565 server.go:1268] "Started kubelet" Jul 12 19:20:00.823359 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.823314 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 12 19:20:00.823803 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.823352 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 12 19:20:00.823856 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.823824 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 12 19:20:00.824027 ip-10-0-129-169 systemd[1]: Started Kubernetes Kubelet. Jul 12 19:20:00.824528 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.824512 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 12 19:20:00.825419 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.825405 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 12 19:20:00.825797 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.825777 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4chp9" Jul 12 19:20:00.828306 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.828287 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 12 19:20:00.828385 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.827375 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-169.ec2.internal.18c1a10c1d91fa51 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-169.ec2.internal,UID:ip-10-0-129-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-169.ec2.internal,},FirstTimestamp:2026-07-12 19:20:00.823212625 +0000 UTC m=+0.285690604,LastTimestamp:2026-07-12 19:20:00.823212625 +0000 UTC m=+0.285690604,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-169.ec2.internal,}" Jul 12 19:20:00.828779 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.828767 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 12 19:20:00.828826 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.828787 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 12 19:20:00.828874 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.828846 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:00.829259 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829246 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 12 19:20:00.829360 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829248 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 12 19:20:00.829360 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829361 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 12 19:20:00.829504 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829482 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 12 19:20:00.829504 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829491 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 12 19:20:00.829600 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829580 2565 factory.go:55] Registering systemd factory Jul 12 19:20:00.829600 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829596 2565 factory.go:223] Registration of the systemd container factory successfully Jul 12 19:20:00.829786 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829774 2565 factory.go:153] Registering CRI-O factory Jul 12 19:20:00.829841 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829788 2565 factory.go:223] Registration of the crio container factory successfully Jul 12 19:20:00.829841 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829827 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 12 19:20:00.829931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829843 2565 factory.go:103] Registering Raw factory Jul 12 19:20:00.829931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.829852 2565 manager.go:1196] Started watching for new ooms in manager Jul 12 19:20:00.830303 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.830291 2565 manager.go:349] Starting recovery of all containers Jul 12 19:20:00.832291 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.832274 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4chp9" Jul 12 19:20:00.839549 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.839526 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:20:00.839859 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.839844 2565 manager.go:354] Recovery completed Jul 12 19:20:00.842158 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.842136 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-169.ec2.internal\" not found" node="ip-10-0-129-169.ec2.internal" Jul 12 19:20:00.845457 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.845445 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:20:00.846955 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.846925 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:20:00.846955 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.846956 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:20:00.847054 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.846966 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:20:00.847489 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.847478 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 12 19:20:00.847548 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.847489 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 12 19:20:00.847548 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.847504 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:20:00.847603 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.847479 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:20:00.849141 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.848583 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:20:00.849141 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.848616 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:20:00.849141 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.848629 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:20:00.850016 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.850001 2565 policy_none.go:49] "None policy: Start" Jul 12 19:20:00.850071 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.850019 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 12 19:20:00.850071 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.850029 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 12 19:20:00.851959 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.851948 2565 policy_none.go:47] "Start" Jul 12 19:20:00.886779 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.886766 2565 manager.go:337] "Starting Device Plugin manager" Jul 12 19:20:00.886850 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.886794 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 12 19:20:00.886850 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.886803 2565 server.go:87] "Starting device plugin registration server" Jul 12 19:20:00.887046 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.887035 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 12 19:20:00.887094 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.887048 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 12 19:20:00.887165 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.887148 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 12 19:20:00.887265 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.887252 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 12 19:20:00.887265 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.887265 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 12 19:20:00.887707 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.887685 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 12 19:20:00.887754 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.887747 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:00.987573 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.987553 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:20:00.989707 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.988345 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:20:00.989707 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.988370 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:20:00.989707 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.988382 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:20:00.989707 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.988405 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-169.ec2.internal" Jul 12 19:20:00.997953 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:00.997938 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-169.ec2.internal" Jul 12 19:20:00.997996 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:00.997956 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-169.ec2.internal\": node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.005166 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.005130 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 12 19:20:01.007582 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.007556 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 12 19:20:01.007944 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.007863 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 12 19:20:01.009641 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.009137 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 12 19:20:01.010044 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.010030 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 12 19:20:01.012595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.012577 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:20:01.042616 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.042594 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.111218 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.111197 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal"] Jul 12 19:20:01.111306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.111266 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:20:01.111998 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.111984 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:20:01.112082 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.112013 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:20:01.112082 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.112026 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:20:01.114364 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.114349 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:20:01.114465 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.114452 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.114512 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.114479 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:20:01.114997 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.114979 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:20:01.115075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.115007 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:20:01.115075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.115022 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:20:01.115075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.115023 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:20:01.115075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.115039 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:20:01.115245 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.115083 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:20:01.117427 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.117412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.117489 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.117436 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:20:01.118056 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.118044 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:20:01.118114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.118067 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:20:01.118114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.118076 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:20:01.130906 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.130885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09c1e6aa5400fb2aac78b9d5f3971a09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal\" (UID: \"09c1e6aa5400fb2aac78b9d5f3971a09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.130986 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.130908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09c1e6aa5400fb2aac78b9d5f3971a09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal\" (UID: \"09c1e6aa5400fb2aac78b9d5f3971a09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.130986 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.130927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8296a02698d224e1818c8cb03a109cf-config\") pod \"kube-apiserver-proxy-ip-10-0-129-169.ec2.internal\" (UID: \"e8296a02698d224e1818c8cb03a109cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.141716 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.141693 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-169.ec2.internal\" not found" node="ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.143397 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.143379 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.145817 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.145803 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-169.ec2.internal\" not found" node="ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.231440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.231422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09c1e6aa5400fb2aac78b9d5f3971a09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal\" (UID: \"09c1e6aa5400fb2aac78b9d5f3971a09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.231503 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.231446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09c1e6aa5400fb2aac78b9d5f3971a09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal\" (UID: \"09c1e6aa5400fb2aac78b9d5f3971a09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.231503 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.231463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8296a02698d224e1818c8cb03a109cf-config\") pod \"kube-apiserver-proxy-ip-10-0-129-169.ec2.internal\" (UID: \"e8296a02698d224e1818c8cb03a109cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.231503 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.231496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8296a02698d224e1818c8cb03a109cf-config\") pod \"kube-apiserver-proxy-ip-10-0-129-169.ec2.internal\" (UID: \"e8296a02698d224e1818c8cb03a109cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.231625 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.231524 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09c1e6aa5400fb2aac78b9d5f3971a09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal\" (UID: \"09c1e6aa5400fb2aac78b9d5f3971a09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.231625 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.231551 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09c1e6aa5400fb2aac78b9d5f3971a09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal\" (UID: \"09c1e6aa5400fb2aac78b9d5f3971a09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.243496 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.243470 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.344217 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.344174 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.444646 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.444627 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.444701 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.444661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.449454 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.449420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" Jul 12 19:20:01.545547 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.545522 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.646010 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.645965 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.746545 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.746521 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.778818 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.778797 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 12 19:20:01.779325 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.778911 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 19:20:01.779325 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.778967 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 12 19:20:01.837247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.837193 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 19:15:01 +0000 UTC" deadline="2028-01-20 16:53:58.758916044 +0000 UTC" Jul 12 19:20:01.837410 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.837251 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13365h33m56.921691123s" Jul 12 19:20:01.847380 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.847363 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.853688 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.853666 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:20:01.948126 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:01.948008 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:01.966337 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:01.966312 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8296a02698d224e1818c8cb03a109cf.slice/crio-e692a41f7b1a1de2cead47528d9465c87b329a2dfc1f2f5f0906214aad4b3a32 WatchSource:0}: Error finding container e692a41f7b1a1de2cead47528d9465c87b329a2dfc1f2f5f0906214aad4b3a32: Status 404 returned error can't find the container with id e692a41f7b1a1de2cead47528d9465c87b329a2dfc1f2f5f0906214aad4b3a32 Jul 12 19:20:01.966691 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:01.966671 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c1e6aa5400fb2aac78b9d5f3971a09.slice/crio-eaba0df2c30ee7ba619fbf91c9e788acbb7a556163ce9ab04f57296ee5d1b3ca WatchSource:0}: Error finding container eaba0df2c30ee7ba619fbf91c9e788acbb7a556163ce9ab04f57296ee5d1b3ca: Status 404 returned error can't find the container with id eaba0df2c30ee7ba619fbf91c9e788acbb7a556163ce9ab04f57296ee5d1b3ca Jul 12 19:20:01.973084 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:01.973070 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:20:02.012495 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:02.012448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" event={"ID":"09c1e6aa5400fb2aac78b9d5f3971a09","Type":"ContainerStarted","Data":"eaba0df2c30ee7ba619fbf91c9e788acbb7a556163ce9ab04f57296ee5d1b3ca"} Jul 12 19:20:02.013341 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:02.013323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" event={"ID":"e8296a02698d224e1818c8cb03a109cf","Type":"ContainerStarted","Data":"e692a41f7b1a1de2cead47528d9465c87b329a2dfc1f2f5f0906214aad4b3a32"} Jul 12 19:20:02.048770 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:02.048750 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-169.ec2.internal\" not found" Jul 12 19:20:02.134293 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:02.134273 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:20:02.228979 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:02.228930 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" Jul 12 19:20:02.239440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:02.239422 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:20:02.240308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:02.240294 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" Jul 12 19:20:02.247871 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:02.247857 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:20:03.853594 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.853562 2565 apiserver.go:52] "Watching apiserver" Jul 12 19:20:03.860375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.860352 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 12 19:20:03.862618 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.862596 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5","openshift-dns/node-resolver-s5rxz","openshift-image-registry/node-ca-bfz54","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal","openshift-multus/multus-additional-cni-plugins-wtfc4","openshift-multus/multus-qkqnk","openshift-multus/network-metrics-daemon-mn2m5","kube-system/global-pull-secret-syncer-tp9xw","kube-system/konnectivity-agent-m45zt","openshift-cluster-node-tuning-operator/tuned-fqtv9","openshift-network-diagnostics/network-check-target-gm5m6","openshift-network-operator/iptables-alerter-2sgj9","openshift-ovn-kubernetes/ovnkube-node-p5vg2"] Jul 12 19:20:03.865054 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.865034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.867225 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.867206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.867635 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.867610 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 12 19:20:03.867732 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.867638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 12 19:20:03.867732 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.867616 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kjbw5\"" Jul 12 19:20:03.867732 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.867712 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 12 19:20:03.867975 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.867947 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 12 19:20:03.869380 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.869362 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 12 19:20:03.869535 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.869515 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 12 19:20:03.869697 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.869679 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g95ld\"" Jul 12 19:20:03.869875 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.869861 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:20:03.870153 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.870136 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.871512 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.871493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:03.872451 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.872354 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 12 19:20:03.872451 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.872383 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 12 19:20:03.872709 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.872696 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hz8mp\"" Jul 12 19:20:03.873810 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.873793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.873892 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.873860 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 12 19:20:03.874069 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.874053 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gww8\"" Jul 12 19:20:03.874129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.873800 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 12 19:20:03.874220 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.874207 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 12 19:20:03.876169 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876150 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 12 19:20:03.876374 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876354 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 12 19:20:03.876467 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876377 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xf5gf\"" Jul 12 19:20:03.876549 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876518 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 12 19:20:03.876678 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876664 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 12 19:20:03.876747 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876735 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 12 19:20:03.876926 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876910 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.877005 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.876949 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 12 19:20:03.878702 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.878686 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:03.878778 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:03.878756 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:03.879082 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879064 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 12 19:20:03.879259 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879244 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 12 19:20:03.879426 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879408 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:20:03.879510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879425 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g8nj9\"" Jul 12 19:20:03.879510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1296f2c1-b211-4c57-a082-6b31de1d0954-tmp\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.879510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-system-cni-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.879510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-etc-kubernetes\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-run\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-cni-bin\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-conf-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-sys\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-var-lib-kubelet\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-cni-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-k8s-cni-cncf-io\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysconfig\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.879690 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-kubernetes\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysctl-d\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-os-release\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-kubelet\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-host\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-tmp-dir\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879913 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gndc5\" (UniqueName: \"kubernetes.io/projected/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-kube-api-access-gndc5\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.879940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-modprobe-d\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880023 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwzb9\" (UniqueName: \"kubernetes.io/projected/1296f2c1-b211-4c57-a082-6b31de1d0954-kube-api-access-wwzb9\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-hosts-file\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-netns\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-hostroot\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880115 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-daemon-config\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysctl-conf\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-systemd\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-tuned\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-cni-multus\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-cnibin\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ac1728-30b7-40b2-83df-1ea018901a2b-cni-binary-copy\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-socket-dir-parent\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880341 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-multus-certs\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwqk\" (UniqueName: \"kubernetes.io/projected/f5ac1728-30b7-40b2-83df-1ea018901a2b-kube-api-access-2rwqk\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.880595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.880385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-lib-modules\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.881653 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.881632 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:03.883946 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.883924 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 12 19:20:03.884068 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.884015 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bvf8d\"" Jul 12 19:20:03.884309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.884289 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 12 19:20:03.884498 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.884483 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.884557 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.884501 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:20:03.887355 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.887335 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 12 19:20:03.887355 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.887352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 12 19:20:03.887508 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.887340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5b8j7\"" Jul 12 19:20:03.887508 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.887396 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 12 19:20:03.889303 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.889288 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:03.891244 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.891225 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 12 19:20:03.891510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.891493 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 12 19:20:03.891599 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.891530 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hlcb7\"" Jul 12 19:20:03.892115 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.892101 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:03.892185 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:03.892152 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:03.894148 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.894132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:03.894212 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:03.894180 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:03.903897 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.903879 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mvhjd" Jul 12 19:20:03.908936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.908914 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mvhjd" Jul 12 19:20:03.924343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.924328 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:20:03.970000 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.969950 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 12 19:20:03.980560 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7sjx\" (UniqueName: \"kubernetes.io/projected/3b759132-7122-4e46-86bd-ba17cd48696a-kube-api-access-d7sjx\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:03.980659 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-registration-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.980659 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stwhz\" (UniqueName: \"kubernetes.io/projected/91118a2a-c38a-44dd-81a9-1781479ad80f-kube-api-access-stwhz\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:03.980659 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980614 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gndc5\" (UniqueName: \"kubernetes.io/projected/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-kube-api-access-gndc5\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.980659 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzb9\" (UniqueName: \"kubernetes.io/projected/1296f2c1-b211-4c57-a082-6b31de1d0954-kube-api-access-wwzb9\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980659 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpzd2\" (UniqueName: \"kubernetes.io/projected/e9fba1c4-09d7-41e8-908a-6bd644526653-kube-api-access-wpzd2\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-system-cni-dir\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-socket-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-hostroot\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-sys-fs\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-cni-multus\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:03.980854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91118a2a-c38a-44dd-81a9-1781479ad80f-serviceca\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwqk\" (UniqueName: \"kubernetes.io/projected/f5ac1728-30b7-40b2-83df-1ea018901a2b-kube-api-access-2rwqk\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980888 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-hostroot\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-systemd\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-cni-multus\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.980988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-cnibin\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91118a2a-c38a-44dd-81a9-1781479ad80f-host\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981056 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-system-cni-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981082 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-etc-kubernetes\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-node-log\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981126 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-ovnkube-script-lib\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981134 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-system-cni-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-kubelet-config\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-etc-kubernetes\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-conf-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.981309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-env-overrides\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-conf-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-device-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhd5\" (UniqueName: \"kubernetes.io/projected/1f427980-287e-47cf-9f34-249cf62f3e01-kube-api-access-wkhd5\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-k8s-cni-cncf-io\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-run-netns\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981463 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-k8s-cni-cncf-io\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981490 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-etc-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-log-socket\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblwk\" (UniqueName: \"kubernetes.io/projected/772d0df4-415d-4630-a386-adc4c8dd0b7a-kube-api-access-bblwk\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981616 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b759132-7122-4e46-86bd-ba17cd48696a-iptables-alerter-script\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-kubernetes\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysctl-d\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-os-release\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-kubelet\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1950439f-375f-4039-8bae-8c4a1e666fd1-konnectivity-ca\") pod \"konnectivity-agent-m45zt\" (UID: \"1950439f-375f-4039-8bae-8c4a1e666fd1\") " pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:03.982130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-kubernetes\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-tmp-dir\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-kubelet\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-modprobe-d\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-systemd-units\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-etc-selinux\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-hosts-file\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-netns\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-daemon-config\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysctl-conf\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-systemd\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-tuned\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.981977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1950439f-375f-4039-8bae-8c4a1e666fd1-agent-certs\") pod \"konnectivity-agent-m45zt\" (UID: \"1950439f-375f-4039-8bae-8c4a1e666fd1\") " pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982004 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-slash\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-cni-bin\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982047 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-dbus\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:03.982874 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-tmp-dir\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-os-release\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982191 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-modprobe-d\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysctl-d\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-hosts-file\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b759132-7122-4e46-86bd-ba17cd48696a-host-slash\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982312 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-systemd\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-netns\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982386 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8565t\" (UniqueName: \"kubernetes.io/projected/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-kube-api-access-8565t\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-cnibin\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ac1728-30b7-40b2-83df-1ea018901a2b-cni-binary-copy\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-socket-dir-parent\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982485 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-multus-certs\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-lib-modules\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1296f2c1-b211-4c57-a082-6b31de1d0954-tmp\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-ovn\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.983652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-ovnkube-config\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-run\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-kubelet\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-cni-netd\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-os-release\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982764 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-cni-bin\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-sys\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-var-lib-kubelet\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-cni-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-run-ovn-kubernetes\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysconfig\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-host\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.982998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-daemon-config\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983002 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-var-lib-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.984422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9fba1c4-09d7-41e8-908a-6bd644526653-ovn-node-metrics-cert\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983075 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-lib-modules\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983116 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-run-multus-certs\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-cnibin\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983185 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysctl-conf\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5ac1728-30b7-40b2-83df-1ea018901a2b-cni-binary-copy\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-socket-dir-parent\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983644 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-host-var-lib-cni-bin\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983665 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-run\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983705 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-sys\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983724 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-sysconfig\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983747 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-var-lib-kubelet\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1296f2c1-b211-4c57-a082-6b31de1d0954-host\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.985196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.983878 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5ac1728-30b7-40b2-83df-1ea018901a2b-multus-cni-dir\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.985788 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.985771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1296f2c1-b211-4c57-a082-6b31de1d0954-tmp\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.986297 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.986277 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1296f2c1-b211-4c57-a082-6b31de1d0954-etc-tuned\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.988307 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.988285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwqk\" (UniqueName: \"kubernetes.io/projected/f5ac1728-30b7-40b2-83df-1ea018901a2b-kube-api-access-2rwqk\") pod \"multus-qkqnk\" (UID: \"f5ac1728-30b7-40b2-83df-1ea018901a2b\") " pod="openshift-multus/multus-qkqnk" Jul 12 19:20:03.988675 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.988656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzb9\" (UniqueName: \"kubernetes.io/projected/1296f2c1-b211-4c57-a082-6b31de1d0954-kube-api-access-wwzb9\") pod \"tuned-fqtv9\" (UID: \"1296f2c1-b211-4c57-a082-6b31de1d0954\") " pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:03.988748 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:03.988675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gndc5\" (UniqueName: \"kubernetes.io/projected/056af27c-be62-4cc4-8cfb-0bdee24f0bc3-kube-api-access-gndc5\") pod \"node-resolver-s5rxz\" (UID: \"056af27c-be62-4cc4-8cfb-0bdee24f0bc3\") " pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:04.058323 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.058300 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:20:04.083645 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-env-overrides\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.083757 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083651 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-device-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.083757 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhd5\" (UniqueName: \"kubernetes.io/projected/1f427980-287e-47cf-9f34-249cf62f3e01-kube-api-access-wkhd5\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.083757 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-run-netns\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.083757 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-etc-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083772 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-device-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-run-netns\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-etc-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-log-socket\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblwk\" (UniqueName: \"kubernetes.io/projected/772d0df4-415d-4630-a386-adc4c8dd0b7a-kube-api-access-bblwk\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-log-socket\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.083982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b759132-7122-4e46-86bd-ba17cd48696a-iptables-alerter-script\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:04.084025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1950439f-375f-4039-8bae-8c4a1e666fd1-konnectivity-ca\") pod \"konnectivity-agent-m45zt\" (UID: \"1950439f-375f-4039-8bae-8c4a1e666fd1\") " pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084031 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-systemd-units\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-etc-selinux\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1950439f-375f-4039-8bae-8c4a1e666fd1-agent-certs\") pod \"konnectivity-agent-m45zt\" (UID: \"1950439f-375f-4039-8bae-8c4a1e666fd1\") " pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-slash\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-cni-bin\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-dbus\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b759132-7122-4e46-86bd-ba17cd48696a-host-slash\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084212 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-etc-selinux\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8565t\" (UniqueName: \"kubernetes.io/projected/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-kube-api-access-8565t\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-env-overrides\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-ovn\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-ovnkube-config\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-systemd-units\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-kubelet\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.084444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-cni-netd\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-dbus\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-os-release\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b759132-7122-4e46-86bd-ba17cd48696a-iptables-alerter-script\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-run-ovn-kubernetes\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084487 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b759132-7122-4e46-86bd-ba17cd48696a-host-slash\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-var-lib-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084551 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-var-lib-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084573 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-cni-netd\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9fba1c4-09d7-41e8-908a-6bd644526653-ovn-node-metrics-cert\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084601 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7sjx\" (UniqueName: \"kubernetes.io/projected/3b759132-7122-4e46-86bd-ba17cd48696a-kube-api-access-d7sjx\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.085168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-registration-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084653 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stwhz\" (UniqueName: \"kubernetes.io/projected/91118a2a-c38a-44dd-81a9-1781479ad80f-kube-api-access-stwhz\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-os-release\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084695 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1950439f-375f-4039-8bae-8c4a1e666fd1-konnectivity-ca\") pod \"konnectivity-agent-m45zt\" (UID: \"1950439f-375f-4039-8bae-8c4a1e666fd1\") " pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpzd2\" (UniqueName: \"kubernetes.io/projected/e9fba1c4-09d7-41e8-908a-6bd644526653-kube-api-access-wpzd2\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-ovn\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-system-cni-dir\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-socket-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-system-cni-dir\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-run-ovn-kubernetes\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-sys-fs\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-slash\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91118a2a-c38a-44dd-81a9-1781479ad80f-serviceca\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084985 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-cni-bin\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.085894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-openvswitch\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-socket-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-ovnkube-config\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.084797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-host-kubelet\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085555 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-sys-fs\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-systemd\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/772d0df4-415d-4630-a386-adc4c8dd0b7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.085644 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-cnibin\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91118a2a-c38a-44dd-81a9-1781479ad80f-host\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f427980-287e-47cf-9f34-249cf62f3e01-registration-dir\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085706 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-cnibin\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085724 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91118a2a-c38a-44dd-81a9-1781479ad80f-host\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.085732 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret podName:bdbd098e-624f-4544-99c7-b7a6e0ef4ae4 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:04.585712296 +0000 UTC m=+3.008460125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret") pod "global-pull-secret-syncer-tp9xw" (UID: "bdbd098e-624f-4544-99c7-b7a6e0ef4ae4") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-run-systemd\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-node-log\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.086544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91118a2a-c38a-44dd-81a9-1781479ad80f-serviceca\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-ovnkube-script-lib\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-kubelet-config\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9fba1c4-09d7-41e8-908a-6bd644526653-node-log\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.085998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-kubelet-config\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.086078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.086173 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.086216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/772d0df4-415d-4630-a386-adc4c8dd0b7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.086235 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:04.5862109 +0000 UTC m=+3.008958727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:04.087139 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.086401 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9fba1c4-09d7-41e8-908a-6bd644526653-ovnkube-script-lib\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.087533 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.087200 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9fba1c4-09d7-41e8-908a-6bd644526653-ovn-node-metrics-cert\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.087585 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.087561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1950439f-375f-4039-8bae-8c4a1e666fd1-agent-certs\") pod \"konnectivity-agent-m45zt\" (UID: \"1950439f-375f-4039-8bae-8c4a1e666fd1\") " pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:04.089283 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.089265 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:04.089364 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.089290 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:04.089364 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.089302 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ds9pl for pod openshift-network-diagnostics/network-check-target-gm5m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:04.089364 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.089355 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl podName:03ff0d17-6362-4c5f-951c-68ee8cc8094a nodeName:}" failed. No retries permitted until 2026-07-12 19:20:04.589340382 +0000 UTC m=+3.012088209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ds9pl" (UniqueName: "kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl") pod "network-check-target-gm5m6" (UID: "03ff0d17-6362-4c5f-951c-68ee8cc8094a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:04.092809 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.092791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblwk\" (UniqueName: \"kubernetes.io/projected/772d0df4-415d-4630-a386-adc4c8dd0b7a-kube-api-access-bblwk\") pod \"multus-additional-cni-plugins-wtfc4\" (UID: \"772d0df4-415d-4630-a386-adc4c8dd0b7a\") " pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.092893 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.092828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8565t\" (UniqueName: \"kubernetes.io/projected/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-kube-api-access-8565t\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:04.092893 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.092869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhd5\" (UniqueName: \"kubernetes.io/projected/1f427980-287e-47cf-9f34-249cf62f3e01-kube-api-access-wkhd5\") pod \"aws-ebs-csi-driver-node-xccp5\" (UID: \"1f427980-287e-47cf-9f34-249cf62f3e01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.093343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.093323 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stwhz\" (UniqueName: \"kubernetes.io/projected/91118a2a-c38a-44dd-81a9-1781479ad80f-kube-api-access-stwhz\") pod \"node-ca-bfz54\" (UID: \"91118a2a-c38a-44dd-81a9-1781479ad80f\") " pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:04.094847 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.094829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpzd2\" (UniqueName: \"kubernetes.io/projected/e9fba1c4-09d7-41e8-908a-6bd644526653-kube-api-access-wpzd2\") pod \"ovnkube-node-p5vg2\" (UID: \"e9fba1c4-09d7-41e8-908a-6bd644526653\") " pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.094923 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.094900 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7sjx\" (UniqueName: \"kubernetes.io/projected/3b759132-7122-4e46-86bd-ba17cd48696a-kube-api-access-d7sjx\") pod \"iptables-alerter-2sgj9\" (UID: \"3b759132-7122-4e46-86bd-ba17cd48696a\") " pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:04.175448 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.175382 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qkqnk" Jul 12 19:20:04.182315 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.182297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" Jul 12 19:20:04.191993 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.191971 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s5rxz" Jul 12 19:20:04.198641 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.198625 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bfz54" Jul 12 19:20:04.206331 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.206309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:04.213101 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.213085 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" Jul 12 19:20:04.220782 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.220766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2sgj9" Jul 12 19:20:04.226762 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.226746 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" Jul 12 19:20:04.231574 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.231555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:04.589786 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.589716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:04.589786 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.589754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:04.589786 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.589784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.589894 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.589895 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.589903 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.589984 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.590005 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ds9pl for pod openshift-network-diagnostics/network-check-target-gm5m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.589943 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:05.589928286 +0000 UTC m=+4.012676114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.590055 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret podName:bdbd098e-624f-4544-99c7-b7a6e0ef4ae4 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:05.590038679 +0000 UTC m=+4.012786508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret") pod "global-pull-secret-syncer-tp9xw" (UID: "bdbd098e-624f-4544-99c7-b7a6e0ef4ae4") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:04.590078 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:04.590072 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl podName:03ff0d17-6362-4c5f-951c-68ee8cc8094a nodeName:}" failed. No retries permitted until 2026-07-12 19:20:05.590064321 +0000 UTC m=+4.012812156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ds9pl" (UniqueName: "kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl") pod "network-check-target-gm5m6" (UID: "03ff0d17-6362-4c5f-951c-68ee8cc8094a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:04.831132 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.831076 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1296f2c1_b211_4c57_a082_6b31de1d0954.slice/crio-010d1c731ca821bc60cb00ce7fc11900e91e5b7bcee8b841b7f39cd29ca1d7a0 WatchSource:0}: Error finding container 010d1c731ca821bc60cb00ce7fc11900e91e5b7bcee8b841b7f39cd29ca1d7a0: Status 404 returned error can't find the container with id 010d1c731ca821bc60cb00ce7fc11900e91e5b7bcee8b841b7f39cd29ca1d7a0 Jul 12 19:20:04.832462 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.832441 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91118a2a_c38a_44dd_81a9_1781479ad80f.slice/crio-8528992e4f3fafa2d64baa66a97a34c21a8763cb3bdebe18cf9bf96ebc37ab6e WatchSource:0}: Error finding container 8528992e4f3fafa2d64baa66a97a34c21a8763cb3bdebe18cf9bf96ebc37ab6e: Status 404 returned error can't find the container with id 8528992e4f3fafa2d64baa66a97a34c21a8763cb3bdebe18cf9bf96ebc37ab6e Jul 12 19:20:04.834124 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.834035 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b759132_7122_4e46_86bd_ba17cd48696a.slice/crio-c1c88c300a9d0dbf41c560ea44a15c0b5afd69989bcbe9093c9b027a8a291162 WatchSource:0}: Error finding container c1c88c300a9d0dbf41c560ea44a15c0b5afd69989bcbe9093c9b027a8a291162: Status 404 returned error can't find the container with id c1c88c300a9d0dbf41c560ea44a15c0b5afd69989bcbe9093c9b027a8a291162 Jul 12 19:20:04.838678 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.838656 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f427980_287e_47cf_9f34_249cf62f3e01.slice/crio-8048da71fd1f80a313055bb143332cee9033b0fa7d630890e4ee4049c861d96b WatchSource:0}: Error finding container 8048da71fd1f80a313055bb143332cee9033b0fa7d630890e4ee4049c861d96b: Status 404 returned error can't find the container with id 8048da71fd1f80a313055bb143332cee9033b0fa7d630890e4ee4049c861d96b Jul 12 19:20:04.839399 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.839371 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056af27c_be62_4cc4_8cfb_0bdee24f0bc3.slice/crio-cff6c3ffaf74bef9307ee365ca410182d770b4f65fc81c01109d483d67be85cc WatchSource:0}: Error finding container cff6c3ffaf74bef9307ee365ca410182d770b4f65fc81c01109d483d67be85cc: Status 404 returned error can't find the container with id cff6c3ffaf74bef9307ee365ca410182d770b4f65fc81c01109d483d67be85cc Jul 12 19:20:04.841066 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.840860 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fba1c4_09d7_41e8_908a_6bd644526653.slice/crio-6f8efaf53b3451404fd29e61921d27fc8d6e0be7f9f388adf16a53fd2d0ebd7f WatchSource:0}: Error finding container 6f8efaf53b3451404fd29e61921d27fc8d6e0be7f9f388adf16a53fd2d0ebd7f: Status 404 returned error can't find the container with id 6f8efaf53b3451404fd29e61921d27fc8d6e0be7f9f388adf16a53fd2d0ebd7f Jul 12 19:20:04.841522 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.841501 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772d0df4_415d_4630_a386_adc4c8dd0b7a.slice/crio-5df600ba21c9e6561752933a1bcb6fd822eb3ba1ba89aae26cdfc3f9b9c7f8ef WatchSource:0}: Error finding container 5df600ba21c9e6561752933a1bcb6fd822eb3ba1ba89aae26cdfc3f9b9c7f8ef: Status 404 returned error can't find the container with id 5df600ba21c9e6561752933a1bcb6fd822eb3ba1ba89aae26cdfc3f9b9c7f8ef Jul 12 19:20:04.842244 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:04.842207 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ac1728_30b7_40b2_83df_1ea018901a2b.slice/crio-9afc69adb56b8f42d20c6528cb7ff5c93f14e3af8f064776128ff935d2a1d1be WatchSource:0}: Error finding container 9afc69adb56b8f42d20c6528cb7ff5c93f14e3af8f064776128ff935d2a1d1be: Status 404 returned error can't find the container with id 9afc69adb56b8f42d20c6528cb7ff5c93f14e3af8f064776128ff935d2a1d1be Jul 12 19:20:04.912512 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.912378 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:15:03 +0000 UTC" deadline="2028-04-11 12:23:23.018571642 +0000 UTC" Jul 12 19:20:04.912512 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:04.912513 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15329h3m18.106063835s" Jul 12 19:20:05.050569 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.050547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:05.050659 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.050641 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:05.057458 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.057428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" event={"ID":"e8296a02698d224e1818c8cb03a109cf","Type":"ContainerStarted","Data":"0d90bd61152313d12ccffcbf0c24d69c29dd0069b7fc628d1da3e658533ed57c"} Jul 12 19:20:05.058555 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.058535 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qkqnk" event={"ID":"f5ac1728-30b7-40b2-83df-1ea018901a2b","Type":"ContainerStarted","Data":"9afc69adb56b8f42d20c6528cb7ff5c93f14e3af8f064776128ff935d2a1d1be"} Jul 12 19:20:05.059666 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.059639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" event={"ID":"1f427980-287e-47cf-9f34-249cf62f3e01","Type":"ContainerStarted","Data":"8048da71fd1f80a313055bb143332cee9033b0fa7d630890e4ee4049c861d96b"} Jul 12 19:20:05.060587 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.060566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bfz54" event={"ID":"91118a2a-c38a-44dd-81a9-1781479ad80f","Type":"ContainerStarted","Data":"8528992e4f3fafa2d64baa66a97a34c21a8763cb3bdebe18cf9bf96ebc37ab6e"} Jul 12 19:20:05.061656 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.061639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" event={"ID":"1296f2c1-b211-4c57-a082-6b31de1d0954","Type":"ContainerStarted","Data":"010d1c731ca821bc60cb00ce7fc11900e91e5b7bcee8b841b7f39cd29ca1d7a0"} Jul 12 19:20:05.062833 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.062800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m45zt" event={"ID":"1950439f-375f-4039-8bae-8c4a1e666fd1","Type":"ContainerStarted","Data":"3ca7be83e3fc038d004648465552e06e9303962fcaa1a54a0b0d3fb50d1f0c6c"} Jul 12 19:20:05.063839 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.063817 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"6f8efaf53b3451404fd29e61921d27fc8d6e0be7f9f388adf16a53fd2d0ebd7f"} Jul 12 19:20:05.064854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.064836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerStarted","Data":"5df600ba21c9e6561752933a1bcb6fd822eb3ba1ba89aae26cdfc3f9b9c7f8ef"} Jul 12 19:20:05.065736 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.065716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s5rxz" event={"ID":"056af27c-be62-4cc4-8cfb-0bdee24f0bc3","Type":"ContainerStarted","Data":"cff6c3ffaf74bef9307ee365ca410182d770b4f65fc81c01109d483d67be85cc"} Jul 12 19:20:05.066593 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.066576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2sgj9" event={"ID":"3b759132-7122-4e46-86bd-ba17cd48696a","Type":"ContainerStarted","Data":"c1c88c300a9d0dbf41c560ea44a15c0b5afd69989bcbe9093c9b027a8a291162"} Jul 12 19:20:05.074085 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.074037 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-169.ec2.internal" podStartSLOduration=2.074025701 podStartE2EDuration="2.074025701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:05.073298141 +0000 UTC m=+3.496045988" watchObservedRunningTime="2026-07-12 19:20:05.074025701 +0000 UTC m=+3.496773549" Jul 12 19:20:05.598054 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.598018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:05.598236 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.598077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:05.598236 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.598112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:05.598236 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598232 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:05.598394 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598291 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:07.598272428 +0000 UTC m=+6.021020260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:05.598731 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598713 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:05.598797 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598738 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:05.598797 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598751 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ds9pl for pod openshift-network-diagnostics/network-check-target-gm5m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:05.598898 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598802 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl podName:03ff0d17-6362-4c5f-951c-68ee8cc8094a nodeName:}" failed. No retries permitted until 2026-07-12 19:20:07.598779518 +0000 UTC m=+6.021527350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ds9pl" (UniqueName: "kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl") pod "network-check-target-gm5m6" (UID: "03ff0d17-6362-4c5f-951c-68ee8cc8094a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:05.598898 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598880 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:05.599018 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:05.598914 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret podName:bdbd098e-624f-4544-99c7-b7a6e0ef4ae4 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:07.598903422 +0000 UTC m=+6.021651254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret") pod "global-pull-secret-syncer-tp9xw" (UID: "bdbd098e-624f-4544-99c7-b7a6e0ef4ae4") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:05.913060 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.912877 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:15:03 +0000 UTC" deadline="2028-01-18 08:14:49.69379407 +0000 UTC" Jul 12 19:20:05.913060 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:05.912937 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13308h54m43.780861237s" Jul 12 19:20:06.053179 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:06.052455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:06.053179 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:06.052579 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:06.053179 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:06.052662 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:06.053179 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:06.052750 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:06.091202 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:06.091170 2565 generic.go:356] "Generic (PLEG): container finished" podID="09c1e6aa5400fb2aac78b9d5f3971a09" containerID="068c40fb61aa89e9efd03df7842bd5e1c035ce2b4dc5fd14d0f4c36edcde45a0" exitCode=0 Jul 12 19:20:06.091577 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:06.091558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" event={"ID":"09c1e6aa5400fb2aac78b9d5f3971a09","Type":"ContainerDied","Data":"068c40fb61aa89e9efd03df7842bd5e1c035ce2b4dc5fd14d0f4c36edcde45a0"} Jul 12 19:20:07.049994 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:07.049748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:07.049994 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.049870 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:07.116850 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:07.116766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" event={"ID":"09c1e6aa5400fb2aac78b9d5f3971a09","Type":"ContainerStarted","Data":"7f82cb6271f3e211da24c2f8d778da0f835fe4e78895af0572f4dfa94c3d61ee"} Jul 12 19:20:07.626497 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:07.626442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:07.626497 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:07.626501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:07.626717 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:07.626535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:07.626717 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.626668 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:07.626823 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.626729 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:11.626709963 +0000 UTC m=+10.049457795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:07.627175 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.627155 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:07.627230 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.627184 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:07.627230 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.627196 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ds9pl for pod openshift-network-diagnostics/network-check-target-gm5m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:07.627299 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.627239 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl podName:03ff0d17-6362-4c5f-951c-68ee8cc8094a nodeName:}" failed. No retries permitted until 2026-07-12 19:20:11.627225065 +0000 UTC m=+10.049972907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ds9pl" (UniqueName: "kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl") pod "network-check-target-gm5m6" (UID: "03ff0d17-6362-4c5f-951c-68ee8cc8094a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:07.627299 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.627294 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:07.627400 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:07.627327 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret podName:bdbd098e-624f-4544-99c7-b7a6e0ef4ae4 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:11.627316561 +0000 UTC m=+10.050064396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret") pod "global-pull-secret-syncer-tp9xw" (UID: "bdbd098e-624f-4544-99c7-b7a6e0ef4ae4") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:08.049917 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:08.049847 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:08.050095 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:08.049998 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:08.050095 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:08.050014 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:08.050465 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:08.050097 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:09.050153 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:09.050121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:09.050570 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:09.050248 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:10.052913 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:10.052410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:10.052913 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:10.052542 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:10.052913 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:10.052567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:10.052913 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:10.052682 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:11.050551 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:11.050517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:11.050734 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.050643 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:11.659264 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:11.659230 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:11.659682 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:11.659282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:11.659682 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:11.659312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:11.659682 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.659452 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:11.659682 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.659512 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:19.659493823 +0000 UTC m=+18.082241654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:11.659951 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.659929 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:11.660035 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.659975 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:11.660035 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.659988 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ds9pl for pod openshift-network-diagnostics/network-check-target-gm5m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:11.660136 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.660036 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl podName:03ff0d17-6362-4c5f-951c-68ee8cc8094a nodeName:}" failed. No retries permitted until 2026-07-12 19:20:19.660020041 +0000 UTC m=+18.082767882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ds9pl" (UniqueName: "kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl") pod "network-check-target-gm5m6" (UID: "03ff0d17-6362-4c5f-951c-68ee8cc8094a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:11.660136 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.660102 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:11.660236 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:11.660137 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret podName:bdbd098e-624f-4544-99c7-b7a6e0ef4ae4 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:19.660125697 +0000 UTC m=+18.082873538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret") pod "global-pull-secret-syncer-tp9xw" (UID: "bdbd098e-624f-4544-99c7-b7a6e0ef4ae4") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:12.050937 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:12.050861 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:12.051100 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:12.050997 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:12.051480 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:12.051456 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:12.051596 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:12.051577 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:13.050606 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:13.050516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:13.051036 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:13.050644 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:14.049840 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:14.049794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:14.050047 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:14.049920 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:14.050370 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:14.050331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:14.050487 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:14.050446 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:15.050379 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:15.050348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:15.050790 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:15.050464 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:16.050117 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:16.050080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:16.050274 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:16.050137 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:16.050344 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:16.050278 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:16.050405 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:16.050391 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:17.050428 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:17.050400 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:17.050810 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:17.050495 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:18.049979 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:18.049933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:18.050127 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:18.050092 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:18.050191 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:18.050172 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:18.050323 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:18.050301 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:19.050321 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:19.050152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:19.050321 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.050272 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:19.723259 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:19.723219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:19.723440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:19.723276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:19.723440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:19.723305 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:19.723440 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723432 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:19.723600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723457 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:19.723600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723466 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ds9pl for pod openshift-network-diagnostics/network-check-target-gm5m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:19.723600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723518 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl podName:03ff0d17-6362-4c5f-951c-68ee8cc8094a nodeName:}" failed. No retries permitted until 2026-07-12 19:20:35.723502981 +0000 UTC m=+34.146250808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ds9pl" (UniqueName: "kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl") pod "network-check-target-gm5m6" (UID: "03ff0d17-6362-4c5f-951c-68ee8cc8094a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:19.723600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723519 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:19.723600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723441 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:19.723600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723549 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:35.723542343 +0000 UTC m=+34.146290170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:19.723600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:19.723572 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret podName:bdbd098e-624f-4544-99c7-b7a6e0ef4ae4 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:35.723555588 +0000 UTC m=+34.146303431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret") pod "global-pull-secret-syncer-tp9xw" (UID: "bdbd098e-624f-4544-99c7-b7a6e0ef4ae4") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:20.049881 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:20.049798 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:20.049881 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:20.049827 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:20.050109 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:20.049941 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:20.050402 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:20.050379 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:21.049823 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:21.049795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:21.050008 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:21.049894 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:22.050295 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:22.050264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:22.050728 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:22.050305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:22.050728 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:22.050384 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:22.050728 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:22.050480 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:23.049838 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:23.049803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:23.050021 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:23.049930 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:24.050316 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.050166 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:24.050888 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.050181 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:24.050888 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:24.050412 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:24.050888 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:24.050484 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:24.147096 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.147069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qkqnk" event={"ID":"f5ac1728-30b7-40b2-83df-1ea018901a2b","Type":"ContainerStarted","Data":"b235afacabcf8e4d0bb6d18b9310aed5bafa498c928718dc723b53ec9e46c9e5"} Jul 12 19:20:24.148435 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.148416 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" event={"ID":"1f427980-287e-47cf-9f34-249cf62f3e01","Type":"ContainerStarted","Data":"dd7aef075119208cf113f8ef8df0d839a96e88ed9ea18a57011b49e0e8c9b1aa"} Jul 12 19:20:24.149669 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.149652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bfz54" event={"ID":"91118a2a-c38a-44dd-81a9-1781479ad80f","Type":"ContainerStarted","Data":"6f53bc8fc61eb6bf67d35e0c9346049e440c9e901ac19adad272bf94b67c7508"} Jul 12 19:20:24.151091 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.151070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" event={"ID":"1296f2c1-b211-4c57-a082-6b31de1d0954","Type":"ContainerStarted","Data":"517287dcf985e3fabfbbb7c58583a7dd819883b82c00f7394f29a7cff47d37db"} Jul 12 19:20:24.152541 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.152524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m45zt" event={"ID":"1950439f-375f-4039-8bae-8c4a1e666fd1","Type":"ContainerStarted","Data":"1254363bbdc27c63edc9674712262457021c8193bb6209b73f2bc9a1a29385c9"} Jul 12 19:20:24.154381 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.154364 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:20:24.154739 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.154718 2565 generic.go:356] "Generic (PLEG): container finished" podID="e9fba1c4-09d7-41e8-908a-6bd644526653" containerID="d0f54a464d0b31ce7c4ee27a993e8e18d56afcb62a3976a93852d79bbff5eb35" exitCode=1 Jul 12 19:20:24.154813 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.154727 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"8b4e966b93b62dfa93c24d79e59861e659f51cc72a8abbc47291e75acc3d25ba"} Jul 12 19:20:24.154852 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.154838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerDied","Data":"d0f54a464d0b31ce7c4ee27a993e8e18d56afcb62a3976a93852d79bbff5eb35"} Jul 12 19:20:24.154898 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.154884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"d9792b80d13726ff9c7bfd3fc07abd026605c0931ece9360b50ecd9df24b2fd4"} Jul 12 19:20:24.156195 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.156177 2565 generic.go:356] "Generic (PLEG): container finished" podID="772d0df4-415d-4630-a386-adc4c8dd0b7a" containerID="399d0661e399fe3a6b33e0deb437d3f2a0fa34593726f373f9e9dc11bf0f5099" exitCode=0 Jul 12 19:20:24.156289 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.156237 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerDied","Data":"399d0661e399fe3a6b33e0deb437d3f2a0fa34593726f373f9e9dc11bf0f5099"} Jul 12 19:20:24.159677 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.159660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s5rxz" event={"ID":"056af27c-be62-4cc4-8cfb-0bdee24f0bc3","Type":"ContainerStarted","Data":"9fb24e8abacac8ae6970c30f17b8ec509e37d630b7ec64005c2cc831ed140092"} Jul 12 19:20:24.162029 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.161992 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-169.ec2.internal" podStartSLOduration=21.161974483 podStartE2EDuration="21.161974483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:07.130118395 +0000 UTC m=+5.552866245" watchObservedRunningTime="2026-07-12 19:20:24.161974483 +0000 UTC m=+22.584722322" Jul 12 19:20:24.162151 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.162131 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qkqnk" podStartSLOduration=3.35881924 podStartE2EDuration="22.162125568s" totalImagesPullingTime="18.803306328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:24.161797016 +0000 UTC m=+22.584544866" watchObservedRunningTime="2026-07-12 19:20:24.162125568 +0000 UTC m=+22.584873417" Jul 12 19:20:24.175811 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.175765 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s5rxz" podStartSLOduration=3.423819235 podStartE2EDuration="22.175751874s" totalImagesPullingTime="18.751932639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:24.17513801 +0000 UTC m=+22.597885859" watchObservedRunningTime="2026-07-12 19:20:24.175751874 +0000 UTC m=+22.598499724" Jul 12 19:20:24.188126 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.188095 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bfz54" podStartSLOduration=3.389609457 podStartE2EDuration="22.188084924s" totalImagesPullingTime="18.798475467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:24.187917543 +0000 UTC m=+22.610665396" watchObservedRunningTime="2026-07-12 19:20:24.188084924 +0000 UTC m=+22.610832772" Jul 12 19:20:24.225332 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.225295 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m45zt" podStartSLOduration=7.813051149 podStartE2EDuration="22.225282961s" totalImagesPullingTime="14.412231812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:24.225075004 +0000 UTC m=+22.647822854" watchObservedRunningTime="2026-07-12 19:20:24.225282961 +0000 UTC m=+22.648030809" Jul 12 19:20:24.244882 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.244775 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fqtv9" podStartSLOduration=3.441405699 podStartE2EDuration="22.244764008s" totalImagesPullingTime="18.803358309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:24.244462228 +0000 UTC m=+22.667210076" watchObservedRunningTime="2026-07-12 19:20:24.244764008 +0000 UTC m=+22.667511869" Jul 12 19:20:24.959743 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:24.959721 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 12 19:20:25.049926 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.049906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:25.050057 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:25.050036 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:25.164734 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.164663 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" event={"ID":"1f427980-287e-47cf-9f34-249cf62f3e01","Type":"ContainerStarted","Data":"c467d3e61b60144f957f6982a8eea3f6467db5282a83939fbd70b40387eb18a7"} Jul 12 19:20:25.167627 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.167609 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:20:25.168044 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.168023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"dd5af008d7e9b914114a0a63913e4fc0c404dda65acb56d537eadee655111a6d"} Jul 12 19:20:25.168154 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.168134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"0f7277ee29f07b6a2d3cdf0a63e174b3dbb90dc65a9bd468aefc3ffbdb7715e4"} Jul 12 19:20:25.168234 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.168190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"9a71f77da3c64960b57cacb862b2aa79d96dd5ac7f1441887ab0999e7167f43d"} Jul 12 19:20:25.169566 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.169541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2sgj9" event={"ID":"3b759132-7122-4e46-86bd-ba17cd48696a","Type":"ContainerStarted","Data":"7712a220271cbe56b5165ac9ea8b391f9351a8eb0f45de8b7ec4382a9121e4d1"} Jul 12 19:20:25.183899 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.183853 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2sgj9" podStartSLOduration=4.427598818 podStartE2EDuration="23.183840535s" totalImagesPullingTime="18.756241717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:25.183520669 +0000 UTC m=+23.606268519" watchObservedRunningTime="2026-07-12 19:20:25.183840535 +0000 UTC m=+23.606588382" Jul 12 19:20:25.939645 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.939520 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T19:20:24.959738497Z","UUID":"36aedb7e-3630-43c8-8d93-404477902603","Handler":null,"Name":"","Endpoint":""} Jul 12 19:20:25.941307 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.941275 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 12 19:20:25.941307 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:25.941314 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 12 19:20:26.050080 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:26.050043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:26.050080 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:26.050080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:26.050316 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:26.050157 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:26.050511 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:26.050493 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:27.050018 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:27.049992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:27.050421 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:27.050086 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:27.176040 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:27.176002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" event={"ID":"1f427980-287e-47cf-9f34-249cf62f3e01","Type":"ContainerStarted","Data":"5c48391daf1a05a0d92733d7b52863b2d74a0ead1b3fb88270059bfbdd18213e"} Jul 12 19:20:27.179720 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:27.179557 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:20:27.180089 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:27.180069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"d5d098d58a3674730195e496901f3ca47eb8ce8ad0913f5eceb704d5c6ece775"} Jul 12 19:20:27.196297 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:27.196244 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xccp5" podStartSLOduration=4.261947527 podStartE2EDuration="25.196227513s" totalImagesPullingTime="20.934279986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:27.195765757 +0000 UTC m=+25.618513605" watchObservedRunningTime="2026-07-12 19:20:27.196227513 +0000 UTC m=+25.618975363" Jul 12 19:20:27.744069 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:27.744038 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:27.744659 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:27.744638 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:28.050550 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:28.050467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:28.050948 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:28.050596 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:28.051037 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:28.050948 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:28.051079 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:28.051060 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:28.182419 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:28.182353 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:28.183430 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:28.183407 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m45zt" Jul 12 19:20:29.050176 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:29.050021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:29.050290 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:29.050244 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:29.185739 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:29.185672 2565 generic.go:356] "Generic (PLEG): container finished" podID="772d0df4-415d-4630-a386-adc4c8dd0b7a" containerID="df9dde41fa5961b582c701166941904a2ae6a04d4121927f990ec46ec5724f2b" exitCode=0 Jul 12 19:20:29.186463 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:29.185759 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerDied","Data":"df9dde41fa5961b582c701166941904a2ae6a04d4121927f990ec46ec5724f2b"} Jul 12 19:20:29.189112 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:29.189096 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:20:29.189432 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:29.189412 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"7695202231c4ad81325f903f32cda9c19a05a05e513f351331d8ccfb1ef559a7"} Jul 12 19:20:29.189841 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:29.189828 2565 scope.go:117] "RemoveContainer" containerID="d0f54a464d0b31ce7c4ee27a993e8e18d56afcb62a3976a93852d79bbff5eb35" Jul 12 19:20:30.050552 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.050525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:30.050720 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:30.050633 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:30.050720 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.050664 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:30.050827 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:30.050722 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:30.195678 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.195615 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:20:30.196087 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.195901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" event={"ID":"e9fba1c4-09d7-41e8-908a-6bd644526653","Type":"ContainerStarted","Data":"27981704c2c0e83fe87ad1d2a4d80d89f52187757d0d9e8044bad5c662c9251d"} Jul 12 19:20:30.196211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.196197 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:30.196346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.196329 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:30.196414 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.196396 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:30.210803 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.210781 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:30.210936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.210922 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:20:30.247125 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.247084 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" podStartSLOduration=9.289863501 podStartE2EDuration="28.24706979s" totalImagesPullingTime="18.957206289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:30.224645616 +0000 UTC m=+28.647393458" watchObservedRunningTime="2026-07-12 19:20:30.24706979 +0000 UTC m=+28.669817638" Jul 12 19:20:30.614047 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.614022 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tp9xw"] Jul 12 19:20:30.614189 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.614174 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:30.614295 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:30.614277 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:30.617107 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.617085 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mn2m5"] Jul 12 19:20:30.617220 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.617206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:30.617344 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:30.617319 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:30.617655 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.617636 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gm5m6"] Jul 12 19:20:30.617746 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:30.617734 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:30.617841 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:30.617822 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:31.200124 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:31.199937 2565 generic.go:356] "Generic (PLEG): container finished" podID="772d0df4-415d-4630-a386-adc4c8dd0b7a" containerID="5a327c50cb1f42ce877075411b1256552b1d33b4c18293fe8b41242ae5ed6cd4" exitCode=0 Jul 12 19:20:31.200545 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:31.200023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerDied","Data":"5a327c50cb1f42ce877075411b1256552b1d33b4c18293fe8b41242ae5ed6cd4"} Jul 12 19:20:32.050389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:32.050359 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:32.050533 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:32.050471 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:32.050533 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:32.050522 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:32.050628 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:32.050609 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:33.049770 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:33.049706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:33.050253 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:33.049804 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:33.209565 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:33.209530 2565 generic.go:356] "Generic (PLEG): container finished" podID="772d0df4-415d-4630-a386-adc4c8dd0b7a" containerID="28741291b3700dbd1a238949db9bab84a5dcf40dc7e18ead8e9c6c915d90aa79" exitCode=0 Jul 12 19:20:33.209697 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:33.209580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerDied","Data":"28741291b3700dbd1a238949db9bab84a5dcf40dc7e18ead8e9c6c915d90aa79"} Jul 12 19:20:34.050046 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:34.050012 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:34.050483 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:34.050138 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:34.050483 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:34.050014 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:34.050483 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:34.050473 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:35.049732 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:35.049661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:35.049916 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.049762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp9xw" podUID="bdbd098e-624f-4544-99c7-b7a6e0ef4ae4" Jul 12 19:20:35.746624 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:35.746595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:35.746624 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:35.746630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:35.746649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746738 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746740 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746755 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746774 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746784 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:07.746771807 +0000 UTC m=+66.169519634 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746786 2565 projected.go:196] Error preparing data for projected volume kube-api-access-ds9pl for pod openshift-network-diagnostics/network-check-target-gm5m6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746798 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret podName:bdbd098e-624f-4544-99c7-b7a6e0ef4ae4 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:07.746791071 +0000 UTC m=+66.169538898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret") pod "global-pull-secret-syncer-tp9xw" (UID: "bdbd098e-624f-4544-99c7-b7a6e0ef4ae4") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:20:35.747101 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:35.746823 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl podName:03ff0d17-6362-4c5f-951c-68ee8cc8094a nodeName:}" failed. No retries permitted until 2026-07-12 19:21:07.746812307 +0000 UTC m=+66.169560134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ds9pl" (UniqueName: "kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl") pod "network-check-target-gm5m6" (UID: "03ff0d17-6362-4c5f-951c-68ee8cc8094a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:20:36.049722 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.049637 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:36.049722 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.049661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:36.049925 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:36.049765 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm5m6" podUID="03ff0d17-6362-4c5f-951c-68ee8cc8094a" Jul 12 19:20:36.049925 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:36.049886 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:20:36.392699 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.392625 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-169.ec2.internal" event="NodeReady" Jul 12 19:20:36.392854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.392758 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 12 19:20:36.433465 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.433435 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-26zj5"] Jul 12 19:20:36.456055 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.456033 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8nn86"] Jul 12 19:20:36.456252 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.456230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.459865 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.459834 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvb27\"" Jul 12 19:20:36.460226 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.460070 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 12 19:20:36.460524 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.460504 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 12 19:20:36.470008 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.469991 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-26zj5"] Jul 12 19:20:36.470082 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.470021 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8nn86"] Jul 12 19:20:36.470082 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.470042 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:36.473575 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.473555 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 12 19:20:36.473660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.473554 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 12 19:20:36.473660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.473555 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 12 19:20:36.473660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.473653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g7h2t\"" Jul 12 19:20:36.552905 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.552876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e91421-8a09-4c33-96b7-db7fd8477878-tmp-dir\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.553070 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.552917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.553070 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.553003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38e91421-8a09-4c33-96b7-db7fd8477878-config-volume\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.553070 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.553034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44bn\" (UniqueName: \"kubernetes.io/projected/38e91421-8a09-4c33-96b7-db7fd8477878-kube-api-access-c44bn\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.654291 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38e91421-8a09-4c33-96b7-db7fd8477878-config-volume\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.654291 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44bn\" (UniqueName: \"kubernetes.io/projected/38e91421-8a09-4c33-96b7-db7fd8477878-kube-api-access-c44bn\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.654456 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444f5432-cc95-45b7-8f56-09c2562181d0-cert\") pod \"ingress-canary-8nn86\" (UID: \"444f5432-cc95-45b7-8f56-09c2562181d0\") " pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:36.654456 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e91421-8a09-4c33-96b7-db7fd8477878-tmp-dir\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.654456 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.654456 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfqn\" (UniqueName: \"kubernetes.io/projected/444f5432-cc95-45b7-8f56-09c2562181d0-kube-api-access-pwfqn\") pod \"ingress-canary-8nn86\" (UID: \"444f5432-cc95-45b7-8f56-09c2562181d0\") " pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:36.654916 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654890 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38e91421-8a09-4c33-96b7-db7fd8477878-config-volume\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.655027 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.654950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e91421-8a09-4c33-96b7-db7fd8477878-tmp-dir\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.655027 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:36.655017 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:36.655130 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:36.655061 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:37.155048314 +0000 UTC m=+35.577796141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:20:36.666629 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.666603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44bn\" (UniqueName: \"kubernetes.io/projected/38e91421-8a09-4c33-96b7-db7fd8477878-kube-api-access-c44bn\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:36.755384 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.755352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfqn\" (UniqueName: \"kubernetes.io/projected/444f5432-cc95-45b7-8f56-09c2562181d0-kube-api-access-pwfqn\") pod \"ingress-canary-8nn86\" (UID: \"444f5432-cc95-45b7-8f56-09c2562181d0\") " pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:36.755814 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.755449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444f5432-cc95-45b7-8f56-09c2562181d0-cert\") pod \"ingress-canary-8nn86\" (UID: \"444f5432-cc95-45b7-8f56-09c2562181d0\") " pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:36.758422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.758260 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444f5432-cc95-45b7-8f56-09c2562181d0-cert\") pod \"ingress-canary-8nn86\" (UID: \"444f5432-cc95-45b7-8f56-09c2562181d0\") " pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:36.762940 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.762898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfqn\" (UniqueName: \"kubernetes.io/projected/444f5432-cc95-45b7-8f56-09c2562181d0-kube-api-access-pwfqn\") pod \"ingress-canary-8nn86\" (UID: \"444f5432-cc95-45b7-8f56-09c2562181d0\") " pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:36.778709 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:36.778689 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8nn86" Jul 12 19:20:37.050103 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:37.050023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:20:37.052830 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:37.052799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:20:37.158513 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:37.158484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:37.158662 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:37.158637 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:37.158746 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:37.158703 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:38.158684111 +0000 UTC m=+36.581431946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:20:37.433742 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:37.433714 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8nn86"] Jul 12 19:20:37.526886 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:20:37.526851 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444f5432_cc95_45b7_8f56_09c2562181d0.slice/crio-ed24423ef4c574bbcf7ed7350bf0a6bd5a380d810ec4778a20db0148eefff3c7 WatchSource:0}: Error finding container ed24423ef4c574bbcf7ed7350bf0a6bd5a380d810ec4778a20db0148eefff3c7: Status 404 returned error can't find the container with id ed24423ef4c574bbcf7ed7350bf0a6bd5a380d810ec4778a20db0148eefff3c7 Jul 12 19:20:38.050431 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.050360 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:20:38.050827 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.050360 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:20:38.052825 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.052800 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brd5c\"" Jul 12 19:20:38.052825 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.052821 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 19:20:38.053021 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.052804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:20:38.053603 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.053584 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkgdj\"" Jul 12 19:20:38.053681 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.053638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 19:20:38.166037 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.166012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:38.166146 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:38.166124 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:38.166199 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:38.166191 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:40.166173664 +0000 UTC m=+38.588921507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:20:38.223321 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.223292 2565 generic.go:356] "Generic (PLEG): container finished" podID="772d0df4-415d-4630-a386-adc4c8dd0b7a" containerID="273230eed3fb22eada35f4a84bb3a42ca92164fcbd1027d7d97d5e70b5040e30" exitCode=0 Jul 12 19:20:38.223448 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.223391 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerDied","Data":"273230eed3fb22eada35f4a84bb3a42ca92164fcbd1027d7d97d5e70b5040e30"} Jul 12 19:20:38.224612 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:38.224590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8nn86" event={"ID":"444f5432-cc95-45b7-8f56-09c2562181d0","Type":"ContainerStarted","Data":"ed24423ef4c574bbcf7ed7350bf0a6bd5a380d810ec4778a20db0148eefff3c7"} Jul 12 19:20:39.230792 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:39.230762 2565 generic.go:356] "Generic (PLEG): container finished" podID="772d0df4-415d-4630-a386-adc4c8dd0b7a" containerID="1d33458dff05ffa5f72db5342af6c9e79b7df99c1996bd604a6a79e11d047af8" exitCode=0 Jul 12 19:20:39.231241 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:39.230825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerDied","Data":"1d33458dff05ffa5f72db5342af6c9e79b7df99c1996bd604a6a79e11d047af8"} Jul 12 19:20:40.178639 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:40.178440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:40.178749 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:40.178579 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:40.178817 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:40.178754 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:44.178735524 +0000 UTC m=+42.601483364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:20:40.234951 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:40.234920 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8nn86" event={"ID":"444f5432-cc95-45b7-8f56-09c2562181d0","Type":"ContainerStarted","Data":"116883a5641b71ed6fcd92b82798fa0fa088aaf24f00c3168b67143379b62c9c"} Jul 12 19:20:40.237835 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:40.237797 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" event={"ID":"772d0df4-415d-4630-a386-adc4c8dd0b7a","Type":"ContainerStarted","Data":"f39fb38fe808987449ea19ba97dd71feb95edb3612721a6f3d972a05c7918e07"} Jul 12 19:20:40.249334 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:40.249292 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8nn86" podStartSLOduration=2.065477675 podStartE2EDuration="4.249280286s" totalImagesPullingTime="2.183802611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:40.248364723 +0000 UTC m=+38.671112571" watchObservedRunningTime="2026-07-12 19:20:40.249280286 +0000 UTC m=+38.672028136" Jul 12 19:20:40.266756 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:40.266723 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wtfc4" podStartSLOduration=8.211073269 podStartE2EDuration="38.266711024s" totalImagesPullingTime="30.055637755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:20:40.266585547 +0000 UTC m=+38.689333397" watchObservedRunningTime="2026-07-12 19:20:40.266711024 +0000 UTC m=+38.689458873" Jul 12 19:20:44.202784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:44.202750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:44.203274 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:44.202900 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:44.203274 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:44.202982 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:20:52.202952089 +0000 UTC m=+50.625699918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:20:52.253704 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:20:52.253671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:20:52.254257 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:52.253852 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:20:52.254257 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:20:52.253934 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:08.253913949 +0000 UTC m=+66.676661795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:21:02.212740 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:02.212714 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p5vg2" Jul 12 19:21:07.754642 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.754611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:21:07.755134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.754654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:21:07.755134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.754680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:21:07.757865 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.757849 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:21:07.757913 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.757854 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:21:07.757913 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.757890 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 19:21:07.765207 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:21:07.765188 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 12 19:21:07.765286 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:21:07.765243 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:11.765228532 +0000 UTC m=+130.187976359 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : secret "metrics-daemon-secret" not found Jul 12 19:21:07.767380 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.767366 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 19:21:07.768695 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.768675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdbd098e-624f-4544-99c7-b7a6e0ef4ae4-original-pull-secret\") pod \"global-pull-secret-syncer-tp9xw\" (UID: \"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4\") " pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:21:07.778430 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.778407 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9pl\" (UniqueName: \"kubernetes.io/projected/03ff0d17-6362-4c5f-951c-68ee8cc8094a-kube-api-access-ds9pl\") pod \"network-check-target-gm5m6\" (UID: \"03ff0d17-6362-4c5f-951c-68ee8cc8094a\") " pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:21:07.961558 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:07.961537 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp9xw" Jul 12 19:21:08.070608 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:08.070586 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkgdj\"" Jul 12 19:21:08.077661 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:08.077637 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tp9xw"] Jul 12 19:21:08.078737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:08.078720 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:21:08.081118 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:21:08.081094 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbd098e_624f_4544_99c7_b7a6e0ef4ae4.slice/crio-9b6019de4e98536bc6719cf3896f2192c117b92e46562a1e57961cfbea779f35 WatchSource:0}: Error finding container 9b6019de4e98536bc6719cf3896f2192c117b92e46562a1e57961cfbea779f35: Status 404 returned error can't find the container with id 9b6019de4e98536bc6719cf3896f2192c117b92e46562a1e57961cfbea779f35 Jul 12 19:21:08.195510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:08.195484 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gm5m6"] Jul 12 19:21:08.199494 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:21:08.199468 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ff0d17_6362_4c5f_951c_68ee8cc8094a.slice/crio-36754c4c6ebc3460a847c819268961ea8479a4707cf02c5b1a7695f0b3dbb0a7 WatchSource:0}: Error finding container 36754c4c6ebc3460a847c819268961ea8479a4707cf02c5b1a7695f0b3dbb0a7: Status 404 returned error can't find the container with id 36754c4c6ebc3460a847c819268961ea8479a4707cf02c5b1a7695f0b3dbb0a7 Jul 12 19:21:08.258268 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:08.258241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:21:08.258372 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:21:08.258358 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:21:08.258438 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:21:08.258410 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:40.258396003 +0000 UTC m=+98.681143830 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:21:08.296204 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:08.296176 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gm5m6" event={"ID":"03ff0d17-6362-4c5f-951c-68ee8cc8094a","Type":"ContainerStarted","Data":"36754c4c6ebc3460a847c819268961ea8479a4707cf02c5b1a7695f0b3dbb0a7"} Jul 12 19:21:08.297074 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:08.297056 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tp9xw" event={"ID":"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4","Type":"ContainerStarted","Data":"9b6019de4e98536bc6719cf3896f2192c117b92e46562a1e57961cfbea779f35"} Jul 12 19:21:13.312371 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:13.312338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gm5m6" event={"ID":"03ff0d17-6362-4c5f-951c-68ee8cc8094a","Type":"ContainerStarted","Data":"d4fc45389413f3d8c78be60489d87379a234c4eab30aa25ad8731d742bb20f5f"} Jul 12 19:21:13.312841 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:13.312428 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:21:13.313724 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:13.313695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tp9xw" event={"ID":"bdbd098e-624f-4544-99c7-b7a6e0ef4ae4","Type":"ContainerStarted","Data":"e32b17b64aba418a288665e70907e9295d394cb54594664cb9f1c39a4942a506"} Jul 12 19:21:13.327062 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:13.327022 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gm5m6" podStartSLOduration=66.66364531400001 podStartE2EDuration="1m11.327009553s" totalImagesPullingTime="4.663364239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:21:13.325704371 +0000 UTC m=+71.748452224" watchObservedRunningTime="2026-07-12 19:21:13.327009553 +0000 UTC m=+71.749757381" Jul 12 19:21:13.338976 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:13.338921 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tp9xw" podStartSLOduration=66.545779737 podStartE2EDuration="1m11.338908999s" totalImagesPullingTime="4.793129262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:21:13.338836548 +0000 UTC m=+71.761584397" watchObservedRunningTime="2026-07-12 19:21:13.338908999 +0000 UTC m=+71.761656848" Jul 12 19:21:40.260704 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:40.260670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:21:40.261081 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:21:40.260766 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:21:40.261081 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:21:40.260824 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:44.260809715 +0000 UTC m=+162.683557544 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:21:44.318751 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:21:44.318720 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gm5m6" Jul 12 19:22:11.859083 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:22:11.859044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:22:11.859532 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:22:11.859188 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 12 19:22:11.859532 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:22:11.859266 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:13.859246843 +0000 UTC m=+252.281994671 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : secret "metrics-daemon-secret" not found Jul 12 19:22:39.470168 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:22:39.470123 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-26zj5" podUID="38e91421-8a09-4c33-96b7-db7fd8477878" Jul 12 19:22:39.509083 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:22:39.509059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26zj5" Jul 12 19:22:41.062446 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:22:41.062397 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:22:44.262655 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:22:44.262622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:22:44.263047 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:22:44.262749 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:44.263047 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:22:44.262801 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls podName:38e91421-8a09-4c33-96b7-db7fd8477878 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:46.262787179 +0000 UTC m=+284.685535007 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls") pod "dns-default-26zj5" (UID: "38e91421-8a09-4c33-96b7-db7fd8477878") : secret "dns-default-metrics-tls" not found Jul 12 19:22:52.050222 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:22:52.050188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:24:07.739614 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.739577 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m"] Jul 12 19:24:07.742373 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.742355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" Jul 12 19:24:07.745055 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.745031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:24:07.745195 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.745037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bhd9q\"" Jul 12 19:24:07.746197 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.746176 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 12 19:24:07.747410 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.747389 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-r85fs"] Jul 12 19:24:07.750125 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.750104 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b6f9bdc88-xhz87"] Jul 12 19:24:07.750244 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.750228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.755836 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.755413 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 12 19:24:07.755836 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.755581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 12 19:24:07.755836 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.755602 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 12 19:24:07.755836 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.755654 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:24:07.755836 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.755746 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-79mvb\"" Jul 12 19:24:07.756736 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.756647 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.757568 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.757545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m"] Jul 12 19:24:07.759200 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.759176 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 12 19:24:07.759352 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.759215 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 12 19:24:07.759606 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.759589 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 12 19:24:07.759996 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.759795 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 12 19:24:07.760186 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.760169 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 12 19:24:07.760349 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.760331 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8rhdj\"" Jul 12 19:24:07.760457 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.760437 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 12 19:24:07.761801 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.761780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 12 19:24:07.764397 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.764373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-r85fs"] Jul 12 19:24:07.767923 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.767899 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b6f9bdc88-xhz87"] Jul 12 19:24:07.803201 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f476e372-7762-4098-a7e0-1ccca0e34c14-config\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.803290 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f476e372-7762-4098-a7e0-1ccca0e34c14-trusted-ca\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.803290 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtgt\" (UniqueName: \"kubernetes.io/projected/f476e372-7762-4098-a7e0-1ccca0e34c14-kube-api-access-jwtgt\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.803290 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803239 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pwtk\" (UniqueName: \"kubernetes.io/projected/bdd55c66-838a-4927-820b-5b708820e9de-kube-api-access-2pwtk\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.803389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f476e372-7762-4098-a7e0-1ccca0e34c14-serving-cert\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.803389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-default-certificate\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.803389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.803478 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4l7l\" (UniqueName: \"kubernetes.io/projected/af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8-kube-api-access-r4l7l\") pod \"volume-data-source-validator-78495597f6-wmm8m\" (UID: \"af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" Jul 12 19:24:07.803478 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.803478 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.803452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-stats-auth\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.837999 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.837977 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54"] Jul 12 19:24:07.840744 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.840724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" Jul 12 19:24:07.841814 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.841792 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66c879658d-8ckcn"] Jul 12 19:24:07.843384 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.843366 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-phjcc\"" Jul 12 19:24:07.844324 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.844308 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.846711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.846587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 12 19:24:07.846931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.846836 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 12 19:24:07.847233 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.847212 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 12 19:24:07.847346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.847328 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-29mst\"" Jul 12 19:24:07.851631 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.851613 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54"] Jul 12 19:24:07.852760 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.852740 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 12 19:24:07.859156 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.859136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66c879658d-8ckcn"] Jul 12 19:24:07.904624 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtgt\" (UniqueName: \"kubernetes.io/projected/f476e372-7762-4098-a7e0-1ccca0e34c14-kube-api-access-jwtgt\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.904712 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pwtk\" (UniqueName: \"kubernetes.io/projected/bdd55c66-838a-4927-820b-5b708820e9de-kube-api-access-2pwtk\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.904712 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-certificates\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.904712 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904668 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73bc10cd-271b-4d4e-95e4-b7768a383d4c-ca-trust-extracted\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.904712 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-installation-pull-secrets\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.904936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f476e372-7762-4098-a7e0-1ccca0e34c14-serving-cert\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.904936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904820 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-image-registry-private-configuration\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.904936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-trusted-ca\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.904936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-kube-api-access-7ljtz\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.904936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9p4\" (UniqueName: \"kubernetes.io/projected/de51f0fe-50e3-41f2-8ae4-534d2b22f53a-kube-api-access-wl9p4\") pod \"network-check-source-869dcfcdcd-khb54\" (UID: \"de51f0fe-50e3-41f2-8ae4-534d2b22f53a\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" Jul 12 19:24:07.904936 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-default-certificate\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.904992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4l7l\" (UniqueName: \"kubernetes.io/projected/af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8-kube-api-access-r4l7l\") pod \"volume-data-source-validator-78495597f6-wmm8m\" (UID: \"af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-stats-auth\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-bound-sa-token\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f476e372-7762-4098-a7e0-1ccca0e34c14-config\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:07.905125 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:07.905212 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:08.405188694 +0000 UTC m=+246.827936538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : configmap references non-existent config key: service-ca.crt Jul 12 19:24:07.905270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f476e372-7762-4098-a7e0-1ccca0e34c14-trusted-ca\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.905734 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:07.905734 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:07.905314 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:08.405280647 +0000 UTC m=+246.828028475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : secret "router-metrics-certs-default" not found Jul 12 19:24:07.905883 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.905863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f476e372-7762-4098-a7e0-1ccca0e34c14-config\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.906430 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.906407 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f476e372-7762-4098-a7e0-1ccca0e34c14-trusted-ca\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.907336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.907318 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-default-certificate\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.907409 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.907367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f476e372-7762-4098-a7e0-1ccca0e34c14-serving-cert\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.907603 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.907586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-stats-auth\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.921160 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.921142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pwtk\" (UniqueName: \"kubernetes.io/projected/bdd55c66-838a-4927-820b-5b708820e9de-kube-api-access-2pwtk\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:07.921243 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.921184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtgt\" (UniqueName: \"kubernetes.io/projected/f476e372-7762-4098-a7e0-1ccca0e34c14-kube-api-access-jwtgt\") pod \"console-operator-7866497494-r85fs\" (UID: \"f476e372-7762-4098-a7e0-1ccca0e34c14\") " pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:07.922210 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:07.922194 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4l7l\" (UniqueName: \"kubernetes.io/projected/af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8-kube-api-access-r4l7l\") pod \"volume-data-source-validator-78495597f6-wmm8m\" (UID: \"af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" Jul 12 19:24:08.006038 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.005987 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-image-registry-private-configuration\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006038 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-trusted-ca\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006038 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-kube-api-access-7ljtz\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006242 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9p4\" (UniqueName: \"kubernetes.io/projected/de51f0fe-50e3-41f2-8ae4-534d2b22f53a-kube-api-access-wl9p4\") pod \"network-check-source-869dcfcdcd-khb54\" (UID: \"de51f0fe-50e3-41f2-8ae4-534d2b22f53a\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" Jul 12 19:24:08.006301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-bound-sa-token\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006396 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-certificates\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006396 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73bc10cd-271b-4d4e-95e4-b7768a383d4c-ca-trust-extracted\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006396 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-installation-pull-secrets\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.006396 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.006384 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:24:08.006552 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.006399 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c879658d-8ckcn: secret "image-registry-tls" not found Jul 12 19:24:08.006552 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.006444 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls podName:73bc10cd-271b-4d4e-95e4-b7768a383d4c nodeName:}" failed. No retries permitted until 2026-07-12 19:24:08.506427379 +0000 UTC m=+246.929175207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls") pod "image-registry-66c879658d-8ckcn" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c") : secret "image-registry-tls" not found Jul 12 19:24:08.006875 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-certificates\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.007136 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73bc10cd-271b-4d4e-95e4-b7768a383d4c-ca-trust-extracted\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.007136 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.006955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-trusted-ca\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.008206 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.008186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-image-registry-private-configuration\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.008445 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.008429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-installation-pull-secrets\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.014591 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.014571 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9p4\" (UniqueName: \"kubernetes.io/projected/de51f0fe-50e3-41f2-8ae4-534d2b22f53a-kube-api-access-wl9p4\") pod \"network-check-source-869dcfcdcd-khb54\" (UID: \"de51f0fe-50e3-41f2-8ae4-534d2b22f53a\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" Jul 12 19:24:08.014847 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.014826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-bound-sa-token\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.014905 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.014862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-kube-api-access-7ljtz\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.056607 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.056591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" Jul 12 19:24:08.066602 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.066584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:08.150358 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.150273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" Jul 12 19:24:08.188887 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.188815 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m"] Jul 12 19:24:08.192340 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:08.192314 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8ac5c6_6e0c_4fbe_8003_e07ee99de6a8.slice/crio-e377cf66aadd05a69854289d5cf34c863d5e378024eb3039256ab8afeae49f8a WatchSource:0}: Error finding container e377cf66aadd05a69854289d5cf34c863d5e378024eb3039256ab8afeae49f8a: Status 404 returned error can't find the container with id e377cf66aadd05a69854289d5cf34c863d5e378024eb3039256ab8afeae49f8a Jul 12 19:24:08.206559 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.206531 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-r85fs"] Jul 12 19:24:08.210788 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:08.210761 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf476e372_7762_4098_a7e0_1ccca0e34c14.slice/crio-1528306bb8c6cf99f2f0eca634478949808c1b3cfe8bce0fede3aec1d5c52246 WatchSource:0}: Error finding container 1528306bb8c6cf99f2f0eca634478949808c1b3cfe8bce0fede3aec1d5c52246: Status 404 returned error can't find the container with id 1528306bb8c6cf99f2f0eca634478949808c1b3cfe8bce0fede3aec1d5c52246 Jul 12 19:24:08.272087 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.272027 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54"] Jul 12 19:24:08.275750 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:08.275724 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde51f0fe_50e3_41f2_8ae4_534d2b22f53a.slice/crio-88eb1fbb69c2adb7a69a5feba94f503e40d7f3e74afab184673ec5fd314b90c2 WatchSource:0}: Error finding container 88eb1fbb69c2adb7a69a5feba94f503e40d7f3e74afab184673ec5fd314b90c2: Status 404 returned error can't find the container with id 88eb1fbb69c2adb7a69a5feba94f503e40d7f3e74afab184673ec5fd314b90c2 Jul 12 19:24:08.409170 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.409143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:08.409304 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.409181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:08.409304 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.409282 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:09.409269668 +0000 UTC m=+247.832017496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : configmap references non-existent config key: service-ca.crt Jul 12 19:24:08.409304 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.409286 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:24:08.409458 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.409349 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:09.409333112 +0000 UTC m=+247.832081128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : secret "router-metrics-certs-default" not found Jul 12 19:24:08.510457 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.510417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:08.515349 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.510670 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:24:08.515349 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.510818 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c879658d-8ckcn: secret "image-registry-tls" not found Jul 12 19:24:08.515349 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:08.510879 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls podName:73bc10cd-271b-4d4e-95e4-b7768a383d4c nodeName:}" failed. No retries permitted until 2026-07-12 19:24:09.510852675 +0000 UTC m=+247.933600566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls") pod "image-registry-66c879658d-8ckcn" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c") : secret "image-registry-tls" not found Jul 12 19:24:08.722154 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.722115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" event={"ID":"de51f0fe-50e3-41f2-8ae4-534d2b22f53a","Type":"ContainerStarted","Data":"22e61bf74d5d58522cc8137a8e5d273df55218002d52a13b1c7958667a61da90"} Jul 12 19:24:08.722318 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.722211 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" event={"ID":"de51f0fe-50e3-41f2-8ae4-534d2b22f53a","Type":"ContainerStarted","Data":"88eb1fbb69c2adb7a69a5feba94f503e40d7f3e74afab184673ec5fd314b90c2"} Jul 12 19:24:08.723445 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.723413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" event={"ID":"af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8","Type":"ContainerStarted","Data":"e377cf66aadd05a69854289d5cf34c863d5e378024eb3039256ab8afeae49f8a"} Jul 12 19:24:08.724627 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:08.724601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r85fs" event={"ID":"f476e372-7762-4098-a7e0-1ccca0e34c14","Type":"ContainerStarted","Data":"1528306bb8c6cf99f2f0eca634478949808c1b3cfe8bce0fede3aec1d5c52246"} Jul 12 19:24:09.419626 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:09.419591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:09.420068 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:09.419656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:09.420068 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:09.419788 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:24:09.420068 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:09.419798 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:11.419781213 +0000 UTC m=+249.842529046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : configmap references non-existent config key: service-ca.crt Jul 12 19:24:09.420068 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:09.419863 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:11.419844366 +0000 UTC m=+249.842592205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : secret "router-metrics-certs-default" not found Jul 12 19:24:09.520726 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:09.520695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:09.520938 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:09.520871 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:24:09.520938 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:09.520889 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c879658d-8ckcn: secret "image-registry-tls" not found Jul 12 19:24:09.521077 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:09.520951 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls podName:73bc10cd-271b-4d4e-95e4-b7768a383d4c nodeName:}" failed. No retries permitted until 2026-07-12 19:24:11.520937131 +0000 UTC m=+249.943684959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls") pod "image-registry-66c879658d-8ckcn" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c") : secret "image-registry-tls" not found Jul 12 19:24:09.729629 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:09.729549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" event={"ID":"af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8","Type":"ContainerStarted","Data":"21466b419ab5297a76c276950e568c4c880576e86aba14bb0a5aede03eba6bb1"} Jul 12 19:24:09.743630 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:09.743169 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-khb54" podStartSLOduration=2.743149708 podStartE2EDuration="2.743149708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:08.736823146 +0000 UTC m=+247.159570995" watchObservedRunningTime="2026-07-12 19:24:09.743149708 +0000 UTC m=+248.165897571" Jul 12 19:24:09.743822 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:09.743794 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-wmm8m" podStartSLOduration=1.841604955 podStartE2EDuration="2.743784924s" totalImagesPullingTime="902.179969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:09.74304591 +0000 UTC m=+248.165793761" watchObservedRunningTime="2026-07-12 19:24:09.743784924 +0000 UTC m=+248.166532773" Jul 12 19:24:10.734830 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:10.734802 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/0.log" Jul 12 19:24:10.735309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:10.734841 2565 generic.go:356] "Generic (PLEG): container finished" podID="f476e372-7762-4098-a7e0-1ccca0e34c14" containerID="7fb92e11b0c913d1efb44557424dda9de0aa2a5280603d253cdb7cd40a7336ba" exitCode=255 Jul 12 19:24:10.735309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:10.734875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r85fs" event={"ID":"f476e372-7762-4098-a7e0-1ccca0e34c14","Type":"ContainerDied","Data":"7fb92e11b0c913d1efb44557424dda9de0aa2a5280603d253cdb7cd40a7336ba"} Jul 12 19:24:10.735309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:10.735165 2565 scope.go:117] "RemoveContainer" containerID="7fb92e11b0c913d1efb44557424dda9de0aa2a5280603d253cdb7cd40a7336ba" Jul 12 19:24:11.434245 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.434214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:11.434384 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.434261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:11.434384 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:11.434366 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:24:11.434457 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:11.434390 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:15.434370011 +0000 UTC m=+253.857117840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : configmap references non-existent config key: service-ca.crt Jul 12 19:24:11.434457 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:11.434424 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:15.434409204 +0000 UTC m=+253.857157032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : secret "router-metrics-certs-default" not found Jul 12 19:24:11.535211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.535179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:11.535324 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:11.535288 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:24:11.535324 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:11.535300 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c879658d-8ckcn: secret "image-registry-tls" not found Jul 12 19:24:11.535395 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:11.535342 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls podName:73bc10cd-271b-4d4e-95e4-b7768a383d4c nodeName:}" failed. No retries permitted until 2026-07-12 19:24:15.535331295 +0000 UTC m=+253.958079123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls") pod "image-registry-66c879658d-8ckcn" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c") : secret "image-registry-tls" not found Jul 12 19:24:11.739563 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.739501 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/1.log" Jul 12 19:24:11.739954 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.739937 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/0.log" Jul 12 19:24:11.740019 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.740004 2565 generic.go:356] "Generic (PLEG): container finished" podID="f476e372-7762-4098-a7e0-1ccca0e34c14" containerID="70ea5fe030ba42958e6f1d01d818beef63d04cd06d237025cfce6d99d87a8634" exitCode=255 Jul 12 19:24:11.740091 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.740073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r85fs" event={"ID":"f476e372-7762-4098-a7e0-1ccca0e34c14","Type":"ContainerDied","Data":"70ea5fe030ba42958e6f1d01d818beef63d04cd06d237025cfce6d99d87a8634"} Jul 12 19:24:11.740165 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.740152 2565 scope.go:117] "RemoveContainer" containerID="7fb92e11b0c913d1efb44557424dda9de0aa2a5280603d253cdb7cd40a7336ba" Jul 12 19:24:11.740333 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:11.740317 2565 scope.go:117] "RemoveContainer" containerID="70ea5fe030ba42958e6f1d01d818beef63d04cd06d237025cfce6d99d87a8634" Jul 12 19:24:11.740504 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:11.740480 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-r85fs_openshift-console-operator(f476e372-7762-4098-a7e0-1ccca0e34c14)\"" pod="openshift-console-operator/console-operator-7866497494-r85fs" podUID="f476e372-7762-4098-a7e0-1ccca0e34c14" Jul 12 19:24:12.264347 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.264309 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w76w5"] Jul 12 19:24:12.268489 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.268466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.271389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.271369 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 12 19:24:12.271730 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.271715 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 12 19:24:12.272565 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.272541 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 12 19:24:12.272565 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.272561 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 12 19:24:12.272744 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.272596 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jgmpf\"" Jul 12 19:24:12.283440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.283418 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w76w5"] Jul 12 19:24:12.340610 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.340584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ae7f2c8-8705-43a4-873f-7586448ec728-data-volume\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.340697 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.340615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.340697 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.340644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ae7f2c8-8705-43a4-873f-7586448ec728-crio-socket\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.340697 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.340663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshxx\" (UniqueName: \"kubernetes.io/projected/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-api-access-rshxx\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.340796 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.340736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.441243 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.441223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.441318 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.441253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ae7f2c8-8705-43a4-873f-7586448ec728-crio-socket\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.441318 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.441268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshxx\" (UniqueName: \"kubernetes.io/projected/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-api-access-rshxx\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.441318 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.441290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.441419 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.441335 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:12.441419 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.441346 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ae7f2c8-8705-43a4-873f-7586448ec728-crio-socket\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.441419 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.441384 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:12.441419 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.441398 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:12.941378106 +0000 UTC m=+251.364125934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : configmap "kube-rbac-proxy" not found Jul 12 19:24:12.441537 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.441422 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:12.941409294 +0000 UTC m=+251.364157122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:12.441537 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.441496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ae7f2c8-8705-43a4-873f-7586448ec728-data-volume\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.441772 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.441757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ae7f2c8-8705-43a4-873f-7586448ec728-data-volume\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.451701 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.451675 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshxx\" (UniqueName: \"kubernetes.io/projected/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-api-access-rshxx\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.744586 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.744562 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/1.log" Jul 12 19:24:12.744920 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.744857 2565 scope.go:117] "RemoveContainer" containerID="70ea5fe030ba42958e6f1d01d818beef63d04cd06d237025cfce6d99d87a8634" Jul 12 19:24:12.745036 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.745015 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-r85fs_openshift-console-operator(f476e372-7762-4098-a7e0-1ccca0e34c14)\"" pod="openshift-console-operator/console-operator-7866497494-r85fs" podUID="f476e372-7762-4098-a7e0-1ccca0e34c14" Jul 12 19:24:12.944791 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.944762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.944931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:12.944802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:12.944931 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.944869 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:12.944931 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.944885 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:12.944931 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.944924 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:13.944909422 +0000 UTC m=+252.367657252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : configmap "kube-rbac-proxy" not found Jul 12 19:24:12.945092 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:12.944936 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:13.944930481 +0000 UTC m=+252.367678309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:13.951395 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:13.951357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:13.951442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:13.951474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:13.951492 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:13.951532 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:13.951561 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:15.951546348 +0000 UTC m=+254.374294178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:13.951575 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:15.951569189 +0000 UTC m=+254.374317016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : configmap "kube-rbac-proxy" not found Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:13.951573 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 12 19:24:13.951854 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:13.951616 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs podName:31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6 nodeName:}" failed. No retries permitted until 2026-07-12 19:26:15.951604775 +0000 UTC m=+374.374352607 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs") pod "network-metrics-daemon-mn2m5" (UID: "31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6") : secret "metrics-daemon-secret" not found Jul 12 19:24:14.272748 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:14.272681 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s5rxz_056af27c-be62-4cc4-8cfb-0bdee24f0bc3/dns-node-resolver/0.log" Jul 12 19:24:15.271944 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.271914 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bfz54_91118a2a-c38a-44dd-81a9-1781479ad80f/node-ca/0.log" Jul 12 19:24:15.448856 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.448826 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-c65dh"] Jul 12 19:24:15.452343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.452321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.454864 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.454835 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 12 19:24:15.455170 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.455152 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 12 19:24:15.455906 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.455884 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 12 19:24:15.456169 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.456149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qssxd\"" Jul 12 19:24:15.456260 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.456172 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 12 19:24:15.459849 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.459820 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-c65dh"] Jul 12 19:24:15.461504 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.461481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:15.461614 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.461595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:15.461677 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.461635 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:24:15.461730 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.461710 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:23.461689078 +0000 UTC m=+261.884436911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : secret "router-metrics-certs-default" not found Jul 12 19:24:15.461790 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.461731 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:23.461720406 +0000 UTC m=+261.884468241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : configmap references non-existent config key: service-ca.crt Jul 12 19:24:15.562883 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.562821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7n9\" (UniqueName: \"kubernetes.io/projected/e976b1b9-1390-41b1-834b-8ea8c9e30b92-kube-api-access-4p7n9\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.562883 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.562857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e976b1b9-1390-41b1-834b-8ea8c9e30b92-signing-cabundle\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.562883 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.562878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:15.563077 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.562951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e976b1b9-1390-41b1-834b-8ea8c9e30b92-signing-key\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.563077 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.563054 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:24:15.563077 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.563071 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c879658d-8ckcn: secret "image-registry-tls" not found Jul 12 19:24:15.563167 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.563118 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls podName:73bc10cd-271b-4d4e-95e4-b7768a383d4c nodeName:}" failed. No retries permitted until 2026-07-12 19:24:23.56310444 +0000 UTC m=+261.985852268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls") pod "image-registry-66c879658d-8ckcn" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c") : secret "image-registry-tls" not found Jul 12 19:24:15.663976 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.663940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e976b1b9-1390-41b1-834b-8ea8c9e30b92-signing-key\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.664099 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.664035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7n9\" (UniqueName: \"kubernetes.io/projected/e976b1b9-1390-41b1-834b-8ea8c9e30b92-kube-api-access-4p7n9\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.664099 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.664073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e976b1b9-1390-41b1-834b-8ea8c9e30b92-signing-cabundle\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.664731 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.664704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e976b1b9-1390-41b1-834b-8ea8c9e30b92-signing-cabundle\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.666693 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.666672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e976b1b9-1390-41b1-834b-8ea8c9e30b92-signing-key\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.672050 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.672026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7n9\" (UniqueName: \"kubernetes.io/projected/e976b1b9-1390-41b1-834b-8ea8c9e30b92-kube-api-access-4p7n9\") pod \"service-ca-79f68b9f4f-c65dh\" (UID: \"e976b1b9-1390-41b1-834b-8ea8c9e30b92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.762077 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.762047 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" Jul 12 19:24:15.872262 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.872215 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8nn86_444f5432-cc95-45b7-8f56-09c2562181d0/serve-healthcheck-canary/0.log" Jul 12 19:24:15.881676 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.881649 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-c65dh"] Jul 12 19:24:15.885357 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:15.885334 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode976b1b9_1390_41b1_834b_8ea8c9e30b92.slice/crio-da899020bcdb2207d9e139a783fd28da86e5671bd58d17f7205ce3005f85e3e6 WatchSource:0}: Error finding container da899020bcdb2207d9e139a783fd28da86e5671bd58d17f7205ce3005f85e3e6: Status 404 returned error can't find the container with id da899020bcdb2207d9e139a783fd28da86e5671bd58d17f7205ce3005f85e3e6 Jul 12 19:24:15.967017 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.966988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:15.967101 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:15.967029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:15.967147 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.967123 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:15.967147 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.967134 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:15.967209 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.967195 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:19.96717746 +0000 UTC m=+258.389925290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : configmap "kube-rbac-proxy" not found Jul 12 19:24:15.967253 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:15.967210 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:19.967203332 +0000 UTC m=+258.389951160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:16.758212 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:16.758178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" event={"ID":"e976b1b9-1390-41b1-834b-8ea8c9e30b92","Type":"ContainerStarted","Data":"da899020bcdb2207d9e139a783fd28da86e5671bd58d17f7205ce3005f85e3e6"} Jul 12 19:24:17.762991 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:17.762949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" event={"ID":"e976b1b9-1390-41b1-834b-8ea8c9e30b92","Type":"ContainerStarted","Data":"db4f13e105fa1a586365b6b67264aba916abaaf15c0cc6d62964894eb7d1276c"} Jul 12 19:24:17.778332 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:17.778245 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-c65dh" podStartSLOduration=1.142528995 podStartE2EDuration="2.778225859s" totalImagesPullingTime="1.635696864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:17.777306599 +0000 UTC m=+256.200054449" watchObservedRunningTime="2026-07-12 19:24:17.778225859 +0000 UTC m=+256.200973711" Jul 12 19:24:18.067597 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:18.067522 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:18.067597 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:18.067571 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:18.067896 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:18.067884 2565 scope.go:117] "RemoveContainer" containerID="70ea5fe030ba42958e6f1d01d818beef63d04cd06d237025cfce6d99d87a8634" Jul 12 19:24:18.068063 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:18.068045 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-r85fs_openshift-console-operator(f476e372-7762-4098-a7e0-1ccca0e34c14)\"" pod="openshift-console-operator/console-operator-7866497494-r85fs" podUID="f476e372-7762-4098-a7e0-1ccca0e34c14" Jul 12 19:24:20.002027 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:20.001944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:20.002027 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:20.002024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:20.002572 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:20.002088 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:24:20.002572 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:20.002150 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:28.002134787 +0000 UTC m=+266.424882615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : configmap "kube-rbac-proxy" not found Jul 12 19:24:20.002572 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:20.002153 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:20.002572 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:20.002195 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:28.002182965 +0000 UTC m=+266.424930793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:23.531239 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:23.531162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:23.531551 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:23.531245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:23.531551 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:23.531359 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle podName:bdd55c66-838a-4927-820b-5b708820e9de nodeName:}" failed. No retries permitted until 2026-07-12 19:24:39.531344833 +0000 UTC m=+277.954092663 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle") pod "router-default-7b6f9bdc88-xhz87" (UID: "bdd55c66-838a-4927-820b-5b708820e9de") : configmap references non-existent config key: service-ca.crt Jul 12 19:24:23.533528 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:23.533508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd55c66-838a-4927-820b-5b708820e9de-metrics-certs\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:23.632346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:23.632315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:23.634563 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:23.634540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"image-registry-66c879658d-8ckcn\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:23.755716 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:23.755687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:23.882653 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:23.882625 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66c879658d-8ckcn"] Jul 12 19:24:23.885663 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:23.885636 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bc10cd_271b_4d4e_95e4_b7768a383d4c.slice/crio-69d8e87c11451d02e792ca27231d7b1148ea87721022cd098bf14b21a89c83fb WatchSource:0}: Error finding container 69d8e87c11451d02e792ca27231d7b1148ea87721022cd098bf14b21a89c83fb: Status 404 returned error can't find the container with id 69d8e87c11451d02e792ca27231d7b1148ea87721022cd098bf14b21a89c83fb Jul 12 19:24:24.787998 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:24.787952 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" event={"ID":"73bc10cd-271b-4d4e-95e4-b7768a383d4c","Type":"ContainerStarted","Data":"f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57"} Jul 12 19:24:24.788443 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:24.788013 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" event={"ID":"73bc10cd-271b-4d4e-95e4-b7768a383d4c","Type":"ContainerStarted","Data":"69d8e87c11451d02e792ca27231d7b1148ea87721022cd098bf14b21a89c83fb"} Jul 12 19:24:24.788443 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:24.788045 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:24.805622 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:24.805576 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" podStartSLOduration=17.805561861 podStartE2EDuration="17.805561861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:24.804223137 +0000 UTC m=+263.226970987" watchObservedRunningTime="2026-07-12 19:24:24.805561861 +0000 UTC m=+263.228309711" Jul 12 19:24:28.068162 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:28.068124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:28.068162 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:28.068166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:28.068583 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:28.068267 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:24:28.068583 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:28.068328 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls podName:0ae7f2c8-8705-43a4-873f-7586448ec728 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:44.068313997 +0000 UTC m=+282.491061825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w76w5" (UID: "0ae7f2c8-8705-43a4-873f-7586448ec728") : secret "insights-runtime-extractor-tls" not found Jul 12 19:24:28.068656 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:28.068635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ae7f2c8-8705-43a4-873f-7586448ec728-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:31.050823 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:31.050791 2565 scope.go:117] "RemoveContainer" containerID="70ea5fe030ba42958e6f1d01d818beef63d04cd06d237025cfce6d99d87a8634" Jul 12 19:24:31.812521 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:31.812490 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:24:31.813013 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:31.812993 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/1.log" Jul 12 19:24:31.813125 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:31.813037 2565 generic.go:356] "Generic (PLEG): container finished" podID="f476e372-7762-4098-a7e0-1ccca0e34c14" containerID="95b2b6505beab01d8161af8f0bd160dde29bd288c023611dca50f66dcaa0e2d6" exitCode=255 Jul 12 19:24:31.813183 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:31.813118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r85fs" event={"ID":"f476e372-7762-4098-a7e0-1ccca0e34c14","Type":"ContainerDied","Data":"95b2b6505beab01d8161af8f0bd160dde29bd288c023611dca50f66dcaa0e2d6"} Jul 12 19:24:31.813238 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:31.813187 2565 scope.go:117] "RemoveContainer" containerID="70ea5fe030ba42958e6f1d01d818beef63d04cd06d237025cfce6d99d87a8634" Jul 12 19:24:31.813502 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:31.813477 2565 scope.go:117] "RemoveContainer" containerID="95b2b6505beab01d8161af8f0bd160dde29bd288c023611dca50f66dcaa0e2d6" Jul 12 19:24:31.813727 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:31.813686 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-r85fs_openshift-console-operator(f476e372-7762-4098-a7e0-1ccca0e34c14)\"" pod="openshift-console-operator/console-operator-7866497494-r85fs" podUID="f476e372-7762-4098-a7e0-1ccca0e34c14" Jul 12 19:24:32.820568 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:32.820539 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:24:34.823045 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:34.823016 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66c879658d-8ckcn"] Jul 12 19:24:38.067420 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:38.067387 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:38.067827 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:38.067436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:24:38.067827 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:38.067685 2565 scope.go:117] "RemoveContainer" containerID="95b2b6505beab01d8161af8f0bd160dde29bd288c023611dca50f66dcaa0e2d6" Jul 12 19:24:38.067890 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:38.067828 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-r85fs_openshift-console-operator(f476e372-7762-4098-a7e0-1ccca0e34c14)\"" pod="openshift-console-operator/console-operator-7866497494-r85fs" podUID="f476e372-7762-4098-a7e0-1ccca0e34c14" Jul 12 19:24:39.555141 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:39.555095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:39.555875 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:39.555845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdd55c66-838a-4927-820b-5b708820e9de-service-ca-bundle\") pod \"router-default-7b6f9bdc88-xhz87\" (UID: \"bdd55c66-838a-4927-820b-5b708820e9de\") " pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:39.573152 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:39.573122 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:39.698499 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:39.698415 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b6f9bdc88-xhz87"] Jul 12 19:24:39.702087 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:39.702062 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd55c66_838a_4927_820b_5b708820e9de.slice/crio-495487e6fab4afb9bfbc3fc3ff2bc0732122518f31f7be281168e912668d4505 WatchSource:0}: Error finding container 495487e6fab4afb9bfbc3fc3ff2bc0732122518f31f7be281168e912668d4505: Status 404 returned error can't find the container with id 495487e6fab4afb9bfbc3fc3ff2bc0732122518f31f7be281168e912668d4505 Jul 12 19:24:39.843458 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:39.843386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" event={"ID":"bdd55c66-838a-4927-820b-5b708820e9de","Type":"ContainerStarted","Data":"8303866663ebe6ad72d01321bb4952eab145d5ca974f8a14f02bfbff32e1d832"} Jul 12 19:24:39.843574 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:39.843467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" event={"ID":"bdd55c66-838a-4927-820b-5b708820e9de","Type":"ContainerStarted","Data":"495487e6fab4afb9bfbc3fc3ff2bc0732122518f31f7be281168e912668d4505"} Jul 12 19:24:39.861721 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:39.861677 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" podStartSLOduration=32.861662624 podStartE2EDuration="32.861662624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:39.860638063 +0000 UTC m=+278.283385912" watchObservedRunningTime="2026-07-12 19:24:39.861662624 +0000 UTC m=+278.284410473" Jul 12 19:24:40.574171 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:40.574134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:40.577016 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:40.576988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:40.846046 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:40.845950 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:40.847288 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:40.847265 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b6f9bdc88-xhz87" Jul 12 19:24:42.510335 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:42.510299 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-26zj5" podUID="38e91421-8a09-4c33-96b7-db7fd8477878" Jul 12 19:24:42.852454 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:42.852372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26zj5" Jul 12 19:24:44.087079 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.087044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:44.089419 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.089393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ae7f2c8-8705-43a4-873f-7586448ec728-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w76w5\" (UID: \"0ae7f2c8-8705-43a4-873f-7586448ec728\") " pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:44.123619 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.123591 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xvhwm"] Jul 12 19:24:44.128415 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.128399 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.131037 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.131011 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 12 19:24:44.132017 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.132000 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 12 19:24:44.132017 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.132024 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 12 19:24:44.132173 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.132003 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 12 19:24:44.132309 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.132292 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 12 19:24:44.132374 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.132360 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2mr5p\"" Jul 12 19:24:44.137503 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.137485 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xvhwm"] Jul 12 19:24:44.288994 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.288953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22f9560-c421-4a96-8581-57789c2943fe-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.289071 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.289038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22f9560-c421-4a96-8581-57789c2943fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.289118 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.289070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzstq\" (UniqueName: \"kubernetes.io/projected/b22f9560-c421-4a96-8581-57789c2943fe-kube-api-access-xzstq\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.289152 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.289111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b22f9560-c421-4a96-8581-57789c2943fe-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.377812 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.377792 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w76w5" Jul 12 19:24:44.389652 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.389626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22f9560-c421-4a96-8581-57789c2943fe-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.389739 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.389687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22f9560-c421-4a96-8581-57789c2943fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.389739 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.389717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzstq\" (UniqueName: \"kubernetes.io/projected/b22f9560-c421-4a96-8581-57789c2943fe-kube-api-access-xzstq\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.389851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.389758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b22f9560-c421-4a96-8581-57789c2943fe-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.390470 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.390402 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b22f9560-c421-4a96-8581-57789c2943fe-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.392319 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.392297 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b22f9560-c421-4a96-8581-57789c2943fe-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.392728 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.392708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b22f9560-c421-4a96-8581-57789c2943fe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.398438 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.398414 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzstq\" (UniqueName: \"kubernetes.io/projected/b22f9560-c421-4a96-8581-57789c2943fe-kube-api-access-xzstq\") pod \"prometheus-operator-64785b6675-xvhwm\" (UID: \"b22f9560-c421-4a96-8581-57789c2943fe\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.437951 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.437921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" Jul 12 19:24:44.515020 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.514995 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w76w5"] Jul 12 19:24:44.518634 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:44.518603 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae7f2c8_8705_43a4_873f_7586448ec728.slice/crio-252cf317c25e5d94bd735e4b8af14179b21271ed277bae46d3b84f82d84a9bd5 WatchSource:0}: Error finding container 252cf317c25e5d94bd735e4b8af14179b21271ed277bae46d3b84f82d84a9bd5: Status 404 returned error can't find the container with id 252cf317c25e5d94bd735e4b8af14179b21271ed277bae46d3b84f82d84a9bd5 Jul 12 19:24:44.567882 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.567817 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xvhwm"] Jul 12 19:24:44.570121 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:44.570096 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22f9560_c421_4a96_8581_57789c2943fe.slice/crio-9450af6d98cb58eee9264d42e678a254563d9bcfea88e2248aefcdb3b23fa4e2 WatchSource:0}: Error finding container 9450af6d98cb58eee9264d42e678a254563d9bcfea88e2248aefcdb3b23fa4e2: Status 404 returned error can't find the container with id 9450af6d98cb58eee9264d42e678a254563d9bcfea88e2248aefcdb3b23fa4e2 Jul 12 19:24:44.828005 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.827898 2565 patch_prober.go:28] interesting pod/image-registry-66c879658d-8ckcn container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 12 19:24:44.828005 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.827978 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" podUID="73bc10cd-271b-4d4e-95e4-b7768a383d4c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:24:44.863282 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.863248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" event={"ID":"b22f9560-c421-4a96-8581-57789c2943fe","Type":"ContainerStarted","Data":"9450af6d98cb58eee9264d42e678a254563d9bcfea88e2248aefcdb3b23fa4e2"} Jul 12 19:24:44.864327 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:44.864304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w76w5" event={"ID":"0ae7f2c8-8705-43a4-873f-7586448ec728","Type":"ContainerStarted","Data":"252cf317c25e5d94bd735e4b8af14179b21271ed277bae46d3b84f82d84a9bd5"} Jul 12 19:24:46.305865 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:46.305816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:24:46.308458 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:46.308434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e91421-8a09-4c33-96b7-db7fd8477878-metrics-tls\") pod \"dns-default-26zj5\" (UID: \"38e91421-8a09-4c33-96b7-db7fd8477878\") " pod="openshift-dns/dns-default-26zj5" Jul 12 19:24:46.455326 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:46.455291 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qvb27\"" Jul 12 19:24:46.463281 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:46.463257 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26zj5" Jul 12 19:24:46.873606 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:46.873573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" event={"ID":"b22f9560-c421-4a96-8581-57789c2943fe","Type":"ContainerStarted","Data":"ccf0ab5401c1152aea5fa6535f411f3393e4150ed1355dcc84b4d72c5343a79c"} Jul 12 19:24:47.879784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:47.879743 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" event={"ID":"b22f9560-c421-4a96-8581-57789c2943fe","Type":"ContainerStarted","Data":"796e476e6667252547127dac5e6ca14e9b941695a967f492959c3e6cdd606d3f"} Jul 12 19:24:47.898266 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:47.898222 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-xvhwm" podStartSLOduration=2.405407328 podStartE2EDuration="3.898208747s" totalImagesPullingTime="1.492801419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:47.897978103 +0000 UTC m=+286.320725944" watchObservedRunningTime="2026-07-12 19:24:47.898208747 +0000 UTC m=+286.320956591" Jul 12 19:24:48.430895 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:48.430865 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-26zj5"] Jul 12 19:24:48.434151 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:48.434126 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e91421_8a09_4c33_96b7_db7fd8477878.slice/crio-e3ab822f8e23febefdabcff310637804317167df4fe5b62097c7b5a68ad3276a WatchSource:0}: Error finding container e3ab822f8e23febefdabcff310637804317167df4fe5b62097c7b5a68ad3276a: Status 404 returned error can't find the container with id e3ab822f8e23febefdabcff310637804317167df4fe5b62097c7b5a68ad3276a Jul 12 19:24:48.885189 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:48.885159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-26zj5" event={"ID":"38e91421-8a09-4c33-96b7-db7fd8477878","Type":"ContainerStarted","Data":"e3ab822f8e23febefdabcff310637804317167df4fe5b62097c7b5a68ad3276a"} Jul 12 19:24:48.886866 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:48.886825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w76w5" event={"ID":"0ae7f2c8-8705-43a4-873f-7586448ec728","Type":"ContainerStarted","Data":"92150c533707563946b9c4603b6c688736e4ded12c82ebfca29ab871c90e19d7"} Jul 12 19:24:49.049843 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.049818 2565 scope.go:117] "RemoveContainer" containerID="95b2b6505beab01d8161af8f0bd160dde29bd288c023611dca50f66dcaa0e2d6" Jul 12 19:24:49.050046 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:49.050018 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-r85fs_openshift-console-operator(f476e372-7762-4098-a7e0-1ccca0e34c14)\"" pod="openshift-console-operator/console-operator-7866497494-r85fs" podUID="f476e372-7762-4098-a7e0-1ccca0e34c14" Jul 12 19:24:49.474712 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.474164 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx"] Jul 12 19:24:49.478733 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.478015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.483877 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.483854 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 12 19:24:49.484118 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.484102 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 12 19:24:49.484301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.484287 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mf2dm\"" Jul 12 19:24:49.504795 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.504773 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-fqjfz"] Jul 12 19:24:49.509694 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.509548 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx"] Jul 12 19:24:49.509840 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.509830 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f5frl"] Jul 12 19:24:49.512008 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.509673 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.514451 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.513939 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-fqjfz"] Jul 12 19:24:49.514451 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.514093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.514990 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.514701 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 12 19:24:49.515063 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.515031 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 12 19:24:49.515181 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.515165 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-69dzq\"" Jul 12 19:24:49.515430 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.515399 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 12 19:24:49.522451 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.522428 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 12 19:24:49.528433 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.528410 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5d6qj\"" Jul 12 19:24:49.528533 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.528512 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 12 19:24:49.528786 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.528769 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 12 19:24:49.636332 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.636447 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.636447 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636386 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.636447 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edde6206-cd48-41df-b389-bf0ab450d3dc-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.636447 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96l82\" (UniqueName: \"kubernetes.io/projected/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-api-access-96l82\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-root\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636497 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636535 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-wtmp\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edde6206-cd48-41df-b389-bf0ab450d3dc-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwlqc\" (UniqueName: \"kubernetes.io/projected/190fdc8d-10ff-453b-8b60-103e9e4823d5-kube-api-access-dwlqc\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8145791c-f93b-41ea-9e4c-089f2775ce52-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636668 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-tls\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-textfile\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-sys\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/190fdc8d-10ff-453b-8b60-103e9e4823d5-metrics-client-ca\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5zk\" (UniqueName: \"kubernetes.io/projected/8145791c-f93b-41ea-9e4c-089f2775ce52-kube-api-access-fp5zk\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.637105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.636842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edde6206-cd48-41df-b389-bf0ab450d3dc-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96l82\" (UniqueName: \"kubernetes.io/projected/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-api-access-96l82\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-root\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-wtmp\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edde6206-cd48-41df-b389-bf0ab450d3dc-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwlqc\" (UniqueName: \"kubernetes.io/projected/190fdc8d-10ff-453b-8b60-103e9e4823d5-kube-api-access-dwlqc\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8145791c-f93b-41ea-9e4c-089f2775ce52-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-tls\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.737925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-textfile\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.740928 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.738003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-sys\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.738027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/190fdc8d-10ff-453b-8b60-103e9e4823d5-metrics-client-ca\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.738060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5zk\" (UniqueName: \"kubernetes.io/projected/8145791c-f93b-41ea-9e4c-089f2775ce52-kube-api-access-fp5zk\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.738086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:49.738663 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:49.738684 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:49.738734 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls podName:8145791c-f93b-41ea-9e4c-089f2775ce52 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:50.238713226 +0000 UTC m=+288.661461057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-htfzx" (UID: "8145791c-f93b-41ea-9e4c-089f2775ce52") : secret "openshift-state-metrics-tls" not found Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:49.738762 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-tls podName:190fdc8d-10ff-453b-8b60-103e9e4823d5 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:50.238742936 +0000 UTC m=+288.661490769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-tls") pod "node-exporter-f5frl" (UID: "190fdc8d-10ff-453b-8b60-103e9e4823d5") : secret "node-exporter-tls" not found Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.738850 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edde6206-cd48-41df-b389-bf0ab450d3dc-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.739093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-sys\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.739144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-root\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.739227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-wtmp\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.739573 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edde6206-cd48-41df-b389-bf0ab450d3dc-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.739810 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/190fdc8d-10ff-453b-8b60-103e9e4823d5-metrics-client-ca\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.740253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.740807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8145791c-f93b-41ea-9e4c-089f2775ce52-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.741773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.740827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.742625 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.741078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-textfile\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.747849 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.745211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.747849 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.745582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.759772 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.759729 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.763275 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.762853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.766236 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.766214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96l82\" (UniqueName: \"kubernetes.io/projected/edde6206-cd48-41df-b389-bf0ab450d3dc-kube-api-access-96l82\") pod \"kube-state-metrics-785778ff74-fqjfz\" (UID: \"edde6206-cd48-41df-b389-bf0ab450d3dc\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.766650 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.766632 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwlqc\" (UniqueName: \"kubernetes.io/projected/190fdc8d-10ff-453b-8b60-103e9e4823d5-kube-api-access-dwlqc\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:49.769600 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.769566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5zk\" (UniqueName: \"kubernetes.io/projected/8145791c-f93b-41ea-9e4c-089f2775ce52-kube-api-access-fp5zk\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:49.847661 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.847552 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" Jul 12 19:24:49.893655 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:49.893626 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w76w5" event={"ID":"0ae7f2c8-8705-43a4-873f-7586448ec728","Type":"ContainerStarted","Data":"3b7965d9b1fccf17b5701eaebaf91cbdc36ca8b0d81d88b5257a08f7588beeca"} Jul 12 19:24:50.243672 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.243062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-tls\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:50.243672 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.243154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:50.243672 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:50.243269 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 12 19:24:50.243672 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:50.243340 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls podName:8145791c-f93b-41ea-9e4c-089f2775ce52 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:51.243319922 +0000 UTC m=+289.666067752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-htfzx" (UID: "8145791c-f93b-41ea-9e4c-089f2775ce52") : secret "openshift-state-metrics-tls" not found Jul 12 19:24:50.246204 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.246155 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/190fdc8d-10ff-453b-8b60-103e9e4823d5-node-exporter-tls\") pod \"node-exporter-f5frl\" (UID: \"190fdc8d-10ff-453b-8b60-103e9e4823d5\") " pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:50.457888 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.457858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f5frl" Jul 12 19:24:50.473267 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:50.473242 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190fdc8d_10ff_453b_8b60_103e9e4823d5.slice/crio-42d8a4d5b52d4fb526f45b3abc735b337e30bf7d42c3907b9860547bdea35172 WatchSource:0}: Error finding container 42d8a4d5b52d4fb526f45b3abc735b337e30bf7d42c3907b9860547bdea35172: Status 404 returned error can't find the container with id 42d8a4d5b52d4fb526f45b3abc735b337e30bf7d42c3907b9860547bdea35172 Jul 12 19:24:50.475783 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.475760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-fqjfz"] Jul 12 19:24:50.482404 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:50.482371 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedde6206_cd48_41df_b389_bf0ab450d3dc.slice/crio-b849cf1bc12036dbfc482ab46812b27d2bfcd0b6d41edaa655c16a11bcc40db8 WatchSource:0}: Error finding container b849cf1bc12036dbfc482ab46812b27d2bfcd0b6d41edaa655c16a11bcc40db8: Status 404 returned error can't find the container with id b849cf1bc12036dbfc482ab46812b27d2bfcd0b6d41edaa655c16a11bcc40db8 Jul 12 19:24:50.898424 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.898390 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5frl" event={"ID":"190fdc8d-10ff-453b-8b60-103e9e4823d5","Type":"ContainerStarted","Data":"42d8a4d5b52d4fb526f45b3abc735b337e30bf7d42c3907b9860547bdea35172"} Jul 12 19:24:50.899770 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.899742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" event={"ID":"edde6206-cd48-41df-b389-bf0ab450d3dc","Type":"ContainerStarted","Data":"b849cf1bc12036dbfc482ab46812b27d2bfcd0b6d41edaa655c16a11bcc40db8"} Jul 12 19:24:50.901724 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.901684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-26zj5" event={"ID":"38e91421-8a09-4c33-96b7-db7fd8477878","Type":"ContainerStarted","Data":"ca43725395098951197992bd156ab3efe481c2e3a74934515b9819d33e385cc9"} Jul 12 19:24:50.901825 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.901735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-26zj5" event={"ID":"38e91421-8a09-4c33-96b7-db7fd8477878","Type":"ContainerStarted","Data":"7b6c3946c88025e3f9fbd9acb9335a8aea17277a0a929e8dab3616dee0a1d5d1"} Jul 12 19:24:50.901825 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.901782 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-26zj5" Jul 12 19:24:50.920175 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:50.920134 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-26zj5" podStartSLOduration=252.975576553 podStartE2EDuration="4m14.920100418s" totalImagesPullingTime="1.944523865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:50.918513762 +0000 UTC m=+289.341261613" watchObservedRunningTime="2026-07-12 19:24:50.920100418 +0000 UTC m=+289.342848268" Jul 12 19:24:51.251946 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:51.251861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:51.254552 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:51.254528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8145791c-f93b-41ea-9e4c-089f2775ce52-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-htfzx\" (UID: \"8145791c-f93b-41ea-9e4c-089f2775ce52\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:51.323571 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:51.323545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" Jul 12 19:24:51.912602 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:51.912217 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx"] Jul 12 19:24:51.914908 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:51.914881 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8145791c_f93b_41ea_9e4c_089f2775ce52.slice/crio-2e6db7d4eb75ad5f46b871162ec4b553d4b137d96e5cd34d34a71cfdd91fe27e WatchSource:0}: Error finding container 2e6db7d4eb75ad5f46b871162ec4b553d4b137d96e5cd34d34a71cfdd91fe27e: Status 404 returned error can't find the container with id 2e6db7d4eb75ad5f46b871162ec4b553d4b137d96e5cd34d34a71cfdd91fe27e Jul 12 19:24:52.910321 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:52.910284 2565 generic.go:356] "Generic (PLEG): container finished" podID="190fdc8d-10ff-453b-8b60-103e9e4823d5" containerID="787c6dc641133d40dc42ab96e0ea0186be2e5571aa97e8b718d2712e789d6078" exitCode=0 Jul 12 19:24:52.910443 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:52.910376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5frl" event={"ID":"190fdc8d-10ff-453b-8b60-103e9e4823d5","Type":"ContainerDied","Data":"787c6dc641133d40dc42ab96e0ea0186be2e5571aa97e8b718d2712e789d6078"} Jul 12 19:24:52.912673 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:52.912610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w76w5" event={"ID":"0ae7f2c8-8705-43a4-873f-7586448ec728","Type":"ContainerStarted","Data":"5afc01ba6d2d7a2d91a8910a9bdc706ef4bd1053a4ada9d199fd2785ce67680a"} Jul 12 19:24:52.914720 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:52.914692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" event={"ID":"8145791c-f93b-41ea-9e4c-089f2775ce52","Type":"ContainerStarted","Data":"032423d7cfb9f7df2108140cfaeb276c6711448fef592094abbc25a883fca83f"} Jul 12 19:24:52.914804 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:52.914753 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" event={"ID":"8145791c-f93b-41ea-9e4c-089f2775ce52","Type":"ContainerStarted","Data":"bb4a00b642172b518986eff364e37fe06099e54d67a97717ce66468262fcf447"} Jul 12 19:24:52.914804 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:52.914781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" event={"ID":"8145791c-f93b-41ea-9e4c-089f2775ce52","Type":"ContainerStarted","Data":"2e6db7d4eb75ad5f46b871162ec4b553d4b137d96e5cd34d34a71cfdd91fe27e"} Jul 12 19:24:52.945566 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:52.945518 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w76w5" podStartSLOduration=33.758546207 podStartE2EDuration="40.945500845s" totalImagesPullingTime="7.186954638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:24:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:52.94474558 +0000 UTC m=+291.367493429" watchObservedRunningTime="2026-07-12 19:24:52.945500845 +0000 UTC m=+291.368248697" Jul 12 19:24:53.920369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.920331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" event={"ID":"edde6206-cd48-41df-b389-bf0ab450d3dc","Type":"ContainerStarted","Data":"c4c6f32f09d29f7db1c8ea6cf0ffad7c6122c5212efa262ef9212a411072df12"} Jul 12 19:24:53.920811 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.920406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" event={"ID":"edde6206-cd48-41df-b389-bf0ab450d3dc","Type":"ContainerStarted","Data":"9f475b7a0440a9a75e297c53e813abcecc4b689dc4cfc7d57f35963778ad45d3"} Jul 12 19:24:53.920811 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.920435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" event={"ID":"edde6206-cd48-41df-b389-bf0ab450d3dc","Type":"ContainerStarted","Data":"38c26acae2bc88f2985fb78f6875205572c2a6737677b2270763e50b99a20041"} Jul 12 19:24:53.922424 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.922398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" event={"ID":"8145791c-f93b-41ea-9e4c-089f2775ce52","Type":"ContainerStarted","Data":"6fbb13b89c4097f5475b0353fb15b9aa2c669ac983915fead2e79e637f9493e0"} Jul 12 19:24:53.924386 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.924364 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5frl" event={"ID":"190fdc8d-10ff-453b-8b60-103e9e4823d5","Type":"ContainerStarted","Data":"9ae6ae47e50df3383a5e3d292917f034f7e2e8c8e2eaa1ccb6f88d4f80bbef30"} Jul 12 19:24:53.924482 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.924412 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5frl" event={"ID":"190fdc8d-10ff-453b-8b60-103e9e4823d5","Type":"ContainerStarted","Data":"caac4f17b7ad1509899888e0ea24b161e6f297aac4ab33251f6bfcc1034a3a4b"} Jul 12 19:24:53.938064 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.938003 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-fqjfz" podStartSLOduration=2.518200856 podStartE2EDuration="4.937986117s" totalImagesPullingTime="2.419785261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:53.936745842 +0000 UTC m=+292.359493691" watchObservedRunningTime="2026-07-12 19:24:53.937986117 +0000 UTC m=+292.360733967" Jul 12 19:24:53.956070 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.956015 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f5frl" podStartSLOduration=2.937362608 podStartE2EDuration="4.955997562s" totalImagesPullingTime="1.938156079s" totalInitContainerRuntime="80.478875ms" isStatefulPod=true podCreationTimestamp="2026-07-12 19:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:53.954248309 +0000 UTC m=+292.376996159" watchObservedRunningTime="2026-07-12 19:24:53.955997562 +0000 UTC m=+292.378745390" Jul 12 19:24:53.970195 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:53.970148 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-htfzx" podStartSLOduration=3.794709254 podStartE2EDuration="4.970132104s" totalImagesPullingTime="1.17542285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:53.969222363 +0000 UTC m=+292.391970216" watchObservedRunningTime="2026-07-12 19:24:53.970132104 +0000 UTC m=+292.392879953" Jul 12 19:24:54.253108 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.253025 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk"] Jul 12 19:24:54.256703 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.256678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:54.259234 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.259205 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 12 19:24:54.259354 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.259257 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7blgg\"" Jul 12 19:24:54.264805 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.264781 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk"] Jul 12 19:24:54.272786 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.272757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxrw\" (UniqueName: \"kubernetes.io/projected/af2c6748-2039-4e58-8e30-5ebe213c10e9-kube-api-access-6hxrw\") pod \"monitoring-plugin-5f68df7f58-q8nvk\" (UID: \"af2c6748-2039-4e58-8e30-5ebe213c10e9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:54.272920 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.272805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af2c6748-2039-4e58-8e30-5ebe213c10e9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q8nvk\" (UID: \"af2c6748-2039-4e58-8e30-5ebe213c10e9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:54.374070 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.374026 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxrw\" (UniqueName: \"kubernetes.io/projected/af2c6748-2039-4e58-8e30-5ebe213c10e9-kube-api-access-6hxrw\") pod \"monitoring-plugin-5f68df7f58-q8nvk\" (UID: \"af2c6748-2039-4e58-8e30-5ebe213c10e9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:54.374242 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.374079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af2c6748-2039-4e58-8e30-5ebe213c10e9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q8nvk\" (UID: \"af2c6748-2039-4e58-8e30-5ebe213c10e9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:54.374242 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:54.374205 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 12 19:24:54.374339 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:54.374269 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2c6748-2039-4e58-8e30-5ebe213c10e9-monitoring-plugin-cert podName:af2c6748-2039-4e58-8e30-5ebe213c10e9 nodeName:}" failed. No retries permitted until 2026-07-12 19:24:54.874252459 +0000 UTC m=+293.297000287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/af2c6748-2039-4e58-8e30-5ebe213c10e9-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-q8nvk" (UID: "af2c6748-2039-4e58-8e30-5ebe213c10e9") : secret "monitoring-plugin-cert" not found Jul 12 19:24:54.382989 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.382928 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxrw\" (UniqueName: \"kubernetes.io/projected/af2c6748-2039-4e58-8e30-5ebe213c10e9-kube-api-access-6hxrw\") pod \"monitoring-plugin-5f68df7f58-q8nvk\" (UID: \"af2c6748-2039-4e58-8e30-5ebe213c10e9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:54.827423 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.827397 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:24:54.877429 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.877385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af2c6748-2039-4e58-8e30-5ebe213c10e9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q8nvk\" (UID: \"af2c6748-2039-4e58-8e30-5ebe213c10e9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:54.879876 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:54.879851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af2c6748-2039-4e58-8e30-5ebe213c10e9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-q8nvk\" (UID: \"af2c6748-2039-4e58-8e30-5ebe213c10e9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:55.051488 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:24:55.051429 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mn2m5" podUID="31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6" Jul 12 19:24:55.167833 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:55.167788 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:55.299523 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:55.299498 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk"] Jul 12 19:24:55.301935 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:24:55.301900 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2c6748_2039_4e58_8e30_5ebe213c10e9.slice/crio-610112c3f2b76d54466d67df5952c75df4c04709cb927a156233c3e0a67c4367 WatchSource:0}: Error finding container 610112c3f2b76d54466d67df5952c75df4c04709cb927a156233c3e0a67c4367: Status 404 returned error can't find the container with id 610112c3f2b76d54466d67df5952c75df4c04709cb927a156233c3e0a67c4367 Jul 12 19:24:55.934573 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:55.934524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" event={"ID":"af2c6748-2039-4e58-8e30-5ebe213c10e9","Type":"ContainerStarted","Data":"610112c3f2b76d54466d67df5952c75df4c04709cb927a156233c3e0a67c4367"} Jul 12 19:24:56.940180 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:56.940138 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" event={"ID":"af2c6748-2039-4e58-8e30-5ebe213c10e9","Type":"ContainerStarted","Data":"da87efdc6bcba082fd1e021deafc8a9c7ff00aa855d95e068756210042aa9ecc"} Jul 12 19:24:56.940637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:56.940205 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:56.945293 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:56.945268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" Jul 12 19:24:56.954655 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:56.954604 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-q8nvk" podStartSLOduration=1.669255754 podStartE2EDuration="2.954588131s" totalImagesPullingTime="1.285332377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:24:56.954123711 +0000 UTC m=+295.376871564" watchObservedRunningTime="2026-07-12 19:24:56.954588131 +0000 UTC m=+295.377335980" Jul 12 19:24:59.843471 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:24:59.843426 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" podUID="73bc10cd-271b-4d4e-95e4-b7768a383d4c" containerName="registry" containerID="cri-o://f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57" gracePeriod=30 Jul 12 19:25:00.094039 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.093944 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:25:00.117416 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117383 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-kube-api-access-7ljtz\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.117609 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117430 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-certificates\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.117609 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117469 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.117609 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117500 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-image-registry-private-configuration\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.117609 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117525 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-trusted-ca\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.117609 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117551 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73bc10cd-271b-4d4e-95e4-b7768a383d4c-ca-trust-extracted\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.117609 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117592 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-bound-sa-token\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.117940 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.117625 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-installation-pull-secrets\") pod \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\" (UID: \"73bc10cd-271b-4d4e-95e4-b7768a383d4c\") " Jul 12 19:25:00.118391 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.118355 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:00.118932 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.118904 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:25:00.120990 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.120926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:00.121082 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.121033 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:00.121166 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.121141 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-kube-api-access-7ljtz" (OuterVolumeSpecName: "kube-api-access-7ljtz") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "kube-api-access-7ljtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:00.121219 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.121154 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:25:00.121458 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.121431 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:25:00.127885 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.127852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73bc10cd-271b-4d4e-95e4-b7768a383d4c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "73bc10cd-271b-4d4e-95e4-b7768a383d4c" (UID: "73bc10cd-271b-4d4e-95e4-b7768a383d4c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:25:00.219069 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219034 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljtz\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-kube-api-access-7ljtz\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.219069 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219065 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-certificates\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.219069 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219075 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-registry-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.219266 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219085 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-image-registry-private-configuration\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.219266 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219095 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73bc10cd-271b-4d4e-95e4-b7768a383d4c-trusted-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.219266 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219103 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73bc10cd-271b-4d4e-95e4-b7768a383d4c-ca-trust-extracted\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.219266 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219112 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73bc10cd-271b-4d4e-95e4-b7768a383d4c-bound-sa-token\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.219266 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.219120 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73bc10cd-271b-4d4e-95e4-b7768a383d4c-installation-pull-secrets\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:25:00.907620 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.907586 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-26zj5" Jul 12 19:25:00.956902 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.956864 2565 generic.go:356] "Generic (PLEG): container finished" podID="73bc10cd-271b-4d4e-95e4-b7768a383d4c" containerID="f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57" exitCode=0 Jul 12 19:25:00.957119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.956915 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" Jul 12 19:25:00.957119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.956923 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" event={"ID":"73bc10cd-271b-4d4e-95e4-b7768a383d4c","Type":"ContainerDied","Data":"f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57"} Jul 12 19:25:00.957119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.956992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c879658d-8ckcn" event={"ID":"73bc10cd-271b-4d4e-95e4-b7768a383d4c","Type":"ContainerDied","Data":"69d8e87c11451d02e792ca27231d7b1148ea87721022cd098bf14b21a89c83fb"} Jul 12 19:25:00.957119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.957022 2565 scope.go:117] "RemoveContainer" containerID="f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57" Jul 12 19:25:00.967223 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.967202 2565 scope.go:117] "RemoveContainer" containerID="f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57" Jul 12 19:25:00.967600 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:25:00.967580 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57\": container with ID starting with f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57 not found: ID does not exist" containerID="f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57" Jul 12 19:25:00.967662 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.967609 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57"} err="failed to get container status \"f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57\": rpc error: code = NotFound desc = could not find container \"f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57\": container with ID starting with f80d9a6df0f926701d39fb9c9844651f5b8a62e6b1815edd10313f6c98ce8b57 not found: ID does not exist" Jul 12 19:25:00.980522 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.980479 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66c879658d-8ckcn"] Jul 12 19:25:00.983801 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:00.983772 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66c879658d-8ckcn"] Jul 12 19:25:01.891487 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:01.891465 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:25:01.891793 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:01.891775 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:25:01.895591 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:01.895563 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:25:01.895717 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:01.895698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:25:01.902731 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:01.902708 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 12 19:25:02.054478 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:02.054329 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73bc10cd-271b-4d4e-95e4-b7768a383d4c" path="/var/lib/kubelet/pods/73bc10cd-271b-4d4e-95e4-b7768a383d4c/volumes" Jul 12 19:25:04.050300 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.050262 2565 scope.go:117] "RemoveContainer" containerID="95b2b6505beab01d8161af8f0bd160dde29bd288c023611dca50f66dcaa0e2d6" Jul 12 19:25:04.671282 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.671249 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-skr8s"] Jul 12 19:25:04.671531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.671519 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73bc10cd-271b-4d4e-95e4-b7768a383d4c" containerName="registry" Jul 12 19:25:04.671578 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.671532 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73bc10cd-271b-4d4e-95e4-b7768a383d4c" containerName="registry" Jul 12 19:25:04.671613 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.671591 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73bc10cd-271b-4d4e-95e4-b7768a383d4c" containerName="registry" Jul 12 19:25:04.674732 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.674693 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-skr8s" Jul 12 19:25:04.677480 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.677456 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 12 19:25:04.677634 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.677456 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 12 19:25:04.677634 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.677456 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-44xpj\"" Jul 12 19:25:04.684074 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.684043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-skr8s"] Jul 12 19:25:04.751537 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.751491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm799\" (UniqueName: \"kubernetes.io/projected/1b37c485-8d55-4a5f-98d7-282690473b4a-kube-api-access-dm799\") pod \"downloads-68c9cbb497-skr8s\" (UID: \"1b37c485-8d55-4a5f-98d7-282690473b4a\") " pod="openshift-console/downloads-68c9cbb497-skr8s" Jul 12 19:25:04.852336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.852291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm799\" (UniqueName: \"kubernetes.io/projected/1b37c485-8d55-4a5f-98d7-282690473b4a-kube-api-access-dm799\") pod \"downloads-68c9cbb497-skr8s\" (UID: \"1b37c485-8d55-4a5f-98d7-282690473b4a\") " pod="openshift-console/downloads-68c9cbb497-skr8s" Jul 12 19:25:04.860245 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.860211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm799\" (UniqueName: \"kubernetes.io/projected/1b37c485-8d55-4a5f-98d7-282690473b4a-kube-api-access-dm799\") pod \"downloads-68c9cbb497-skr8s\" (UID: \"1b37c485-8d55-4a5f-98d7-282690473b4a\") " pod="openshift-console/downloads-68c9cbb497-skr8s" Jul 12 19:25:04.973166 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.973081 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:25:04.973308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.973182 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-r85fs" event={"ID":"f476e372-7762-4098-a7e0-1ccca0e34c14","Type":"ContainerStarted","Data":"c525ee0fc371c46504068e3b4cc26df6917d1e977b9897c27db4642b5a520521"} Jul 12 19:25:04.973486 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.973459 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:25:04.978331 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.978303 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-r85fs" Jul 12 19:25:04.985540 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.985517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-skr8s" Jul 12 19:25:04.988931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:04.988888 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-r85fs" podStartSLOduration=56.25501077 podStartE2EDuration="57.988875059s" totalImagesPullingTime="1.733864289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:25:04.988576963 +0000 UTC m=+303.411324816" watchObservedRunningTime="2026-07-12 19:25:04.988875059 +0000 UTC m=+303.411622968" Jul 12 19:25:05.138449 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:05.138416 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-skr8s"] Jul 12 19:25:05.141283 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:25:05.141248 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b37c485_8d55_4a5f_98d7_282690473b4a.slice/crio-2b6fa2433d6d1ac49e06dcb13796f6c564492796258bda3878c4d77fdd9af8f6 WatchSource:0}: Error finding container 2b6fa2433d6d1ac49e06dcb13796f6c564492796258bda3878c4d77fdd9af8f6: Status 404 returned error can't find the container with id 2b6fa2433d6d1ac49e06dcb13796f6c564492796258bda3878c4d77fdd9af8f6 Jul 12 19:25:05.143333 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:05.143314 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:25:05.978668 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:05.978615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-skr8s" event={"ID":"1b37c485-8d55-4a5f-98d7-282690473b4a","Type":"ContainerStarted","Data":"2b6fa2433d6d1ac49e06dcb13796f6c564492796258bda3878c4d77fdd9af8f6"} Jul 12 19:25:06.050125 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:06.050086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:25:15.189593 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.189533 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59867f586c-2l28w"] Jul 12 19:25:15.194227 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.194196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.196741 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.196703 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 12 19:25:15.196897 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.196785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 12 19:25:15.198003 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.197720 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 12 19:25:15.198003 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.197747 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-74cxb\"" Jul 12 19:25:15.198003 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.197801 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 12 19:25:15.198003 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.197855 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 12 19:25:15.203743 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.203716 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59867f586c-2l28w"] Jul 12 19:25:15.249064 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.249030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-service-ca\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.249237 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.249085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-oauth-serving-cert\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.249237 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.249129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl52h\" (UniqueName: \"kubernetes.io/projected/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-kube-api-access-hl52h\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.249334 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.249247 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-oauth-config\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.249334 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.249280 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-serving-cert\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.249334 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.249304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-config\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.350592 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.350543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-serving-cert\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.350592 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.350600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-config\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.350864 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.350626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-service-ca\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.350864 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.350658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-oauth-serving-cert\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.350983 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.350864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl52h\" (UniqueName: \"kubernetes.io/projected/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-kube-api-access-hl52h\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.351051 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.350985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-oauth-config\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.351665 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.351627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-config\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.351782 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.351762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-service-ca\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.351864 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.351844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-oauth-serving-cert\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.354159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.354134 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-oauth-config\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.354159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.354152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-serving-cert\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.358792 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.358761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl52h\" (UniqueName: \"kubernetes.io/projected/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-kube-api-access-hl52h\") pod \"console-59867f586c-2l28w\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:15.507660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:15.507560 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:25:22.584414 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:22.584379 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59867f586c-2l28w"] Jul 12 19:25:23.049156 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:23.049116 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59867f586c-2l28w" event={"ID":"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b","Type":"ContainerStarted","Data":"ce8cb6127a9e619b8e5fe809eb8ef8dd788ddcefc8e612ead6dd1e7ce8582d50"} Jul 12 19:25:23.050506 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:23.050477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-skr8s" event={"ID":"1b37c485-8d55-4a5f-98d7-282690473b4a","Type":"ContainerStarted","Data":"fecee06cfc2c5275d91400245a06341119b7545c8a52ca7ea6d79ea079819310"} Jul 12 19:25:23.050719 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:23.050702 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-skr8s" Jul 12 19:25:23.052247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:23.052224 2565 patch_prober.go:28] interesting pod/downloads-68c9cbb497-skr8s container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.19:8080/\": dial tcp 10.134.0.19:8080: connect: connection refused" start-of-body= Jul 12 19:25:23.052346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:23.052270 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-skr8s" podUID="1b37c485-8d55-4a5f-98d7-282690473b4a" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.19:8080/\": dial tcp 10.134.0.19:8080: connect: connection refused" Jul 12 19:25:23.069090 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:23.068990 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-skr8s" podStartSLOduration=1.402179818 podStartE2EDuration="19.068952579s" totalImagesPullingTime="17.666772761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:25:23.0670897 +0000 UTC m=+321.489837549" watchObservedRunningTime="2026-07-12 19:25:23.068952579 +0000 UTC m=+321.491700429" Jul 12 19:25:23.930825 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:23.930707 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fqjfz_edde6206-cd48-41df-b389-bf0ab450d3dc/kube-state-metrics/0.log" Jul 12 19:25:24.067844 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.067806 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-skr8s" Jul 12 19:25:24.129877 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.129750 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fqjfz_edde6206-cd48-41df-b389-bf0ab450d3dc/kube-rbac-proxy-main/0.log" Jul 12 19:25:24.330721 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.330619 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fqjfz_edde6206-cd48-41df-b389-bf0ab450d3dc/kube-rbac-proxy-self/0.log" Jul 12 19:25:24.580526 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.580487 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74dd7b67d7-sw4c8"] Jul 12 19:25:24.615372 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.615333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dd7b67d7-sw4c8"] Jul 12 19:25:24.615563 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.615470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.624820 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.624791 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 12 19:25:24.729408 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.729376 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-q8nvk_af2c6748-2039-4e58-8e30-5ebe213c10e9/monitoring-plugin/0.log" Jul 12 19:25:24.741272 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.741227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-oauth-config\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.741531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.741282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxk5\" (UniqueName: \"kubernetes.io/projected/685d2f09-86e3-41f3-a5fb-6d41671a73c7-kube-api-access-kwxk5\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.741531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.741321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-serving-cert\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.741531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.741344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-service-ca\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.741531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.741371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-trusted-ca-bundle\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.741531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.741418 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-config\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.741531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.741460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-oauth-serving-cert\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.842771 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.842735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-oauth-serving-cert\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843002 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.842812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-oauth-config\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843002 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.842840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxk5\" (UniqueName: \"kubernetes.io/projected/685d2f09-86e3-41f3-a5fb-6d41671a73c7-kube-api-access-kwxk5\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843002 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.842874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-serving-cert\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843002 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.842894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-service-ca\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843002 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.842915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-trusted-ca-bundle\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843002 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.842974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-config\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843706 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.843607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-oauth-serving-cert\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843805 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.843656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-config\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.843978 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.843907 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-service-ca\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.844346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.844323 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-trusted-ca-bundle\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.845772 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.845748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-oauth-config\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.846921 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.846884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-serving-cert\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.851083 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.851058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxk5\" (UniqueName: \"kubernetes.io/projected/685d2f09-86e3-41f3-a5fb-6d41671a73c7-kube-api-access-kwxk5\") pod \"console-74dd7b67d7-sw4c8\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:24.930395 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:24.930258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:25.530222 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:25.530187 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5frl_190fdc8d-10ff-453b-8b60-103e9e4823d5/init-textfile/0.log" Jul 12 19:25:25.730728 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:25.730690 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5frl_190fdc8d-10ff-453b-8b60-103e9e4823d5/node-exporter/0.log" Jul 12 19:25:25.930113 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:25.930014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5frl_190fdc8d-10ff-453b-8b60-103e9e4823d5/kube-rbac-proxy/0.log" Jul 12 19:25:26.465688 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:26.465603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dd7b67d7-sw4c8"] Jul 12 19:25:26.468754 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:25:26.468722 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685d2f09_86e3_41f3_a5fb_6d41671a73c7.slice/crio-24c170e49940f524f752224fc8e84aa9e489cd95260ec7d75d23c6d3ad50705c WatchSource:0}: Error finding container 24c170e49940f524f752224fc8e84aa9e489cd95260ec7d75d23c6d3ad50705c: Status 404 returned error can't find the container with id 24c170e49940f524f752224fc8e84aa9e489cd95260ec7d75d23c6d3ad50705c Jul 12 19:25:26.729134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:26.729100 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-htfzx_8145791c-f93b-41ea-9e4c-089f2775ce52/kube-rbac-proxy-main/0.log" Jul 12 19:25:26.932242 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:26.932214 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-htfzx_8145791c-f93b-41ea-9e4c-089f2775ce52/kube-rbac-proxy-self/0.log" Jul 12 19:25:27.071796 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:27.071692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59867f586c-2l28w" event={"ID":"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b","Type":"ContainerStarted","Data":"42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38"} Jul 12 19:25:27.073052 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:27.073021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dd7b67d7-sw4c8" event={"ID":"685d2f09-86e3-41f3-a5fb-6d41671a73c7","Type":"ContainerStarted","Data":"24c170e49940f524f752224fc8e84aa9e489cd95260ec7d75d23c6d3ad50705c"} Jul 12 19:25:27.089146 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:27.089088 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59867f586c-2l28w" podStartSLOduration=7.941900438 podStartE2EDuration="12.089066554s" totalImagesPullingTime="4.147166116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:25:27.087668782 +0000 UTC m=+325.510416623" watchObservedRunningTime="2026-07-12 19:25:27.089066554 +0000 UTC m=+325.511814405" Jul 12 19:25:27.129631 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:27.129586 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-htfzx_8145791c-f93b-41ea-9e4c-089f2775ce52/openshift-state-metrics/0.log" Jul 12 19:25:28.079814 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:28.079764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dd7b67d7-sw4c8" event={"ID":"685d2f09-86e3-41f3-a5fb-6d41671a73c7","Type":"ContainerStarted","Data":"52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458"} Jul 12 19:25:28.098429 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:28.098372 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74dd7b67d7-sw4c8" podStartSLOduration=3.424527717 podStartE2EDuration="4.098354112s" totalImagesPullingTime="673.826395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:25:28.096826882 +0000 UTC m=+326.519574733" watchObservedRunningTime="2026-07-12 19:25:28.098354112 +0000 UTC m=+326.521101992" Jul 12 19:25:28.731184 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:28.731142 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xvhwm_b22f9560-c421-4a96-8581-57789c2943fe/prometheus-operator/0.log" Jul 12 19:25:28.929135 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:28.929098 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xvhwm_b22f9560-c421-4a96-8581-57789c2943fe/kube-rbac-proxy/0.log" Jul 12 19:25:30.730374 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:30.730334 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:25:30.932169 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:30.932131 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/3.log" Jul 12 19:25:31.135667 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:31.135625 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59867f586c-2l28w_ab3d7f0b-c0f6-4678-bac7-e5b588fde81b/console/0.log" Jul 12 19:25:31.331728 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:31.331678 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-skr8s_1b37c485-8d55-4a5f-98d7-282690473b4a/download-server/0.log" Jul 12 19:25:34.931546 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:34.930772 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:34.931546 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:34.931494 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:34.936535 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:34.936507 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:35.111863 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:35.111795 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:25:35.155012 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:35.154955 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59867f586c-2l28w"] Jul 12 19:25:35.508737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:25:35.508699 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:26:00.190498 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.190393 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59867f586c-2l28w" podUID="ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" containerName="console" containerID="cri-o://42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38" gracePeriod=15 Jul 12 19:26:00.463553 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.463464 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:26:00.552544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.552503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-oauth-config\") pod \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " Jul 12 19:26:00.552737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.552559 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-oauth-serving-cert\") pod \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " Jul 12 19:26:00.552737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.552585 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-config\") pod \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " Jul 12 19:26:00.552737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.552606 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-serving-cert\") pod \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " Jul 12 19:26:00.552737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.552635 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-service-ca\") pod \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " Jul 12 19:26:00.552737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.552671 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl52h\" (UniqueName: \"kubernetes.io/projected/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-kube-api-access-hl52h\") pod \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\" (UID: \"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b\") " Jul 12 19:26:00.553093 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.553069 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" (UID: "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:26:00.553161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.553066 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-config" (OuterVolumeSpecName: "console-config") pod "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" (UID: "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:26:00.553161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.553111 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-service-ca" (OuterVolumeSpecName: "service-ca") pod "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" (UID: "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:26:00.555078 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.555048 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" (UID: "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:26:00.555134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.555057 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-kube-api-access-hl52h" (OuterVolumeSpecName: "kube-api-access-hl52h") pod "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" (UID: "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b"). InnerVolumeSpecName "kube-api-access-hl52h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:26:00.555221 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.555195 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" (UID: "ab3d7f0b-c0f6-4678-bac7-e5b588fde81b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:26:00.654031 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.653955 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:00.654031 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.654024 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl52h\" (UniqueName: \"kubernetes.io/projected/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-kube-api-access-hl52h\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:00.654031 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.654039 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-oauth-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:00.654271 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.654053 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-oauth-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:00.654271 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.654071 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:00.654271 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:00.654083 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b-console-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:01.206932 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.206893 2565 generic.go:356] "Generic (PLEG): container finished" podID="ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" containerID="42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38" exitCode=0 Jul 12 19:26:01.207359 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.206978 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59867f586c-2l28w" Jul 12 19:26:01.207359 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.206989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59867f586c-2l28w" event={"ID":"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b","Type":"ContainerDied","Data":"42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38"} Jul 12 19:26:01.207359 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.207048 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59867f586c-2l28w" event={"ID":"ab3d7f0b-c0f6-4678-bac7-e5b588fde81b","Type":"ContainerDied","Data":"ce8cb6127a9e619b8e5fe809eb8ef8dd788ddcefc8e612ead6dd1e7ce8582d50"} Jul 12 19:26:01.207359 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.207068 2565 scope.go:117] "RemoveContainer" containerID="42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38" Jul 12 19:26:01.216026 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.216003 2565 scope.go:117] "RemoveContainer" containerID="42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38" Jul 12 19:26:01.216335 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:26:01.216314 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38\": container with ID starting with 42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38 not found: ID does not exist" containerID="42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38" Jul 12 19:26:01.216406 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.216345 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38"} err="failed to get container status \"42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38\": rpc error: code = NotFound desc = could not find container \"42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38\": container with ID starting with 42ce250d12c064c43463342e58221a5a5961d755f14239e712f0734748b31b38 not found: ID does not exist" Jul 12 19:26:01.227090 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.227061 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59867f586c-2l28w"] Jul 12 19:26:01.229820 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:01.229798 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59867f586c-2l28w"] Jul 12 19:26:02.054298 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:02.054257 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" path="/var/lib/kubelet/pods/ab3d7f0b-c0f6-4678-bac7-e5b588fde81b/volumes" Jul 12 19:26:13.705842 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.705802 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d74949d6-hc9r2"] Jul 12 19:26:13.706343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.706290 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" containerName="console" Jul 12 19:26:13.706343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.706307 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" containerName="console" Jul 12 19:26:13.706414 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.706386 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3d7f0b-c0f6-4678-bac7-e5b588fde81b" containerName="console" Jul 12 19:26:13.710292 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.710261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.713031 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.713000 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 12 19:26:13.713194 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.713126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 12 19:26:13.713194 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.713165 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cfb22\"" Jul 12 19:26:13.713312 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.713226 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 12 19:26:13.713312 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.713276 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 12 19:26:13.713407 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.713340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 12 19:26:13.720223 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.720196 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 12 19:26:13.721359 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.721330 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d74949d6-hc9r2"] Jul 12 19:26:13.761931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.761886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.761931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.761931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-metrics-client-ca\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.762168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.761950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-telemeter-client-tls\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.762168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.762083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-federate-client-tls\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.762168 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.762123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-secret-telemeter-client\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.762260 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.762204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtsgf\" (UniqueName: \"kubernetes.io/projected/5e907c8c-112c-4dca-8e59-00a3f8c687a1-kube-api-access-qtsgf\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.762260 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.762233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-serving-certs-ca-bundle\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.762260 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.762256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.862949 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.862901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtsgf\" (UniqueName: \"kubernetes.io/projected/5e907c8c-112c-4dca-8e59-00a3f8c687a1-kube-api-access-qtsgf\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.862973 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-serving-certs-ca-bundle\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.863041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.863104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863246 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.863140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-metrics-client-ca\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863246 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.863171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-telemeter-client-tls\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863246 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.863209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-federate-client-tls\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863377 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.863247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-secret-telemeter-client\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.863859 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.863834 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-serving-certs-ca-bundle\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.864259 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.864061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-metrics-client-ca\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.864259 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.864165 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e907c8c-112c-4dca-8e59-00a3f8c687a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.865900 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.865879 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-secret-telemeter-client\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.866035 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.866012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.866317 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.866295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-federate-client-tls\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.866398 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.866381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e907c8c-112c-4dca-8e59-00a3f8c687a1-telemeter-client-tls\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:13.871191 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:13.871148 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtsgf\" (UniqueName: \"kubernetes.io/projected/5e907c8c-112c-4dca-8e59-00a3f8c687a1-kube-api-access-qtsgf\") pod \"telemeter-client-5d74949d6-hc9r2\" (UID: \"5e907c8c-112c-4dca-8e59-00a3f8c687a1\") " pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:14.023910 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:14.023819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" Jul 12 19:26:14.188922 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:14.188888 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d74949d6-hc9r2"] Jul 12 19:26:14.192289 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:26:14.192235 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e907c8c_112c_4dca_8e59_00a3f8c687a1.slice/crio-3f4a99afc74cea45bfa75664c1ead9324d92134f907ec6cb78af6ee6f74f3bab WatchSource:0}: Error finding container 3f4a99afc74cea45bfa75664c1ead9324d92134f907ec6cb78af6ee6f74f3bab: Status 404 returned error can't find the container with id 3f4a99afc74cea45bfa75664c1ead9324d92134f907ec6cb78af6ee6f74f3bab Jul 12 19:26:14.257224 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:14.256950 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" event={"ID":"5e907c8c-112c-4dca-8e59-00a3f8c687a1","Type":"ContainerStarted","Data":"3f4a99afc74cea45bfa75664c1ead9324d92134f907ec6cb78af6ee6f74f3bab"} Jul 12 19:26:15.980710 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:15.980667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:26:15.983408 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:15.983380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6-metrics-certs\") pod \"network-metrics-daemon-mn2m5\" (UID: \"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6\") " pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:26:16.254265 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:16.254176 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brd5c\"" Jul 12 19:26:16.262232 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:16.262205 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mn2m5" Jul 12 19:26:16.268737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:16.268697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" event={"ID":"5e907c8c-112c-4dca-8e59-00a3f8c687a1","Type":"ContainerStarted","Data":"47a475dfb8273b85bc2729ffda0c67dff4c602216ecf9789cf046bb945c78c79"} Jul 12 19:26:16.400728 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:16.400694 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mn2m5"] Jul 12 19:26:16.404105 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:26:16.404073 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e30d3f_8aaa_4b90_8c55_e3de6cd7f3a6.slice/crio-f94aacc51281ed382054e04f9208cdcd8dbe1560133b44ede69e7a1df6d70db9 WatchSource:0}: Error finding container f94aacc51281ed382054e04f9208cdcd8dbe1560133b44ede69e7a1df6d70db9: Status 404 returned error can't find the container with id f94aacc51281ed382054e04f9208cdcd8dbe1560133b44ede69e7a1df6d70db9 Jul 12 19:26:17.273998 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:17.273931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mn2m5" event={"ID":"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6","Type":"ContainerStarted","Data":"f94aacc51281ed382054e04f9208cdcd8dbe1560133b44ede69e7a1df6d70db9"} Jul 12 19:26:18.280153 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:18.280115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" event={"ID":"5e907c8c-112c-4dca-8e59-00a3f8c687a1","Type":"ContainerStarted","Data":"990228164a1c60cad38ff8f2555172f0200ca77ae7ce75d317d9d14272317bb3"} Jul 12 19:26:18.280584 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:18.280188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" event={"ID":"5e907c8c-112c-4dca-8e59-00a3f8c687a1","Type":"ContainerStarted","Data":"aa39f827d9ec172cbf4f7ab25f62c59c0f55ab5f6aa654fae0f932e8c2b5a704"} Jul 12 19:26:18.281811 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:18.281791 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mn2m5" event={"ID":"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6","Type":"ContainerStarted","Data":"2fc7be1a61379fc4bedc032682c1d8cd295b460a0070f4cc01275253500d73f1"} Jul 12 19:26:18.281894 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:18.281827 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mn2m5" event={"ID":"31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6","Type":"ContainerStarted","Data":"05f4ad83785aa70d6fe77d7bb10a28c142d80af5e23b631e3bd79aa577ca6a43"} Jul 12 19:26:18.302145 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:18.301920 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d74949d6-hc9r2" podStartSLOduration=2.071317407 podStartE2EDuration="5.301902747s" totalImagesPullingTime="3.23058534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:26:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:26:18.300252046 +0000 UTC m=+376.722999896" watchObservedRunningTime="2026-07-12 19:26:18.301902747 +0000 UTC m=+376.724650597" Jul 12 19:26:18.314713 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:18.314648 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mn2m5" podStartSLOduration=375.235513941 podStartE2EDuration="6m16.314625496s" totalImagesPullingTime="1.079111555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:26:18.314116599 +0000 UTC m=+376.736864449" watchObservedRunningTime="2026-07-12 19:26:18.314625496 +0000 UTC m=+376.737373347" Jul 12 19:26:19.066216 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.066181 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-559b5fd578-thhjs"] Jul 12 19:26:19.069747 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.069717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.083925 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.083897 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559b5fd578-thhjs"] Jul 12 19:26:19.209764 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.209727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-oauth-serving-cert\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.209764 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.209767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-oauth-config\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.210145 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.209790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-serving-cert\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.210145 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.209825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8s2\" (UniqueName: \"kubernetes.io/projected/ea902359-0f1b-4f31-b4b7-aa583b951f33-kube-api-access-9m8s2\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.210145 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.209851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-config\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.210145 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.209888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-trusted-ca-bundle\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.210145 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.210022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-service-ca\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.311263 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.311224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-trusted-ca-bundle\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.311263 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.311263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-service-ca\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.311784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.311288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-oauth-serving-cert\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.311784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.311333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-oauth-config\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.311784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.311372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-serving-cert\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.311784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.311399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8s2\" (UniqueName: \"kubernetes.io/projected/ea902359-0f1b-4f31-b4b7-aa583b951f33-kube-api-access-9m8s2\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.311784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.311438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-config\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.312277 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.312154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-oauth-serving-cert\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.312345 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.312281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-config\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.312345 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.312214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-service-ca\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.312531 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.312506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-trusted-ca-bundle\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.313914 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.313893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-oauth-config\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.314114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.314094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-serving-cert\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.319988 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.319919 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8s2\" (UniqueName: \"kubernetes.io/projected/ea902359-0f1b-4f31-b4b7-aa583b951f33-kube-api-access-9m8s2\") pod \"console-559b5fd578-thhjs\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.381181 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.381142 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:19.518261 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:19.518224 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559b5fd578-thhjs"] Jul 12 19:26:19.522529 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:26:19.522501 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea902359_0f1b_4f31_b4b7_aa583b951f33.slice/crio-21600f2d249f314cf7bbb0446b4404b0d1c42a106112a72f383b9474a6d2ae2e WatchSource:0}: Error finding container 21600f2d249f314cf7bbb0446b4404b0d1c42a106112a72f383b9474a6d2ae2e: Status 404 returned error can't find the container with id 21600f2d249f314cf7bbb0446b4404b0d1c42a106112a72f383b9474a6d2ae2e Jul 12 19:26:20.290306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:20.290273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b5fd578-thhjs" event={"ID":"ea902359-0f1b-4f31-b4b7-aa583b951f33","Type":"ContainerStarted","Data":"b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a"} Jul 12 19:26:20.290519 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:20.290321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b5fd578-thhjs" event={"ID":"ea902359-0f1b-4f31-b4b7-aa583b951f33","Type":"ContainerStarted","Data":"21600f2d249f314cf7bbb0446b4404b0d1c42a106112a72f383b9474a6d2ae2e"} Jul 12 19:26:20.308217 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:20.308167 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-559b5fd578-thhjs" podStartSLOduration=1.308150454 podStartE2EDuration="1.308150454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:26:20.307144866 +0000 UTC m=+378.729892716" watchObservedRunningTime="2026-07-12 19:26:20.308150454 +0000 UTC m=+378.730898304" Jul 12 19:26:29.381395 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:29.381351 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:29.381787 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:29.381411 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:29.386262 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:29.386234 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:30.337110 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:30.337075 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:26:30.386075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:30.386041 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dd7b67d7-sw4c8"] Jul 12 19:26:55.409747 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.409697 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74dd7b67d7-sw4c8" podUID="685d2f09-86e3-41f3-a5fb-6d41671a73c7" containerName="console" containerID="cri-o://52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458" gracePeriod=15 Jul 12 19:26:55.657164 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.657132 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:26:55.714119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.713945 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-serving-cert\") pod \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " Jul 12 19:26:55.714119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714027 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwxk5\" (UniqueName: \"kubernetes.io/projected/685d2f09-86e3-41f3-a5fb-6d41671a73c7-kube-api-access-kwxk5\") pod \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " Jul 12 19:26:55.714119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714086 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-config\") pod \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " Jul 12 19:26:55.714119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714114 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-oauth-config\") pod \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " Jul 12 19:26:55.714462 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714149 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-trusted-ca-bundle\") pod \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " Jul 12 19:26:55.714462 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714193 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-oauth-serving-cert\") pod \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " Jul 12 19:26:55.714462 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714230 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-service-ca\") pod \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\" (UID: \"685d2f09-86e3-41f3-a5fb-6d41671a73c7\") " Jul 12 19:26:55.714683 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714652 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-config" (OuterVolumeSpecName: "console-config") pod "685d2f09-86e3-41f3-a5fb-6d41671a73c7" (UID: "685d2f09-86e3-41f3-a5fb-6d41671a73c7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:26:55.714803 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714661 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "685d2f09-86e3-41f3-a5fb-6d41671a73c7" (UID: "685d2f09-86e3-41f3-a5fb-6d41671a73c7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:26:55.714803 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714758 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "685d2f09-86e3-41f3-a5fb-6d41671a73c7" (UID: "685d2f09-86e3-41f3-a5fb-6d41671a73c7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:26:55.714948 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.714924 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-service-ca" (OuterVolumeSpecName: "service-ca") pod "685d2f09-86e3-41f3-a5fb-6d41671a73c7" (UID: "685d2f09-86e3-41f3-a5fb-6d41671a73c7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:26:55.716437 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.716406 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "685d2f09-86e3-41f3-a5fb-6d41671a73c7" (UID: "685d2f09-86e3-41f3-a5fb-6d41671a73c7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:26:55.716437 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.716426 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "685d2f09-86e3-41f3-a5fb-6d41671a73c7" (UID: "685d2f09-86e3-41f3-a5fb-6d41671a73c7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:26:55.716578 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.716490 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685d2f09-86e3-41f3-a5fb-6d41671a73c7-kube-api-access-kwxk5" (OuterVolumeSpecName: "kube-api-access-kwxk5") pod "685d2f09-86e3-41f3-a5fb-6d41671a73c7" (UID: "685d2f09-86e3-41f3-a5fb-6d41671a73c7"). InnerVolumeSpecName "kube-api-access-kwxk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:26:55.815094 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.815057 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:55.815094 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.815090 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-oauth-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:55.815094 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.815104 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-trusted-ca-bundle\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:55.815355 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.815116 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-oauth-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:55.815355 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.815128 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/685d2f09-86e3-41f3-a5fb-6d41671a73c7-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:55.815355 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.815139 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d2f09-86e3-41f3-a5fb-6d41671a73c7-console-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:55.815355 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:55.815151 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxk5\" (UniqueName: \"kubernetes.io/projected/685d2f09-86e3-41f3-a5fb-6d41671a73c7-kube-api-access-kwxk5\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:26:56.438450 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.438409 2565 generic.go:356] "Generic (PLEG): container finished" podID="685d2f09-86e3-41f3-a5fb-6d41671a73c7" containerID="52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458" exitCode=0 Jul 12 19:26:56.438911 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.438464 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dd7b67d7-sw4c8" Jul 12 19:26:56.438911 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.438497 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dd7b67d7-sw4c8" event={"ID":"685d2f09-86e3-41f3-a5fb-6d41671a73c7","Type":"ContainerDied","Data":"52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458"} Jul 12 19:26:56.438911 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.438552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dd7b67d7-sw4c8" event={"ID":"685d2f09-86e3-41f3-a5fb-6d41671a73c7","Type":"ContainerDied","Data":"24c170e49940f524f752224fc8e84aa9e489cd95260ec7d75d23c6d3ad50705c"} Jul 12 19:26:56.438911 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.438574 2565 scope.go:117] "RemoveContainer" containerID="52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458" Jul 12 19:26:56.448753 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.448734 2565 scope.go:117] "RemoveContainer" containerID="52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458" Jul 12 19:26:56.449102 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:26:56.449080 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458\": container with ID starting with 52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458 not found: ID does not exist" containerID="52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458" Jul 12 19:26:56.449165 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.449110 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458"} err="failed to get container status \"52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458\": rpc error: code = NotFound desc = could not find container \"52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458\": container with ID starting with 52665d1d0e6093454e910a772a953aefe1b92e9870a0826c5a45b10f293b9458 not found: ID does not exist" Jul 12 19:26:56.456198 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.456168 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74dd7b67d7-sw4c8"] Jul 12 19:26:56.463742 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:56.463690 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74dd7b67d7-sw4c8"] Jul 12 19:26:58.054825 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:26:58.054789 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="685d2f09-86e3-41f3-a5fb-6d41671a73c7" path="/var/lib/kubelet/pods/685d2f09-86e3-41f3-a5fb-6d41671a73c7/volumes" Jul 12 19:27:41.854479 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.854440 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-694c577648-pwm9z"] Jul 12 19:27:41.855217 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.855183 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685d2f09-86e3-41f3-a5fb-6d41671a73c7" containerName="console" Jul 12 19:27:41.855336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.855290 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="685d2f09-86e3-41f3-a5fb-6d41671a73c7" containerName="console" Jul 12 19:27:41.855528 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.855510 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="685d2f09-86e3-41f3-a5fb-6d41671a73c7" containerName="console" Jul 12 19:27:41.859848 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.859818 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:41.863320 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.863296 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-694c577648-pwm9z"] Jul 12 19:27:41.981108 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.981075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-serving-cert\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:41.981281 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.981122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-service-ca\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:41.981281 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.981209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tc9l\" (UniqueName: \"kubernetes.io/projected/f214dfdc-97c9-44a0-8606-789d4e5cd48e-kube-api-access-9tc9l\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:41.981281 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.981251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-oauth-config\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:41.981281 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.981274 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-trusted-ca-bundle\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:41.981413 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.981294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-config\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:41.981413 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:41.981311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-oauth-serving-cert\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.082463 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.082424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tc9l\" (UniqueName: \"kubernetes.io/projected/f214dfdc-97c9-44a0-8606-789d4e5cd48e-kube-api-access-9tc9l\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.082463 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.082462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-oauth-config\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.082723 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.082483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-trusted-ca-bundle\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.082723 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.082502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-config\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.082723 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.082518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-oauth-serving-cert\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.082723 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.082554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-serving-cert\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.082723 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.082596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-service-ca\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.083411 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.083341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-config\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.083411 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.083397 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-service-ca\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.083411 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.083428 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-oauth-serving-cert\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.083411 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.083491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-trusted-ca-bundle\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.085262 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.085236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-oauth-config\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.085356 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.085285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-serving-cert\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.090722 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.090692 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tc9l\" (UniqueName: \"kubernetes.io/projected/f214dfdc-97c9-44a0-8606-789d4e5cd48e-kube-api-access-9tc9l\") pod \"console-694c577648-pwm9z\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.171978 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.171936 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:42.307935 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.307797 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-694c577648-pwm9z"] Jul 12 19:27:42.310682 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:27:42.310645 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf214dfdc_97c9_44a0_8606_789d4e5cd48e.slice/crio-5da7d395f7fc29931f2e53d3e1c1dcdb6e7e2fa7897558e1c6ff616928a3cd59 WatchSource:0}: Error finding container 5da7d395f7fc29931f2e53d3e1c1dcdb6e7e2fa7897558e1c6ff616928a3cd59: Status 404 returned error can't find the container with id 5da7d395f7fc29931f2e53d3e1c1dcdb6e7e2fa7897558e1c6ff616928a3cd59 Jul 12 19:27:42.615852 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.615823 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694c577648-pwm9z" event={"ID":"f214dfdc-97c9-44a0-8606-789d4e5cd48e","Type":"ContainerStarted","Data":"1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e"} Jul 12 19:27:42.616050 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.615871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694c577648-pwm9z" event={"ID":"f214dfdc-97c9-44a0-8606-789d4e5cd48e","Type":"ContainerStarted","Data":"5da7d395f7fc29931f2e53d3e1c1dcdb6e7e2fa7897558e1c6ff616928a3cd59"} Jul 12 19:27:42.633389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:42.633333 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-694c577648-pwm9z" podStartSLOduration=1.633314567 podStartE2EDuration="1.633314567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:42.631336334 +0000 UTC m=+461.054084187" watchObservedRunningTime="2026-07-12 19:27:42.633314567 +0000 UTC m=+461.056062471" Jul 12 19:27:52.172413 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:52.172375 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:52.173045 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:52.172540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:52.177358 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:52.177337 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:52.657151 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:52.657124 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:27:52.704186 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:27:52.704152 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559b5fd578-thhjs"] Jul 12 19:28:12.750153 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.750108 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75"] Jul 12 19:28:12.753685 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.753663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.756282 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.756260 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sg447\"" Jul 12 19:28:12.756430 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.756267 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 12 19:28:12.757265 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.757247 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 12 19:28:12.761771 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.761739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75"] Jul 12 19:28:12.827866 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.827826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.828090 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.827889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.828090 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.827993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnxn\" (UniqueName: \"kubernetes.io/projected/f13555d6-40d9-4e79-9f63-83090072c17b-kube-api-access-gdnxn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.929022 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.928952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnxn\" (UniqueName: \"kubernetes.io/projected/f13555d6-40d9-4e79-9f63-83090072c17b-kube-api-access-gdnxn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.929227 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.929059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.929227 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.929101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.929447 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.929430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.929518 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.929496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:12.937635 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:12.937607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnxn\" (UniqueName: \"kubernetes.io/projected/f13555d6-40d9-4e79-9f63-83090072c17b-kube-api-access-gdnxn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:13.064040 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:13.063920 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:13.198212 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:13.198184 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75"] Jul 12 19:28:13.200884 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:28:13.200856 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13555d6_40d9_4e79_9f63_83090072c17b.slice/crio-437e3d5323ee7148644ebb5a515d3df9afff73ebea8f2284867a4dd0cc598836 WatchSource:0}: Error finding container 437e3d5323ee7148644ebb5a515d3df9afff73ebea8f2284867a4dd0cc598836: Status 404 returned error can't find the container with id 437e3d5323ee7148644ebb5a515d3df9afff73ebea8f2284867a4dd0cc598836 Jul 12 19:28:13.736000 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:13.735948 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" event={"ID":"f13555d6-40d9-4e79-9f63-83090072c17b","Type":"ContainerStarted","Data":"437e3d5323ee7148644ebb5a515d3df9afff73ebea8f2284867a4dd0cc598836"} Jul 12 19:28:17.737101 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:17.737058 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-559b5fd578-thhjs" podUID="ea902359-0f1b-4f31-b4b7-aa583b951f33" containerName="console" containerID="cri-o://b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a" gracePeriod=15 Jul 12 19:28:19.206729 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.206692 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:28:19.286917 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.286877 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-trusted-ca-bundle\") pod \"ea902359-0f1b-4f31-b4b7-aa583b951f33\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " Jul 12 19:28:19.287134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.286936 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-oauth-config\") pod \"ea902359-0f1b-4f31-b4b7-aa583b951f33\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " Jul 12 19:28:19.287134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.286955 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-config\") pod \"ea902359-0f1b-4f31-b4b7-aa583b951f33\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " Jul 12 19:28:19.287134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.286998 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-oauth-serving-cert\") pod \"ea902359-0f1b-4f31-b4b7-aa583b951f33\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " Jul 12 19:28:19.287134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.287020 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-service-ca\") pod \"ea902359-0f1b-4f31-b4b7-aa583b951f33\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " Jul 12 19:28:19.287134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.287055 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m8s2\" (UniqueName: \"kubernetes.io/projected/ea902359-0f1b-4f31-b4b7-aa583b951f33-kube-api-access-9m8s2\") pod \"ea902359-0f1b-4f31-b4b7-aa583b951f33\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " Jul 12 19:28:19.287134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.287115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-serving-cert\") pod \"ea902359-0f1b-4f31-b4b7-aa583b951f33\" (UID: \"ea902359-0f1b-4f31-b4b7-aa583b951f33\") " Jul 12 19:28:19.287468 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.287363 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ea902359-0f1b-4f31-b4b7-aa583b951f33" (UID: "ea902359-0f1b-4f31-b4b7-aa583b951f33"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:28:19.287530 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.287499 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ea902359-0f1b-4f31-b4b7-aa583b951f33" (UID: "ea902359-0f1b-4f31-b4b7-aa583b951f33"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:28:19.287586 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.287560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-service-ca" (OuterVolumeSpecName: "service-ca") pod "ea902359-0f1b-4f31-b4b7-aa583b951f33" (UID: "ea902359-0f1b-4f31-b4b7-aa583b951f33"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:28:19.287640 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.287612 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-config" (OuterVolumeSpecName: "console-config") pod "ea902359-0f1b-4f31-b4b7-aa583b951f33" (UID: "ea902359-0f1b-4f31-b4b7-aa583b951f33"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:28:19.289397 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.289373 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea902359-0f1b-4f31-b4b7-aa583b951f33-kube-api-access-9m8s2" (OuterVolumeSpecName: "kube-api-access-9m8s2") pod "ea902359-0f1b-4f31-b4b7-aa583b951f33" (UID: "ea902359-0f1b-4f31-b4b7-aa583b951f33"). InnerVolumeSpecName "kube-api-access-9m8s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:28:19.289481 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.289409 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ea902359-0f1b-4f31-b4b7-aa583b951f33" (UID: "ea902359-0f1b-4f31-b4b7-aa583b951f33"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:28:19.289481 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.289431 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ea902359-0f1b-4f31-b4b7-aa583b951f33" (UID: "ea902359-0f1b-4f31-b4b7-aa583b951f33"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:28:19.388319 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.388279 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-trusted-ca-bundle\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:19.388319 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.388312 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-oauth-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:19.388319 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.388321 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:19.388319 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.388330 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-oauth-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:19.388571 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.388338 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea902359-0f1b-4f31-b4b7-aa583b951f33-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:19.388571 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.388345 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m8s2\" (UniqueName: \"kubernetes.io/projected/ea902359-0f1b-4f31-b4b7-aa583b951f33-kube-api-access-9m8s2\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:19.388571 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.388353 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea902359-0f1b-4f31-b4b7-aa583b951f33-console-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:19.765446 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.765410 2565 generic.go:356] "Generic (PLEG): container finished" podID="f13555d6-40d9-4e79-9f63-83090072c17b" containerID="d88c47a911d46d19c297251255bc5fc7137b84ab2cb8c354c246536650f57864" exitCode=0 Jul 12 19:28:19.765616 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.765496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" event={"ID":"f13555d6-40d9-4e79-9f63-83090072c17b","Type":"ContainerDied","Data":"d88c47a911d46d19c297251255bc5fc7137b84ab2cb8c354c246536650f57864"} Jul 12 19:28:19.766925 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.766901 2565 generic.go:356] "Generic (PLEG): container finished" podID="ea902359-0f1b-4f31-b4b7-aa583b951f33" containerID="b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a" exitCode=0 Jul 12 19:28:19.767021 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.766927 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559b5fd578-thhjs" Jul 12 19:28:19.767021 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.766995 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b5fd578-thhjs" event={"ID":"ea902359-0f1b-4f31-b4b7-aa583b951f33","Type":"ContainerDied","Data":"b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a"} Jul 12 19:28:19.767163 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.767036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559b5fd578-thhjs" event={"ID":"ea902359-0f1b-4f31-b4b7-aa583b951f33","Type":"ContainerDied","Data":"21600f2d249f314cf7bbb0446b4404b0d1c42a106112a72f383b9474a6d2ae2e"} Jul 12 19:28:19.767163 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.767062 2565 scope.go:117] "RemoveContainer" containerID="b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a" Jul 12 19:28:19.775681 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.775663 2565 scope.go:117] "RemoveContainer" containerID="b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a" Jul 12 19:28:19.775949 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:28:19.775926 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a\": container with ID starting with b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a not found: ID does not exist" containerID="b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a" Jul 12 19:28:19.776023 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.775978 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a"} err="failed to get container status \"b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a\": rpc error: code = NotFound desc = could not find container \"b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a\": container with ID starting with b33974d63466522b7c9d5294c7724f15e9a49aaa77f3ec00beb4fb42520b430a not found: ID does not exist" Jul 12 19:28:19.792816 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.792730 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559b5fd578-thhjs"] Jul 12 19:28:19.794322 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:19.794299 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-559b5fd578-thhjs"] Jul 12 19:28:20.055146 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:20.055043 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea902359-0f1b-4f31-b4b7-aa583b951f33" path="/var/lib/kubelet/pods/ea902359-0f1b-4f31-b4b7-aa583b951f33/volumes" Jul 12 19:28:21.778407 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:21.778371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" event={"ID":"f13555d6-40d9-4e79-9f63-83090072c17b","Type":"ContainerStarted","Data":"9f520f8d98b36d5a80dbe9742f5ba297a0fb9d9af4986e5f59f424ded1793cdf"} Jul 12 19:28:22.784296 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:22.784263 2565 generic.go:356] "Generic (PLEG): container finished" podID="f13555d6-40d9-4e79-9f63-83090072c17b" containerID="9f520f8d98b36d5a80dbe9742f5ba297a0fb9d9af4986e5f59f424ded1793cdf" exitCode=0 Jul 12 19:28:22.784663 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:22.784346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" event={"ID":"f13555d6-40d9-4e79-9f63-83090072c17b","Type":"ContainerDied","Data":"9f520f8d98b36d5a80dbe9742f5ba297a0fb9d9af4986e5f59f424ded1793cdf"} Jul 12 19:28:29.818046 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:29.818006 2565 generic.go:356] "Generic (PLEG): container finished" podID="f13555d6-40d9-4e79-9f63-83090072c17b" containerID="2fbf168bbfc13acc125a0c82f67c2f8cfa0675751d38778028be66be35f60f22" exitCode=0 Jul 12 19:28:29.818456 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:29.818097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" event={"ID":"f13555d6-40d9-4e79-9f63-83090072c17b","Type":"ContainerDied","Data":"2fbf168bbfc13acc125a0c82f67c2f8cfa0675751d38778028be66be35f60f22"} Jul 12 19:28:30.952105 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:30.952072 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:31.089674 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.089585 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-util\") pod \"f13555d6-40d9-4e79-9f63-83090072c17b\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " Jul 12 19:28:31.089674 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.089632 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdnxn\" (UniqueName: \"kubernetes.io/projected/f13555d6-40d9-4e79-9f63-83090072c17b-kube-api-access-gdnxn\") pod \"f13555d6-40d9-4e79-9f63-83090072c17b\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " Jul 12 19:28:31.089674 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.089674 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-bundle\") pod \"f13555d6-40d9-4e79-9f63-83090072c17b\" (UID: \"f13555d6-40d9-4e79-9f63-83090072c17b\") " Jul 12 19:28:31.090416 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.090392 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-bundle" (OuterVolumeSpecName: "bundle") pod "f13555d6-40d9-4e79-9f63-83090072c17b" (UID: "f13555d6-40d9-4e79-9f63-83090072c17b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:28:31.092059 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.092033 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13555d6-40d9-4e79-9f63-83090072c17b-kube-api-access-gdnxn" (OuterVolumeSpecName: "kube-api-access-gdnxn") pod "f13555d6-40d9-4e79-9f63-83090072c17b" (UID: "f13555d6-40d9-4e79-9f63-83090072c17b"). InnerVolumeSpecName "kube-api-access-gdnxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:28:31.093615 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.093589 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-util" (OuterVolumeSpecName: "util") pod "f13555d6-40d9-4e79-9f63-83090072c17b" (UID: "f13555d6-40d9-4e79-9f63-83090072c17b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:28:31.191114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.191065 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-bundle\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:31.191114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.191106 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f13555d6-40d9-4e79-9f63-83090072c17b-util\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:31.191114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.191118 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdnxn\" (UniqueName: \"kubernetes.io/projected/f13555d6-40d9-4e79-9f63-83090072c17b-kube-api-access-gdnxn\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:28:31.827924 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.827884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" event={"ID":"f13555d6-40d9-4e79-9f63-83090072c17b","Type":"ContainerDied","Data":"437e3d5323ee7148644ebb5a515d3df9afff73ebea8f2284867a4dd0cc598836"} Jul 12 19:28:31.828129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.827936 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="437e3d5323ee7148644ebb5a515d3df9afff73ebea8f2284867a4dd0cc598836" Jul 12 19:28:31.828129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:31.827894 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m5s75" Jul 12 19:28:35.678223 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678187 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf"] Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678485 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea902359-0f1b-4f31-b4b7-aa583b951f33" containerName="console" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678495 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea902359-0f1b-4f31-b4b7-aa583b951f33" containerName="console" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678512 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13555d6-40d9-4e79-9f63-83090072c17b" containerName="pull" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678518 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13555d6-40d9-4e79-9f63-83090072c17b" containerName="pull" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678527 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13555d6-40d9-4e79-9f63-83090072c17b" containerName="util" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678533 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13555d6-40d9-4e79-9f63-83090072c17b" containerName="util" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678539 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13555d6-40d9-4e79-9f63-83090072c17b" containerName="extract" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678547 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13555d6-40d9-4e79-9f63-83090072c17b" containerName="extract" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678590 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13555d6-40d9-4e79-9f63-83090072c17b" containerName="extract" Jul 12 19:28:35.678643 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.678599 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea902359-0f1b-4f31-b4b7-aa583b951f33" containerName="console" Jul 12 19:28:35.719385 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.719339 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf"] Jul 12 19:28:35.719564 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.719473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:35.722394 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.722370 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hrvck\"" Jul 12 19:28:35.722548 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.722414 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:28:35.722548 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.722414 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 12 19:28:35.825227 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.825186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40eb1bd-a44b-4f72-a67a-b491751ecd51-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tnhmf\" (UID: \"a40eb1bd-a44b-4f72-a67a-b491751ecd51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:35.825420 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.825250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4j6r\" (UniqueName: \"kubernetes.io/projected/a40eb1bd-a44b-4f72-a67a-b491751ecd51-kube-api-access-j4j6r\") pod \"cert-manager-operator-controller-manager-55c444bf86-tnhmf\" (UID: \"a40eb1bd-a44b-4f72-a67a-b491751ecd51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:35.925683 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.925636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4j6r\" (UniqueName: \"kubernetes.io/projected/a40eb1bd-a44b-4f72-a67a-b491751ecd51-kube-api-access-j4j6r\") pod \"cert-manager-operator-controller-manager-55c444bf86-tnhmf\" (UID: \"a40eb1bd-a44b-4f72-a67a-b491751ecd51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:35.925889 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.925720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40eb1bd-a44b-4f72-a67a-b491751ecd51-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tnhmf\" (UID: \"a40eb1bd-a44b-4f72-a67a-b491751ecd51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:35.926107 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.926089 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a40eb1bd-a44b-4f72-a67a-b491751ecd51-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tnhmf\" (UID: \"a40eb1bd-a44b-4f72-a67a-b491751ecd51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:35.937136 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:35.937063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4j6r\" (UniqueName: \"kubernetes.io/projected/a40eb1bd-a44b-4f72-a67a-b491751ecd51-kube-api-access-j4j6r\") pod \"cert-manager-operator-controller-manager-55c444bf86-tnhmf\" (UID: \"a40eb1bd-a44b-4f72-a67a-b491751ecd51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:36.029308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:36.029273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" Jul 12 19:28:36.193609 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:36.193531 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf"] Jul 12 19:28:36.197947 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:28:36.197920 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40eb1bd_a44b_4f72_a67a_b491751ecd51.slice/crio-9843d0014a79b508967b38a50761ad526d7ed2eed99c04f9e07e645cca0863a8 WatchSource:0}: Error finding container 9843d0014a79b508967b38a50761ad526d7ed2eed99c04f9e07e645cca0863a8: Status 404 returned error can't find the container with id 9843d0014a79b508967b38a50761ad526d7ed2eed99c04f9e07e645cca0863a8 Jul 12 19:28:36.850097 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:36.850065 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" event={"ID":"a40eb1bd-a44b-4f72-a67a-b491751ecd51","Type":"ContainerStarted","Data":"9843d0014a79b508967b38a50761ad526d7ed2eed99c04f9e07e645cca0863a8"} Jul 12 19:28:38.860664 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:38.860621 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" event={"ID":"a40eb1bd-a44b-4f72-a67a-b491751ecd51","Type":"ContainerStarted","Data":"b4ff0b1eae3e68459302cb726dc5224ac6b2326dbc39f63e7c317d9aef646999"} Jul 12 19:28:38.884016 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:38.883939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tnhmf" podStartSLOduration=1.844840169 podStartE2EDuration="3.88392153s" totalImagesPullingTime="2.039081361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:28:38.882557865 +0000 UTC m=+517.305305717" watchObservedRunningTime="2026-07-12 19:28:38.88392153 +0000 UTC m=+517.306669380" Jul 12 19:28:39.035834 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.035798 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lwdww"] Jul 12 19:28:39.057346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.057309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lwdww"] Jul 12 19:28:39.057509 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.057430 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.060179 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.060149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4vwqw\"" Jul 12 19:28:39.060322 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.060180 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 12 19:28:39.060322 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.060226 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 12 19:28:39.151705 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.151665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qq9\" (UniqueName: \"kubernetes.io/projected/cf802594-aba8-4147-a0b9-8a589bee70e4-kube-api-access-88qq9\") pod \"cert-manager-webhook-6486ffd997-lwdww\" (UID: \"cf802594-aba8-4147-a0b9-8a589bee70e4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.151891 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.151721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf802594-aba8-4147-a0b9-8a589bee70e4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lwdww\" (UID: \"cf802594-aba8-4147-a0b9-8a589bee70e4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.252183 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.252151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qq9\" (UniqueName: \"kubernetes.io/projected/cf802594-aba8-4147-a0b9-8a589bee70e4-kube-api-access-88qq9\") pod \"cert-manager-webhook-6486ffd997-lwdww\" (UID: \"cf802594-aba8-4147-a0b9-8a589bee70e4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.252363 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.252200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf802594-aba8-4147-a0b9-8a589bee70e4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lwdww\" (UID: \"cf802594-aba8-4147-a0b9-8a589bee70e4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.261838 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.261803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf802594-aba8-4147-a0b9-8a589bee70e4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lwdww\" (UID: \"cf802594-aba8-4147-a0b9-8a589bee70e4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.262004 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.261910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qq9\" (UniqueName: \"kubernetes.io/projected/cf802594-aba8-4147-a0b9-8a589bee70e4-kube-api-access-88qq9\") pod \"cert-manager-webhook-6486ffd997-lwdww\" (UID: \"cf802594-aba8-4147-a0b9-8a589bee70e4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.376501 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.376467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:39.546746 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.546710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lwdww"] Jul 12 19:28:39.549258 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:28:39.549223 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf802594_aba8_4147_a0b9_8a589bee70e4.slice/crio-027d9e09a8c8adc7a62f95340199aa43cca6c84cb263d424dde11490a5a7a077 WatchSource:0}: Error finding container 027d9e09a8c8adc7a62f95340199aa43cca6c84cb263d424dde11490a5a7a077: Status 404 returned error can't find the container with id 027d9e09a8c8adc7a62f95340199aa43cca6c84cb263d424dde11490a5a7a077 Jul 12 19:28:39.866527 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:39.866492 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" event={"ID":"cf802594-aba8-4147-a0b9-8a589bee70e4","Type":"ContainerStarted","Data":"027d9e09a8c8adc7a62f95340199aa43cca6c84cb263d424dde11490a5a7a077"} Jul 12 19:28:43.888028 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:43.887988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" event={"ID":"cf802594-aba8-4147-a0b9-8a589bee70e4","Type":"ContainerStarted","Data":"ec1ec6afc40a22586df54d5bccb0141f75a711ded05578d4dd5877f29f85e065"} Jul 12 19:28:43.888440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:43.888158 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:28:43.902590 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:43.902535 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" podStartSLOduration=1.503842948 podStartE2EDuration="4.902518619s" totalImagesPullingTime="3.398675671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:28:43.901982066 +0000 UTC m=+522.324729911" watchObservedRunningTime="2026-07-12 19:28:43.902518619 +0000 UTC m=+522.325266469" Jul 12 19:28:47.228651 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.228609 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf"] Jul 12 19:28:47.232171 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.232147 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.234588 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.234565 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9b65w\"" Jul 12 19:28:47.239753 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.239729 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf"] Jul 12 19:28:47.324425 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.324389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67mm\" (UniqueName: \"kubernetes.io/projected/6c326c10-cb90-4233-b3cb-4a996e272714-kube-api-access-l67mm\") pod \"cert-manager-cainjector-559cdd8c87-8xnlf\" (UID: \"6c326c10-cb90-4233-b3cb-4a996e272714\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.324614 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.324469 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c326c10-cb90-4233-b3cb-4a996e272714-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8xnlf\" (UID: \"6c326c10-cb90-4233-b3cb-4a996e272714\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.425913 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.425873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c326c10-cb90-4233-b3cb-4a996e272714-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8xnlf\" (UID: \"6c326c10-cb90-4233-b3cb-4a996e272714\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.426107 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.425925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67mm\" (UniqueName: \"kubernetes.io/projected/6c326c10-cb90-4233-b3cb-4a996e272714-kube-api-access-l67mm\") pod \"cert-manager-cainjector-559cdd8c87-8xnlf\" (UID: \"6c326c10-cb90-4233-b3cb-4a996e272714\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.433738 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.433709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c326c10-cb90-4233-b3cb-4a996e272714-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8xnlf\" (UID: \"6c326c10-cb90-4233-b3cb-4a996e272714\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.433863 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.433786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67mm\" (UniqueName: \"kubernetes.io/projected/6c326c10-cb90-4233-b3cb-4a996e272714-kube-api-access-l67mm\") pod \"cert-manager-cainjector-559cdd8c87-8xnlf\" (UID: \"6c326c10-cb90-4233-b3cb-4a996e272714\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.542370 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.542271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" Jul 12 19:28:47.678338 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.678311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf"] Jul 12 19:28:47.680641 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:28:47.680615 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c326c10_cb90_4233_b3cb_4a996e272714.slice/crio-62b5a7d465518ce2003d20fbb70b4dd328e00ba4ccbbeb13c5694938c177022d WatchSource:0}: Error finding container 62b5a7d465518ce2003d20fbb70b4dd328e00ba4ccbbeb13c5694938c177022d: Status 404 returned error can't find the container with id 62b5a7d465518ce2003d20fbb70b4dd328e00ba4ccbbeb13c5694938c177022d Jul 12 19:28:47.908783 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.908744 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" event={"ID":"6c326c10-cb90-4233-b3cb-4a996e272714","Type":"ContainerStarted","Data":"a649065a24f8a4000de7463da7ce70d3fd675f5333980c026990e44d51407809"} Jul 12 19:28:47.908991 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.908790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" event={"ID":"6c326c10-cb90-4233-b3cb-4a996e272714","Type":"ContainerStarted","Data":"62b5a7d465518ce2003d20fbb70b4dd328e00ba4ccbbeb13c5694938c177022d"} Jul 12 19:28:47.926162 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:47.926078 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8xnlf" podStartSLOduration=0.926053769 podStartE2EDuration="926.053769ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:28:47.924071974 +0000 UTC m=+526.346819843" watchObservedRunningTime="2026-07-12 19:28:47.926053769 +0000 UTC m=+526.348801620" Jul 12 19:28:49.895459 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:28:49.895427 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lwdww" Jul 12 19:29:09.039582 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.039539 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v"] Jul 12 19:29:09.047320 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.047282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.049842 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.049806 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 12 19:29:09.050027 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.049807 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sg447\"" Jul 12 19:29:09.050252 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.050225 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v"] Jul 12 19:29:09.050828 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.050810 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 12 19:29:09.105873 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.105831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.106074 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.105887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qh2v\" (UniqueName: \"kubernetes.io/projected/95e455d8-f038-412a-ad98-71df8e1d9bd8-kube-api-access-7qh2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.106074 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.105909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.207113 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.207071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.207287 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.207124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qh2v\" (UniqueName: \"kubernetes.io/projected/95e455d8-f038-412a-ad98-71df8e1d9bd8-kube-api-access-7qh2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.207287 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.207145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.207477 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.207461 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.207546 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.207528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.215010 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.214975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qh2v\" (UniqueName: \"kubernetes.io/projected/95e455d8-f038-412a-ad98-71df8e1d9bd8-kube-api-access-7qh2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.357404 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.357312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:09.494273 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:09.494241 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v"] Jul 12 19:29:09.497513 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:29:09.497484 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e455d8_f038_412a_ad98_71df8e1d9bd8.slice/crio-086c79c9fcf61e88072bb43eb775723fcba6acbe3e98e1490bbc36b8e289ebc1 WatchSource:0}: Error finding container 086c79c9fcf61e88072bb43eb775723fcba6acbe3e98e1490bbc36b8e289ebc1: Status 404 returned error can't find the container with id 086c79c9fcf61e88072bb43eb775723fcba6acbe3e98e1490bbc36b8e289ebc1 Jul 12 19:29:10.007632 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:10.007540 2565 generic.go:356] "Generic (PLEG): container finished" podID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerID="9fedb114e59dbd1606b8adba8c1cf1b3c6facc20eb9d331c440f354fb87dfc14" exitCode=0 Jul 12 19:29:10.007785 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:10.007623 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" event={"ID":"95e455d8-f038-412a-ad98-71df8e1d9bd8","Type":"ContainerDied","Data":"9fedb114e59dbd1606b8adba8c1cf1b3c6facc20eb9d331c440f354fb87dfc14"} Jul 12 19:29:10.007785 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:10.007677 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" event={"ID":"95e455d8-f038-412a-ad98-71df8e1d9bd8","Type":"ContainerStarted","Data":"086c79c9fcf61e88072bb43eb775723fcba6acbe3e98e1490bbc36b8e289ebc1"} Jul 12 19:29:11.013422 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:11.013388 2565 generic.go:356] "Generic (PLEG): container finished" podID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerID="e375e4dcdb0580bda9fcbcd32eb8596c59caa4565415dc130b71ddead852d1ac" exitCode=0 Jul 12 19:29:11.013839 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:11.013480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" event={"ID":"95e455d8-f038-412a-ad98-71df8e1d9bd8","Type":"ContainerDied","Data":"e375e4dcdb0580bda9fcbcd32eb8596c59caa4565415dc130b71ddead852d1ac"} Jul 12 19:29:12.020618 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:12.020583 2565 generic.go:356] "Generic (PLEG): container finished" podID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerID="14a9df8d8aa60dc90fa821e09dd71d5eed30feebb19691d6af7ed06eb6336c1b" exitCode=0 Jul 12 19:29:12.021022 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:12.020660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" event={"ID":"95e455d8-f038-412a-ad98-71df8e1d9bd8","Type":"ContainerDied","Data":"14a9df8d8aa60dc90fa821e09dd71d5eed30feebb19691d6af7ed06eb6336c1b"} Jul 12 19:29:13.152730 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.152687 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:13.241399 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.241362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qh2v\" (UniqueName: \"kubernetes.io/projected/95e455d8-f038-412a-ad98-71df8e1d9bd8-kube-api-access-7qh2v\") pod \"95e455d8-f038-412a-ad98-71df8e1d9bd8\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " Jul 12 19:29:13.241569 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.241458 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-util\") pod \"95e455d8-f038-412a-ad98-71df8e1d9bd8\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " Jul 12 19:29:13.241640 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.241619 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-bundle\") pod \"95e455d8-f038-412a-ad98-71df8e1d9bd8\" (UID: \"95e455d8-f038-412a-ad98-71df8e1d9bd8\") " Jul 12 19:29:13.242303 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.242265 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-bundle" (OuterVolumeSpecName: "bundle") pod "95e455d8-f038-412a-ad98-71df8e1d9bd8" (UID: "95e455d8-f038-412a-ad98-71df8e1d9bd8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:29:13.243693 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.243666 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e455d8-f038-412a-ad98-71df8e1d9bd8-kube-api-access-7qh2v" (OuterVolumeSpecName: "kube-api-access-7qh2v") pod "95e455d8-f038-412a-ad98-71df8e1d9bd8" (UID: "95e455d8-f038-412a-ad98-71df8e1d9bd8"). InnerVolumeSpecName "kube-api-access-7qh2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:29:13.247193 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.247165 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-util" (OuterVolumeSpecName: "util") pod "95e455d8-f038-412a-ad98-71df8e1d9bd8" (UID: "95e455d8-f038-412a-ad98-71df8e1d9bd8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:29:13.343308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.343218 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-bundle\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:29:13.343308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.343250 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qh2v\" (UniqueName: \"kubernetes.io/projected/95e455d8-f038-412a-ad98-71df8e1d9bd8-kube-api-access-7qh2v\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:29:13.343308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.343261 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95e455d8-f038-412a-ad98-71df8e1d9bd8-util\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:29:13.535837 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.535797 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-89775f59b-qvc57"] Jul 12 19:29:13.536163 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.536149 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerName="extract" Jul 12 19:29:13.536208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.536165 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerName="extract" Jul 12 19:29:13.536208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.536181 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerName="util" Jul 12 19:29:13.536208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.536186 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerName="util" Jul 12 19:29:13.536208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.536193 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerName="pull" Jul 12 19:29:13.536208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.536198 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerName="pull" Jul 12 19:29:13.536389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.536247 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95e455d8-f038-412a-ad98-71df8e1d9bd8" containerName="extract" Jul 12 19:29:13.540644 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.540621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.544068 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.544039 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 12 19:29:13.546338 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.546313 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:29:13.546478 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.546359 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 12 19:29:13.546539 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.546469 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-89775f59b-qvc57"] Jul 12 19:29:13.546770 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.546750 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 12 19:29:13.546876 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.546756 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 12 19:29:13.546876 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.546814 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vs8gx\"" Jul 12 19:29:13.646521 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.646477 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmcmq\" (UniqueName: \"kubernetes.io/projected/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-kube-api-access-lmcmq\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.646685 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.646577 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-metrics-cert\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.646685 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.646610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-manager-config\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.646685 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.646634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-cert\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.747879 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.747843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-metrics-cert\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.747879 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.747884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-manager-config\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.748110 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.747901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-cert\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.748110 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.747935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmcmq\" (UniqueName: \"kubernetes.io/projected/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-kube-api-access-lmcmq\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.748700 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.748670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-manager-config\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.750542 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.750517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-metrics-cert\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.750630 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.750564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-cert\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.755901 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.755875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmcmq\" (UniqueName: \"kubernetes.io/projected/1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c-kube-api-access-lmcmq\") pod \"lws-controller-manager-89775f59b-qvc57\" (UID: \"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c\") " pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.850945 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.850903 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:13.985356 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:13.985331 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-89775f59b-qvc57"] Jul 12 19:29:13.987668 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:29:13.987637 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0f32f0_8c78_4f6b_80dc_8a55c86fd70c.slice/crio-bf0398141f8bd63d9566eccb82f963b574b2ad6453fb4bfcb9c464a9d6c28259 WatchSource:0}: Error finding container bf0398141f8bd63d9566eccb82f963b574b2ad6453fb4bfcb9c464a9d6c28259: Status 404 returned error can't find the container with id bf0398141f8bd63d9566eccb82f963b574b2ad6453fb4bfcb9c464a9d6c28259 Jul 12 19:29:14.033285 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:14.033259 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" Jul 12 19:29:14.033474 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:14.033257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhqm5v" event={"ID":"95e455d8-f038-412a-ad98-71df8e1d9bd8","Type":"ContainerDied","Data":"086c79c9fcf61e88072bb43eb775723fcba6acbe3e98e1490bbc36b8e289ebc1"} Jul 12 19:29:14.033474 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:14.033397 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="086c79c9fcf61e88072bb43eb775723fcba6acbe3e98e1490bbc36b8e289ebc1" Jul 12 19:29:14.034747 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:14.034719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" event={"ID":"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c","Type":"ContainerStarted","Data":"bf0398141f8bd63d9566eccb82f963b574b2ad6453fb4bfcb9c464a9d6c28259"} Jul 12 19:29:17.052005 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:17.051944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" event={"ID":"1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c","Type":"ContainerStarted","Data":"bfc8e8eb3836a6b4a7a60307651bc604b239cf57de0f05a09c813c3890eb2074"} Jul 12 19:29:17.052475 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:17.052021 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:17.069065 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:17.069005 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" podStartSLOduration=1.695486665 podStartE2EDuration="4.068987048s" totalImagesPullingTime="2.373500383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:17.066657677 +0000 UTC m=+555.489405524" watchObservedRunningTime="2026-07-12 19:29:17.068987048 +0000 UTC m=+555.491734895" Jul 12 19:29:24.548973 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.548925 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml"] Jul 12 19:29:24.552952 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.552928 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.555878 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.555842 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 12 19:29:24.556037 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.555944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4cqmv\"" Jul 12 19:29:24.556037 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.555999 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 12 19:29:24.556372 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.556343 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 12 19:29:24.556473 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.556396 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 12 19:29:24.571822 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.571765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml"] Jul 12 19:29:24.650123 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.650082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de2e3825-6753-417f-adc7-db4d85ea6e26-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.650330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.650156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8z9l\" (UniqueName: \"kubernetes.io/projected/de2e3825-6753-417f-adc7-db4d85ea6e26-kube-api-access-s8z9l\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.650330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.650210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2e3825-6753-417f-adc7-db4d85ea6e26-tmp\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.650330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.650260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de2e3825-6753-417f-adc7-db4d85ea6e26-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.650330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.650281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de2e3825-6753-417f-adc7-db4d85ea6e26-manifests\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.751047 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.750993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2e3825-6753-417f-adc7-db4d85ea6e26-tmp\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.751047 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.751050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de2e3825-6753-417f-adc7-db4d85ea6e26-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.751305 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.751080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de2e3825-6753-417f-adc7-db4d85ea6e26-manifests\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.751305 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.751124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de2e3825-6753-417f-adc7-db4d85ea6e26-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.751305 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.751276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8z9l\" (UniqueName: \"kubernetes.io/projected/de2e3825-6753-417f-adc7-db4d85ea6e26-kube-api-access-s8z9l\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.751529 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.751502 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de2e3825-6753-417f-adc7-db4d85ea6e26-manifests\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.753454 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.753424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de2e3825-6753-417f-adc7-db4d85ea6e26-tmp\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.753717 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.753700 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de2e3825-6753-417f-adc7-db4d85ea6e26-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.753763 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.753713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de2e3825-6753-417f-adc7-db4d85ea6e26-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.760498 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.760446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8z9l\" (UniqueName: \"kubernetes.io/projected/de2e3825-6753-417f-adc7-db4d85ea6e26-kube-api-access-s8z9l\") pod \"opendatahub-operator-controller-manager-68dff8c9d6-s5pml\" (UID: \"de2e3825-6753-417f-adc7-db4d85ea6e26\") " pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:24.864138 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:24.864067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:25.024058 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:25.024010 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml"] Jul 12 19:29:25.027999 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:29:25.027937 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2e3825_6753_417f_adc7_db4d85ea6e26.slice/crio-ba6b2e5dd548e7cecd93b28895326749029a79ae11b5471c39130a7952e81306 WatchSource:0}: Error finding container ba6b2e5dd548e7cecd93b28895326749029a79ae11b5471c39130a7952e81306: Status 404 returned error can't find the container with id ba6b2e5dd548e7cecd93b28895326749029a79ae11b5471c39130a7952e81306 Jul 12 19:29:25.090751 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:25.090717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" event={"ID":"de2e3825-6753-417f-adc7-db4d85ea6e26","Type":"ContainerStarted","Data":"ba6b2e5dd548e7cecd93b28895326749029a79ae11b5471c39130a7952e81306"} Jul 12 19:29:28.059764 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:28.059730 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-89775f59b-qvc57" Jul 12 19:29:28.112895 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:28.112860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" event={"ID":"de2e3825-6753-417f-adc7-db4d85ea6e26","Type":"ContainerStarted","Data":"b67d7cbdfb3e245461e2c6c58ad7800fbd81d44b2e9653780e4b41311b8e59fc"} Jul 12 19:29:30.123571 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:30.123533 2565 generic.go:356] "Generic (PLEG): container finished" podID="de2e3825-6753-417f-adc7-db4d85ea6e26" containerID="b67d7cbdfb3e245461e2c6c58ad7800fbd81d44b2e9653780e4b41311b8e59fc" exitCode=0 Jul 12 19:29:30.124004 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:30.123585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" event={"ID":"de2e3825-6753-417f-adc7-db4d85ea6e26","Type":"ContainerDied","Data":"b67d7cbdfb3e245461e2c6c58ad7800fbd81d44b2e9653780e4b41311b8e59fc"} Jul 12 19:29:31.129906 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:31.129873 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" event={"ID":"de2e3825-6753-417f-adc7-db4d85ea6e26","Type":"ContainerStarted","Data":"ce9caad8d98f769768fedf32858759a3297349c17d4942ea6ce1dc84ddaf0e57"} Jul 12 19:29:31.130330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:31.130110 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:31.153043 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:31.152991 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" podStartSLOduration=2.746661665 podStartE2EDuration="7.152953159s" totalImagesPullingTime="3.30784565s" totalInitContainerRuntime="1.098445844s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:31.151906847 +0000 UTC m=+569.574654697" watchObservedRunningTime="2026-07-12 19:29:31.152953159 +0000 UTC m=+569.575701009" Jul 12 19:29:42.137948 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:42.137917 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68dff8c9d6-s5pml" Jul 12 19:29:52.553330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.553284 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q"] Jul 12 19:29:52.571052 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.571014 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q"] Jul 12 19:29:52.571214 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.571119 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.573807 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.573728 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 12 19:29:52.573807 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.573751 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 12 19:29:52.573807 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.573800 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 12 19:29:52.574115 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.573751 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 12 19:29:52.574115 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.573756 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hxpsr\"" Jul 12 19:29:52.604067 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.604018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6znm\" (UniqueName: \"kubernetes.io/projected/c7ad74f3-5542-4f1d-9a22-61a31173181c-kube-api-access-d6znm\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.604067 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.604069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad74f3-5542-4f1d-9a22-61a31173181c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.604336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.604130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.604336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.604182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.604336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.604233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.604336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.604272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.604336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.604294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.705463 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.705416 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.705463 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.705467 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.705727 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.705488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.705727 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.705547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6znm\" (UniqueName: \"kubernetes.io/projected/c7ad74f3-5542-4f1d-9a22-61a31173181c-kube-api-access-d6znm\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.705727 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.705575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad74f3-5542-4f1d-9a22-61a31173181c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.705727 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.705614 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.705727 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.705668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.706289 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.706263 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.708167 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.708143 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.708267 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.708149 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.708371 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.708352 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ad74f3-5542-4f1d-9a22-61a31173181c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.708494 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.708479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.713601 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.713568 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6znm\" (UniqueName: \"kubernetes.io/projected/c7ad74f3-5542-4f1d-9a22-61a31173181c-kube-api-access-d6znm\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.713802 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.713780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ad74f3-5542-4f1d-9a22-61a31173181c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rwd2q\" (UID: \"c7ad74f3-5542-4f1d-9a22-61a31173181c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:52.881640 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:52.881602 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:53.044532 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:53.044467 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q"] Jul 12 19:29:53.233946 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:53.233850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" event={"ID":"c7ad74f3-5542-4f1d-9a22-61a31173181c","Type":"ContainerStarted","Data":"b56ca6c23e0be1abe139c51e63c26ce5ee63d0d1cc9162ae7ebf6a733cdc273f"} Jul 12 19:29:55.434453 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:55.434413 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Jul 12 19:29:55.434829 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:55.434497 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Jul 12 19:29:56.264385 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:56.264344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" event={"ID":"c7ad74f3-5542-4f1d-9a22-61a31173181c","Type":"ContainerStarted","Data":"99a85c09f6a46bcad589c31744d8f2a96c87395dccae40efe0acec8b1ca32d42"} Jul 12 19:29:56.271202 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:56.265002 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:29:56.271202 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:56.267498 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-rwd2q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 12 19:29:56.271202 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:56.267561 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" podUID="c7ad74f3-5542-4f1d-9a22-61a31173181c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:29:56.292858 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:56.292797 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" podStartSLOduration=1.9135050009999999 podStartE2EDuration="4.292779593s" totalImagesPullingTime="2.379274592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:56.29016187 +0000 UTC m=+594.712909701" watchObservedRunningTime="2026-07-12 19:29:56.292779593 +0000 UTC m=+594.715527443" Jul 12 19:29:57.270381 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:29:57.270349 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rwd2q" Jul 12 19:30:01.920682 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:30:01.920654 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:30:01.921212 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:30:01.921194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:30:01.924559 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:30:01.924539 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:30:01.924878 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:30:01.924854 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:31:24.225783 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.225739 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d5d7957d8-5htd4"] Jul 12 19:31:24.229795 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.229766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.239327 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.239288 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d5d7957d8-5htd4"] Jul 12 19:31:24.397419 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.397378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-oauth-config\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.397607 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.397488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-config\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.397607 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.397547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-serving-cert\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.397607 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.397585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-service-ca\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.397762 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.397613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-trusted-ca-bundle\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.397762 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.397728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-oauth-serving-cert\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.397851 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.397770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhpf\" (UniqueName: \"kubernetes.io/projected/604b0ea5-098d-4f09-9e29-662e7c7dc326-kube-api-access-5fhpf\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.498813 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.498708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-oauth-config\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.498813 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.498761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-config\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.498813 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.498787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-serving-cert\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.498813 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.498805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-service-ca\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.499198 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.499007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-trusted-ca-bundle\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.499198 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.499092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-oauth-serving-cert\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.499198 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.499127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhpf\" (UniqueName: \"kubernetes.io/projected/604b0ea5-098d-4f09-9e29-662e7c7dc326-kube-api-access-5fhpf\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.499782 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.499637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-service-ca\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.499782 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.499655 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-config\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.499951 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.499935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-trusted-ca-bundle\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.500075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.500048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604b0ea5-098d-4f09-9e29-662e7c7dc326-oauth-serving-cert\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.501570 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.501548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-oauth-config\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.501678 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.501640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604b0ea5-098d-4f09-9e29-662e7c7dc326-console-serving-cert\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.507119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.507087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhpf\" (UniqueName: \"kubernetes.io/projected/604b0ea5-098d-4f09-9e29-662e7c7dc326-kube-api-access-5fhpf\") pod \"console-6d5d7957d8-5htd4\" (UID: \"604b0ea5-098d-4f09-9e29-662e7c7dc326\") " pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.540784 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.540739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:24.686547 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:24.686422 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d5d7957d8-5htd4"] Jul 12 19:31:24.689544 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:31:24.689511 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604b0ea5_098d_4f09_9e29_662e7c7dc326.slice/crio-243c0ea0d930527429e9354888f4d5766ee5601f7d3a51a696c6ed4bed42c4d6 WatchSource:0}: Error finding container 243c0ea0d930527429e9354888f4d5766ee5601f7d3a51a696c6ed4bed42c4d6: Status 404 returned error can't find the container with id 243c0ea0d930527429e9354888f4d5766ee5601f7d3a51a696c6ed4bed42c4d6 Jul 12 19:31:25.685310 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:25.685270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5d7957d8-5htd4" event={"ID":"604b0ea5-098d-4f09-9e29-662e7c7dc326","Type":"ContainerStarted","Data":"c786e8ed3f6ef71e242b27ee8d45fc8c82df4fdd0fd25d90d5e2b83dfeba5f77"} Jul 12 19:31:25.685778 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:25.685329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5d7957d8-5htd4" event={"ID":"604b0ea5-098d-4f09-9e29-662e7c7dc326","Type":"ContainerStarted","Data":"243c0ea0d930527429e9354888f4d5766ee5601f7d3a51a696c6ed4bed42c4d6"} Jul 12 19:31:25.724250 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:25.724189 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d5d7957d8-5htd4" podStartSLOduration=1.724169822 podStartE2EDuration="1.724169822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:25.721268791 +0000 UTC m=+684.144016641" watchObservedRunningTime="2026-07-12 19:31:25.724169822 +0000 UTC m=+684.146917672" Jul 12 19:31:28.609490 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.609450 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg"] Jul 12 19:31:28.612941 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.612922 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" Jul 12 19:31:28.615407 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.615382 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 12 19:31:28.615540 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.615522 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 12 19:31:28.616358 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.616340 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 12 19:31:28.616467 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.616340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-29xv2\"" Jul 12 19:31:28.622767 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.622739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg"] Jul 12 19:31:28.636792 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.636757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6n4\" (UniqueName: \"kubernetes.io/projected/c2032a6a-c3c0-4134-836b-be2e8467e9ef-kube-api-access-bs6n4\") pod \"dns-operator-controller-manager-864dd78c4c-x5qlg\" (UID: \"c2032a6a-c3c0-4134-836b-be2e8467e9ef\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" Jul 12 19:31:28.737306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.737263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6n4\" (UniqueName: \"kubernetes.io/projected/c2032a6a-c3c0-4134-836b-be2e8467e9ef-kube-api-access-bs6n4\") pod \"dns-operator-controller-manager-864dd78c4c-x5qlg\" (UID: \"c2032a6a-c3c0-4134-836b-be2e8467e9ef\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" Jul 12 19:31:28.745302 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.745271 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6n4\" (UniqueName: \"kubernetes.io/projected/c2032a6a-c3c0-4134-836b-be2e8467e9ef-kube-api-access-bs6n4\") pod \"dns-operator-controller-manager-864dd78c4c-x5qlg\" (UID: \"c2032a6a-c3c0-4134-836b-be2e8467e9ef\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" Jul 12 19:31:28.924817 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:28.924783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" Jul 12 19:31:29.074909 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:29.074833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg"] Jul 12 19:31:29.077922 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:31:29.077882 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2032a6a_c3c0_4134_836b_be2e8467e9ef.slice/crio-3be42d3ccc609c82d91c74c749a54ed8dcfb327e18035df8a86df14c632d81ee WatchSource:0}: Error finding container 3be42d3ccc609c82d91c74c749a54ed8dcfb327e18035df8a86df14c632d81ee: Status 404 returned error can't find the container with id 3be42d3ccc609c82d91c74c749a54ed8dcfb327e18035df8a86df14c632d81ee Jul 12 19:31:29.080277 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:29.080257 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:31:29.705478 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:29.705445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" event={"ID":"c2032a6a-c3c0-4134-836b-be2e8467e9ef","Type":"ContainerStarted","Data":"3be42d3ccc609c82d91c74c749a54ed8dcfb327e18035df8a86df14c632d81ee"} Jul 12 19:31:31.719421 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:31.719379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" event={"ID":"c2032a6a-c3c0-4134-836b-be2e8467e9ef","Type":"ContainerStarted","Data":"7fb79996b5ed4e2a6ae4152b86de1ac8de9bb0db481034ba62fd813a60ee557f"} Jul 12 19:31:31.719824 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:31.719472 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" Jul 12 19:31:31.736824 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:31.736774 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" podStartSLOduration=1.489023037 podStartE2EDuration="3.736751652s" totalImagesPullingTime="2.247728615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:31.735583909 +0000 UTC m=+690.158331755" watchObservedRunningTime="2026-07-12 19:31:31.736751652 +0000 UTC m=+690.159499507" Jul 12 19:31:34.541182 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:34.541126 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:34.541658 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:34.541333 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:34.546724 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:34.546693 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:34.737051 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:34.737020 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d5d7957d8-5htd4" Jul 12 19:31:34.790012 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:34.789978 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-694c577648-pwm9z"] Jul 12 19:31:42.199812 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.199772 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww"] Jul 12 19:31:42.203241 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.203220 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.206342 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.206318 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kcwqf\"" Jul 12 19:31:42.214752 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.214719 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww"] Jul 12 19:31:42.262195 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.262158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" (UID: \"08cbe0b0-365b-4429-8e75-068f9e1c7eb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.262373 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.262212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2rm\" (UniqueName: \"kubernetes.io/projected/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-kube-api-access-5f2rm\") pod \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" (UID: \"08cbe0b0-365b-4429-8e75-068f9e1c7eb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.348186 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.348149 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww"] Jul 12 19:31:42.348489 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:31:42.348462 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-5f2rm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" podUID="08cbe0b0-365b-4429-8e75-068f9e1c7eb5" Jul 12 19:31:42.351739 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.351701 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww"] Jul 12 19:31:42.363019 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.362977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2rm\" (UniqueName: \"kubernetes.io/projected/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-kube-api-access-5f2rm\") pod \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" (UID: \"08cbe0b0-365b-4429-8e75-068f9e1c7eb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.363224 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.363081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" (UID: \"08cbe0b0-365b-4429-8e75-068f9e1c7eb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.363500 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.363480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" (UID: \"08cbe0b0-365b-4429-8e75-068f9e1c7eb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.365913 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:31:42.365877 2565 projected.go:196] Error preparing data for projected volume kube-api-access-5f2rm for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-129-169.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-129-169.ec2.internal' and this object Jul 12 19:31:42.366118 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:31:42.365997 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-kube-api-access-5f2rm podName:08cbe0b0-365b-4429-8e75-068f9e1c7eb5 nodeName:}" failed. No retries permitted until 2026-07-12 19:31:42.865972687 +0000 UTC m=+701.288720530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5f2rm" (UniqueName: "kubernetes.io/projected/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-kube-api-access-5f2rm") pod "kuadrant-operator-controller-manager-86878c769f-dn4ww" (UID: "08cbe0b0-365b-4429-8e75-068f9e1c7eb5") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-129-169.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-129-169.ec2.internal' and this object Jul 12 19:31:42.726917 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.726883 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-x5qlg" Jul 12 19:31:42.778630 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.778601 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.780907 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:31:42.780870 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" is forbidden: User \"system:node:ip-10-0-129-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-169.ec2.internal' and this object" podUID="08cbe0b0-365b-4429-8e75-068f9e1c7eb5" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.785902 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.785875 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.788019 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:31:42.787992 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" is forbidden: User \"system:node:ip-10-0-129-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-169.ec2.internal' and this object" podUID="08cbe0b0-365b-4429-8e75-068f9e1c7eb5" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:42.868163 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.868122 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-extensions-socket-volume\") pod \"08cbe0b0-365b-4429-8e75-068f9e1c7eb5\" (UID: \"08cbe0b0-365b-4429-8e75-068f9e1c7eb5\") " Jul 12 19:31:42.868366 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.868352 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f2rm\" (UniqueName: \"kubernetes.io/projected/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-kube-api-access-5f2rm\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:31:42.868517 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.868492 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "08cbe0b0-365b-4429-8e75-068f9e1c7eb5" (UID: "08cbe0b0-365b-4429-8e75-068f9e1c7eb5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:31:42.969634 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:42.969591 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/08cbe0b0-365b-4429-8e75-068f9e1c7eb5-extensions-socket-volume\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:31:43.783506 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:43.783465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:43.785788 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:31:43.785753 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" is forbidden: User \"system:node:ip-10-0-129-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-169.ec2.internal' and this object" podUID="08cbe0b0-365b-4429-8e75-068f9e1c7eb5" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:43.791484 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:31:43.791449 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dn4ww\" is forbidden: User \"system:node:ip-10-0-129-169.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-169.ec2.internal' and this object" podUID="08cbe0b0-365b-4429-8e75-068f9e1c7eb5" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dn4ww" Jul 12 19:31:44.056182 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:44.056099 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08cbe0b0-365b-4429-8e75-068f9e1c7eb5" path="/var/lib/kubelet/pods/08cbe0b0-365b-4429-8e75-068f9e1c7eb5/volumes" Jul 12 19:31:59.821917 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:31:59.821802 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-694c577648-pwm9z" podUID="f214dfdc-97c9-44a0-8606-789d4e5cd48e" containerName="console" containerID="cri-o://1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e" gracePeriod=15 Jul 12 19:32:00.080505 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.080417 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.222824 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-trusted-ca-bundle\") pod \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.222884 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-oauth-config\") pod \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.222949 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-config\") pod \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.223012 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-serving-cert\") pod \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.223096 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-oauth-serving-cert\") pod \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.223124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-service-ca\") pod \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.223170 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tc9l\" (UniqueName: \"kubernetes.io/projected/f214dfdc-97c9-44a0-8606-789d4e5cd48e-kube-api-access-9tc9l\") pod \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\" (UID: \"f214dfdc-97c9-44a0-8606-789d4e5cd48e\") " Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.223320 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f214dfdc-97c9-44a0-8606-789d4e5cd48e" (UID: "f214dfdc-97c9-44a0-8606-789d4e5cd48e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.223488 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-trusted-ca-bundle\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.223922 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f214dfdc-97c9-44a0-8606-789d4e5cd48e" (UID: "f214dfdc-97c9-44a0-8606-789d4e5cd48e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.224180 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-service-ca" (OuterVolumeSpecName: "service-ca") pod "f214dfdc-97c9-44a0-8606-789d4e5cd48e" (UID: "f214dfdc-97c9-44a0-8606-789d4e5cd48e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.224703 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-config" (OuterVolumeSpecName: "console-config") pod "f214dfdc-97c9-44a0-8606-789d4e5cd48e" (UID: "f214dfdc-97c9-44a0-8606-789d4e5cd48e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:32:00.228369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.226291 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f214dfdc-97c9-44a0-8606-789d4e5cd48e" (UID: "f214dfdc-97c9-44a0-8606-789d4e5cd48e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:32:00.229134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.228745 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f214dfdc-97c9-44a0-8606-789d4e5cd48e" (UID: "f214dfdc-97c9-44a0-8606-789d4e5cd48e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:32:00.229134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.229112 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f214dfdc-97c9-44a0-8606-789d4e5cd48e-kube-api-access-9tc9l" (OuterVolumeSpecName: "kube-api-access-9tc9l") pod "f214dfdc-97c9-44a0-8606-789d4e5cd48e" (UID: "f214dfdc-97c9-44a0-8606-789d4e5cd48e"). InnerVolumeSpecName "kube-api-access-9tc9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:32:00.324634 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.324597 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tc9l\" (UniqueName: \"kubernetes.io/projected/f214dfdc-97c9-44a0-8606-789d4e5cd48e-kube-api-access-9tc9l\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:32:00.324634 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.324629 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-oauth-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:32:00.324634 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.324639 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-config\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:32:00.324872 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.324648 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f214dfdc-97c9-44a0-8606-789d4e5cd48e-console-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:32:00.324872 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.324657 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-oauth-serving-cert\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:32:00.324872 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.324666 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f214dfdc-97c9-44a0-8606-789d4e5cd48e-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:32:00.872396 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.872364 2565 generic.go:356] "Generic (PLEG): container finished" podID="f214dfdc-97c9-44a0-8606-789d4e5cd48e" containerID="1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e" exitCode=0 Jul 12 19:32:00.873162 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.872415 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694c577648-pwm9z" Jul 12 19:32:00.873162 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.872430 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694c577648-pwm9z" event={"ID":"f214dfdc-97c9-44a0-8606-789d4e5cd48e","Type":"ContainerDied","Data":"1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e"} Jul 12 19:32:00.873162 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.872473 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694c577648-pwm9z" event={"ID":"f214dfdc-97c9-44a0-8606-789d4e5cd48e","Type":"ContainerDied","Data":"5da7d395f7fc29931f2e53d3e1c1dcdb6e7e2fa7897558e1c6ff616928a3cd59"} Jul 12 19:32:00.873162 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.872492 2565 scope.go:117] "RemoveContainer" containerID="1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e" Jul 12 19:32:00.884492 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.884476 2565 scope.go:117] "RemoveContainer" containerID="1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e" Jul 12 19:32:00.885025 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:32:00.885000 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e\": container with ID starting with 1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e not found: ID does not exist" containerID="1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e" Jul 12 19:32:00.885120 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.885039 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e"} err="failed to get container status \"1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e\": rpc error: code = NotFound desc = could not find container \"1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e\": container with ID starting with 1fa2eaa3589ee124502d15c872cbeb9ab8be36a351a4280dfc9ce3ff092c389e not found: ID does not exist" Jul 12 19:32:00.897781 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.897745 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-694c577648-pwm9z"] Jul 12 19:32:00.904628 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:00.904598 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-694c577648-pwm9z"] Jul 12 19:32:02.055925 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:02.055892 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f214dfdc-97c9-44a0-8606-789d4e5cd48e" path="/var/lib/kubelet/pods/f214dfdc-97c9-44a0-8606-789d4e5cd48e/volumes" Jul 12 19:32:26.471583 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.471548 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:32:26.471993 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.471916 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f214dfdc-97c9-44a0-8606-789d4e5cd48e" containerName="console" Jul 12 19:32:26.471993 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.471927 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f214dfdc-97c9-44a0-8606-789d4e5cd48e" containerName="console" Jul 12 19:32:26.472076 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.472005 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f214dfdc-97c9-44a0-8606-789d4e5cd48e" containerName="console" Jul 12 19:32:26.474915 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.474896 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.477353 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.477319 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 12 19:32:26.477486 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.477416 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qlwpj\"" Jul 12 19:32:26.481586 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.481559 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:32:26.509040 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.509004 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:32:26.560398 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.560349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65djx\" (UniqueName: \"kubernetes.io/projected/48414c6c-09ae-44ce-b9f5-848857fa82b5-kube-api-access-65djx\") pod \"limitador-limitador-54d6d988d5-94q64\" (UID: \"48414c6c-09ae-44ce-b9f5-848857fa82b5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.560398 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.560398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48414c6c-09ae-44ce-b9f5-848857fa82b5-config-file\") pod \"limitador-limitador-54d6d988d5-94q64\" (UID: \"48414c6c-09ae-44ce-b9f5-848857fa82b5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.661639 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.661599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65djx\" (UniqueName: \"kubernetes.io/projected/48414c6c-09ae-44ce-b9f5-848857fa82b5-kube-api-access-65djx\") pod \"limitador-limitador-54d6d988d5-94q64\" (UID: \"48414c6c-09ae-44ce-b9f5-848857fa82b5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.661812 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.661649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48414c6c-09ae-44ce-b9f5-848857fa82b5-config-file\") pod \"limitador-limitador-54d6d988d5-94q64\" (UID: \"48414c6c-09ae-44ce-b9f5-848857fa82b5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.662330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.662310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48414c6c-09ae-44ce-b9f5-848857fa82b5-config-file\") pod \"limitador-limitador-54d6d988d5-94q64\" (UID: \"48414c6c-09ae-44ce-b9f5-848857fa82b5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.669380 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.669355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65djx\" (UniqueName: \"kubernetes.io/projected/48414c6c-09ae-44ce-b9f5-848857fa82b5-kube-api-access-65djx\") pod \"limitador-limitador-54d6d988d5-94q64\" (UID: \"48414c6c-09ae-44ce-b9f5-848857fa82b5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.786556 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.786452 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:26.926209 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:26.926180 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:32:26.928249 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:32:26.928213 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48414c6c_09ae_44ce_b9f5_848857fa82b5.slice/crio-70bb772bcdd0d70631fa50895ef15d521d13289a4de1984053809e2d76f1ad58 WatchSource:0}: Error finding container 70bb772bcdd0d70631fa50895ef15d521d13289a4de1984053809e2d76f1ad58: Status 404 returned error can't find the container with id 70bb772bcdd0d70631fa50895ef15d521d13289a4de1984053809e2d76f1ad58 Jul 12 19:32:27.004899 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:27.004868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" event={"ID":"48414c6c-09ae-44ce-b9f5-848857fa82b5","Type":"ContainerStarted","Data":"70bb772bcdd0d70631fa50895ef15d521d13289a4de1984053809e2d76f1ad58"} Jul 12 19:32:30.024221 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:30.024183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" event={"ID":"48414c6c-09ae-44ce-b9f5-848857fa82b5","Type":"ContainerStarted","Data":"a64ea25c9f1e81a23a2bde144ca122437f01ae2ee21332e733135ab3bb10b693"} Jul 12 19:32:30.024684 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:30.024263 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:32:30.039364 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:30.039312 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" podStartSLOduration=1.481013805 podStartE2EDuration="4.039294988s" totalImagesPullingTime="2.558281183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:32:30.038533703 +0000 UTC m=+748.461281554" watchObservedRunningTime="2026-07-12 19:32:30.039294988 +0000 UTC m=+748.462042838" Jul 12 19:32:41.030410 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:32:41.030372 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-94q64" Jul 12 19:33:38.232209 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.232098 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x"] Jul 12 19:33:38.235602 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.235583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.238385 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.238359 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v78b6\"" Jul 12 19:33:38.238529 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.238407 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 12 19:33:38.238529 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.238415 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 12 19:33:38.239346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.239319 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 12 19:33:38.246123 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.246095 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x"] Jul 12 19:33:38.414673 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.414638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwjd\" (UniqueName: \"kubernetes.io/projected/7a877aed-88cc-4244-8f74-df46fc878c9c-kube-api-access-tpwjd\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.414897 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.414693 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a877aed-88cc-4244-8f74-df46fc878c9c-maas-api-tls\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.414897 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.414720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a877aed-88cc-4244-8f74-df46fc878c9c-openshift-service-ca\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.515728 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.515627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a877aed-88cc-4244-8f74-df46fc878c9c-maas-api-tls\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.515728 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.515673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a877aed-88cc-4244-8f74-df46fc878c9c-openshift-service-ca\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.516006 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.515742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwjd\" (UniqueName: \"kubernetes.io/projected/7a877aed-88cc-4244-8f74-df46fc878c9c-kube-api-access-tpwjd\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.516006 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:33:38.515776 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 12 19:33:38.516006 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:33:38.515855 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a877aed-88cc-4244-8f74-df46fc878c9c-maas-api-tls podName:7a877aed-88cc-4244-8f74-df46fc878c9c nodeName:}" failed. No retries permitted until 2026-07-12 19:33:39.015837926 +0000 UTC m=+817.438585754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a877aed-88cc-4244-8f74-df46fc878c9c-maas-api-tls") pod "maas-api-b9ccf5cf-cx64x" (UID: "7a877aed-88cc-4244-8f74-df46fc878c9c") : secret "maas-api-serving-cert" not found Jul 12 19:33:38.516470 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.516450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a877aed-88cc-4244-8f74-df46fc878c9c-openshift-service-ca\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.527247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.527206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwjd\" (UniqueName: \"kubernetes.io/projected/7a877aed-88cc-4244-8f74-df46fc878c9c-kube-api-access-tpwjd\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:38.795545 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:38.795438 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:33:39.021496 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:39.021454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a877aed-88cc-4244-8f74-df46fc878c9c-maas-api-tls\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:39.024252 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:39.024211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a877aed-88cc-4244-8f74-df46fc878c9c-maas-api-tls\") pod \"maas-api-b9ccf5cf-cx64x\" (UID: \"7a877aed-88cc-4244-8f74-df46fc878c9c\") " pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:39.147341 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:39.147298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:39.290698 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:39.290656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x"] Jul 12 19:33:39.294082 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:33:39.294053 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a877aed_88cc_4244_8f74_df46fc878c9c.slice/crio-ffc1311e66ae06dc3fb43ad7932e298e608f8183540b689ab5ffa0cf2be17481 WatchSource:0}: Error finding container ffc1311e66ae06dc3fb43ad7932e298e608f8183540b689ab5ffa0cf2be17481: Status 404 returned error can't find the container with id ffc1311e66ae06dc3fb43ad7932e298e608f8183540b689ab5ffa0cf2be17481 Jul 12 19:33:39.350156 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:39.350120 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" event={"ID":"7a877aed-88cc-4244-8f74-df46fc878c9c","Type":"ContainerStarted","Data":"ffc1311e66ae06dc3fb43ad7932e298e608f8183540b689ab5ffa0cf2be17481"} Jul 12 19:33:41.361519 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:41.361480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" event={"ID":"7a877aed-88cc-4244-8f74-df46fc878c9c","Type":"ContainerStarted","Data":"953283ea4566dec278ef612ab11bbddbeacf78467db1057e6850827881ef7e58"} Jul 12 19:33:41.361904 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:41.361583 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:33:41.382117 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:41.382053 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" podStartSLOduration=1.8838996460000001 podStartE2EDuration="3.382032578s" totalImagesPullingTime="1.498132932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:33:41.379722036 +0000 UTC m=+819.802469885" watchObservedRunningTime="2026-07-12 19:33:41.382032578 +0000 UTC m=+819.804780430" Jul 12 19:33:47.373193 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:33:47.373163 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b9ccf5cf-cx64x" Jul 12 19:34:07.770059 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:07.770016 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:34:18.963790 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:18.963748 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv"] Jul 12 19:34:18.968931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:18.968901 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:18.971540 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:18.971522 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c45j6\"" Jul 12 19:34:18.972618 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:18.972596 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 12 19:34:18.972736 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:18.972596 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 12 19:34:18.972736 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:18.972667 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 12 19:34:18.977404 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:18.977382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv"] Jul 12 19:34:19.054920 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.054874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.055129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.054924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.055129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.055017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823bdb78-adb7-496b-beaf-97371ba1c254-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.055129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.055045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.055129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.055069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.055129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.055088 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586b7\" (UniqueName: \"kubernetes.io/projected/823bdb78-adb7-496b-beaf-97371ba1c254-kube-api-access-586b7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156084 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156084 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156091 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823bdb78-adb7-496b-beaf-97371ba1c254-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586b7\" (UniqueName: \"kubernetes.io/projected/823bdb78-adb7-496b-beaf-97371ba1c254-kube-api-access-586b7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156549 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156549 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.156651 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.156586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.158703 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.158672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823bdb78-adb7-496b-beaf-97371ba1c254-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.158833 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.158817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823bdb78-adb7-496b-beaf-97371ba1c254-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.163650 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.163624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-586b7\" (UniqueName: \"kubernetes.io/projected/823bdb78-adb7-496b-beaf-97371ba1c254-kube-api-access-586b7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-qbdcv\" (UID: \"823bdb78-adb7-496b-beaf-97371ba1c254\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.281658 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.281554 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:19.418119 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.418095 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv"] Jul 12 19:34:19.420381 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:34:19.420349 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823bdb78_adb7_496b_beaf_97371ba1c254.slice/crio-08a516c9feb3ea74d4f77bc75b08f9efa25fe271bf35cfc8132ba202114bbb94 WatchSource:0}: Error finding container 08a516c9feb3ea74d4f77bc75b08f9efa25fe271bf35cfc8132ba202114bbb94: Status 404 returned error can't find the container with id 08a516c9feb3ea74d4f77bc75b08f9efa25fe271bf35cfc8132ba202114bbb94 Jul 12 19:34:19.551149 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.551059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" event={"ID":"823bdb78-adb7-496b-beaf-97371ba1c254","Type":"ContainerStarted","Data":"08a516c9feb3ea74d4f77bc75b08f9efa25fe271bf35cfc8132ba202114bbb94"} Jul 12 19:34:19.666007 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:19.665952 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:34:20.558063 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:20.557998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" event={"ID":"823bdb78-adb7-496b-beaf-97371ba1c254","Type":"ContainerStarted","Data":"31bc9ecd63356cafb6027324f7fa69a6eeff886cdd0026be3a869120f146b55b"} Jul 12 19:34:20.575299 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:20.575245 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" podStartSLOduration=1.574324481 podStartE2EDuration="2.575227976s" totalImagesPullingTime="1.000903495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:34:20.574796364 +0000 UTC m=+858.997544215" watchObservedRunningTime="2026-07-12 19:34:20.575227976 +0000 UTC m=+858.997975826" Jul 12 19:34:29.282554 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:29.282507 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:29.283189 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:29.282606 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:29.295143 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:29.295113 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:29.612154 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:29.612071 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-qbdcv" Jul 12 19:34:30.768886 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:30.768843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:34:35.364585 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:35.364548 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:34:42.664308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:34:42.664263 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:35:01.952342 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:35:01.952263 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:35:01.953999 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:35:01.953955 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:35:01.956573 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:35:01.956552 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:35:01.957999 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:35:01.957948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:35:03.167259 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:35:03.167219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:36:35.865248 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:36:35.865155 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:36:45.265418 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:36:45.265376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:36:56.272924 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:36:56.272878 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:37:05.273538 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:37:05.273498 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:37:16.265107 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:37:16.265065 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:37:42.665794 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:37:42.665749 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:37:56.170308 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:37:56.170220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:38:27.368146 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:38:27.368102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:38:37.061999 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:38:37.061941 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:38:46.764353 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:38:46.764311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:38:54.367638 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:38:54.367596 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:39:06.062850 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:39:06.062813 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:39:16.778468 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:39:16.778430 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:39:28.061816 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:39:28.061726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:39:37.266934 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:39:37.266893 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:39:59.668296 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:39:59.668256 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:40:01.982301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:01.982262 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:40:01.985647 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:01.985598 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:40:01.990038 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:01.990014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:40:01.992522 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:01.992500 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:40:08.876205 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:08.876161 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:40:32.272696 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:32.272653 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:40:42.064944 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:42.064897 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:40:57.772601 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:40:57.772496 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:41:07.866929 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:41:07.866891 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:41:17.864700 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:41:17.864660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:41:27.864497 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:41:27.864454 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:41:37.360089 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:41:37.360048 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:43:25.567021 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:43:25.566952 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:43:34.773694 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:43:34.773650 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:43:47.864331 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:43:47.864290 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:43:58.870008 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:43:58.869898 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:44:10.070646 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:44:10.070609 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:44:20.970833 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:44:20.970791 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:44:30.765664 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:44:30.765618 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:44:40.969681 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:44:40.969642 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:44:51.765347 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:44:51.765297 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:45:00.137585 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.137547 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt"] Jul 12 19:45:00.141034 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.141013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" Jul 12 19:45:00.154529 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.154499 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt"] Jul 12 19:45:00.274907 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.274864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8787\" (UniqueName: \"kubernetes.io/projected/73da0717-a5fc-4ab0-a32e-bd34454c73c1-kube-api-access-m8787\") pod \"maas-api-key-cleanup-29731425-hmlwt\" (UID: \"73da0717-a5fc-4ab0-a32e-bd34454c73c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" Jul 12 19:45:00.375401 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.375355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8787\" (UniqueName: \"kubernetes.io/projected/73da0717-a5fc-4ab0-a32e-bd34454c73c1-kube-api-access-m8787\") pod \"maas-api-key-cleanup-29731425-hmlwt\" (UID: \"73da0717-a5fc-4ab0-a32e-bd34454c73c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" Jul 12 19:45:00.383695 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.383658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8787\" (UniqueName: \"kubernetes.io/projected/73da0717-a5fc-4ab0-a32e-bd34454c73c1-kube-api-access-m8787\") pod \"maas-api-key-cleanup-29731425-hmlwt\" (UID: \"73da0717-a5fc-4ab0-a32e-bd34454c73c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" Jul 12 19:45:00.451648 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.451558 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" Jul 12 19:45:00.596600 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.596572 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt"] Jul 12 19:45:00.599173 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:45:00.599147 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73da0717_a5fc_4ab0_a32e_bd34454c73c1.slice/crio-4e259ab3de158b8d3ed47465ad494ee9f23fac95ccf1e8e9c386d3098bde0b34 WatchSource:0}: Error finding container 4e259ab3de158b8d3ed47465ad494ee9f23fac95ccf1e8e9c386d3098bde0b34: Status 404 returned error can't find the container with id 4e259ab3de158b8d3ed47465ad494ee9f23fac95ccf1e8e9c386d3098bde0b34 Jul 12 19:45:00.601231 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.601215 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:45:00.717014 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:00.716898 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" event={"ID":"73da0717-a5fc-4ab0-a32e-bd34454c73c1","Type":"ContainerStarted","Data":"4e259ab3de158b8d3ed47465ad494ee9f23fac95ccf1e8e9c386d3098bde0b34"} Jul 12 19:45:01.274116 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:01.274074 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:45:02.020984 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:02.020938 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:45:02.022197 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:02.022165 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:45:02.025405 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:02.025379 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:45:02.026701 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:02.026682 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:45:03.735507 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:03.735465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" event={"ID":"73da0717-a5fc-4ab0-a32e-bd34454c73c1","Type":"ContainerStarted","Data":"b1dd7f1a8d62bde7255f3d57708f0a1a0e0b1cdca6ae0b8d689c11e9150f81df"} Jul 12 19:45:03.750846 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:03.750791 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" podStartSLOduration=1.527729593 podStartE2EDuration="3.750774728s" totalImagesPullingTime="2.223045135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:45:03.750117637 +0000 UTC m=+1502.172865489" watchObservedRunningTime="2026-07-12 19:45:03.750774728 +0000 UTC m=+1502.173522579" Jul 12 19:45:12.970604 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:12.970558 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:45:37.673298 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:37.673256 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:45:45.170518 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:45.170473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:45:55.468311 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:45:55.468266 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:46:00.009901 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:00.009862 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt"] Jul 12 19:46:00.010386 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:00.010110 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" podUID="73da0717-a5fc-4ab0-a32e-bd34454c73c1" containerName="cleanup" containerID="cri-o://b1dd7f1a8d62bde7255f3d57708f0a1a0e0b1cdca6ae0b8d689c11e9150f81df" gracePeriod=30 Jul 12 19:46:05.364761 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:05.364715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:46:14.380004 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:14.379947 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:46:26.973976 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:26.973912 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:46:30.160840 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:30.160804 2565 generic.go:356] "Generic (PLEG): container finished" podID="73da0717-a5fc-4ab0-a32e-bd34454c73c1" containerID="b1dd7f1a8d62bde7255f3d57708f0a1a0e0b1cdca6ae0b8d689c11e9150f81df" exitCode=137 Jul 12 19:46:30.161234 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:30.160874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" event={"ID":"73da0717-a5fc-4ab0-a32e-bd34454c73c1","Type":"ContainerDied","Data":"b1dd7f1a8d62bde7255f3d57708f0a1a0e0b1cdca6ae0b8d689c11e9150f81df"} Jul 12 19:46:30.213091 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:30.213060 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" Jul 12 19:46:30.285233 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:30.285137 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8787\" (UniqueName: \"kubernetes.io/projected/73da0717-a5fc-4ab0-a32e-bd34454c73c1-kube-api-access-m8787\") pod \"73da0717-a5fc-4ab0-a32e-bd34454c73c1\" (UID: \"73da0717-a5fc-4ab0-a32e-bd34454c73c1\") " Jul 12 19:46:30.287476 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:30.287442 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73da0717-a5fc-4ab0-a32e-bd34454c73c1-kube-api-access-m8787" (OuterVolumeSpecName: "kube-api-access-m8787") pod "73da0717-a5fc-4ab0-a32e-bd34454c73c1" (UID: "73da0717-a5fc-4ab0-a32e-bd34454c73c1"). InnerVolumeSpecName "kube-api-access-m8787". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:46:30.386708 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:30.386651 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8787\" (UniqueName: \"kubernetes.io/projected/73da0717-a5fc-4ab0-a32e-bd34454c73c1-kube-api-access-m8787\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:46:31.166773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:31.166746 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" Jul 12 19:46:31.167239 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:31.166741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt" event={"ID":"73da0717-a5fc-4ab0-a32e-bd34454c73c1","Type":"ContainerDied","Data":"4e259ab3de158b8d3ed47465ad494ee9f23fac95ccf1e8e9c386d3098bde0b34"} Jul 12 19:46:31.167239 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:31.166886 2565 scope.go:117] "RemoveContainer" containerID="b1dd7f1a8d62bde7255f3d57708f0a1a0e0b1cdca6ae0b8d689c11e9150f81df" Jul 12 19:46:31.188558 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:31.188526 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt"] Jul 12 19:46:31.191915 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:31.191882 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731425-hmlwt"] Jul 12 19:46:32.056093 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:32.056060 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73da0717-a5fc-4ab0-a32e-bd34454c73c1" path="/var/lib/kubelet/pods/73da0717-a5fc-4ab0-a32e-bd34454c73c1/volumes" Jul 12 19:46:37.566510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:37.566471 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:46:47.966750 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:47.966708 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:46:55.766627 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:46:55.766532 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:47:07.169878 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:47:07.169827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:47:17.869713 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:47:17.869679 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:47:29.564091 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:47:29.564040 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:47:38.972880 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:47:38.972838 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:47:53.461443 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:47:53.461407 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:48:01.467937 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:48:01.467894 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:48:14.264767 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:48:14.264728 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:48:23.072616 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:48:23.072574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:48:35.762546 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:48:35.762511 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:48:47.774079 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:48:47.774038 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:49:00.760360 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:00.760304 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:49:09.868519 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:09.868477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:49:30.565781 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:30.565737 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:49:36.574816 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.574783 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4"] Jul 12 19:49:36.575324 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.575287 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73da0717-a5fc-4ab0-a32e-bd34454c73c1" containerName="cleanup" Jul 12 19:49:36.575324 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.575307 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73da0717-a5fc-4ab0-a32e-bd34454c73c1" containerName="cleanup" Jul 12 19:49:36.575411 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.575398 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73da0717-a5fc-4ab0-a32e-bd34454c73c1" containerName="cleanup" Jul 12 19:49:36.578402 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.578377 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.580946 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.580923 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e8a6ab99-serving-cert\"" Jul 12 19:49:36.589872 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.589841 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4"] Jul 12 19:49:36.650890 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.650851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec9851d-95dd-44da-a546-19e4575c80c5-openshift-service-ca\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.651130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.650909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbbgj\" (UniqueName: \"kubernetes.io/projected/5ec9851d-95dd-44da-a546-19e4575c80c5-kube-api-access-jbbgj\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.651130 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.650944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec9851d-95dd-44da-a546-19e4575c80c5-maas-api-tls\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.752380 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.752334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec9851d-95dd-44da-a546-19e4575c80c5-maas-api-tls\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.752601 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.752454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec9851d-95dd-44da-a546-19e4575c80c5-openshift-service-ca\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.752601 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.752510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbbgj\" (UniqueName: \"kubernetes.io/projected/5ec9851d-95dd-44da-a546-19e4575c80c5-kube-api-access-jbbgj\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.753313 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.753228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec9851d-95dd-44da-a546-19e4575c80c5-openshift-service-ca\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.756189 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.756133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec9851d-95dd-44da-a546-19e4575c80c5-maas-api-tls\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.760632 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.760610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbbgj\" (UniqueName: \"kubernetes.io/projected/5ec9851d-95dd-44da-a546-19e4575c80c5-kube-api-access-jbbgj\") pod \"maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:36.892633 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:36.892595 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:37.034400 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:37.034251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4"] Jul 12 19:49:37.037335 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:49:37.037291 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec9851d_95dd_44da_a546_19e4575c80c5.slice/crio-110435cfb5b29298f3ab42d2cb5599f6db4437a437417e077bc9f8570eed3e1f WatchSource:0}: Error finding container 110435cfb5b29298f3ab42d2cb5599f6db4437a437417e077bc9f8570eed3e1f: Status 404 returned error can't find the container with id 110435cfb5b29298f3ab42d2cb5599f6db4437a437417e077bc9f8570eed3e1f Jul 12 19:49:37.090033 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:37.089995 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" event={"ID":"5ec9851d-95dd-44da-a546-19e4575c80c5","Type":"ContainerStarted","Data":"110435cfb5b29298f3ab42d2cb5599f6db4437a437417e077bc9f8570eed3e1f"} Jul 12 19:49:38.096503 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:38.096461 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" event={"ID":"5ec9851d-95dd-44da-a546-19e4575c80c5","Type":"ContainerStarted","Data":"0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0"} Jul 12 19:49:38.097035 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:38.096628 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:38.113349 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:38.113296 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" podStartSLOduration=1.692068246 podStartE2EDuration="2.113278255s" totalImagesPullingTime="421.210009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:49:38.111660839 +0000 UTC m=+1776.534408693" watchObservedRunningTime="2026-07-12 19:49:38.113278255 +0000 UTC m=+1776.536026128" Jul 12 19:49:41.371056 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:41.371013 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:49:41.809432 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:41.809335 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4"] Jul 12 19:49:41.809661 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:41.809631 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" podUID="5ec9851d-95dd-44da-a546-19e4575c80c5" containerName="maas-api" containerID="cri-o://0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0" gracePeriod=30 Jul 12 19:49:42.067209 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.067132 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:42.118204 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.118163 2565 generic.go:356] "Generic (PLEG): container finished" podID="5ec9851d-95dd-44da-a546-19e4575c80c5" containerID="0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0" exitCode=0 Jul 12 19:49:42.118405 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.118218 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" Jul 12 19:49:42.118405 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.118262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" event={"ID":"5ec9851d-95dd-44da-a546-19e4575c80c5","Type":"ContainerDied","Data":"0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0"} Jul 12 19:49:42.118405 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.118323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4" event={"ID":"5ec9851d-95dd-44da-a546-19e4575c80c5","Type":"ContainerDied","Data":"110435cfb5b29298f3ab42d2cb5599f6db4437a437417e077bc9f8570eed3e1f"} Jul 12 19:49:42.118405 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.118353 2565 scope.go:117] "RemoveContainer" containerID="0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0" Jul 12 19:49:42.129272 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.129243 2565 scope.go:117] "RemoveContainer" containerID="0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0" Jul 12 19:49:42.129598 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:49:42.129579 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0\": container with ID starting with 0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0 not found: ID does not exist" containerID="0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0" Jul 12 19:49:42.129642 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.129610 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0"} err="failed to get container status \"0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0\": rpc error: code = NotFound desc = could not find container \"0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0\": container with ID starting with 0157def4a07a7dd6467d4a90f0a5eaf39c9cf6c971de1094fbbeebfbff0b29e0 not found: ID does not exist" Jul 12 19:49:42.206275 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.206239 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbbgj\" (UniqueName: \"kubernetes.io/projected/5ec9851d-95dd-44da-a546-19e4575c80c5-kube-api-access-jbbgj\") pod \"5ec9851d-95dd-44da-a546-19e4575c80c5\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " Jul 12 19:49:42.206486 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.206307 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec9851d-95dd-44da-a546-19e4575c80c5-maas-api-tls\") pod \"5ec9851d-95dd-44da-a546-19e4575c80c5\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " Jul 12 19:49:42.206486 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.206348 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec9851d-95dd-44da-a546-19e4575c80c5-openshift-service-ca\") pod \"5ec9851d-95dd-44da-a546-19e4575c80c5\" (UID: \"5ec9851d-95dd-44da-a546-19e4575c80c5\") " Jul 12 19:49:42.206766 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.206742 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec9851d-95dd-44da-a546-19e4575c80c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ec9851d-95dd-44da-a546-19e4575c80c5" (UID: "5ec9851d-95dd-44da-a546-19e4575c80c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:49:42.208521 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.208487 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec9851d-95dd-44da-a546-19e4575c80c5-kube-api-access-jbbgj" (OuterVolumeSpecName: "kube-api-access-jbbgj") pod "5ec9851d-95dd-44da-a546-19e4575c80c5" (UID: "5ec9851d-95dd-44da-a546-19e4575c80c5"). InnerVolumeSpecName "kube-api-access-jbbgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:42.208612 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.208525 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec9851d-95dd-44da-a546-19e4575c80c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ec9851d-95dd-44da-a546-19e4575c80c5" (UID: "5ec9851d-95dd-44da-a546-19e4575c80c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:49:42.307230 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.307186 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ec9851d-95dd-44da-a546-19e4575c80c5-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:49:42.307230 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.307223 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec9851d-95dd-44da-a546-19e4575c80c5-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:49:42.307230 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.307232 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbbgj\" (UniqueName: \"kubernetes.io/projected/5ec9851d-95dd-44da-a546-19e4575c80c5-kube-api-access-jbbgj\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:49:42.434867 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:49:42.434833 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec9851d_95dd_44da_a546_19e4575c80c5.slice\": RecentStats: unable to find data in memory cache]" Jul 12 19:49:42.440658 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.440625 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4"] Jul 12 19:49:42.446239 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:42.446211 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8a6ab99-69dfcb678c-kpvg4"] Jul 12 19:49:44.057860 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:44.057823 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ec9851d-95dd-44da-a546-19e4575c80c5" path="/var/lib/kubelet/pods/5ec9851d-95dd-44da-a546-19e4575c80c5/volumes" Jul 12 19:49:48.847085 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.847042 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs"] Jul 12 19:49:48.847509 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.847399 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec9851d-95dd-44da-a546-19e4575c80c5" containerName="maas-api" Jul 12 19:49:48.847509 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.847410 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec9851d-95dd-44da-a546-19e4575c80c5" containerName="maas-api" Jul 12 19:49:48.847509 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.847490 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec9851d-95dd-44da-a546-19e4575c80c5" containerName="maas-api" Jul 12 19:49:48.850900 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.850875 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:48.853575 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.853554 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1b63ffec-serving-cert\"" Jul 12 19:49:48.861131 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.861101 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs"] Jul 12 19:49:48.964597 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.964551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:48.964783 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.964615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-openshift-service-ca\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:48.964783 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:48.964730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblc7\" (UniqueName: \"kubernetes.io/projected/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-kube-api-access-bblc7\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.065551 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.065512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-openshift-service-ca\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.065743 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.065620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblc7\" (UniqueName: \"kubernetes.io/projected/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-kube-api-access-bblc7\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.065743 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.065669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.065887 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:49:49.065790 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-serving-cert: secret "maas-api-e2e-ait-1b63ffec-serving-cert" not found Jul 12 19:49:49.065887 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:49:49.065874 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls podName:64b4c88f-7f5a-4e79-9ea5-1992619cf0aa nodeName:}" failed. No retries permitted until 2026-07-12 19:49:49.565850761 +0000 UTC m=+1787.988598608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls") pod "maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" (UID: "64b4c88f-7f5a-4e79-9ea5-1992619cf0aa") : secret "maas-api-e2e-ait-1b63ffec-serving-cert" not found Jul 12 19:49:49.066270 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.066247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-openshift-service-ca\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.075300 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.075270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblc7\" (UniqueName: \"kubernetes.io/projected/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-kube-api-access-bblc7\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.571388 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.571347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.574694 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.574667 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls\") pod \"maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.762512 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.762472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:49.915036 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:49.915003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs"] Jul 12 19:49:49.918009 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:49:49.917978 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b4c88f_7f5a_4e79_9ea5_1992619cf0aa.slice/crio-d4ed4844c34f593de8aa7d8d2eb31efeed250d8cfde3f9d6d59388f2ca66c412 WatchSource:0}: Error finding container d4ed4844c34f593de8aa7d8d2eb31efeed250d8cfde3f9d6d59388f2ca66c412: Status 404 returned error can't find the container with id d4ed4844c34f593de8aa7d8d2eb31efeed250d8cfde3f9d6d59388f2ca66c412 Jul 12 19:49:50.159608 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:50.159571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" event={"ID":"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa","Type":"ContainerStarted","Data":"d4ed4844c34f593de8aa7d8d2eb31efeed250d8cfde3f9d6d59388f2ca66c412"} Jul 12 19:49:51.165659 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:51.165614 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" event={"ID":"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa","Type":"ContainerStarted","Data":"b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3"} Jul 12 19:49:51.166157 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:51.165780 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:51.184153 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:51.184100 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" podStartSLOduration=2.821879869 podStartE2EDuration="3.184083061s" totalImagesPullingTime="362.203192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:49:51.182036907 +0000 UTC m=+1789.604784760" watchObservedRunningTime="2026-07-12 19:49:51.184083061 +0000 UTC m=+1789.606830911" Jul 12 19:49:51.964260 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:51.964219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:49:54.225452 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.225359 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs"] Jul 12 19:49:54.225986 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.225591 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" podUID="64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" containerName="maas-api" containerID="cri-o://b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3" gracePeriod=30 Jul 12 19:49:54.475909 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.475839 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:54.621828 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.621790 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bblc7\" (UniqueName: \"kubernetes.io/projected/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-kube-api-access-bblc7\") pod \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " Jul 12 19:49:54.621828 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.621832 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-openshift-service-ca\") pod \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " Jul 12 19:49:54.622097 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.621875 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls\") pod \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\" (UID: \"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa\") " Jul 12 19:49:54.622312 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.622288 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" (UID: "64b4c88f-7f5a-4e79-9ea5-1992619cf0aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:49:54.624128 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.624094 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-kube-api-access-bblc7" (OuterVolumeSpecName: "kube-api-access-bblc7") pod "64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" (UID: "64b4c88f-7f5a-4e79-9ea5-1992619cf0aa"). InnerVolumeSpecName "kube-api-access-bblc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:54.624128 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.624114 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" (UID: "64b4c88f-7f5a-4e79-9ea5-1992619cf0aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:49:54.723394 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.723356 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bblc7\" (UniqueName: \"kubernetes.io/projected/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-kube-api-access-bblc7\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:49:54.723394 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.723388 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:49:54.723394 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:54.723400 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:49:55.187179 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.187141 2565 generic.go:356] "Generic (PLEG): container finished" podID="64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" containerID="b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3" exitCode=0 Jul 12 19:49:55.187327 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.187184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" Jul 12 19:49:55.187327 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.187188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" event={"ID":"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa","Type":"ContainerDied","Data":"b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3"} Jul 12 19:49:55.187327 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.187289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs" event={"ID":"64b4c88f-7f5a-4e79-9ea5-1992619cf0aa","Type":"ContainerDied","Data":"d4ed4844c34f593de8aa7d8d2eb31efeed250d8cfde3f9d6d59388f2ca66c412"} Jul 12 19:49:55.187327 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.187309 2565 scope.go:117] "RemoveContainer" containerID="b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3" Jul 12 19:49:55.196529 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.196509 2565 scope.go:117] "RemoveContainer" containerID="b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3" Jul 12 19:49:55.196796 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:49:55.196774 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3\": container with ID starting with b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3 not found: ID does not exist" containerID="b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3" Jul 12 19:49:55.196857 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.196806 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3"} err="failed to get container status \"b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3\": rpc error: code = NotFound desc = could not find container \"b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3\": container with ID starting with b1ebccd9614fa4ee821124dabbd5e7bbccf32b27da2273731ff58fad830d81f3 not found: ID does not exist" Jul 12 19:49:55.208041 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.208003 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs"] Jul 12 19:49:55.211761 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:55.211725 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b63ffec-7c87c5cb8b-x66vs"] Jul 12 19:49:56.056019 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:49:56.055984 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" path="/var/lib/kubelet/pods/64b4c88f-7f5a-4e79-9ea5-1992619cf0aa/volumes" Jul 12 19:50:02.054914 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:02.054728 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:50:02.061075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:02.061042 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:50:02.062738 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:02.062716 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:50:02.067129 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:02.067109 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:50:03.668028 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:03.667986 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:50:07.153150 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.153117 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7"] Jul 12 19:50:07.153710 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.153497 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" containerName="maas-api" Jul 12 19:50:07.153710 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.153509 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" containerName="maas-api" Jul 12 19:50:07.153710 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.153595 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b4c88f-7f5a-4e79-9ea5-1992619cf0aa" containerName="maas-api" Jul 12 19:50:07.156906 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.156878 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.159228 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.159201 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9fa1f168-serving-cert\"" Jul 12 19:50:07.168275 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.168247 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7"] Jul 12 19:50:07.238592 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.238485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4006e7ec-9f90-4807-abc7-5cd87d2b225b-openshift-service-ca\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.238592 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.238541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfw6s\" (UniqueName: \"kubernetes.io/projected/4006e7ec-9f90-4807-abc7-5cd87d2b225b-kube-api-access-qfw6s\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.238800 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.238624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.339910 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.339865 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw6s\" (UniqueName: \"kubernetes.io/projected/4006e7ec-9f90-4807-abc7-5cd87d2b225b-kube-api-access-qfw6s\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.340244 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.340223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.340448 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.340432 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4006e7ec-9f90-4807-abc7-5cd87d2b225b-openshift-service-ca\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.341368 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.341342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4006e7ec-9f90-4807-abc7-5cd87d2b225b-openshift-service-ca\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.346431 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:50:07.341885 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-serving-cert: secret "maas-api-e2e-derive-9fa1f168-serving-cert" not found Jul 12 19:50:07.346431 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:50:07.341997 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls podName:4006e7ec-9f90-4807-abc7-5cd87d2b225b nodeName:}" failed. No retries permitted until 2026-07-12 19:50:07.841947643 +0000 UTC m=+1806.264695473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls") pod "maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" (UID: "4006e7ec-9f90-4807-abc7-5cd87d2b225b") : secret "maas-api-e2e-derive-9fa1f168-serving-cert" not found Jul 12 19:50:07.350317 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.350284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw6s\" (UniqueName: \"kubernetes.io/projected/4006e7ec-9f90-4807-abc7-5cd87d2b225b-kube-api-access-qfw6s\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.845556 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.845514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:07.848343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:07.848311 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls\") pod \"maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:08.072388 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:08.072357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:08.220361 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:08.220335 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7"] Jul 12 19:50:08.222765 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:50:08.222723 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4006e7ec_9f90_4807_abc7_5cd87d2b225b.slice/crio-3cac6d4adafe568b45cb83a4e070bd2e826cc9e5fd26cdd207b051fc12e6ca34 WatchSource:0}: Error finding container 3cac6d4adafe568b45cb83a4e070bd2e826cc9e5fd26cdd207b051fc12e6ca34: Status 404 returned error can't find the container with id 3cac6d4adafe568b45cb83a4e070bd2e826cc9e5fd26cdd207b051fc12e6ca34 Jul 12 19:50:08.224316 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:08.224290 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:50:08.255203 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:08.255165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" event={"ID":"4006e7ec-9f90-4807-abc7-5cd87d2b225b","Type":"ContainerStarted","Data":"3cac6d4adafe568b45cb83a4e070bd2e826cc9e5fd26cdd207b051fc12e6ca34"} Jul 12 19:50:09.262849 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:09.262801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" event={"ID":"4006e7ec-9f90-4807-abc7-5cd87d2b225b","Type":"ContainerStarted","Data":"daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd"} Jul 12 19:50:09.263363 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:09.262895 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:09.280495 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:09.280445 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" podStartSLOduration=1.9678715900000001 podStartE2EDuration="2.280426487s" totalImagesPullingTime="312.554897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:50:09.278033243 +0000 UTC m=+1807.700781097" watchObservedRunningTime="2026-07-12 19:50:09.280426487 +0000 UTC m=+1807.703174337" Jul 12 19:50:11.807292 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:11.807250 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7"] Jul 12 19:50:11.807807 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:11.807500 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" podUID="4006e7ec-9f90-4807-abc7-5cd87d2b225b" containerName="maas-api" containerID="cri-o://daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd" gracePeriod=30 Jul 12 19:50:11.873364 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:11.873326 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:50:12.076025 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.075999 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:12.182005 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.181935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfw6s\" (UniqueName: \"kubernetes.io/projected/4006e7ec-9f90-4807-abc7-5cd87d2b225b-kube-api-access-qfw6s\") pod \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " Jul 12 19:50:12.182211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.182028 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls\") pod \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " Jul 12 19:50:12.182211 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.182103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4006e7ec-9f90-4807-abc7-5cd87d2b225b-openshift-service-ca\") pod \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\" (UID: \"4006e7ec-9f90-4807-abc7-5cd87d2b225b\") " Jul 12 19:50:12.182471 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.182445 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4006e7ec-9f90-4807-abc7-5cd87d2b225b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4006e7ec-9f90-4807-abc7-5cd87d2b225b" (UID: "4006e7ec-9f90-4807-abc7-5cd87d2b225b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:50:12.184271 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.184244 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4006e7ec-9f90-4807-abc7-5cd87d2b225b" (UID: "4006e7ec-9f90-4807-abc7-5cd87d2b225b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:50:12.184397 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.184376 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4006e7ec-9f90-4807-abc7-5cd87d2b225b-kube-api-access-qfw6s" (OuterVolumeSpecName: "kube-api-access-qfw6s") pod "4006e7ec-9f90-4807-abc7-5cd87d2b225b" (UID: "4006e7ec-9f90-4807-abc7-5cd87d2b225b"). InnerVolumeSpecName "kube-api-access-qfw6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:50:12.278940 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.278906 2565 generic.go:356] "Generic (PLEG): container finished" podID="4006e7ec-9f90-4807-abc7-5cd87d2b225b" containerID="daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd" exitCode=0 Jul 12 19:50:12.279159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.278955 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" Jul 12 19:50:12.279159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.278993 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" event={"ID":"4006e7ec-9f90-4807-abc7-5cd87d2b225b","Type":"ContainerDied","Data":"daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd"} Jul 12 19:50:12.279159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.279051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7" event={"ID":"4006e7ec-9f90-4807-abc7-5cd87d2b225b","Type":"ContainerDied","Data":"3cac6d4adafe568b45cb83a4e070bd2e826cc9e5fd26cdd207b051fc12e6ca34"} Jul 12 19:50:12.279159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.279074 2565 scope.go:117] "RemoveContainer" containerID="daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd" Jul 12 19:50:12.285632 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.285583 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfw6s\" (UniqueName: \"kubernetes.io/projected/4006e7ec-9f90-4807-abc7-5cd87d2b225b-kube-api-access-qfw6s\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:50:12.285632 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.285622 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4006e7ec-9f90-4807-abc7-5cd87d2b225b-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:50:12.285923 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.285638 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4006e7ec-9f90-4807-abc7-5cd87d2b225b-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:50:12.294041 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.294014 2565 scope.go:117] "RemoveContainer" containerID="daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd" Jul 12 19:50:12.294348 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:50:12.294325 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd\": container with ID starting with daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd not found: ID does not exist" containerID="daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd" Jul 12 19:50:12.294455 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.294357 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd"} err="failed to get container status \"daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd\": rpc error: code = NotFound desc = could not find container \"daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd\": container with ID starting with daa5d280dac1726272adbf8fd2ed401bdc617bab43a1e6c29857024c1c9de5bd not found: ID does not exist" Jul 12 19:50:12.304525 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.304487 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7"] Jul 12 19:50:12.307926 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:12.307890 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9fa1f168-5dcdd47bdc-sfsk7"] Jul 12 19:50:14.056859 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:14.056814 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4006e7ec-9f90-4807-abc7-5cd87d2b225b" path="/var/lib/kubelet/pods/4006e7ec-9f90-4807-abc7-5cd87d2b225b/volumes" Jul 12 19:50:21.572183 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:50:21.572135 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:51:39.667184 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:39.667083 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:51:48.565425 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:48.565386 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:51:54.649859 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.649820 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k"] Jul 12 19:51:54.650440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.650224 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4006e7ec-9f90-4807-abc7-5cd87d2b225b" containerName="maas-api" Jul 12 19:51:54.650440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.650237 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4006e7ec-9f90-4807-abc7-5cd87d2b225b" containerName="maas-api" Jul 12 19:51:54.650440 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.650301 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4006e7ec-9f90-4807-abc7-5cd87d2b225b" containerName="maas-api" Jul 12 19:51:54.653515 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.653489 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.656099 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.656074 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ac7c507d-serving-cert\"" Jul 12 19:51:54.666667 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.666625 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k"] Jul 12 19:51:54.685555 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.685514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8mf\" (UniqueName: \"kubernetes.io/projected/d49b5336-72ba-48f3-a779-8777adad079a-kube-api-access-lm8mf\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.685752 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.685574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d49b5336-72ba-48f3-a779-8777adad079a-openshift-service-ca\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.685752 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.685610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.786645 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.786612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8mf\" (UniqueName: \"kubernetes.io/projected/d49b5336-72ba-48f3-a779-8777adad079a-kube-api-access-lm8mf\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.786645 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.786659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d49b5336-72ba-48f3-a779-8777adad079a-openshift-service-ca\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.786931 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.786700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.786931 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:51:54.786866 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-serving-cert: secret "maas-api-e2e-mt-ac7c507d-serving-cert" not found Jul 12 19:51:54.787112 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:51:54.786946 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls podName:d49b5336-72ba-48f3-a779-8777adad079a nodeName:}" failed. No retries permitted until 2026-07-12 19:51:55.286924564 +0000 UTC m=+1913.709672395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls") pod "maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" (UID: "d49b5336-72ba-48f3-a779-8777adad079a") : secret "maas-api-e2e-mt-ac7c507d-serving-cert" not found Jul 12 19:51:54.787430 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.787406 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d49b5336-72ba-48f3-a779-8777adad079a-openshift-service-ca\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:54.801218 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:54.801184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8mf\" (UniqueName: \"kubernetes.io/projected/d49b5336-72ba-48f3-a779-8777adad079a-kube-api-access-lm8mf\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:55.292081 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:55.292038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:55.294753 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:55.294716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls\") pod \"maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:55.565667 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:55.565563 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:55.713794 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:55.713665 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k"] Jul 12 19:51:55.716859 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:51:55.716823 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49b5336_72ba_48f3_a779_8777adad079a.slice/crio-277b1a2d5a591cd2cef7e3c3a831952401b53416a58c4c02aeb60dc663d47025 WatchSource:0}: Error finding container 277b1a2d5a591cd2cef7e3c3a831952401b53416a58c4c02aeb60dc663d47025: Status 404 returned error can't find the container with id 277b1a2d5a591cd2cef7e3c3a831952401b53416a58c4c02aeb60dc663d47025 Jul 12 19:51:55.795560 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:55.795518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" event={"ID":"d49b5336-72ba-48f3-a779-8777adad079a","Type":"ContainerStarted","Data":"277b1a2d5a591cd2cef7e3c3a831952401b53416a58c4c02aeb60dc663d47025"} Jul 12 19:51:56.802466 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:56.802430 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" event={"ID":"d49b5336-72ba-48f3-a779-8777adad079a","Type":"ContainerStarted","Data":"fc1b70fdc12bd44828bca4fe35c365d763fefb3389abd57c8c00cb88099aeb1f"} Jul 12 19:51:56.802862 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:56.802508 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:51:56.820238 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:56.820177 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" podStartSLOduration=2.468577501 podStartE2EDuration="2.820159976s" totalImagesPullingTime="351.582475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:51:56.817738541 +0000 UTC m=+1915.240486390" watchObservedRunningTime="2026-07-12 19:51:56.820159976 +0000 UTC m=+1915.242907825" Jul 12 19:51:59.371247 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:51:59.371209 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:52:02.813115 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:02.813082 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:52:06.616411 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.616371 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k"] Jul 12 19:52:06.616880 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.616602 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" podUID="d49b5336-72ba-48f3-a779-8777adad079a" containerName="maas-api" containerID="cri-o://fc1b70fdc12bd44828bca4fe35c365d763fefb3389abd57c8c00cb88099aeb1f" gracePeriod=30 Jul 12 19:52:06.853646 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.853609 2565 generic.go:356] "Generic (PLEG): container finished" podID="d49b5336-72ba-48f3-a779-8777adad079a" containerID="fc1b70fdc12bd44828bca4fe35c365d763fefb3389abd57c8c00cb88099aeb1f" exitCode=0 Jul 12 19:52:06.853646 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.853655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" event={"ID":"d49b5336-72ba-48f3-a779-8777adad079a","Type":"ContainerDied","Data":"fc1b70fdc12bd44828bca4fe35c365d763fefb3389abd57c8c00cb88099aeb1f"} Jul 12 19:52:06.870802 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.870730 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:52:06.900049 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.900013 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm8mf\" (UniqueName: \"kubernetes.io/projected/d49b5336-72ba-48f3-a779-8777adad079a-kube-api-access-lm8mf\") pod \"d49b5336-72ba-48f3-a779-8777adad079a\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " Jul 12 19:52:06.900252 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.900127 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls\") pod \"d49b5336-72ba-48f3-a779-8777adad079a\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " Jul 12 19:52:06.900252 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.900158 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d49b5336-72ba-48f3-a779-8777adad079a-openshift-service-ca\") pod \"d49b5336-72ba-48f3-a779-8777adad079a\" (UID: \"d49b5336-72ba-48f3-a779-8777adad079a\") " Jul 12 19:52:06.900596 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.900569 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49b5336-72ba-48f3-a779-8777adad079a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d49b5336-72ba-48f3-a779-8777adad079a" (UID: "d49b5336-72ba-48f3-a779-8777adad079a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:52:06.902330 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.902298 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49b5336-72ba-48f3-a779-8777adad079a-kube-api-access-lm8mf" (OuterVolumeSpecName: "kube-api-access-lm8mf") pod "d49b5336-72ba-48f3-a779-8777adad079a" (UID: "d49b5336-72ba-48f3-a779-8777adad079a"). InnerVolumeSpecName "kube-api-access-lm8mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:52:06.902445 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:06.902339 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d49b5336-72ba-48f3-a779-8777adad079a" (UID: "d49b5336-72ba-48f3-a779-8777adad079a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:52:07.000711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.000675 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8mf\" (UniqueName: \"kubernetes.io/projected/d49b5336-72ba-48f3-a779-8777adad079a-kube-api-access-lm8mf\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:52:07.000711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.000705 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d49b5336-72ba-48f3-a779-8777adad079a-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:52:07.000711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.000715 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d49b5336-72ba-48f3-a779-8777adad079a-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:52:07.861482 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.861450 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" Jul 12 19:52:07.862160 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.861477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k" event={"ID":"d49b5336-72ba-48f3-a779-8777adad079a","Type":"ContainerDied","Data":"277b1a2d5a591cd2cef7e3c3a831952401b53416a58c4c02aeb60dc663d47025"} Jul 12 19:52:07.862293 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.862237 2565 scope.go:117] "RemoveContainer" containerID="fc1b70fdc12bd44828bca4fe35c365d763fefb3389abd57c8c00cb88099aeb1f" Jul 12 19:52:07.899417 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.899383 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k"] Jul 12 19:52:07.912080 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:07.912043 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7c507d-54f8f999-s5m2k"] Jul 12 19:52:08.055886 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:08.055849 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49b5336-72ba-48f3-a779-8777adad079a" path="/var/lib/kubelet/pods/d49b5336-72ba-48f3-a779-8777adad079a/volumes" Jul 12 19:52:10.962980 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:10.962929 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:52:34.865301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:34.865262 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8"] Jul 12 19:52:34.865819 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:34.865804 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49b5336-72ba-48f3-a779-8777adad079a" containerName="maas-api" Jul 12 19:52:34.865865 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:34.865822 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49b5336-72ba-48f3-a779-8777adad079a" containerName="maas-api" Jul 12 19:52:34.865922 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:34.865911 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49b5336-72ba-48f3-a779-8777adad079a" containerName="maas-api" Jul 12 19:52:34.869287 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:34.869260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:34.871681 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:34.871651 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-81bf5d18-serving-cert\"" Jul 12 19:52:34.880133 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:34.880101 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8"] Jul 12 19:52:35.063758 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.063721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72vq\" (UniqueName: \"kubernetes.io/projected/daaaea67-e449-49d7-97bf-3fdd68505ca5-kube-api-access-w72vq\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.063995 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.063786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daaaea67-e449-49d7-97bf-3fdd68505ca5-maas-api-tls\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.063995 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.063865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaaea67-e449-49d7-97bf-3fdd68505ca5-openshift-service-ca\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.164628 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.164522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72vq\" (UniqueName: \"kubernetes.io/projected/daaaea67-e449-49d7-97bf-3fdd68505ca5-kube-api-access-w72vq\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.164628 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.164581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daaaea67-e449-49d7-97bf-3fdd68505ca5-maas-api-tls\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.164873 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.164641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaaea67-e449-49d7-97bf-3fdd68505ca5-openshift-service-ca\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.165410 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.165381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaaea67-e449-49d7-97bf-3fdd68505ca5-openshift-service-ca\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.167172 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.167152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daaaea67-e449-49d7-97bf-3fdd68505ca5-maas-api-tls\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.173387 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.173351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72vq\" (UniqueName: \"kubernetes.io/projected/daaaea67-e449-49d7-97bf-3fdd68505ca5-kube-api-access-w72vq\") pod \"maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.182155 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.182125 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:35.334126 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:35.334099 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8"] Jul 12 19:52:35.336946 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:52:35.336919 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaaaea67_e449_49d7_97bf_3fdd68505ca5.slice/crio-c707b95f7220e24107b72f6de02699caacd0e3b8ed0d6366c4ac4fba17b0d239 WatchSource:0}: Error finding container c707b95f7220e24107b72f6de02699caacd0e3b8ed0d6366c4ac4fba17b0d239: Status 404 returned error can't find the container with id c707b95f7220e24107b72f6de02699caacd0e3b8ed0d6366c4ac4fba17b0d239 Jul 12 19:52:36.008812 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:36.008710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" event={"ID":"daaaea67-e449-49d7-97bf-3fdd68505ca5","Type":"ContainerStarted","Data":"522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9"} Jul 12 19:52:36.008812 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:36.008770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" event={"ID":"daaaea67-e449-49d7-97bf-3fdd68505ca5","Type":"ContainerStarted","Data":"c707b95f7220e24107b72f6de02699caacd0e3b8ed0d6366c4ac4fba17b0d239"} Jul 12 19:52:36.008812 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:36.008800 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:36.025269 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:36.025213 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" podStartSLOduration=1.69413027 podStartE2EDuration="2.025193635s" totalImagesPullingTime="331.063365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:52:36.023510358 +0000 UTC m=+1954.446258210" watchObservedRunningTime="2026-07-12 19:52:36.025193635 +0000 UTC m=+1954.447941486" Jul 12 19:52:37.465238 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:37.465195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:52:42.019603 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:42.019571 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:52:46.451186 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.451151 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh"] Jul 12 19:52:46.455091 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.455067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.458740 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.458712 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-81bf5d18\"/\"openshift-service-ca.crt\"" Jul 12 19:52:46.458888 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.458740 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-81bf5d18\"/\"default-dockercfg-68snf\"" Jul 12 19:52:46.458888 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.458757 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-81bf5d18\"/\"kube-root-ca.crt\"" Jul 12 19:52:46.459146 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.459107 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-81bf5d18\"/\"e2e-lifecycle-model-81bf5d18-kserve-self-signed-certs\"" Jul 12 19:52:46.464985 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.464938 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh"] Jul 12 19:52:46.572409 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.572374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-model-cache\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.572591 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.572443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-dshm\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.572591 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.572467 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-home\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.572591 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.572483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tmp-dir\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.572591 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.572520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tls-certs\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.572591 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.572549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htc58\" (UniqueName: \"kubernetes.io/projected/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-kube-api-access-htc58\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.673737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.673700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-dshm\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.673737 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.673734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-home\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.673990 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.673754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tmp-dir\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.673990 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.673785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tls-certs\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.673990 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.673835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htc58\" (UniqueName: \"kubernetes.io/projected/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-kube-api-access-htc58\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.673990 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.673905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-model-cache\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.674279 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.674251 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-home\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.674279 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.674273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tmp-dir\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.674493 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.674367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-model-cache\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.676320 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.676296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-dshm\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.676446 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.676430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tls-certs\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.681268 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.681248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htc58\" (UniqueName: \"kubernetes.io/projected/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-kube-api-access-htc58\") pod \"e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.766551 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.766465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:46.901635 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:46.901610 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh"] Jul 12 19:52:46.903602 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:52:46.903573 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb325bcd6_15ae_4f63_8ff3_47a5c55c958e.slice/crio-0b3bda7db7f310bd8b9c5a4b72753e880d6c703472992e57923173b4aabbb7d8 WatchSource:0}: Error finding container 0b3bda7db7f310bd8b9c5a4b72753e880d6c703472992e57923173b4aabbb7d8: Status 404 returned error can't find the container with id 0b3bda7db7f310bd8b9c5a4b72753e880d6c703472992e57923173b4aabbb7d8 Jul 12 19:52:47.064947 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:47.064852 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" event={"ID":"b325bcd6-15ae-4f63-8ff3-47a5c55c958e","Type":"ContainerStarted","Data":"d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9"} Jul 12 19:52:47.064947 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:47.064904 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" event={"ID":"b325bcd6-15ae-4f63-8ff3-47a5c55c958e","Type":"ContainerStarted","Data":"0b3bda7db7f310bd8b9c5a4b72753e880d6c703472992e57923173b4aabbb7d8"} Jul 12 19:52:47.081345 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:47.081293 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" podStartSLOduration=1.081277913 podStartE2EDuration="1.081277913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:52:47.079218868 +0000 UTC m=+1965.501966719" watchObservedRunningTime="2026-07-12 19:52:47.081277913 +0000 UTC m=+1965.504025763" Jul 12 19:52:51.762887 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:51.762843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:52:56.767176 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:56.767075 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:56.767176 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:56.767141 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:56.780209 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:56.780175 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:52:57.125489 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:52:57.125458 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:53:06.065174 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:06.065136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:53:11.940090 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:11.940053 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8"] Jul 12 19:53:11.940572 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:11.940315 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" podUID="daaaea67-e449-49d7-97bf-3fdd68505ca5" containerName="maas-api" containerID="cri-o://522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9" gracePeriod=30 Jul 12 19:53:12.014828 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.014781 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" podUID="daaaea67-e449-49d7-97bf-3fdd68505ca5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": dial tcp 10.134.0.44:8443: connect: connection refused" Jul 12 19:53:12.188552 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.188522 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:53:12.191279 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.191202 2565 generic.go:356] "Generic (PLEG): container finished" podID="daaaea67-e449-49d7-97bf-3fdd68505ca5" containerID="522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9" exitCode=0 Jul 12 19:53:12.191279 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.191248 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" Jul 12 19:53:12.191279 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.191267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" event={"ID":"daaaea67-e449-49d7-97bf-3fdd68505ca5","Type":"ContainerDied","Data":"522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9"} Jul 12 19:53:12.191460 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.191312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8" event={"ID":"daaaea67-e449-49d7-97bf-3fdd68505ca5","Type":"ContainerDied","Data":"c707b95f7220e24107b72f6de02699caacd0e3b8ed0d6366c4ac4fba17b0d239"} Jul 12 19:53:12.191460 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.191333 2565 scope.go:117] "RemoveContainer" containerID="522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9" Jul 12 19:53:12.200218 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.200196 2565 scope.go:117] "RemoveContainer" containerID="522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9" Jul 12 19:53:12.200536 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:53:12.200518 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9\": container with ID starting with 522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9 not found: ID does not exist" containerID="522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9" Jul 12 19:53:12.200580 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.200549 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9"} err="failed to get container status \"522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9\": rpc error: code = NotFound desc = could not find container \"522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9\": container with ID starting with 522df9a7485b1b57e74333fdff26f075dfd6c20dd31706694f80be19af5feaf9 not found: ID does not exist" Jul 12 19:53:12.225518 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.225472 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daaaea67-e449-49d7-97bf-3fdd68505ca5-maas-api-tls\") pod \"daaaea67-e449-49d7-97bf-3fdd68505ca5\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " Jul 12 19:53:12.225725 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.225557 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w72vq\" (UniqueName: \"kubernetes.io/projected/daaaea67-e449-49d7-97bf-3fdd68505ca5-kube-api-access-w72vq\") pod \"daaaea67-e449-49d7-97bf-3fdd68505ca5\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " Jul 12 19:53:12.225725 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.225578 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaaea67-e449-49d7-97bf-3fdd68505ca5-openshift-service-ca\") pod \"daaaea67-e449-49d7-97bf-3fdd68505ca5\" (UID: \"daaaea67-e449-49d7-97bf-3fdd68505ca5\") " Jul 12 19:53:12.226037 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.226013 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daaaea67-e449-49d7-97bf-3fdd68505ca5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "daaaea67-e449-49d7-97bf-3fdd68505ca5" (UID: "daaaea67-e449-49d7-97bf-3fdd68505ca5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:53:12.227744 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.227709 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daaaea67-e449-49d7-97bf-3fdd68505ca5-kube-api-access-w72vq" (OuterVolumeSpecName: "kube-api-access-w72vq") pod "daaaea67-e449-49d7-97bf-3fdd68505ca5" (UID: "daaaea67-e449-49d7-97bf-3fdd68505ca5"). InnerVolumeSpecName "kube-api-access-w72vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:53:12.227838 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.227735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daaaea67-e449-49d7-97bf-3fdd68505ca5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "daaaea67-e449-49d7-97bf-3fdd68505ca5" (UID: "daaaea67-e449-49d7-97bf-3fdd68505ca5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:53:12.326161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.326117 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daaaea67-e449-49d7-97bf-3fdd68505ca5-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:12.326161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.326151 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72vq\" (UniqueName: \"kubernetes.io/projected/daaaea67-e449-49d7-97bf-3fdd68505ca5-kube-api-access-w72vq\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:12.326161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.326160 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaaea67-e449-49d7-97bf-3fdd68505ca5-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:12.519935 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.519898 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8"] Jul 12 19:53:12.523907 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:12.523875 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81bf5d18-547ccf698b-mvwx8"] Jul 12 19:53:14.056506 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:14.056470 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daaaea67-e449-49d7-97bf-3fdd68505ca5" path="/var/lib/kubelet/pods/daaaea67-e449-49d7-97bf-3fdd68505ca5/volumes" Jul 12 19:53:16.367337 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:16.367294 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:53:22.636925 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:22.636888 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh"] Jul 12 19:53:26.376916 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:26.376876 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:53:37.658773 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.658701 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" podUID="b325bcd6-15ae-4f63-8ff3-47a5c55c958e" containerName="main" containerID="cri-o://d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9" gracePeriod=45 Jul 12 19:53:37.912094 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.912021 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:53:37.944407 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944373 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-model-cache\") pod \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " Jul 12 19:53:37.944407 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944410 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-dshm\") pod \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " Jul 12 19:53:37.944660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944428 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-home\") pod \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " Jul 12 19:53:37.944660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944455 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htc58\" (UniqueName: \"kubernetes.io/projected/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-kube-api-access-htc58\") pod \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " Jul 12 19:53:37.944660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944534 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tmp-dir\") pod \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " Jul 12 19:53:37.944660 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944565 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tls-certs\") pod \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\" (UID: \"b325bcd6-15ae-4f63-8ff3-47a5c55c958e\") " Jul 12 19:53:37.944880 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944712 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b325bcd6-15ae-4f63-8ff3-47a5c55c958e" (UID: "b325bcd6-15ae-4f63-8ff3-47a5c55c958e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:37.945253 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944752 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-home" (OuterVolumeSpecName: "home") pod "b325bcd6-15ae-4f63-8ff3-47a5c55c958e" (UID: "b325bcd6-15ae-4f63-8ff3-47a5c55c958e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:37.945253 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.944810 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b325bcd6-15ae-4f63-8ff3-47a5c55c958e" (UID: "b325bcd6-15ae-4f63-8ff3-47a5c55c958e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:37.945253 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.945188 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tmp-dir\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:37.945253 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.945209 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-model-cache\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:37.945253 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.945222 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-home\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:37.947018 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.946936 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b325bcd6-15ae-4f63-8ff3-47a5c55c958e" (UID: "b325bcd6-15ae-4f63-8ff3-47a5c55c958e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:53:37.947134 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.947030 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-dshm" (OuterVolumeSpecName: "dshm") pod "b325bcd6-15ae-4f63-8ff3-47a5c55c958e" (UID: "b325bcd6-15ae-4f63-8ff3-47a5c55c958e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:53:37.947321 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:37.947303 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-kube-api-access-htc58" (OuterVolumeSpecName: "kube-api-access-htc58") pod "b325bcd6-15ae-4f63-8ff3-47a5c55c958e" (UID: "b325bcd6-15ae-4f63-8ff3-47a5c55c958e"). InnerVolumeSpecName "kube-api-access-htc58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:53:38.046375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.046334 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-dshm\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:38.046375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.046369 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htc58\" (UniqueName: \"kubernetes.io/projected/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-kube-api-access-htc58\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:38.046375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.046381 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b325bcd6-15ae-4f63-8ff3-47a5c55c958e-tls-certs\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:53:38.328114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.328012 2565 generic.go:356] "Generic (PLEG): container finished" podID="b325bcd6-15ae-4f63-8ff3-47a5c55c958e" containerID="d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9" exitCode=0 Jul 12 19:53:38.328114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.328058 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" Jul 12 19:53:38.328114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.328097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" event={"ID":"b325bcd6-15ae-4f63-8ff3-47a5c55c958e","Type":"ContainerDied","Data":"d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9"} Jul 12 19:53:38.328407 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.328155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh" event={"ID":"b325bcd6-15ae-4f63-8ff3-47a5c55c958e","Type":"ContainerDied","Data":"0b3bda7db7f310bd8b9c5a4b72753e880d6c703472992e57923173b4aabbb7d8"} Jul 12 19:53:38.328407 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.328177 2565 scope.go:117] "RemoveContainer" containerID="d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9" Jul 12 19:53:38.339990 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.339951 2565 scope.go:117] "RemoveContainer" containerID="d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9" Jul 12 19:53:38.340288 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:53:38.340268 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9\": container with ID starting with d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9 not found: ID does not exist" containerID="d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9" Jul 12 19:53:38.340336 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.340298 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9"} err="failed to get container status \"d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9\": rpc error: code = NotFound desc = could not find container \"d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9\": container with ID starting with d482446d123ee28653fd7a9320e23a3d79711823d74edc03f54fbb788a65d1a9 not found: ID does not exist" Jul 12 19:53:38.343326 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.343300 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh"] Jul 12 19:53:38.346477 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:38.346454 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-81bf5d18/e2e-lifecycle-model-81bf5d18-kserve-6995674fd7-lt2gh"] Jul 12 19:53:40.056493 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:53:40.056460 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b325bcd6-15ae-4f63-8ff3-47a5c55c958e" path="/var/lib/kubelet/pods/b325bcd6-15ae-4f63-8ff3-47a5c55c958e/volumes" Jul 12 19:54:02.768362 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:54:02.768314 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:54:23.368707 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:54:23.368623 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:54:37.364919 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:54:37.364877 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:55:02.086612 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:02.086577 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:55:02.090788 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:02.090763 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:55:02.094080 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:02.094047 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 19:55:02.097755 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:02.097730 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 19:55:15.705987 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.705929 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw"] Jul 12 19:55:15.706494 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.706321 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b325bcd6-15ae-4f63-8ff3-47a5c55c958e" containerName="main" Jul 12 19:55:15.706494 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.706334 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b325bcd6-15ae-4f63-8ff3-47a5c55c958e" containerName="main" Jul 12 19:55:15.706494 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.706356 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daaaea67-e449-49d7-97bf-3fdd68505ca5" containerName="maas-api" Jul 12 19:55:15.706494 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.706362 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="daaaea67-e449-49d7-97bf-3fdd68505ca5" containerName="maas-api" Jul 12 19:55:15.706494 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.706428 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b325bcd6-15ae-4f63-8ff3-47a5c55c958e" containerName="main" Jul 12 19:55:15.706494 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.706436 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="daaaea67-e449-49d7-97bf-3fdd68505ca5" containerName="maas-api" Jul 12 19:55:15.709739 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.709714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.712291 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.712266 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e5b064-serving-cert\"" Jul 12 19:55:15.721072 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.721045 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw"] Jul 12 19:55:15.808114 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.808069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.808349 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.808141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.808349 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.808167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvtx7\" (UniqueName: \"kubernetes.io/projected/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-kube-api-access-qvtx7\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.908972 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.908920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.909171 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.909004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.909171 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.909027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtx7\" (UniqueName: \"kubernetes.io/projected/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-kube-api-access-qvtx7\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.909700 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.909674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.911527 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.911506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:15.924732 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:15.924699 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvtx7\" (UniqueName: \"kubernetes.io/projected/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-kube-api-access-qvtx7\") pod \"maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:16.022870 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:16.022778 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:16.170299 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:16.170264 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw"] Jul 12 19:55:16.173166 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:55:16.173137 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7010e8_38e7_4211_a36e_a972c5d4d8d8.slice/crio-4c285a4db8ef9f471cb39904ee84591ea4a03746dd6d1b573831b33e4dc71102 WatchSource:0}: Error finding container 4c285a4db8ef9f471cb39904ee84591ea4a03746dd6d1b573831b33e4dc71102: Status 404 returned error can't find the container with id 4c285a4db8ef9f471cb39904ee84591ea4a03746dd6d1b573831b33e4dc71102 Jul 12 19:55:16.174545 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:16.174525 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:55:16.817412 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:16.817371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" event={"ID":"2b7010e8-38e7-4211-a36e-a972c5d4d8d8","Type":"ContainerStarted","Data":"09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231"} Jul 12 19:55:16.817816 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:16.817436 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" event={"ID":"2b7010e8-38e7-4211-a36e-a972c5d4d8d8","Type":"ContainerStarted","Data":"4c285a4db8ef9f471cb39904ee84591ea4a03746dd6d1b573831b33e4dc71102"} Jul 12 19:55:16.817816 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:16.817589 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:16.833667 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:16.833553 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" podStartSLOduration=1.42879998 podStartE2EDuration="1.833529975s" totalImagesPullingTime="404.729995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:55:16.833429373 +0000 UTC m=+2115.256177224" watchObservedRunningTime="2026-07-12 19:55:16.833529975 +0000 UTC m=+2115.256277827" Jul 12 19:55:22.829155 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:22.829117 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:55:22.965792 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:22.965758 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:55:26.553586 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.553542 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568"] Jul 12 19:55:26.557807 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.557780 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.560300 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.560275 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1f14a4-serving-cert\"" Jul 12 19:55:26.568227 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.568199 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568"] Jul 12 19:55:26.604733 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.604697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7tn\" (UniqueName: \"kubernetes.io/projected/8898e386-b1e7-495c-81e7-e8b67fcb336e-kube-api-access-4w7tn\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.604921 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.604814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8898e386-b1e7-495c-81e7-e8b67fcb336e-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.604921 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.604874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8898e386-b1e7-495c-81e7-e8b67fcb336e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.705450 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.705410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7tn\" (UniqueName: \"kubernetes.io/projected/8898e386-b1e7-495c-81e7-e8b67fcb336e-kube-api-access-4w7tn\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.705626 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.705492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8898e386-b1e7-495c-81e7-e8b67fcb336e-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.705626 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.705517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8898e386-b1e7-495c-81e7-e8b67fcb336e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.706313 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.706284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8898e386-b1e7-495c-81e7-e8b67fcb336e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.708120 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.708102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8898e386-b1e7-495c-81e7-e8b67fcb336e-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.713544 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.713506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7tn\" (UniqueName: \"kubernetes.io/projected/8898e386-b1e7-495c-81e7-e8b67fcb336e-kube-api-access-4w7tn\") pod \"maas-api-e2e-inf-b-1f14a4-6d65964757-lk568\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:26.872479 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:26.872433 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:27.013620 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:27.013592 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568"] Jul 12 19:55:27.015843 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:55:27.015815 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8898e386_b1e7_495c_81e7_e8b67fcb336e.slice/crio-7b8ec87ed0e04e60cb77458b4d3f64518657ce2cc3ad3e93998b8478c0dcc28f WatchSource:0}: Error finding container 7b8ec87ed0e04e60cb77458b4d3f64518657ce2cc3ad3e93998b8478c0dcc28f: Status 404 returned error can't find the container with id 7b8ec87ed0e04e60cb77458b4d3f64518657ce2cc3ad3e93998b8478c0dcc28f Jul 12 19:55:27.883631 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:27.883595 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" event={"ID":"8898e386-b1e7-495c-81e7-e8b67fcb336e","Type":"ContainerStarted","Data":"f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0"} Jul 12 19:55:27.884140 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:27.883686 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:27.884140 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:27.883707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" event={"ID":"8898e386-b1e7-495c-81e7-e8b67fcb336e","Type":"ContainerStarted","Data":"7b8ec87ed0e04e60cb77458b4d3f64518657ce2cc3ad3e93998b8478c0dcc28f"} Jul 12 19:55:27.899369 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:27.899303 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" podStartSLOduration=1.5319316170000001 podStartE2EDuration="1.899283995s" totalImagesPullingTime="367.352378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:55:27.898599125 +0000 UTC m=+2126.321346973" watchObservedRunningTime="2026-07-12 19:55:27.899283995 +0000 UTC m=+2126.322031845" Jul 12 19:55:33.893693 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:33.893659 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:55:37.362762 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:37.362720 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:55:53.169529 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:55:53.169486 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:56:04.573183 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:04.573134 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:56:16.872621 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.872581 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk"] Jul 12 19:56:16.876506 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.876477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:16.879197 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.879168 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1f14a4\"/\"test-model-1f14a4-kserve-self-signed-certs\"" Jul 12 19:56:16.880460 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.880435 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1f14a4\"/\"default-dockercfg-sp7tf\"" Jul 12 19:56:16.880600 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.880498 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1f14a4\"/\"kube-root-ca.crt\"" Jul 12 19:56:16.880600 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.880498 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1f14a4\"/\"openshift-service-ca.crt\"" Jul 12 19:56:16.893762 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.890117 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk"] Jul 12 19:56:16.957481 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.957438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528bq\" (UniqueName: \"kubernetes.io/projected/ada246df-2baf-4862-badc-45160b72ff30-kube-api-access-528bq\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:16.957661 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.957553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-tmp-dir\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:16.957661 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.957585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-dshm\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:16.957741 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.957661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-model-cache\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:16.957741 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.957691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada246df-2baf-4862-badc-45160b72ff30-tls-certs\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:16.957741 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:16.957726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-home\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059013 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.058935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528bq\" (UniqueName: \"kubernetes.io/projected/ada246df-2baf-4862-badc-45160b72ff30-kube-api-access-528bq\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-tmp-dir\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-dshm\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-model-cache\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada246df-2baf-4862-badc-45160b72ff30-tls-certs\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059208 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-home\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059617 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-tmp-dir\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059617 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059602 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-model-cache\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.059786 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.059630 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-home\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.061510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.061484 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-dshm\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.061809 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.061791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada246df-2baf-4862-badc-45160b72ff30-tls-certs\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.067749 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.067720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-528bq\" (UniqueName: \"kubernetes.io/projected/ada246df-2baf-4862-badc-45160b72ff30-kube-api-access-528bq\") pod \"test-model-1f14a4-kserve-58cc8d975f-s8qwk\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.196197 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.196089 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:17.340226 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:17.340198 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk"] Jul 12 19:56:17.343076 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:56:17.343044 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podada246df_2baf_4862_badc_45160b72ff30.slice/crio-fb923466ee3e7cc0f6ae7666ae17e5ab324ef62e666c966374abdc40f1429ee3 WatchSource:0}: Error finding container fb923466ee3e7cc0f6ae7666ae17e5ab324ef62e666c966374abdc40f1429ee3: Status 404 returned error can't find the container with id fb923466ee3e7cc0f6ae7666ae17e5ab324ef62e666c966374abdc40f1429ee3 Jul 12 19:56:18.137018 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:18.136982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" event={"ID":"ada246df-2baf-4862-badc-45160b72ff30","Type":"ContainerStarted","Data":"04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7"} Jul 12 19:56:18.137018 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:18.137023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" event={"ID":"ada246df-2baf-4862-badc-45160b72ff30","Type":"ContainerStarted","Data":"fb923466ee3e7cc0f6ae7666ae17e5ab324ef62e666c966374abdc40f1429ee3"} Jul 12 19:56:18.152817 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:18.152761 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" podStartSLOduration=2.152743506 podStartE2EDuration="2.152743506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:56:18.151658617 +0000 UTC m=+2176.574406467" watchObservedRunningTime="2026-07-12 19:56:18.152743506 +0000 UTC m=+2176.575491356" Jul 12 19:56:22.171324 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:22.171282 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:56:27.197205 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:27.197168 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:27.197639 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:27.197231 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:27.209754 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:27.209717 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:28.198154 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:28.198121 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:35.547491 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:35.547443 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk"] Jul 12 19:56:36.434978 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:36.434922 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw"] Jul 12 19:56:36.435197 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:36.435161 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" podUID="2b7010e8-38e7-4211-a36e-a972c5d4d8d8" containerName="maas-api" containerID="cri-o://09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231" gracePeriod=30 Jul 12 19:56:36.893370 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:36.893343 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:56:37.041107 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.041000 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvtx7\" (UniqueName: \"kubernetes.io/projected/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-kube-api-access-qvtx7\") pod \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " Jul 12 19:56:37.041107 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.041100 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-openshift-service-ca\") pod \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " Jul 12 19:56:37.041352 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.041146 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-maas-api-tls\") pod \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\" (UID: \"2b7010e8-38e7-4211-a36e-a972c5d4d8d8\") " Jul 12 19:56:37.041480 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.041456 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b7010e8-38e7-4211-a36e-a972c5d4d8d8" (UID: "2b7010e8-38e7-4211-a36e-a972c5d4d8d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:56:37.043343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.043316 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-kube-api-access-qvtx7" (OuterVolumeSpecName: "kube-api-access-qvtx7") pod "2b7010e8-38e7-4211-a36e-a972c5d4d8d8" (UID: "2b7010e8-38e7-4211-a36e-a972c5d4d8d8"). InnerVolumeSpecName "kube-api-access-qvtx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:37.043343 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.043333 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b7010e8-38e7-4211-a36e-a972c5d4d8d8" (UID: "2b7010e8-38e7-4211-a36e-a972c5d4d8d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:56:37.142301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.142255 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvtx7\" (UniqueName: \"kubernetes.io/projected/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-kube-api-access-qvtx7\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:37.142301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.142292 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:37.142301 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.142306 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b7010e8-38e7-4211-a36e-a972c5d4d8d8-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:37.236124 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.236084 2565 generic.go:356] "Generic (PLEG): container finished" podID="2b7010e8-38e7-4211-a36e-a972c5d4d8d8" containerID="09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231" exitCode=0 Jul 12 19:56:37.236306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.236132 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" Jul 12 19:56:37.236306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.236178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" event={"ID":"2b7010e8-38e7-4211-a36e-a972c5d4d8d8","Type":"ContainerDied","Data":"09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231"} Jul 12 19:56:37.236306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.236231 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw" event={"ID":"2b7010e8-38e7-4211-a36e-a972c5d4d8d8","Type":"ContainerDied","Data":"4c285a4db8ef9f471cb39904ee84591ea4a03746dd6d1b573831b33e4dc71102"} Jul 12 19:56:37.236306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.236254 2565 scope.go:117] "RemoveContainer" containerID="09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231" Jul 12 19:56:37.245674 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.245656 2565 scope.go:117] "RemoveContainer" containerID="09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231" Jul 12 19:56:37.246062 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:56:37.245944 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231\": container with ID starting with 09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231 not found: ID does not exist" containerID="09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231" Jul 12 19:56:37.246143 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.246073 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231"} err="failed to get container status \"09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231\": rpc error: code = NotFound desc = could not find container \"09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231\": container with ID starting with 09a454a76ca647ef8cb9522355cbcc93289ef6d265c81ee6637a26dce3906231 not found: ID does not exist" Jul 12 19:56:37.257225 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.257189 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw"] Jul 12 19:56:37.262625 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:37.262597 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5b064-599c958c47-9jvpw"] Jul 12 19:56:38.055754 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:38.055717 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7010e8-38e7-4211-a36e-a972c5d4d8d8" path="/var/lib/kubelet/pods/2b7010e8-38e7-4211-a36e-a972c5d4d8d8/volumes" Jul 12 19:56:39.567441 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:39.567402 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:56:50.569571 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.569527 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" podUID="ada246df-2baf-4862-badc-45160b72ff30" containerName="main" containerID="cri-o://04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7" gracePeriod=45 Jul 12 19:56:50.840133 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.840098 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:50.969090 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969045 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-home\") pod \"ada246df-2baf-4862-badc-45160b72ff30\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " Jul 12 19:56:50.969302 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969162 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-528bq\" (UniqueName: \"kubernetes.io/projected/ada246df-2baf-4862-badc-45160b72ff30-kube-api-access-528bq\") pod \"ada246df-2baf-4862-badc-45160b72ff30\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " Jul 12 19:56:50.969302 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969190 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-dshm\") pod \"ada246df-2baf-4862-badc-45160b72ff30\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " Jul 12 19:56:50.969302 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969233 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-tmp-dir\") pod \"ada246df-2baf-4862-badc-45160b72ff30\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " Jul 12 19:56:50.969302 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-model-cache\") pod \"ada246df-2baf-4862-badc-45160b72ff30\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " Jul 12 19:56:50.969302 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada246df-2baf-4862-badc-45160b72ff30-tls-certs\") pod \"ada246df-2baf-4862-badc-45160b72ff30\" (UID: \"ada246df-2baf-4862-badc-45160b72ff30\") " Jul 12 19:56:50.969551 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969470 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-home" (OuterVolumeSpecName: "home") pod "ada246df-2baf-4862-badc-45160b72ff30" (UID: "ada246df-2baf-4862-badc-45160b72ff30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:50.969613 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969562 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-model-cache" (OuterVolumeSpecName: "model-cache") pod "ada246df-2baf-4862-badc-45160b72ff30" (UID: "ada246df-2baf-4862-badc-45160b72ff30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:50.969672 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969607 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-home\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.969729 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.969666 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ada246df-2baf-4862-badc-45160b72ff30" (UID: "ada246df-2baf-4862-badc-45160b72ff30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:50.971744 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.971709 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada246df-2baf-4862-badc-45160b72ff30-kube-api-access-528bq" (OuterVolumeSpecName: "kube-api-access-528bq") pod "ada246df-2baf-4862-badc-45160b72ff30" (UID: "ada246df-2baf-4862-badc-45160b72ff30"). InnerVolumeSpecName "kube-api-access-528bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:50.971884 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.971752 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada246df-2baf-4862-badc-45160b72ff30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ada246df-2baf-4862-badc-45160b72ff30" (UID: "ada246df-2baf-4862-badc-45160b72ff30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:56:50.971884 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:50.971764 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-dshm" (OuterVolumeSpecName: "dshm") pod "ada246df-2baf-4862-badc-45160b72ff30" (UID: "ada246df-2baf-4862-badc-45160b72ff30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:56:51.070736 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.070694 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528bq\" (UniqueName: \"kubernetes.io/projected/ada246df-2baf-4862-badc-45160b72ff30-kube-api-access-528bq\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:51.070736 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.070728 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-dshm\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:51.070736 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.070740 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-tmp-dir\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:51.071021 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.070750 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada246df-2baf-4862-badc-45160b72ff30-model-cache\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:51.071021 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.070758 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada246df-2baf-4862-badc-45160b72ff30-tls-certs\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:56:51.308172 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.308136 2565 generic.go:356] "Generic (PLEG): container finished" podID="ada246df-2baf-4862-badc-45160b72ff30" containerID="04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7" exitCode=0 Jul 12 19:56:51.308375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.308198 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" Jul 12 19:56:51.308375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.308221 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" event={"ID":"ada246df-2baf-4862-badc-45160b72ff30","Type":"ContainerDied","Data":"04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7"} Jul 12 19:56:51.308375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.308264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk" event={"ID":"ada246df-2baf-4862-badc-45160b72ff30","Type":"ContainerDied","Data":"fb923466ee3e7cc0f6ae7666ae17e5ab324ef62e666c966374abdc40f1429ee3"} Jul 12 19:56:51.308375 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.308281 2565 scope.go:117] "RemoveContainer" containerID="04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7" Jul 12 19:56:51.317635 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.317608 2565 scope.go:117] "RemoveContainer" containerID="04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7" Jul 12 19:56:51.317945 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:56:51.317924 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7\": container with ID starting with 04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7 not found: ID does not exist" containerID="04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7" Jul 12 19:56:51.318039 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.317955 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7"} err="failed to get container status \"04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7\": rpc error: code = NotFound desc = could not find container \"04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7\": container with ID starting with 04c79e565e7ca45e4442ef0521cd25caadae9e47ae42a9ea42c31d84dd183db7 not found: ID does not exist" Jul 12 19:56:51.328483 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.328450 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk"] Jul 12 19:56:51.334197 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:51.334161 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1f14a4/test-model-1f14a4-kserve-58cc8d975f-s8qwk"] Jul 12 19:56:52.056763 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:56:52.056726 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada246df-2baf-4862-badc-45160b72ff30" path="/var/lib/kubelet/pods/ada246df-2baf-4862-badc-45160b72ff30/volumes" Jul 12 19:57:19.091180 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.091142 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568"] Jul 12 19:57:19.091763 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.091387 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" podUID="8898e386-b1e7-495c-81e7-e8b67fcb336e" containerName="maas-api" containerID="cri-o://f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0" gracePeriod=30 Jul 12 19:57:19.343138 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.343058 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:57:19.423427 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.423384 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w7tn\" (UniqueName: \"kubernetes.io/projected/8898e386-b1e7-495c-81e7-e8b67fcb336e-kube-api-access-4w7tn\") pod \"8898e386-b1e7-495c-81e7-e8b67fcb336e\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " Jul 12 19:57:19.423632 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.423440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8898e386-b1e7-495c-81e7-e8b67fcb336e-openshift-service-ca\") pod \"8898e386-b1e7-495c-81e7-e8b67fcb336e\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " Jul 12 19:57:19.423632 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.423464 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8898e386-b1e7-495c-81e7-e8b67fcb336e-maas-api-tls\") pod \"8898e386-b1e7-495c-81e7-e8b67fcb336e\" (UID: \"8898e386-b1e7-495c-81e7-e8b67fcb336e\") " Jul 12 19:57:19.423849 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.423823 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8898e386-b1e7-495c-81e7-e8b67fcb336e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8898e386-b1e7-495c-81e7-e8b67fcb336e" (UID: "8898e386-b1e7-495c-81e7-e8b67fcb336e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:57:19.425859 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.425824 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8898e386-b1e7-495c-81e7-e8b67fcb336e-kube-api-access-4w7tn" (OuterVolumeSpecName: "kube-api-access-4w7tn") pod "8898e386-b1e7-495c-81e7-e8b67fcb336e" (UID: "8898e386-b1e7-495c-81e7-e8b67fcb336e"). InnerVolumeSpecName "kube-api-access-4w7tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:19.425859 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.425840 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8898e386-b1e7-495c-81e7-e8b67fcb336e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8898e386-b1e7-495c-81e7-e8b67fcb336e" (UID: "8898e386-b1e7-495c-81e7-e8b67fcb336e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:57:19.450583 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.450550 2565 generic.go:356] "Generic (PLEG): container finished" podID="8898e386-b1e7-495c-81e7-e8b67fcb336e" containerID="f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0" exitCode=0 Jul 12 19:57:19.450750 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.450623 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" Jul 12 19:57:19.450750 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.450644 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" event={"ID":"8898e386-b1e7-495c-81e7-e8b67fcb336e","Type":"ContainerDied","Data":"f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0"} Jul 12 19:57:19.450750 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.450725 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568" event={"ID":"8898e386-b1e7-495c-81e7-e8b67fcb336e","Type":"ContainerDied","Data":"7b8ec87ed0e04e60cb77458b4d3f64518657ce2cc3ad3e93998b8478c0dcc28f"} Jul 12 19:57:19.450750 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.450749 2565 scope.go:117] "RemoveContainer" containerID="f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0" Jul 12 19:57:19.460034 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.460015 2565 scope.go:117] "RemoveContainer" containerID="f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0" Jul 12 19:57:19.460356 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:57:19.460335 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0\": container with ID starting with f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0 not found: ID does not exist" containerID="f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0" Jul 12 19:57:19.460399 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.460366 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0"} err="failed to get container status \"f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0\": rpc error: code = NotFound desc = could not find container \"f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0\": container with ID starting with f734ee96d8373cf0b859c858c888f6aa6202e4eba68802e6b091fc7af8e718e0 not found: ID does not exist" Jul 12 19:57:19.471585 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.471545 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568"] Jul 12 19:57:19.473325 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.473301 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f14a4-6d65964757-lk568"] Jul 12 19:57:19.524770 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.524724 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w7tn\" (UniqueName: \"kubernetes.io/projected/8898e386-b1e7-495c-81e7-e8b67fcb336e-kube-api-access-4w7tn\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:19.524770 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.524759 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8898e386-b1e7-495c-81e7-e8b67fcb336e-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:19.524770 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:19.524772 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8898e386-b1e7-495c-81e7-e8b67fcb336e-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:20.057072 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:20.057039 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8898e386-b1e7-495c-81e7-e8b67fcb336e" path="/var/lib/kubelet/pods/8898e386-b1e7-495c-81e7-e8b67fcb336e/volumes" Jul 12 19:57:21.361616 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:21.361574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:57:29.141759 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.141668 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4"] Jul 12 19:57:29.142288 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142269 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada246df-2baf-4862-badc-45160b72ff30" containerName="main" Jul 12 19:57:29.142338 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142292 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada246df-2baf-4862-badc-45160b72ff30" containerName="main" Jul 12 19:57:29.142338 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142319 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7010e8-38e7-4211-a36e-a972c5d4d8d8" containerName="maas-api" Jul 12 19:57:29.142338 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142329 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7010e8-38e7-4211-a36e-a972c5d4d8d8" containerName="maas-api" Jul 12 19:57:29.142448 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142347 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8898e386-b1e7-495c-81e7-e8b67fcb336e" containerName="maas-api" Jul 12 19:57:29.142448 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142358 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8898e386-b1e7-495c-81e7-e8b67fcb336e" containerName="maas-api" Jul 12 19:57:29.142511 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142461 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8898e386-b1e7-495c-81e7-e8b67fcb336e" containerName="maas-api" Jul 12 19:57:29.142511 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142474 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7010e8-38e7-4211-a36e-a972c5d4d8d8" containerName="maas-api" Jul 12 19:57:29.142511 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.142486 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada246df-2baf-4862-badc-45160b72ff30" containerName="main" Jul 12 19:57:29.147278 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.147240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.150076 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.150046 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-39b82d-serving-cert\"" Jul 12 19:57:29.155063 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.155018 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4"] Jul 12 19:57:29.319998 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.319930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58878705-092b-4e09-b332-1b70b57786ed-maas-api-tls\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.320178 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.320035 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58878705-092b-4e09-b332-1b70b57786ed-openshift-service-ca\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.320178 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.320090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n5fb\" (UniqueName: \"kubernetes.io/projected/58878705-092b-4e09-b332-1b70b57786ed-kube-api-access-7n5fb\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.421268 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.421172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58878705-092b-4e09-b332-1b70b57786ed-maas-api-tls\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.421268 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.421222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58878705-092b-4e09-b332-1b70b57786ed-openshift-service-ca\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.421268 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.421261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n5fb\" (UniqueName: \"kubernetes.io/projected/58878705-092b-4e09-b332-1b70b57786ed-kube-api-access-7n5fb\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.422013 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.421990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58878705-092b-4e09-b332-1b70b57786ed-openshift-service-ca\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.423796 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.423774 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58878705-092b-4e09-b332-1b70b57786ed-maas-api-tls\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.429858 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.429837 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n5fb\" (UniqueName: \"kubernetes.io/projected/58878705-092b-4e09-b332-1b70b57786ed-kube-api-access-7n5fb\") pod \"maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.461955 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.461909 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:29.609858 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:29.609820 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4"] Jul 12 19:57:29.613995 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:57:29.613925 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58878705_092b_4e09_b332_1b70b57786ed.slice/crio-e473fbf638726b8b1acee08f271341cae44720cc9c514fff34cb7496ec000ed3 WatchSource:0}: Error finding container e473fbf638726b8b1acee08f271341cae44720cc9c514fff34cb7496ec000ed3: Status 404 returned error can't find the container with id e473fbf638726b8b1acee08f271341cae44720cc9c514fff34cb7496ec000ed3 Jul 12 19:57:30.520750 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:30.520710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" event={"ID":"58878705-092b-4e09-b332-1b70b57786ed","Type":"ContainerStarted","Data":"65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06"} Jul 12 19:57:30.521196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:30.520792 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" event={"ID":"58878705-092b-4e09-b332-1b70b57786ed","Type":"ContainerStarted","Data":"e473fbf638726b8b1acee08f271341cae44720cc9c514fff34cb7496ec000ed3"} Jul 12 19:57:30.521196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:30.520840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:30.538188 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:30.538135 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" podStartSLOduration=1.126232302 podStartE2EDuration="1.538115409s" totalImagesPullingTime="411.883107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:30.536819953 +0000 UTC m=+2248.959567802" watchObservedRunningTime="2026-07-12 19:57:30.538115409 +0000 UTC m=+2248.960863260" Jul 12 19:57:35.019500 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:35.019463 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:57:36.533429 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:36.533399 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:37.443986 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.442353 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx"] Jul 12 19:57:37.448196 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.448152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.451944 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.451911 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5a7d9b-serving-cert\"" Jul 12 19:57:37.454409 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.454366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx"] Jul 12 19:57:37.491093 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.491056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.491277 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.491129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.491277 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.491187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6x9k\" (UniqueName: \"kubernetes.io/projected/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-kube-api-access-b6x9k\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.591698 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.591663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.592159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.591728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6x9k\" (UniqueName: \"kubernetes.io/projected/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-kube-api-access-b6x9k\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.592159 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.591763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.592159 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:57:37.591841 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-serving-cert: secret "maas-api-e2e-api-b-5a7d9b-serving-cert" not found Jul 12 19:57:37.592159 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:57:37.591928 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls podName:bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae nodeName:}" failed. No retries permitted until 2026-07-12 19:57:38.091905019 +0000 UTC m=+2256.514652848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls") pod "maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" (UID: "bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae") : secret "maas-api-e2e-api-b-5a7d9b-serving-cert" not found Jul 12 19:57:37.592487 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.592468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:37.601059 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:37.601024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6x9k\" (UniqueName: \"kubernetes.io/projected/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-kube-api-access-b6x9k\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:38.096731 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:38.096694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:38.099332 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:38.099299 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:38.363223 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:38.363127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:38.510808 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:38.510771 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx"] Jul 12 19:57:38.514286 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:57:38.514251 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3e7bb4_0db4_45a8_a1a9_a33d11c2fbae.slice/crio-917a478d03e45d9b435769c502038fb9b7b0a328d4a9f7af7e8815275a926222 WatchSource:0}: Error finding container 917a478d03e45d9b435769c502038fb9b7b0a328d4a9f7af7e8815275a926222: Status 404 returned error can't find the container with id 917a478d03e45d9b435769c502038fb9b7b0a328d4a9f7af7e8815275a926222 Jul 12 19:57:38.563248 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:38.563214 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" event={"ID":"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae","Type":"ContainerStarted","Data":"917a478d03e45d9b435769c502038fb9b7b0a328d4a9f7af7e8815275a926222"} Jul 12 19:57:39.570548 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:39.570510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" event={"ID":"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae","Type":"ContainerStarted","Data":"54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c"} Jul 12 19:57:39.571026 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:39.570649 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:39.586323 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:39.586240 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" podStartSLOduration=2.202618553 podStartE2EDuration="2.586222659s" totalImagesPullingTime="383.604106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:39.585930594 +0000 UTC m=+2258.008678448" watchObservedRunningTime="2026-07-12 19:57:39.586222659 +0000 UTC m=+2258.008970511" Jul 12 19:57:45.580397 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:45.580367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:50.124622 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:50.124581 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:57:50.766136 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:50.766100 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4"] Jul 12 19:57:50.766389 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:50.766363 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" podUID="58878705-092b-4e09-b332-1b70b57786ed" containerName="maas-api" containerID="cri-o://65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06" gracePeriod=30 Jul 12 19:57:51.034016 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.033941 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:51.120059 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.120017 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58878705-092b-4e09-b332-1b70b57786ed-openshift-service-ca\") pod \"58878705-092b-4e09-b332-1b70b57786ed\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " Jul 12 19:57:51.120274 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.120100 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58878705-092b-4e09-b332-1b70b57786ed-maas-api-tls\") pod \"58878705-092b-4e09-b332-1b70b57786ed\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " Jul 12 19:57:51.120274 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.120128 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n5fb\" (UniqueName: \"kubernetes.io/projected/58878705-092b-4e09-b332-1b70b57786ed-kube-api-access-7n5fb\") pod \"58878705-092b-4e09-b332-1b70b57786ed\" (UID: \"58878705-092b-4e09-b332-1b70b57786ed\") " Jul 12 19:57:51.120443 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.120409 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58878705-092b-4e09-b332-1b70b57786ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58878705-092b-4e09-b332-1b70b57786ed" (UID: "58878705-092b-4e09-b332-1b70b57786ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:57:51.122404 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.122378 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58878705-092b-4e09-b332-1b70b57786ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58878705-092b-4e09-b332-1b70b57786ed" (UID: "58878705-092b-4e09-b332-1b70b57786ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:57:51.122508 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.122481 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58878705-092b-4e09-b332-1b70b57786ed-kube-api-access-7n5fb" (OuterVolumeSpecName: "kube-api-access-7n5fb") pod "58878705-092b-4e09-b332-1b70b57786ed" (UID: "58878705-092b-4e09-b332-1b70b57786ed"). InnerVolumeSpecName "kube-api-access-7n5fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:51.221552 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.221504 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58878705-092b-4e09-b332-1b70b57786ed-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:51.221552 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.221542 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58878705-092b-4e09-b332-1b70b57786ed-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:51.221552 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.221553 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n5fb\" (UniqueName: \"kubernetes.io/projected/58878705-092b-4e09-b332-1b70b57786ed-kube-api-access-7n5fb\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:51.630419 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.630382 2565 generic.go:356] "Generic (PLEG): container finished" podID="58878705-092b-4e09-b332-1b70b57786ed" containerID="65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06" exitCode=0 Jul 12 19:57:51.630595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.630433 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" Jul 12 19:57:51.630595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.630473 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" event={"ID":"58878705-092b-4e09-b332-1b70b57786ed","Type":"ContainerDied","Data":"65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06"} Jul 12 19:57:51.630595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.630538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4" event={"ID":"58878705-092b-4e09-b332-1b70b57786ed","Type":"ContainerDied","Data":"e473fbf638726b8b1acee08f271341cae44720cc9c514fff34cb7496ec000ed3"} Jul 12 19:57:51.630595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.630565 2565 scope.go:117] "RemoveContainer" containerID="65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06" Jul 12 19:57:51.640096 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.640071 2565 scope.go:117] "RemoveContainer" containerID="65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06" Jul 12 19:57:51.640453 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:57:51.640432 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06\": container with ID starting with 65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06 not found: ID does not exist" containerID="65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06" Jul 12 19:57:51.640533 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.640462 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06"} err="failed to get container status \"65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06\": rpc error: code = NotFound desc = could not find container \"65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06\": container with ID starting with 65208b9f3dacfd18e57e65c068cb6bce23257bb37d2ff6bba70909cf1d421c06 not found: ID does not exist" Jul 12 19:57:51.651781 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.651749 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4"] Jul 12 19:57:51.655276 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:51.655250 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-39b82d-7d8b674cc8-2jnk4"] Jul 12 19:57:52.055922 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:52.055833 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58878705-092b-4e09-b332-1b70b57786ed" path="/var/lib/kubelet/pods/58878705-092b-4e09-b332-1b70b57786ed/volumes" Jul 12 19:57:57.349644 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.349605 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx"] Jul 12 19:57:57.350170 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.349822 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" podUID="bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" containerName="maas-api" containerID="cri-o://54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c" gracePeriod=30 Jul 12 19:57:57.599377 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.599342 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:57.669666 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.669632 2565 generic.go:356] "Generic (PLEG): container finished" podID="bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" containerID="54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c" exitCode=0 Jul 12 19:57:57.669858 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.669685 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" Jul 12 19:57:57.669858 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.669722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" event={"ID":"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae","Type":"ContainerDied","Data":"54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c"} Jul 12 19:57:57.669858 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.669783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx" event={"ID":"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae","Type":"ContainerDied","Data":"917a478d03e45d9b435769c502038fb9b7b0a328d4a9f7af7e8815275a926222"} Jul 12 19:57:57.669858 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.669809 2565 scope.go:117] "RemoveContainer" containerID="54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c" Jul 12 19:57:57.678674 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.678650 2565 scope.go:117] "RemoveContainer" containerID="54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c" Jul 12 19:57:57.679079 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:57:57.679051 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c\": container with ID starting with 54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c not found: ID does not exist" containerID="54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c" Jul 12 19:57:57.679182 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.679086 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c"} err="failed to get container status \"54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c\": rpc error: code = NotFound desc = could not find container \"54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c\": container with ID starting with 54d6f2cf3b492e170704a2eeca36859ce9ee98a37635992d81848d49998e892c not found: ID does not exist" Jul 12 19:57:57.680306 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.680282 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls\") pod \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " Jul 12 19:57:57.680411 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.680330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-openshift-service-ca\") pod \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " Jul 12 19:57:57.680469 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.680443 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6x9k\" (UniqueName: \"kubernetes.io/projected/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-kube-api-access-b6x9k\") pod \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\" (UID: \"bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae\") " Jul 12 19:57:57.680711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.680685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" (UID: "bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:57:57.682637 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.682610 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" (UID: "bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:57:57.682763 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.682679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-kube-api-access-b6x9k" (OuterVolumeSpecName: "kube-api-access-b6x9k") pod "bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" (UID: "bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae"). InnerVolumeSpecName "kube-api-access-b6x9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:57.781161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.781120 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6x9k\" (UniqueName: \"kubernetes.io/projected/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-kube-api-access-b6x9k\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:57.781161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.781156 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:57.781161 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.781169 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 19:57:57.990221 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.990181 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx"] Jul 12 19:57:57.995041 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:57.995003 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7d9b-c4b9c9df-6zdkx"] Jul 12 19:57:58.056868 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:57:58.056830 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" path="/var/lib/kubelet/pods/bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae/volumes" Jul 12 19:58:02.368537 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:02.368487 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:58:07.343095 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.343001 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt"] Jul 12 19:58:07.344396 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.344372 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58878705-092b-4e09-b332-1b70b57786ed" containerName="maas-api" Jul 12 19:58:07.344539 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.344527 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58878705-092b-4e09-b332-1b70b57786ed" containerName="maas-api" Jul 12 19:58:07.344688 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.344678 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" containerName="maas-api" Jul 12 19:58:07.344761 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.344752 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" containerName="maas-api" Jul 12 19:58:07.345053 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.345039 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="58878705-092b-4e09-b332-1b70b57786ed" containerName="maas-api" Jul 12 19:58:07.345151 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.345141 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf3e7bb4-0db4-45a8-a1a9-a33d11c2fbae" containerName="maas-api" Jul 12 19:58:07.349595 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.349564 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.353346 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.353312 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-dc7943-serving-cert\"" Jul 12 19:58:07.356184 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.356157 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt"] Jul 12 19:58:07.466997 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.466930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e542077b-227d-4d9b-b01b-e9af57bcba72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.466997 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.466995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2s6\" (UniqueName: \"kubernetes.io/projected/e542077b-227d-4d9b-b01b-e9af57bcba72-kube-api-access-sg2s6\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.467227 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.467089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.567943 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.567896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e542077b-227d-4d9b-b01b-e9af57bcba72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.567943 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.567940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2s6\" (UniqueName: \"kubernetes.io/projected/e542077b-227d-4d9b-b01b-e9af57bcba72-kube-api-access-sg2s6\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.568239 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.568014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.568239 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:58:07.568156 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-serving-cert: secret "maas-api-e2e-shared-a-dc7943-serving-cert" not found Jul 12 19:58:07.568239 ip-10-0-129-169 kubenswrapper[2565]: E0712 19:58:07.568215 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls podName:e542077b-227d-4d9b-b01b-e9af57bcba72 nodeName:}" failed. No retries permitted until 2026-07-12 19:58:08.068198276 +0000 UTC m=+2286.490946104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls") pod "maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" (UID: "e542077b-227d-4d9b-b01b-e9af57bcba72") : secret "maas-api-e2e-shared-a-dc7943-serving-cert" not found Jul 12 19:58:07.568610 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.568586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e542077b-227d-4d9b-b01b-e9af57bcba72-openshift-service-ca\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:07.585923 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:07.585889 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2s6\" (UniqueName: \"kubernetes.io/projected/e542077b-227d-4d9b-b01b-e9af57bcba72-kube-api-access-sg2s6\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:08.072945 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:08.072900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:08.075668 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:08.075635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls\") pod \"maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:08.263439 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:08.263398 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:08.407718 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:08.407638 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt"] Jul 12 19:58:08.410912 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:58:08.410872 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode542077b_227d_4d9b_b01b_e9af57bcba72.slice/crio-4f8adf50a6f3561f151c72eb70e5ceb13c25b05299bdcca8edb4dce48aa775bd WatchSource:0}: Error finding container 4f8adf50a6f3561f151c72eb70e5ceb13c25b05299bdcca8edb4dce48aa775bd: Status 404 returned error can't find the container with id 4f8adf50a6f3561f151c72eb70e5ceb13c25b05299bdcca8edb4dce48aa775bd Jul 12 19:58:08.732615 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:08.732582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" event={"ID":"e542077b-227d-4d9b-b01b-e9af57bcba72","Type":"ContainerStarted","Data":"4f8adf50a6f3561f151c72eb70e5ceb13c25b05299bdcca8edb4dce48aa775bd"} Jul 12 19:58:09.739783 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:09.739732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" event={"ID":"e542077b-227d-4d9b-b01b-e9af57bcba72","Type":"ContainerStarted","Data":"b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee"} Jul 12 19:58:09.740213 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:09.739854 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:09.756081 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:09.756024 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" podStartSLOduration=2.4667505370000002 podStartE2EDuration="2.756006486s" totalImagesPullingTime="289.255949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:58:09.754505286 +0000 UTC m=+2288.177253138" watchObservedRunningTime="2026-07-12 19:58:09.756006486 +0000 UTC m=+2288.178754336" Jul 12 19:58:14.582879 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:14.582833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 19:58:15.517444 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.517404 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt"] Jul 12 19:58:15.521305 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.521278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.523769 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.523730 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c62cfb-serving-cert\"" Jul 12 19:58:15.536079 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.535346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt"] Jul 12 19:58:15.646075 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.646030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-maas-api-tls\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.646464 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.646107 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.646464 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.646155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2njc\" (UniqueName: \"kubernetes.io/projected/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-kube-api-access-w2njc\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.746678 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.746605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2njc\" (UniqueName: \"kubernetes.io/projected/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-kube-api-access-w2njc\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.746865 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.746711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-maas-api-tls\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.746865 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.746750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.747510 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.747486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.749906 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.749881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-maas-api-tls\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.752058 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.752036 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 19:58:15.754629 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.754604 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2njc\" (UniqueName: \"kubernetes.io/projected/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-kube-api-access-w2njc\") pod \"maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.836373 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.836269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:15.981798 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:15.981760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt"] Jul 12 19:58:15.985538 ip-10-0-129-169 kubenswrapper[2565]: W0712 19:58:15.985500 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce4cdf6_00b0_4ad8_94ea_c942616d9360.slice/crio-a667940b278b3a968ade9c05772b6ea4e320e6f95835bedee7931bd005bf85b9 WatchSource:0}: Error finding container a667940b278b3a968ade9c05772b6ea4e320e6f95835bedee7931bd005bf85b9: Status 404 returned error can't find the container with id a667940b278b3a968ade9c05772b6ea4e320e6f95835bedee7931bd005bf85b9 Jul 12 19:58:16.777711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:16.777615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" event={"ID":"6ce4cdf6-00b0-4ad8-94ea-c942616d9360","Type":"ContainerStarted","Data":"3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1"} Jul 12 19:58:16.777711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:16.777665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" event={"ID":"6ce4cdf6-00b0-4ad8-94ea-c942616d9360","Type":"ContainerStarted","Data":"a667940b278b3a968ade9c05772b6ea4e320e6f95835bedee7931bd005bf85b9"} Jul 12 19:58:16.777711 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:16.777697 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:16.794464 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:16.794409 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" podStartSLOduration=1.286121474 podStartE2EDuration="1.794391421s" totalImagesPullingTime="508.269947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:58:16.792777439 +0000 UTC m=+2295.215525288" watchObservedRunningTime="2026-07-12 19:58:16.794391421 +0000 UTC m=+2295.217139271" Jul 12 19:58:22.789269 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:22.789236 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 19:58:26.065688 ip-10-0-129-169 kubenswrapper[2565]: I0712 19:58:26.065642 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 20:00:00.140465 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.140426 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n"] Jul 12 20:00:00.144218 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.144190 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4"] Jul 12 20:00:00.144405 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.144324 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" Jul 12 20:00:00.147487 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.147464 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj"] Jul 12 20:00:00.147607 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.147576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" Jul 12 20:00:00.150795 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.150694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" Jul 12 20:00:00.153812 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.153573 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n"] Jul 12 20:00:00.156160 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.156129 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4"] Jul 12 20:00:00.169004 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.168941 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj"] Jul 12 20:00:00.195800 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.195757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kvc\" (UniqueName: \"kubernetes.io/projected/5973471c-16d1-4493-8cdf-306f3266d265-kube-api-access-65kvc\") pod \"maas-api-key-cleanup-29731440-srbb4\" (UID: \"5973471c-16d1-4493-8cdf-306f3266d265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" Jul 12 20:00:00.196160 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.195860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgbs6\" (UniqueName: \"kubernetes.io/projected/5c8b1893-543f-4a81-abe1-51c5126efab6-kube-api-access-bgbs6\") pod \"maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj\" (UID: \"5c8b1893-543f-4a81-abe1-51c5126efab6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" Jul 12 20:00:00.196160 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.195903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlntw\" (UniqueName: \"kubernetes.io/projected/e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c-kube-api-access-jlntw\") pod \"maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n\" (UID: \"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" Jul 12 20:00:00.296571 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.296520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kvc\" (UniqueName: \"kubernetes.io/projected/5973471c-16d1-4493-8cdf-306f3266d265-kube-api-access-65kvc\") pod \"maas-api-key-cleanup-29731440-srbb4\" (UID: \"5973471c-16d1-4493-8cdf-306f3266d265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" Jul 12 20:00:00.296763 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.296610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgbs6\" (UniqueName: \"kubernetes.io/projected/5c8b1893-543f-4a81-abe1-51c5126efab6-kube-api-access-bgbs6\") pod \"maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj\" (UID: \"5c8b1893-543f-4a81-abe1-51c5126efab6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" Jul 12 20:00:00.296763 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.296644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlntw\" (UniqueName: \"kubernetes.io/projected/e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c-kube-api-access-jlntw\") pod \"maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n\" (UID: \"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" Jul 12 20:00:00.304813 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.304782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kvc\" (UniqueName: \"kubernetes.io/projected/5973471c-16d1-4493-8cdf-306f3266d265-kube-api-access-65kvc\") pod \"maas-api-key-cleanup-29731440-srbb4\" (UID: \"5973471c-16d1-4493-8cdf-306f3266d265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" Jul 12 20:00:00.304980 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.304943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgbs6\" (UniqueName: \"kubernetes.io/projected/5c8b1893-543f-4a81-abe1-51c5126efab6-kube-api-access-bgbs6\") pod \"maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj\" (UID: \"5c8b1893-543f-4a81-abe1-51c5126efab6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" Jul 12 20:00:00.305032 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.304943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlntw\" (UniqueName: \"kubernetes.io/projected/e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c-kube-api-access-jlntw\") pod \"maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n\" (UID: \"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" Jul 12 20:00:00.458242 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.458138 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" Jul 12 20:00:00.468560 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.468523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" Jul 12 20:00:00.476728 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.476696 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" Jul 12 20:00:00.618643 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.618600 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n"] Jul 12 20:00:00.621072 ip-10-0-129-169 kubenswrapper[2565]: W0712 20:00:00.621035 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f55e3a_3e39_4ca4_8e2b_eb4fe0528a6c.slice/crio-517a8f817026df0b21943e75413bcc067590ed44d380ca46bb6c910060690bc7 WatchSource:0}: Error finding container 517a8f817026df0b21943e75413bcc067590ed44d380ca46bb6c910060690bc7: Status 404 returned error can't find the container with id 517a8f817026df0b21943e75413bcc067590ed44d380ca46bb6c910060690bc7 Jul 12 20:00:00.858893 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.858518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj"] Jul 12 20:00:00.860682 ip-10-0-129-169 kubenswrapper[2565]: W0712 20:00:00.860653 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8b1893_543f_4a81_abe1_51c5126efab6.slice/crio-0aebc1782c2e5f1f451d0240b069513d494095a6ec937b19f8e80874a4237493 WatchSource:0}: Error finding container 0aebc1782c2e5f1f451d0240b069513d494095a6ec937b19f8e80874a4237493: Status 404 returned error can't find the container with id 0aebc1782c2e5f1f451d0240b069513d494095a6ec937b19f8e80874a4237493 Jul 12 20:00:00.860793 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:00.860774 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4"] Jul 12 20:00:00.862782 ip-10-0-129-169 kubenswrapper[2565]: W0712 20:00:00.862753 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5973471c_16d1_4493_8cdf_306f3266d265.slice/crio-3c83032e2bc6939143b8e171dc7817418ff6cffc94df682db2890c13771af54e WatchSource:0}: Error finding container 3c83032e2bc6939143b8e171dc7817418ff6cffc94df682db2890c13771af54e: Status 404 returned error can't find the container with id 3c83032e2bc6939143b8e171dc7817418ff6cffc94df682db2890c13771af54e Jul 12 20:00:01.322544 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.322502 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" event={"ID":"5973471c-16d1-4493-8cdf-306f3266d265","Type":"ContainerStarted","Data":"e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab"} Jul 12 20:00:01.323016 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.322562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" event={"ID":"5973471c-16d1-4493-8cdf-306f3266d265","Type":"ContainerStarted","Data":"3c83032e2bc6939143b8e171dc7817418ff6cffc94df682db2890c13771af54e"} Jul 12 20:00:01.324367 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.324331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" event={"ID":"5c8b1893-543f-4a81-abe1-51c5126efab6","Type":"ContainerStarted","Data":"b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc"} Jul 12 20:00:01.324520 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.324392 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" event={"ID":"5c8b1893-543f-4a81-abe1-51c5126efab6","Type":"ContainerStarted","Data":"0aebc1782c2e5f1f451d0240b069513d494095a6ec937b19f8e80874a4237493"} Jul 12 20:00:01.326624 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.326596 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" event={"ID":"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c","Type":"ContainerStarted","Data":"6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7"} Jul 12 20:00:01.326749 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.326645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" event={"ID":"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c","Type":"ContainerStarted","Data":"517a8f817026df0b21943e75413bcc067590ed44d380ca46bb6c910060690bc7"} Jul 12 20:00:01.338782 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.338732 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" podStartSLOduration=1.338716878 podStartE2EDuration="1.338716878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:01.337488268 +0000 UTC m=+2399.760236119" watchObservedRunningTime="2026-07-12 20:00:01.338716878 +0000 UTC m=+2399.761464785" Jul 12 20:00:01.352172 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.352118 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" podStartSLOduration=1.352099413 podStartE2EDuration="1.352099413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:01.35150503 +0000 UTC m=+2399.774252882" watchObservedRunningTime="2026-07-12 20:00:01.352099413 +0000 UTC m=+2399.774847263" Jul 12 20:00:01.367524 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:01.367467 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" podStartSLOduration=1.367450947 podStartE2EDuration="1.367450947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:01.365618658 +0000 UTC m=+2399.788366509" watchObservedRunningTime="2026-07-12 20:00:01.367450947 +0000 UTC m=+2399.790198798" Jul 12 20:00:02.128364 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:02.128329 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 20:00:02.132599 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:02.132572 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 20:00:02.136270 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:02.136233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 20:00:02.140388 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:00:02.140361 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p5vg2_e9fba1c4-09d7-41e8-908a-6bd644526653/ovn-acl-logging/0.log" Jul 12 20:01:00.010293 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:00.010244 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj"] Jul 12 20:01:00.011285 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:00.011252 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" podUID="5c8b1893-543f-4a81-abe1-51c5126efab6" containerName="cleanup" containerID="cri-o://b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc" gracePeriod=30 Jul 12 20:01:00.012846 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:00.012824 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4"] Jul 12 20:01:00.013050 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:00.013030 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" podUID="5973471c-16d1-4493-8cdf-306f3266d265" containerName="cleanup" containerID="cri-o://e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab" gracePeriod=30 Jul 12 20:01:00.015081 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:00.015056 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n"] Jul 12 20:01:00.015288 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:00.015267 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" podUID="e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c" containerName="cleanup" containerID="cri-o://6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7" gracePeriod=30 Jul 12 20:01:25.243763 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.243722 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt"] Jul 12 20:01:25.244300 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.243997 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" podUID="e542077b-227d-4d9b-b01b-e9af57bcba72" containerName="maas-api" containerID="cri-o://b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee" gracePeriod=30 Jul 12 20:01:25.498881 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.498805 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 20:01:25.680211 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.680169 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e542077b-227d-4d9b-b01b-e9af57bcba72-openshift-service-ca\") pod \"e542077b-227d-4d9b-b01b-e9af57bcba72\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " Jul 12 20:01:25.680211 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.680213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls\") pod \"e542077b-227d-4d9b-b01b-e9af57bcba72\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " Jul 12 20:01:25.680441 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.680293 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg2s6\" (UniqueName: \"kubernetes.io/projected/e542077b-227d-4d9b-b01b-e9af57bcba72-kube-api-access-sg2s6\") pod \"e542077b-227d-4d9b-b01b-e9af57bcba72\" (UID: \"e542077b-227d-4d9b-b01b-e9af57bcba72\") " Jul 12 20:01:25.680615 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.680594 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e542077b-227d-4d9b-b01b-e9af57bcba72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e542077b-227d-4d9b-b01b-e9af57bcba72" (UID: "e542077b-227d-4d9b-b01b-e9af57bcba72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 20:01:25.682561 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.682529 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e542077b-227d-4d9b-b01b-e9af57bcba72" (UID: "e542077b-227d-4d9b-b01b-e9af57bcba72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:01:25.682677 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.682629 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e542077b-227d-4d9b-b01b-e9af57bcba72-kube-api-access-sg2s6" (OuterVolumeSpecName: "kube-api-access-sg2s6") pod "e542077b-227d-4d9b-b01b-e9af57bcba72" (UID: "e542077b-227d-4d9b-b01b-e9af57bcba72"). InnerVolumeSpecName "kube-api-access-sg2s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:01:25.759839 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.759751 2565 generic.go:356] "Generic (PLEG): container finished" podID="e542077b-227d-4d9b-b01b-e9af57bcba72" containerID="b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee" exitCode=0 Jul 12 20:01:25.759839 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.759797 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" Jul 12 20:01:25.759839 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.759816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" event={"ID":"e542077b-227d-4d9b-b01b-e9af57bcba72","Type":"ContainerDied","Data":"b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee"} Jul 12 20:01:25.760131 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.759861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt" event={"ID":"e542077b-227d-4d9b-b01b-e9af57bcba72","Type":"ContainerDied","Data":"4f8adf50a6f3561f151c72eb70e5ceb13c25b05299bdcca8edb4dce48aa775bd"} Jul 12 20:01:25.760131 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.759883 2565 scope.go:117] "RemoveContainer" containerID="b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee" Jul 12 20:01:25.769502 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.769479 2565 scope.go:117] "RemoveContainer" containerID="b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee" Jul 12 20:01:25.769812 ip-10-0-129-169 kubenswrapper[2565]: E0712 20:01:25.769791 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee\": container with ID starting with b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee not found: ID does not exist" containerID="b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee" Jul 12 20:01:25.769878 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.769822 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee"} err="failed to get container status \"b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee\": rpc error: code = NotFound desc = could not find container \"b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee\": container with ID starting with b8f99a12dfc5b5a3e81443284af1832da69f159154f066571ca178adc094a8ee not found: ID does not exist" Jul 12 20:01:25.780907 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.780870 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt"] Jul 12 20:01:25.781066 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.780988 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e542077b-227d-4d9b-b01b-e9af57bcba72-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:25.781066 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.781010 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e542077b-227d-4d9b-b01b-e9af57bcba72-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:25.781066 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.781020 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg2s6\" (UniqueName: \"kubernetes.io/projected/e542077b-227d-4d9b-b01b-e9af57bcba72-kube-api-access-sg2s6\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:25.786135 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:25.786106 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-dc7943-65699fb6b5-q5vpt"] Jul 12 20:01:26.056457 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:26.056370 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e542077b-227d-4d9b-b01b-e9af57bcba72" path="/var/lib/kubelet/pods/e542077b-227d-4d9b-b01b-e9af57bcba72/volumes" Jul 12 20:01:28.462985 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:28.462935 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 20:01:30.226543 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.226505 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" Jul 12 20:01:30.254336 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.254307 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" Jul 12 20:01:30.280151 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.280115 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" Jul 12 20:01:30.322641 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.322603 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65kvc\" (UniqueName: \"kubernetes.io/projected/5973471c-16d1-4493-8cdf-306f3266d265-kube-api-access-65kvc\") pod \"5973471c-16d1-4493-8cdf-306f3266d265\" (UID: \"5973471c-16d1-4493-8cdf-306f3266d265\") " Jul 12 20:01:30.324905 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.324869 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5973471c-16d1-4493-8cdf-306f3266d265-kube-api-access-65kvc" (OuterVolumeSpecName: "kube-api-access-65kvc") pod "5973471c-16d1-4493-8cdf-306f3266d265" (UID: "5973471c-16d1-4493-8cdf-306f3266d265"). InnerVolumeSpecName "kube-api-access-65kvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:01:30.424181 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.424130 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgbs6\" (UniqueName: \"kubernetes.io/projected/5c8b1893-543f-4a81-abe1-51c5126efab6-kube-api-access-bgbs6\") pod \"5c8b1893-543f-4a81-abe1-51c5126efab6\" (UID: \"5c8b1893-543f-4a81-abe1-51c5126efab6\") " Jul 12 20:01:30.424181 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.424183 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlntw\" (UniqueName: \"kubernetes.io/projected/e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c-kube-api-access-jlntw\") pod \"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c\" (UID: \"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c\") " Jul 12 20:01:30.424462 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.424386 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65kvc\" (UniqueName: \"kubernetes.io/projected/5973471c-16d1-4493-8cdf-306f3266d265-kube-api-access-65kvc\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:30.426464 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.426435 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c-kube-api-access-jlntw" (OuterVolumeSpecName: "kube-api-access-jlntw") pod "e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c" (UID: "e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c"). InnerVolumeSpecName "kube-api-access-jlntw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:01:30.426571 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.426473 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8b1893-543f-4a81-abe1-51c5126efab6-kube-api-access-bgbs6" (OuterVolumeSpecName: "kube-api-access-bgbs6") pod "5c8b1893-543f-4a81-abe1-51c5126efab6" (UID: "5c8b1893-543f-4a81-abe1-51c5126efab6"). InnerVolumeSpecName "kube-api-access-bgbs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:01:30.525862 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.525765 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgbs6\" (UniqueName: \"kubernetes.io/projected/5c8b1893-543f-4a81-abe1-51c5126efab6-kube-api-access-bgbs6\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:30.525862 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.525800 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlntw\" (UniqueName: \"kubernetes.io/projected/e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c-kube-api-access-jlntw\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:30.787062 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.786944 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c" containerID="6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7" exitCode=137 Jul 12 20:01:30.787062 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.787000 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" Jul 12 20:01:30.787062 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.787044 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" event={"ID":"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c","Type":"ContainerDied","Data":"6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7"} Jul 12 20:01:30.787460 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.787104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n" event={"ID":"e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c","Type":"ContainerDied","Data":"517a8f817026df0b21943e75413bcc067590ed44d380ca46bb6c910060690bc7"} Jul 12 20:01:30.787460 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.787131 2565 scope.go:117] "RemoveContainer" containerID="6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7" Jul 12 20:01:30.788410 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.788382 2565 generic.go:356] "Generic (PLEG): container finished" podID="5973471c-16d1-4493-8cdf-306f3266d265" containerID="e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab" exitCode=137 Jul 12 20:01:30.788538 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.788411 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" event={"ID":"5973471c-16d1-4493-8cdf-306f3266d265","Type":"ContainerDied","Data":"e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab"} Jul 12 20:01:30.788538 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.788437 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" Jul 12 20:01:30.788538 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.788468 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4" event={"ID":"5973471c-16d1-4493-8cdf-306f3266d265","Type":"ContainerDied","Data":"3c83032e2bc6939143b8e171dc7817418ff6cffc94df682db2890c13771af54e"} Jul 12 20:01:30.789994 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.789941 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c8b1893-543f-4a81-abe1-51c5126efab6" containerID="b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc" exitCode=137 Jul 12 20:01:30.789994 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.789986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" event={"ID":"5c8b1893-543f-4a81-abe1-51c5126efab6","Type":"ContainerDied","Data":"b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc"} Jul 12 20:01:30.790209 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.790030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" event={"ID":"5c8b1893-543f-4a81-abe1-51c5126efab6","Type":"ContainerDied","Data":"0aebc1782c2e5f1f451d0240b069513d494095a6ec937b19f8e80874a4237493"} Jul 12 20:01:30.790209 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.790069 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj" Jul 12 20:01:30.798376 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.798346 2565 scope.go:117] "RemoveContainer" containerID="6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7" Jul 12 20:01:30.798781 ip-10-0-129-169 kubenswrapper[2565]: E0712 20:01:30.798755 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7\": container with ID starting with 6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7 not found: ID does not exist" containerID="6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7" Jul 12 20:01:30.798889 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.798787 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7"} err="failed to get container status \"6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7\": rpc error: code = NotFound desc = could not find container \"6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7\": container with ID starting with 6e8d40dc62644290eb1600dbe60442ddbe4ce57c00c026f809744f72a79e68f7 not found: ID does not exist" Jul 12 20:01:30.798889 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.798806 2565 scope.go:117] "RemoveContainer" containerID="e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab" Jul 12 20:01:30.807790 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.807769 2565 scope.go:117] "RemoveContainer" containerID="e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab" Jul 12 20:01:30.808154 ip-10-0-129-169 kubenswrapper[2565]: E0712 20:01:30.808131 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab\": container with ID starting with e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab not found: ID does not exist" containerID="e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab" Jul 12 20:01:30.808234 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.808162 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab"} err="failed to get container status \"e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab\": rpc error: code = NotFound desc = could not find container \"e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab\": container with ID starting with e116015ff6c809901690a12ede6cf330fe88d7c9c8e9efa3e1d127cf8a4434ab not found: ID does not exist" Jul 12 20:01:30.808234 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.808184 2565 scope.go:117] "RemoveContainer" containerID="b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc" Jul 12 20:01:30.814393 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.814364 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n"] Jul 12 20:01:30.817056 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.816567 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-dc7943-29731440-zfv9n"] Jul 12 20:01:30.819506 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.818493 2565 scope.go:117] "RemoveContainer" containerID="b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc" Jul 12 20:01:30.819506 ip-10-0-129-169 kubenswrapper[2565]: E0712 20:01:30.819022 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc\": container with ID starting with b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc not found: ID does not exist" containerID="b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc" Jul 12 20:01:30.819506 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.819080 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc"} err="failed to get container status \"b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc\": rpc error: code = NotFound desc = could not find container \"b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc\": container with ID starting with b91e8692f56b008125fe2d4214aa5c15d8820a6441236761365d36db55d1e5fc not found: ID does not exist" Jul 12 20:01:30.825615 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.825584 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj"] Jul 12 20:01:30.831327 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.831299 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c62cfb-29731440-hv4tj"] Jul 12 20:01:30.840720 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.840686 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4"] Jul 12 20:01:30.842622 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:30.842593 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731440-srbb4"] Jul 12 20:01:32.056209 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.056170 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5973471c-16d1-4493-8cdf-306f3266d265" path="/var/lib/kubelet/pods/5973471c-16d1-4493-8cdf-306f3266d265/volumes" Jul 12 20:01:32.056616 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.056501 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8b1893-543f-4a81-abe1-51c5126efab6" path="/var/lib/kubelet/pods/5c8b1893-543f-4a81-abe1-51c5126efab6/volumes" Jul 12 20:01:32.056777 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.056763 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c" path="/var/lib/kubelet/pods/e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c/volumes" Jul 12 20:01:32.302152 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.302114 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt"] Jul 12 20:01:32.302405 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.302374 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" podUID="6ce4cdf6-00b0-4ad8-94ea-c942616d9360" containerName="maas-api" containerID="cri-o://3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1" gracePeriod=30 Jul 12 20:01:32.556939 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.556912 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 20:01:32.648806 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.648769 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-openshift-service-ca\") pod \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " Jul 12 20:01:32.649023 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.648840 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-maas-api-tls\") pod \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " Jul 12 20:01:32.649023 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.648883 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2njc\" (UniqueName: \"kubernetes.io/projected/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-kube-api-access-w2njc\") pod \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\" (UID: \"6ce4cdf6-00b0-4ad8-94ea-c942616d9360\") " Jul 12 20:01:32.649196 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.649173 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ce4cdf6-00b0-4ad8-94ea-c942616d9360" (UID: "6ce4cdf6-00b0-4ad8-94ea-c942616d9360"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 20:01:32.651144 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.651114 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-kube-api-access-w2njc" (OuterVolumeSpecName: "kube-api-access-w2njc") pod "6ce4cdf6-00b0-4ad8-94ea-c942616d9360" (UID: "6ce4cdf6-00b0-4ad8-94ea-c942616d9360"). InnerVolumeSpecName "kube-api-access-w2njc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:01:32.651274 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.651254 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ce4cdf6-00b0-4ad8-94ea-c942616d9360" (UID: "6ce4cdf6-00b0-4ad8-94ea-c942616d9360"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:01:32.750125 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.750072 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-openshift-service-ca\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:32.750125 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.750115 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-maas-api-tls\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:32.750125 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.750125 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2njc\" (UniqueName: \"kubernetes.io/projected/6ce4cdf6-00b0-4ad8-94ea-c942616d9360-kube-api-access-w2njc\") on node \"ip-10-0-129-169.ec2.internal\" DevicePath \"\"" Jul 12 20:01:32.804248 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.804211 2565 generic.go:356] "Generic (PLEG): container finished" podID="6ce4cdf6-00b0-4ad8-94ea-c942616d9360" containerID="3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1" exitCode=0 Jul 12 20:01:32.804447 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.804251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" event={"ID":"6ce4cdf6-00b0-4ad8-94ea-c942616d9360","Type":"ContainerDied","Data":"3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1"} Jul 12 20:01:32.804447 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.804313 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" event={"ID":"6ce4cdf6-00b0-4ad8-94ea-c942616d9360","Type":"ContainerDied","Data":"a667940b278b3a968ade9c05772b6ea4e320e6f95835bedee7931bd005bf85b9"} Jul 12 20:01:32.804447 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.804335 2565 scope.go:117] "RemoveContainer" containerID="3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1" Jul 12 20:01:32.804447 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.804266 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt" Jul 12 20:01:32.813816 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.813793 2565 scope.go:117] "RemoveContainer" containerID="3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1" Jul 12 20:01:32.814112 ip-10-0-129-169 kubenswrapper[2565]: E0712 20:01:32.814088 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1\": container with ID starting with 3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1 not found: ID does not exist" containerID="3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1" Jul 12 20:01:32.814197 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.814123 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1"} err="failed to get container status \"3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1\": rpc error: code = NotFound desc = could not find container \"3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1\": container with ID starting with 3726eb49e8940f92fa7fa55c1b6f6c58dd7f1f9621d865ce43e23ba4e492a7b1 not found: ID does not exist" Jul 12 20:01:32.826012 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.825978 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt"] Jul 12 20:01:32.829568 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:32.829542 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c62cfb-599c58565d-ngxjt"] Jul 12 20:01:34.056102 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:34.056067 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce4cdf6-00b0-4ad8-94ea-c942616d9360" path="/var/lib/kubelet/pods/6ce4cdf6-00b0-4ad8-94ea-c942616d9360/volumes" Jul 12 20:01:39.768230 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:39.768187 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-94q64"] Jul 12 20:01:43.569279 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:43.569246 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-s5pml_de2e3825-6753-417f-adc7-db4d85ea6e26/manager/0.log" Jul 12 20:01:43.575338 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:43.575308 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-s5pml_de2e3825-6753-417f-adc7-db4d85ea6e26/copy-manifests/0.log" Jul 12 20:01:45.178227 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:45.178189 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-x5qlg_c2032a6a-c3c0-4134-836b-be2e8467e9ef/manager/0.log" Jul 12 20:01:45.623900 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:45.623870 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-94q64_48414c6c-09ae-44ce-b9f5-848857fa82b5/limitador/0.log" Jul 12 20:01:46.185199 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:46.185164 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rwd2q_c7ad74f3-5542-4f1d-9a22-61a31173181c/discovery/0.log" Jul 12 20:01:46.863572 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:46.863539 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b6f9bdc88-xhz87_bdd55c66-838a-4927-820b-5b708820e9de/router/0.log" Jul 12 20:01:47.300927 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:47.300894 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-qbdcv_823bdb78-adb7-496b-beaf-97371ba1c254/main/0.log" Jul 12 20:01:54.955915 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:54.955826 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tp9xw_bdbd098e-624f-4544-99c7-b7a6e0ef4ae4/global-pull-secret-syncer/0.log" Jul 12 20:01:55.035550 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:55.035512 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m45zt_1950439f-375f-4039-8bae-8c4a1e666fd1/konnectivity-agent/0.log" Jul 12 20:01:55.086100 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:01:55.086063 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-169.ec2.internal_e8296a02698d224e1818c8cb03a109cf/haproxy/0.log" Jul 12 20:02:00.325255 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:00.325221 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-x5qlg_c2032a6a-c3c0-4134-836b-be2e8467e9ef/manager/0.log" Jul 12 20:02:00.598545 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:00.598464 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-94q64_48414c6c-09ae-44ce-b9f5-848857fa82b5/limitador/0.log" Jul 12 20:02:02.307872 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.307840 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fqjfz_edde6206-cd48-41df-b389-bf0ab450d3dc/kube-state-metrics/0.log" Jul 12 20:02:02.325986 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.325940 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fqjfz_edde6206-cd48-41df-b389-bf0ab450d3dc/kube-rbac-proxy-main/0.log" Jul 12 20:02:02.347060 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.347041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-fqjfz_edde6206-cd48-41df-b389-bf0ab450d3dc/kube-rbac-proxy-self/0.log" Jul 12 20:02:02.403991 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.403946 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-q8nvk_af2c6748-2039-4e58-8e30-5ebe213c10e9/monitoring-plugin/0.log" Jul 12 20:02:02.554671 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.554617 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5frl_190fdc8d-10ff-453b-8b60-103e9e4823d5/node-exporter/0.log" Jul 12 20:02:02.575580 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.575553 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5frl_190fdc8d-10ff-453b-8b60-103e9e4823d5/kube-rbac-proxy/0.log" Jul 12 20:02:02.603773 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.603738 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5frl_190fdc8d-10ff-453b-8b60-103e9e4823d5/init-textfile/0.log" Jul 12 20:02:02.702865 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.702834 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-htfzx_8145791c-f93b-41ea-9e4c-089f2775ce52/kube-rbac-proxy-main/0.log" Jul 12 20:02:02.725628 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.725599 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-htfzx_8145791c-f93b-41ea-9e4c-089f2775ce52/kube-rbac-proxy-self/0.log" Jul 12 20:02:02.747260 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.747225 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-htfzx_8145791c-f93b-41ea-9e4c-089f2775ce52/openshift-state-metrics/0.log" Jul 12 20:02:02.949001 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.948953 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xvhwm_b22f9560-c421-4a96-8581-57789c2943fe/prometheus-operator/0.log" Jul 12 20:02:02.968077 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:02.968045 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xvhwm_b22f9560-c421-4a96-8581-57789c2943fe/kube-rbac-proxy/0.log" Jul 12 20:02:03.021707 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:03.021671 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d74949d6-hc9r2_5e907c8c-112c-4dca-8e59-00a3f8c687a1/telemeter-client/0.log" Jul 12 20:02:03.042143 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:03.042114 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d74949d6-hc9r2_5e907c8c-112c-4dca-8e59-00a3f8c687a1/reload/0.log" Jul 12 20:02:03.062501 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:03.062471 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d74949d6-hc9r2_5e907c8c-112c-4dca-8e59-00a3f8c687a1/kube-rbac-proxy/0.log" Jul 12 20:02:05.028364 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.028323 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/2.log" Jul 12 20:02:05.032311 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.032292 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-r85fs_f476e372-7762-4098-a7e0-1ccca0e34c14/console-operator/3.log" Jul 12 20:02:05.373449 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373414 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc"] Jul 12 20:02:05.373795 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373782 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8b1893-543f-4a81-abe1-51c5126efab6" containerName="cleanup" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373797 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8b1893-543f-4a81-abe1-51c5126efab6" containerName="cleanup" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373812 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5973471c-16d1-4493-8cdf-306f3266d265" containerName="cleanup" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373818 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5973471c-16d1-4493-8cdf-306f3266d265" containerName="cleanup" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373829 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e542077b-227d-4d9b-b01b-e9af57bcba72" containerName="maas-api" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373834 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e542077b-227d-4d9b-b01b-e9af57bcba72" containerName="maas-api" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373843 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce4cdf6-00b0-4ad8-94ea-c942616d9360" containerName="maas-api" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373848 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce4cdf6-00b0-4ad8-94ea-c942616d9360" containerName="maas-api" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373854 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c" containerName="cleanup" Jul 12 20:02:05.373860 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373859 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c" containerName="cleanup" Jul 12 20:02:05.374382 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373914 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e542077b-227d-4d9b-b01b-e9af57bcba72" containerName="maas-api" Jul 12 20:02:05.374382 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373925 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f55e3a-3e39-4ca4-8e2b-eb4fe0528a6c" containerName="cleanup" Jul 12 20:02:05.374382 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373930 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5973471c-16d1-4493-8cdf-306f3266d265" containerName="cleanup" Jul 12 20:02:05.374382 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373941 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce4cdf6-00b0-4ad8-94ea-c942616d9360" containerName="maas-api" Jul 12 20:02:05.374382 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.373950 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8b1893-543f-4a81-abe1-51c5126efab6" containerName="cleanup" Jul 12 20:02:05.378755 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.378733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.381051 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.381032 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txwrt\"/\"kube-root-ca.crt\"" Jul 12 20:02:05.382365 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.382346 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-txwrt\"/\"openshift-service-ca.crt\"" Jul 12 20:02:05.382459 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.382353 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-txwrt\"/\"default-dockercfg-44w77\"" Jul 12 20:02:05.385482 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.385354 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc"] Jul 12 20:02:05.416803 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.416763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-lib-modules\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.416803 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.416809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-sys\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.417058 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.416834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-proc\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.417058 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.416903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-podres\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.417058 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.416937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4l9\" (UniqueName: \"kubernetes.io/projected/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-kube-api-access-ft4l9\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.517939 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.517895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-sys\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518148 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.517947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-proc\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518148 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.518021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-podres\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518148 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.518049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-proc\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518148 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.518070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4l9\" (UniqueName: \"kubernetes.io/projected/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-kube-api-access-ft4l9\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518148 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.518049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-sys\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518347 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.518174 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-podres\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518347 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.518187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-lib-modules\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.518347 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.518293 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-lib-modules\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.527569 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.527533 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4l9\" (UniqueName: \"kubernetes.io/projected/0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403-kube-api-access-ft4l9\") pod \"perf-node-gather-daemonset-fv2gc\" (UID: \"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403\") " pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.582981 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.582926 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d5d7957d8-5htd4_604b0ea5-098d-4f09-9e29-662e7c7dc326/console/0.log" Jul 12 20:02:05.612110 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.612074 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-skr8s_1b37c485-8d55-4a5f-98d7-282690473b4a/download-server/0.log" Jul 12 20:02:05.690836 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.690742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.830607 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.830580 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc"] Jul 12 20:02:05.832687 ip-10-0-129-169 kubenswrapper[2565]: W0712 20:02:05.832664 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0dcbbcb3_ddc4_4ed0_865a_0a8b08f63403.slice/crio-27e231f0fbf82e8e811c585d343deefc51382a9e767cc083f7b68cf93d47cb34 WatchSource:0}: Error finding container 27e231f0fbf82e8e811c585d343deefc51382a9e767cc083f7b68cf93d47cb34: Status 404 returned error can't find the container with id 27e231f0fbf82e8e811c585d343deefc51382a9e767cc083f7b68cf93d47cb34 Jul 12 20:02:05.977287 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.977200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" event={"ID":"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403","Type":"ContainerStarted","Data":"0b001d60ad1c2fa76d153f26a3214154a70570e1b83990a800aa9ad37448ef7c"} Jul 12 20:02:05.977287 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.977261 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" event={"ID":"0dcbbcb3-ddc4-4ed0-865a-0a8b08f63403","Type":"ContainerStarted","Data":"27e231f0fbf82e8e811c585d343deefc51382a9e767cc083f7b68cf93d47cb34"} Jul 12 20:02:05.977479 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.977335 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:05.991522 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:05.991454 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" podStartSLOduration=0.991431365 podStartE2EDuration="991.431365ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 20:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:02:05.990989534 +0000 UTC m=+2524.413737382" watchObservedRunningTime="2026-07-12 20:02:05.991431365 +0000 UTC m=+2524.414179218" Jul 12 20:02:06.176436 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:06.176409 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-wmm8m_af8ac5c6-6e0c-4fbe-8003-e07ee99de6a8/volume-data-source-validator/0.log" Jul 12 20:02:07.142772 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:07.142746 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-26zj5_38e91421-8a09-4c33-96b7-db7fd8477878/dns/0.log" Jul 12 20:02:07.164632 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:07.164571 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-26zj5_38e91421-8a09-4c33-96b7-db7fd8477878/kube-rbac-proxy/0.log" Jul 12 20:02:07.293900 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:07.293813 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s5rxz_056af27c-be62-4cc4-8cfb-0bdee24f0bc3/dns-node-resolver/0.log" Jul 12 20:02:07.887475 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:07.887438 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bfz54_91118a2a-c38a-44dd-81a9-1781479ad80f/node-ca/0.log" Jul 12 20:02:08.898710 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:08.898654 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rwd2q_c7ad74f3-5542-4f1d-9a22-61a31173181c/discovery/0.log" Jul 12 20:02:09.171572 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:09.171478 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b6f9bdc88-xhz87_bdd55c66-838a-4927-820b-5b708820e9de/router/0.log" Jul 12 20:02:09.799948 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:09.799919 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8nn86_444f5432-cc95-45b7-8f56-09c2562181d0/serve-healthcheck-canary/0.log" Jul 12 20:02:10.441341 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:10.441304 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w76w5_0ae7f2c8-8705-43a4-873f-7586448ec728/kube-rbac-proxy/0.log" Jul 12 20:02:10.461591 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:10.461563 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w76w5_0ae7f2c8-8705-43a4-873f-7586448ec728/exporter/0.log" Jul 12 20:02:10.481644 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:10.481618 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w76w5_0ae7f2c8-8705-43a4-873f-7586448ec728/extractor/0.log" Jul 12 20:02:11.992886 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:11.992858 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-txwrt/perf-node-gather-daemonset-fv2gc" Jul 12 20:02:12.986127 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:12.986085 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-s5pml_de2e3825-6753-417f-adc7-db4d85ea6e26/manager/0.log" Jul 12 20:02:13.031284 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:13.031252 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dff8c9d6-s5pml_de2e3825-6753-417f-adc7-db4d85ea6e26/copy-manifests/0.log" Jul 12 20:02:14.454262 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:14.454223 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-89775f59b-qvc57_1b0f32f0-8c78-4f6b-80dc-8a55c86fd70c/manager/0.log" Jul 12 20:02:15.715458 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:15.715417 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rwd2q_c7ad74f3-5542-4f1d-9a22-61a31173181c/discovery/0.log" Jul 12 20:02:15.943082 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:15.943043 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b6f9bdc88-xhz87_bdd55c66-838a-4927-820b-5b708820e9de/router/0.log" Jul 12 20:02:17.299952 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:17.299910 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rwd2q_c7ad74f3-5542-4f1d-9a22-61a31173181c/discovery/0.log" Jul 12 20:02:17.520589 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:17.520476 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b6f9bdc88-xhz87_bdd55c66-838a-4927-820b-5b708820e9de/router/0.log" Jul 12 20:02:20.965854 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:20.965820 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtfc4_772d0df4-415d-4630-a386-adc4c8dd0b7a/kube-multus-additional-cni-plugins/0.log" Jul 12 20:02:20.985776 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:20.985740 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtfc4_772d0df4-415d-4630-a386-adc4c8dd0b7a/egress-router-binary-copy/0.log" Jul 12 20:02:21.005408 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.005376 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtfc4_772d0df4-415d-4630-a386-adc4c8dd0b7a/cni-plugins/0.log" Jul 12 20:02:21.026706 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.026666 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtfc4_772d0df4-415d-4630-a386-adc4c8dd0b7a/bond-cni-plugin/0.log" Jul 12 20:02:21.045637 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.045603 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtfc4_772d0df4-415d-4630-a386-adc4c8dd0b7a/routeoverride-cni/0.log" Jul 12 20:02:21.066091 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.066053 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtfc4_772d0df4-415d-4630-a386-adc4c8dd0b7a/whereabouts-cni-bincopy/0.log" Jul 12 20:02:21.086109 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.086078 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wtfc4_772d0df4-415d-4630-a386-adc4c8dd0b7a/whereabouts-cni/0.log" Jul 12 20:02:21.325383 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.325284 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qkqnk_f5ac1728-30b7-40b2-83df-1ea018901a2b/kube-multus/0.log" Jul 12 20:02:21.429411 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.429367 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mn2m5_31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6/network-metrics-daemon/0.log" Jul 12 20:02:21.447240 ip-10-0-129-169 kubenswrapper[2565]: I0712 20:02:21.447206 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mn2m5_31e30d3f-8aaa-4b90-8c55-e3de6cd7f3a6/kube-rbac-proxy/0.log"