Jul 13 17:32:23.615773 ip-10-0-139-16 systemd[1]: Starting Kubernetes Kubelet... Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.309365 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315254 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315267 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315271 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315274 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315277 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315281 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315285 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 17:32:25.385515 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315288 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315292 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315296 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315298 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315302 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315305 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315308 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315310 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315313 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315316 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315319 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315321 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315324 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315327 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315329 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315332 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315334 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315337 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315339 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 17:32:25.386638 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315342 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315344 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315348 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315352 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315355 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315358 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315361 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315364 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315367 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315369 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315373 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315376 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315379 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315382 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315384 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315386 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315389 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315392 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315395 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315397 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 17:32:25.387324 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315400 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315402 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315405 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315407 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315410 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315412 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315415 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315417 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315420 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315423 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315425 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315428 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315430 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315433 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315435 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315437 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315440 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315442 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315446 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 17:32:25.388174 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315450 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315453 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315456 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315460 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315463 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315465 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315471 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315474 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315477 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315480 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315483 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315486 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315489 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315491 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315494 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315496 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315499 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315501 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315504 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315506 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 17:32:25.388848 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315509 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315511 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315514 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315516 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315519 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315521 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315524 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315526 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315528 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315531 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315533 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315536 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315538 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.315541 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318176 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318184 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318188 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318191 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318210 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318215 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 17:32:25.389545 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318220 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318223 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318228 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318253 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318255 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318258 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318261 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318263 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318266 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318268 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318271 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318273 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318276 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318279 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318281 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318284 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318286 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318289 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318291 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318293 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 17:32:25.390093 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318296 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318298 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318301 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318304 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318306 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318309 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318312 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318318 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318321 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318324 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318327 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318330 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318334 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318337 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318340 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318342 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318345 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318348 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318351 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318354 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 17:32:25.390748 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318357 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318360 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318362 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318365 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318367 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318370 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318372 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318375 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318378 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318380 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318383 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318385 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318388 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318391 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318393 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318396 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318400 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318404 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318406 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 17:32:25.391648 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318409 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 17:32:25.392228 ip-10-0-139-16 systemd[1]: Started Kubernetes Kubelet. Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318411 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318414 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318416 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318419 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318421 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318424 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318427 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318430 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318433 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318437 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318439 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318442 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318445 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318448 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318450 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318453 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318455 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318458 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 17:32:25.392597 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318461 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318463 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318466 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318469 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318471 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318474 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318476 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318479 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318481 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318484 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318486 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318489 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318491 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318494 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.318497 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319000 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319006 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319010 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319013 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319016 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 17:32:25.393675 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319019 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319022 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319025 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319028 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319030 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319033 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319036 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319038 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319041 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319044 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319047 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319049 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319052 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319054 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319057 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319059 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319062 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319065 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319068 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319070 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 17:32:25.396016 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319074 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319077 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319080 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319082 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319085 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319088 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319091 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319094 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319096 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319099 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319101 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319104 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319107 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319109 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319112 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319115 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319117 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319120 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319122 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319125 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 17:32:25.397071 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319127 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319131 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319133 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319136 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319138 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319141 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319143 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319146 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319148 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319151 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319153 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319156 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319158 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319161 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319163 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319166 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319169 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319171 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319175 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319178 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 17:32:25.398565 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319181 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319183 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319186 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319188 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319191 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319194 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319196 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319199 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319201 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319204 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319208 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319210 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319213 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319215 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319218 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319220 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319223 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319225 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319228 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 17:32:25.399346 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319250 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319253 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319256 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319258 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319261 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319264 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319266 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319269 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319271 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319274 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319278 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319281 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319284 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319287 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.319290 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319359 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319367 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319373 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319378 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319383 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319386 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 17:32:25.509703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319391 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319396 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319399 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319402 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319405 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319409 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319412 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319415 2564 flags.go:64] FLAG: --cgroup-root="" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319418 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319421 2564 flags.go:64] FLAG: --client-ca-file="" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319424 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319427 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319432 2564 flags.go:64] FLAG: --cluster-domain="" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319435 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319438 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319441 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319445 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319449 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319452 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319456 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319459 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319463 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319466 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319469 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 17:32:25.510540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319472 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319475 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319483 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319486 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319489 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319492 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319495 2564 flags.go:64] FLAG: --enable-server="true" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319499 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319504 2564 flags.go:64] FLAG: --event-burst="100" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319507 2564 flags.go:64] FLAG: --event-qps="50" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319510 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319513 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319516 2564 flags.go:64] FLAG: --eviction-hard="" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319520 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319523 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319526 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319529 2564 flags.go:64] FLAG: --eviction-soft="" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319532 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319535 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319538 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319541 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319544 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319547 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319551 2564 flags.go:64] FLAG: --feature-gates="" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319554 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 17:32:25.512293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319558 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319561 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319564 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319567 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319570 2564 flags.go:64] FLAG: --help="false" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319573 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319576 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319579 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319582 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319587 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319591 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319594 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319597 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319600 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319603 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319607 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319610 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319613 2564 flags.go:64] FLAG: --kube-reserved="" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319616 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319620 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319623 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319625 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319628 2564 flags.go:64] FLAG: --lock-file="" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319631 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 17:32:25.513267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319634 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319638 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319643 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319646 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319649 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319652 2564 flags.go:64] FLAG: --logging-format="text" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319655 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319658 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319661 2564 flags.go:64] FLAG: --manifest-url="" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319664 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319669 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319672 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319676 2564 flags.go:64] FLAG: --max-pods="110" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319679 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319683 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319686 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319689 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319696 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319699 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319702 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319709 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319712 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319716 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 17:32:25.514054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319719 2564 flags.go:64] FLAG: --pod-cidr="" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319723 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319729 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319732 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319736 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319739 2564 flags.go:64] FLAG: --port="10250" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319742 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319745 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-026cf0bc9346c21ca" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319748 2564 flags.go:64] FLAG: --qos-reserved="" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319759 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319763 2564 flags.go:64] FLAG: --register-node="true" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319765 2564 flags.go:64] FLAG: --register-with-taints="" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319769 2564 flags.go:64] FLAG: --registry-burst="10" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319772 2564 flags.go:64] FLAG: --registry-qps="5" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319775 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319778 2564 flags.go:64] FLAG: --reserved-memory="" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319782 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319785 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319788 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319791 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319794 2564 flags.go:64] FLAG: --runonce="false" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319796 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319799 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319802 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319805 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 17:32:25.514750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319810 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319813 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319817 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319820 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319824 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319827 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319829 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319832 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319835 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319839 2564 flags.go:64] FLAG: --system-cgroups="" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319842 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319847 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319850 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319853 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319858 2564 flags.go:64] FLAG: --tls-min-version="" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319861 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319864 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319867 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319870 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319873 2564 flags.go:64] FLAG: --v="2" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319877 2564 flags.go:64] FLAG: --version="false" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319881 2564 flags.go:64] FLAG: --vmodule="" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319885 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.319888 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320010 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 17:32:25.515604 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320014 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320017 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320021 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320024 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320027 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320029 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320032 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320036 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320038 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320042 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320045 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320048 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320050 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320053 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320055 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320058 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320060 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320064 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320066 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 17:32:25.516357 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320069 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320071 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320074 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320076 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320079 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320081 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320085 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320089 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320092 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320095 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320098 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320100 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320103 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320106 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320108 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320111 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320114 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320116 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320118 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320121 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 17:32:25.517081 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320127 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320129 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320133 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320136 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320138 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320141 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320143 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320146 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320149 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320151 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320154 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320158 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320162 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320164 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320167 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320169 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320172 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320174 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320177 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320179 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 17:32:25.517664 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320182 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320184 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320187 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320190 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320192 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320195 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320197 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320199 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320202 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320205 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320207 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320209 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320213 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320217 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320222 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320225 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320227 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320246 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320249 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 17:32:25.518371 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320251 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320254 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320256 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320259 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320261 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320265 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320268 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320271 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320273 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320276 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320278 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320281 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320283 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320286 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320288 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320291 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320293 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320296 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320298 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 17:32:25.518918 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.320300 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.320917 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.328219 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.328250 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.328274 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.328280 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328358 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328363 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328367 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328370 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328373 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328376 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328379 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328382 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328384 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 17:32:25.519716 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328387 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328389 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328392 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328395 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328397 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328400 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328402 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328405 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328408 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328410 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328413 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328415 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328418 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328420 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328423 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328426 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328429 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328431 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328434 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328437 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 17:32:25.520156 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328440 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328445 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328448 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328451 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328454 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328457 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328459 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328462 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328464 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328467 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328470 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328472 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328475 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328477 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328480 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328483 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328486 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328488 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328491 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328493 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 17:32:25.520937 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328496 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328499 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328501 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328504 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328507 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328509 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328515 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328517 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328520 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328522 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328525 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328528 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328530 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328533 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328536 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328539 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328542 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328544 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328547 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 17:32:25.627190 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328549 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328552 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328554 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328557 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328559 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328562 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328564 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328567 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328569 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328572 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328576 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328580 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328583 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328585 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328588 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328590 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328593 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328595 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328598 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 17:32:25.645805 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328603 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328606 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328609 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328612 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328615 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328617 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328620 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328623 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328626 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328630 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328634 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328637 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.328642 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328758 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 17:32:25.646634 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328763 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328766 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328769 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328772 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328774 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328777 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328780 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328783 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328785 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328788 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328790 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328793 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328795 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328798 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328800 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328803 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328805 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328808 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328810 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328814 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 17:32:25.648498 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328818 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328821 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328823 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328826 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328829 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328831 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328834 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328837 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328840 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328843 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328845 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328848 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328850 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328853 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328855 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328858 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328860 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328863 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328866 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328868 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 17:32:25.649123 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328871 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328873 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328876 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328878 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328881 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328884 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328886 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328889 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328892 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328894 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328896 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328900 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328906 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328910 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328914 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328917 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328920 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328923 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328925 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 17:32:25.650001 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328928 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328932 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328936 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328939 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328941 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328944 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328946 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328949 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328951 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328954 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328956 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328958 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328961 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328963 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328966 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328968 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328970 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328973 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328976 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 17:32:25.650592 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328978 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328981 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328983 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328986 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328988 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328992 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328996 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.328999 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329001 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329004 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329006 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329009 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329012 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329014 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329016 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329020 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329022 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329025 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329027 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 17:32:25.651414 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:25.329030 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.329034 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.329148 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.331419 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.333082 2564 server.go:1019] "Starting client certificate rotation" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.334429 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.334977 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.337993 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.339172 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.354372 2564 log.go:25] "Validated CRI v1 runtime API" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.357054 2564 log.go:25] "Validated CRI v1 image API" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.357946 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 17:32:25.651974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.361741 2564 fs.go:135] Filesystem UUIDs: map[1e100878-5e94-459b-9f43-ca44f6e0341d:/dev/nvme0n1p3 52407729-8f4f-4aee-a99a-e45ffb390431:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 13 17:32:25.652534 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.361756 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 17:32:25.652534 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.361924 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.367646 2564 manager.go:216] Machine: {Timestamp:2026-07-13 17:32:25.366302017 +0000 UTC m=+0.373704251 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086696 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e292c23489b289838dd805e2ca1ac SystemUUID:ec2e292c-2348-9b28-9838-dd805e2ca1ac BootID:6f7ace60-a0b3-4e96-bdcf-dd761810697f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:88:d2:33:f7:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:88:d2:33:f7:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:1d:e2:d7:67:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.367753 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.367874 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.370976 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.370999 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.371778 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.371788 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.371795 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.372487 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.374035 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.377667 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.377774 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.377802 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.377812 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.377831 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.377840 2564 kubelet.go:398] "Adding apiserver pod source" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.377850 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 17:32:25.652889 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.378266 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.378284 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.382356 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.384945 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.384979 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387143 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7g745" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387152 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387208 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387217 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387223 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387250 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387257 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387266 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387271 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387278 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387283 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387291 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387297 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387319 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.387325 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.387884 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.387895 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.390651 2564 server.go:1268] "Started kubelet" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.390706 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.390732 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.391036 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.410090 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.410464 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.410909 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7g745" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.411070 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.411584 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.415638 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-16.ec2.internal.18c1e9c1a91cc460 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-16.ec2.internal,UID:ip-10-0-139-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-16.ec2.internal,},FirstTimestamp:2026-07-13 17:32:25.390621792 +0000 UTC m=+0.398024076,LastTimestamp:2026-07-13 17:32:25.390621792 +0000 UTC m=+0.398024076,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-16.ec2.internal,}" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.417843 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.417846 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.417969 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.418526 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.418543 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.418542 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.418671 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.418682 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 13 17:32:25.653492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.418834 2564 factory.go:55] Registering systemd factory Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.418851 2564 factory.go:223] Registration of the systemd container factory successfully Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.419071 2564 factory.go:153] Registering CRI-O factory Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.419083 2564 factory.go:223] Registration of the crio container factory successfully Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.419153 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.419175 2564 factory.go:103] Registering Raw factory Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.419185 2564 manager.go:1196] Started watching for new ooms in manager Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.419622 2564 manager.go:349] Starting recovery of all containers Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.420863 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.425088 2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-16.ec2.internal\" not found" node="ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.435616 2564 manager.go:354] Recovery completed Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.440003 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.442089 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.442116 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.442129 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.442816 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.442827 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.442846 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.444062 2564 policy_none.go:49] "None policy: Start" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.444076 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.444087 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.445273 2564 policy_none.go:47] "Start" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.446035 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.446862 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.446892 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.446903 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.489844 2564 manager.go:337] "Starting Device Plugin manager" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.490024 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.490037 2564 server.go:87] "Starting device plugin registration server" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.490309 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.490321 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.490418 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.490516 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.490529 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.491001 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.491046 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.591444 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.592499 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.592532 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.592547 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.592579 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.600216 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.600252 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-16.ec2.internal\": node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.609506 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.630031 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.631184 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.631217 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.631258 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.631300 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 17:32:25.654861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.633205 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.712178 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.731623 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal"] Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.731703 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.732825 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.732854 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.732866 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.734188 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.734326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.734357 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.734965 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.734969 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.734989 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.735011 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.734989 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.735075 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.736271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.736295 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.736964 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientMemory" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.736980 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.736989 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeHasSufficientPID" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.779056 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-16.ec2.internal\" not found" node="ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.787589 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.782580 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-16.ec2.internal\" not found" node="ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.868722 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.818328 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:25.868722 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.821599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/771081612d1c7db4aa39e9060dd20e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"771081612d1c7db4aa39e9060dd20e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.868722 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.821623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/771081612d1c7db4aa39e9060dd20e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"771081612d1c7db4aa39e9060dd20e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.868722 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.821639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df8d47ff6ebf83b9a6270b9d8e65671-config\") pod \"kube-apiserver-proxy-ip-10-0-139-16.ec2.internal\" (UID: \"8df8d47ff6ebf83b9a6270b9d8e65671\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.960954 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:25.918590 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:25.960954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.921725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/771081612d1c7db4aa39e9060dd20e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"771081612d1c7db4aa39e9060dd20e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.960954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.921751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/771081612d1c7db4aa39e9060dd20e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"771081612d1c7db4aa39e9060dd20e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.960954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.921771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df8d47ff6ebf83b9a6270b9d8e65671-config\") pod \"kube-apiserver-proxy-ip-10-0-139-16.ec2.internal\" (UID: \"8df8d47ff6ebf83b9a6270b9d8e65671\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.960954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.921800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df8d47ff6ebf83b9a6270b9d8e65671-config\") pod \"kube-apiserver-proxy-ip-10-0-139-16.ec2.internal\" (UID: \"8df8d47ff6ebf83b9a6270b9d8e65671\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.960954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.921813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/771081612d1c7db4aa39e9060dd20e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"771081612d1c7db4aa39e9060dd20e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:25.960954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:25.921823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/771081612d1c7db4aa39e9060dd20e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal\" (UID: \"771081612d1c7db4aa39e9060dd20e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:26.018932 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:26.018896 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:26.081504 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.081455 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:26.087617 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.087142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" Jul 13 17:32:26.119099 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:26.119028 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:26.219541 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:26.219452 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:26.319914 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:26.319882 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-16.ec2.internal\" not found" Jul 13 17:32:26.332192 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.332173 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 17:32:26.332368 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.332346 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 17:32:26.332421 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.332360 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 17:32:26.405738 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.405706 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 17:32:26.415762 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.415734 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 17:27:25 +0000 UTC" deadline="2028-04-14 04:53:14.462264982 +0000 UTC" Jul 13 17:32:26.415762 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.415757 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15371h20m48.046510216s" Jul 13 17:32:26.418912 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.418893 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" Jul 13 17:32:26.418912 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.418913 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 17:32:26.426468 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.426449 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 17:32:26.427945 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.427932 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" Jul 13 17:32:26.429709 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.429694 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 17:32:26.439808 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.439792 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 17:32:26.453531 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.453513 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2wt6l" Jul 13 17:32:26.460784 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.460769 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2wt6l" Jul 13 17:32:26.692657 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.692629 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 17:32:26.721525 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:26.721491 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df8d47ff6ebf83b9a6270b9d8e65671.slice/crio-a14fbb024a4e4363f9d202dbc4a22a2cbd3c9785ae2a7488d5f860d9f233fadb WatchSource:0}: Error finding container a14fbb024a4e4363f9d202dbc4a22a2cbd3c9785ae2a7488d5f860d9f233fadb: Status 404 returned error can't find the container with id a14fbb024a4e4363f9d202dbc4a22a2cbd3c9785ae2a7488d5f860d9f233fadb Jul 13 17:32:26.721736 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:26.721718 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771081612d1c7db4aa39e9060dd20e70.slice/crio-98f89ac7a4201010634446a0a6ed21ce637e23efd25564a089660ce6668f2b98 WatchSource:0}: Error finding container 98f89ac7a4201010634446a0a6ed21ce637e23efd25564a089660ce6668f2b98: Status 404 returned error can't find the container with id 98f89ac7a4201010634446a0a6ed21ce637e23efd25564a089660ce6668f2b98 Jul 13 17:32:26.725667 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:26.725653 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 17:32:27.162352 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.162319 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 17:32:27.378844 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.378808 2564 apiserver.go:52] "Watching apiserver" Jul 13 17:32:27.385917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.385660 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 17:32:27.388614 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.388263 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-9tx5d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal","openshift-multus/multus-qqt4v","openshift-network-diagnostics/network-check-target-fx7fh","openshift-network-operator/iptables-alerter-vm859","openshift-ovn-kubernetes/ovnkube-node-74zgv","kube-system/konnectivity-agent-27mpb","kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg","openshift-cluster-node-tuning-operator/tuned-brs7q","openshift-image-registry/node-ca-jfss5","openshift-multus/multus-additional-cni-plugins-bcvkf","openshift-multus/network-metrics-daemon-7xvlp","kube-system/global-pull-secret-syncer-n8t6r"] Jul 13 17:32:27.389886 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.389864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.391378 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.391344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:27.391522 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.391490 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:27.392061 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.392016 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 17:32:27.392201 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.392169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 17:32:27.392347 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.392333 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8n9xl\"" Jul 13 17:32:27.392752 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.392737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.393880 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.393860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:27.393976 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.393934 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:27.395221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.394674 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 17:32:27.395221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.394681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 17:32:27.395221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.394878 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 17:32:27.395221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.395058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d5jqr\"" Jul 13 17:32:27.395221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.395199 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 17:32:27.395513 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.395500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.396637 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.396620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.397604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.397465 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 17:32:27.397604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.397493 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 17:32:27.397604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.397523 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 17:32:27.398096 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.397964 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vch8s\"" Jul 13 17:32:27.398435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.398358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.398544 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.398474 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 17:32:27.398544 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.398492 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hsdkw\"" Jul 13 17:32:27.402435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.401065 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 17:32:27.402435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.401125 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 17:32:27.402435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.401583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 17:32:27.402435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.401610 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 17:32:27.402435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.401889 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 17:32:27.402435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.401941 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 17:32:27.402435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.401947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5jj2x\"" Jul 13 17:32:27.402834 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.402549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.402834 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.402802 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 17:32:27.404188 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.403835 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 17:32:27.405588 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.404814 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b58qp\"" Jul 13 17:32:27.405588 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.404887 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 17:32:27.405588 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.405091 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 17:32:27.405588 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.405147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.405588 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.405250 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:27.407184 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.407158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.408866 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.408850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.409068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.409049 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w8bbl\"" Jul 13 17:32:27.409328 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.409174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 17:32:27.409423 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.409374 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 17:32:27.410970 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.410950 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 17:32:27.411252 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.411062 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vp779\"" Jul 13 17:32:27.411252 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.411139 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 17:32:27.411400 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.411292 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 17:32:27.412218 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.412193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.414173 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.414131 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m5khn\"" Jul 13 17:32:27.414260 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.414170 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 17:32:27.414321 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.414132 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 17:32:27.419892 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.419876 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 17:32:27.421940 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.421921 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 17:32:27.427609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-daemon-config\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.427721 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7c7n\" (UniqueName: \"kubernetes.io/projected/d143e32b-1fb1-44cf-ad56-a44feb4804f2-kube-api-access-w7c7n\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.427721 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30053369-dbea-4ebb-8e82-4ee42689e422-dbus\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.427721 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52c44237-67a1-466a-a213-73d437bd48de-serviceca\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.427721 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.427721 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-kubelet\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysconfig\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d874e0f-b94b-4561-b80a-c28cb5434d65-tmp\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427816 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-env-overrides\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-system-cni-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-modprobe-d\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-lib-modules\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d12f0ad1-bc89-4309-9162-54e55c73e79f-agent-certs\") pod \"konnectivity-agent-27mpb\" (UID: \"d12f0ad1-bc89-4309-9162-54e55c73e79f\") " pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427974 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-etc-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.427997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-cni-bin\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-cnibin\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.428068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-iptables-alerter-script\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428102 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-registration-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thg2f\" (UniqueName: \"kubernetes.io/projected/57acac65-9ea0-4727-90d8-ac9ade3c7055-kube-api-access-thg2f\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-node-log\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-cni-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-cni-bin\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca2937f1-bf59-4da2-a238-5d057c9bcec2-hosts-file\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-host\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-cni-binary-copy\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef4276ff-3000-4fac-8c12-2a48c565929b-cni-binary-copy\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-sys-fs\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysctl-d\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysctl-conf\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-os-release\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-run-ovn-kubernetes\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-ovnkube-config\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.428664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kv2s\" (UniqueName: \"kubernetes.io/projected/f88cd6c6-7717-46de-8c19-a15c236db534-kube-api-access-8kv2s\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-var-lib-kubelet\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-cnibin\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-multus-certs\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30053369-dbea-4ebb-8e82-4ee42689e422-kubelet-config\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-systemd\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428662 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-host-slash\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-systemd\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-ovnkube-script-lib\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-k8s-cni-cncf-io\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-kubelet\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-etc-kubernetes\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.429295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-kubernetes\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.428996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-run\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-cni-netd\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0ad1-bc89-4309-9162-54e55c73e79f-konnectivity-ca\") pod \"konnectivity-agent-27mpb\" (UID: \"d12f0ad1-bc89-4309-9162-54e55c73e79f\") " pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-cni-multus\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgmr\" (UniqueName: \"kubernetes.io/projected/ef4276ff-3000-4fac-8c12-2a48c565929b-kube-api-access-wxgmr\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-system-cni-dir\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-ovn\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f88cd6c6-7717-46de-8c19-a15c236db534-ovn-node-metrics-cert\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-os-release\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-netns\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-systemd-units\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.430022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-var-lib-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-hostroot\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-tuned\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tsz\" (UniqueName: \"kubernetes.io/projected/3d874e0f-b94b-4561-b80a-c28cb5434d65-kube-api-access-w9tsz\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52c44237-67a1-466a-a213-73d437bd48de-host\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhkk\" (UniqueName: \"kubernetes.io/projected/52c44237-67a1-466a-a213-73d437bd48de-kube-api-access-wrhkk\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dn6\" (UniqueName: \"kubernetes.io/projected/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-kube-api-access-z2dn6\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-log-socket\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-run-netns\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxln\" (UniqueName: \"kubernetes.io/projected/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-kube-api-access-2gxln\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-device-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca2937f1-bf59-4da2-a238-5d057c9bcec2-tmp-dir\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksdz\" (UniqueName: \"kubernetes.io/projected/ca2937f1-bf59-4da2-a238-5d057c9bcec2-kube-api-access-sksdz\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-socket-dir-parent\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-conf-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-socket-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-sys\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.430900 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.429894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-slash\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.463904 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.463875 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 17:27:26 +0000 UTC" deadline="2027-12-26 13:02:26.951726933 +0000 UTC" Jul 13 17:32:27.463904 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.463903 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12739h29m59.487826835s" Jul 13 17:32:27.531166 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-tuned\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.531346 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tsz\" (UniqueName: \"kubernetes.io/projected/3d874e0f-b94b-4561-b80a-c28cb5434d65-kube-api-access-w9tsz\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.531346 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52c44237-67a1-466a-a213-73d437bd48de-host\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.531346 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52c44237-67a1-466a-a213-73d437bd48de-host\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.531346 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhkk\" (UniqueName: \"kubernetes.io/projected/52c44237-67a1-466a-a213-73d437bd48de-kube-api-access-wrhkk\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.531346 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dn6\" (UniqueName: \"kubernetes.io/projected/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-kube-api-access-z2dn6\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-log-socket\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-run-netns\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxln\" (UniqueName: \"kubernetes.io/projected/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-kube-api-access-2gxln\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-device-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca2937f1-bf59-4da2-a238-5d057c9bcec2-tmp-dir\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-log-socket\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-run-netns\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksdz\" (UniqueName: \"kubernetes.io/projected/ca2937f1-bf59-4da2-a238-5d057c9bcec2-kube-api-access-sksdz\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-socket-dir-parent\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-conf-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531574 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 17:32:27.531609 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-conf-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-device-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-socket-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-sys\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-socket-dir-parent\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-slash\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-daemon-config\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-sys\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-socket-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-slash\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531798 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca2937f1-bf59-4da2-a238-5d057c9bcec2-tmp-dir\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7c7n\" (UniqueName: \"kubernetes.io/projected/d143e32b-1fb1-44cf-ad56-a44feb4804f2-kube-api-access-w7c7n\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30053369-dbea-4ebb-8e82-4ee42689e422-dbus\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52c44237-67a1-466a-a213-73d437bd48de-serviceca\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-kubelet\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysconfig\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d874e0f-b94b-4561-b80a-c28cb5434d65-tmp\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.531982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-env-overrides\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30053369-dbea-4ebb-8e82-4ee42689e422-dbus\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-system-cni-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-modprobe-d\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-kubelet\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-system-cni-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-lib-modules\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysconfig\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-lib-modules\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d12f0ad1-bc89-4309-9162-54e55c73e79f-agent-certs\") pod \"konnectivity-agent-27mpb\" (UID: \"d12f0ad1-bc89-4309-9162-54e55c73e79f\") " pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-modprobe-d\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-etc-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-etc-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-cni-bin\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-cnibin\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.532999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-daemon-config\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52c44237-67a1-466a-a213-73d437bd48de-serviceca\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-cni-bin\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-iptables-alerter-script\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-registration-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532499 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-cnibin\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thg2f\" (UniqueName: \"kubernetes.io/projected/57acac65-9ea0-4727-90d8-ac9ade3c7055-kube-api-access-thg2f\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-env-overrides\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-node-log\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532564 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-registration-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-cni-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-cni-bin\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-node-log\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca2937f1-bf59-4da2-a238-5d057c9bcec2-hosts-file\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-host\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-cni-binary-copy\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-multus-cni-dir\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-cni-bin\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.533863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef4276ff-3000-4fac-8c12-2a48c565929b-cni-binary-copy\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-host\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-sys-fs\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysctl-d\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532745 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca2937f1-bf59-4da2-a238-5d057c9bcec2-hosts-file\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysctl-conf\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-os-release\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-sys-fs\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-run-ovn-kubernetes\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-ovnkube-config\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kv2s\" (UniqueName: \"kubernetes.io/projected/f88cd6c6-7717-46de-8c19-a15c236db534-kube-api-access-8kv2s\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-var-lib-kubelet\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysctl-conf\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-cnibin\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-multus-certs\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30053369-dbea-4ebb-8e82-4ee42689e422-kubelet-config\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-systemd\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.534693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-sysctl-d\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.532983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-host-slash\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-systemd\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533011 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-var-lib-kubelet\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533028 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533046 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-iptables-alerter-script\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-os-release\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-cnibin\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-ovnkube-script-lib\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-k8s-cni-cncf-io\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-systemd\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-cni-binary-copy\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-kubelet\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.533254 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-etc-kubernetes\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533102 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-systemd\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.535466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-multus-certs\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.533381 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs podName:bbc4ffda-5366-46b0-88bc-32e697a9d2ff nodeName:}" failed. No retries permitted until 2026-07-13 17:32:28.033358573 +0000 UTC m=+3.040760817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs") pod "network-metrics-daemon-7xvlp" (UID: "bbc4ffda-5366-46b0-88bc-32e697a9d2ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-etc-kubernetes\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-run-ovn-kubernetes\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-host-slash\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-ovnkube-config\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-k8s-cni-cncf-io\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-kubernetes\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-kubelet\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-run\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30053369-dbea-4ebb-8e82-4ee42689e422-kubelet-config\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef4276ff-3000-4fac-8c12-2a48c565929b-cni-binary-copy\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-run\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.536305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-cni-netd\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533811 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-kubernetes\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.533853 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0ad1-bc89-4309-9162-54e55c73e79f-konnectivity-ca\") pod \"konnectivity-agent-27mpb\" (UID: \"d12f0ad1-bc89-4309-9162-54e55c73e79f\") " pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-cni-multus\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.533911 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret podName:30053369-dbea-4ebb-8e82-4ee42689e422 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:28.033893292 +0000 UTC m=+3.041295527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret") pod "global-pull-secret-syncer-n8t6r" (UID: "30053369-dbea-4ebb-8e82-4ee42689e422") : object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533930 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-var-lib-cni-multus\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgmr\" (UniqueName: \"kubernetes.io/projected/ef4276ff-3000-4fac-8c12-2a48c565929b-kube-api-access-wxgmr\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-host-cni-netd\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-system-cni-dir\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57acac65-9ea0-4727-90d8-ac9ade3c7055-system-cni-dir\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.533996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-ovn\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f88cd6c6-7717-46de-8c19-a15c236db534-ovn-node-metrics-cert\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-os-release\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-netns\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.537165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-systemd-units\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-var-lib-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-hostroot\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534225 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534311 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-hostroot\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-host-run-netns\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f88cd6c6-7717-46de-8c19-a15c236db534-ovnkube-script-lib\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0ad1-bc89-4309-9162-54e55c73e79f-konnectivity-ca\") pod \"konnectivity-agent-27mpb\" (UID: \"d12f0ad1-bc89-4309-9162-54e55c73e79f\") " pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-systemd-units\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534582 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-var-lib-openvswitch\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef4276ff-3000-4fac-8c12-2a48c565929b-os-release\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f88cd6c6-7717-46de-8c19-a15c236db534-run-ovn\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.534760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d143e32b-1fb1-44cf-ad56-a44feb4804f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.535201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57acac65-9ea0-4727-90d8-ac9ade3c7055-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.536142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d874e0f-b94b-4561-b80a-c28cb5434d65-tmp\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.536473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d12f0ad1-bc89-4309-9162-54e55c73e79f-agent-certs\") pod \"konnectivity-agent-27mpb\" (UID: \"d12f0ad1-bc89-4309-9162-54e55c73e79f\") " pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.536892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f88cd6c6-7717-46de-8c19-a15c236db534-ovn-node-metrics-cert\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.537989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.537114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d874e0f-b94b-4561-b80a-c28cb5434d65-etc-tuned\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.541423 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.541297 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 17:32:27.541423 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.541324 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 17:32:27.541423 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.541337 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sppk5 for pod openshift-network-diagnostics/network-check-target-fx7fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:27.541423 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:27.541399 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5 podName:06123f7e-7ce5-4f55-8b0a-f858c91860a0 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:28.041382847 +0000 UTC m=+3.048785088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sppk5" (UniqueName: "kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5") pod "network-check-target-fx7fh" (UID: "06123f7e-7ce5-4f55-8b0a-f858c91860a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:27.542286 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.542203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksdz\" (UniqueName: \"kubernetes.io/projected/ca2937f1-bf59-4da2-a238-5d057c9bcec2-kube-api-access-sksdz\") pod \"node-resolver-9tx5d\" (UID: \"ca2937f1-bf59-4da2-a238-5d057c9bcec2\") " pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.542398 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.542220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxln\" (UniqueName: \"kubernetes.io/projected/da7f42e5-55f2-48b2-8e1f-f91cc2311eab-kube-api-access-2gxln\") pod \"iptables-alerter-vm859\" (UID: \"da7f42e5-55f2-48b2-8e1f-f91cc2311eab\") " pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.542398 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.542348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7c7n\" (UniqueName: \"kubernetes.io/projected/d143e32b-1fb1-44cf-ad56-a44feb4804f2-kube-api-access-w7c7n\") pod \"aws-ebs-csi-driver-node-dwrvg\" (UID: \"d143e32b-1fb1-44cf-ad56-a44feb4804f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.542511 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.542450 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhkk\" (UniqueName: \"kubernetes.io/projected/52c44237-67a1-466a-a213-73d437bd48de-kube-api-access-wrhkk\") pod \"node-ca-jfss5\" (UID: \"52c44237-67a1-466a-a213-73d437bd48de\") " pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.542756 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.542739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tsz\" (UniqueName: \"kubernetes.io/projected/3d874e0f-b94b-4561-b80a-c28cb5434d65-kube-api-access-w9tsz\") pod \"tuned-brs7q\" (UID: \"3d874e0f-b94b-4561-b80a-c28cb5434d65\") " pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.544398 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.544318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kv2s\" (UniqueName: \"kubernetes.io/projected/f88cd6c6-7717-46de-8c19-a15c236db534-kube-api-access-8kv2s\") pod \"ovnkube-node-74zgv\" (UID: \"f88cd6c6-7717-46de-8c19-a15c236db534\") " pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.544398 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.544321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thg2f\" (UniqueName: \"kubernetes.io/projected/57acac65-9ea0-4727-90d8-ac9ade3c7055-kube-api-access-thg2f\") pod \"multus-additional-cni-plugins-bcvkf\" (UID: \"57acac65-9ea0-4727-90d8-ac9ade3c7055\") " pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.545066 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.545045 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dn6\" (UniqueName: \"kubernetes.io/projected/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-kube-api-access-z2dn6\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:27.545732 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.545707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgmr\" (UniqueName: \"kubernetes.io/projected/ef4276ff-3000-4fac-8c12-2a48c565929b-kube-api-access-wxgmr\") pod \"multus-qqt4v\" (UID: \"ef4276ff-3000-4fac-8c12-2a48c565929b\") " pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.636260 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.636175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" event={"ID":"8df8d47ff6ebf83b9a6270b9d8e65671","Type":"ContainerStarted","Data":"a14fbb024a4e4363f9d202dbc4a22a2cbd3c9785ae2a7488d5f860d9f233fadb"} Jul 13 17:32:27.637366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.637342 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" event={"ID":"771081612d1c7db4aa39e9060dd20e70","Type":"ContainerStarted","Data":"98f89ac7a4201010634446a0a6ed21ce637e23efd25564a089660ce6668f2b98"} Jul 13 17:32:27.706660 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.706581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:27.714764 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.714741 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqt4v" Jul 13 17:32:27.717382 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.717351 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12f0ad1_bc89_4309_9162_54e55c73e79f.slice/crio-a7c1a27fbc2f48d920a203ab8b665cd0666b87d23419d5a27f87d643911cad0d WatchSource:0}: Error finding container a7c1a27fbc2f48d920a203ab8b665cd0666b87d23419d5a27f87d643911cad0d: Status 404 returned error can't find the container with id a7c1a27fbc2f48d920a203ab8b665cd0666b87d23419d5a27f87d643911cad0d Jul 13 17:32:27.725769 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.725747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brs7q" Jul 13 17:32:27.730980 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.730947 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4276ff_3000_4fac_8c12_2a48c565929b.slice/crio-b644bd0c30da91e11063ef61d867ed9e2e1251b0e12d11901e3361ed2ea49414 WatchSource:0}: Error finding container b644bd0c30da91e11063ef61d867ed9e2e1251b0e12d11901e3361ed2ea49414: Status 404 returned error can't find the container with id b644bd0c30da91e11063ef61d867ed9e2e1251b0e12d11901e3361ed2ea49414 Jul 13 17:32:27.732651 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.732623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vm859" Jul 13 17:32:27.735387 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.735351 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d874e0f_b94b_4561_b80a_c28cb5434d65.slice/crio-b8413c0eb1507cbf28590c66fa3b4842fb797609667662928ea62c65bc699fae WatchSource:0}: Error finding container b8413c0eb1507cbf28590c66fa3b4842fb797609667662928ea62c65bc699fae: Status 404 returned error can't find the container with id b8413c0eb1507cbf28590c66fa3b4842fb797609667662928ea62c65bc699fae Jul 13 17:32:27.740737 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.740713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:27.741062 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.741036 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7f42e5_55f2_48b2_8e1f_f91cc2311eab.slice/crio-14c41b640be71e161b911c7a513b16d3160c9b04d169f00800a20b2d9bafae98 WatchSource:0}: Error finding container 14c41b640be71e161b911c7a513b16d3160c9b04d169f00800a20b2d9bafae98: Status 404 returned error can't find the container with id 14c41b640be71e161b911c7a513b16d3160c9b04d169f00800a20b2d9bafae98 Jul 13 17:32:27.747832 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.747812 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" Jul 13 17:32:27.750014 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.749985 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88cd6c6_7717_46de_8c19_a15c236db534.slice/crio-b56c0ae8b87a1751e4fa4d33715ac0e44ab4dde5d0226498868d068aeb384fdc WatchSource:0}: Error finding container b56c0ae8b87a1751e4fa4d33715ac0e44ab4dde5d0226498868d068aeb384fdc: Status 404 returned error can't find the container with id b56c0ae8b87a1751e4fa4d33715ac0e44ab4dde5d0226498868d068aeb384fdc Jul 13 17:32:27.755276 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.755222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9tx5d" Jul 13 17:32:27.759273 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.759224 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd143e32b_1fb1_44cf_ad56_a44feb4804f2.slice/crio-bdff601b61fbc757522f5616c625c4d721c4044343836abbf954e6599ba31287 WatchSource:0}: Error finding container bdff601b61fbc757522f5616c625c4d721c4044343836abbf954e6599ba31287: Status 404 returned error can't find the container with id bdff601b61fbc757522f5616c625c4d721c4044343836abbf954e6599ba31287 Jul 13 17:32:27.764407 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.763067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jfss5" Jul 13 17:32:27.766522 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.766022 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2937f1_bf59_4da2_a238_5d057c9bcec2.slice/crio-7136d306fbfc53e742c1895a6ea5d5eb23bdaa240061c38cc6cdfd3f702a63dd WatchSource:0}: Error finding container 7136d306fbfc53e742c1895a6ea5d5eb23bdaa240061c38cc6cdfd3f702a63dd: Status 404 returned error can't find the container with id 7136d306fbfc53e742c1895a6ea5d5eb23bdaa240061c38cc6cdfd3f702a63dd Jul 13 17:32:27.768884 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:27.768863 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" Jul 13 17:32:27.773374 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.773354 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c44237_67a1_466a_a213_73d437bd48de.slice/crio-236d4cf3cd55fb144a4a8b3470dbab3446660d65eab24a1613afc9ec501b9b31 WatchSource:0}: Error finding container 236d4cf3cd55fb144a4a8b3470dbab3446660d65eab24a1613afc9ec501b9b31: Status 404 returned error can't find the container with id 236d4cf3cd55fb144a4a8b3470dbab3446660d65eab24a1613afc9ec501b9b31 Jul 13 17:32:27.778430 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:32:27.778410 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57acac65_9ea0_4727_90d8_ac9ade3c7055.slice/crio-de050cc0cdc392b8830cc4978ec78ab7b3a236e008eb12822fd76f099b02c1ab WatchSource:0}: Error finding container de050cc0cdc392b8830cc4978ec78ab7b3a236e008eb12822fd76f099b02c1ab: Status 404 returned error can't find the container with id de050cc0cdc392b8830cc4978ec78ab7b3a236e008eb12822fd76f099b02c1ab Jul 13 17:32:28.037982 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.037850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:28.037982 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.037904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:28.037982 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.037968 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:28.038326 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.038011 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:28.038326 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.038041 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs podName:bbc4ffda-5366-46b0-88bc-32e697a9d2ff nodeName:}" failed. No retries permitted until 2026-07-13 17:32:29.038021574 +0000 UTC m=+4.045423818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs") pod "network-metrics-daemon-7xvlp" (UID: "bbc4ffda-5366-46b0-88bc-32e697a9d2ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:28.038326 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.038059 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret podName:30053369-dbea-4ebb-8e82-4ee42689e422 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:29.038050355 +0000 UTC m=+4.045452585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret") pod "global-pull-secret-syncer-n8t6r" (UID: "30053369-dbea-4ebb-8e82-4ee42689e422") : object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:28.138967 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.138934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:28.139116 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.139096 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 17:32:28.139116 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.139115 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 17:32:28.139188 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.139125 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sppk5 for pod openshift-network-diagnostics/network-check-target-fx7fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:28.139188 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.139177 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5 podName:06123f7e-7ce5-4f55-8b0a-f858c91860a0 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:29.13916239 +0000 UTC m=+4.146564633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sppk5" (UniqueName: "kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5") pod "network-check-target-fx7fh" (UID: "06123f7e-7ce5-4f55-8b0a-f858c91860a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:28.464435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.464312 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 17:27:26 +0000 UTC" deadline="2027-12-11 20:35:06.004154221 +0000 UTC" Jul 13 17:32:28.464435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.464379 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12387h2m37.539779791s" Jul 13 17:32:28.632957 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.632391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:28.632957 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:28.632541 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:28.642948 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.642857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerStarted","Data":"de050cc0cdc392b8830cc4978ec78ab7b3a236e008eb12822fd76f099b02c1ab"} Jul 13 17:32:28.644169 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.644090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9tx5d" event={"ID":"ca2937f1-bf59-4da2-a238-5d057c9bcec2","Type":"ContainerStarted","Data":"7136d306fbfc53e742c1895a6ea5d5eb23bdaa240061c38cc6cdfd3f702a63dd"} Jul 13 17:32:28.646226 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.646176 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" event={"ID":"d143e32b-1fb1-44cf-ad56-a44feb4804f2","Type":"ContainerStarted","Data":"bdff601b61fbc757522f5616c625c4d721c4044343836abbf954e6599ba31287"} Jul 13 17:32:28.648224 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.648199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"b56c0ae8b87a1751e4fa4d33715ac0e44ab4dde5d0226498868d068aeb384fdc"} Jul 13 17:32:28.650707 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.650650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vm859" event={"ID":"da7f42e5-55f2-48b2-8e1f-f91cc2311eab","Type":"ContainerStarted","Data":"14c41b640be71e161b911c7a513b16d3160c9b04d169f00800a20b2d9bafae98"} Jul 13 17:32:28.654963 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.654919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brs7q" event={"ID":"3d874e0f-b94b-4561-b80a-c28cb5434d65","Type":"ContainerStarted","Data":"b8413c0eb1507cbf28590c66fa3b4842fb797609667662928ea62c65bc699fae"} Jul 13 17:32:28.657508 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.657482 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jfss5" event={"ID":"52c44237-67a1-466a-a213-73d437bd48de","Type":"ContainerStarted","Data":"236d4cf3cd55fb144a4a8b3470dbab3446660d65eab24a1613afc9ec501b9b31"} Jul 13 17:32:28.658899 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.658833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqt4v" event={"ID":"ef4276ff-3000-4fac-8c12-2a48c565929b","Type":"ContainerStarted","Data":"b644bd0c30da91e11063ef61d867ed9e2e1251b0e12d11901e3361ed2ea49414"} Jul 13 17:32:28.660775 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:28.660729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-27mpb" event={"ID":"d12f0ad1-bc89-4309-9162-54e55c73e79f","Type":"ContainerStarted","Data":"a7c1a27fbc2f48d920a203ab8b665cd0666b87d23419d5a27f87d643911cad0d"} Jul 13 17:32:29.046498 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.045905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:29.046498 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.045954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:29.046498 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.046084 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:29.046498 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.046097 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:29.046498 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.046149 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret podName:30053369-dbea-4ebb-8e82-4ee42689e422 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:31.046129804 +0000 UTC m=+6.053532028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret") pod "global-pull-secret-syncer-n8t6r" (UID: "30053369-dbea-4ebb-8e82-4ee42689e422") : object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:29.046498 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.046169 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs podName:bbc4ffda-5366-46b0-88bc-32e697a9d2ff nodeName:}" failed. No retries permitted until 2026-07-13 17:32:31.046159199 +0000 UTC m=+6.053561455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs") pod "network-metrics-daemon-7xvlp" (UID: "bbc4ffda-5366-46b0-88bc-32e697a9d2ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:29.147739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.147702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:29.148081 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.147859 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 17:32:29.148081 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.147881 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 17:32:29.148081 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.147894 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sppk5 for pod openshift-network-diagnostics/network-check-target-fx7fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:29.148081 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.147951 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5 podName:06123f7e-7ce5-4f55-8b0a-f858c91860a0 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:31.147933149 +0000 UTC m=+6.155335384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sppk5" (UniqueName: "kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5") pod "network-check-target-fx7fh" (UID: "06123f7e-7ce5-4f55-8b0a-f858c91860a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:29.634209 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.631929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:29.634209 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.632286 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:29.634209 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.632697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:29.634209 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:29.632801 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:29.677980 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.677948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" event={"ID":"8df8d47ff6ebf83b9a6270b9d8e65671","Type":"ContainerStarted","Data":"8de5e3d80d4b772beac52bbcb3213990fe8cb4832056005e0da5c0571c3237ff"} Jul 13 17:32:29.683228 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.683199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" event={"ID":"771081612d1c7db4aa39e9060dd20e70","Type":"ContainerStarted","Data":"7fa2b380b328b8599d95f9f702957b4a820cfd08082a564609a625e2e47f3d51"} Jul 13 17:32:29.691917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:29.691060 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-16.ec2.internal" podStartSLOduration=3.691043357 podStartE2EDuration="3.691043357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:29.690972584 +0000 UTC m=+4.698374830" watchObservedRunningTime="2026-07-13 17:32:29.691043357 +0000 UTC m=+4.698445608" Jul 13 17:32:30.632562 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:30.632523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:30.632765 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:30.632674 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:30.705366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:30.704461 2564 generic.go:356] "Generic (PLEG): container finished" podID="771081612d1c7db4aa39e9060dd20e70" containerID="7fa2b380b328b8599d95f9f702957b4a820cfd08082a564609a625e2e47f3d51" exitCode=0 Jul 13 17:32:30.705366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:30.705353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" event={"ID":"771081612d1c7db4aa39e9060dd20e70","Type":"ContainerDied","Data":"7fa2b380b328b8599d95f9f702957b4a820cfd08082a564609a625e2e47f3d51"} Jul 13 17:32:31.065813 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:31.065091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:31.065813 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:31.065144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:31.065813 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.065309 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:31.065813 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.065375 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret podName:30053369-dbea-4ebb-8e82-4ee42689e422 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:35.065356817 +0000 UTC m=+10.072759058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret") pod "global-pull-secret-syncer-n8t6r" (UID: "30053369-dbea-4ebb-8e82-4ee42689e422") : object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:31.065813 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.065455 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:31.065813 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.065487 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs podName:bbc4ffda-5366-46b0-88bc-32e697a9d2ff nodeName:}" failed. No retries permitted until 2026-07-13 17:32:35.065475656 +0000 UTC m=+10.072877883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs") pod "network-metrics-daemon-7xvlp" (UID: "bbc4ffda-5366-46b0-88bc-32e697a9d2ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:31.165837 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:31.165797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:31.166021 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.165974 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 17:32:31.166021 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.165997 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 17:32:31.166021 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.166009 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sppk5 for pod openshift-network-diagnostics/network-check-target-fx7fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:31.166187 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.166073 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5 podName:06123f7e-7ce5-4f55-8b0a-f858c91860a0 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:35.166055904 +0000 UTC m=+10.173458127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sppk5" (UniqueName: "kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5") pod "network-check-target-fx7fh" (UID: "06123f7e-7ce5-4f55-8b0a-f858c91860a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:31.636667 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:31.636634 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:31.636877 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.636781 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:31.637122 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:31.637103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:31.637246 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:31.637209 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:32.632112 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:32.631611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:32.632112 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:32.631754 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:33.631739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:33.631693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:33.631917 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:33.631815 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:33.632158 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:33.631706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:33.632158 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:33.632101 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:34.632620 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:34.632072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:34.632620 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:34.632222 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:35.108665 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:35.108619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:35.108866 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:35.108678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:35.108866 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.108773 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:35.108866 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.108774 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:35.108866 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.108837 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret podName:30053369-dbea-4ebb-8e82-4ee42689e422 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:43.108815479 +0000 UTC m=+18.116217722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret") pod "global-pull-secret-syncer-n8t6r" (UID: "30053369-dbea-4ebb-8e82-4ee42689e422") : object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:35.108866 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.108862 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs podName:bbc4ffda-5366-46b0-88bc-32e697a9d2ff nodeName:}" failed. No retries permitted until 2026-07-13 17:32:43.108848998 +0000 UTC m=+18.116251222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs") pod "network-metrics-daemon-7xvlp" (UID: "bbc4ffda-5366-46b0-88bc-32e697a9d2ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:35.210008 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:35.209974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:35.210192 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.210175 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 17:32:35.210271 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.210196 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 17:32:35.210271 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.210208 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sppk5 for pod openshift-network-diagnostics/network-check-target-fx7fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:35.210413 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.210287 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5 podName:06123f7e-7ce5-4f55-8b0a-f858c91860a0 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:43.210267981 +0000 UTC m=+18.217670224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sppk5" (UniqueName: "kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5") pod "network-check-target-fx7fh" (UID: "06123f7e-7ce5-4f55-8b0a-f858c91860a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:35.633424 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:35.632911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:35.633424 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.633028 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:35.633424 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:35.633078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:35.633424 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:35.633194 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:36.632312 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:36.632276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:36.632500 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:36.632415 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:37.631550 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:37.631515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:37.631550 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:37.631554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:37.632035 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:37.631657 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:37.632035 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:37.631937 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:38.631831 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:38.631799 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:38.632203 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:38.631904 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:39.632290 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:39.632255 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:39.632759 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:39.632433 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:39.632759 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:39.632489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:39.632759 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:39.632602 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:40.632132 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:40.632102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:40.632303 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:40.632216 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:41.632338 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:41.632298 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:41.632767 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:41.632347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:41.632767 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:41.632435 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:41.632767 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:41.632568 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:42.632427 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:42.632394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:42.632931 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:42.632518 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:43.169497 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:43.169455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:43.169713 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:43.169514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:43.169713 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.169635 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:43.169713 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.169638 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:43.169713 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.169700 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret podName:30053369-dbea-4ebb-8e82-4ee42689e422 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:59.169683883 +0000 UTC m=+34.177086124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret") pod "global-pull-secret-syncer-n8t6r" (UID: "30053369-dbea-4ebb-8e82-4ee42689e422") : object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:43.169713 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.169715 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs podName:bbc4ffda-5366-46b0-88bc-32e697a9d2ff nodeName:}" failed. No retries permitted until 2026-07-13 17:32:59.169708649 +0000 UTC m=+34.177110870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs") pod "network-metrics-daemon-7xvlp" (UID: "bbc4ffda-5366-46b0-88bc-32e697a9d2ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:43.270518 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:43.270481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:43.270707 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.270667 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 17:32:43.270707 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.270690 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 17:32:43.270707 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.270703 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sppk5 for pod openshift-network-diagnostics/network-check-target-fx7fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:43.270866 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.270763 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5 podName:06123f7e-7ce5-4f55-8b0a-f858c91860a0 nodeName:}" failed. No retries permitted until 2026-07-13 17:32:59.270745119 +0000 UTC m=+34.278147340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sppk5" (UniqueName: "kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5") pod "network-check-target-fx7fh" (UID: "06123f7e-7ce5-4f55-8b0a-f858c91860a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:43.632390 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:43.632355 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:43.632584 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.632496 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:43.632584 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:43.632533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:43.632930 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:43.632612 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:44.631982 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:44.631949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:44.632168 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:44.632080 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:45.633119 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:45.633091 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:45.633575 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:45.633254 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:45.633575 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:45.633295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:45.633575 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:45.633387 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:46.632452 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:46.632417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:46.632615 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:46.632596 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:47.634818 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.634659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:47.635333 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.634713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:47.635333 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:47.634905 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:47.635333 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:47.634986 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:47.740334 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.740303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" event={"ID":"771081612d1c7db4aa39e9060dd20e70","Type":"ContainerStarted","Data":"fc55d8bb6117572d57fba6bd6cd45d452db49538bf11e82dc7bc5abbed00ba7f"} Jul 13 17:32:47.742332 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.742304 2564 generic.go:356] "Generic (PLEG): container finished" podID="57acac65-9ea0-4727-90d8-ac9ade3c7055" containerID="e4d775e199473eaf3567040b2943cbf2c8d8dc323457f25414e0d7c839406415" exitCode=0 Jul 13 17:32:47.742462 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.742379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerDied","Data":"e4d775e199473eaf3567040b2943cbf2c8d8dc323457f25414e0d7c839406415"} Jul 13 17:32:47.745683 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.745656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9tx5d" event={"ID":"ca2937f1-bf59-4da2-a238-5d057c9bcec2","Type":"ContainerStarted","Data":"4fa1b8479ee4274c61f61398426bb39b968b23f366ed255b9ff8c2429f62cc0b"} Jul 13 17:32:47.747271 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.747220 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" event={"ID":"d143e32b-1fb1-44cf-ad56-a44feb4804f2","Type":"ContainerStarted","Data":"82988c2f61c87d0c8a5ab9f3fe3d7ef3b57450d26e2473b810fb0145001591a5"} Jul 13 17:32:47.750692 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.750672 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:32:47.751117 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.751092 2564 generic.go:356] "Generic (PLEG): container finished" podID="f88cd6c6-7717-46de-8c19-a15c236db534" containerID="78f48f70e4e6e7da547af283c18fd0f14e040a909cc91168274d92f83bc41b28" exitCode=1 Jul 13 17:32:47.751249 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.751161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"fa552f6a1b000bd6c5b0963158d07df15f11c963a4b148b7b339c96667b7e658"} Jul 13 17:32:47.751249 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.751227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"0cc0af01c3a5d035672df1594ed8475205da7af87857433ca001c098e1b39ae8"} Jul 13 17:32:47.751359 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.751279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerDied","Data":"78f48f70e4e6e7da547af283c18fd0f14e040a909cc91168274d92f83bc41b28"} Jul 13 17:32:47.751359 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.751323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"d6b9b24b131647fb4a25573ba92a25a87a86aa0784c145b12732ecafe7c43792"} Jul 13 17:32:47.754371 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.753290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brs7q" event={"ID":"3d874e0f-b94b-4561-b80a-c28cb5434d65","Type":"ContainerStarted","Data":"e7d6665648ad2b34499960dd297c447c51e5d9c1e1963d93c28de85e3633e5ae"} Jul 13 17:32:47.756180 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.756133 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-16.ec2.internal" podStartSLOduration=21.756117299 podStartE2EDuration="21.756117299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:47.755921372 +0000 UTC m=+22.763323616" watchObservedRunningTime="2026-07-13 17:32:47.756117299 +0000 UTC m=+22.763519543" Jul 13 17:32:47.758563 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.758527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jfss5" event={"ID":"52c44237-67a1-466a-a213-73d437bd48de","Type":"ContainerStarted","Data":"a6f4fbbe493288e5811db46461dd3c47bcea97aeda4634d7a248d2dba2aebe42"} Jul 13 17:32:47.760965 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.760816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqt4v" event={"ID":"ef4276ff-3000-4fac-8c12-2a48c565929b","Type":"ContainerStarted","Data":"df32c8783def622aea209d225fa95e901fb6cd7d5d7f69e8167ed6b16755553d"} Jul 13 17:32:47.762859 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.762824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-27mpb" event={"ID":"d12f0ad1-bc89-4309-9162-54e55c73e79f","Type":"ContainerStarted","Data":"85f114d2f173b368ca6fc298528c56283c29498172615d6b1dad259de62813c1"} Jul 13 17:32:47.773116 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.773068 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-brs7q" podStartSLOduration=3.547642621 podStartE2EDuration="22.773050568s" totalImagesPullingTime="19.225407947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:47.772468757 +0000 UTC m=+22.779870999" watchObservedRunningTime="2026-07-13 17:32:47.773050568 +0000 UTC m=+22.780452814" Jul 13 17:32:47.810185 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.810135 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9tx5d" podStartSLOduration=3.620752478 podStartE2EDuration="22.810103176s" totalImagesPullingTime="19.189350698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:47.809534247 +0000 UTC m=+22.816936490" watchObservedRunningTime="2026-07-13 17:32:47.810103176 +0000 UTC m=+22.817505419" Jul 13 17:32:47.848114 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.848062 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qqt4v" podStartSLOduration=3.602755505 podStartE2EDuration="22.848045073s" totalImagesPullingTime="19.245289568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:47.847845972 +0000 UTC m=+22.855248217" watchObservedRunningTime="2026-07-13 17:32:47.848045073 +0000 UTC m=+22.855447316" Jul 13 17:32:47.848425 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.848394 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-27mpb" podStartSLOduration=3.606689339 podStartE2EDuration="22.848386368s" totalImagesPullingTime="19.241697029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:47.826295924 +0000 UTC m=+22.833698168" watchObservedRunningTime="2026-07-13 17:32:47.848386368 +0000 UTC m=+22.855788652" Jul 13 17:32:47.861406 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:47.861361 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jfss5" podStartSLOduration=3.676993517 podStartE2EDuration="22.861344722s" totalImagesPullingTime="19.184351205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:47.8608196 +0000 UTC m=+22.868221847" watchObservedRunningTime="2026-07-13 17:32:47.861344722 +0000 UTC m=+22.868746965" Jul 13 17:32:48.069092 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.069051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:48.276981 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.276958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:48.505251 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.505209 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 17:32:48.631775 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.631706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:48.631920 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:48.631812 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:48.681031 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.681003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:48.766466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.766434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" event={"ID":"d143e32b-1fb1-44cf-ad56-a44feb4804f2","Type":"ContainerStarted","Data":"aea0c12e6588634bad600694089dbb204b77f9163ffaaaf4b22b68c2d7a73c52"} Jul 13 17:32:48.769206 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.769186 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:32:48.769575 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.769550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"af02b5f6333972bf27983199eb4928006d36cdac94f7ba963de55141dad38dd9"} Jul 13 17:32:48.769688 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.769622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"50310418da4df07be6989bf8dba8d3d5338ff1d5c3e38919a234d3ae59879f1f"} Jul 13 17:32:48.770859 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.770824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vm859" event={"ID":"da7f42e5-55f2-48b2-8e1f-f91cc2311eab","Type":"ContainerStarted","Data":"06f1414f4d771022433dc00ece0c7f7640e9c8b017694385df0ed62934781740"} Jul 13 17:32:48.785713 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:48.785670 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vm859" podStartSLOduration=9.083189286 podStartE2EDuration="23.785656446s" totalImagesPullingTime="14.70246716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:48.785141536 +0000 UTC m=+23.792543780" watchObservedRunningTime="2026-07-13 17:32:48.785656446 +0000 UTC m=+23.793058689" Jul 13 17:32:49.271602 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.271568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:49.470003 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.469964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:49.505183 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.505020 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T17:32:48.505228477Z","UUID":"5a8730c5-2ecc-494e-b4bf-a679a99ed74c","Handler":null,"Name":"","Endpoint":""} Jul 13 17:32:49.506455 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.506439 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 17:32:49.506535 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.506485 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 17:32:49.588881 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.588844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:49.631636 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.631604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:49.631797 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:49.631711 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:49.631797 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.631771 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:49.631935 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:49.631913 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:49.690393 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:49.690355 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:50.074990 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.074839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:50.268296 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.268198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:50.369913 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.369880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:50.631517 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.631484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:50.631706 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:50.631598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:50.778205 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.778168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" event={"ID":"d143e32b-1fb1-44cf-ad56-a44feb4804f2","Type":"ContainerStarted","Data":"0b477b9c1a8aeb8630b80d9736059ce655f0a9b24caafde28e9b4fc0df2beefc"} Jul 13 17:32:50.782018 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.781988 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:32:50.782463 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.782416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"85a2197162555df0d92c947566f822fafa9be672c2a60b5a8fd571d77240d5a0"} Jul 13 17:32:50.870137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:50.870102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:51.199789 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.199758 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:51.501071 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.501000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:51.632001 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.631973 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:51.632154 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:51.632096 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:51.632567 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.632535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:51.632705 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:51.632665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:51.670931 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.670894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:51.836818 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.836788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:51.837399 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.837382 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:51.850847 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.850794 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dwrvg" podStartSLOduration=5.068335328 podStartE2EDuration="26.850776118s" totalImagesPullingTime="21.78244079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:50.794968148 +0000 UTC m=+25.802370391" watchObservedRunningTime="2026-07-13 17:32:51.850776118 +0000 UTC m=+26.858178362" Jul 13 17:32:51.873648 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.873618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:51.975967 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:51.975938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:52.269926 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:52.269888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:52.632122 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:52.632095 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:52.632300 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:52.632215 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:52.715028 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:52.714989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:53.198509 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.198485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:53.296359 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.296171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:53.634169 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.634150 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:53.634286 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.634151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:53.634286 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:53.634269 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:53.634378 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:53.634313 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:53.790615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.790586 2564 generic.go:356] "Generic (PLEG): container finished" podID="57acac65-9ea0-4727-90d8-ac9ade3c7055" containerID="56eb2957293c9eef6b1d5a7eadf5af92e96f921df0d025f3507357956f844938" exitCode=0 Jul 13 17:32:53.790837 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.790672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerDied","Data":"56eb2957293c9eef6b1d5a7eadf5af92e96f921df0d025f3507357956f844938"} Jul 13 17:32:53.794289 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.794267 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:32:53.797708 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.797679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"78f6c9a675d23b7d81b528b90dd1b1220b9f18e48b559213c2518e77447ede46"} Jul 13 17:32:53.797877 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.797866 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:53.797947 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.797903 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:53.798050 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.798032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:53.798170 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.798152 2564 scope.go:117] "RemoveContainer" containerID="78f48f70e4e6e7da547af283c18fd0f14e040a909cc91168274d92f83bc41b28" Jul 13 17:32:53.814841 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.814805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:53.814984 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.814957 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:32:53.883906 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:53.883874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:54.071270 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.071166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:54.294659 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.294624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:54.631595 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.631567 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:54.631814 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:54.631677 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:54.671064 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.671035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:54.803008 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.802981 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:32:54.803390 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.803361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" event={"ID":"f88cd6c6-7717-46de-8c19-a15c236db534","Type":"ContainerStarted","Data":"1c48337fad8897c3927c8aa45fecfd8f39bfc77d51cda48fef2bd8be5bd67963"} Jul 13 17:32:54.832465 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.832415 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" podStartSLOduration=10.391253328 podStartE2EDuration="29.832399267s" totalImagesPullingTime="19.441145939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:32:54.832163425 +0000 UTC m=+29.839565669" watchObservedRunningTime="2026-07-13 17:32:54.832399267 +0000 UTC m=+29.839801509" Jul 13 17:32:54.869811 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.869748 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:54.973510 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:54.973450 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.071867 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.071835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.171922 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.171897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.272144 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.272073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.550706 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.550676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.635094 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.635066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:55.635284 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:55.635152 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:55.635284 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.635208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:55.635400 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:55.635319 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:55.651618 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.651589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.752052 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.752020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.807815 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.807737 2564 generic.go:356] "Generic (PLEG): container finished" podID="57acac65-9ea0-4727-90d8-ac9ade3c7055" containerID="602f1f92ca6b87a9190dae2b44929a1f850b8e8907159251af0dba40860e868d" exitCode=0 Jul 13 17:32:55.807815 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.807799 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerDied","Data":"602f1f92ca6b87a9190dae2b44929a1f850b8e8907159251af0dba40860e868d"} Jul 13 17:32:55.874533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.874502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:55.972003 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:55.971973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:56.095338 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.095264 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:56.095480 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.095381 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 13 17:32:56.095844 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.095828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-27mpb" Jul 13 17:32:56.183651 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.183594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:56.281284 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.281251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:56.382109 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.382023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:56.479274 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.479223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:56.535200 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.535164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:56.591308 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.591282 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:56.632384 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.632315 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:56.632507 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:56.632430 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:56.873941 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:56.873907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:57.286109 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:57.286081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:57.341188 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:57.341155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:57.632412 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:57.632182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:57.632782 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:57.632219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:57.632782 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:57.632467 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:57.632782 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:57.632507 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:57.791848 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:57.791814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:57.813754 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:57.813724 2564 generic.go:356] "Generic (PLEG): container finished" podID="57acac65-9ea0-4727-90d8-ac9ade3c7055" containerID="73d76e983392fead14acff0f9b62fb78c43a8190c8e258b65741e684930bd8a6" exitCode=0 Jul 13 17:32:57.813875 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:57.813783 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerDied","Data":"73d76e983392fead14acff0f9b62fb78c43a8190c8e258b65741e684930bd8a6"} Jul 13 17:32:58.632122 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:58.632073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:58.632321 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:58.632284 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:32:59.193930 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.193888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:32:59.194405 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.193939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:59.194405 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.194012 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:59.194405 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.194076 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:59.194405 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.194087 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs podName:bbc4ffda-5366-46b0-88bc-32e697a9d2ff nodeName:}" failed. No retries permitted until 2026-07-13 17:33:31.194064973 +0000 UTC m=+66.201467199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs") pod "network-metrics-daemon-7xvlp" (UID: "bbc4ffda-5366-46b0-88bc-32e697a9d2ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 17:32:59.194405 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.194120 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret podName:30053369-dbea-4ebb-8e82-4ee42689e422 nodeName:}" failed. No retries permitted until 2026-07-13 17:33:31.194103342 +0000 UTC m=+66.201505567 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret") pod "global-pull-secret-syncer-n8t6r" (UID: "30053369-dbea-4ebb-8e82-4ee42689e422") : object "kube-system"/"original-pull-secret" not registered Jul 13 17:32:59.294841 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.294792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:59.295038 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.294937 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 17:32:59.295038 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.294965 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 17:32:59.295038 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.294978 2564 projected.go:196] Error preparing data for projected volume kube-api-access-sppk5 for pod openshift-network-diagnostics/network-check-target-fx7fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:59.295208 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.295045 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5 podName:06123f7e-7ce5-4f55-8b0a-f858c91860a0 nodeName:}" failed. No retries permitted until 2026-07-13 17:33:31.295024421 +0000 UTC m=+66.302426660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sppk5" (UniqueName: "kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5") pod "network-check-target-fx7fh" (UID: "06123f7e-7ce5-4f55-8b0a-f858c91860a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 17:32:59.332939 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.332902 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:59.410432 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.410362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:59.523778 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.523688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:59.616738 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.616588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:59.631958 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.631906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:32:59.632214 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.632035 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:32:59.632214 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.632082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:32:59.632214 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:32:59.632194 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:32:59.703249 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.703179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:32:59.761674 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:32:59.761637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:00.567333 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:00.567288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fx7fh"] Jul 13 17:33:00.568069 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:00.567468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:00.568069 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:00.567613 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:33:00.568191 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:00.568161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7xvlp"] Jul 13 17:33:00.568635 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:00.568609 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:33:00.568777 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:00.568755 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:33:00.578610 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:00.578548 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n8t6r"] Jul 13 17:33:00.578767 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:00.578657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:33:00.578767 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:00.578750 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:33:01.626961 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:01.626886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:01.687981 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:01.687933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:01.901174 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:01.901149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:02.401321 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:02.401134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:02.631568 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:02.631540 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:02.631933 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:02.631541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:33:02.631933 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:02.631635 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:33:02.631933 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:02.631541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:33:02.631933 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:02.631719 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:33:02.631933 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:02.631775 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:33:02.828081 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:02.828048 2564 generic.go:356] "Generic (PLEG): container finished" podID="57acac65-9ea0-4727-90d8-ac9ade3c7055" containerID="97ba7af6fc00356aa7e52a74f4e316d6d49fc6223bcedb01ea1c36e0435834af" exitCode=0 Jul 13 17:33:02.828249 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:02.828119 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerDied","Data":"97ba7af6fc00356aa7e52a74f4e316d6d49fc6223bcedb01ea1c36e0435834af"} Jul 13 17:33:03.833470 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:03.833438 2564 generic.go:356] "Generic (PLEG): container finished" podID="57acac65-9ea0-4727-90d8-ac9ade3c7055" containerID="15dddaf854c75129235bc97aa88c34329641271896b081cef3ce549675c182c4" exitCode=0 Jul 13 17:33:03.833922 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:03.833496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerDied","Data":"15dddaf854c75129235bc97aa88c34329641271896b081cef3ce549675c182c4"} Jul 13 17:33:04.631651 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.631623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:04.631768 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.631724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:33:04.631768 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:04.631735 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fx7fh" podUID="06123f7e-7ce5-4f55-8b0a-f858c91860a0" Jul 13 17:33:04.631768 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.631751 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:33:04.631916 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:04.631819 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n8t6r" podUID="30053369-dbea-4ebb-8e82-4ee42689e422" Jul 13 17:33:04.631916 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:04.631900 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7xvlp" podUID="bbc4ffda-5366-46b0-88bc-32e697a9d2ff" Jul 13 17:33:04.817730 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.817675 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-16.ec2.internal" event="NodeReady" Jul 13 17:33:04.817814 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.817772 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 17:33:04.839242 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.839215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" event={"ID":"57acac65-9ea0-4727-90d8-ac9ade3c7055","Type":"ContainerStarted","Data":"00d9a743484b9b9fc497900b719636e0436599bc76c3f097bc3416d105b5e311"} Jul 13 17:33:04.863623 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.863582 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bcvkf" podStartSLOduration=8.542322434999999 podStartE2EDuration="39.86356502s" totalImagesPullingTime="31.321242585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:04.862043841 +0000 UTC m=+39.869446084" watchObservedRunningTime="2026-07-13 17:33:04.86356502 +0000 UTC m=+39.870967266" Jul 13 17:33:04.868751 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.868732 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nvsmc"] Jul 13 17:33:04.893082 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.893063 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hdllh"] Jul 13 17:33:04.893207 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.893188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:04.895206 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.895186 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 17:33:04.895309 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.895287 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-225kz\"" Jul 13 17:33:04.895382 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.895366 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 17:33:04.913489 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.913467 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdllh"] Jul 13 17:33:04.913569 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.913495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nvsmc"] Jul 13 17:33:04.913569 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.913566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:04.915618 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.915604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 17:33:04.915777 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.915764 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 17:33:04.915913 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.915900 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dnkkg\"" Jul 13 17:33:04.915980 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.915939 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 17:33:04.965279 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.965255 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7qr6m"] Jul 13 17:33:04.984406 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.984385 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qr6m"] Jul 13 17:33:04.984484 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.984460 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:04.988663 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.988641 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 17:33:04.988778 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.988664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 17:33:04.988778 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.988673 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 17:33:04.988880 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.988851 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6jxkx\"" Jul 13 17:33:04.989052 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.989030 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 17:33:04.993714 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:04.993691 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:05.043136 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.043113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7tnf\" (UniqueName: \"kubernetes.io/projected/3636a3d0-240f-4f02-a5dd-19c8530fd533-kube-api-access-d7tnf\") pod \"ingress-canary-hdllh\" (UID: \"3636a3d0-240f-4f02-a5dd-19c8530fd533\") " pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:05.043221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.043146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3636a3d0-240f-4f02-a5dd-19c8530fd533-cert\") pod \"ingress-canary-hdllh\" (UID: \"3636a3d0-240f-4f02-a5dd-19c8530fd533\") " pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:05.043221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.043200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15fb73-676b-4e3a-9f03-e7a50131d82e-config-volume\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.043360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.043251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kwj\" (UniqueName: \"kubernetes.io/projected/ff15fb73-676b-4e3a-9f03-e7a50131d82e-kube-api-access-k8kwj\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.043360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.043293 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff15fb73-676b-4e3a-9f03-e7a50131d82e-tmp-dir\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.043360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.043326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff15fb73-676b-4e3a-9f03-e7a50131d82e-metrics-tls\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.143959 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.143910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ead76a25-f4a9-453c-b1bb-4449cff1edc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.143959 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.143938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kwj\" (UniqueName: \"kubernetes.io/projected/ff15fb73-676b-4e3a-9f03-e7a50131d82e-kube-api-access-k8kwj\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.143959 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.143956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsb87\" (UniqueName: \"kubernetes.io/projected/ead76a25-f4a9-453c-b1bb-4449cff1edc0-kube-api-access-dsb87\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.144155 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.143982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff15fb73-676b-4e3a-9f03-e7a50131d82e-tmp-dir\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.144155 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff15fb73-676b-4e3a-9f03-e7a50131d82e-metrics-tls\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.144155 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead76a25-f4a9-453c-b1bb-4449cff1edc0-data-volume\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.144155 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ead76a25-f4a9-453c-b1bb-4449cff1edc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.144386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7tnf\" (UniqueName: \"kubernetes.io/projected/3636a3d0-240f-4f02-a5dd-19c8530fd533-kube-api-access-d7tnf\") pod \"ingress-canary-hdllh\" (UID: \"3636a3d0-240f-4f02-a5dd-19c8530fd533\") " pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:05.144386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3636a3d0-240f-4f02-a5dd-19c8530fd533-cert\") pod \"ingress-canary-hdllh\" (UID: \"3636a3d0-240f-4f02-a5dd-19c8530fd533\") " pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:05.144386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ead76a25-f4a9-453c-b1bb-4449cff1edc0-crio-socket\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.144386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15fb73-676b-4e3a-9f03-e7a50131d82e-config-volume\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.144386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144310 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff15fb73-676b-4e3a-9f03-e7a50131d82e-tmp-dir\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.144786 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.144767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15fb73-676b-4e3a-9f03-e7a50131d82e-config-volume\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.147809 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.147791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff15fb73-676b-4e3a-9f03-e7a50131d82e-metrics-tls\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.147942 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.147927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3636a3d0-240f-4f02-a5dd-19c8530fd533-cert\") pod \"ingress-canary-hdllh\" (UID: \"3636a3d0-240f-4f02-a5dd-19c8530fd533\") " pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:05.156288 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.156271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kwj\" (UniqueName: \"kubernetes.io/projected/ff15fb73-676b-4e3a-9f03-e7a50131d82e-kube-api-access-k8kwj\") pod \"dns-default-nvsmc\" (UID: \"ff15fb73-676b-4e3a-9f03-e7a50131d82e\") " pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.156379 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.156288 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7tnf\" (UniqueName: \"kubernetes.io/projected/3636a3d0-240f-4f02-a5dd-19c8530fd533-kube-api-access-d7tnf\") pod \"ingress-canary-hdllh\" (UID: \"3636a3d0-240f-4f02-a5dd-19c8530fd533\") " pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:05.197797 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.197778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:05.201792 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.201774 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:05.221740 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.221723 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdllh" Jul 13 17:33:05.244843 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.244821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead76a25-f4a9-453c-b1bb-4449cff1edc0-data-volume\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.244929 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.244853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ead76a25-f4a9-453c-b1bb-4449cff1edc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.244929 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.244879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ead76a25-f4a9-453c-b1bb-4449cff1edc0-crio-socket\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.245033 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.245015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ead76a25-f4a9-453c-b1bb-4449cff1edc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.245121 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.245047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ead76a25-f4a9-453c-b1bb-4449cff1edc0-crio-socket\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.245121 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.245053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsb87\" (UniqueName: \"kubernetes.io/projected/ead76a25-f4a9-453c-b1bb-4449cff1edc0-kube-api-access-dsb87\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.245317 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.245298 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ead76a25-f4a9-453c-b1bb-4449cff1edc0-data-volume\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.245760 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.245731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ead76a25-f4a9-453c-b1bb-4449cff1edc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.247358 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.247336 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ead76a25-f4a9-453c-b1bb-4449cff1edc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.258998 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.258975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsb87\" (UniqueName: \"kubernetes.io/projected/ead76a25-f4a9-453c-b1bb-4449cff1edc0-kube-api-access-dsb87\") pod \"insights-runtime-extractor-7qr6m\" (UID: \"ead76a25-f4a9-453c-b1bb-4449cff1edc0\") " pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.295254 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.295198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qr6m" Jul 13 17:33:05.330542 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.330513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:05.378220 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.378196 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nvsmc"] Jul 13 17:33:05.381874 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:05.381849 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff15fb73_676b_4e3a_9f03_e7a50131d82e.slice/crio-8d87a79cd9b7ab2e55b80dbaf0da813b5cc9423ce439567631808832174ecaf6 WatchSource:0}: Error finding container 8d87a79cd9b7ab2e55b80dbaf0da813b5cc9423ce439567631808832174ecaf6: Status 404 returned error can't find the container with id 8d87a79cd9b7ab2e55b80dbaf0da813b5cc9423ce439567631808832174ecaf6 Jul 13 17:33:05.397451 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.397389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdllh"] Jul 13 17:33:05.397890 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.397864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:05.400559 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:05.400536 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3636a3d0_240f_4f02_a5dd_19c8530fd533.slice/crio-c0d3b9ac3c72d82c96a5f3f2b2d03737fe794a8a62fda919e01694f4cc9e78bb WatchSource:0}: Error finding container c0d3b9ac3c72d82c96a5f3f2b2d03737fe794a8a62fda919e01694f4cc9e78bb: Status 404 returned error can't find the container with id c0d3b9ac3c72d82c96a5f3f2b2d03737fe794a8a62fda919e01694f4cc9e78bb Jul 13 17:33:05.434201 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.434181 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qr6m"] Jul 13 17:33:05.437005 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:05.436987 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead76a25_f4a9_453c_b1bb_4449cff1edc0.slice/crio-7151c2465672f8d7e4ea110b7ea15a0b3959b6fdcdec09f2111d0b5588ba8f7c WatchSource:0}: Error finding container 7151c2465672f8d7e4ea110b7ea15a0b3959b6fdcdec09f2111d0b5588ba8f7c: Status 404 returned error can't find the container with id 7151c2465672f8d7e4ea110b7ea15a0b3959b6fdcdec09f2111d0b5588ba8f7c Jul 13 17:33:05.651985 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.651928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:05.748761 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.748730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:05.842795 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.842770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nvsmc" event={"ID":"ff15fb73-676b-4e3a-9f03-e7a50131d82e","Type":"ContainerStarted","Data":"8d87a79cd9b7ab2e55b80dbaf0da813b5cc9423ce439567631808832174ecaf6"} Jul 13 17:33:05.843905 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.843873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qr6m" event={"ID":"ead76a25-f4a9-453c-b1bb-4449cff1edc0","Type":"ContainerStarted","Data":"7151c2465672f8d7e4ea110b7ea15a0b3959b6fdcdec09f2111d0b5588ba8f7c"} Jul 13 17:33:05.844982 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:05.844962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdllh" event={"ID":"3636a3d0-240f-4f02-a5dd-19c8530fd533","Type":"ContainerStarted","Data":"c0d3b9ac3c72d82c96a5f3f2b2d03737fe794a8a62fda919e01694f4cc9e78bb"} Jul 13 17:33:06.040582 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.040554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:06.140978 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.140912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:06.216927 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.216900 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84"] Jul 13 17:33:06.217674 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.217551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:06.242013 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.241990 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84"] Jul 13 17:33:06.242151 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.242111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:06.244181 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.244160 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-crqbt\"" Jul 13 17:33:06.244315 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.244186 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 13 17:33:06.353071 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.352879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pdl84\" (UID: \"9eb98afd-67bb-4a40-bc76-166bede84957\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:06.454021 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.453569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pdl84\" (UID: \"9eb98afd-67bb-4a40-bc76-166bede84957\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:06.454021 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:06.453720 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 13 17:33:06.454021 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:06.453782 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates podName:9eb98afd-67bb-4a40-bc76-166bede84957 nodeName:}" failed. No retries permitted until 2026-07-13 17:33:06.953763043 +0000 UTC m=+41.961165269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-pdl84" (UID: "9eb98afd-67bb-4a40-bc76-166bede84957") : secret "prometheus-operator-admission-webhook-tls" not found Jul 13 17:33:06.632575 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.632500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:06.632841 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.632818 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:33:06.634835 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.633032 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:33:06.637602 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.637512 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 17:33:06.638343 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.637967 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 17:33:06.641495 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.638618 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 17:33:06.641495 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.639037 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4jvwf\"" Jul 13 17:33:06.644264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.644221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vn7td\"" Jul 13 17:33:06.648021 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.647807 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 17:33:06.959388 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:06.959308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pdl84\" (UID: \"9eb98afd-67bb-4a40-bc76-166bede84957\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:06.959818 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:06.959492 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 13 17:33:06.959818 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:06.959550 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates podName:9eb98afd-67bb-4a40-bc76-166bede84957 nodeName:}" failed. No retries permitted until 2026-07-13 17:33:07.95952913 +0000 UTC m=+42.966931352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-pdl84" (UID: "9eb98afd-67bb-4a40-bc76-166bede84957") : secret "prometheus-operator-admission-webhook-tls" not found Jul 13 17:33:07.668997 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:07.668958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:07.967871 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:07.967780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pdl84\" (UID: \"9eb98afd-67bb-4a40-bc76-166bede84957\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:07.970452 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:07.970420 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9eb98afd-67bb-4a40-bc76-166bede84957-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-pdl84\" (UID: \"9eb98afd-67bb-4a40-bc76-166bede84957\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:08.057349 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:08.057321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:08.217465 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:08.217434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:08.284088 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:08.284052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:10.181669 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:10.181623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:10.238311 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:10.238208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:10.749151 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:10.749114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:10.848518 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:10.848479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:11.726181 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.726156 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84"] Jul 13 17:33:11.729154 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:11.729112 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb98afd_67bb_4a40_bc76_166bede84957.slice/crio-91b2641ef88c7ee7a14a7e2a504561a358eb9203d0300cef5780ff83ea0bbd7a WatchSource:0}: Error finding container 91b2641ef88c7ee7a14a7e2a504561a358eb9203d0300cef5780ff83ea0bbd7a: Status 404 returned error can't find the container with id 91b2641ef88c7ee7a14a7e2a504561a358eb9203d0300cef5780ff83ea0bbd7a Jul 13 17:33:11.803704 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.803528 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85749c8588-g2jbb"] Jul 13 17:33:11.810371 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.810348 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:11.816286 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.816267 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.817857 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.817833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85749c8588-g2jbb"] Jul 13 17:33:11.819473 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.819450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 17:33:11.819733 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.819715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cjph7\"" Jul 13 17:33:11.819960 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.819942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 17:33:11.822564 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.820601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 17:33:11.824264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.823542 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 17:33:11.824264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.823615 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 17:33:11.824264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.823781 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 17:33:11.824264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.823833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 17:33:11.831849 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.831826 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 17:33:11.864291 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.864208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdllh" event={"ID":"3636a3d0-240f-4f02-a5dd-19c8530fd533","Type":"ContainerStarted","Data":"61256aee8c058911ea6f615f146256d8b5ece9c2474d35698eb5fc7e3848da04"} Jul 13 17:33:11.866249 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.866203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nvsmc" event={"ID":"ff15fb73-676b-4e3a-9f03-e7a50131d82e","Type":"ContainerStarted","Data":"1d399266f455faf3b054367e8ba4029e29572ba8d69a76286cd40027c1a81c11"} Jul 13 17:33:11.867647 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.867625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" event={"ID":"9eb98afd-67bb-4a40-bc76-166bede84957","Type":"ContainerStarted","Data":"91b2641ef88c7ee7a14a7e2a504561a358eb9203d0300cef5780ff83ea0bbd7a"} Jul 13 17:33:11.869306 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.869275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:11.879966 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.879912 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hdllh" podStartSLOduration=1.677397813 podStartE2EDuration="7.879894783s" totalImagesPullingTime="6.20249697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:11.879185126 +0000 UTC m=+46.886587371" watchObservedRunningTime="2026-07-13 17:33:11.879894783 +0000 UTC m=+46.887297028" Jul 13 17:33:11.897563 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.897545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-serving-cert\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.898188 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.898172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-config\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.898332 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.898314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-service-ca\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.898404 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.898345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-trusted-ca-bundle\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.898404 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.898384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-oauth-serving-cert\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.898511 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.898419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vslp\" (UniqueName: \"kubernetes.io/projected/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-kube-api-access-6vslp\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.898565 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.898532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-oauth-config\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:11.999894 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.999861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-oauth-config\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.000023 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:11.999910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-serving-cert\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.000023 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.000004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-config\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.000129 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.000044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-service-ca\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.000184 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.000143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-trusted-ca-bundle\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.000283 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.000197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-oauth-serving-cert\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.000283 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.000256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vslp\" (UniqueName: \"kubernetes.io/projected/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-kube-api-access-6vslp\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.000715 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.000696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-config\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.001012 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.000987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-service-ca\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.001155 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.001132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-trusted-ca-bundle\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.001452 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.001432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-oauth-serving-cert\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.002355 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.002340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-oauth-config\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.002498 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.002481 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-serving-cert\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.008945 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.008928 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vslp\" (UniqueName: \"kubernetes.io/projected/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-kube-api-access-6vslp\") pod \"console-85749c8588-g2jbb\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.145552 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.145501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:12.292407 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.292377 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85749c8588-g2jbb"] Jul 13 17:33:12.295334 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:12.295292 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca7519c_c7ac_4f10_8ebc_41dd0eac624f.slice/crio-53aca214b9e28ac55c0c3247da4b78b3d3f1405ade812090fa6783861547c1bc WatchSource:0}: Error finding container 53aca214b9e28ac55c0c3247da4b78b3d3f1405ade812090fa6783861547c1bc: Status 404 returned error can't find the container with id 53aca214b9e28ac55c0c3247da4b78b3d3f1405ade812090fa6783861547c1bc Jul 13 17:33:12.872052 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.872014 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qr6m" event={"ID":"ead76a25-f4a9-453c-b1bb-4449cff1edc0","Type":"ContainerStarted","Data":"0ddb1aa9163aa2c0fcc450b929c163b95609f8976431fbbf813d6e5d0d6f4b24"} Jul 13 17:33:12.873875 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.873850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nvsmc" event={"ID":"ff15fb73-676b-4e3a-9f03-e7a50131d82e","Type":"ContainerStarted","Data":"e0e868870d986753436282a779e41a8b1fcb1a01c2ee9132f1803e8453d8e771"} Jul 13 17:33:12.874074 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.874056 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:12.875192 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.875165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85749c8588-g2jbb" event={"ID":"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f","Type":"ContainerStarted","Data":"53aca214b9e28ac55c0c3247da4b78b3d3f1405ade812090fa6783861547c1bc"} Jul 13 17:33:12.895118 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:12.895073 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nvsmc" podStartSLOduration=2.674186496 podStartE2EDuration="8.895049047s" totalImagesPullingTime="6.220862551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:12.893891536 +0000 UTC m=+47.901293781" watchObservedRunningTime="2026-07-13 17:33:12.895049047 +0000 UTC m=+47.902451310" Jul 13 17:33:13.883936 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:13.883846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qr6m" event={"ID":"ead76a25-f4a9-453c-b1bb-4449cff1edc0","Type":"ContainerStarted","Data":"f559a718c90665078622323f620e4996c3394bfbab84c588290dbe154aeef829"} Jul 13 17:33:13.885721 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:13.885693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" event={"ID":"9eb98afd-67bb-4a40-bc76-166bede84957","Type":"ContainerStarted","Data":"a82f6bb11463063936dc7dfa81076bb4dd9aa82c4a284086d17022453080e891"} Jul 13 17:33:13.899755 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:13.899714 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" podStartSLOduration=6.093335724 podStartE2EDuration="7.899702507s" totalImagesPullingTime="1.806366783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:13.899373549 +0000 UTC m=+48.906775792" watchObservedRunningTime="2026-07-13 17:33:13.899702507 +0000 UTC m=+48.907104750" Jul 13 17:33:14.622260 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:14.622207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:14.724127 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:14.724031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:14.889431 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:14.889355 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:14.895137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:14.895111 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-pdl84" Jul 13 17:33:15.091139 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.091100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:15.245333 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.245213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:15.282747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.282719 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6w656"] Jul 13 17:33:15.300022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.299986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6w656"] Jul 13 17:33:15.300159 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.300104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.305511 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.305491 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 17:33:15.309923 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.309709 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 17:33:15.309923 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.309741 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 13 17:33:15.309923 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.309759 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 13 17:33:15.309923 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.309709 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nxm4j\"" Jul 13 17:33:15.309923 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.309920 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 17:33:15.326824 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.326790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9498398-0795-44e3-a8d2-4de79b211633-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.326936 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.326840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9498398-0795-44e3-a8d2-4de79b211633-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.326936 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.326914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9498398-0795-44e3-a8d2-4de79b211633-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.327045 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.326953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5zx\" (UniqueName: \"kubernetes.io/projected/f9498398-0795-44e3-a8d2-4de79b211633-kube-api-access-wx5zx\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.427292 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.427254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9498398-0795-44e3-a8d2-4de79b211633-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.427422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.427324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9498398-0795-44e3-a8d2-4de79b211633-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.427422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.427359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5zx\" (UniqueName: \"kubernetes.io/projected/f9498398-0795-44e3-a8d2-4de79b211633-kube-api-access-wx5zx\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.427627 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.427520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9498398-0795-44e3-a8d2-4de79b211633-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.428258 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.428169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:15.428440 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.428421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9498398-0795-44e3-a8d2-4de79b211633-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.430003 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.429978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9498398-0795-44e3-a8d2-4de79b211633-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.430102 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.430044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9498398-0795-44e3-a8d2-4de79b211633-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.440659 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.440639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5zx\" (UniqueName: \"kubernetes.io/projected/f9498398-0795-44e3-a8d2-4de79b211633-kube-api-access-wx5zx\") pod \"prometheus-operator-64785b6675-6w656\" (UID: \"f9498398-0795-44e3-a8d2-4de79b211633\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.485368 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.485340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:15.611541 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.611509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" Jul 13 17:33:15.709333 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:15.709301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:17.814073 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:17.814049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6w656"] Jul 13 17:33:17.817045 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:17.817022 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9498398_0795_44e3_a8d2_4de79b211633.slice/crio-79eca377f76b9952feb0287f348ca93bef0bfd8d362bfcdb18eef27dcfa6bd6f WatchSource:0}: Error finding container 79eca377f76b9952feb0287f348ca93bef0bfd8d362bfcdb18eef27dcfa6bd6f: Status 404 returned error can't find the container with id 79eca377f76b9952feb0287f348ca93bef0bfd8d362bfcdb18eef27dcfa6bd6f Jul 13 17:33:17.900185 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:17.900155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qr6m" event={"ID":"ead76a25-f4a9-453c-b1bb-4449cff1edc0","Type":"ContainerStarted","Data":"f2d36935045dedff6aa8223b0a5e31b3660661348ab8c61a2cda3885452da3b2"} Jul 13 17:33:17.901664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:17.901642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85749c8588-g2jbb" event={"ID":"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f","Type":"ContainerStarted","Data":"b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a"} Jul 13 17:33:17.902774 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:17.902753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" event={"ID":"f9498398-0795-44e3-a8d2-4de79b211633","Type":"ContainerStarted","Data":"79eca377f76b9952feb0287f348ca93bef0bfd8d362bfcdb18eef27dcfa6bd6f"} Jul 13 17:33:17.916570 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:17.916518 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7qr6m" podStartSLOduration=2.051318853 podStartE2EDuration="13.916504148s" totalImagesPullingTime="11.865185295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:33:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:17.915692615 +0000 UTC m=+52.923094861" watchObservedRunningTime="2026-07-13 17:33:17.916504148 +0000 UTC m=+52.923906392" Jul 13 17:33:17.931697 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:17.931623 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85749c8588-g2jbb" podStartSLOduration=1.53160158 podStartE2EDuration="6.931609538s" totalImagesPullingTime="5.400007958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:17.93043095 +0000 UTC m=+52.937833194" watchObservedRunningTime="2026-07-13 17:33:17.931609538 +0000 UTC m=+52.939011781" Jul 13 17:33:19.911315 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:19.911273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" event={"ID":"f9498398-0795-44e3-a8d2-4de79b211633","Type":"ContainerStarted","Data":"7fa379b69c33e11086d816bd7031418eae04102d9065d33c9799d0b7e64185d3"} Jul 13 17:33:19.998018 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:19.997992 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:20.091310 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:20.091216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:20.145674 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:20.145642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:20.916003 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:20.915969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" event={"ID":"f9498398-0795-44e3-a8d2-4de79b211633","Type":"ContainerStarted","Data":"9d6e51d7ac2faa9e36ead8b869e6d90e86efce9ff9dce4ee67a53f7b4e77007b"} Jul 13 17:33:21.183091 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:21.183018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:21.283877 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:21.283848 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:21.686752 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:21.686709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:21.881155 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:21.881125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:21.938999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:21.938921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:22.146387 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.146358 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:22.146540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.146414 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:22.150972 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.150950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:22.167620 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.167571 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-6w656" podStartSLOduration=5.199676912 podStartE2EDuration="7.167557587s" totalImagesPullingTime="1.967880675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:20.931421494 +0000 UTC m=+55.938823737" watchObservedRunningTime="2026-07-13 17:33:22.167557587 +0000 UTC m=+57.174959830" Jul 13 17:33:22.387206 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.387171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:22.578038 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.578003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:22.616628 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.616590 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nd4nm"] Jul 13 17:33:22.619391 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.619366 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.621938 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.621913 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 13 17:33:22.622087 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.621958 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 13 17:33:22.622604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.622591 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lh2bq\"" Jul 13 17:33:22.624894 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.624874 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 13 17:33:22.634183 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.634156 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nd4nm"] Jul 13 17:33:22.644285 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.644199 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5"] Jul 13 17:33:22.646473 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.646454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.649161 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.649140 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 13 17:33:22.649293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.649161 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kcxgz\"" Jul 13 17:33:22.649357 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.649149 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 13 17:33:22.659325 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.659304 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5"] Jul 13 17:33:22.660434 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.660418 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9nr6w"] Jul 13 17:33:22.662765 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.662748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.664924 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.664904 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 17:33:22.665016 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.664937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vzcd9\"" Jul 13 17:33:22.665064 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.665043 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 17:33:22.665321 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.665307 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 17:33:22.679431 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.679561 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679448 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.679561 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.679561 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49dfab4f-19d7-459a-9634-023750e51552-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.679747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49dfab4f-19d7-459a-9634-023750e51552-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.679747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrf7\" (UniqueName: \"kubernetes.io/projected/49dfab4f-19d7-459a-9634-023750e51552-kube-api-access-cwrf7\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.679747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.679747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.679747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.679747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9lf\" (UniqueName: \"kubernetes.io/projected/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-kube-api-access-2x9lf\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.680050 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.679978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:22.751354 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.751323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:22.780592 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9lf\" (UniqueName: \"kubernetes.io/projected/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-kube-api-access-2x9lf\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.780739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-metrics-client-ca\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.780739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.780739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbj5\" (UniqueName: \"kubernetes.io/projected/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-kube-api-access-lgbj5\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.780739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.780739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.780996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-tls\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.780996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.780996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780816 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-sys\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.780996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49dfab4f-19d7-459a-9634-023750e51552-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.780996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49dfab4f-19d7-459a-9634-023750e51552-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.780996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrf7\" (UniqueName: \"kubernetes.io/projected/49dfab4f-19d7-459a-9634-023750e51552-kube-api-access-cwrf7\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.780996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.780998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-root\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-wtmp\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-textfile\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/49dfab4f-19d7-459a-9634-023750e51552-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.781360 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:22.781291 2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 13 17:33:22.781738 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:22.781371 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls podName:ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a nodeName:}" failed. No retries permitted until 2026-07-13 17:33:23.281352477 +0000 UTC m=+58.288754718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-hg4v5" (UID: "ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a") : secret "openshift-state-metrics-tls" not found Jul 13 17:33:22.781738 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49dfab4f-19d7-459a-9634-023750e51552-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.781738 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.782004 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.781886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.783330 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.783308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.783441 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.783360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.783441 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.783373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/49dfab4f-19d7-459a-9634-023750e51552-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.792500 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.792472 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrf7\" (UniqueName: \"kubernetes.io/projected/49dfab4f-19d7-459a-9634-023750e51552-kube-api-access-cwrf7\") pod \"kube-state-metrics-785778ff74-nd4nm\" (UID: \"49dfab4f-19d7-459a-9634-023750e51552\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.792722 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.792702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9lf\" (UniqueName: \"kubernetes.io/projected/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-kube-api-access-2x9lf\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:22.881738 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.881712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-metrics-client-ca\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.881841 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.881751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbj5\" (UniqueName: \"kubernetes.io/projected/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-kube-api-access-lgbj5\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.881841 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.881804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.881841 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.881838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-tls\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.881999 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:22.881903 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 17:33:22.881999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.881949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-sys\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.881999 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:22.881961 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-tls podName:15e4507a-4eb4-4e02-8afd-e6b9055e9af8 nodeName:}" failed. No retries permitted until 2026-07-13 17:33:23.38194371 +0000 UTC m=+58.389345932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-tls") pod "node-exporter-9nr6w" (UID: "15e4507a-4eb4-4e02-8afd-e6b9055e9af8") : secret "node-exporter-tls" not found Jul 13 17:33:22.882153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-root\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882056 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-sys\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-wtmp\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-textfile\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-root\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882475 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882344 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882475 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-textfile\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882475 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-metrics-client-ca\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.882475 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.882473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-wtmp\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.883996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.883975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.887666 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.887634 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nvsmc" Jul 13 17:33:22.889927 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.889908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbj5\" (UniqueName: \"kubernetes.io/projected/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-kube-api-access-lgbj5\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:22.928322 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.928268 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" Jul 13 17:33:22.929512 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.929491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:22.929880 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:22.929861 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:33:23.006206 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.006172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.061443 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.061414 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nd4nm"] Jul 13 17:33:23.064118 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:23.064093 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49dfab4f_19d7_459a_9634_023750e51552.slice/crio-b9a7f26e6d4ab185110169ba868d6c383ed56589033d530c6c384197fa805bbc WatchSource:0}: Error finding container b9a7f26e6d4ab185110169ba868d6c383ed56589033d530c6c384197fa805bbc: Status 404 returned error can't find the container with id b9a7f26e6d4ab185110169ba868d6c383ed56589033d530c6c384197fa805bbc Jul 13 17:33:23.073629 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.073604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.249344 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.249290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.285785 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.285761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:23.285916 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:23.285900 2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 13 17:33:23.285970 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:23.285960 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls podName:ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a nodeName:}" failed. No retries permitted until 2026-07-13 17:33:24.28594238 +0000 UTC m=+59.293344605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-hg4v5" (UID: "ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a") : secret "openshift-state-metrics-tls" not found Jul 13 17:33:23.348066 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.348033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.386663 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.386635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-tls\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:23.388800 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.388784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/15e4507a-4eb4-4e02-8afd-e6b9055e9af8-node-exporter-tls\") pod \"node-exporter-9nr6w\" (UID: \"15e4507a-4eb4-4e02-8afd-e6b9055e9af8\") " pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:23.511410 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.511346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.571695 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.571670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9nr6w" Jul 13 17:33:23.581044 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:23.581019 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e4507a_4eb4_4e02_8afd_e6b9055e9af8.slice/crio-186ebcc76d992b3ee7a7f02c4fa081fd8295e106d201cdb57bd8d3d882150d75 WatchSource:0}: Error finding container 186ebcc76d992b3ee7a7f02c4fa081fd8295e106d201cdb57bd8d3d882150d75: Status 404 returned error can't find the container with id 186ebcc76d992b3ee7a7f02c4fa081fd8295e106d201cdb57bd8d3d882150d75 Jul 13 17:33:23.615085 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.615046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.680444 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.680416 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 17:33:23.683835 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.683806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.683941 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.683867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.685904 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.685889 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 13 17:33:23.685988 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.685904 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 13 17:33:23.686138 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.686121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 13 17:33:23.686194 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.686157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 13 17:33:23.686194 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.686185 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 13 17:33:23.686322 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.686219 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jbbc9\"" Jul 13 17:33:23.686452 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.686436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 13 17:33:23.686535 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.686519 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 13 17:33:23.686589 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.686539 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 13 17:33:23.696154 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.694503 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 13 17:33:23.701173 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.701155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 17:33:23.789192 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789344 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ebb5417-10a2-4288-beea-1a8ee2c30776-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789344 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789344 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb5417-10a2-4288-beea-1a8ee2c30776-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789451 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789451 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789451 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789542 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ebb5417-10a2-4288-beea-1a8ee2c30776-config-out\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789542 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ebb5417-10a2-4288-beea-1a8ee2c30776-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789542 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ebb5417-10a2-4288-beea-1a8ee2c30776-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789640 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789640 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-web-config\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.789640 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.789586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x55qk\" (UniqueName: \"kubernetes.io/projected/3ebb5417-10a2-4288-beea-1a8ee2c30776-kube-api-access-x55qk\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.823853 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.823830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.879070 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.879046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:23.890157 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890255 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ebb5417-10a2-4288-beea-1a8ee2c30776-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890255 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890255 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb5417-10a2-4288-beea-1a8ee2c30776-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890255 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890445 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890445 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890445 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ebb5417-10a2-4288-beea-1a8ee2c30776-config-out\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.890445 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.890345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ebb5417-10a2-4288-beea-1a8ee2c30776-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.891264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.891217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ebb5417-10a2-4288-beea-1a8ee2c30776-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.891366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.891319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.891366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.891354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-web-config\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.891473 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.891383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55qk\" (UniqueName: \"kubernetes.io/projected/3ebb5417-10a2-4288-beea-1a8ee2c30776-kube-api-access-x55qk\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.891531 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.891473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ebb5417-10a2-4288-beea-1a8ee2c30776-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.891719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.891473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ebb5417-10a2-4288-beea-1a8ee2c30776-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.891799 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.891720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ebb5417-10a2-4288-beea-1a8ee2c30776-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.893068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.893037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.893165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.893112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.893500 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.893458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ebb5417-10a2-4288-beea-1a8ee2c30776-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.893722 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.893702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ebb5417-10a2-4288-beea-1a8ee2c30776-config-out\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.893782 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.893749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.894119 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.894097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.894466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.894445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-web-config\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.894562 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.894543 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.894695 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.894681 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ebb5417-10a2-4288-beea-1a8ee2c30776-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.899435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.899418 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55qk\" (UniqueName: \"kubernetes.io/projected/3ebb5417-10a2-4288-beea-1a8ee2c30776-kube-api-access-x55qk\") pod \"alertmanager-main-0\" (UID: \"3ebb5417-10a2-4288-beea-1a8ee2c30776\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:23.929278 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.929254 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9nr6w" event={"ID":"15e4507a-4eb4-4e02-8afd-e6b9055e9af8","Type":"ContainerStarted","Data":"186ebcc76d992b3ee7a7f02c4fa081fd8295e106d201cdb57bd8d3d882150d75"} Jul 13 17:33:23.930454 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.930436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" event={"ID":"49dfab4f-19d7-459a-9634-023750e51552","Type":"ContainerStarted","Data":"b9a7f26e6d4ab185110169ba868d6c383ed56589033d530c6c384197fa805bbc"} Jul 13 17:33:23.996083 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:23.996067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 17:33:24.133704 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.133680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 17:33:24.136392 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:24.136361 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebb5417_10a2_4288_beea_1a8ee2c30776.slice/crio-8f1107ccd0f12250067968880fdaebdf46c1f61f90e7b5b847027a8f414a6d12 WatchSource:0}: Error finding container 8f1107ccd0f12250067968880fdaebdf46c1f61f90e7b5b847027a8f414a6d12: Status 404 returned error can't find the container with id 8f1107ccd0f12250067968880fdaebdf46c1f61f90e7b5b847027a8f414a6d12 Jul 13 17:33:24.294639 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.294473 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:24.297831 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.297795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-hg4v5\" (UID: \"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:24.455339 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.455309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" Jul 13 17:33:24.693164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.692343 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79f46758fc-2cnvm"] Jul 13 17:33:24.695341 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.695313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.697659 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.697563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9drdk\"" Jul 13 17:33:24.697802 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.697712 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 13 17:33:24.697869 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.697827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 13 17:33:24.697869 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.697853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 13 17:33:24.698019 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.697868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 13 17:33:24.698486 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.698367 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-703mm04430ps\"" Jul 13 17:33:24.698486 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.698380 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 13 17:33:24.706507 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.706488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79f46758fc-2cnvm"] Jul 13 17:33:24.798938 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.798912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-grpc-tls\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.799065 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.798966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.799121 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.799077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.799164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.799114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.799164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.799145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5ed53-33d0-4f7b-846f-f5603bba2992-metrics-client-ca\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.799273 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.799170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2m6\" (UniqueName: \"kubernetes.io/projected/c2b5ed53-33d0-4f7b-846f-f5603bba2992-kube-api-access-lh2m6\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.799273 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.799247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-tls\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.799352 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.799298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900502 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-grpc-tls\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900697 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900769 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900827 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900827 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5ed53-33d0-4f7b-846f-f5603bba2992-metrics-client-ca\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900928 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2m6\" (UniqueName: \"kubernetes.io/projected/c2b5ed53-33d0-4f7b-846f-f5603bba2992-kube-api-access-lh2m6\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900928 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-tls\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.900928 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.900904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.901861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.901790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5ed53-33d0-4f7b-846f-f5603bba2992-metrics-client-ca\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.903774 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.903752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.903774 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.903768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.904333 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.904315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-tls\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.904426 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.904341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.904689 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.904662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-grpc-tls\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.905105 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.905083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c2b5ed53-33d0-4f7b-846f-f5603bba2992-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.908915 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.908896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2m6\" (UniqueName: \"kubernetes.io/projected/c2b5ed53-33d0-4f7b-846f-f5603bba2992-kube-api-access-lh2m6\") pod \"thanos-querier-79f46758fc-2cnvm\" (UID: \"c2b5ed53-33d0-4f7b-846f-f5603bba2992\") " pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:24.934585 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:24.934556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerStarted","Data":"8f1107ccd0f12250067968880fdaebdf46c1f61f90e7b5b847027a8f414a6d12"} Jul 13 17:33:25.007025 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.006876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:25.106687 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.104498 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5"] Jul 13 17:33:25.201223 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.201069 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79f46758fc-2cnvm"] Jul 13 17:33:25.217437 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:25.217404 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b5ed53_33d0_4f7b_846f_f5603bba2992.slice/crio-6357f22659e3d9922367586ad5251e6b4f11d9b96af7b46beae252156cb612d3 WatchSource:0}: Error finding container 6357f22659e3d9922367586ad5251e6b4f11d9b96af7b46beae252156cb612d3: Status 404 returned error can't find the container with id 6357f22659e3d9922367586ad5251e6b4f11d9b96af7b46beae252156cb612d3 Jul 13 17:33:25.559073 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.559041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:25.618555 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.618527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:25.818085 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.818051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74zgv" Jul 13 17:33:25.822316 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.822295 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:25.938519 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.938487 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" event={"ID":"c2b5ed53-33d0-4f7b-846f-f5603bba2992","Type":"ContainerStarted","Data":"6357f22659e3d9922367586ad5251e6b4f11d9b96af7b46beae252156cb612d3"} Jul 13 17:33:25.940208 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.940178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" event={"ID":"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a","Type":"ContainerStarted","Data":"7cd97b67ed15bdb2add65179365afb8c9c7257feff23a87a130ffd1fb9a8e491"} Jul 13 17:33:25.940317 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.940241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" event={"ID":"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a","Type":"ContainerStarted","Data":"2aac64a7bed8ebc9079044be7d94993409187ff01bee4187f12f181267597584"} Jul 13 17:33:25.940317 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.940262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" event={"ID":"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a","Type":"ContainerStarted","Data":"8bb4a6df0bbee40eeb969ce1d257d34208fb53f59d761006b28356b27d7f0e5e"} Jul 13 17:33:25.941554 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.941526 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ebb5417-10a2-4288-beea-1a8ee2c30776" containerID="c24e67cadefb7218474ee5c0ab8d0222fe3119e74cd542ad7cabd8765994a6ba" exitCode=0 Jul 13 17:33:25.941643 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.941607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerDied","Data":"c24e67cadefb7218474ee5c0ab8d0222fe3119e74cd542ad7cabd8765994a6ba"} Jul 13 17:33:25.943138 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.943121 2564 generic.go:356] "Generic (PLEG): container finished" podID="15e4507a-4eb4-4e02-8afd-e6b9055e9af8" containerID="75c70ffc9d111fb1db36f8cc2e09107ef444eaa98aa73e1d4b4a58a5d2b1c625" exitCode=0 Jul 13 17:33:25.943208 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.943191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9nr6w" event={"ID":"15e4507a-4eb4-4e02-8afd-e6b9055e9af8","Type":"ContainerDied","Data":"75c70ffc9d111fb1db36f8cc2e09107ef444eaa98aa73e1d4b4a58a5d2b1c625"} Jul 13 17:33:25.945435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.945408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" event={"ID":"49dfab4f-19d7-459a-9634-023750e51552","Type":"ContainerStarted","Data":"873e1e9077cb533763d5b40e2468d320fa6d331ef89f4ac9656fc8bbea8deead"} Jul 13 17:33:25.945543 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.945461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" event={"ID":"49dfab4f-19d7-459a-9634-023750e51552","Type":"ContainerStarted","Data":"e4c84d2b60b3aba28ba3ebc5580142f978199662b848d2edda934880fd197b32"} Jul 13 17:33:25.945543 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.945495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" event={"ID":"49dfab4f-19d7-459a-9634-023750e51552","Type":"ContainerStarted","Data":"cf19d94c0ae38b9d721a4bd9bb2fab4a1c11cc9ed210344076ab33413e8fdaa6"} Jul 13 17:33:25.972656 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:25.972636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:26.000731 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.000684 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-nd4nm" podStartSLOduration=2.113981797 podStartE2EDuration="4.000664145s" totalImagesPullingTime="1.886682348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:25.998025988 +0000 UTC m=+61.005428255" watchObservedRunningTime="2026-07-13 17:33:26.000664145 +0000 UTC m=+61.008066393" Jul 13 17:33:26.037566 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.037495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:26.139698 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.139657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:26.889052 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.889021 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8"] Jul 13 17:33:26.912345 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.912312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8"] Jul 13 17:33:26.912485 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.912325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:26.917553 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.916389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 13 17:33:26.917553 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.916646 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6rggz\"" Jul 13 17:33:26.917553 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.916724 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 13 17:33:26.917553 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.916749 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 13 17:33:26.917553 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.916817 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 17:33:26.918349 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.918327 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cf0jm0592l5j0\"" Jul 13 17:33:26.952035 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.952009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9nr6w" event={"ID":"15e4507a-4eb4-4e02-8afd-e6b9055e9af8","Type":"ContainerStarted","Data":"2a114916c8d67dc80d8ede15efd9d97e5e05ee1656afe2d091926faa8f0d45fc"} Jul 13 17:33:26.952157 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.952084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9nr6w" event={"ID":"15e4507a-4eb4-4e02-8afd-e6b9055e9af8","Type":"ContainerStarted","Data":"9ac2c25f768b962c7d726e67c19fce39561b564ade20983e276e4b7227a48d0d"} Jul 13 17:33:26.969750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:26.969707 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9nr6w" podStartSLOduration=3.464777308 podStartE2EDuration="4.969695514s" totalImagesPullingTime="1.36773082s" totalInitContainerRuntime="137.187386ms" isStatefulPod=true podCreationTimestamp="2026-07-13 17:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:26.967811059 +0000 UTC m=+61.975213322" watchObservedRunningTime="2026-07-13 17:33:26.969695514 +0000 UTC m=+61.977097756" Jul 13 17:33:27.025672 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.025639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-metrics-server-audit-profiles\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.025788 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.025684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-secret-metrics-server-tls\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.025788 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.025723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.025788 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.025754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjm76\" (UniqueName: \"kubernetes.io/projected/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-kube-api-access-bjm76\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.026001 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.025801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-client-ca-bundle\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.026001 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.025909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-audit-log\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.026110 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.026038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-secret-metrics-server-client-certs\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.127145 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.127119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-client-ca-bundle\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.127323 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.127168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-audit-log\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.127323 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.127264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-secret-metrics-server-client-certs\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.127437 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.127321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-metrics-server-audit-profiles\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.127437 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.127349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-secret-metrics-server-tls\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.127437 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.127378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.127437 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.127409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjm76\" (UniqueName: \"kubernetes.io/projected/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-kube-api-access-bjm76\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.128297 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.128227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-audit-log\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.128404 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.128306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.128617 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.128567 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-metrics-server-audit-profiles\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.130328 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.130307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-secret-metrics-server-client-certs\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.130466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.130446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-secret-metrics-server-tls\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.130813 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.130779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-client-ca-bundle\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.135546 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.135523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjm76\" (UniqueName: \"kubernetes.io/projected/b06cdc92-4954-4afa-bc32-6e2dfdac1faa-kube-api-access-bjm76\") pod \"metrics-server-6dcd9bfc4d-td6c8\" (UID: \"b06cdc92-4954-4afa-bc32-6e2dfdac1faa\") " pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.222880 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.222793 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:27.290413 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.290378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:27.390514 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.390484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:27.424766 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.424736 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27"] Jul 13 17:33:27.440446 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.440308 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d986664cd-xt686"] Jul 13 17:33:27.440446 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.440347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:27.443097 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.443070 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 13 17:33:27.443717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.443694 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gx8w7\"" Jul 13 17:33:27.453763 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.453743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27"] Jul 13 17:33:27.453873 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.453778 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d986664cd-xt686"] Jul 13 17:33:27.453974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.453921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.463686 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.463607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:27.529863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.529796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-oauth-serving-cert\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.529863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.529830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51db2690-8634-4f9c-b6a9-69f00b05ba2e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hjb27\" (UID: \"51db2690-8634-4f9c-b6a9-69f00b05ba2e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:27.529863 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.529858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmlmh\" (UniqueName: \"kubernetes.io/projected/51db2690-8634-4f9c-b6a9-69f00b05ba2e-kube-api-access-fmlmh\") pod \"monitoring-plugin-5f68df7f58-hjb27\" (UID: \"51db2690-8634-4f9c-b6a9-69f00b05ba2e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:27.530108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.529884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-config\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.530108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.529901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-trusted-ca-bundle\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.530108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.529934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-service-ca\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.530108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.530059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-serving-cert\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.530108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.530103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-oauth-config\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.530356 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.530169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2lj\" (UniqueName: \"kubernetes.io/projected/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-kube-api-access-sh2lj\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2lj\" (UniqueName: \"kubernetes.io/projected/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-kube-api-access-sh2lj\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-oauth-serving-cert\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51db2690-8634-4f9c-b6a9-69f00b05ba2e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hjb27\" (UID: \"51db2690-8634-4f9c-b6a9-69f00b05ba2e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmlmh\" (UniqueName: \"kubernetes.io/projected/51db2690-8634-4f9c-b6a9-69f00b05ba2e-kube-api-access-fmlmh\") pod \"monitoring-plugin-5f68df7f58-hjb27\" (UID: \"51db2690-8634-4f9c-b6a9-69f00b05ba2e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-config\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-trusted-ca-bundle\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-service-ca\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-serving-cert\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.631925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.631714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-oauth-config\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.632648 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.632578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-oauth-serving-cert\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.632719 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:27.632683 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 13 17:33:27.632767 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:33:27.632737 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51db2690-8634-4f9c-b6a9-69f00b05ba2e-monitoring-plugin-cert podName:51db2690-8634-4f9c-b6a9-69f00b05ba2e nodeName:}" failed. No retries permitted until 2026-07-13 17:33:28.132718176 +0000 UTC m=+63.140120402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/51db2690-8634-4f9c-b6a9-69f00b05ba2e-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-hjb27" (UID: "51db2690-8634-4f9c-b6a9-69f00b05ba2e") : secret "monitoring-plugin-cert" not found Jul 13 17:33:27.633886 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.633607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-trusted-ca-bundle\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.634913 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.634139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-config\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.635208 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.635172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-service-ca\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.638464 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.638441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-serving-cert\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.640133 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.640083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-oauth-config\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.640398 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.640358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2lj\" (UniqueName: \"kubernetes.io/projected/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-kube-api-access-sh2lj\") pod \"console-5d986664cd-xt686\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.642225 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.642201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmlmh\" (UniqueName: \"kubernetes.io/projected/51db2690-8634-4f9c-b6a9-69f00b05ba2e-kube-api-access-fmlmh\") pod \"monitoring-plugin-5f68df7f58-hjb27\" (UID: \"51db2690-8634-4f9c-b6a9-69f00b05ba2e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:27.670366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.670331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:27.748816 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.748793 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8"] Jul 13 17:33:27.749955 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.749746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:27.767205 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.767181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:27.793974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.793943 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt"] Jul 13 17:33:27.796331 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.796297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.798714 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.798696 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 13 17:33:27.798899 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.798879 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xvnln\"" Jul 13 17:33:27.798977 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.798934 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 13 17:33:27.799046 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.798990 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 13 17:33:27.799046 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.799004 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 13 17:33:27.799224 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.799210 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 13 17:33:27.808651 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.808629 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 13 17:33:27.812524 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.812504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt"] Jul 13 17:33:27.833353 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.833454 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-telemeter-client-tls\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.833510 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lss7x\" (UniqueName: \"kubernetes.io/projected/b59cfd5f-2222-44d3-bb81-3caaf98b632e-kube-api-access-lss7x\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.833551 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-metrics-client-ca\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.833626 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-serving-certs-ca-bundle\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.833665 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-federate-client-tls\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.833709 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-secret-telemeter-client\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.833752 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.833721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.871161 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.871140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:27.934752 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.934719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-telemeter-client-tls\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.934820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lss7x\" (UniqueName: \"kubernetes.io/projected/b59cfd5f-2222-44d3-bb81-3caaf98b632e-kube-api-access-lss7x\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.934852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-metrics-client-ca\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.934890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-serving-certs-ca-bundle\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.934936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-federate-client-tls\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.934972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-secret-telemeter-client\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.935079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.935147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935752 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.935699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-metrics-client-ca\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.935752 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.935726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-serving-certs-ca-bundle\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.936000 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.935960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b59cfd5f-2222-44d3-bb81-3caaf98b632e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.937775 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.937756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-secret-telemeter-client\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.937861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.937781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.937963 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.937936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-federate-client-tls\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.937963 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.937954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b59cfd5f-2222-44d3-bb81-3caaf98b632e-telemeter-client-tls\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.944106 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.944083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lss7x\" (UniqueName: \"kubernetes.io/projected/b59cfd5f-2222-44d3-bb81-3caaf98b632e-kube-api-access-lss7x\") pod \"telemeter-client-5c8fbf89f4-fk4kt\" (UID: \"b59cfd5f-2222-44d3-bb81-3caaf98b632e\") " pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:27.958501 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.958467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" event={"ID":"ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a","Type":"ContainerStarted","Data":"ee265ec0f49bad5c01c371218b67bcb76154297f00c7f85420af6faf53711628"} Jul 13 17:33:27.975189 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:27.975149 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-hg4v5" podStartSLOduration=3.659452753 podStartE2EDuration="5.975135754s" totalImagesPullingTime="2.315683001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:27.974065213 +0000 UTC m=+62.981467480" watchObservedRunningTime="2026-07-13 17:33:27.975135754 +0000 UTC m=+62.982537999" Jul 13 17:33:28.088266 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:28.088160 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06cdc92_4954_4afa_bc32_6e2dfdac1faa.slice/crio-fa2d2143e8be7a5d572da900c85f621d18596908d0ab666f4da2f9070cac1b0b WatchSource:0}: Error finding container fa2d2143e8be7a5d572da900c85f621d18596908d0ab666f4da2f9070cac1b0b: Status 404 returned error can't find the container with id fa2d2143e8be7a5d572da900c85f621d18596908d0ab666f4da2f9070cac1b0b Jul 13 17:33:28.119390 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.118548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" Jul 13 17:33:28.137991 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.137950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51db2690-8634-4f9c-b6a9-69f00b05ba2e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hjb27\" (UID: \"51db2690-8634-4f9c-b6a9-69f00b05ba2e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:28.141124 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.141075 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51db2690-8634-4f9c-b6a9-69f00b05ba2e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-hjb27\" (UID: \"51db2690-8634-4f9c-b6a9-69f00b05ba2e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:28.272492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.272416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d986664cd-xt686"] Jul 13 17:33:28.278822 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:28.278780 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795b2b37_b7bc_4aac_b4da_6f21fc1eac94.slice/crio-ea8823009f0516d725b031efb573e7c36ae27d755321a9c5182ecd1463c42b8c WatchSource:0}: Error finding container ea8823009f0516d725b031efb573e7c36ae27d755321a9c5182ecd1463c42b8c: Status 404 returned error can't find the container with id ea8823009f0516d725b031efb573e7c36ae27d755321a9c5182ecd1463c42b8c Jul 13 17:33:28.317186 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.315757 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt"] Jul 13 17:33:28.322674 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:28.322642 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59cfd5f_2222_44d3_bb81_3caaf98b632e.slice/crio-9c1f1d324fd7582e92758769043d4047e5c96017a56efc4e4c3cbf10f0d7739f WatchSource:0}: Error finding container 9c1f1d324fd7582e92758769043d4047e5c96017a56efc4e4c3cbf10f0d7739f: Status 404 returned error can't find the container with id 9c1f1d324fd7582e92758769043d4047e5c96017a56efc4e4c3cbf10f0d7739f Jul 13 17:33:28.357000 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.356968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:28.513679 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.513655 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27"] Jul 13 17:33:28.519204 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:28.519175 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51db2690_8634_4f9c_b6a9_69f00b05ba2e.slice/crio-8f75363a7e98079668eea36c1854f8b4aa2e9b12b96a6590ac56658299a89f36 WatchSource:0}: Error finding container 8f75363a7e98079668eea36c1854f8b4aa2e9b12b96a6590ac56658299a89f36: Status 404 returned error can't find the container with id 8f75363a7e98079668eea36c1854f8b4aa2e9b12b96a6590ac56658299a89f36 Jul 13 17:33:28.811531 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.811452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:28.818901 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.818868 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 17:33:28.822896 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.822874 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.825457 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.825589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.825668 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.825739 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b3j6h58ldcbrq\"" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.825970 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.826796 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.827054 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 17:33:28.827615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.827578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 17:33:28.828137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.827679 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mnp5k\"" Jul 13 17:33:28.828137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.827931 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 17:33:28.828137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.827986 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 17:33:28.828137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.827933 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 17:33:28.828137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.828129 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 17:33:28.828439 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.828366 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 17:33:28.828439 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.828386 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 17:33:28.834633 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.834605 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 17:33:28.835361 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.835336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 17:33:28.946039 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.945889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-config\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8f4\" (UniqueName: \"kubernetes.io/projected/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-kube-api-access-lb8f4\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-web-config\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-config-out\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.946996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.947054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.947078 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.947110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.947533 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.947148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.948422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.947194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.948422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.947218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.948422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.947278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:28.972295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.972072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" event={"ID":"c2b5ed53-33d0-4f7b-846f-f5603bba2992","Type":"ContainerStarted","Data":"ff6f2b70bf41b4695ff6cb7508d65a1021e5628bf4d16ddfcc405211ea6c8e6d"} Jul 13 17:33:28.972295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.972161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" event={"ID":"c2b5ed53-33d0-4f7b-846f-f5603bba2992","Type":"ContainerStarted","Data":"f3f2f00415945c545bdf8daf2a5deb72f7b7d6430c6d2f30b5a9a102c4eb6484"} Jul 13 17:33:28.972295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.972200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" event={"ID":"c2b5ed53-33d0-4f7b-846f-f5603bba2992","Type":"ContainerStarted","Data":"0594baaf57799939f014ebac86a228ea6bb8602a7dd980f12a7000ae057abf53"} Jul 13 17:33:28.977287 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.976922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d986664cd-xt686" event={"ID":"795b2b37-b7bc-4aac-b4da-6f21fc1eac94","Type":"ContainerStarted","Data":"0aa95a61ecc55b73b0aaea12c9934603c13f42c2032da7d66ae4d07adc550900"} Jul 13 17:33:28.977287 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.976974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d986664cd-xt686" event={"ID":"795b2b37-b7bc-4aac-b4da-6f21fc1eac94","Type":"ContainerStarted","Data":"ea8823009f0516d725b031efb573e7c36ae27d755321a9c5182ecd1463c42b8c"} Jul 13 17:33:28.987717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.987453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerStarted","Data":"9106e36d5e138e023e921fa3b8191cd28a00aceda71c93c4c521e610a0513abc"} Jul 13 17:33:28.987717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.987526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerStarted","Data":"dc8f2e7b43f581d01b4bff49c518166af16d37cfe37e6c2eb13456802f9bfbf4"} Jul 13 17:33:28.987717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.987568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerStarted","Data":"ce155a0a3a41e78465e7f6b61e62bb52a2eec178415c452c13685bfcfaf4d717"} Jul 13 17:33:28.987717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.987610 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerStarted","Data":"9f6cfe53496a4a753d881868caac926a6f2ba99c6e8afea37652959f6eaf541a"} Jul 13 17:33:28.987717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.987646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerStarted","Data":"e8193140477d001ae940f733694303b9b6d31a9cf4953528074605e2c00c9b71"} Jul 13 17:33:28.989353 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.989287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" event={"ID":"51db2690-8634-4f9c-b6a9-69f00b05ba2e","Type":"ContainerStarted","Data":"8f75363a7e98079668eea36c1854f8b4aa2e9b12b96a6590ac56658299a89f36"} Jul 13 17:33:28.991656 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.991576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" event={"ID":"b59cfd5f-2222-44d3-bb81-3caaf98b632e","Type":"ContainerStarted","Data":"9c1f1d324fd7582e92758769043d4047e5c96017a56efc4e4c3cbf10f0d7739f"} Jul 13 17:33:28.997557 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.996090 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d986664cd-xt686" podStartSLOduration=1.996074198 podStartE2EDuration="1.996074198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:28.996068696 +0000 UTC m=+64.003470940" watchObservedRunningTime="2026-07-13 17:33:28.996074198 +0000 UTC m=+64.003476425" Jul 13 17:33:28.997557 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:28.997505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" event={"ID":"b06cdc92-4954-4afa-bc32-6e2dfdac1faa","Type":"ContainerStarted","Data":"fa2d2143e8be7a5d572da900c85f621d18596908d0ab666f4da2f9070cac1b0b"} Jul 13 17:33:29.048121 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048288 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-config-out\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.048807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.048924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-config\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8f4\" (UniqueName: \"kubernetes.io/projected/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-kube-api-access-lb8f4\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049603 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049603 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-web-config\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.049603 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.049343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.052813 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.051806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.054015 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.053991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.054107 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.053989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.056719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.054513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.056719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.054791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.056719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.055078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.056719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.055491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-web-config\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.056719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.056222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.056719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.056662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.061977 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.061899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.065824 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.063662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.065824 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.063764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.067496 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.067436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.067496 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.067452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.068017 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.067875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8f4\" (UniqueName: \"kubernetes.io/projected/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-kube-api-access-lb8f4\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.068089 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.068014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.069649 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.069583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-config\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.069649 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.069588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-config-out\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.070575 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.070549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.071624 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.071581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.146526 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.146490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:29.158939 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.158855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:29.259018 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.258929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:29.510033 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:29.509610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 17:33:29.513049 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:29.513023 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87bfe57_0e9d_4ce4_96bb_3c3eed39cf61.slice/crio-b8bfd9e75a5c80ff78aa212054785af3124c9db8f0fbc4c5fed8632279d9bf9b WatchSource:0}: Error finding container b8bfd9e75a5c80ff78aa212054785af3124c9db8f0fbc4c5fed8632279d9bf9b: Status 404 returned error can't find the container with id b8bfd9e75a5c80ff78aa212054785af3124c9db8f0fbc4c5fed8632279d9bf9b Jul 13 17:33:30.002960 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.002874 2564 generic.go:356] "Generic (PLEG): container finished" podID="b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61" containerID="76185fb0b217e4ee96b786e6c97304132d1db1b8057f662f88551c40034ab20f" exitCode=0 Jul 13 17:33:30.003425 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.002963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerDied","Data":"76185fb0b217e4ee96b786e6c97304132d1db1b8057f662f88551c40034ab20f"} Jul 13 17:33:30.003425 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.003044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerStarted","Data":"b8bfd9e75a5c80ff78aa212054785af3124c9db8f0fbc4c5fed8632279d9bf9b"} Jul 13 17:33:30.006849 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.006818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" event={"ID":"c2b5ed53-33d0-4f7b-846f-f5603bba2992","Type":"ContainerStarted","Data":"ce57278c6d67e8dbad5fbd5d63ab7573b496891206f51be69de4741a4b3abe4d"} Jul 13 17:33:30.006954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.006892 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" event={"ID":"c2b5ed53-33d0-4f7b-846f-f5603bba2992","Type":"ContainerStarted","Data":"023cf2c1800d019cc5af00b1bd07608b8d6525c875e953e48db492e34b2486c1"} Jul 13 17:33:30.006954 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.006940 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:30.007056 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.006980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" event={"ID":"c2b5ed53-33d0-4f7b-846f-f5603bba2992","Type":"ContainerStarted","Data":"6896b06a5b46b18ddb97658815a1554ffce923b89a1b431f5f5e8311123bca51"} Jul 13 17:33:30.010855 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.010834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ebb5417-10a2-4288-beea-1a8ee2c30776","Type":"ContainerStarted","Data":"a4b4328d597509c7c19b152a552822fe5308b7d63f10f6d4c9bbe66dd03ab3fb"} Jul 13 17:33:30.048690 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.048629 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" podStartSLOduration=2.210860033 podStartE2EDuration="6.04860946s" totalImagesPullingTime="3.837749427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:30.047306787 +0000 UTC m=+65.054709031" watchObservedRunningTime="2026-07-13 17:33:30.04860946 +0000 UTC m=+65.056011705" Jul 13 17:33:30.071549 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:30.071500 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.33533763 podStartE2EDuration="7.071485485s" totalImagesPullingTime="4.722289411s" totalInitContainerRuntime="13.858444ms" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:30.070087444 +0000 UTC m=+65.077489687" watchObservedRunningTime="2026-07-13 17:33:30.071485485 +0000 UTC m=+65.078887728" Jul 13 17:33:31.016957 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.016924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" event={"ID":"b59cfd5f-2222-44d3-bb81-3caaf98b632e","Type":"ContainerStarted","Data":"e9fa60f299a6be2bc296cdb76b2b13d89ca9652d85b3fc4c0b1331fd8171a869"} Jul 13 17:33:31.017286 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.017029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" event={"ID":"b59cfd5f-2222-44d3-bb81-3caaf98b632e","Type":"ContainerStarted","Data":"2ba71786bb36b8608590c5c296a46f6ea47df1ab264c09b5e064304a043bf9e6"} Jul 13 17:33:31.018506 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.018428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" event={"ID":"b06cdc92-4954-4afa-bc32-6e2dfdac1faa","Type":"ContainerStarted","Data":"748e0d50dff7d42379bcd9dd5391df07960fd7e36f0375f6a6ede1cb9ec3adc9"} Jul 13 17:33:31.020467 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.020379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" event={"ID":"51db2690-8634-4f9c-b6a9-69f00b05ba2e","Type":"ContainerStarted","Data":"17058986e462c0830bd4ea475647a4e991c43ca68fe4e32cbcf92e568a0621a6"} Jul 13 17:33:31.020783 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.020734 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:31.025758 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.025732 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" Jul 13 17:33:31.036593 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.036552 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" podStartSLOduration=2.288624527 podStartE2EDuration="5.036540715s" totalImagesPullingTime="2.747916188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:31.035114967 +0000 UTC m=+66.042517211" watchObservedRunningTime="2026-07-13 17:33:31.036540715 +0000 UTC m=+66.043942957" Jul 13 17:33:31.053633 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.053569 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-hjb27" podStartSLOduration=1.7328329390000001 podStartE2EDuration="4.053547977s" totalImagesPullingTime="2.320715038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:31.051607402 +0000 UTC m=+66.059009647" watchObservedRunningTime="2026-07-13 17:33:31.053547977 +0000 UTC m=+66.060950221" Jul 13 17:33:31.280732 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.280655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:33:31.280732 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.280694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:33:31.282855 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.282834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 17:33:31.282906 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.282834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 17:33:31.293798 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.293771 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ffda-5366-46b0-88bc-32e697a9d2ff-metrics-certs\") pod \"network-metrics-daemon-7xvlp\" (UID: \"bbc4ffda-5366-46b0-88bc-32e697a9d2ff\") " pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:33:31.293904 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.293852 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30053369-dbea-4ebb-8e82-4ee42689e422-original-pull-secret\") pod \"global-pull-secret-syncer-n8t6r\" (UID: \"30053369-dbea-4ebb-8e82-4ee42689e422\") " pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:33:31.382185 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.382150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:31.384509 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.384487 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 17:33:31.395283 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.395261 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 17:33:31.406463 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.406436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppk5\" (UniqueName: \"kubernetes.io/projected/06123f7e-7ce5-4f55-8b0a-f858c91860a0-kube-api-access-sppk5\") pod \"network-check-target-fx7fh\" (UID: \"06123f7e-7ce5-4f55-8b0a-f858c91860a0\") " pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:31.557906 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.557878 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vn7td\"" Jul 13 17:33:31.566535 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.566490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:31.569560 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.569541 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4jvwf\"" Jul 13 17:33:31.574143 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.574124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n8t6r" Jul 13 17:33:31.578483 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.578448 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7xvlp" Jul 13 17:33:31.741773 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.741746 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fx7fh"] Jul 13 17:33:31.745261 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:31.745214 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06123f7e_7ce5_4f55_8b0a_f858c91860a0.slice/crio-096dbb98896ea6791ce772a09664f7cfe9230f0038d0f04acf668f6118ab0d11 WatchSource:0}: Error finding container 096dbb98896ea6791ce772a09664f7cfe9230f0038d0f04acf668f6118ab0d11: Status 404 returned error can't find the container with id 096dbb98896ea6791ce772a09664f7cfe9230f0038d0f04acf668f6118ab0d11 Jul 13 17:33:31.971477 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.971406 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n8t6r"] Jul 13 17:33:31.974445 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:31.974357 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7xvlp"] Jul 13 17:33:31.974508 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:31.974435 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30053369_dbea_4ebb_8e82_4ee42689e422.slice/crio-81f56482174164328d5037c428b8477f4a81c9407b13586a6cd889dc83db8f00 WatchSource:0}: Error finding container 81f56482174164328d5037c428b8477f4a81c9407b13586a6cd889dc83db8f00: Status 404 returned error can't find the container with id 81f56482174164328d5037c428b8477f4a81c9407b13586a6cd889dc83db8f00 Jul 13 17:33:31.976602 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:33:31.976575 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc4ffda_5366_46b0_88bc_32e697a9d2ff.slice/crio-e0f430c62694e2e5061b8175f07c71e79bd89f5280df730a8965321e5a6d28e4 WatchSource:0}: Error finding container e0f430c62694e2e5061b8175f07c71e79bd89f5280df730a8965321e5a6d28e4: Status 404 returned error can't find the container with id e0f430c62694e2e5061b8175f07c71e79bd89f5280df730a8965321e5a6d28e4 Jul 13 17:33:32.017919 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.017895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:32.025339 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.025312 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n8t6r" event={"ID":"30053369-dbea-4ebb-8e82-4ee42689e422","Type":"ContainerStarted","Data":"81f56482174164328d5037c428b8477f4a81c9407b13586a6cd889dc83db8f00"} Jul 13 17:33:32.026690 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.026640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7xvlp" event={"ID":"bbc4ffda-5366-46b0-88bc-32e697a9d2ff","Type":"ContainerStarted","Data":"e0f430c62694e2e5061b8175f07c71e79bd89f5280df730a8965321e5a6d28e4"} Jul 13 17:33:32.029084 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.029064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" event={"ID":"b59cfd5f-2222-44d3-bb81-3caaf98b632e","Type":"ContainerStarted","Data":"4af8d590ae650321a5ad0cfa2b4e0bd5f75cb3db019b54cbffc0ea247f1e777f"} Jul 13 17:33:32.030409 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.030386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fx7fh" event={"ID":"06123f7e-7ce5-4f55-8b0a-f858c91860a0","Type":"ContainerStarted","Data":"096dbb98896ea6791ce772a09664f7cfe9230f0038d0f04acf668f6118ab0d11"} Jul 13 17:33:32.053291 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.053210 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5c8fbf89f4-fk4kt" podStartSLOduration=2.533532286 podStartE2EDuration="5.053193005s" totalImagesPullingTime="2.519660719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:32.05165579 +0000 UTC m=+67.059058051" watchObservedRunningTime="2026-07-13 17:33:32.053193005 +0000 UTC m=+67.060595251" Jul 13 17:33:32.118756 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.118734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:32.906578 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.906514 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d986664cd-xt686"] Jul 13 17:33:32.938951 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.937321 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5876448bbd-mg684"] Jul 13 17:33:32.942266 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.942013 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:32.949571 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:32.949521 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5876448bbd-mg684"] Jul 13 17:33:33.104737 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.104628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-oauth-config\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.104737 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.104715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-config\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.105270 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.104741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-service-ca\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.105270 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.104767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlrb\" (UniqueName: \"kubernetes.io/projected/beb82c81-f106-4a23-a4e4-c0e7c37c4087-kube-api-access-2mlrb\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.105270 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.104808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-trusted-ca-bundle\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.105270 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.104870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-oauth-serving-cert\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.105270 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.104902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-serving-cert\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.207997 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.207914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-trusted-ca-bundle\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.208123 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.208024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-oauth-serving-cert\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.208123 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.208064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-serving-cert\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.208123 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.208097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-oauth-config\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.208537 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.208316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-config\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.208537 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.208343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-service-ca\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.208537 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.208372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlrb\" (UniqueName: \"kubernetes.io/projected/beb82c81-f106-4a23-a4e4-c0e7c37c4087-kube-api-access-2mlrb\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.211207 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.210208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-config\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.211207 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.210660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-trusted-ca-bundle\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.211207 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.210819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-service-ca\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.211498 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.211429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-oauth-serving-cert\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.212194 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.212155 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-serving-cert\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.215404 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.215365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-oauth-config\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.218221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.218031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlrb\" (UniqueName: \"kubernetes.io/projected/beb82c81-f106-4a23-a4e4-c0e7c37c4087-kube-api-access-2mlrb\") pod \"console-5876448bbd-mg684\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.234832 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.234746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:33.256933 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.256847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:33.334829 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.334735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:33.462326 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:33.462297 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5876448bbd-mg684"] Jul 13 17:33:34.061264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.060929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7xvlp" event={"ID":"bbc4ffda-5366-46b0-88bc-32e697a9d2ff","Type":"ContainerStarted","Data":"5eab902eb07dc524e343097bc28f66b46ee9c3b0be0271871211513ac1c61ac8"} Jul 13 17:33:34.061264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.060991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7xvlp" event={"ID":"bbc4ffda-5366-46b0-88bc-32e697a9d2ff","Type":"ContainerStarted","Data":"cc5a148be65e880baad69b2b12f249c7a070e146364cba7cc312fcc01084e4f7"} Jul 13 17:33:34.064848 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.064423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5876448bbd-mg684" event={"ID":"beb82c81-f106-4a23-a4e4-c0e7c37c4087","Type":"ContainerStarted","Data":"e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c"} Jul 13 17:33:34.064848 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.064479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5876448bbd-mg684" event={"ID":"beb82c81-f106-4a23-a4e4-c0e7c37c4087","Type":"ContainerStarted","Data":"2886f4bbf7419497b3e7b05773e2aeba8156a373b10d77a83e03569cb5bf9c5c"} Jul 13 17:33:34.078543 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.078435 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7xvlp" podStartSLOduration=67.97599017 podStartE2EDuration="1m9.078412883s" totalImagesPullingTime="1.102422713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:34.074930016 +0000 UTC m=+69.082332275" watchObservedRunningTime="2026-07-13 17:33:34.078412883 +0000 UTC m=+69.085815129" Jul 13 17:33:34.095127 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.095071 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5876448bbd-mg684" podStartSLOduration=2.09505331 podStartE2EDuration="2.09505331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:34.093175965 +0000 UTC m=+69.100578207" watchObservedRunningTime="2026-07-13 17:33:34.09505331 +0000 UTC m=+69.102455553" Jul 13 17:33:34.197464 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.197379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:34.298523 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:34.298479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:36.028334 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:36.028305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79f46758fc-2cnvm" Jul 13 17:33:37.767637 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:37.767613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:38.088511 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.088475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fx7fh" event={"ID":"06123f7e-7ce5-4f55-8b0a-f858c91860a0","Type":"ContainerStarted","Data":"aa220e45992c451feeab7b07a4beb66510b736eccb5dd9f652babd57144639b1"} Jul 13 17:33:38.088711 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.088690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:33:38.090481 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.090421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n8t6r" event={"ID":"30053369-dbea-4ebb-8e82-4ee42689e422","Type":"ContainerStarted","Data":"d7e536792066305e57614757eec8227cb3268c7906bb9a89ed50d5a866347700"} Jul 13 17:33:38.094015 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.093907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerStarted","Data":"82c5d67728db789460a6dca39809e3a207cb5f79a7918952ce5c24c9ca14e97c"} Jul 13 17:33:38.094015 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.093973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerStarted","Data":"37d0bd447d9b68f5c899fb5b07c81677d5661e0bf4b877e6351bb07dde7a8b86"} Jul 13 17:33:38.094015 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.094015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerStarted","Data":"0ef03e72434de7d22f1030e8311e31c413ea5e9d7a24379a418e3184e9db9ca4"} Jul 13 17:33:38.094248 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.094057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerStarted","Data":"21c8ec04cdf2c7555d2071bf1e9d569f1ef9c1f6b1df414406daeb682dd3756e"} Jul 13 17:33:38.107194 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.105621 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fx7fh" podStartSLOduration=67.067712876 podStartE2EDuration="1m13.105603724s" totalImagesPullingTime="6.037890848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:38.104316362 +0000 UTC m=+73.111718605" watchObservedRunningTime="2026-07-13 17:33:38.105603724 +0000 UTC m=+73.113005968" Jul 13 17:33:38.122181 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.120937 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n8t6r" podStartSLOduration=67.304073 podStartE2EDuration="1m13.120919418s" totalImagesPullingTime="5.816846418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 17:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:38.120365331 +0000 UTC m=+73.127767573" watchObservedRunningTime="2026-07-13 17:33:38.120919418 +0000 UTC m=+73.128321662" Jul 13 17:33:38.163022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.162994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:38.261810 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:38.261786 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:39.101564 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:39.101524 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerStarted","Data":"35411c119cdf0650b66f315312bb331815e09993d4a5280c51c83dda1fd00158"} Jul 13 17:33:39.101920 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:39.101618 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61","Type":"ContainerStarted","Data":"78ad44c2b8abaaf5998207ee26f233c31375e5de6c3d02de8df6a2ce62675259"} Jul 13 17:33:39.128637 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:39.128586 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.334292919 podStartE2EDuration="11.128569591s" totalImagesPullingTime="7.773182695s" totalInitContainerRuntime="21.093977ms" isStatefulPod=false podCreationTimestamp="2026-07-13 17:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:33:39.127885722 +0000 UTC m=+74.135287989" watchObservedRunningTime="2026-07-13 17:33:39.128569591 +0000 UTC m=+74.135971835" Jul 13 17:33:39.147891 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:39.147862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:33:42.288111 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:42.288077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:42.345659 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:42.345633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:43.257037 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:43.257006 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:43.257197 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:43.257065 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:43.261643 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:43.261624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:44.129926 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:44.129897 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:33:44.181448 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:44.181414 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85749c8588-g2jbb"] Jul 13 17:33:44.309056 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:44.309028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:44.370170 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:44.370145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:44.429309 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:44.429252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:47.228018 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:47.227895 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:47.228471 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:47.228291 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:33:56.823816 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:56.823781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:56.925254 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:56.925205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:56.977908 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:56.977865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:57.290079 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:57.290043 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:57.352796 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:57.352770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:33:57.953032 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:57.953001 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d986664cd-xt686" podUID="795b2b37-b7bc-4aac-b4da-6f21fc1eac94" containerName="console" containerID="cri-o://0aa95a61ecc55b73b0aaea12c9934603c13f42c2032da7d66ae4d07adc550900" gracePeriod=15 Jul 13 17:33:58.199462 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.199434 2564 generic.go:356] "Generic (PLEG): container finished" podID="795b2b37-b7bc-4aac-b4da-6f21fc1eac94" containerID="0aa95a61ecc55b73b0aaea12c9934603c13f42c2032da7d66ae4d07adc550900" exitCode=0 Jul 13 17:33:58.199592 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.199511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d986664cd-xt686" event={"ID":"795b2b37-b7bc-4aac-b4da-6f21fc1eac94","Type":"ContainerDied","Data":"0aa95a61ecc55b73b0aaea12c9934603c13f42c2032da7d66ae4d07adc550900"} Jul 13 17:33:58.233690 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.233671 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:58.328257 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328210 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-config\") pod \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " Jul 13 17:33:58.328392 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328270 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-oauth-serving-cert\") pod \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " Jul 13 17:33:58.328392 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328372 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-service-ca\") pod \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " Jul 13 17:33:58.328508 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328439 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-serving-cert\") pod \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " Jul 13 17:33:58.328508 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328467 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh2lj\" (UniqueName: \"kubernetes.io/projected/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-kube-api-access-sh2lj\") pod \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " Jul 13 17:33:58.328621 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328511 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-oauth-config\") pod \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " Jul 13 17:33:58.328621 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-trusted-ca-bundle\") pod \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\" (UID: \"795b2b37-b7bc-4aac-b4da-6f21fc1eac94\") " Jul 13 17:33:58.328621 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328590 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-config" (OuterVolumeSpecName: "console-config") pod "795b2b37-b7bc-4aac-b4da-6f21fc1eac94" (UID: "795b2b37-b7bc-4aac-b4da-6f21fc1eac94"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:33:58.328756 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "795b2b37-b7bc-4aac-b4da-6f21fc1eac94" (UID: "795b2b37-b7bc-4aac-b4da-6f21fc1eac94"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:33:58.328756 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328663 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-service-ca" (OuterVolumeSpecName: "service-ca") pod "795b2b37-b7bc-4aac-b4da-6f21fc1eac94" (UID: "795b2b37-b7bc-4aac-b4da-6f21fc1eac94"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:33:58.328996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328975 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:33:58.328996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "795b2b37-b7bc-4aac-b4da-6f21fc1eac94" (UID: "795b2b37-b7bc-4aac-b4da-6f21fc1eac94"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:33:58.329122 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.328998 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-oauth-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:33:58.329122 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.329013 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:33:58.330793 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.330769 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "795b2b37-b7bc-4aac-b4da-6f21fc1eac94" (UID: "795b2b37-b7bc-4aac-b4da-6f21fc1eac94"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:33:58.330941 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.330914 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "795b2b37-b7bc-4aac-b4da-6f21fc1eac94" (UID: "795b2b37-b7bc-4aac-b4da-6f21fc1eac94"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:33:58.330977 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.330940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-kube-api-access-sh2lj" (OuterVolumeSpecName: "kube-api-access-sh2lj") pod "795b2b37-b7bc-4aac-b4da-6f21fc1eac94" (UID: "795b2b37-b7bc-4aac-b4da-6f21fc1eac94"). InnerVolumeSpecName "kube-api-access-sh2lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:33:58.430229 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.430206 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:33:58.430229 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.430226 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2lj\" (UniqueName: \"kubernetes.io/projected/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-kube-api-access-sh2lj\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:33:58.430360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.430256 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-console-oauth-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:33:58.430360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:58.430265 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795b2b37-b7bc-4aac-b4da-6f21fc1eac94-trusted-ca-bundle\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:33:59.205054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:59.205020 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d986664cd-xt686" Jul 13 17:33:59.205054 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:59.205033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d986664cd-xt686" event={"ID":"795b2b37-b7bc-4aac-b4da-6f21fc1eac94","Type":"ContainerDied","Data":"ea8823009f0516d725b031efb573e7c36ae27d755321a9c5182ecd1463c42b8c"} Jul 13 17:33:59.205576 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:59.205099 2564 scope.go:117] "RemoveContainer" containerID="0aa95a61ecc55b73b0aaea12c9934603c13f42c2032da7d66ae4d07adc550900" Jul 13 17:33:59.229219 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:59.229190 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d986664cd-xt686"] Jul 13 17:33:59.235635 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:59.235604 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d986664cd-xt686"] Jul 13 17:33:59.637704 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:33:59.637673 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795b2b37-b7bc-4aac-b4da-6f21fc1eac94" path="/var/lib/kubelet/pods/795b2b37-b7bc-4aac-b4da-6f21fc1eac94/volumes" Jul 13 17:34:07.229472 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:07.229440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:34:07.234142 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:07.234117 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dcd9bfc4d-td6c8" Jul 13 17:34:09.104484 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.104456 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fx7fh" Jul 13 17:34:09.201846 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.201806 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85749c8588-g2jbb" podUID="8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" containerName="console" containerID="cri-o://b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a" gracePeriod=15 Jul 13 17:34:09.481420 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.481400 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:34:09.531010 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.530984 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-service-ca\") pod \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " Jul 13 17:34:09.531116 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531040 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vslp\" (UniqueName: \"kubernetes.io/projected/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-kube-api-access-6vslp\") pod \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " Jul 13 17:34:09.531116 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-config\") pod \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " Jul 13 17:34:09.531116 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531089 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-serving-cert\") pod \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " Jul 13 17:34:09.531116 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531113 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-oauth-config\") pod \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " Jul 13 17:34:09.531347 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531139 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-trusted-ca-bundle\") pod \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " Jul 13 17:34:09.531347 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531206 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-oauth-serving-cert\") pod \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\" (UID: \"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f\") " Jul 13 17:34:09.531451 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531399 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-service-ca" (OuterVolumeSpecName: "service-ca") pod "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" (UID: "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:34:09.531519 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531496 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-config" (OuterVolumeSpecName: "console-config") pod "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" (UID: "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:34:09.531615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531597 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:34:09.531679 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531620 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:34:09.531733 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531694 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" (UID: "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:34:09.531790 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.531701 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" (UID: "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:34:09.533277 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.533228 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" (UID: "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:34:09.533394 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.533349 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-kube-api-access-6vslp" (OuterVolumeSpecName: "kube-api-access-6vslp") pod "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" (UID: "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f"). InnerVolumeSpecName "kube-api-access-6vslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:34:09.533394 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.533354 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" (UID: "8ca7519c-c7ac-4f10-8ebc-41dd0eac624f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:34:09.632975 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.632930 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:34:09.632975 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.632950 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-console-oauth-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:34:09.632975 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.632960 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-trusted-ca-bundle\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:34:09.632975 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.632968 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-oauth-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:34:09.632975 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:09.632977 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vslp\" (UniqueName: \"kubernetes.io/projected/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f-kube-api-access-6vslp\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:34:10.261092 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.261009 2564 generic.go:356] "Generic (PLEG): container finished" podID="8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" containerID="b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a" exitCode=0 Jul 13 17:34:10.261092 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.261062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85749c8588-g2jbb" event={"ID":"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f","Type":"ContainerDied","Data":"b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a"} Jul 13 17:34:10.261092 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.261071 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85749c8588-g2jbb" Jul 13 17:34:10.261634 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.261106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85749c8588-g2jbb" event={"ID":"8ca7519c-c7ac-4f10-8ebc-41dd0eac624f","Type":"ContainerDied","Data":"53aca214b9e28ac55c0c3247da4b78b3d3f1405ade812090fa6783861547c1bc"} Jul 13 17:34:10.261634 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.261127 2564 scope.go:117] "RemoveContainer" containerID="b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a" Jul 13 17:34:10.269805 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.269789 2564 scope.go:117] "RemoveContainer" containerID="b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a" Jul 13 17:34:10.270080 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:34:10.270055 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a\": container with ID starting with b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a not found: ID does not exist" containerID="b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a" Jul 13 17:34:10.270123 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.270088 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a"} err="failed to get container status \"b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a\": rpc error: code = NotFound desc = could not find container \"b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a\": container with ID starting with b8a79f1fc5d103c492b829d557a30d855a33d3274e59560ec95f9357d368195a not found: ID does not exist" Jul 13 17:34:10.283475 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.283454 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85749c8588-g2jbb"] Jul 13 17:34:10.284833 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:10.284815 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85749c8588-g2jbb"] Jul 13 17:34:11.637430 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:11.637397 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" path="/var/lib/kubelet/pods/8ca7519c-c7ac-4f10-8ebc-41dd0eac624f/volumes" Jul 13 17:34:29.147044 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:29.147005 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:34:29.166122 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:29.166095 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:34:29.355390 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:29.355366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 17:34:29.515851 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:29.515767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:38.703731 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:38.703695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:38.843627 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:38.843599 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:45.691783 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:45.691746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:45.749397 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:45.749363 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:50.613467 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:50.613432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:50.666528 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:50.666495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:56.062176 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:56.062141 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5876448bbd-mg684"] Jul 13 17:34:56.171572 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:56.171516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:56.220728 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:56.220694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:56.272083 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:56.272042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:57.297120 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:57.297083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:57.396182 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:57.396150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:34:57.443992 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:34:57.443960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:21.081978 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.081932 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5876448bbd-mg684" podUID="beb82c81-f106-4a23-a4e4-c0e7c37c4087" containerName="console" containerID="cri-o://e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c" gracePeriod=15 Jul 13 17:35:21.321408 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.321384 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:35:21.461444 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-serving-cert\") pod \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " Jul 13 17:35:21.461444 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461390 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mlrb\" (UniqueName: \"kubernetes.io/projected/beb82c81-f106-4a23-a4e4-c0e7c37c4087-kube-api-access-2mlrb\") pod \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " Jul 13 17:35:21.461444 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461414 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-oauth-serving-cert\") pod \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " Jul 13 17:35:21.461717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-trusted-ca-bundle\") pod \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " Jul 13 17:35:21.461717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461503 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-config\") pod \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " Jul 13 17:35:21.461717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461583 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-oauth-config\") pod \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " Jul 13 17:35:21.461717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-service-ca\") pod \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\" (UID: \"beb82c81-f106-4a23-a4e4-c0e7c37c4087\") " Jul 13 17:35:21.461886 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461841 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "beb82c81-f106-4a23-a4e4-c0e7c37c4087" (UID: "beb82c81-f106-4a23-a4e4-c0e7c37c4087"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:35:21.461886 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.461866 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "beb82c81-f106-4a23-a4e4-c0e7c37c4087" (UID: "beb82c81-f106-4a23-a4e4-c0e7c37c4087"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:35:21.462084 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.462053 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-config" (OuterVolumeSpecName: "console-config") pod "beb82c81-f106-4a23-a4e4-c0e7c37c4087" (UID: "beb82c81-f106-4a23-a4e4-c0e7c37c4087"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:35:21.462197 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.462153 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-service-ca" (OuterVolumeSpecName: "service-ca") pod "beb82c81-f106-4a23-a4e4-c0e7c37c4087" (UID: "beb82c81-f106-4a23-a4e4-c0e7c37c4087"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:35:21.463608 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.463582 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "beb82c81-f106-4a23-a4e4-c0e7c37c4087" (UID: "beb82c81-f106-4a23-a4e4-c0e7c37c4087"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:35:21.463693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.463674 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "beb82c81-f106-4a23-a4e4-c0e7c37c4087" (UID: "beb82c81-f106-4a23-a4e4-c0e7c37c4087"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:35:21.463750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.463706 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb82c81-f106-4a23-a4e4-c0e7c37c4087-kube-api-access-2mlrb" (OuterVolumeSpecName: "kube-api-access-2mlrb") pod "beb82c81-f106-4a23-a4e4-c0e7c37c4087" (UID: "beb82c81-f106-4a23-a4e4-c0e7c37c4087"). InnerVolumeSpecName "kube-api-access-2mlrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:35:21.545850 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.545818 2564 generic.go:356] "Generic (PLEG): container finished" podID="beb82c81-f106-4a23-a4e4-c0e7c37c4087" containerID="e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c" exitCode=0 Jul 13 17:35:21.546020 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.545864 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5876448bbd-mg684" Jul 13 17:35:21.546020 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.545908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5876448bbd-mg684" event={"ID":"beb82c81-f106-4a23-a4e4-c0e7c37c4087","Type":"ContainerDied","Data":"e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c"} Jul 13 17:35:21.546020 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.545961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5876448bbd-mg684" event={"ID":"beb82c81-f106-4a23-a4e4-c0e7c37c4087","Type":"ContainerDied","Data":"2886f4bbf7419497b3e7b05773e2aeba8156a373b10d77a83e03569cb5bf9c5c"} Jul 13 17:35:21.546020 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.545986 2564 scope.go:117] "RemoveContainer" containerID="e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c" Jul 13 17:35:21.554380 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.554363 2564 scope.go:117] "RemoveContainer" containerID="e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c" Jul 13 17:35:21.554618 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:35:21.554600 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c\": container with ID starting with e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c not found: ID does not exist" containerID="e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c" Jul 13 17:35:21.554670 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.554626 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c"} err="failed to get container status \"e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c\": rpc error: code = NotFound desc = could not find container \"e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c\": container with ID starting with e48ffb08e801c0fd4766ddebde17b60acc63608171f8018d8ed0db396a679d4c not found: ID does not exist" Jul 13 17:35:21.563213 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.563192 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-oauth-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:35:21.563213 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.563213 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:35:21.563348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.563223 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:35:21.563348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.563245 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlrb\" (UniqueName: \"kubernetes.io/projected/beb82c81-f106-4a23-a4e4-c0e7c37c4087-kube-api-access-2mlrb\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:35:21.563348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.563264 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-oauth-serving-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:35:21.563348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.563272 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-trusted-ca-bundle\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:35:21.563348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.563280 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/beb82c81-f106-4a23-a4e4-c0e7c37c4087-console-config\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:35:21.566051 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.566030 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5876448bbd-mg684"] Jul 13 17:35:21.569859 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.569838 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5876448bbd-mg684"] Jul 13 17:35:21.636822 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:21.636787 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb82c81-f106-4a23-a4e4-c0e7c37c4087" path="/var/lib/kubelet/pods/beb82c81-f106-4a23-a4e4-c0e7c37c4087/volumes" Jul 13 17:35:23.866192 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:23.866165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:23.917504 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:23.917472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:25.934828 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:25.934796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:26.033748 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:26.033719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:52.319952 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:52.319916 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:53.668587 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:53.668551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:55.927892 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:55.927860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:56.027876 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:56.027846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:56.641419 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:56.641386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:56.761330 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:56.761301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:58.009890 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:58.009861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:35:58.075685 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:35:58.075656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:00.441347 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:00.441302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:00.917100 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:00.917069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:01.709739 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.709703 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl"] Jul 13 17:36:01.710096 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710064 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb82c81-f106-4a23-a4e4-c0e7c37c4087" containerName="console" Jul 13 17:36:01.710096 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710074 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb82c81-f106-4a23-a4e4-c0e7c37c4087" containerName="console" Jul 13 17:36:01.710096 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710087 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795b2b37-b7bc-4aac-b4da-6f21fc1eac94" containerName="console" Jul 13 17:36:01.710096 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710095 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="795b2b37-b7bc-4aac-b4da-6f21fc1eac94" containerName="console" Jul 13 17:36:01.710217 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710105 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" containerName="console" Jul 13 17:36:01.710217 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710110 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" containerName="console" Jul 13 17:36:01.710217 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710156 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="795b2b37-b7bc-4aac-b4da-6f21fc1eac94" containerName="console" Jul 13 17:36:01.710217 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710164 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb82c81-f106-4a23-a4e4-c0e7c37c4087" containerName="console" Jul 13 17:36:01.710217 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.710172 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca7519c-c7ac-4f10-8ebc-41dd0eac624f" containerName="console" Jul 13 17:36:01.712199 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.712183 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:01.714172 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.714155 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 13 17:36:01.714754 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.714736 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-c99ph\"" Jul 13 17:36:01.714852 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.714773 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 13 17:36:01.720724 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.720705 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl"] Jul 13 17:36:01.782472 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.782428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w8gv\" (UniqueName: \"kubernetes.io/projected/f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b-kube-api-access-2w8gv\") pod \"cert-manager-cainjector-559cdd8c87-mg8gl\" (UID: \"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:01.782644 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.782530 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mg8gl\" (UID: \"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:01.883171 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.883135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mg8gl\" (UID: \"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:01.883323 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.883221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w8gv\" (UniqueName: \"kubernetes.io/projected/f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b-kube-api-access-2w8gv\") pod \"cert-manager-cainjector-559cdd8c87-mg8gl\" (UID: \"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:01.892168 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.892128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mg8gl\" (UID: \"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:01.892330 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:01.892190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w8gv\" (UniqueName: \"kubernetes.io/projected/f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b-kube-api-access-2w8gv\") pod \"cert-manager-cainjector-559cdd8c87-mg8gl\" (UID: \"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:02.031316 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:02.031220 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" Jul 13 17:36:02.159710 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:02.159687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl"] Jul 13 17:36:02.162144 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:36:02.162115 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b5cebf_b6ca_4078_93f8_f8a57ff19a6b.slice/crio-f620263a6f0aba3c87e52201b1a493addce64bff16838f6ec78c1d97f19585c8 WatchSource:0}: Error finding container f620263a6f0aba3c87e52201b1a493addce64bff16838f6ec78c1d97f19585c8: Status 404 returned error can't find the container with id f620263a6f0aba3c87e52201b1a493addce64bff16838f6ec78c1d97f19585c8 Jul 13 17:36:02.704737 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:02.704701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" event={"ID":"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b","Type":"ContainerStarted","Data":"f620263a6f0aba3c87e52201b1a493addce64bff16838f6ec78c1d97f19585c8"} Jul 13 17:36:03.154607 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:03.154566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:03.255644 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:03.255603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:05.721300 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:05.721261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" event={"ID":"f3b5cebf-b6ca-4078-93f8-f8a57ff19a6b","Type":"ContainerStarted","Data":"7d73fbf19f6fa665e4efc505fc85d5daea52b777e3ad828bbc6b9812fbac3ee2"} Jul 13 17:36:05.738314 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:05.738211 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mg8gl" podStartSLOduration=1.645620687 podStartE2EDuration="4.738197768s" totalImagesPullingTime="3.092577081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:36:05.736878911 +0000 UTC m=+220.744281157" watchObservedRunningTime="2026-07-13 17:36:05.738197768 +0000 UTC m=+220.745600009" Jul 13 17:36:07.932653 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:07.932623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:08.032075 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.032044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:08.085101 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.085072 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:08.402681 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.402645 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mnqb4"] Jul 13 17:36:08.404980 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.404959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.406953 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.406929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8g9cl\"" Jul 13 17:36:08.413330 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.413307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mnqb4"] Jul 13 17:36:08.546001 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.545959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd38e56-62ff-4106-86e1-153b4607d8fc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mnqb4\" (UID: \"ebd38e56-62ff-4106-86e1-153b4607d8fc\") " pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.546174 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.546094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krdw\" (UniqueName: \"kubernetes.io/projected/ebd38e56-62ff-4106-86e1-153b4607d8fc-kube-api-access-9krdw\") pod \"cert-manager-55b4d954d5-mnqb4\" (UID: \"ebd38e56-62ff-4106-86e1-153b4607d8fc\") " pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.647418 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.647380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krdw\" (UniqueName: \"kubernetes.io/projected/ebd38e56-62ff-4106-86e1-153b4607d8fc-kube-api-access-9krdw\") pod \"cert-manager-55b4d954d5-mnqb4\" (UID: \"ebd38e56-62ff-4106-86e1-153b4607d8fc\") " pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.647577 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.647441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd38e56-62ff-4106-86e1-153b4607d8fc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mnqb4\" (UID: \"ebd38e56-62ff-4106-86e1-153b4607d8fc\") " pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.656693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.656631 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebd38e56-62ff-4106-86e1-153b4607d8fc-bound-sa-token\") pod \"cert-manager-55b4d954d5-mnqb4\" (UID: \"ebd38e56-62ff-4106-86e1-153b4607d8fc\") " pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.656812 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.656762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krdw\" (UniqueName: \"kubernetes.io/projected/ebd38e56-62ff-4106-86e1-153b4607d8fc-kube-api-access-9krdw\") pod \"cert-manager-55b4d954d5-mnqb4\" (UID: \"ebd38e56-62ff-4106-86e1-153b4607d8fc\") " pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.714341 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.714297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mnqb4" Jul 13 17:36:08.837442 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:08.837327 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mnqb4"] Jul 13 17:36:08.839929 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:36:08.839905 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebd38e56_62ff_4106_86e1_153b4607d8fc.slice/crio-84aff42c9d74618cb11db575152530a06037c3dced0f41a34c03976a9b641da3 WatchSource:0}: Error finding container 84aff42c9d74618cb11db575152530a06037c3dced0f41a34c03976a9b641da3: Status 404 returned error can't find the container with id 84aff42c9d74618cb11db575152530a06037c3dced0f41a34c03976a9b641da3 Jul 13 17:36:09.509438 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:09.509405 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:09.610128 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:09.610092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:09.739617 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:09.739579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mnqb4" event={"ID":"ebd38e56-62ff-4106-86e1-153b4607d8fc","Type":"ContainerStarted","Data":"6b442c3a2b6295db36d6427e36184ff6f561e92ec381280ef9db8fd8e6b4322b"} Jul 13 17:36:09.739617 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:09.739628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mnqb4" event={"ID":"ebd38e56-62ff-4106-86e1-153b4607d8fc","Type":"ContainerStarted","Data":"84aff42c9d74618cb11db575152530a06037c3dced0f41a34c03976a9b641da3"} Jul 13 17:36:09.756139 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:09.756099 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mnqb4" podStartSLOduration=1.756085547 podStartE2EDuration="1.756085547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:36:09.754567604 +0000 UTC m=+224.761969848" watchObservedRunningTime="2026-07-13 17:36:09.756085547 +0000 UTC m=+224.763487789" Jul 13 17:36:15.122178 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:15.122144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:15.219777 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:15.219743 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:16.936805 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:16.936769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:29.356654 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:29.356615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:29.461399 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:29.461347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:29.945719 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:29.945677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.112678 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.112646 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.211408 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.211328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.234840 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.234803 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc"] Jul 13 17:36:30.239944 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.239917 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.242457 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.242408 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 17:36:30.242633 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.242610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s8bgx\"" Jul 13 17:36:30.242733 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.242682 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 17:36:30.242824 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.242810 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 17:36:30.242883 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.242866 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 17:36:30.253939 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.253906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc"] Jul 13 17:36:30.340288 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.340213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-656xh\" (UniqueName: \"kubernetes.io/projected/fdf44874-a2e4-4769-834c-948b0d47384c-kube-api-access-656xh\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.340488 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.340343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fdf44874-a2e4-4769-834c-948b0d47384c-manifests\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.340488 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.340411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdf44874-a2e4-4769-834c-948b0d47384c-webhook-cert\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.340488 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.340456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf44874-a2e4-4769-834c-948b0d47384c-tmp\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.340488 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.340485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fdf44874-a2e4-4769-834c-948b0d47384c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.342163 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.342137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.441631 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.441593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fdf44874-a2e4-4769-834c-948b0d47384c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.442050 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.441670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-656xh\" (UniqueName: \"kubernetes.io/projected/fdf44874-a2e4-4769-834c-948b0d47384c-kube-api-access-656xh\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.442050 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.441716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fdf44874-a2e4-4769-834c-948b0d47384c-manifests\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.442050 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.441753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdf44874-a2e4-4769-834c-948b0d47384c-webhook-cert\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.442050 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.441799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf44874-a2e4-4769-834c-948b0d47384c-tmp\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.442317 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.442127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fdf44874-a2e4-4769-834c-948b0d47384c-manifests\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.443564 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.443537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.444207 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.444175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf44874-a2e4-4769-834c-948b0d47384c-tmp\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.444207 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.444188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdf44874-a2e4-4769-834c-948b0d47384c-webhook-cert\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.444420 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.444316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fdf44874-a2e4-4769-834c-948b0d47384c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.449555 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.449535 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-656xh\" (UniqueName: \"kubernetes.io/projected/fdf44874-a2e4-4769-834c-948b0d47384c-kube-api-access-656xh\") pod \"opendatahub-operator-controller-manager-59fbf78564-n4ckc\" (UID: \"fdf44874-a2e4-4769-834c-948b0d47384c\") " pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.540995 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.540963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.551326 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.551293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:30.648322 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.648281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.692859 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.692835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc"] Jul 13 17:36:30.694993 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:36:30.694965 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf44874_a2e4_4769_834c_948b0d47384c.slice/crio-2c073e7a7e9946be4555e2fce28d4cd02ac210d1e789518f6066ae6961653591 WatchSource:0}: Error finding container 2c073e7a7e9946be4555e2fce28d4cd02ac210d1e789518f6066ae6961653591: Status 404 returned error can't find the container with id 2c073e7a7e9946be4555e2fce28d4cd02ac210d1e789518f6066ae6961653591 Jul 13 17:36:30.752755 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.752723 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:30.827762 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:30.827682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" event={"ID":"fdf44874-a2e4-4769-834c-948b0d47384c","Type":"ContainerStarted","Data":"2c073e7a7e9946be4555e2fce28d4cd02ac210d1e789518f6066ae6961653591"} Jul 13 17:36:34.850348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:34.850313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" event={"ID":"fdf44874-a2e4-4769-834c-948b0d47384c","Type":"ContainerStarted","Data":"35288235e870c7d31d3da4f7f9420e40389f4457d316e5fd9fd4ddf025952d52"} Jul 13 17:36:35.855781 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:35.855745 2564 generic.go:356] "Generic (PLEG): container finished" podID="fdf44874-a2e4-4769-834c-948b0d47384c" containerID="35288235e870c7d31d3da4f7f9420e40389f4457d316e5fd9fd4ddf025952d52" exitCode=0 Jul 13 17:36:35.856188 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:35.855829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" event={"ID":"fdf44874-a2e4-4769-834c-948b0d47384c","Type":"ContainerDied","Data":"35288235e870c7d31d3da4f7f9420e40389f4457d316e5fd9fd4ddf025952d52"} Jul 13 17:36:36.862080 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:36.862044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" event={"ID":"fdf44874-a2e4-4769-834c-948b0d47384c","Type":"ContainerStarted","Data":"dd703254e2d91314478a8830fb976a330bd86e9af4858248e46c8314ca4b1d09"} Jul 13 17:36:36.862574 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:36.862175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:36.909184 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:36.909124 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" podStartSLOduration=2.166037748 podStartE2EDuration="6.909106988s" totalImagesPullingTime="3.653468557s" totalInitContainerRuntime="1.089600683s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:36:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:36:36.906664025 +0000 UTC m=+251.914066278" watchObservedRunningTime="2026-07-13 17:36:36.909106988 +0000 UTC m=+251.916509300" Jul 13 17:36:42.861288 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:42.861193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:42.952370 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:42.952337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:47.869394 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:47.869315 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59fbf78564-n4ckc" Jul 13 17:36:49.649451 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:49.649420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:49.751643 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:49.751605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:50.168549 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:50.168519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:50.288188 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:50.288152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:50.417855 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:50.417812 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:50.517936 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:50.517855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:50.618972 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:50.618943 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:51.122059 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:51.122028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:51.339402 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:51.339365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:51.445307 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:51.445192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:52.278204 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:52.278174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:58.450610 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.450578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:58.550741 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.550706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:58.729342 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.729260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:58.820636 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.820600 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26"] Jul 13 17:36:58.824164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.824146 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:58.828665 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.828641 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 17:36:58.829057 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.829040 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 17:36:58.829057 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.829043 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xkxn6\"" Jul 13 17:36:58.829209 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.829103 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 17:36:58.852547 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.852525 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26"] Jul 13 17:36:58.974623 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.974585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:58.999780 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.999703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:58.999780 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.999741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:58.999780 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.999769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bcvf\" (UniqueName: \"kubernetes.io/projected/4f9a8244-cb35-479c-928a-7a21b0dcf134-kube-api-access-2bcvf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:58.999985 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.999823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:58.999985 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.999852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:58.999985 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.999883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:58.999985 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:58.999906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9a8244-cb35-479c-928a-7a21b0dcf134-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.000110 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.000014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.000110 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.000044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.063416 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.063379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:36:59.100788 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.100788 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101005 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100810 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bcvf\" (UniqueName: \"kubernetes.io/projected/4f9a8244-cb35-479c-928a-7a21b0dcf134-kube-api-access-2bcvf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101005 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101005 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101005 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101005 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9a8244-cb35-479c-928a-7a21b0dcf134-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101005 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101005 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.100964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101352 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.101217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101352 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.101263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101498 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.101473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101622 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.101506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.101669 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.101619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9a8244-cb35-479c-928a-7a21b0dcf134-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.103316 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.103289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.103419 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.103290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.111676 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.111656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9a8244-cb35-479c-928a-7a21b0dcf134-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.111945 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.111928 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bcvf\" (UniqueName: \"kubernetes.io/projected/4f9a8244-cb35-479c-928a-7a21b0dcf134-kube-api-access-2bcvf\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xtj26\" (UID: \"4f9a8244-cb35-479c-928a-7a21b0dcf134\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.134858 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.134813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:36:59.276824 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.276799 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26"] Jul 13 17:36:59.279176 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:36:59.279140 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9a8244_cb35_479c_928a_7a21b0dcf134.slice/crio-a328607a0e495a0a3a347d6314cd83dca8825958fa9517965f6fc9b9854a5ac9 WatchSource:0}: Error finding container a328607a0e495a0a3a347d6314cd83dca8825958fa9517965f6fc9b9854a5ac9: Status 404 returned error can't find the container with id a328607a0e495a0a3a347d6314cd83dca8825958fa9517965f6fc9b9854a5ac9 Jul 13 17:36:59.967648 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:36:59.967600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" event={"ID":"4f9a8244-cb35-479c-928a-7a21b0dcf134","Type":"ContainerStarted","Data":"a328607a0e495a0a3a347d6314cd83dca8825958fa9517965f6fc9b9854a5ac9"} Jul 13 17:37:01.239747 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.239706 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs"] Jul 13 17:37:01.243785 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.243762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.246255 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.246180 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 13 17:37:01.246255 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.246206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qbdvt\"" Jul 13 17:37:01.246440 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.246183 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 13 17:37:01.246591 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.246569 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 13 17:37:01.246715 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.246595 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 13 17:37:01.246715 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.246619 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 13 17:37:01.253431 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.253373 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs"] Jul 13 17:37:01.422153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.422118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32cc088-a455-4bf1-b259-5bec26ff5918-cert\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.422341 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.422161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jw99\" (UniqueName: \"kubernetes.io/projected/b32cc088-a455-4bf1-b259-5bec26ff5918-kube-api-access-5jw99\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.422341 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.422306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b32cc088-a455-4bf1-b259-5bec26ff5918-manager-config\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.422468 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.422349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b32cc088-a455-4bf1-b259-5bec26ff5918-metrics-cert\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.523628 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.523555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32cc088-a455-4bf1-b259-5bec26ff5918-cert\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.523628 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.523589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jw99\" (UniqueName: \"kubernetes.io/projected/b32cc088-a455-4bf1-b259-5bec26ff5918-kube-api-access-5jw99\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.523844 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.523633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b32cc088-a455-4bf1-b259-5bec26ff5918-manager-config\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.523844 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.523686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b32cc088-a455-4bf1-b259-5bec26ff5918-metrics-cert\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.524278 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.524256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b32cc088-a455-4bf1-b259-5bec26ff5918-manager-config\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.526247 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.526212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32cc088-a455-4bf1-b259-5bec26ff5918-cert\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.526352 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.526267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b32cc088-a455-4bf1-b259-5bec26ff5918-metrics-cert\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.537024 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.537003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jw99\" (UniqueName: \"kubernetes.io/projected/b32cc088-a455-4bf1-b259-5bec26ff5918-kube-api-access-5jw99\") pod \"lws-controller-manager-84567cbc5f-68grs\" (UID: \"b32cc088-a455-4bf1-b259-5bec26ff5918\") " pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.557440 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.557419 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:01.702084 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.702049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs"] Jul 13 17:37:01.704866 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:37:01.704835 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32cc088_a455_4bf1_b259_5bec26ff5918.slice/crio-a1c8deb3507a55ab964a7c4620b8ba4873b687c56b39b2afd288e5ac8636d124 WatchSource:0}: Error finding container a1c8deb3507a55ab964a7c4620b8ba4873b687c56b39b2afd288e5ac8636d124: Status 404 returned error can't find the container with id a1c8deb3507a55ab964a7c4620b8ba4873b687c56b39b2afd288e5ac8636d124 Jul 13 17:37:01.785280 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.785205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:01.834160 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.834127 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v"] Jul 13 17:37:01.839065 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.839042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:01.841353 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.841332 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 13 17:37:01.841485 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.841353 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qb6bx\"" Jul 13 17:37:01.843913 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.843893 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt"] Jul 13 17:37:01.847917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.847768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v"] Jul 13 17:37:01.847917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.847865 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:01.850065 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.850045 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 17:37:01.850282 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.850264 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4xhwb\"" Jul 13 17:37:01.858202 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.858180 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt"] Jul 13 17:37:01.880338 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.880305 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:01.927776 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.927743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccf2v\" (UID: \"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:01.927917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.927879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmj5c\" (UniqueName: \"kubernetes.io/projected/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-kube-api-access-nmj5c\") pod \"kserve-controller-manager-d6f4b65b8-ccf2v\" (UID: \"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:01.980949 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:01.980914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" event={"ID":"b32cc088-a455-4bf1-b259-5bec26ff5918","Type":"ContainerStarted","Data":"a1c8deb3507a55ab964a7c4620b8ba4873b687c56b39b2afd288e5ac8636d124"} Jul 13 17:37:02.029379 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.029351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74ddef11-460c-445b-a5f5-59d0843387c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-spqdt\" (UID: \"74ddef11-460c-445b-a5f5-59d0843387c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.029539 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.029421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccf2v\" (UID: \"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:02.029539 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.029465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmj5c\" (UniqueName: \"kubernetes.io/projected/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-kube-api-access-nmj5c\") pod \"kserve-controller-manager-d6f4b65b8-ccf2v\" (UID: \"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:02.029539 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.029517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4bg\" (UniqueName: \"kubernetes.io/projected/74ddef11-460c-445b-a5f5-59d0843387c8-kube-api-access-lr4bg\") pod \"llmisvc-controller-manager-777c6bd7d5-spqdt\" (UID: \"74ddef11-460c-445b-a5f5-59d0843387c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.029692 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:37:02.029546 2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 13 17:37:02.029692 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:37:02.029610 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-cert podName:4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541 nodeName:}" failed. No retries permitted until 2026-07-13 17:37:02.529587591 +0000 UTC m=+277.536989818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-cert") pod "kserve-controller-manager-d6f4b65b8-ccf2v" (UID: "4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541") : secret "kserve-webhook-server-cert" not found Jul 13 17:37:02.049144 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.049067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmj5c\" (UniqueName: \"kubernetes.io/projected/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-kube-api-access-nmj5c\") pod \"kserve-controller-manager-d6f4b65b8-ccf2v\" (UID: \"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:02.075657 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.075621 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:02.130182 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.130151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4bg\" (UniqueName: \"kubernetes.io/projected/74ddef11-460c-445b-a5f5-59d0843387c8-kube-api-access-lr4bg\") pod \"llmisvc-controller-manager-777c6bd7d5-spqdt\" (UID: \"74ddef11-460c-445b-a5f5-59d0843387c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.130390 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.130220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74ddef11-460c-445b-a5f5-59d0843387c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-spqdt\" (UID: \"74ddef11-460c-445b-a5f5-59d0843387c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.130459 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:37:02.130417 2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 13 17:37:02.130513 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:37:02.130490 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddef11-460c-445b-a5f5-59d0843387c8-cert podName:74ddef11-460c-445b-a5f5-59d0843387c8 nodeName:}" failed. No retries permitted until 2026-07-13 17:37:02.630468602 +0000 UTC m=+277.637870828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/74ddef11-460c-445b-a5f5-59d0843387c8-cert") pod "llmisvc-controller-manager-777c6bd7d5-spqdt" (UID: "74ddef11-460c-445b-a5f5-59d0843387c8") : secret "llmisvc-webhook-server-cert" not found Jul 13 17:37:02.142073 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.142049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4bg\" (UniqueName: \"kubernetes.io/projected/74ddef11-460c-445b-a5f5-59d0843387c8-kube-api-access-lr4bg\") pod \"llmisvc-controller-manager-777c6bd7d5-spqdt\" (UID: \"74ddef11-460c-445b-a5f5-59d0843387c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.173581 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.173544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:02.274651 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.274616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:02.370512 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.370429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:02.533179 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.533141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccf2v\" (UID: \"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:02.536168 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.536143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccf2v\" (UID: \"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:02.634228 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.634136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74ddef11-460c-445b-a5f5-59d0843387c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-spqdt\" (UID: \"74ddef11-460c-445b-a5f5-59d0843387c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.636815 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.636789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74ddef11-460c-445b-a5f5-59d0843387c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-spqdt\" (UID: \"74ddef11-460c-445b-a5f5-59d0843387c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.754029 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.753991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:02.765928 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.765899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:02.968403 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.968328 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:37:02.968531 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.968429 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:37:02.968531 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:02.968466 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:37:03.064156 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:03.064132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt"] Jul 13 17:37:03.067037 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:37:03.067009 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod74ddef11_460c_445b_a5f5_59d0843387c8.slice/crio-a2de64abfaac1b430f8dcb3aa7bb60346986cd4a9361a077f2a5a98da515cd07 WatchSource:0}: Error finding container a2de64abfaac1b430f8dcb3aa7bb60346986cd4a9361a077f2a5a98da515cd07: Status 404 returned error can't find the container with id a2de64abfaac1b430f8dcb3aa7bb60346986cd4a9361a077f2a5a98da515cd07 Jul 13 17:37:03.079169 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:03.079140 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v"] Jul 13 17:37:03.081541 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:37:03.081512 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2a1d4b_bdf6_4ecc_9e92_42144bdd9541.slice/crio-09cf84ae6bea58e62c0081502ab1ef6e74d6ba88a6100c9b5f8ddb41cddd3e4e WatchSource:0}: Error finding container 09cf84ae6bea58e62c0081502ab1ef6e74d6ba88a6100c9b5f8ddb41cddd3e4e: Status 404 returned error can't find the container with id 09cf84ae6bea58e62c0081502ab1ef6e74d6ba88a6100c9b5f8ddb41cddd3e4e Jul 13 17:37:03.997651 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:03.997614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" event={"ID":"74ddef11-460c-445b-a5f5-59d0843387c8","Type":"ContainerStarted","Data":"a2de64abfaac1b430f8dcb3aa7bb60346986cd4a9361a077f2a5a98da515cd07"} Jul 13 17:37:03.999046 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:03.999012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" event={"ID":"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541","Type":"ContainerStarted","Data":"09cf84ae6bea58e62c0081502ab1ef6e74d6ba88a6100c9b5f8ddb41cddd3e4e"} Jul 13 17:37:04.000611 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:04.000586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" event={"ID":"4f9a8244-cb35-479c-928a-7a21b0dcf134","Type":"ContainerStarted","Data":"614ca39a3a6b551f53f7b5a0cce7de4fcb9fa6c8bc5eb5b425b3d8d3ac61e266"} Jul 13 17:37:04.023619 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:04.023545 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" podStartSLOduration=2.337037458 podStartE2EDuration="6.023525931s" totalImagesPullingTime="3.686488473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:37:04.021322356 +0000 UTC m=+279.028724621" watchObservedRunningTime="2026-07-13 17:37:04.023525931 +0000 UTC m=+279.030928175" Jul 13 17:37:04.135494 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:04.135429 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:37:04.142394 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:04.142360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:37:05.008618 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:05.008569 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:37:05.009640 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:05.009617 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xtj26" Jul 13 17:37:07.021294 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:07.021249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" event={"ID":"4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541","Type":"ContainerStarted","Data":"4d818e223602f734c7d0104468a44c7afe15ddc2519b86849896f91053a686e3"} Jul 13 17:37:07.049565 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:07.049511 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" podStartSLOduration=2.766127139 podStartE2EDuration="6.049493904s" totalImagesPullingTime="3.283366765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:37:07.048476053 +0000 UTC m=+282.055878297" watchObservedRunningTime="2026-07-13 17:37:07.049493904 +0000 UTC m=+282.056896147" Jul 13 17:37:08.026459 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:08.026375 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:09.032928 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:09.032882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" event={"ID":"74ddef11-460c-445b-a5f5-59d0843387c8","Type":"ContainerStarted","Data":"c40fcddec14c5033ff922ea41b0a76e980444b96c706335eafe22bbf4be4bc7b"} Jul 13 17:37:09.033417 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:09.032946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:09.034421 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:09.034391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" event={"ID":"b32cc088-a455-4bf1-b259-5bec26ff5918","Type":"ContainerStarted","Data":"374ecfceb7bbad3a1e72744bbd32fb68a7a97f46788e5a6ebedf5201984de763"} Jul 13 17:37:09.034559 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:09.034521 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:09.049970 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:09.049929 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" podStartSLOduration=3.048235592 podStartE2EDuration="8.049916512s" totalImagesPullingTime="5.00168092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:37:09.047864071 +0000 UTC m=+284.055266314" watchObservedRunningTime="2026-07-13 17:37:09.049916512 +0000 UTC m=+284.057318755" Jul 13 17:37:09.064765 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:09.064722 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" podStartSLOduration=1.660459696 podStartE2EDuration="8.064708703s" totalImagesPullingTime="6.404249007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:37:09.063032221 +0000 UTC m=+284.070434489" watchObservedRunningTime="2026-07-13 17:37:09.064708703 +0000 UTC m=+284.072110949" Jul 13 17:37:20.041227 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:20.041197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-84567cbc5f-68grs" Jul 13 17:37:20.198471 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:20.198434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:25.445118 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:25.445092 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:37:25.448560 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:25.448538 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:37:25.449411 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:25.449390 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 17:37:39.039541 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:39.039505 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccf2v" Jul 13 17:37:40.041842 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:40.041816 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-spqdt" Jul 13 17:37:51.108777 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:51.108734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:51.205981 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:51.205934 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:51.333876 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:51.333847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:52.052301 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:52.052272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:52.254770 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:52.254726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:37:59.335704 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:37:59.335667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:20.026937 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:20.026886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:23.620679 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:23.620645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:26.654192 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:26.654157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:28.203903 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:28.203859 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:28.311170 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:28.311138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:28.826565 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:28.826533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:31.474133 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:31.474101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:32.438085 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:32.438053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:34.255153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.255117 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm"] Jul 13 17:38:34.258501 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.258482 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:34.260801 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.260777 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 17:38:34.261698 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.261682 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 17:38:34.261766 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.261700 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-88sms\"" Jul 13 17:38:34.270366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.270341 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm"] Jul 13 17:38:34.284534 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.284502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:34.370504 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.370476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2z7\" (UniqueName: \"kubernetes.io/projected/6e6535b8-f926-4b83-99a6-25491a32052e-kube-api-access-xr2z7\") pod \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" (UID: \"6e6535b8-f926-4b83-99a6-25491a32052e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:34.471714 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.471680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2z7\" (UniqueName: \"kubernetes.io/projected/6e6535b8-f926-4b83-99a6-25491a32052e-kube-api-access-xr2z7\") pod \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" (UID: \"6e6535b8-f926-4b83-99a6-25491a32052e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:34.486753 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.486725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2z7\" (UniqueName: \"kubernetes.io/projected/6e6535b8-f926-4b83-99a6-25491a32052e-kube-api-access-xr2z7\") pod \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" (UID: \"6e6535b8-f926-4b83-99a6-25491a32052e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:34.571336 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.571303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:34.703513 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.703490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm"] Jul 13 17:38:34.705569 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:38:34.705541 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6535b8_f926_4b83_99a6_25491a32052e.slice/crio-13b5828b2c998f17687eeb4cae1938c84fee2702f2ef28123113472598fc168b WatchSource:0}: Error finding container 13b5828b2c998f17687eeb4cae1938c84fee2702f2ef28123113472598fc168b: Status 404 returned error can't find the container with id 13b5828b2c998f17687eeb4cae1938c84fee2702f2ef28123113472598fc168b Jul 13 17:38:34.707704 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:34.707690 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 17:38:35.444429 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:35.444395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" event={"ID":"6e6535b8-f926-4b83-99a6-25491a32052e","Type":"ContainerStarted","Data":"13b5828b2c998f17687eeb4cae1938c84fee2702f2ef28123113472598fc168b"} Jul 13 17:38:35.836819 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:35.836780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:35.933660 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:35.933628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:37.455703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.455666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" event={"ID":"6e6535b8-f926-4b83-99a6-25491a32052e","Type":"ContainerStarted","Data":"11e1d7bb841afc007fe12e0fd9a2ab7c50727ad0a63f6de4b20cf330e41aacaf"} Jul 13 17:38:37.456077 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.455733 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:37.471755 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.471691 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" podStartSLOduration=1.4318754710000001 podStartE2EDuration="3.471676814s" totalImagesPullingTime="2.039801343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:38:37.470696377 +0000 UTC m=+372.478098647" watchObservedRunningTime="2026-07-13 17:38:37.471676814 +0000 UTC m=+372.479079056" Jul 13 17:38:37.971506 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.971469 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p"] Jul 13 17:38:37.974959 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.974935 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:37.977137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.977104 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 13 17:38:37.977269 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.977137 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 13 17:38:37.977269 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.977164 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l446t\"" Jul 13 17:38:37.982089 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.982066 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p"] Jul 13 17:38:37.984577 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:37.984548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:38.005386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.005366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.005475 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.005419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.005475 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.005443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvpc\" (UniqueName: \"kubernetes.io/projected/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-kube-api-access-9dvpc\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.106429 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.106406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvpc\" (UniqueName: \"kubernetes.io/projected/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-kube-api-access-9dvpc\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.106520 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.106476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.106559 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.106549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.106635 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:38:38.106618 2564 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 13 17:38:38.106685 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:38:38.106675 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-plugin-serving-cert podName:20e6b5f0-fd39-4dd9-8ee0-598baa5f295f nodeName:}" failed. No retries permitted until 2026-07-13 17:38:38.606660688 +0000 UTC m=+373.614062908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-dgq4p" (UID: "20e6b5f0-fd39-4dd9-8ee0-598baa5f295f") : secret "plugin-serving-cert" not found Jul 13 17:38:38.107095 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.107077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.117789 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.117761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvpc\" (UniqueName: \"kubernetes.io/projected/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-kube-api-access-9dvpc\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.440111 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.440076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:38.612317 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.612276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.614672 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.614642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/20e6b5f0-fd39-4dd9-8ee0-598baa5f295f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dgq4p\" (UID: \"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.802335 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.802294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:38.885679 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.885646 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" Jul 13 17:38:38.901927 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:38.901897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:39.007192 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:39.007160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:39.015740 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:39.015721 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p"] Jul 13 17:38:39.018139 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:38:39.018112 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20e6b5f0_fd39_4dd9_8ee0_598baa5f295f.slice/crio-e9ed916f8fa1dfcfd615996372e951f1078db4855887faa742d57cebbdeb2975 WatchSource:0}: Error finding container e9ed916f8fa1dfcfd615996372e951f1078db4855887faa742d57cebbdeb2975: Status 404 returned error can't find the container with id e9ed916f8fa1dfcfd615996372e951f1078db4855887faa742d57cebbdeb2975 Jul 13 17:38:39.428043 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:39.428008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:39.466810 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:39.466774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" event={"ID":"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f","Type":"ContainerStarted","Data":"e9ed916f8fa1dfcfd615996372e951f1078db4855887faa742d57cebbdeb2975"} Jul 13 17:38:39.528273 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:39.528227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:39.628925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:39.628886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:43.871928 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:43.871553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:43.935351 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:43.935316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:45.869107 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:45.869069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:45.970006 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:45.969975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:48.462974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.462942 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:48.676415 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.676375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:48.773614 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.773539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:48.915789 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.915761 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv"] Jul 13 17:38:48.919578 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.919549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:48.921729 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.921710 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5wrtc\"" Jul 13 17:38:48.937447 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.937424 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv"] Jul 13 17:38:48.945184 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.945162 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm"] Jul 13 17:38:48.945389 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.945364 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" containerName="manager" containerID="cri-o://11e1d7bb841afc007fe12e0fd9a2ab7c50727ad0a63f6de4b20cf330e41aacaf" gracePeriod=2 Jul 13 17:38:48.954482 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:48.954462 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm"] Jul 13 17:38:48.977280 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:38:48.977217 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:38:49.016661 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.016636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djqnn\" (UniqueName: \"kubernetes.io/projected/bfaa3785-6442-497c-9781-3f4006fb977f-kube-api-access-djqnn\") pod \"kuadrant-operator-controller-manager-55467d9f7-v6ttv\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:49.016794 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.016671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa3785-6442-497c-9781-3f4006fb977f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v6ttv\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:49.117523 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.117495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqnn\" (UniqueName: \"kubernetes.io/projected/bfaa3785-6442-497c-9781-3f4006fb977f-kube-api-access-djqnn\") pod \"kuadrant-operator-controller-manager-55467d9f7-v6ttv\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:49.117684 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.117533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa3785-6442-497c-9781-3f4006fb977f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v6ttv\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:49.117967 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.117939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa3785-6442-497c-9781-3f4006fb977f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v6ttv\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:49.127974 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.127951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqnn\" (UniqueName: \"kubernetes.io/projected/bfaa3785-6442-497c-9781-3f4006fb977f-kube-api-access-djqnn\") pod \"kuadrant-operator-controller-manager-55467d9f7-v6ttv\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:49.231675 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.231641 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:38:49.352496 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.352458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:49.526772 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.526691 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e6535b8-f926-4b83-99a6-25491a32052e" containerID="11e1d7bb841afc007fe12e0fd9a2ab7c50727ad0a63f6de4b20cf330e41aacaf" exitCode=0 Jul 13 17:38:49.529428 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.529398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:49.623917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:49.623880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:51.661433 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:51.661403 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:51.761910 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:51.761877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:54.240458 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:54.240415 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:54.342138 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:54.342101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:59.267628 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:59.267577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:38:59.324574 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:38:59.324510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:03.441608 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.441585 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv"] Jul 13 17:39:03.443825 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:39:03.443799 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaa3785_6442_497c_9781_3f4006fb977f.slice/crio-806677b672644238c03ca95e77e585f470f681ef33dca82cefddc38d7da90249 WatchSource:0}: Error finding container 806677b672644238c03ca95e77e585f470f681ef33dca82cefddc38d7da90249: Status 404 returned error can't find the container with id 806677b672644238c03ca95e77e585f470f681ef33dca82cefddc38d7da90249 Jul 13 17:39:03.477778 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.477752 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:39:03.479749 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:39:03.479720 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:39:03.608313 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.608220 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:39:03.608313 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.608261 2564 scope.go:117] "RemoveContainer" containerID="11e1d7bb841afc007fe12e0fd9a2ab7c50727ad0a63f6de4b20cf330e41aacaf" Jul 13 17:39:03.609664 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.609635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" event={"ID":"bfaa3785-6442-497c-9781-3f4006fb977f","Type":"ContainerStarted","Data":"806677b672644238c03ca95e77e585f470f681ef33dca82cefddc38d7da90249"} Jul 13 17:39:03.610735 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:39:03.610706 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:39:03.658961 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.658928 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr2z7\" (UniqueName: \"kubernetes.io/projected/6e6535b8-f926-4b83-99a6-25491a32052e-kube-api-access-xr2z7\") pod \"6e6535b8-f926-4b83-99a6-25491a32052e\" (UID: \"6e6535b8-f926-4b83-99a6-25491a32052e\") " Jul 13 17:39:03.660909 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.660887 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6535b8-f926-4b83-99a6-25491a32052e-kube-api-access-xr2z7" (OuterVolumeSpecName: "kube-api-access-xr2z7") pod "6e6535b8-f926-4b83-99a6-25491a32052e" (UID: "6e6535b8-f926-4b83-99a6-25491a32052e"). InnerVolumeSpecName "kube-api-access-xr2z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:39:03.760262 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:03.760216 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2z7\" (UniqueName: \"kubernetes.io/projected/6e6535b8-f926-4b83-99a6-25491a32052e-kube-api-access-xr2z7\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:39:03.918323 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:39:03.918290 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:39:04.617051 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:04.616970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" event={"ID":"20e6b5f0-fd39-4dd9-8ee0-598baa5f295f","Type":"ContainerStarted","Data":"2608b9ef70369be621d147e2d187a988f205574c87192319f2958892adfe5142"} Jul 13 17:39:04.618814 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:39:04.618787 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:39:04.635952 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:04.635890 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dgq4p" podStartSLOduration=2.940878725 podStartE2EDuration="27.635868611s" totalImagesPullingTime="24.694989886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:39:04.632855936 +0000 UTC m=+399.640258192" watchObservedRunningTime="2026-07-13 17:39:04.635868611 +0000 UTC m=+399.643270855" Jul 13 17:39:05.637053 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:39:05.637009 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s8wkm\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s8wkm" Jul 13 17:39:05.639419 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:05.639394 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" path="/var/lib/kubelet/pods/6e6535b8-f926-4b83-99a6-25491a32052e/volumes" Jul 13 17:39:09.040640 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:09.040548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:09.099862 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:09.099781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:09.244021 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:09.243984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:09.304992 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:09.304922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:10.650693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:10.650602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" event={"ID":"bfaa3785-6442-497c-9781-3f4006fb977f","Type":"ContainerStarted","Data":"69e072aec018a674ac81d89d9281b5399d61a1c90280d1f56f5f0b1d7516153f"} Jul 13 17:39:10.650693 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:10.650676 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:39:10.671146 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:10.671095 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" podStartSLOduration=15.753139303 podStartE2EDuration="22.671077533s" totalImagesPullingTime="6.91793823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:39:10.669227847 +0000 UTC m=+405.676630090" watchObservedRunningTime="2026-07-13 17:39:10.671077533 +0000 UTC m=+405.678479776" Jul 13 17:39:12.452617 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:12.452574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:12.552131 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:12.552098 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:21.657018 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:21.656989 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:39:34.324495 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:34.324459 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv"] Jul 13 17:39:34.324915 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:34.324724 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" podUID="bfaa3785-6442-497c-9781-3f4006fb977f" containerName="manager" containerID="cri-o://69e072aec018a674ac81d89d9281b5399d61a1c90280d1f56f5f0b1d7516153f" gracePeriod=10 Jul 13 17:39:34.762267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:34.762213 2564 generic.go:356] "Generic (PLEG): container finished" podID="bfaa3785-6442-497c-9781-3f4006fb977f" containerID="69e072aec018a674ac81d89d9281b5399d61a1c90280d1f56f5f0b1d7516153f" exitCode=0 Jul 13 17:39:34.762267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:34.762261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" event={"ID":"bfaa3785-6442-497c-9781-3f4006fb977f","Type":"ContainerDied","Data":"69e072aec018a674ac81d89d9281b5399d61a1c90280d1f56f5f0b1d7516153f"} Jul 13 17:39:35.075453 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.075427 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:39:35.133171 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.133126 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa3785-6442-497c-9781-3f4006fb977f-extensions-socket-volume\") pod \"bfaa3785-6442-497c-9781-3f4006fb977f\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " Jul 13 17:39:35.133368 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.133208 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djqnn\" (UniqueName: \"kubernetes.io/projected/bfaa3785-6442-497c-9781-3f4006fb977f-kube-api-access-djqnn\") pod \"bfaa3785-6442-497c-9781-3f4006fb977f\" (UID: \"bfaa3785-6442-497c-9781-3f4006fb977f\") " Jul 13 17:39:35.133467 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.133446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaa3785-6442-497c-9781-3f4006fb977f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bfaa3785-6442-497c-9781-3f4006fb977f" (UID: "bfaa3785-6442-497c-9781-3f4006fb977f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 17:39:35.133571 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.133550 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfaa3785-6442-497c-9781-3f4006fb977f-extensions-socket-volume\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:39:35.135291 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.135268 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaa3785-6442-497c-9781-3f4006fb977f-kube-api-access-djqnn" (OuterVolumeSpecName: "kube-api-access-djqnn") pod "bfaa3785-6442-497c-9781-3f4006fb977f" (UID: "bfaa3785-6442-497c-9781-3f4006fb977f"). InnerVolumeSpecName "kube-api-access-djqnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:39:35.234681 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.234594 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djqnn\" (UniqueName: \"kubernetes.io/projected/bfaa3785-6442-497c-9781-3f4006fb977f-kube-api-access-djqnn\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:39:35.768503 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.768465 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" Jul 13 17:39:35.768503 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.768470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv" event={"ID":"bfaa3785-6442-497c-9781-3f4006fb977f","Type":"ContainerDied","Data":"806677b672644238c03ca95e77e585f470f681ef33dca82cefddc38d7da90249"} Jul 13 17:39:35.769022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.768537 2564 scope.go:117] "RemoveContainer" containerID="69e072aec018a674ac81d89d9281b5399d61a1c90280d1f56f5f0b1d7516153f" Jul 13 17:39:35.786100 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.786069 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv"] Jul 13 17:39:35.790345 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:35.790315 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v6ttv"] Jul 13 17:39:37.638164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:37.638127 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfaa3785-6442-497c-9781-3f4006fb977f" path="/var/lib/kubelet/pods/bfaa3785-6442-497c-9781-3f4006fb977f/volumes" Jul 13 17:39:38.134004 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.133967 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm"] Jul 13 17:39:38.134595 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.134573 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" containerName="manager" Jul 13 17:39:38.134595 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.134594 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" containerName="manager" Jul 13 17:39:38.134729 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.134640 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaa3785-6442-497c-9781-3f4006fb977f" containerName="manager" Jul 13 17:39:38.134729 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.134648 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaa3785-6442-497c-9781-3f4006fb977f" containerName="manager" Jul 13 17:39:38.134793 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.134736 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e6535b8-f926-4b83-99a6-25491a32052e" containerName="manager" Jul 13 17:39:38.134793 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.134752 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfaa3785-6442-497c-9781-3f4006fb977f" containerName="manager" Jul 13 17:39:38.137093 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.137071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.139382 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.139361 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dbv2d\"" Jul 13 17:39:38.153070 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.153036 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm"] Jul 13 17:39:38.161506 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.161471 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:38.261867 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.261818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262087 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.261884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b20046-7323-4547-b96c-0931ce17020a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262087 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.262004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262087 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.262037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b20046-7323-4547-b96c-0931ce17020a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262087 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.262065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262315 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.262115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26md8\" (UniqueName: \"kubernetes.io/projected/88b20046-7323-4547-b96c-0931ce17020a-kube-api-access-26md8\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262315 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.262155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b20046-7323-4547-b96c-0931ce17020a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262315 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.262193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.262315 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.262213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363063 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b20046-7323-4547-b96c-0931ce17020a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b20046-7323-4547-b96c-0931ce17020a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26md8\" (UniqueName: \"kubernetes.io/projected/88b20046-7323-4547-b96c-0931ce17020a-kube-api-access-26md8\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b20046-7323-4547-b96c-0931ce17020a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363766 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363766 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363766 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363766 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363631 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363977 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.363977 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.363959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b20046-7323-4547-b96c-0931ce17020a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.366025 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.365994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b20046-7323-4547-b96c-0931ce17020a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.366170 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.366152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b20046-7323-4547-b96c-0931ce17020a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.372698 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.372665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b20046-7323-4547-b96c-0931ce17020a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.372925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.372904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26md8\" (UniqueName: \"kubernetes.io/projected/88b20046-7323-4547-b96c-0931ce17020a-kube-api-access-26md8\") pod \"maas-default-gateway-openshift-default-56654dcf7-87fqm\" (UID: \"88b20046-7323-4547-b96c-0931ce17020a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.453813 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.453712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:38.461976 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.461942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:38.595851 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.595827 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm"] Jul 13 17:39:38.598062 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:39:38.598029 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b20046_7323_4547_b96c_0931ce17020a.slice/crio-e312e91b355decb70a6f2f943182190a53363d7afda1f9e0063ed4d4112451ad WatchSource:0}: Error finding container e312e91b355decb70a6f2f943182190a53363d7afda1f9e0063ed4d4112451ad: Status 404 returned error can't find the container with id e312e91b355decb70a6f2f943182190a53363d7afda1f9e0063ed4d4112451ad Jul 13 17:39:38.600602 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.600572 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:39:38.600668 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.600636 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:39:38.600708 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.600669 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:39:38.789675 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.789641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" event={"ID":"88b20046-7323-4547-b96c-0931ce17020a","Type":"ContainerStarted","Data":"a0dadd517cc683b05f20645b5c168b990e22b8bdfbf8014cb9d7a67cbe59c1d2"} Jul 13 17:39:38.790197 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.789706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" event={"ID":"88b20046-7323-4547-b96c-0931ce17020a","Type":"ContainerStarted","Data":"e312e91b355decb70a6f2f943182190a53363d7afda1f9e0063ed4d4112451ad"} Jul 13 17:39:38.813083 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:38.813014 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" podStartSLOduration=0.812994985 podStartE2EDuration="812.994985ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:39:38.811472219 +0000 UTC m=+433.818874473" watchObservedRunningTime="2026-07-13 17:39:38.812994985 +0000 UTC m=+433.820397228" Jul 13 17:39:39.454514 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:39.454341 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:39.459517 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:39.459478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:39.794635 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:39.794600 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:39.795684 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:39.795665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:39:41.288126 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.288088 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9b6f8"] Jul 13 17:39:41.290294 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.290268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:41.290870 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.290853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.293118 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.293096 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 13 17:39:41.301806 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.301786 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9b6f8"] Jul 13 17:39:41.390221 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.390184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9b6f8"] Jul 13 17:39:41.394147 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.394122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5g45\" (UniqueName: \"kubernetes.io/projected/ac24439a-b281-4059-89c3-b5e727be4787-kube-api-access-h5g45\") pod \"limitador-limitador-64c94fd64f-9b6f8\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.394306 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.394263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac24439a-b281-4059-89c3-b5e727be4787-config-file\") pod \"limitador-limitador-64c94fd64f-9b6f8\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.403398 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.403377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:41.467021 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.466991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:41.495296 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.495257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac24439a-b281-4059-89c3-b5e727be4787-config-file\") pod \"limitador-limitador-64c94fd64f-9b6f8\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.495516 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.495349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5g45\" (UniqueName: \"kubernetes.io/projected/ac24439a-b281-4059-89c3-b5e727be4787-kube-api-access-h5g45\") pod \"limitador-limitador-64c94fd64f-9b6f8\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.496065 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.496039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac24439a-b281-4059-89c3-b5e727be4787-config-file\") pod \"limitador-limitador-64c94fd64f-9b6f8\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.503162 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.503142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5g45\" (UniqueName: \"kubernetes.io/projected/ac24439a-b281-4059-89c3-b5e727be4787-kube-api-access-h5g45\") pod \"limitador-limitador-64c94fd64f-9b6f8\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.601753 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.601645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:41.735411 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.735346 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9b6f8"] Jul 13 17:39:41.737802 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:39:41.737774 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac24439a_b281_4059_89c3_b5e727be4787.slice/crio-34089cf22c12e4de3fbc571e63f2841a69de9191c1f68dac9e7d7797be11184c WatchSource:0}: Error finding container 34089cf22c12e4de3fbc571e63f2841a69de9191c1f68dac9e7d7797be11184c: Status 404 returned error can't find the container with id 34089cf22c12e4de3fbc571e63f2841a69de9191c1f68dac9e7d7797be11184c Jul 13 17:39:41.804098 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:41.804059 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" event={"ID":"ac24439a-b281-4059-89c3-b5e727be4787","Type":"ContainerStarted","Data":"34089cf22c12e4de3fbc571e63f2841a69de9191c1f68dac9e7d7797be11184c"} Jul 13 17:39:42.135969 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.135933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:42.195075 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.195036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:39:42.325095 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.325060 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-59r76"] Jul 13 17:39:42.328299 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.328274 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-59r76" Jul 13 17:39:42.330487 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.330453 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-89qfm\"" Jul 13 17:39:42.332765 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.332739 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-59r76"] Jul 13 17:39:42.405650 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.405580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbz28\" (UniqueName: \"kubernetes.io/projected/c2a72620-0f02-4882-918a-f0cd251c0258-kube-api-access-lbz28\") pod \"authorino-75d556bd77-59r76\" (UID: \"c2a72620-0f02-4882-918a-f0cd251c0258\") " pod="kuadrant-system/authorino-75d556bd77-59r76" Jul 13 17:39:42.506444 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.506406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbz28\" (UniqueName: \"kubernetes.io/projected/c2a72620-0f02-4882-918a-f0cd251c0258-kube-api-access-lbz28\") pod \"authorino-75d556bd77-59r76\" (UID: \"c2a72620-0f02-4882-918a-f0cd251c0258\") " pod="kuadrant-system/authorino-75d556bd77-59r76" Jul 13 17:39:42.514416 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.514380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbz28\" (UniqueName: \"kubernetes.io/projected/c2a72620-0f02-4882-918a-f0cd251c0258-kube-api-access-lbz28\") pod \"authorino-75d556bd77-59r76\" (UID: \"c2a72620-0f02-4882-918a-f0cd251c0258\") " pod="kuadrant-system/authorino-75d556bd77-59r76" Jul 13 17:39:42.638572 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.638531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-59r76" Jul 13 17:39:42.832297 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:42.832197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-59r76"] Jul 13 17:39:42.833693 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:39:42.833645 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2a72620_0f02_4882_918a_f0cd251c0258.slice/crio-806c5ade353ff3939f10a33781c6c34b7dd1f7768109a09168e9b503f58aef92 WatchSource:0}: Error finding container 806c5ade353ff3939f10a33781c6c34b7dd1f7768109a09168e9b503f58aef92: Status 404 returned error can't find the container with id 806c5ade353ff3939f10a33781c6c34b7dd1f7768109a09168e9b503f58aef92 Jul 13 17:39:43.822728 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:43.822670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-59r76" event={"ID":"c2a72620-0f02-4882-918a-f0cd251c0258","Type":"ContainerStarted","Data":"806c5ade353ff3939f10a33781c6c34b7dd1f7768109a09168e9b503f58aef92"} Jul 13 17:39:45.835825 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:45.835784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" event={"ID":"ac24439a-b281-4059-89c3-b5e727be4787","Type":"ContainerStarted","Data":"3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7"} Jul 13 17:39:45.836260 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:45.835851 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:45.853272 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:45.853206 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" podStartSLOduration=1.850807864 podStartE2EDuration="4.85318967s" totalImagesPullingTime="3.002381806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:39:45.850840853 +0000 UTC m=+440.858243099" watchObservedRunningTime="2026-07-13 17:39:45.85318967 +0000 UTC m=+440.860591912" Jul 13 17:39:47.846929 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:47.846891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-59r76" event={"ID":"c2a72620-0f02-4882-918a-f0cd251c0258","Type":"ContainerStarted","Data":"00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6"} Jul 13 17:39:47.864195 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:47.864145 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-59r76" podStartSLOduration=1.621196121 podStartE2EDuration="5.864130571s" totalImagesPullingTime="4.24293445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:39:47.86169822 +0000 UTC m=+442.869100463" watchObservedRunningTime="2026-07-13 17:39:47.864130571 +0000 UTC m=+442.871532813" Jul 13 17:39:56.841726 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:56.841694 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:58.222007 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.221972 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9b6f8"] Jul 13 17:39:58.222457 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.222129 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" podUID="ac24439a-b281-4059-89c3-b5e727be4787" containerName="limitador" containerID="cri-o://3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7" gracePeriod=30 Jul 13 17:39:58.769902 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.769881 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:58.858416 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.858387 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac24439a-b281-4059-89c3-b5e727be4787-config-file\") pod \"ac24439a-b281-4059-89c3-b5e727be4787\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " Jul 13 17:39:58.858555 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.858470 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5g45\" (UniqueName: \"kubernetes.io/projected/ac24439a-b281-4059-89c3-b5e727be4787-kube-api-access-h5g45\") pod \"ac24439a-b281-4059-89c3-b5e727be4787\" (UID: \"ac24439a-b281-4059-89c3-b5e727be4787\") " Jul 13 17:39:58.858779 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.858752 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac24439a-b281-4059-89c3-b5e727be4787-config-file" (OuterVolumeSpecName: "config-file") pod "ac24439a-b281-4059-89c3-b5e727be4787" (UID: "ac24439a-b281-4059-89c3-b5e727be4787"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:39:58.860421 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.860399 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac24439a-b281-4059-89c3-b5e727be4787-kube-api-access-h5g45" (OuterVolumeSpecName: "kube-api-access-h5g45") pod "ac24439a-b281-4059-89c3-b5e727be4787" (UID: "ac24439a-b281-4059-89c3-b5e727be4787"). InnerVolumeSpecName "kube-api-access-h5g45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:39:58.900325 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.900292 2564 generic.go:356] "Generic (PLEG): container finished" podID="ac24439a-b281-4059-89c3-b5e727be4787" containerID="3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7" exitCode=0 Jul 13 17:39:58.900422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.900331 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" Jul 13 17:39:58.900422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.900355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" event={"ID":"ac24439a-b281-4059-89c3-b5e727be4787","Type":"ContainerDied","Data":"3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7"} Jul 13 17:39:58.900422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.900395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9b6f8" event={"ID":"ac24439a-b281-4059-89c3-b5e727be4787","Type":"ContainerDied","Data":"34089cf22c12e4de3fbc571e63f2841a69de9191c1f68dac9e7d7797be11184c"} Jul 13 17:39:58.900422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.900415 2564 scope.go:117] "RemoveContainer" containerID="3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7" Jul 13 17:39:58.909211 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.909193 2564 scope.go:117] "RemoveContainer" containerID="3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7" Jul 13 17:39:58.909483 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:39:58.909463 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7\": container with ID starting with 3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7 not found: ID does not exist" containerID="3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7" Jul 13 17:39:58.909538 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.909492 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7"} err="failed to get container status \"3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7\": rpc error: code = NotFound desc = could not find container \"3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7\": container with ID starting with 3eda6e96b40ad19ff6b5e6a7dcd197391b888beabf8cf2c1a1f8ac149dbb2cf7 not found: ID does not exist" Jul 13 17:39:58.921239 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.921210 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9b6f8"] Jul 13 17:39:58.928678 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.928659 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9b6f8"] Jul 13 17:39:58.959480 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.959462 2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac24439a-b281-4059-89c3-b5e727be4787-config-file\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:39:58.959480 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:58.959479 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5g45\" (UniqueName: \"kubernetes.io/projected/ac24439a-b281-4059-89c3-b5e727be4787-kube-api-access-h5g45\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:39:59.638417 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:39:59.638384 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac24439a-b281-4059-89c3-b5e727be4787" path="/var/lib/kubelet/pods/ac24439a-b281-4059-89c3-b5e727be4787/volumes" Jul 13 17:40:01.311016 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.310981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:01.409079 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.409047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:01.775507 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.775427 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn"] Jul 13 17:40:01.776016 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.775997 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac24439a-b281-4059-89c3-b5e727be4787" containerName="limitador" Jul 13 17:40:01.776063 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.776022 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac24439a-b281-4059-89c3-b5e727be4787" containerName="limitador" Jul 13 17:40:01.776153 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.776142 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac24439a-b281-4059-89c3-b5e727be4787" containerName="limitador" Jul 13 17:40:01.780289 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.780269 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:01.783603 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.782938 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 17:40:01.783603 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.783321 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 13 17:40:01.783603 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.783532 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bccfg\"" Jul 13 17:40:01.783854 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.783816 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 17:40:01.786479 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.786458 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn"] Jul 13 17:40:01.804774 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.804746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:01.881470 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.881442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/41f09881-4c99-4a5f-a14c-ac0d7c5e43c1-data\") pod \"postgres-69fb4594cf-lbrrn\" (UID: \"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:01.881587 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.881483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xjn\" (UniqueName: \"kubernetes.io/projected/41f09881-4c99-4a5f-a14c-ac0d7c5e43c1-kube-api-access-c5xjn\") pod \"postgres-69fb4594cf-lbrrn\" (UID: \"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:01.982138 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.982115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/41f09881-4c99-4a5f-a14c-ac0d7c5e43c1-data\") pod \"postgres-69fb4594cf-lbrrn\" (UID: \"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:01.982273 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.982154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xjn\" (UniqueName: \"kubernetes.io/projected/41f09881-4c99-4a5f-a14c-ac0d7c5e43c1-kube-api-access-c5xjn\") pod \"postgres-69fb4594cf-lbrrn\" (UID: \"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:01.982537 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.982512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/41f09881-4c99-4a5f-a14c-ac0d7c5e43c1-data\") pod \"postgres-69fb4594cf-lbrrn\" (UID: \"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:01.990491 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:01.990468 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xjn\" (UniqueName: \"kubernetes.io/projected/41f09881-4c99-4a5f-a14c-ac0d7c5e43c1-kube-api-access-c5xjn\") pod \"postgres-69fb4594cf-lbrrn\" (UID: \"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:02.092958 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:02.092923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:02.219484 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:02.219456 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn"] Jul 13 17:40:02.222606 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:02.222578 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f09881_4c99_4a5f_a14c_ac0d7c5e43c1.slice/crio-d62f55f5c8b3612c8f491d28b3ea7b818fdef2d3b4ec408c31bb7131b2eec4b7 WatchSource:0}: Error finding container d62f55f5c8b3612c8f491d28b3ea7b818fdef2d3b4ec408c31bb7131b2eec4b7: Status 404 returned error can't find the container with id d62f55f5c8b3612c8f491d28b3ea7b818fdef2d3b4ec408c31bb7131b2eec4b7 Jul 13 17:40:02.922506 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:02.922468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" event={"ID":"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1","Type":"ContainerStarted","Data":"d62f55f5c8b3612c8f491d28b3ea7b818fdef2d3b4ec408c31bb7131b2eec4b7"} Jul 13 17:40:08.963728 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:08.963684 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" event={"ID":"41f09881-4c99-4a5f-a14c-ac0d7c5e43c1","Type":"ContainerStarted","Data":"148833642e83a568a18e14aa3e55c057397f44ef421cd15c42c000346e7ca32d"} Jul 13 17:40:08.964177 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:08.963797 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:08.979443 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:08.979388 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" podStartSLOduration=1.340184939 podStartE2EDuration="7.979370729s" totalImagesPullingTime="6.63918579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:40:08.978461084 +0000 UTC m=+463.985863327" watchObservedRunningTime="2026-07-13 17:40:08.979370729 +0000 UTC m=+463.986772971" Jul 13 17:40:14.998971 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:14.998940 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lbrrn" Jul 13 17:40:15.348697 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.348659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:15.456453 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.456414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:15.623217 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.623147 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ww6d5"] Jul 13 17:40:15.628518 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.628486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" Jul 13 17:40:15.638993 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.638966 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ww6d5"] Jul 13 17:40:15.710264 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.710215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shpj\" (UniqueName: \"kubernetes.io/projected/d7a6d793-028d-4c76-a834-3c9444e78e9d-kube-api-access-4shpj\") pod \"authorino-5d4b67c446-ww6d5\" (UID: \"d7a6d793-028d-4c76-a834-3c9444e78e9d\") " pod="kuadrant-system/authorino-5d4b67c446-ww6d5" Jul 13 17:40:15.811486 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.811455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shpj\" (UniqueName: \"kubernetes.io/projected/d7a6d793-028d-4c76-a834-3c9444e78e9d-kube-api-access-4shpj\") pod \"authorino-5d4b67c446-ww6d5\" (UID: \"d7a6d793-028d-4c76-a834-3c9444e78e9d\") " pod="kuadrant-system/authorino-5d4b67c446-ww6d5" Jul 13 17:40:15.821407 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.821379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shpj\" (UniqueName: \"kubernetes.io/projected/d7a6d793-028d-4c76-a834-3c9444e78e9d-kube-api-access-4shpj\") pod \"authorino-5d4b67c446-ww6d5\" (UID: \"d7a6d793-028d-4c76-a834-3c9444e78e9d\") " pod="kuadrant-system/authorino-5d4b67c446-ww6d5" Jul 13 17:40:15.878760 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.878676 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ww6d5"] Jul 13 17:40:15.878999 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.878981 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" Jul 13 17:40:15.900139 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.900115 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-756c5bdd7d-f768v"] Jul 13 17:40:15.904772 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.904752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-756c5bdd7d-f768v" Jul 13 17:40:15.910578 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:15.910555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-756c5bdd7d-f768v"] Jul 13 17:40:16.010640 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.010614 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ww6d5"] Jul 13 17:40:16.013219 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.013145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mzm\" (UniqueName: \"kubernetes.io/projected/c85ccd97-dde4-4c79-a958-7e015f5e978d-kube-api-access-g6mzm\") pod \"authorino-756c5bdd7d-f768v\" (UID: \"c85ccd97-dde4-4c79-a958-7e015f5e978d\") " pod="kuadrant-system/authorino-756c5bdd7d-f768v" Jul 13 17:40:16.013600 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:16.013578 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7a6d793_028d_4c76_a834_3c9444e78e9d.slice/crio-4fe5adb1e0ad3c5212643284f3566a246a4a5a295eef9158b099bb6ba64a2f8d WatchSource:0}: Error finding container 4fe5adb1e0ad3c5212643284f3566a246a4a5a295eef9158b099bb6ba64a2f8d: Status 404 returned error can't find the container with id 4fe5adb1e0ad3c5212643284f3566a246a4a5a295eef9158b099bb6ba64a2f8d Jul 13 17:40:16.114537 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.114498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mzm\" (UniqueName: \"kubernetes.io/projected/c85ccd97-dde4-4c79-a958-7e015f5e978d-kube-api-access-g6mzm\") pod \"authorino-756c5bdd7d-f768v\" (UID: \"c85ccd97-dde4-4c79-a958-7e015f5e978d\") " pod="kuadrant-system/authorino-756c5bdd7d-f768v" Jul 13 17:40:16.116141 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.116118 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-756c5bdd7d-f768v"] Jul 13 17:40:16.116423 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:40:16.116403 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g6mzm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-756c5bdd7d-f768v" podUID="c85ccd97-dde4-4c79-a958-7e015f5e978d" Jul 13 17:40:16.127538 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.127511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mzm\" (UniqueName: \"kubernetes.io/projected/c85ccd97-dde4-4c79-a958-7e015f5e978d-kube-api-access-g6mzm\") pod \"authorino-756c5bdd7d-f768v\" (UID: \"c85ccd97-dde4-4c79-a958-7e015f5e978d\") " pod="kuadrant-system/authorino-756c5bdd7d-f768v" Jul 13 17:40:16.145142 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.145062 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58c446fc8b-sj5gd"] Jul 13 17:40:16.149089 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.149066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.151440 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.151420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 13 17:40:16.157796 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.157768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58c446fc8b-sj5gd"] Jul 13 17:40:16.215112 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.215072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d85eb78-5dea-4273-8040-ec8f1820e18c-tls-cert\") pod \"authorino-58c446fc8b-sj5gd\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.215314 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.215119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9zck\" (UniqueName: \"kubernetes.io/projected/9d85eb78-5dea-4273-8040-ec8f1820e18c-kube-api-access-z9zck\") pod \"authorino-58c446fc8b-sj5gd\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.316634 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.316591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d85eb78-5dea-4273-8040-ec8f1820e18c-tls-cert\") pod \"authorino-58c446fc8b-sj5gd\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.316634 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.316632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9zck\" (UniqueName: \"kubernetes.io/projected/9d85eb78-5dea-4273-8040-ec8f1820e18c-kube-api-access-z9zck\") pod \"authorino-58c446fc8b-sj5gd\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.319022 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.318998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d85eb78-5dea-4273-8040-ec8f1820e18c-tls-cert\") pod \"authorino-58c446fc8b-sj5gd\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.325522 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.325502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9zck\" (UniqueName: \"kubernetes.io/projected/9d85eb78-5dea-4273-8040-ec8f1820e18c-kube-api-access-z9zck\") pod \"authorino-58c446fc8b-sj5gd\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.459877 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.459846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:40:16.600440 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:16.600408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58c446fc8b-sj5gd"] Jul 13 17:40:16.603408 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:16.603372 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d85eb78_5dea_4273_8040_ec8f1820e18c.slice/crio-35cde4bc20e08fe92f1b563e33e1e77ed51ba5c3356c055e6e21e1149cbdb161 WatchSource:0}: Error finding container 35cde4bc20e08fe92f1b563e33e1e77ed51ba5c3356c055e6e21e1149cbdb161: Status 404 returned error can't find the container with id 35cde4bc20e08fe92f1b563e33e1e77ed51ba5c3356c055e6e21e1149cbdb161 Jul 13 17:40:17.004309 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.004271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" event={"ID":"9d85eb78-5dea-4273-8040-ec8f1820e18c","Type":"ContainerStarted","Data":"35cde4bc20e08fe92f1b563e33e1e77ed51ba5c3356c055e6e21e1149cbdb161"} Jul 13 17:40:17.005762 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.005727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" event={"ID":"d7a6d793-028d-4c76-a834-3c9444e78e9d","Type":"ContainerStarted","Data":"9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe"} Jul 13 17:40:17.005879 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.005767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-756c5bdd7d-f768v" Jul 13 17:40:17.005879 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.005767 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" podUID="d7a6d793-028d-4c76-a834-3c9444e78e9d" containerName="authorino" containerID="cri-o://9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe" gracePeriod=30 Jul 13 17:40:17.005879 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.005771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" event={"ID":"d7a6d793-028d-4c76-a834-3c9444e78e9d","Type":"ContainerStarted","Data":"4fe5adb1e0ad3c5212643284f3566a246a4a5a295eef9158b099bb6ba64a2f8d"} Jul 13 17:40:17.014504 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.014486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-756c5bdd7d-f768v" Jul 13 17:40:17.022523 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.022483 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" podStartSLOduration=1.658792042 podStartE2EDuration="2.022470005s" totalImagesPullingTime="363.677963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:40:17.021531463 +0000 UTC m=+472.028933703" watchObservedRunningTime="2026-07-13 17:40:17.022470005 +0000 UTC m=+472.029872247" Jul 13 17:40:17.124856 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.124829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6mzm\" (UniqueName: \"kubernetes.io/projected/c85ccd97-dde4-4c79-a958-7e015f5e978d-kube-api-access-g6mzm\") pod \"c85ccd97-dde4-4c79-a958-7e015f5e978d\" (UID: \"c85ccd97-dde4-4c79-a958-7e015f5e978d\") " Jul 13 17:40:17.126750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.126721 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85ccd97-dde4-4c79-a958-7e015f5e978d-kube-api-access-g6mzm" (OuterVolumeSpecName: "kube-api-access-g6mzm") pod "c85ccd97-dde4-4c79-a958-7e015f5e978d" (UID: "c85ccd97-dde4-4c79-a958-7e015f5e978d"). InnerVolumeSpecName "kube-api-access-g6mzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:40:17.227218 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.227178 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mzm\" (UniqueName: \"kubernetes.io/projected/c85ccd97-dde4-4c79-a958-7e015f5e978d-kube-api-access-g6mzm\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:40:17.267326 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.267272 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" Jul 13 17:40:17.327481 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.327458 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4shpj\" (UniqueName: \"kubernetes.io/projected/d7a6d793-028d-4c76-a834-3c9444e78e9d-kube-api-access-4shpj\") pod \"d7a6d793-028d-4c76-a834-3c9444e78e9d\" (UID: \"d7a6d793-028d-4c76-a834-3c9444e78e9d\") " Jul 13 17:40:17.329220 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.329197 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a6d793-028d-4c76-a834-3c9444e78e9d-kube-api-access-4shpj" (OuterVolumeSpecName: "kube-api-access-4shpj") pod "d7a6d793-028d-4c76-a834-3c9444e78e9d" (UID: "d7a6d793-028d-4c76-a834-3c9444e78e9d"). InnerVolumeSpecName "kube-api-access-4shpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:40:17.428898 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:17.428863 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4shpj\" (UniqueName: \"kubernetes.io/projected/d7a6d793-028d-4c76-a834-3c9444e78e9d-kube-api-access-4shpj\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:40:18.013538 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.013442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" event={"ID":"9d85eb78-5dea-4273-8040-ec8f1820e18c","Type":"ContainerStarted","Data":"d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391"} Jul 13 17:40:18.014615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.014587 2564 generic.go:356] "Generic (PLEG): container finished" podID="d7a6d793-028d-4c76-a834-3c9444e78e9d" containerID="9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe" exitCode=0 Jul 13 17:40:18.014994 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.014616 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" Jul 13 17:40:18.014994 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.014653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" event={"ID":"d7a6d793-028d-4c76-a834-3c9444e78e9d","Type":"ContainerDied","Data":"9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe"} Jul 13 17:40:18.014994 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.014678 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ww6d5" event={"ID":"d7a6d793-028d-4c76-a834-3c9444e78e9d","Type":"ContainerDied","Data":"4fe5adb1e0ad3c5212643284f3566a246a4a5a295eef9158b099bb6ba64a2f8d"} Jul 13 17:40:18.014994 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.014695 2564 scope.go:117] "RemoveContainer" containerID="9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe" Jul 13 17:40:18.014994 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.014804 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-756c5bdd7d-f768v" Jul 13 17:40:18.023274 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.023257 2564 scope.go:117] "RemoveContainer" containerID="9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe" Jul 13 17:40:18.023530 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:40:18.023504 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe\": container with ID starting with 9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe not found: ID does not exist" containerID="9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe" Jul 13 17:40:18.023574 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.023532 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe"} err="failed to get container status \"9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe\": rpc error: code = NotFound desc = could not find container \"9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe\": container with ID starting with 9b14db71c4d36d810ade156358e934ebac239bee7f85915cd77f1ec5a5049ffe not found: ID does not exist" Jul 13 17:40:18.028590 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.028532 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" podStartSLOduration=1.337412736 podStartE2EDuration="2.028519376s" totalImagesPullingTime="691.10664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:40:18.026827034 +0000 UTC m=+473.034229296" watchObservedRunningTime="2026-07-13 17:40:18.028519376 +0000 UTC m=+473.035921619" Jul 13 17:40:18.054404 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.054376 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-756c5bdd7d-f768v"] Jul 13 17:40:18.056947 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.056918 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-59r76"] Jul 13 17:40:18.057133 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.057110 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-59r76" podUID="c2a72620-0f02-4882-918a-f0cd251c0258" containerName="authorino" containerID="cri-o://00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6" gracePeriod=30 Jul 13 17:40:18.058724 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.058699 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-756c5bdd7d-f768v"] Jul 13 17:40:18.074969 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.074943 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ww6d5"] Jul 13 17:40:18.080917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.080892 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ww6d5"] Jul 13 17:40:18.337644 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.337614 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-59r76" Jul 13 17:40:18.440652 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.440622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbz28\" (UniqueName: \"kubernetes.io/projected/c2a72620-0f02-4882-918a-f0cd251c0258-kube-api-access-lbz28\") pod \"c2a72620-0f02-4882-918a-f0cd251c0258\" (UID: \"c2a72620-0f02-4882-918a-f0cd251c0258\") " Jul 13 17:40:18.442669 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.442633 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a72620-0f02-4882-918a-f0cd251c0258-kube-api-access-lbz28" (OuterVolumeSpecName: "kube-api-access-lbz28") pod "c2a72620-0f02-4882-918a-f0cd251c0258" (UID: "c2a72620-0f02-4882-918a-f0cd251c0258"). InnerVolumeSpecName "kube-api-access-lbz28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:40:18.542025 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:18.541992 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbz28\" (UniqueName: \"kubernetes.io/projected/c2a72620-0f02-4882-918a-f0cd251c0258-kube-api-access-lbz28\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:40:19.021971 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.021930 2564 generic.go:356] "Generic (PLEG): container finished" podID="c2a72620-0f02-4882-918a-f0cd251c0258" containerID="00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6" exitCode=0 Jul 13 17:40:19.021971 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.021973 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-59r76" Jul 13 17:40:19.022483 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.022010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-59r76" event={"ID":"c2a72620-0f02-4882-918a-f0cd251c0258","Type":"ContainerDied","Data":"00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6"} Jul 13 17:40:19.022483 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.022049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-59r76" event={"ID":"c2a72620-0f02-4882-918a-f0cd251c0258","Type":"ContainerDied","Data":"806c5ade353ff3939f10a33781c6c34b7dd1f7768109a09168e9b503f58aef92"} Jul 13 17:40:19.022483 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.022065 2564 scope.go:117] "RemoveContainer" containerID="00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6" Jul 13 17:40:19.030498 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.030482 2564 scope.go:117] "RemoveContainer" containerID="00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6" Jul 13 17:40:19.030739 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:40:19.030719 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6\": container with ID starting with 00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6 not found: ID does not exist" containerID="00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6" Jul 13 17:40:19.030782 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.030747 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6"} err="failed to get container status \"00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6\": rpc error: code = NotFound desc = could not find container \"00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6\": container with ID starting with 00749d969f07bcb37f3acd4bd6371be6141b1ab851ecc17ea7eb6d895e57e8f6 not found: ID does not exist" Jul 13 17:40:19.045000 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.044977 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-59r76"] Jul 13 17:40:19.049069 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.049042 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-59r76"] Jul 13 17:40:19.059504 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.059473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:19.637993 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.637958 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a72620-0f02-4882-918a-f0cd251c0258" path="/var/lib/kubelet/pods/c2a72620-0f02-4882-918a-f0cd251c0258/volumes" Jul 13 17:40:19.638322 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.638308 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85ccd97-dde4-4c79-a958-7e015f5e978d" path="/var/lib/kubelet/pods/c85ccd97-dde4-4c79-a958-7e015f5e978d/volumes" Jul 13 17:40:19.638516 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:19.638504 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a6d793-028d-4c76-a834-3c9444e78e9d" path="/var/lib/kubelet/pods/d7a6d793-028d-4c76-a834-3c9444e78e9d/volumes" Jul 13 17:40:23.126317 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.126281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:23.141431 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.141405 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f998c59c-d7sd4"] Jul 13 17:40:23.141801 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.141788 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a72620-0f02-4882-918a-f0cd251c0258" containerName="authorino" Jul 13 17:40:23.141855 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.141803 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a72620-0f02-4882-918a-f0cd251c0258" containerName="authorino" Jul 13 17:40:23.141855 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.141821 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a6d793-028d-4c76-a834-3c9444e78e9d" containerName="authorino" Jul 13 17:40:23.141855 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.141827 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a6d793-028d-4c76-a834-3c9444e78e9d" containerName="authorino" Jul 13 17:40:23.141941 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.141892 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a6d793-028d-4c76-a834-3c9444e78e9d" containerName="authorino" Jul 13 17:40:23.141941 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.141903 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a72620-0f02-4882-918a-f0cd251c0258" containerName="authorino" Jul 13 17:40:23.158753 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.158725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f998c59c-d7sd4"] Jul 13 17:40:23.158897 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.158824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.161146 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.161124 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 17:40:23.161309 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.161290 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 17:40:23.161366 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.161316 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zmk4b\"" Jul 13 17:40:23.227161 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.227127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:23.286443 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.286404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00b54b56-422a-4108-a1c5-34228213a625-webhook-cert\") pod \"maas-controller-5f998c59c-d7sd4\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.286607 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.286537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5wx\" (UniqueName: \"kubernetes.io/projected/00b54b56-422a-4108-a1c5-34228213a625-kube-api-access-mf5wx\") pod \"maas-controller-5f998c59c-d7sd4\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.314617 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.314588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:23.315454 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.315433 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-7tb2b"] Jul 13 17:40:23.317967 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.317950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.326104 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.326079 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-7tb2b"] Jul 13 17:40:23.370490 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.370461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:23.387885 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.387809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-webhook-cert\") pod \"maas-controller-6cbcd4b6b8-7tb2b\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.387885 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.387846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5wx\" (UniqueName: \"kubernetes.io/projected/00b54b56-422a-4108-a1c5-34228213a625-kube-api-access-mf5wx\") pod \"maas-controller-5f998c59c-d7sd4\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.387885 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.387865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rc2\" (UniqueName: \"kubernetes.io/projected/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-kube-api-access-t7rc2\") pod \"maas-controller-6cbcd4b6b8-7tb2b\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.388119 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.388006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00b54b56-422a-4108-a1c5-34228213a625-webhook-cert\") pod \"maas-controller-5f998c59c-d7sd4\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.390561 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.390531 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00b54b56-422a-4108-a1c5-34228213a625-webhook-cert\") pod \"maas-controller-5f998c59c-d7sd4\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.401798 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.401773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5wx\" (UniqueName: \"kubernetes.io/projected/00b54b56-422a-4108-a1c5-34228213a625-kube-api-access-mf5wx\") pod \"maas-controller-5f998c59c-d7sd4\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.472492 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.472462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:23.488934 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.488901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rc2\" (UniqueName: \"kubernetes.io/projected/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-kube-api-access-t7rc2\") pod \"maas-controller-6cbcd4b6b8-7tb2b\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.489128 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.489107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-webhook-cert\") pod \"maas-controller-6cbcd4b6b8-7tb2b\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.491391 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.491369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-webhook-cert\") pod \"maas-controller-6cbcd4b6b8-7tb2b\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.497921 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.497897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rc2\" (UniqueName: \"kubernetes.io/projected/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-kube-api-access-t7rc2\") pod \"maas-controller-6cbcd4b6b8-7tb2b\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.609540 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.609505 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f998c59c-d7sd4"] Jul 13 17:40:23.612793 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:23.612762 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b54b56_422a_4108_a1c5_34228213a625.slice/crio-5c7b197803ad7992b92b0dc7253ea9b0cf3ef1b0a6203201a15c2e52f04c03d1 WatchSource:0}: Error finding container 5c7b197803ad7992b92b0dc7253ea9b0cf3ef1b0a6203201a15c2e52f04c03d1: Status 404 returned error can't find the container with id 5c7b197803ad7992b92b0dc7253ea9b0cf3ef1b0a6203201a15c2e52f04c03d1 Jul 13 17:40:23.628060 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.628040 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:23.754944 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:23.754903 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-7tb2b"] Jul 13 17:40:23.756896 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:23.756864 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed2598e_0c9d_4872_bcd8_96926a40e1e2.slice/crio-73b421a144cf833984e1c1cce00333980cdde96b57c94692941b20b4d55b5990 WatchSource:0}: Error finding container 73b421a144cf833984e1c1cce00333980cdde96b57c94692941b20b4d55b5990: Status 404 returned error can't find the container with id 73b421a144cf833984e1c1cce00333980cdde96b57c94692941b20b4d55b5990 Jul 13 17:40:24.050880 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:24.050845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" event={"ID":"0ed2598e-0c9d-4872-bcd8-96926a40e1e2","Type":"ContainerStarted","Data":"73b421a144cf833984e1c1cce00333980cdde96b57c94692941b20b4d55b5990"} Jul 13 17:40:24.051988 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:24.051966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f998c59c-d7sd4" event={"ID":"00b54b56-422a-4108-a1c5-34228213a625","Type":"ContainerStarted","Data":"5c7b197803ad7992b92b0dc7253ea9b0cf3ef1b0a6203201a15c2e52f04c03d1"} Jul 13 17:40:25.911026 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:25.910998 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 17:40:26.072983 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.072926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" event={"ID":"0ed2598e-0c9d-4872-bcd8-96926a40e1e2","Type":"ContainerStarted","Data":"26a463200ffdc72bda5f6b4615790b0ee1cf0feaba8c47cfacbdacb023cc0394"} Jul 13 17:40:26.073159 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.073001 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:26.074418 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.074393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f998c59c-d7sd4" event={"ID":"00b54b56-422a-4108-a1c5-34228213a625","Type":"ContainerStarted","Data":"24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d"} Jul 13 17:40:26.074532 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.074477 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:26.090125 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.089839 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" podStartSLOduration=0.930716621 podStartE2EDuration="3.089820335s" totalImagesPullingTime="2.159103714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:40:26.089003736 +0000 UTC m=+481.096405979" watchObservedRunningTime="2026-07-13 17:40:26.089820335 +0000 UTC m=+481.097222579" Jul 13 17:40:26.109116 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.109060 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f998c59c-d7sd4" podStartSLOduration=0.814110005 podStartE2EDuration="3.109043106s" totalImagesPullingTime="2.294933101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:40:26.106885961 +0000 UTC m=+481.114288255" watchObservedRunningTime="2026-07-13 17:40:26.109043106 +0000 UTC m=+481.116445354" Jul 13 17:40:26.140858 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.140745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:26.241593 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:26.241559 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:37.081462 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.081427 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:40:37.081871 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.081508 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:37.176911 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.176878 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f998c59c-d7sd4"] Jul 13 17:40:37.177131 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.177104 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f998c59c-d7sd4" podUID="00b54b56-422a-4108-a1c5-34228213a625" containerName="manager" containerID="cri-o://24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d" gracePeriod=30 Jul 13 17:40:37.425663 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.425638 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:37.527519 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.527485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf5wx\" (UniqueName: \"kubernetes.io/projected/00b54b56-422a-4108-a1c5-34228213a625-kube-api-access-mf5wx\") pod \"00b54b56-422a-4108-a1c5-34228213a625\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " Jul 13 17:40:37.527673 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.527535 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00b54b56-422a-4108-a1c5-34228213a625-webhook-cert\") pod \"00b54b56-422a-4108-a1c5-34228213a625\" (UID: \"00b54b56-422a-4108-a1c5-34228213a625\") " Jul 13 17:40:37.529705 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.529676 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b54b56-422a-4108-a1c5-34228213a625-kube-api-access-mf5wx" (OuterVolumeSpecName: "kube-api-access-mf5wx") pod "00b54b56-422a-4108-a1c5-34228213a625" (UID: "00b54b56-422a-4108-a1c5-34228213a625"). InnerVolumeSpecName "kube-api-access-mf5wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:40:37.529705 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.529692 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b54b56-422a-4108-a1c5-34228213a625-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "00b54b56-422a-4108-a1c5-34228213a625" (UID: "00b54b56-422a-4108-a1c5-34228213a625"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:40:37.628441 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.628366 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf5wx\" (UniqueName: \"kubernetes.io/projected/00b54b56-422a-4108-a1c5-34228213a625-kube-api-access-mf5wx\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:40:37.628441 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:37.628394 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00b54b56-422a-4108-a1c5-34228213a625-webhook-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:40:38.134298 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.134263 2564 generic.go:356] "Generic (PLEG): container finished" podID="00b54b56-422a-4108-a1c5-34228213a625" containerID="24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d" exitCode=0 Jul 13 17:40:38.134769 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.134320 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f998c59c-d7sd4" Jul 13 17:40:38.134769 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.134354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f998c59c-d7sd4" event={"ID":"00b54b56-422a-4108-a1c5-34228213a625","Type":"ContainerDied","Data":"24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d"} Jul 13 17:40:38.134769 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.134420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f998c59c-d7sd4" event={"ID":"00b54b56-422a-4108-a1c5-34228213a625","Type":"ContainerDied","Data":"5c7b197803ad7992b92b0dc7253ea9b0cf3ef1b0a6203201a15c2e52f04c03d1"} Jul 13 17:40:38.134769 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.134456 2564 scope.go:117] "RemoveContainer" containerID="24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d" Jul 13 17:40:38.143982 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.143961 2564 scope.go:117] "RemoveContainer" containerID="24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d" Jul 13 17:40:38.144257 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:40:38.144223 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d\": container with ID starting with 24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d not found: ID does not exist" containerID="24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d" Jul 13 17:40:38.144330 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.144265 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d"} err="failed to get container status \"24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d\": rpc error: code = NotFound desc = could not find container \"24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d\": container with ID starting with 24d6aa4bdf0eb027971fd6c792179534d9c108eb37230d1c97b39e24b331fc2d not found: ID does not exist" Jul 13 17:40:38.151644 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.151620 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f998c59c-d7sd4"] Jul 13 17:40:38.158348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:38.158326 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f998c59c-d7sd4"] Jul 13 17:40:39.637629 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:39.637592 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b54b56-422a-4108-a1c5-34228213a625" path="/var/lib/kubelet/pods/00b54b56-422a-4108-a1c5-34228213a625/volumes" Jul 13 17:40:56.129305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.129257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:56.225987 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.225956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:56.595108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.595076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:56.711901 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.709088 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr"] Jul 13 17:40:56.711901 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.709958 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b54b56-422a-4108-a1c5-34228213a625" containerName="manager" Jul 13 17:40:56.711901 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.709979 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b54b56-422a-4108-a1c5-34228213a625" containerName="manager" Jul 13 17:40:56.711901 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.710164 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b54b56-422a-4108-a1c5-34228213a625" containerName="manager" Jul 13 17:40:56.714310 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.714289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.718383 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.718363 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pjdj4\"" Jul 13 17:40:56.718505 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.718394 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 13 17:40:56.722750 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.722731 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr"] Jul 13 17:40:56.809185 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.809156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdhb\" (UniqueName: \"kubernetes.io/projected/4dafd306-8edf-40b5-8bf1-421805ba4210-kube-api-access-5rdhb\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.809309 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.809190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dafd306-8edf-40b5-8bf1-421805ba4210-maas-api-tls\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.809309 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.809218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dafd306-8edf-40b5-8bf1-421805ba4210-openshift-service-ca\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.909731 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.909667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdhb\" (UniqueName: \"kubernetes.io/projected/4dafd306-8edf-40b5-8bf1-421805ba4210-kube-api-access-5rdhb\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.909731 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.909700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dafd306-8edf-40b5-8bf1-421805ba4210-maas-api-tls\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.909898 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.909735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dafd306-8edf-40b5-8bf1-421805ba4210-openshift-service-ca\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.909898 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:40:56.909814 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 13 17:40:56.910001 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:40:56.909922 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dafd306-8edf-40b5-8bf1-421805ba4210-maas-api-tls podName:4dafd306-8edf-40b5-8bf1-421805ba4210 nodeName:}" failed. No retries permitted until 2026-07-13 17:40:57.409902778 +0000 UTC m=+512.417304999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dafd306-8edf-40b5-8bf1-421805ba4210-maas-api-tls") pod "maas-api-78fc4dddbd-b5jqr" (UID: "4dafd306-8edf-40b5-8bf1-421805ba4210") : secret "maas-api-serving-cert" not found Jul 13 17:40:56.910443 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.910422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dafd306-8edf-40b5-8bf1-421805ba4210-openshift-service-ca\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:56.919875 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:56.919854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdhb\" (UniqueName: \"kubernetes.io/projected/4dafd306-8edf-40b5-8bf1-421805ba4210-kube-api-access-5rdhb\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:57.064055 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.064023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:57.150085 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.150051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:40:57.150932 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.150914 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-9gggq"] Jul 13 17:40:57.154615 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.154593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.156702 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.156680 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sxjhp\"" Jul 13 17:40:57.156702 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.156698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 13 17:40:57.164481 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.164461 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-9gggq"] Jul 13 17:40:57.205770 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.205738 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-pdp4t"] Jul 13 17:40:57.209280 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.209258 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.223451 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.223427 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-pdp4t"] Jul 13 17:40:57.312633 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.312597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cs4\" (UniqueName: \"kubernetes.io/projected/ac4d8a62-5774-4a52-95c1-8ff6643c7598-kube-api-access-k8cs4\") pod \"payload-pre-processing-74584bb9f8-9gggq\" (UID: \"ac4d8a62-5774-4a52-95c1-8ff6643c7598\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.312800 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.312650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8bec71-e814-400f-b48f-9932bfc4776d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-pdp4t\" (UID: \"ac8bec71-e814-400f-b48f-9932bfc4776d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.312866 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.312799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4d8a62-5774-4a52-95c1-8ff6643c7598-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-9gggq\" (UID: \"ac4d8a62-5774-4a52-95c1-8ff6643c7598\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.312866 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.312857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbg2\" (UniqueName: \"kubernetes.io/projected/ac8bec71-e814-400f-b48f-9932bfc4776d-kube-api-access-kgbg2\") pod \"payload-processing-f9b8d9b9b-pdp4t\" (UID: \"ac8bec71-e814-400f-b48f-9932bfc4776d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.414222 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.414145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dafd306-8edf-40b5-8bf1-421805ba4210-maas-api-tls\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:57.414222 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.414179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4d8a62-5774-4a52-95c1-8ff6643c7598-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-9gggq\" (UID: \"ac4d8a62-5774-4a52-95c1-8ff6643c7598\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.414222 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.414212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbg2\" (UniqueName: \"kubernetes.io/projected/ac8bec71-e814-400f-b48f-9932bfc4776d-kube-api-access-kgbg2\") pod \"payload-processing-f9b8d9b9b-pdp4t\" (UID: \"ac8bec71-e814-400f-b48f-9932bfc4776d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.414553 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.414292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cs4\" (UniqueName: \"kubernetes.io/projected/ac4d8a62-5774-4a52-95c1-8ff6643c7598-kube-api-access-k8cs4\") pod \"payload-pre-processing-74584bb9f8-9gggq\" (UID: \"ac4d8a62-5774-4a52-95c1-8ff6643c7598\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.414553 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.414321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8bec71-e814-400f-b48f-9932bfc4776d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-pdp4t\" (UID: \"ac8bec71-e814-400f-b48f-9932bfc4776d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.415144 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.415120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8bec71-e814-400f-b48f-9932bfc4776d-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-pdp4t\" (UID: \"ac8bec71-e814-400f-b48f-9932bfc4776d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.415194 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.415121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4d8a62-5774-4a52-95c1-8ff6643c7598-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-9gggq\" (UID: \"ac4d8a62-5774-4a52-95c1-8ff6643c7598\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.416601 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.416579 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dafd306-8edf-40b5-8bf1-421805ba4210-maas-api-tls\") pod \"maas-api-78fc4dddbd-b5jqr\" (UID: \"4dafd306-8edf-40b5-8bf1-421805ba4210\") " pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:57.424101 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.424076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cs4\" (UniqueName: \"kubernetes.io/projected/ac4d8a62-5774-4a52-95c1-8ff6643c7598-kube-api-access-k8cs4\") pod \"payload-pre-processing-74584bb9f8-9gggq\" (UID: \"ac4d8a62-5774-4a52-95c1-8ff6643c7598\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.424327 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.424308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbg2\" (UniqueName: \"kubernetes.io/projected/ac8bec71-e814-400f-b48f-9932bfc4776d-kube-api-access-kgbg2\") pod \"payload-processing-f9b8d9b9b-pdp4t\" (UID: \"ac8bec71-e814-400f-b48f-9932bfc4776d\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.466321 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.466295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:40:57.518638 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.518591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:40:57.610833 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.610803 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-9gggq"] Jul 13 17:40:57.613290 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:57.613263 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4d8a62_5774_4a52_95c1_8ff6643c7598.slice/crio-a1817e390779851efde1443fb4236bfba3d98a9c90d5747bee7ea7938c8df766 WatchSource:0}: Error finding container a1817e390779851efde1443fb4236bfba3d98a9c90d5747bee7ea7938c8df766: Status 404 returned error can't find the container with id a1817e390779851efde1443fb4236bfba3d98a9c90d5747bee7ea7938c8df766 Jul 13 17:40:57.625614 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.625548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:40:57.663490 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.663438 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-pdp4t"] Jul 13 17:40:57.664961 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:57.664896 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8bec71_e814_400f_b48f_9932bfc4776d.slice/crio-d5059e4f71684fbdb0d4ea69cb38098ad1bbbc55e52100f2b0c55410b194395f WatchSource:0}: Error finding container d5059e4f71684fbdb0d4ea69cb38098ad1bbbc55e52100f2b0c55410b194395f: Status 404 returned error can't find the container with id d5059e4f71684fbdb0d4ea69cb38098ad1bbbc55e52100f2b0c55410b194395f Jul 13 17:40:57.809611 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:57.809576 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr"] Jul 13 17:40:57.810536 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:40:57.810511 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dafd306_8edf_40b5_8bf1_421805ba4210.slice/crio-a9b4b9da79eb50ea6ba632eaaaddc71d9be6f684ed58300f563da7ac56586e87 WatchSource:0}: Error finding container a9b4b9da79eb50ea6ba632eaaaddc71d9be6f684ed58300f563da7ac56586e87: Status 404 returned error can't find the container with id a9b4b9da79eb50ea6ba632eaaaddc71d9be6f684ed58300f563da7ac56586e87 Jul 13 17:40:58.233917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:58.233875 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" event={"ID":"ac8bec71-e814-400f-b48f-9932bfc4776d","Type":"ContainerStarted","Data":"d5059e4f71684fbdb0d4ea69cb38098ad1bbbc55e52100f2b0c55410b194395f"} Jul 13 17:40:58.235085 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:58.235060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" event={"ID":"ac4d8a62-5774-4a52-95c1-8ff6643c7598","Type":"ContainerStarted","Data":"a1817e390779851efde1443fb4236bfba3d98a9c90d5747bee7ea7938c8df766"} Jul 13 17:40:58.236108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:40:58.236076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" event={"ID":"4dafd306-8edf-40b5-8bf1-421805ba4210","Type":"ContainerStarted","Data":"a9b4b9da79eb50ea6ba632eaaaddc71d9be6f684ed58300f563da7ac56586e87"} Jul 13 17:41:00.250284 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.250226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" event={"ID":"4dafd306-8edf-40b5-8bf1-421805ba4210","Type":"ContainerStarted","Data":"662897975d97cb98074cd909f06db6db9a8742cd79257bc700bd076345347ef9"} Jul 13 17:41:00.250698 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.250310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:41:00.251818 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.251754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" event={"ID":"ac8bec71-e814-400f-b48f-9932bfc4776d","Type":"ContainerStarted","Data":"ff4bde1243460c2be1657cf3da12f08fa82faccb6424240a83929bee4b0ebc5b"} Jul 13 17:41:00.251818 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.251816 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:41:00.253143 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.253111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" event={"ID":"ac4d8a62-5774-4a52-95c1-8ff6643c7598","Type":"ContainerStarted","Data":"bceb5feddd99863a4649cbbcaf3a8031f2fe0fdbdf6b1af54a7a4e2722c183ae"} Jul 13 17:41:00.253271 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.253169 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:41:00.266131 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.266076 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" podStartSLOduration=1.992539579 podStartE2EDuration="4.266060104s" totalImagesPullingTime="2.273520525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:41:00.265593086 +0000 UTC m=+515.272995365" watchObservedRunningTime="2026-07-13 17:41:00.266060104 +0000 UTC m=+515.273462348" Jul 13 17:41:00.298309 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.298185 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" podStartSLOduration=0.876799628 podStartE2EDuration="3.298165057s" totalImagesPullingTime="2.421365429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:41:00.29698658 +0000 UTC m=+515.304388824" watchObservedRunningTime="2026-07-13 17:41:00.298165057 +0000 UTC m=+515.305567301" Jul 13 17:41:00.299373 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:00.299336 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" podStartSLOduration=0.932728896 podStartE2EDuration="3.299324395s" totalImagesPullingTime="2.366595499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:41:00.282196102 +0000 UTC m=+515.289598348" watchObservedRunningTime="2026-07-13 17:41:00.299324395 +0000 UTC m=+515.306726631" Jul 13 17:41:05.284339 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:05.284308 2564 generic.go:356] "Generic (PLEG): container finished" podID="88b20046-7323-4547-b96c-0931ce17020a" containerID="a0dadd517cc683b05f20645b5c168b990e22b8bdfbf8014cb9d7a67cbe59c1d2" exitCode=0 Jul 13 17:41:05.284751 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:05.284380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" event={"ID":"88b20046-7323-4547-b96c-0931ce17020a","Type":"ContainerDied","Data":"a0dadd517cc683b05f20645b5c168b990e22b8bdfbf8014cb9d7a67cbe59c1d2"} Jul 13 17:41:05.284802 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:05.284779 2564 scope.go:117] "RemoveContainer" containerID="a0dadd517cc683b05f20645b5c168b990e22b8bdfbf8014cb9d7a67cbe59c1d2" Jul 13 17:41:05.285760 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:05.285729 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:41:05.285833 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:05.285815 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:41:05.285873 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:05.285861 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 17:41:06.275578 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:06.275536 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78fc4dddbd-b5jqr" Jul 13 17:41:06.292922 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:06.292889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" event={"ID":"88b20046-7323-4547-b96c-0931ce17020a","Type":"ContainerStarted","Data":"527f8f9bc00d23423c1dbe7f74c43dc3dcca5656101df7a5b956e83a61b45771"} Jul 13 17:41:06.454377 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:06.454348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:41:06.455518 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:06.455485 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-87fqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 13 17:41:06.455662 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:06.455546 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" podUID="88b20046-7323-4547-b96c-0931ce17020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 13 17:41:07.280202 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:07.280165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:07.385681 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:07.385644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:07.454453 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:07.454427 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-87fqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 13 17:41:07.454600 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:07.454473 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" podUID="88b20046-7323-4547-b96c-0931ce17020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 13 17:41:08.316124 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:08.316090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:08.416287 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:08.416254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:08.454645 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:08.454616 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-87fqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 13 17:41:08.454780 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:08.454664 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" podUID="88b20046-7323-4547-b96c-0931ce17020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 13 17:41:09.454546 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:09.454515 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-87fqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 13 17:41:09.454980 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:09.454575 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" podUID="88b20046-7323-4547-b96c-0931ce17020a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 13 17:41:09.794665 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:09.794631 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:41:10.458261 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:10.458218 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:41:10.459667 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:10.459648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-87fqm" Jul 13 17:41:11.263661 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:11.263630 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-pdp4t" Jul 13 17:41:11.264116 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:11.264095 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-9gggq" Jul 13 17:41:13.088042 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:13.088007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:13.946686 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:13.946657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:14.047967 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:14.047941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:16.129418 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.129382 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4"] Jul 13 17:41:16.131991 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.131974 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.134358 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.134337 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 17:41:16.134467 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.134406 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 17:41:16.135393 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.135371 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 13 17:41:16.135393 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.135384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-87j4q\"" Jul 13 17:41:16.141898 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.141877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4"] Jul 13 17:41:16.224227 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.224189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:16.310566 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.310531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.310757 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.310709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2p6d\" (UniqueName: \"kubernetes.io/projected/2b22d4ba-fb4a-4179-953d-3bf5322de980-kube-api-access-v2p6d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.310757 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.310745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.310919 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.310780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.310919 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.310852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.310919 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.310894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b22d4ba-fb4a-4179-953d-3bf5322de980-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.411967 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.411879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2p6d\" (UniqueName: \"kubernetes.io/projected/2b22d4ba-fb4a-4179-953d-3bf5322de980-kube-api-access-v2p6d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.411967 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.411921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.412267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.412044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.412267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.412100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.412267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.412138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b22d4ba-fb4a-4179-953d-3bf5322de980-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.412267 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.412218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.412491 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.412348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.412491 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.412380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.412491 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.412441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.414518 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.414497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b22d4ba-fb4a-4179-953d-3bf5322de980-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.414602 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.414557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b22d4ba-fb4a-4179-953d-3bf5322de980-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.419899 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.419877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2p6d\" (UniqueName: \"kubernetes.io/projected/2b22d4ba-fb4a-4179-953d-3bf5322de980-kube-api-access-v2p6d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4\" (UID: \"2b22d4ba-fb4a-4179-953d-3bf5322de980\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.443213 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.443190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:16.573302 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:16.573275 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4"] Jul 13 17:41:16.575193 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:41:16.575165 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b22d4ba_fb4a_4179_953d_3bf5322de980.slice/crio-93c9a4d3ba9b70acbf9366c98f23912d1c2a42343eb8a690f7752878026c3340 WatchSource:0}: Error finding container 93c9a4d3ba9b70acbf9366c98f23912d1c2a42343eb8a690f7752878026c3340: Status 404 returned error can't find the container with id 93c9a4d3ba9b70acbf9366c98f23912d1c2a42343eb8a690f7752878026c3340 Jul 13 17:41:17.349825 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:17.349749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" event={"ID":"2b22d4ba-fb4a-4179-953d-3bf5322de980","Type":"ContainerStarted","Data":"93c9a4d3ba9b70acbf9366c98f23912d1c2a42343eb8a690f7752878026c3340"} Jul 13 17:41:18.357306 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:18.357272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" event={"ID":"2b22d4ba-fb4a-4179-953d-3bf5322de980","Type":"ContainerStarted","Data":"8daa7062ca1d1a247f8b18c93675e5558d25f29f246f9c4a8321d7c849f57d66"} Jul 13 17:41:18.374152 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:18.374091 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" podStartSLOduration=1.136781412 podStartE2EDuration="2.374071676s" totalImagesPullingTime="1.237290264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:41:18.37265298 +0000 UTC m=+533.380055225" watchObservedRunningTime="2026-07-13 17:41:18.374071676 +0000 UTC m=+533.381473919" Jul 13 17:41:19.516386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:19.516357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:19.574649 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:19.574622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:26.443359 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:26.443314 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:26.443807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:26.443566 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:26.455956 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:26.455927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:27.414142 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:27.414112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4" Jul 13 17:41:30.142516 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:30.142482 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:48.849554 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:48.849517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:58.258105 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:58.258068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:41:59.103955 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:41:59.103925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:42:05.008305 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.008276 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp"] Jul 13 17:42:05.014449 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.014427 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.016608 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.016588 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 13 17:42:05.021086 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.021049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp"] Jul 13 17:42:05.034318 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.034290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:42:05.059322 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.059292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.059472 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.059331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.059472 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.059366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48465a2d-2a39-49db-a243-a1c25af2d3c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.059472 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.059456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wms2q\" (UniqueName: \"kubernetes.io/projected/48465a2d-2a39-49db-a243-a1c25af2d3c5-kube-api-access-wms2q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.059641 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.059502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.059641 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.059558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160222 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160425 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160425 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48465a2d-2a39-49db-a243-a1c25af2d3c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160425 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wms2q\" (UniqueName: \"kubernetes.io/projected/48465a2d-2a39-49db-a243-a1c25af2d3c5-kube-api-access-wms2q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160595 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160595 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160704 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160780 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.160833 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.160774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.162567 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.162544 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48465a2d-2a39-49db-a243-a1c25af2d3c5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.162902 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.162881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48465a2d-2a39-49db-a243-a1c25af2d3c5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.169294 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.169267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wms2q\" (UniqueName: \"kubernetes.io/projected/48465a2d-2a39-49db-a243-a1c25af2d3c5-kube-api-access-wms2q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cmjvp\" (UID: \"48465a2d-2a39-49db-a243-a1c25af2d3c5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.326759 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.326719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:05.463709 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.463683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp"] Jul 13 17:42:05.466102 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:42:05.466077 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48465a2d_2a39_49db_a243_a1c25af2d3c5.slice/crio-f7e48eaab3ab051bda07932716577d776a2547b1e252b9ca8399a24779705367 WatchSource:0}: Error finding container f7e48eaab3ab051bda07932716577d776a2547b1e252b9ca8399a24779705367: Status 404 returned error can't find the container with id f7e48eaab3ab051bda07932716577d776a2547b1e252b9ca8399a24779705367 Jul 13 17:42:05.595748 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:05.595670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" event={"ID":"48465a2d-2a39-49db-a243-a1c25af2d3c5","Type":"ContainerStarted","Data":"f7e48eaab3ab051bda07932716577d776a2547b1e252b9ca8399a24779705367"} Jul 13 17:42:06.602672 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:06.602626 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" event={"ID":"48465a2d-2a39-49db-a243-a1c25af2d3c5","Type":"ContainerStarted","Data":"e9536831bb48d6f7ef9d53c2b9adece646171f7ec7dd31b026282d5c569d96f4"} Jul 13 17:42:06.619064 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:06.618985 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" podStartSLOduration=2.414580718 podStartE2EDuration="2.618955657s" totalImagesPullingTime="204.374939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:42:06.618289521 +0000 UTC m=+581.625691780" watchObservedRunningTime="2026-07-13 17:42:06.618955657 +0000 UTC m=+581.626357903" Jul 13 17:42:12.382368 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:12.382334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:42:14.451731 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:14.451692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:42:15.326913 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:15.326883 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:15.327170 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:15.327147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:15.339537 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:15.339515 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:15.660558 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:15.660464 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cmjvp" Jul 13 17:42:20.962185 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:20.962141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:42:25.481858 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:25.481830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:42:25.482506 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:42:25.482484 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:43:17.491257 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:17.491204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:43:18.144655 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.144611 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58c446fc8b-sj5gd"] Jul 13 17:43:18.144901 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.144848 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" podUID="9d85eb78-5dea-4273-8040-ec8f1820e18c" containerName="authorino" containerID="cri-o://d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391" gracePeriod=30 Jul 13 17:43:18.401521 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.401460 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:43:18.480907 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.480876 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9zck\" (UniqueName: \"kubernetes.io/projected/9d85eb78-5dea-4273-8040-ec8f1820e18c-kube-api-access-z9zck\") pod \"9d85eb78-5dea-4273-8040-ec8f1820e18c\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " Jul 13 17:43:18.481060 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.480997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d85eb78-5dea-4273-8040-ec8f1820e18c-tls-cert\") pod \"9d85eb78-5dea-4273-8040-ec8f1820e18c\" (UID: \"9d85eb78-5dea-4273-8040-ec8f1820e18c\") " Jul 13 17:43:18.482972 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.482941 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d85eb78-5dea-4273-8040-ec8f1820e18c-kube-api-access-z9zck" (OuterVolumeSpecName: "kube-api-access-z9zck") pod "9d85eb78-5dea-4273-8040-ec8f1820e18c" (UID: "9d85eb78-5dea-4273-8040-ec8f1820e18c"). InnerVolumeSpecName "kube-api-access-z9zck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:43:18.491879 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.491852 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d85eb78-5dea-4273-8040-ec8f1820e18c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9d85eb78-5dea-4273-8040-ec8f1820e18c" (UID: "9d85eb78-5dea-4273-8040-ec8f1820e18c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:43:18.562266 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.562204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:43:18.581646 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.581623 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d85eb78-5dea-4273-8040-ec8f1820e18c-tls-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:43:18.581646 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.581645 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9zck\" (UniqueName: \"kubernetes.io/projected/9d85eb78-5dea-4273-8040-ec8f1820e18c-kube-api-access-z9zck\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:43:18.775084 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.774989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:43:18.875588 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.875550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:43:18.977367 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.977331 2564 generic.go:356] "Generic (PLEG): container finished" podID="9d85eb78-5dea-4273-8040-ec8f1820e18c" containerID="d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391" exitCode=0 Jul 13 17:43:18.977532 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.977392 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" Jul 13 17:43:18.977532 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.977417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" event={"ID":"9d85eb78-5dea-4273-8040-ec8f1820e18c","Type":"ContainerDied","Data":"d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391"} Jul 13 17:43:18.977532 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.977459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58c446fc8b-sj5gd" event={"ID":"9d85eb78-5dea-4273-8040-ec8f1820e18c","Type":"ContainerDied","Data":"35cde4bc20e08fe92f1b563e33e1e77ed51ba5c3356c055e6e21e1149cbdb161"} Jul 13 17:43:18.977532 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.977476 2564 scope.go:117] "RemoveContainer" containerID="d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391" Jul 13 17:43:18.987085 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.987067 2564 scope.go:117] "RemoveContainer" containerID="d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391" Jul 13 17:43:18.987385 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:43:18.987365 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391\": container with ID starting with d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391 not found: ID does not exist" containerID="d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391" Jul 13 17:43:18.987457 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:18.987393 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391"} err="failed to get container status \"d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391\": rpc error: code = NotFound desc = could not find container \"d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391\": container with ID starting with d42c4b084aabad90128ae2d4b493d6cfc086924c370a455bd4ce442d77ba7391 not found: ID does not exist" Jul 13 17:43:19.002754 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:19.002733 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58c446fc8b-sj5gd"] Jul 13 17:43:19.008501 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:19.008480 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58c446fc8b-sj5gd"] Jul 13 17:43:19.638809 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:43:19.638776 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d85eb78-5dea-4273-8040-ec8f1820e18c" path="/var/lib/kubelet/pods/9d85eb78-5dea-4273-8040-ec8f1820e18c/volumes" Jul 13 17:44:31.026038 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.026004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:44:31.161616 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.161568 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-7tb2b"] Jul 13 17:44:31.161962 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.161931 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" podUID="0ed2598e-0c9d-4872-bcd8-96926a40e1e2" containerName="manager" containerID="cri-o://26a463200ffdc72bda5f6b4615790b0ee1cf0feaba8c47cfacbdacb023cc0394" gracePeriod=30 Jul 13 17:44:31.335429 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.335395 2564 generic.go:356] "Generic (PLEG): container finished" podID="0ed2598e-0c9d-4872-bcd8-96926a40e1e2" containerID="26a463200ffdc72bda5f6b4615790b0ee1cf0feaba8c47cfacbdacb023cc0394" exitCode=0 Jul 13 17:44:31.335595 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.335441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" event={"ID":"0ed2598e-0c9d-4872-bcd8-96926a40e1e2","Type":"ContainerDied","Data":"26a463200ffdc72bda5f6b4615790b0ee1cf0feaba8c47cfacbdacb023cc0394"} Jul 13 17:44:31.427504 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.427472 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:44:31.506657 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.506618 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7rc2\" (UniqueName: \"kubernetes.io/projected/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-kube-api-access-t7rc2\") pod \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " Jul 13 17:44:31.506867 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.506716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-webhook-cert\") pod \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\" (UID: \"0ed2598e-0c9d-4872-bcd8-96926a40e1e2\") " Jul 13 17:44:31.508809 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.508770 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0ed2598e-0c9d-4872-bcd8-96926a40e1e2" (UID: "0ed2598e-0c9d-4872-bcd8-96926a40e1e2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:44:31.508931 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.508804 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-kube-api-access-t7rc2" (OuterVolumeSpecName: "kube-api-access-t7rc2") pod "0ed2598e-0c9d-4872-bcd8-96926a40e1e2" (UID: "0ed2598e-0c9d-4872-bcd8-96926a40e1e2"). InnerVolumeSpecName "kube-api-access-t7rc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:44:31.607525 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.607488 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7rc2\" (UniqueName: \"kubernetes.io/projected/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-kube-api-access-t7rc2\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:44:31.607525 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:31.607519 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed2598e-0c9d-4872-bcd8-96926a40e1e2-webhook-cert\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:44:32.342376 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:32.342346 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" Jul 13 17:44:32.342809 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:32.342381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbcd4b6b8-7tb2b" event={"ID":"0ed2598e-0c9d-4872-bcd8-96926a40e1e2","Type":"ContainerDied","Data":"73b421a144cf833984e1c1cce00333980cdde96b57c94692941b20b4d55b5990"} Jul 13 17:44:32.342809 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:32.342439 2564 scope.go:117] "RemoveContainer" containerID="26a463200ffdc72bda5f6b4615790b0ee1cf0feaba8c47cfacbdacb023cc0394" Jul 13 17:44:32.360481 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:32.360448 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-7tb2b"] Jul 13 17:44:32.365183 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:32.365149 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-7tb2b"] Jul 13 17:44:33.043946 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.043915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:44:33.159282 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.159242 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-hh2pd"] Jul 13 17:44:33.159753 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.159734 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d85eb78-5dea-4273-8040-ec8f1820e18c" containerName="authorino" Jul 13 17:44:33.159840 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.159756 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d85eb78-5dea-4273-8040-ec8f1820e18c" containerName="authorino" Jul 13 17:44:33.159840 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.159768 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed2598e-0c9d-4872-bcd8-96926a40e1e2" containerName="manager" Jul 13 17:44:33.159840 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.159776 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed2598e-0c9d-4872-bcd8-96926a40e1e2" containerName="manager" Jul 13 17:44:33.160002 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.159912 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d85eb78-5dea-4273-8040-ec8f1820e18c" containerName="authorino" Jul 13 17:44:33.160002 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.159928 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed2598e-0c9d-4872-bcd8-96926a40e1e2" containerName="manager" Jul 13 17:44:33.164461 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.164433 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.167393 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.167366 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 17:44:33.167393 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.167385 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zmk4b\"" Jul 13 17:44:33.167576 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.167397 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 17:44:33.171709 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.171678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-hh2pd"] Jul 13 17:44:33.224323 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.224278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p6td\" (UniqueName: \"kubernetes.io/projected/b2b285b0-632a-48d2-b5a8-90015cc80bb8-kube-api-access-5p6td\") pod \"maas-controller-6cbcd4b6b8-hh2pd\" (UID: \"b2b285b0-632a-48d2-b5a8-90015cc80bb8\") " pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.224512 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.224343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b285b0-632a-48d2-b5a8-90015cc80bb8-webhook-cert\") pod \"maas-controller-6cbcd4b6b8-hh2pd\" (UID: \"b2b285b0-632a-48d2-b5a8-90015cc80bb8\") " pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.326026 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.325927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p6td\" (UniqueName: \"kubernetes.io/projected/b2b285b0-632a-48d2-b5a8-90015cc80bb8-kube-api-access-5p6td\") pod \"maas-controller-6cbcd4b6b8-hh2pd\" (UID: \"b2b285b0-632a-48d2-b5a8-90015cc80bb8\") " pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.326026 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.325973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b285b0-632a-48d2-b5a8-90015cc80bb8-webhook-cert\") pod \"maas-controller-6cbcd4b6b8-hh2pd\" (UID: \"b2b285b0-632a-48d2-b5a8-90015cc80bb8\") " pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.328478 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.328456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2b285b0-632a-48d2-b5a8-90015cc80bb8-webhook-cert\") pod \"maas-controller-6cbcd4b6b8-hh2pd\" (UID: \"b2b285b0-632a-48d2-b5a8-90015cc80bb8\") " pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.334581 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.334551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p6td\" (UniqueName: \"kubernetes.io/projected/b2b285b0-632a-48d2-b5a8-90015cc80bb8-kube-api-access-5p6td\") pod \"maas-controller-6cbcd4b6b8-hh2pd\" (UID: \"b2b285b0-632a-48d2-b5a8-90015cc80bb8\") " pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.477619 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.477583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:33.620017 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.619990 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbcd4b6b8-hh2pd"] Jul 13 17:44:33.621852 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:44:33.621823 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b285b0_632a_48d2_b5a8_90015cc80bb8.slice/crio-8c8ed959613ff1c0e28e2808aff428cb42eb15e62e803e5b97871ce89963e7a7 WatchSource:0}: Error finding container 8c8ed959613ff1c0e28e2808aff428cb42eb15e62e803e5b97871ce89963e7a7: Status 404 returned error can't find the container with id 8c8ed959613ff1c0e28e2808aff428cb42eb15e62e803e5b97871ce89963e7a7 Jul 13 17:44:33.623450 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.623428 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 17:44:33.638295 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:33.638261 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed2598e-0c9d-4872-bcd8-96926a40e1e2" path="/var/lib/kubelet/pods/0ed2598e-0c9d-4872-bcd8-96926a40e1e2/volumes" Jul 13 17:44:34.356433 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:34.356388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" event={"ID":"b2b285b0-632a-48d2-b5a8-90015cc80bb8","Type":"ContainerStarted","Data":"9df36a758aab9e1f20d79972afdeab93cd51ed001f7ddd5ed3a644467bd64de3"} Jul 13 17:44:34.356605 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:34.356555 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:44:34.356653 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:34.356630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" event={"ID":"b2b285b0-632a-48d2-b5a8-90015cc80bb8","Type":"ContainerStarted","Data":"8c8ed959613ff1c0e28e2808aff428cb42eb15e62e803e5b97871ce89963e7a7"} Jul 13 17:44:34.374830 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:34.374773 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" podStartSLOduration=0.830439231 podStartE2EDuration="1.374754569s" totalImagesPullingTime="544.315338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:44:34.372748507 +0000 UTC m=+729.380150767" watchObservedRunningTime="2026-07-13 17:44:34.374754569 +0000 UTC m=+729.382156811" Jul 13 17:44:45.365445 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:44:45.365412 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cbcd4b6b8-hh2pd" Jul 13 17:45:00.136178 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:00.136140 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw"] Jul 13 17:45:00.140086 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:00.140066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" Jul 13 17:45:00.143391 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:00.143070 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 17:45:00.145341 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:00.145311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw"] Jul 13 17:45:00.153702 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:00.153674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" Jul 13 17:45:00.291455 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:00.291363 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw"] Jul 13 17:45:00.293884 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:45:00.293852 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac522ace_8fed_40a1_8c79_24927b111be0.slice/crio-fb938dcd19aa8979ed903dd2d15dcbd3857209e6cfb19fb5b141cf18031f3a51 WatchSource:0}: Error finding container fb938dcd19aa8979ed903dd2d15dcbd3857209e6cfb19fb5b141cf18031f3a51: Status 404 returned error can't find the container with id fb938dcd19aa8979ed903dd2d15dcbd3857209e6cfb19fb5b141cf18031f3a51 Jul 13 17:45:00.499960 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:00.499869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" event={"ID":"ac522ace-8fed-40a1-8c79-24927b111be0","Type":"ContainerStarted","Data":"fb938dcd19aa8979ed903dd2d15dcbd3857209e6cfb19fb5b141cf18031f3a51"} Jul 13 17:45:03.518816 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:03.518778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" event={"ID":"ac522ace-8fed-40a1-8c79-24927b111be0","Type":"ContainerStarted","Data":"9299d556d30be7c31b9144d1cbcd65a6f72aa409b81b32dc7ecd469f1214f83e"} Jul 13 17:45:03.534858 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:03.534805 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" podStartSLOduration=1.128132814 podStartE2EDuration="3.534787944s" totalImagesPullingTime="2.40665513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:45:03.532306585 +0000 UTC m=+758.539708828" watchObservedRunningTime="2026-07-13 17:45:03.534787944 +0000 UTC m=+758.542190187" Jul 13 17:45:04.524950 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:04.524911 2564 generic.go:356] "Generic (PLEG): container finished" podID="ac522ace-8fed-40a1-8c79-24927b111be0" containerID="9299d556d30be7c31b9144d1cbcd65a6f72aa409b81b32dc7ecd469f1214f83e" exitCode=0 Jul 13 17:45:04.525359 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:04.524971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" event={"ID":"ac522ace-8fed-40a1-8c79-24927b111be0","Type":"ContainerDied","Data":"9299d556d30be7c31b9144d1cbcd65a6f72aa409b81b32dc7ecd469f1214f83e"} Jul 13 17:45:05.672093 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:05.672064 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" Jul 13 17:45:06.538209 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:06.538176 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" Jul 13 17:45:06.538394 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:06.538180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732745-bbbxw" event={"ID":"ac522ace-8fed-40a1-8c79-24927b111be0","Type":"ContainerDied","Data":"fb938dcd19aa8979ed903dd2d15dcbd3857209e6cfb19fb5b141cf18031f3a51"} Jul 13 17:45:06.538394 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:06.538333 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb938dcd19aa8979ed903dd2d15dcbd3857209e6cfb19fb5b141cf18031f3a51" Jul 13 17:45:14.039768 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:14.039729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:14.140225 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:14.140188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:14.278419 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:14.278385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:20.387099 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:20.387049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:20.488973 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:20.488935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:20.587400 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:20.587364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:20.762099 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:20.762019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:26.596879 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:26.596835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:26.661476 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:26.661448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:45:45.161947 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:45:45.161903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:47:00.642097 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:47:00.642058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:47:25.516352 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:47:25.516322 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:47:25.517162 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:47:25.517076 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:48:18.833635 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:48:18.833599 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:48:20.169506 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:48:20.169466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:49:17.314115 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:49:17.314015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:51:12.236496 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:51:12.236462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:52:25.548705 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:52:25.548677 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:52:25.556947 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:52:25.556928 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:53:18.529077 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:53:18.528996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:53:18.619773 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:53:18.619739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:55:44.640736 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:55:44.640703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:21.542286 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:21.542250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:27.816039 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:27.816002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:35.468173 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:35.468138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:44.532435 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:44.532406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:48.923678 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:48.923633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:52.024011 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:52.023975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:52.123068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:52.123035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:52.612083 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:52.612051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:58.353822 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:58.353788 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:58.457223 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:58.457193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:58.573812 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:58.573774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:58.648979 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:58.648889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:59.764291 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.764219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:59.868702 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.868670 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6"] Jul 13 17:56:59.869161 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.869147 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac522ace-8fed-40a1-8c79-24927b111be0" containerName="cleanup" Jul 13 17:56:59.869206 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.869164 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac522ace-8fed-40a1-8c79-24927b111be0" containerName="cleanup" Jul 13 17:56:59.869308 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.869295 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac522ace-8fed-40a1-8c79-24927b111be0" containerName="cleanup" Jul 13 17:56:59.871545 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.871502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:56:59.872642 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.872625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:56:59.875848 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.875828 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a4635dfb-serving-cert\"" Jul 13 17:56:59.885185 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.885163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6"] Jul 13 17:56:59.909210 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.909184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglck\" (UniqueName: \"kubernetes.io/projected/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-kube-api-access-lglck\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:56:59.909340 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.909215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-openshift-service-ca\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:56:59.909394 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.909338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:56:59.981422 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:56:59.981393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:57:00.010048 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.010025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.010166 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.010091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglck\" (UniqueName: \"kubernetes.io/projected/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-kube-api-access-lglck\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.010166 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.010131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-openshift-service-ca\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.010307 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:57:00.010165 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-serving-cert: secret "maas-api-e2e-ait-a4635dfb-serving-cert" not found Jul 13 17:57:00.010307 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:57:00.010254 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls podName:ffa8c090-ce1e-48b6-af89-b1cffd0abeaa nodeName:}" failed. No retries permitted until 2026-07-13 17:57:00.510219775 +0000 UTC m=+1475.517621996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls") pod "maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" (UID: "ffa8c090-ce1e-48b6-af89-b1cffd0abeaa") : secret "maas-api-e2e-ait-a4635dfb-serving-cert" not found Jul 13 17:57:00.010776 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.010759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-openshift-service-ca\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.021628 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.021572 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglck\" (UniqueName: \"kubernetes.io/projected/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-kube-api-access-lglck\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.294777 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.294746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:57:00.422743 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.422707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:57:00.515386 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.515351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.517703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.517682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls\") pod \"maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.526989 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.526963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:57:00.782916 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.782887 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:00.916189 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.916165 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6"] Jul 13 17:57:00.918210 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:57:00.918183 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa8c090_ce1e_48b6_af89_b1cffd0abeaa.slice/crio-c1bd3011a65ce364d4afe03b4fd0a2673aca03c57349c0c205166154926e001b WatchSource:0}: Error finding container c1bd3011a65ce364d4afe03b4fd0a2673aca03c57349c0c205166154926e001b: Status 404 returned error can't find the container with id c1bd3011a65ce364d4afe03b4fd0a2673aca03c57349c0c205166154926e001b Jul 13 17:57:00.919696 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:00.919675 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 17:57:01.170345 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:01.170270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" event={"ID":"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa","Type":"ContainerStarted","Data":"c1bd3011a65ce364d4afe03b4fd0a2673aca03c57349c0c205166154926e001b"} Jul 13 17:57:02.177294 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:02.177198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" event={"ID":"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa","Type":"ContainerStarted","Data":"45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691"} Jul 13 17:57:02.177637 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:02.177328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:02.194644 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:02.194600 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" podStartSLOduration=2.201623127 podStartE2EDuration="3.194587194s" totalImagesPullingTime="992.964067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:57:02.192371114 +0000 UTC m=+1477.199773557" watchObservedRunningTime="2026-07-13 17:57:02.194587194 +0000 UTC m=+1477.201989436" Jul 13 17:57:02.286176 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:02.286144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:57:04.795315 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:04.795289 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt"] Jul 13 17:57:04.798951 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:04.798930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" Jul 13 17:57:04.801034 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:04.801011 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 17:57:04.805431 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:04.805407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt"] Jul 13 17:57:04.861569 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:04.861537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721321a3-d9de-4316-bf69-7c18b5e3c226-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt\" (UID: \"721321a3-d9de-4316-bf69-7c18b5e3c226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" Jul 13 17:57:04.962424 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:04.962396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721321a3-d9de-4316-bf69-7c18b5e3c226-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt\" (UID: \"721321a3-d9de-4316-bf69-7c18b5e3c226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" Jul 13 17:57:04.963051 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:04.963031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721321a3-d9de-4316-bf69-7c18b5e3c226-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt\" (UID: \"721321a3-d9de-4316-bf69-7c18b5e3c226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" Jul 13 17:57:05.110905 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:05.110840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" Jul 13 17:57:05.243020 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:05.242996 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt"] Jul 13 17:57:05.246308 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:57:05.246276 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721321a3_d9de_4316_bf69_7c18b5e3c226.slice/crio-8abb99a2f3a6f6a9cb7d789175a743008dcfac70c440f7b96f1d4eeff805a3fe WatchSource:0}: Error finding container 8abb99a2f3a6f6a9cb7d789175a743008dcfac70c440f7b96f1d4eeff805a3fe: Status 404 returned error can't find the container with id 8abb99a2f3a6f6a9cb7d789175a743008dcfac70c440f7b96f1d4eeff805a3fe Jul 13 17:57:06.199433 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:06.199328 2564 generic.go:356] "Generic (PLEG): container finished" podID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerID="7b7e5c726565e4ac93fdd7b9dbf662a70acf9429cf02d656ddd2e9a55de4bff9" exitCode=7 Jul 13 17:57:06.199433 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:06.199412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" event={"ID":"721321a3-d9de-4316-bf69-7c18b5e3c226","Type":"ContainerDied","Data":"7b7e5c726565e4ac93fdd7b9dbf662a70acf9429cf02d656ddd2e9a55de4bff9"} Jul 13 17:57:06.199937 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:06.199487 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" event={"ID":"721321a3-d9de-4316-bf69-7c18b5e3c226","Type":"ContainerStarted","Data":"8abb99a2f3a6f6a9cb7d789175a743008dcfac70c440f7b96f1d4eeff805a3fe"} Jul 13 17:57:06.199937 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:06.199630 2564 scope.go:117] "RemoveContainer" containerID="7b7e5c726565e4ac93fdd7b9dbf662a70acf9429cf02d656ddd2e9a55de4bff9" Jul 13 17:57:07.206195 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:07.206159 2564 generic.go:356] "Generic (PLEG): container finished" podID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerID="3e863c82a373dcb28f922910483d90197c40d3dd22b85b05f85679f1006f371e" exitCode=7 Jul 13 17:57:07.206712 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:07.206210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" event={"ID":"721321a3-d9de-4316-bf69-7c18b5e3c226","Type":"ContainerDied","Data":"3e863c82a373dcb28f922910483d90197c40d3dd22b85b05f85679f1006f371e"} Jul 13 17:57:07.206712 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:07.206280 2564 scope.go:117] "RemoveContainer" containerID="7b7e5c726565e4ac93fdd7b9dbf662a70acf9429cf02d656ddd2e9a55de4bff9" Jul 13 17:57:07.206712 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:07.206475 2564 scope.go:117] "RemoveContainer" containerID="3e863c82a373dcb28f922910483d90197c40d3dd22b85b05f85679f1006f371e" Jul 13 17:57:07.206871 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:57:07.206751 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt_odh-ai-gateway-infra(721321a3-d9de-4316-bf69-7c18b5e3c226)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" Jul 13 17:57:08.188001 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:08.187965 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:57:08.213038 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:08.213007 2564 scope.go:117] "RemoveContainer" containerID="3e863c82a373dcb28f922910483d90197c40d3dd22b85b05f85679f1006f371e" Jul 13 17:57:08.213542 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:57:08.213225 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt_odh-ai-gateway-infra(721321a3-d9de-4316-bf69-7c18b5e3c226)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" Jul 13 17:57:21.632000 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:21.631967 2564 scope.go:117] "RemoveContainer" containerID="3e863c82a373dcb28f922910483d90197c40d3dd22b85b05f85679f1006f371e" Jul 13 17:57:22.285449 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:22.285420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" event={"ID":"721321a3-d9de-4316-bf69-7c18b5e3c226","Type":"ContainerStarted","Data":"b2b84cb7bb41aaf31ad564ef600ae8f1a4b2dc449b7fd770be4b9cb0c454bd58"} Jul 13 17:57:22.301746 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:22.301700 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" podStartSLOduration=18.301685473 podStartE2EDuration="18.301685473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:57:22.300521816 +0000 UTC m=+1497.307924062" watchObservedRunningTime="2026-07-13 17:57:22.301685473 +0000 UTC m=+1497.309087716" Jul 13 17:57:23.291951 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:23.291918 2564 generic.go:356] "Generic (PLEG): container finished" podID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerID="b2b84cb7bb41aaf31ad564ef600ae8f1a4b2dc449b7fd770be4b9cb0c454bd58" exitCode=0 Jul 13 17:57:23.292677 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:23.291991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" event={"ID":"721321a3-d9de-4316-bf69-7c18b5e3c226","Type":"ContainerDied","Data":"b2b84cb7bb41aaf31ad564ef600ae8f1a4b2dc449b7fd770be4b9cb0c454bd58"} Jul 13 17:57:23.292677 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:23.292066 2564 scope.go:117] "RemoveContainer" containerID="3e863c82a373dcb28f922910483d90197c40d3dd22b85b05f85679f1006f371e" Jul 13 17:57:23.313958 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:23.313935 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt"] Jul 13 17:57:24.436293 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:24.436270 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" Jul 13 17:57:24.537381 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:24.537352 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721321a3-d9de-4316-bf69-7c18b5e3c226-maas-api-service-ca\") pod \"721321a3-d9de-4316-bf69-7c18b5e3c226\" (UID: \"721321a3-d9de-4316-bf69-7c18b5e3c226\") " Jul 13 17:57:24.537639 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:24.537619 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721321a3-d9de-4316-bf69-7c18b5e3c226-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "721321a3-d9de-4316-bf69-7c18b5e3c226" (UID: "721321a3-d9de-4316-bf69-7c18b5e3c226"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:57:24.638075 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:24.638011 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721321a3-d9de-4316-bf69-7c18b5e3c226-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:57:25.305714 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.305687 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" Jul 13 17:57:25.305875 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.305728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt" event={"ID":"721321a3-d9de-4316-bf69-7c18b5e3c226","Type":"ContainerDied","Data":"8abb99a2f3a6f6a9cb7d789175a743008dcfac70c440f7b96f1d4eeff805a3fe"} Jul 13 17:57:25.305875 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.305777 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8abb99a2f3a6f6a9cb7d789175a743008dcfac70c440f7b96f1d4eeff805a3fe" Jul 13 17:57:25.335360 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.335335 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt"] Jul 13 17:57:25.339031 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.339010 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-lzvnt"] Jul 13 17:57:25.584638 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.584470 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:57:25.597164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.597141 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 17:57:25.638042 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:25.638017 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" path="/var/lib/kubelet/pods/721321a3-d9de-4316-bf69-7c18b5e3c226/volumes" Jul 13 17:57:41.873371 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:41.873333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:57:52.663430 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:57:52.663349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:04.849996 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.849963 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j"] Jul 13 17:58:04.850499 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850421 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850499 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850433 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850499 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850442 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850499 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850450 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850499 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850465 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850499 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850470 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850720 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850542 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850720 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850551 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.850720 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.850558 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="721321a3-d9de-4316-bf69-7c18b5e3c226" containerName="revoke-keys" Jul 13 17:58:04.853821 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.853804 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" Jul 13 17:58:04.855909 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.855889 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 17:58:04.862055 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:04.862015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j"] Jul 13 17:58:05.003332 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.003303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36272dc3-c046-48cc-8889-36c110c773ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j\" (UID: \"36272dc3-c046-48cc-8889-36c110c773ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" Jul 13 17:58:05.104003 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.103937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36272dc3-c046-48cc-8889-36c110c773ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j\" (UID: \"36272dc3-c046-48cc-8889-36c110c773ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" Jul 13 17:58:05.104552 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.104533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36272dc3-c046-48cc-8889-36c110c773ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j\" (UID: \"36272dc3-c046-48cc-8889-36c110c773ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" Jul 13 17:58:05.166127 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.166101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" Jul 13 17:58:05.297194 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.297171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j"] Jul 13 17:58:05.299713 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:58:05.299681 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36272dc3_c046_48cc_8889_36c110c773ef.slice/crio-c8c81fa3ff0a6cfc784658bde35fbfb08e5ba266f4c95d968e613f9664e13dd6 WatchSource:0}: Error finding container c8c81fa3ff0a6cfc784658bde35fbfb08e5ba266f4c95d968e613f9664e13dd6: Status 404 returned error can't find the container with id c8c81fa3ff0a6cfc784658bde35fbfb08e5ba266f4c95d968e613f9664e13dd6 Jul 13 17:58:05.512957 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.512929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" event={"ID":"36272dc3-c046-48cc-8889-36c110c773ef","Type":"ContainerStarted","Data":"ffd5c9c2b437a679f57be22285e590a6a87c5ce9f6b80a22c9550f8e68eabd7a"} Jul 13 17:58:05.513056 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.512975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" event={"ID":"36272dc3-c046-48cc-8889-36c110c773ef","Type":"ContainerStarted","Data":"c8c81fa3ff0a6cfc784658bde35fbfb08e5ba266f4c95d968e613f9664e13dd6"} Jul 13 17:58:05.528108 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:05.528067 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" podStartSLOduration=1.5280533200000002 podStartE2EDuration="1.52805332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:58:05.526438657 +0000 UTC m=+1540.533840901" watchObservedRunningTime="2026-07-13 17:58:05.52805332 +0000 UTC m=+1540.535455566" Jul 13 17:58:06.526059 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:06.526021 2564 generic.go:356] "Generic (PLEG): container finished" podID="36272dc3-c046-48cc-8889-36c110c773ef" containerID="ffd5c9c2b437a679f57be22285e590a6a87c5ce9f6b80a22c9550f8e68eabd7a" exitCode=0 Jul 13 17:58:06.526455 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:06.526098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" event={"ID":"36272dc3-c046-48cc-8889-36c110c773ef","Type":"ContainerDied","Data":"ffd5c9c2b437a679f57be22285e590a6a87c5ce9f6b80a22c9550f8e68eabd7a"} Jul 13 17:58:07.666415 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:07.666394 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" Jul 13 17:58:07.830761 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:07.830739 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36272dc3-c046-48cc-8889-36c110c773ef-maas-api-service-ca\") pod \"36272dc3-c046-48cc-8889-36c110c773ef\" (UID: \"36272dc3-c046-48cc-8889-36c110c773ef\") " Jul 13 17:58:07.831056 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:07.831034 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36272dc3-c046-48cc-8889-36c110c773ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36272dc3-c046-48cc-8889-36c110c773ef" (UID: "36272dc3-c046-48cc-8889-36c110c773ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:58:07.931778 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:07.931755 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36272dc3-c046-48cc-8889-36c110c773ef-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:58:08.537891 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:08.537858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" event={"ID":"36272dc3-c046-48cc-8889-36c110c773ef","Type":"ContainerDied","Data":"c8c81fa3ff0a6cfc784658bde35fbfb08e5ba266f4c95d968e613f9664e13dd6"} Jul 13 17:58:08.537891 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:08.537885 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j" Jul 13 17:58:08.538067 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:08.537907 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c81fa3ff0a6cfc784658bde35fbfb08e5ba266f4c95d968e613f9664e13dd6" Jul 13 17:58:15.255538 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.255499 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6"] Jul 13 17:58:15.255929 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.255745 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" podUID="ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" containerName="maas-api" containerID="cri-o://45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691" gracePeriod=30 Jul 13 17:58:15.278033 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.278007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:15.500781 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.500755 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:58:15.581423 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.581393 2564 generic.go:356] "Generic (PLEG): container finished" podID="ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" containerID="45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691" exitCode=0 Jul 13 17:58:15.581582 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.581442 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" Jul 13 17:58:15.581582 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.581473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" event={"ID":"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa","Type":"ContainerDied","Data":"45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691"} Jul 13 17:58:15.581582 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.581522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6" event={"ID":"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa","Type":"ContainerDied","Data":"c1bd3011a65ce364d4afe03b4fd0a2673aca03c57349c0c205166154926e001b"} Jul 13 17:58:15.581582 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.581545 2564 scope.go:117] "RemoveContainer" containerID="45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691" Jul 13 17:58:15.589984 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.589970 2564 scope.go:117] "RemoveContainer" containerID="45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691" Jul 13 17:58:15.590221 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:58:15.590204 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691\": container with ID starting with 45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691 not found: ID does not exist" containerID="45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691" Jul 13 17:58:15.590298 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.590228 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691"} err="failed to get container status \"45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691\": rpc error: code = NotFound desc = could not find container \"45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691\": container with ID starting with 45dc7a6d83ac437cd1e7081f043382575d3fce480476241ce0911c1f868c3691 not found: ID does not exist" Jul 13 17:58:15.603038 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.603019 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-openshift-service-ca\") pod \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " Jul 13 17:58:15.603101 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.603056 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lglck\" (UniqueName: \"kubernetes.io/projected/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-kube-api-access-lglck\") pod \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " Jul 13 17:58:15.603141 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.603122 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls\") pod \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\" (UID: \"ffa8c090-ce1e-48b6-af89-b1cffd0abeaa\") " Jul 13 17:58:15.603347 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.603327 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" (UID: "ffa8c090-ce1e-48b6-af89-b1cffd0abeaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:58:15.605080 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.605059 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-kube-api-access-lglck" (OuterVolumeSpecName: "kube-api-access-lglck") pod "ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" (UID: "ffa8c090-ce1e-48b6-af89-b1cffd0abeaa"). InnerVolumeSpecName "kube-api-access-lglck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:58:15.605167 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.605059 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" (UID: "ffa8c090-ce1e-48b6-af89-b1cffd0abeaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:58:15.704691 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.704667 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lglck\" (UniqueName: \"kubernetes.io/projected/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-kube-api-access-lglck\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:58:15.704813 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.704696 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:58:15.704813 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.704712 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:58:15.897324 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.897255 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6"] Jul 13 17:58:15.903086 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:15.903065 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4635dfb-7d579fcf4d-qzcr6"] Jul 13 17:58:17.638498 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:17.638465 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" path="/var/lib/kubelet/pods/ffa8c090-ce1e-48b6-af89-b1cffd0abeaa/volumes" Jul 13 17:58:19.934815 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:19.934782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:25.839641 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:25.839610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:25.905715 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:25.905690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:30.190046 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:30.190008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:30.634021 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:30.633988 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:30.733816 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:30.733787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:30.842604 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:30.842577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:32.084749 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.084719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:32.156336 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.156307 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5"] Jul 13 17:58:32.156804 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.156788 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36272dc3-c046-48cc-8889-36c110c773ef" containerName="revoke-keys" Jul 13 17:58:32.156804 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.156805 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="36272dc3-c046-48cc-8889-36c110c773ef" containerName="revoke-keys" Jul 13 17:58:32.156957 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.156826 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" containerName="maas-api" Jul 13 17:58:32.156957 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.156835 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" containerName="maas-api" Jul 13 17:58:32.157057 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.156956 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa8c090-ce1e-48b6-af89-b1cffd0abeaa" containerName="maas-api" Jul 13 17:58:32.157057 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.156971 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="36272dc3-c046-48cc-8889-36c110c773ef" containerName="revoke-keys" Jul 13 17:58:32.161602 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.161579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.163707 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.163686 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bcf3d6aa-serving-cert\"" Jul 13 17:58:32.169207 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.169182 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5"] Jul 13 17:58:32.189085 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.189065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:32.244708 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.244682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.244816 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.244715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flq4c\" (UniqueName: \"kubernetes.io/projected/638927bd-9cfb-4e2c-9998-eb47c42490ac-kube-api-access-flq4c\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.244816 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.244753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638927bd-9cfb-4e2c-9998-eb47c42490ac-openshift-service-ca\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.345912 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.345825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.345912 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.345865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flq4c\" (UniqueName: \"kubernetes.io/projected/638927bd-9cfb-4e2c-9998-eb47c42490ac-kube-api-access-flq4c\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.346103 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:58:32.345964 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-serving-cert: secret "maas-api-e2e-migrate-bcf3d6aa-serving-cert" not found Jul 13 17:58:32.346103 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.345980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638927bd-9cfb-4e2c-9998-eb47c42490ac-openshift-service-ca\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.346103 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:58:32.346042 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls podName:638927bd-9cfb-4e2c-9998-eb47c42490ac nodeName:}" failed. No retries permitted until 2026-07-13 17:58:32.846020379 +0000 UTC m=+1567.853422601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls") pod "maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" (UID: "638927bd-9cfb-4e2c-9998-eb47c42490ac") : secret "maas-api-e2e-migrate-bcf3d6aa-serving-cert" not found Jul 13 17:58:32.346652 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.346633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638927bd-9cfb-4e2c-9998-eb47c42490ac-openshift-service-ca\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.354851 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.354831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flq4c\" (UniqueName: \"kubernetes.io/projected/638927bd-9cfb-4e2c-9998-eb47c42490ac-kube-api-access-flq4c\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.610181 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.610086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:32.711251 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.711173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:32.809250 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.809204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:32.851791 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.851760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:32.854373 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:32.854345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls\") pod \"maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:33.074908 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:33.074872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:33.214827 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:33.214796 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5"] Jul 13 17:58:33.216302 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:58:33.216274 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638927bd_9cfb_4e2c_9998_eb47c42490ac.slice/crio-ffd95759f0adc5324d3c16e730c887eee4d4ce6c0f421add7f2f2160d1f1ca74 WatchSource:0}: Error finding container ffd95759f0adc5324d3c16e730c887eee4d4ce6c0f421add7f2f2160d1f1ca74: Status 404 returned error can't find the container with id ffd95759f0adc5324d3c16e730c887eee4d4ce6c0f421add7f2f2160d1f1ca74 Jul 13 17:58:33.674844 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:33.674809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" event={"ID":"638927bd-9cfb-4e2c-9998-eb47c42490ac","Type":"ContainerStarted","Data":"ffd95759f0adc5324d3c16e730c887eee4d4ce6c0f421add7f2f2160d1f1ca74"} Jul 13 17:58:34.681733 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:34.681700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" event={"ID":"638927bd-9cfb-4e2c-9998-eb47c42490ac","Type":"ContainerStarted","Data":"a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee"} Jul 13 17:58:34.682103 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:34.681829 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:34.698599 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:34.698555 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" podStartSLOduration=2.114704733 podStartE2EDuration="2.698540904s" totalImagesPullingTime="583.836171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:58:34.696338237 +0000 UTC m=+1569.703740505" watchObservedRunningTime="2026-07-13 17:58:34.698540904 +0000 UTC m=+1569.705943146" Jul 13 17:58:36.650336 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.650304 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw"] Jul 13 17:58:36.654440 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.654423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" Jul 13 17:58:36.657012 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.656978 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 17:58:36.659686 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.659663 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw"] Jul 13 17:58:36.787882 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.787849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b768f346-bca9-4e10-9b91-6c0cd39b2cd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw\" (UID: \"b768f346-bca9-4e10-9b91-6c0cd39b2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" Jul 13 17:58:36.889185 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.889158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b768f346-bca9-4e10-9b91-6c0cd39b2cd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw\" (UID: \"b768f346-bca9-4e10-9b91-6c0cd39b2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" Jul 13 17:58:36.889797 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.889766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b768f346-bca9-4e10-9b91-6c0cd39b2cd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw\" (UID: \"b768f346-bca9-4e10-9b91-6c0cd39b2cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" Jul 13 17:58:36.967288 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:36.967197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" Jul 13 17:58:37.103946 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:37.103919 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw"] Jul 13 17:58:37.106264 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:58:37.106217 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb768f346_bca9_4e10_9b91_6c0cd39b2cd2.slice/crio-26b02aba76e809a22ec0f3152f1a876ea06be08345b22f3b4cd1446ccb9844bc WatchSource:0}: Error finding container 26b02aba76e809a22ec0f3152f1a876ea06be08345b22f3b4cd1446ccb9844bc: Status 404 returned error can't find the container with id 26b02aba76e809a22ec0f3152f1a876ea06be08345b22f3b4cd1446ccb9844bc Jul 13 17:58:37.700093 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:37.700016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" event={"ID":"b768f346-bca9-4e10-9b91-6c0cd39b2cd2","Type":"ContainerStarted","Data":"3195a67b5a4ec7fb4801bd6a92fc369d21eee953b186fba34385668484552a85"} Jul 13 17:58:37.700093 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:37.700063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" event={"ID":"b768f346-bca9-4e10-9b91-6c0cd39b2cd2","Type":"ContainerStarted","Data":"26b02aba76e809a22ec0f3152f1a876ea06be08345b22f3b4cd1446ccb9844bc"} Jul 13 17:58:37.715046 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:37.715002 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" podStartSLOduration=1.714988094 podStartE2EDuration="1.714988094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:58:37.713801686 +0000 UTC m=+1572.721203930" watchObservedRunningTime="2026-07-13 17:58:37.714988094 +0000 UTC m=+1572.722390338" Jul 13 17:58:38.710181 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:38.710149 2564 generic.go:356] "Generic (PLEG): container finished" podID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerID="3195a67b5a4ec7fb4801bd6a92fc369d21eee953b186fba34385668484552a85" exitCode=7 Jul 13 17:58:38.710565 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:38.710190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" event={"ID":"b768f346-bca9-4e10-9b91-6c0cd39b2cd2","Type":"ContainerDied","Data":"3195a67b5a4ec7fb4801bd6a92fc369d21eee953b186fba34385668484552a85"} Jul 13 17:58:38.710565 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:38.710396 2564 scope.go:117] "RemoveContainer" containerID="3195a67b5a4ec7fb4801bd6a92fc369d21eee953b186fba34385668484552a85" Jul 13 17:58:39.717532 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:39.717494 2564 generic.go:356] "Generic (PLEG): container finished" podID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerID="e63143fd1e80571ec74ee211f0d5ea6d0003f747fd8865896c594e61afb2d553" exitCode=7 Jul 13 17:58:39.717925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:39.717574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" event={"ID":"b768f346-bca9-4e10-9b91-6c0cd39b2cd2","Type":"ContainerDied","Data":"e63143fd1e80571ec74ee211f0d5ea6d0003f747fd8865896c594e61afb2d553"} Jul 13 17:58:39.717925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:39.717639 2564 scope.go:117] "RemoveContainer" containerID="3195a67b5a4ec7fb4801bd6a92fc369d21eee953b186fba34385668484552a85" Jul 13 17:58:39.717925 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:39.717855 2564 scope.go:117] "RemoveContainer" containerID="e63143fd1e80571ec74ee211f0d5ea6d0003f747fd8865896c594e61afb2d553" Jul 13 17:58:39.718125 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:58:39.718084 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw_odh-ai-gateway-infra(b768f346-bca9-4e10-9b91-6c0cd39b2cd2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" Jul 13 17:58:40.692818 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:40.692789 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:58:40.725165 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:40.725142 2564 scope.go:117] "RemoveContainer" containerID="e63143fd1e80571ec74ee211f0d5ea6d0003f747fd8865896c594e61afb2d553" Jul 13 17:58:40.725538 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:58:40.725335 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw_odh-ai-gateway-infra(b768f346-bca9-4e10-9b91-6c0cd39b2cd2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" Jul 13 17:58:40.742906 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:40.742878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:41.591038 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:41.591005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:41.687821 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:41.687782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:45.124861 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:45.124829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:58:52.632011 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:52.631980 2564 scope.go:117] "RemoveContainer" containerID="e63143fd1e80571ec74ee211f0d5ea6d0003f747fd8865896c594e61afb2d553" Jul 13 17:58:53.797950 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:53.797917 2564 generic.go:356] "Generic (PLEG): container finished" podID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerID="7a022e4d3bcc21d8e1185ce3949463d6c88243479cd9ff0b72221acdf2b56dcb" exitCode=0 Jul 13 17:58:53.798428 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:53.797980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" event={"ID":"b768f346-bca9-4e10-9b91-6c0cd39b2cd2","Type":"ContainerDied","Data":"7a022e4d3bcc21d8e1185ce3949463d6c88243479cd9ff0b72221acdf2b56dcb"} Jul 13 17:58:53.798428 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:53.798074 2564 scope.go:117] "RemoveContainer" containerID="e63143fd1e80571ec74ee211f0d5ea6d0003f747fd8865896c594e61afb2d553" Jul 13 17:58:54.829068 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:54.829036 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw"] Jul 13 17:58:54.942712 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:54.942690 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" Jul 13 17:58:55.041128 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.041098 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b768f346-bca9-4e10-9b91-6c0cd39b2cd2-maas-api-service-ca\") pod \"b768f346-bca9-4e10-9b91-6c0cd39b2cd2\" (UID: \"b768f346-bca9-4e10-9b91-6c0cd39b2cd2\") " Jul 13 17:58:55.041466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.041446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b768f346-bca9-4e10-9b91-6c0cd39b2cd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b768f346-bca9-4e10-9b91-6c0cd39b2cd2" (UID: "b768f346-bca9-4e10-9b91-6c0cd39b2cd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:58:55.142216 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.142152 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b768f346-bca9-4e10-9b91-6c0cd39b2cd2-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:58:55.812164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.812127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" event={"ID":"b768f346-bca9-4e10-9b91-6c0cd39b2cd2","Type":"ContainerDied","Data":"26b02aba76e809a22ec0f3152f1a876ea06be08345b22f3b4cd1446ccb9844bc"} Jul 13 17:58:55.812164 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.812145 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw" Jul 13 17:58:55.812375 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.812179 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b02aba76e809a22ec0f3152f1a876ea06be08345b22f3b4cd1446ccb9844bc" Jul 13 17:58:55.842410 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.842386 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw"] Jul 13 17:58:55.846211 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:55.846189 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-gbfjw"] Jul 13 17:58:57.637843 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:58:57.637804 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" path="/var/lib/kubelet/pods/b768f346-bca9-4e10-9b91-6c0cd39b2cd2/volumes" Jul 13 17:59:26.710599 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.710558 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m"] Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711032 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711044 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711055 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711060 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711067 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711074 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711134 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711142 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.711193 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.711150 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b768f346-bca9-4e10-9b91-6c0cd39b2cd2" containerName="revoke-keys" Jul 13 17:59:26.714292 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.714268 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" Jul 13 17:59:26.716269 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.716228 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 17:59:26.721624 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.721597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m"] Jul 13 17:59:26.834192 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.834145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5da0444-4c29-4540-a2ae-8bddb45bbf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m\" (UID: \"c5da0444-4c29-4540-a2ae-8bddb45bbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" Jul 13 17:59:26.935691 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.935648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5da0444-4c29-4540-a2ae-8bddb45bbf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m\" (UID: \"c5da0444-4c29-4540-a2ae-8bddb45bbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" Jul 13 17:59:26.936358 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:26.936333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5da0444-4c29-4540-a2ae-8bddb45bbf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m\" (UID: \"c5da0444-4c29-4540-a2ae-8bddb45bbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" Jul 13 17:59:27.028097 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:27.028010 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" Jul 13 17:59:27.170729 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:27.170694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m"] Jul 13 17:59:27.172663 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:59:27.172634 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5da0444_4c29_4540_a2ae_8bddb45bbf5e.slice/crio-45170565a260b0053106330920f60d7ab770922ae8ec7816f2fd0beb444f38fe WatchSource:0}: Error finding container 45170565a260b0053106330920f60d7ab770922ae8ec7816f2fd0beb444f38fe: Status 404 returned error can't find the container with id 45170565a260b0053106330920f60d7ab770922ae8ec7816f2fd0beb444f38fe Jul 13 17:59:27.980732 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:27.980698 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5da0444-4c29-4540-a2ae-8bddb45bbf5e" containerID="7d5a43901a9eb557eb6a716eaf1320e9a16061f8ef5992ffca3185f62ea52111" exitCode=0 Jul 13 17:59:27.981099 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:27.980784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" event={"ID":"c5da0444-4c29-4540-a2ae-8bddb45bbf5e","Type":"ContainerDied","Data":"7d5a43901a9eb557eb6a716eaf1320e9a16061f8ef5992ffca3185f62ea52111"} Jul 13 17:59:27.981099 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:27.980835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" event={"ID":"c5da0444-4c29-4540-a2ae-8bddb45bbf5e","Type":"ContainerStarted","Data":"45170565a260b0053106330920f60d7ab770922ae8ec7816f2fd0beb444f38fe"} Jul 13 17:59:29.130678 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:29.130644 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" Jul 13 17:59:29.262044 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:29.261944 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5da0444-4c29-4540-a2ae-8bddb45bbf5e-maas-api-service-ca\") pod \"c5da0444-4c29-4540-a2ae-8bddb45bbf5e\" (UID: \"c5da0444-4c29-4540-a2ae-8bddb45bbf5e\") " Jul 13 17:59:29.262384 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:29.262356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5da0444-4c29-4540-a2ae-8bddb45bbf5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5da0444-4c29-4540-a2ae-8bddb45bbf5e" (UID: "c5da0444-4c29-4540-a2ae-8bddb45bbf5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:59:29.363348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:29.363317 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5da0444-4c29-4540-a2ae-8bddb45bbf5e-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:59:29.994302 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:29.994193 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" event={"ID":"c5da0444-4c29-4540-a2ae-8bddb45bbf5e","Type":"ContainerDied","Data":"45170565a260b0053106330920f60d7ab770922ae8ec7816f2fd0beb444f38fe"} Jul 13 17:59:29.994302 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:29.994203 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m" Jul 13 17:59:29.994302 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:29.994260 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45170565a260b0053106330920f60d7ab770922ae8ec7816f2fd0beb444f38fe" Jul 13 17:59:36.797082 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:36.797045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:36.914273 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:36.914211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:37.108556 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.108468 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5"] Jul 13 17:59:37.108729 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.108700 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" podUID="638927bd-9cfb-4e2c-9998-eb47c42490ac" containerName="maas-api" containerID="cri-o://a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee" gracePeriod=30 Jul 13 17:59:37.135631 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.135607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:37.367964 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.367894 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:59:37.443407 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.443372 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638927bd-9cfb-4e2c-9998-eb47c42490ac-openshift-service-ca\") pod \"638927bd-9cfb-4e2c-9998-eb47c42490ac\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " Jul 13 17:59:37.443590 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.443425 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls\") pod \"638927bd-9cfb-4e2c-9998-eb47c42490ac\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " Jul 13 17:59:37.443590 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.443477 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flq4c\" (UniqueName: \"kubernetes.io/projected/638927bd-9cfb-4e2c-9998-eb47c42490ac-kube-api-access-flq4c\") pod \"638927bd-9cfb-4e2c-9998-eb47c42490ac\" (UID: \"638927bd-9cfb-4e2c-9998-eb47c42490ac\") " Jul 13 17:59:37.443809 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.443780 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638927bd-9cfb-4e2c-9998-eb47c42490ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "638927bd-9cfb-4e2c-9998-eb47c42490ac" (UID: "638927bd-9cfb-4e2c-9998-eb47c42490ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 17:59:37.445629 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.445601 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638927bd-9cfb-4e2c-9998-eb47c42490ac-kube-api-access-flq4c" (OuterVolumeSpecName: "kube-api-access-flq4c") pod "638927bd-9cfb-4e2c-9998-eb47c42490ac" (UID: "638927bd-9cfb-4e2c-9998-eb47c42490ac"). InnerVolumeSpecName "kube-api-access-flq4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 17:59:37.445717 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.445636 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "638927bd-9cfb-4e2c-9998-eb47c42490ac" (UID: "638927bd-9cfb-4e2c-9998-eb47c42490ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 17:59:37.545027 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.544986 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638927bd-9cfb-4e2c-9998-eb47c42490ac-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:59:37.545027 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.545022 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638927bd-9cfb-4e2c-9998-eb47c42490ac-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:59:37.545027 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:37.545031 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flq4c\" (UniqueName: \"kubernetes.io/projected/638927bd-9cfb-4e2c-9998-eb47c42490ac-kube-api-access-flq4c\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 17:59:38.037649 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.037611 2564 generic.go:356] "Generic (PLEG): container finished" podID="638927bd-9cfb-4e2c-9998-eb47c42490ac" containerID="a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee" exitCode=0 Jul 13 17:59:38.038029 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.037658 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" Jul 13 17:59:38.038029 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.037692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" event={"ID":"638927bd-9cfb-4e2c-9998-eb47c42490ac","Type":"ContainerDied","Data":"a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee"} Jul 13 17:59:38.038029 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.037746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5" event={"ID":"638927bd-9cfb-4e2c-9998-eb47c42490ac","Type":"ContainerDied","Data":"ffd95759f0adc5324d3c16e730c887eee4d4ce6c0f421add7f2f2160d1f1ca74"} Jul 13 17:59:38.038029 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.037768 2564 scope.go:117] "RemoveContainer" containerID="a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee" Jul 13 17:59:38.046917 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.046899 2564 scope.go:117] "RemoveContainer" containerID="a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee" Jul 13 17:59:38.047188 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:59:38.047167 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee\": container with ID starting with a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee not found: ID does not exist" containerID="a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee" Jul 13 17:59:38.047287 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.047198 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee"} err="failed to get container status \"a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee\": rpc error: code = NotFound desc = could not find container \"a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee\": container with ID starting with a81fa91c2fa0ac7b20a9325ecc76ba872ffe4412d25627e298ca7cf0984a6dee not found: ID does not exist" Jul 13 17:59:38.054024 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.054004 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5"] Jul 13 17:59:38.058466 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:38.058445 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bcf3d6aa-9df746dcd-sdwm5"] Jul 13 17:59:39.638244 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:39.638195 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="638927bd-9cfb-4e2c-9998-eb47c42490ac" path="/var/lib/kubelet/pods/638927bd-9cfb-4e2c-9998-eb47c42490ac/volumes" Jul 13 17:59:42.968560 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:42.968526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:43.084879 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:43.084842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:43.235290 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:43.235178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:43.328940 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:43.328898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:43.428489 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:43.428457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:44.418807 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.418771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:44.507348 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.507315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:44.558163 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.558125 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh"] Jul 13 17:59:44.559084 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.559060 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="638927bd-9cfb-4e2c-9998-eb47c42490ac" containerName="maas-api" Jul 13 17:59:44.559246 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.559216 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="638927bd-9cfb-4e2c-9998-eb47c42490ac" containerName="maas-api" Jul 13 17:59:44.559359 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.559345 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5da0444-4c29-4540-a2ae-8bddb45bbf5e" containerName="revoke-keys" Jul 13 17:59:44.559426 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.559377 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5da0444-4c29-4540-a2ae-8bddb45bbf5e" containerName="revoke-keys" Jul 13 17:59:44.563145 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.559541 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="638927bd-9cfb-4e2c-9998-eb47c42490ac" containerName="maas-api" Jul 13 17:59:44.563145 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.559568 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5da0444-4c29-4540-a2ae-8bddb45bbf5e" containerName="revoke-keys" Jul 13 17:59:44.565060 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.565033 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.567407 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.567385 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ee0971b4-serving-cert\"" Jul 13 17:59:44.573196 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.573174 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh"] Jul 13 17:59:44.716303 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.716207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-maas-api-tls\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.716303 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.716289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-openshift-service-ca\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.716495 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.716312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh5l8\" (UniqueName: \"kubernetes.io/projected/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-kube-api-access-gh5l8\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.741818 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.741773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:44.817399 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.817375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-maas-api-tls\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.817531 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.817442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-openshift-service-ca\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.817531 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.817465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh5l8\" (UniqueName: \"kubernetes.io/projected/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-kube-api-access-gh5l8\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.818129 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.818103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-openshift-service-ca\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.819709 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.819689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-maas-api-tls\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.825960 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.825934 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh5l8\" (UniqueName: \"kubernetes.io/projected/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-kube-api-access-gh5l8\") pod \"maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:44.842597 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.842571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:44.880703 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:44.880678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:45.027485 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:45.027450 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh"] Jul 13 17:59:45.029375 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:59:45.029350 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2a5930_7993_4b48_a262_f8a3cd3c3d2d.slice/crio-1a3973b234419aa548e341ecbb240b9ee92266086ff452812cca2d53d6c43292 WatchSource:0}: Error finding container 1a3973b234419aa548e341ecbb240b9ee92266086ff452812cca2d53d6c43292: Status 404 returned error can't find the container with id 1a3973b234419aa548e341ecbb240b9ee92266086ff452812cca2d53d6c43292 Jul 13 17:59:45.078779 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:45.078753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" event={"ID":"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d","Type":"ContainerStarted","Data":"1a3973b234419aa548e341ecbb240b9ee92266086ff452812cca2d53d6c43292"} Jul 13 17:59:45.091899 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:45.091870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:45.189470 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:45.189434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:46.085087 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:46.085044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" event={"ID":"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d","Type":"ContainerStarted","Data":"496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c"} Jul 13 17:59:46.085733 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:46.085157 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:46.102844 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:46.102794 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" podStartSLOduration=1.557277134 podStartE2EDuration="2.102776815s" totalImagesPullingTime="545.499681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 17:59:46.100588162 +0000 UTC m=+1641.107990434" watchObservedRunningTime="2026-07-13 17:59:46.102776815 +0000 UTC m=+1641.110179059" Jul 13 17:59:47.307137 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:47.307100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:47.442868 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:47.442828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:47.513647 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:47.513611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:49.511890 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.511852 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9"] Jul 13 17:59:49.516376 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.516349 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" Jul 13 17:59:49.518848 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.518808 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 17:59:49.519780 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.519756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9"] Jul 13 17:59:49.665190 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.665157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91dd2e3-4e8b-4340-ae2e-10c9f0563271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9\" (UID: \"b91dd2e3-4e8b-4340-ae2e-10c9f0563271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" Jul 13 17:59:49.766031 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.765961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91dd2e3-4e8b-4340-ae2e-10c9f0563271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9\" (UID: \"b91dd2e3-4e8b-4340-ae2e-10c9f0563271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" Jul 13 17:59:49.766621 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.766602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91dd2e3-4e8b-4340-ae2e-10c9f0563271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9\" (UID: \"b91dd2e3-4e8b-4340-ae2e-10c9f0563271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" Jul 13 17:59:49.829638 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.829608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" Jul 13 17:59:49.962145 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:49.962120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9"] Jul 13 17:59:49.964445 ip-10-0-139-16 kubenswrapper[2564]: W0713 17:59:49.964416 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91dd2e3_4e8b_4340_ae2e_10c9f0563271.slice/crio-a7e453f72f3d7f66db909b56bea1422f7e7f3b92381beff4e2154891189898c3 WatchSource:0}: Error finding container a7e453f72f3d7f66db909b56bea1422f7e7f3b92381beff4e2154891189898c3: Status 404 returned error can't find the container with id a7e453f72f3d7f66db909b56bea1422f7e7f3b92381beff4e2154891189898c3 Jul 13 17:59:50.107480 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:50.107446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" event={"ID":"b91dd2e3-4e8b-4340-ae2e-10c9f0563271","Type":"ContainerStarted","Data":"a7e453f72f3d7f66db909b56bea1422f7e7f3b92381beff4e2154891189898c3"} Jul 13 17:59:51.114770 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:51.114735 2564 generic.go:356] "Generic (PLEG): container finished" podID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerID="f3e530566f6835c2bb998f684b64d3be5bea344eb918c64e0fdeadb695ee8e0b" exitCode=7 Jul 13 17:59:51.115160 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:51.114826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" event={"ID":"b91dd2e3-4e8b-4340-ae2e-10c9f0563271","Type":"ContainerDied","Data":"f3e530566f6835c2bb998f684b64d3be5bea344eb918c64e0fdeadb695ee8e0b"} Jul 13 17:59:51.115160 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:51.115006 2564 scope.go:117] "RemoveContainer" containerID="f3e530566f6835c2bb998f684b64d3be5bea344eb918c64e0fdeadb695ee8e0b" Jul 13 17:59:52.095710 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:52.095682 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 17:59:52.124069 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:52.124037 2564 generic.go:356] "Generic (PLEG): container finished" podID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerID="7adb12599af2f20caf2854a54d06462b64ac7938e7ea22bc8e5bd57f90420114" exitCode=7 Jul 13 17:59:52.124511 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:52.124123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" event={"ID":"b91dd2e3-4e8b-4340-ae2e-10c9f0563271","Type":"ContainerDied","Data":"7adb12599af2f20caf2854a54d06462b64ac7938e7ea22bc8e5bd57f90420114"} Jul 13 17:59:52.124511 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:52.124207 2564 scope.go:117] "RemoveContainer" containerID="f3e530566f6835c2bb998f684b64d3be5bea344eb918c64e0fdeadb695ee8e0b" Jul 13 17:59:52.124511 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:52.124485 2564 scope.go:117] "RemoveContainer" containerID="7adb12599af2f20caf2854a54d06462b64ac7938e7ea22bc8e5bd57f90420114" Jul 13 17:59:52.124755 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:59:52.124732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9_odh-ai-gateway-infra(b91dd2e3-4e8b-4340-ae2e-10c9f0563271)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" Jul 13 17:59:53.131084 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:53.131052 2564 scope.go:117] "RemoveContainer" containerID="7adb12599af2f20caf2854a54d06462b64ac7938e7ea22bc8e5bd57f90420114" Jul 13 17:59:53.131530 ip-10-0-139-16 kubenswrapper[2564]: E0713 17:59:53.131322 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9_odh-ai-gateway-infra(b91dd2e3-4e8b-4340-ae2e-10c9f0563271)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" Jul 13 17:59:55.089885 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:55.089855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 17:59:55.191059 ip-10-0-139-16 kubenswrapper[2564]: I0713 17:59:55.191021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:00:00.128953 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.128921 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4"] Jul 13 18:00:00.132500 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.132484 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r"] Jul 13 18:00:00.132608 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.132590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" Jul 13 18:00:00.135784 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.135765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" Jul 13 18:00:00.141416 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.141306 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r"] Jul 13 18:00:00.143686 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.143663 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4"] Jul 13 18:00:00.146949 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.146922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" Jul 13 18:00:00.152801 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.152781 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" Jul 13 18:00:00.293274 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.293039 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4"] Jul 13 18:00:00.296002 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:00:00.295969 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80c131f_a5ce_4f30_b764_1dcd8100683b.slice/crio-0aedf5b9c7b1f2d92ed2e0e6240bddd450696ab8fdb4abdcba9398e142673675 WatchSource:0}: Error finding container 0aedf5b9c7b1f2d92ed2e0e6240bddd450696ab8fdb4abdcba9398e142673675: Status 404 returned error can't find the container with id 0aedf5b9c7b1f2d92ed2e0e6240bddd450696ab8fdb4abdcba9398e142673675 Jul 13 18:00:00.319046 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:00.319017 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r"] Jul 13 18:00:00.320271 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:00:00.320223 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41735e63_82a0_4687_91cd_f8b9ae4055c3.slice/crio-ad5b918e712a61a08d285c0eeeb45e49c5eb1a598612540a45317d8c335df579 WatchSource:0}: Error finding container ad5b918e712a61a08d285c0eeeb45e49c5eb1a598612540a45317d8c335df579: Status 404 returned error can't find the container with id ad5b918e712a61a08d285c0eeeb45e49c5eb1a598612540a45317d8c335df579 Jul 13 18:00:01.182074 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:01.182044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" event={"ID":"41735e63-82a0-4687-91cd-f8b9ae4055c3","Type":"ContainerStarted","Data":"0d8463adbb9ec31fb58614bdf6cb4591682bc70659db0115e016ed4d71d5d973"} Jul 13 18:00:01.182548 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:01.182105 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" event={"ID":"41735e63-82a0-4687-91cd-f8b9ae4055c3","Type":"ContainerStarted","Data":"ad5b918e712a61a08d285c0eeeb45e49c5eb1a598612540a45317d8c335df579"} Jul 13 18:00:01.183689 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:01.183667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" event={"ID":"b80c131f-a5ce-4f30-b764-1dcd8100683b","Type":"ContainerStarted","Data":"75f3eb800c7168a8d893e4d781bb47d681ce9aaa1ff1ff6c9b0b9ca5c11de042"} Jul 13 18:00:01.183804 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:01.183715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" event={"ID":"b80c131f-a5ce-4f30-b764-1dcd8100683b","Type":"ContainerStarted","Data":"0aedf5b9c7b1f2d92ed2e0e6240bddd450696ab8fdb4abdcba9398e142673675"} Jul 13 18:00:01.200870 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:01.200821 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" podStartSLOduration=1.200806758 podStartE2EDuration="1.200806758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:00:01.198838511 +0000 UTC m=+1656.206240756" watchObservedRunningTime="2026-07-13 18:00:01.200806758 +0000 UTC m=+1656.208209067" Jul 13 18:00:01.212380 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:01.212334 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" podStartSLOduration=1.212317246 podStartE2EDuration="1.212317246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:00:01.211744344 +0000 UTC m=+1656.219146586" watchObservedRunningTime="2026-07-13 18:00:01.212317246 +0000 UTC m=+1656.219719488" Jul 13 18:00:02.190458 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:02.190424 2564 generic.go:356] "Generic (PLEG): container finished" podID="b80c131f-a5ce-4f30-b764-1dcd8100683b" containerID="75f3eb800c7168a8d893e4d781bb47d681ce9aaa1ff1ff6c9b0b9ca5c11de042" exitCode=0 Jul 13 18:00:02.190864 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:02.190511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" event={"ID":"b80c131f-a5ce-4f30-b764-1dcd8100683b","Type":"ContainerDied","Data":"75f3eb800c7168a8d893e4d781bb47d681ce9aaa1ff1ff6c9b0b9ca5c11de042"} Jul 13 18:00:02.191995 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:02.191975 2564 generic.go:356] "Generic (PLEG): container finished" podID="41735e63-82a0-4687-91cd-f8b9ae4055c3" containerID="0d8463adbb9ec31fb58614bdf6cb4591682bc70659db0115e016ed4d71d5d973" exitCode=0 Jul 13 18:00:02.192070 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:02.192009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" event={"ID":"41735e63-82a0-4687-91cd-f8b9ae4055c3","Type":"ContainerDied","Data":"0d8463adbb9ec31fb58614bdf6cb4591682bc70659db0115e016ed4d71d5d973"} Jul 13 18:00:03.329409 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:03.329385 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" Jul 13 18:00:03.366967 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:03.366942 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" Jul 13 18:00:04.204396 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:04.204364 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" Jul 13 18:00:04.204574 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:04.204402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4" event={"ID":"b80c131f-a5ce-4f30-b764-1dcd8100683b","Type":"ContainerDied","Data":"0aedf5b9c7b1f2d92ed2e0e6240bddd450696ab8fdb4abdcba9398e142673675"} Jul 13 18:00:04.204574 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:04.204485 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aedf5b9c7b1f2d92ed2e0e6240bddd450696ab8fdb4abdcba9398e142673675" Jul 13 18:00:04.205684 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:04.205665 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" Jul 13 18:00:04.205815 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:04.205659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732760-xm76r" event={"ID":"41735e63-82a0-4687-91cd-f8b9ae4055c3","Type":"ContainerDied","Data":"ad5b918e712a61a08d285c0eeeb45e49c5eb1a598612540a45317d8c335df579"} Jul 13 18:00:04.205815 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:04.205779 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5b918e712a61a08d285c0eeeb45e49c5eb1a598612540a45317d8c335df579" Jul 13 18:00:06.632095 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:06.632067 2564 scope.go:117] "RemoveContainer" containerID="7adb12599af2f20caf2854a54d06462b64ac7938e7ea22bc8e5bd57f90420114" Jul 13 18:00:07.224298 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:07.224185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" event={"ID":"b91dd2e3-4e8b-4340-ae2e-10c9f0563271","Type":"ContainerStarted","Data":"ae9a710238549b85148ddf562221be5e552584650dc0009e236d8dca95519a80"} Jul 13 18:00:07.241085 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:07.241033 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" podStartSLOduration=18.241019183 podStartE2EDuration="18.241019183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 17:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:00:07.238840762 +0000 UTC m=+1662.246243024" watchObservedRunningTime="2026-07-13 18:00:07.241019183 +0000 UTC m=+1662.248421425" Jul 13 18:00:08.234553 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:08.234517 2564 generic.go:356] "Generic (PLEG): container finished" podID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerID="ae9a710238549b85148ddf562221be5e552584650dc0009e236d8dca95519a80" exitCode=0 Jul 13 18:00:08.235010 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:08.234591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" event={"ID":"b91dd2e3-4e8b-4340-ae2e-10c9f0563271","Type":"ContainerDied","Data":"ae9a710238549b85148ddf562221be5e552584650dc0009e236d8dca95519a80"} Jul 13 18:00:08.235010 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:08.234671 2564 scope.go:117] "RemoveContainer" containerID="7adb12599af2f20caf2854a54d06462b64ac7938e7ea22bc8e5bd57f90420114" Jul 13 18:00:08.250926 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:08.250900 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9"] Jul 13 18:00:09.381876 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:09.381854 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" Jul 13 18:00:09.448933 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:09.448900 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91dd2e3-4e8b-4340-ae2e-10c9f0563271-maas-api-service-ca\") pod \"b91dd2e3-4e8b-4340-ae2e-10c9f0563271\" (UID: \"b91dd2e3-4e8b-4340-ae2e-10c9f0563271\") " Jul 13 18:00:09.449271 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:09.449228 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91dd2e3-4e8b-4340-ae2e-10c9f0563271-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b91dd2e3-4e8b-4340-ae2e-10c9f0563271" (UID: "b91dd2e3-4e8b-4340-ae2e-10c9f0563271"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:00:09.550403 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:09.550378 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91dd2e3-4e8b-4340-ae2e-10c9f0563271-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:00:10.248713 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:10.248682 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" Jul 13 18:00:10.248915 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:10.248711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9" event={"ID":"b91dd2e3-4e8b-4340-ae2e-10c9f0563271","Type":"ContainerDied","Data":"a7e453f72f3d7f66db909b56bea1422f7e7f3b92381beff4e2154891189898c3"} Jul 13 18:00:10.248915 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:10.248754 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7e453f72f3d7f66db909b56bea1422f7e7f3b92381beff4e2154891189898c3" Jul 13 18:00:10.276702 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:10.276674 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9"] Jul 13 18:00:10.281318 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:10.281292 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-4ncg9"] Jul 13 18:00:10.904177 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:10.904136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:00:11.006390 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:11.006336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:00:11.639052 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:11.639018 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" path="/var/lib/kubelet/pods/b91dd2e3-4e8b-4340-ae2e-10c9f0563271/volumes" Jul 13 18:00:49.564085 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.563995 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484"] Jul 13 18:00:49.564709 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564690 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.564799 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564713 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.564799 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564726 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.564799 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564734 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.564799 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564752 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80c131f-a5ce-4f30-b764-1dcd8100683b" containerName="cleanup" Jul 13 18:00:49.564799 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564760 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80c131f-a5ce-4f30-b764-1dcd8100683b" containerName="cleanup" Jul 13 18:00:49.564799 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564781 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.564799 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564790 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.565139 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564812 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41735e63-82a0-4687-91cd-f8b9ae4055c3" containerName="cleanup" Jul 13 18:00:49.565139 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564822 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="41735e63-82a0-4687-91cd-f8b9ae4055c3" containerName="cleanup" Jul 13 18:00:49.565139 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564921 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.565139 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564934 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:00:49.565139 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564944 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80c131f-a5ce-4f30-b764-1dcd8100683b" containerName="cleanup" Jul 13 18:00:49.565139 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.564961 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="41735e63-82a0-4687-91cd-f8b9ae4055c3" containerName="cleanup" Jul 13 18:00:49.569815 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.569794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" Jul 13 18:00:49.572516 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.572490 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:00:49.575391 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.575368 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484"] Jul 13 18:00:49.644070 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.644039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73491f11-7890-4a1b-8e6d-65b4d34c4a6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee0971b4-cg484\" (UID: \"73491f11-7890-4a1b-8e6d-65b4d34c4a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" Jul 13 18:00:49.745355 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.745324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73491f11-7890-4a1b-8e6d-65b4d34c4a6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee0971b4-cg484\" (UID: \"73491f11-7890-4a1b-8e6d-65b4d34c4a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" Jul 13 18:00:49.745961 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.745944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73491f11-7890-4a1b-8e6d-65b4d34c4a6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee0971b4-cg484\" (UID: \"73491f11-7890-4a1b-8e6d-65b4d34c4a6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" Jul 13 18:00:49.882846 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:49.882764 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" Jul 13 18:00:50.020558 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:50.020532 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484"] Jul 13 18:00:50.022194 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:00:50.022165 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73491f11_7890_4a1b_8e6d_65b4d34c4a6e.slice/crio-d9872d8819a272d89ff80dca8c7f9f9d077c17b32d1c3f31e787beb18c534beb WatchSource:0}: Error finding container d9872d8819a272d89ff80dca8c7f9f9d077c17b32d1c3f31e787beb18c534beb: Status 404 returned error can't find the container with id d9872d8819a272d89ff80dca8c7f9f9d077c17b32d1c3f31e787beb18c534beb Jul 13 18:00:50.474034 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:50.473938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" event={"ID":"73491f11-7890-4a1b-8e6d-65b4d34c4a6e","Type":"ContainerStarted","Data":"14d18febe793b3eb58d51d5a16f08e46e9136c166a7b416f13164704922434bf"} Jul 13 18:00:50.474034 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:50.473995 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" event={"ID":"73491f11-7890-4a1b-8e6d-65b4d34c4a6e","Type":"ContainerStarted","Data":"d9872d8819a272d89ff80dca8c7f9f9d077c17b32d1c3f31e787beb18c534beb"} Jul 13 18:00:50.492376 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:50.492326 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" podStartSLOduration=1.492312351 podStartE2EDuration="1.492312351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:00:50.490619172 +0000 UTC m=+1705.498021448" watchObservedRunningTime="2026-07-13 18:00:50.492312351 +0000 UTC m=+1705.499714593" Jul 13 18:00:51.488184 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:51.488146 2564 generic.go:356] "Generic (PLEG): container finished" podID="73491f11-7890-4a1b-8e6d-65b4d34c4a6e" containerID="14d18febe793b3eb58d51d5a16f08e46e9136c166a7b416f13164704922434bf" exitCode=0 Jul 13 18:00:51.488623 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:51.488203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" event={"ID":"73491f11-7890-4a1b-8e6d-65b4d34c4a6e","Type":"ContainerDied","Data":"14d18febe793b3eb58d51d5a16f08e46e9136c166a7b416f13164704922434bf"} Jul 13 18:00:52.636523 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:52.636500 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" Jul 13 18:00:52.769868 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:52.769801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73491f11-7890-4a1b-8e6d-65b4d34c4a6e-maas-api-service-ca\") pod \"73491f11-7890-4a1b-8e6d-65b4d34c4a6e\" (UID: \"73491f11-7890-4a1b-8e6d-65b4d34c4a6e\") " Jul 13 18:00:52.770172 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:52.770151 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73491f11-7890-4a1b-8e6d-65b4d34c4a6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73491f11-7890-4a1b-8e6d-65b4d34c4a6e" (UID: "73491f11-7890-4a1b-8e6d-65b4d34c4a6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:00:52.871311 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:52.871282 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73491f11-7890-4a1b-8e6d-65b4d34c4a6e-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:00:53.500979 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:53.500949 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" Jul 13 18:00:53.501117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:53.500993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee0971b4-cg484" event={"ID":"73491f11-7890-4a1b-8e6d-65b4d34c4a6e","Type":"ContainerDied","Data":"d9872d8819a272d89ff80dca8c7f9f9d077c17b32d1c3f31e787beb18c534beb"} Jul 13 18:00:53.501117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:00:53.501038 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9872d8819a272d89ff80dca8c7f9f9d077c17b32d1c3f31e787beb18c534beb" Jul 13 18:01:00.145469 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.145432 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh"] Jul 13 18:01:00.145948 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.145694 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" podUID="7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" containerName="maas-api" containerID="cri-o://496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c" gracePeriod=30 Jul 13 18:01:00.174533 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.174496 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4"] Jul 13 18:01:00.179179 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.179157 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ee0971b4-29732760-qkvl4"] Jul 13 18:01:00.184446 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.184422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:00.400514 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.400459 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 18:01:00.540631 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540604 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-openshift-service-ca\") pod \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " Jul 13 18:01:00.540805 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540641 2564 generic.go:356] "Generic (PLEG): container finished" podID="7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" containerID="496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c" exitCode=0 Jul 13 18:01:00.540805 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540724 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" Jul 13 18:01:00.540805 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" event={"ID":"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d","Type":"ContainerDied","Data":"496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c"} Jul 13 18:01:00.540805 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540772 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh" event={"ID":"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d","Type":"ContainerDied","Data":"1a3973b234419aa548e341ecbb240b9ee92266086ff452812cca2d53d6c43292"} Jul 13 18:01:00.540805 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540778 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh5l8\" (UniqueName: \"kubernetes.io/projected/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-kube-api-access-gh5l8\") pod \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " Jul 13 18:01:00.541063 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540816 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-maas-api-tls\") pod \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\" (UID: \"7d2a5930-7993-4b48-a262-f8a3cd3c3d2d\") " Jul 13 18:01:00.541063 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540796 2564 scope.go:117] "RemoveContainer" containerID="496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c" Jul 13 18:01:00.541063 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.540985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" (UID: "7d2a5930-7993-4b48-a262-f8a3cd3c3d2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:01:00.541206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.541109 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:01:00.542993 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.542960 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-kube-api-access-gh5l8" (OuterVolumeSpecName: "kube-api-access-gh5l8") pod "7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" (UID: "7d2a5930-7993-4b48-a262-f8a3cd3c3d2d"). InnerVolumeSpecName "kube-api-access-gh5l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:01:00.542993 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.542986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" (UID: "7d2a5930-7993-4b48-a262-f8a3cd3c3d2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:01:00.566521 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.566504 2564 scope.go:117] "RemoveContainer" containerID="496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c" Jul 13 18:01:00.566788 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:01:00.566763 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c\": container with ID starting with 496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c not found: ID does not exist" containerID="496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c" Jul 13 18:01:00.566825 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.566797 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c"} err="failed to get container status \"496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c\": rpc error: code = NotFound desc = could not find container \"496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c\": container with ID starting with 496f9739e2832db6b09fc11f9e2e48af39d8ba68ee3671d423ad361509821f6c not found: ID does not exist" Jul 13 18:01:00.642028 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.642000 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh5l8\" (UniqueName: \"kubernetes.io/projected/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-kube-api-access-gh5l8\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:01:00.642028 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.642024 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:01:00.862932 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.862903 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh"] Jul 13 18:01:00.866624 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:00.866603 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ee0971b4-5bb79669b-zbvnh"] Jul 13 18:01:01.638407 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:01.638375 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" path="/var/lib/kubelet/pods/7d2a5930-7993-4b48-a262-f8a3cd3c3d2d/volumes" Jul 13 18:01:01.638770 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:01.638694 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b80c131f-a5ce-4f30-b764-1dcd8100683b" path="/var/lib/kubelet/pods/b80c131f-a5ce-4f30-b764-1dcd8100683b/volumes" Jul 13 18:01:04.650150 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:04.650122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:10.797731 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:10.797697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:10.934581 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:10.934552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:15.734295 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:15.734261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:16.080456 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:16.080427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:17.161426 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.161390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:17.239681 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.239651 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw"] Jul 13 18:01:17.240107 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.240095 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73491f11-7890-4a1b-8e6d-65b4d34c4a6e" containerName="revoke-keys" Jul 13 18:01:17.240149 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.240108 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="73491f11-7890-4a1b-8e6d-65b4d34c4a6e" containerName="revoke-keys" Jul 13 18:01:17.240149 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.240127 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" containerName="maas-api" Jul 13 18:01:17.240149 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.240132 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" containerName="maas-api" Jul 13 18:01:17.240266 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.240198 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="73491f11-7890-4a1b-8e6d-65b4d34c4a6e" containerName="revoke-keys" Jul 13 18:01:17.240266 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.240207 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91dd2e3-4e8b-4340-ae2e-10c9f0563271" containerName="revoke-keys" Jul 13 18:01:17.240266 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.240217 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2a5930-7993-4b48-a262-f8a3cd3c3d2d" containerName="maas-api" Jul 13 18:01:17.246178 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.246151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.248576 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.248557 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-338f4926-serving-cert\"" Jul 13 18:01:17.255174 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.255149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw"] Jul 13 18:01:17.257288 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.257262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:17.300356 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.300325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee881572-0827-4494-9c1d-629acfeb36a4-maas-api-tls\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.300501 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.300382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qsgt\" (UniqueName: \"kubernetes.io/projected/ee881572-0827-4494-9c1d-629acfeb36a4-kube-api-access-9qsgt\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.300566 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.300526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee881572-0827-4494-9c1d-629acfeb36a4-openshift-service-ca\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.321668 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.321637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:17.401552 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.401511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qsgt\" (UniqueName: \"kubernetes.io/projected/ee881572-0827-4494-9c1d-629acfeb36a4-kube-api-access-9qsgt\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.401762 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.401638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee881572-0827-4494-9c1d-629acfeb36a4-openshift-service-ca\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.401762 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.401692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee881572-0827-4494-9c1d-629acfeb36a4-maas-api-tls\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.402486 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.402455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee881572-0827-4494-9c1d-629acfeb36a4-openshift-service-ca\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.404342 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.404318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee881572-0827-4494-9c1d-629acfeb36a4-maas-api-tls\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.409622 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.409592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qsgt\" (UniqueName: \"kubernetes.io/projected/ee881572-0827-4494-9c1d-629acfeb36a4-kube-api-access-9qsgt\") pod \"maas-api-e2e-derive-338f4926-79cd9746f9-psvdw\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.420221 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.420165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:17.559188 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.559154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:17.699152 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.699129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw"] Jul 13 18:01:17.701296 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:01:17.701265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee881572_0827_4494_9c1d_629acfeb36a4.slice/crio-6e69720ddb38fcdd6154c8bfd136ca332b29ac8de5f69cf7f87e44a22e765193 WatchSource:0}: Error finding container 6e69720ddb38fcdd6154c8bfd136ca332b29ac8de5f69cf7f87e44a22e765193: Status 404 returned error can't find the container with id 6e69720ddb38fcdd6154c8bfd136ca332b29ac8de5f69cf7f87e44a22e765193 Jul 13 18:01:17.830615 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.830584 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:17.918690 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:17.918652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:18.637638 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:18.637598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" event={"ID":"ee881572-0827-4494-9c1d-629acfeb36a4","Type":"ContainerStarted","Data":"6e69720ddb38fcdd6154c8bfd136ca332b29ac8de5f69cf7f87e44a22e765193"} Jul 13 18:01:19.643741 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:19.643708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" event={"ID":"ee881572-0827-4494-9c1d-629acfeb36a4","Type":"ContainerStarted","Data":"90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3"} Jul 13 18:01:19.644186 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:19.643840 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:19.661036 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:19.660992 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" podStartSLOduration=0.910992098 podStartE2EDuration="2.660979665s" totalImagesPullingTime="1.749987567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:01:19.658617628 +0000 UTC m=+1734.666019896" watchObservedRunningTime="2026-07-13 18:01:19.660979665 +0000 UTC m=+1734.668381908" Jul 13 18:01:21.583188 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.583152 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9"] Jul 13 18:01:21.586967 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.586939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" Jul 13 18:01:21.589039 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.589018 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:01:21.593960 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.593940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9"] Jul 13 18:01:21.640527 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.640504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e92a50-71ae-4605-aff3-511c39b08ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-338f4926-tqng9\" (UID: \"b9e92a50-71ae-4605-aff3-511c39b08ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" Jul 13 18:01:21.741202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.741179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e92a50-71ae-4605-aff3-511c39b08ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-338f4926-tqng9\" (UID: \"b9e92a50-71ae-4605-aff3-511c39b08ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" Jul 13 18:01:21.741812 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.741793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e92a50-71ae-4605-aff3-511c39b08ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-338f4926-tqng9\" (UID: \"b9e92a50-71ae-4605-aff3-511c39b08ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" Jul 13 18:01:21.899906 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.899813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" Jul 13 18:01:21.987202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:21.987152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:22.050315 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:22.050283 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9"] Jul 13 18:01:22.051890 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:22.051851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:22.053740 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:01:22.053716 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e92a50_71ae_4605_aff3_511c39b08ba4.slice/crio-b1dcc04609da0fe5ad97f5ce9f665b10ba6abf52187f0b04bd6411fc20ca169e WatchSource:0}: Error finding container b1dcc04609da0fe5ad97f5ce9f665b10ba6abf52187f0b04bd6411fc20ca169e: Status 404 returned error can't find the container with id b1dcc04609da0fe5ad97f5ce9f665b10ba6abf52187f0b04bd6411fc20ca169e Jul 13 18:01:22.660881 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:22.660838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" event={"ID":"b9e92a50-71ae-4605-aff3-511c39b08ba4","Type":"ContainerStarted","Data":"fd2fdb8b6a637257ea2d220897d517fa3adaf1a1aa711b59483ee88ea5214b0c"} Jul 13 18:01:22.661272 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:22.660901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" event={"ID":"b9e92a50-71ae-4605-aff3-511c39b08ba4","Type":"ContainerStarted","Data":"b1dcc04609da0fe5ad97f5ce9f665b10ba6abf52187f0b04bd6411fc20ca169e"} Jul 13 18:01:22.681162 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:22.681122 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" podStartSLOduration=1.681109178 podStartE2EDuration="1.681109178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:01:22.678802515 +0000 UTC m=+1737.686204779" watchObservedRunningTime="2026-07-13 18:01:22.681109178 +0000 UTC m=+1737.688511420" Jul 13 18:01:23.667164 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:23.667124 2564 generic.go:356] "Generic (PLEG): container finished" podID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerID="fd2fdb8b6a637257ea2d220897d517fa3adaf1a1aa711b59483ee88ea5214b0c" exitCode=7 Jul 13 18:01:23.667582 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:23.667204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" event={"ID":"b9e92a50-71ae-4605-aff3-511c39b08ba4","Type":"ContainerDied","Data":"fd2fdb8b6a637257ea2d220897d517fa3adaf1a1aa711b59483ee88ea5214b0c"} Jul 13 18:01:23.667582 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:23.667387 2564 scope.go:117] "RemoveContainer" containerID="fd2fdb8b6a637257ea2d220897d517fa3adaf1a1aa711b59483ee88ea5214b0c" Jul 13 18:01:24.674031 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:24.673994 2564 generic.go:356] "Generic (PLEG): container finished" podID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerID="6762257affc1b613d39ff6e121b672b28888cef4dd332dc6b17b2759a0b39ac7" exitCode=7 Jul 13 18:01:24.674457 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:24.674079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" event={"ID":"b9e92a50-71ae-4605-aff3-511c39b08ba4","Type":"ContainerDied","Data":"6762257affc1b613d39ff6e121b672b28888cef4dd332dc6b17b2759a0b39ac7"} Jul 13 18:01:24.674457 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:24.674158 2564 scope.go:117] "RemoveContainer" containerID="fd2fdb8b6a637257ea2d220897d517fa3adaf1a1aa711b59483ee88ea5214b0c" Jul 13 18:01:24.674457 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:24.674268 2564 scope.go:117] "RemoveContainer" containerID="6762257affc1b613d39ff6e121b672b28888cef4dd332dc6b17b2759a0b39ac7" Jul 13 18:01:24.674632 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:01:24.674453 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-338f4926-tqng9_odh-ai-gateway-infra(b9e92a50-71ae-4605-aff3-511c39b08ba4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" Jul 13 18:01:25.494270 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:25.494215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:01:25.654331 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:25.654296 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:01:25.681201 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:25.681172 2564 scope.go:117] "RemoveContainer" containerID="6762257affc1b613d39ff6e121b672b28888cef4dd332dc6b17b2759a0b39ac7" Jul 13 18:01:25.681560 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:01:25.681443 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-338f4926-tqng9_odh-ai-gateway-infra(b9e92a50-71ae-4605-aff3-511c39b08ba4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" Jul 13 18:01:37.632344 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:37.632311 2564 scope.go:117] "RemoveContainer" containerID="6762257affc1b613d39ff6e121b672b28888cef4dd332dc6b17b2759a0b39ac7" Jul 13 18:01:38.757614 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:38.757532 2564 generic.go:356] "Generic (PLEG): container finished" podID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerID="492170cd5895f350920e31d9a6ecbf024a9633216ee860f0cbb063ad12f03404" exitCode=0 Jul 13 18:01:38.757982 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:38.757611 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" event={"ID":"b9e92a50-71ae-4605-aff3-511c39b08ba4","Type":"ContainerDied","Data":"492170cd5895f350920e31d9a6ecbf024a9633216ee860f0cbb063ad12f03404"} Jul 13 18:01:38.757982 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:38.757674 2564 scope.go:117] "RemoveContainer" containerID="6762257affc1b613d39ff6e121b672b28888cef4dd332dc6b17b2759a0b39ac7" Jul 13 18:01:39.788568 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:39.788538 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9"] Jul 13 18:01:39.900601 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:39.900570 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" Jul 13 18:01:40.011888 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.011852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e92a50-71ae-4605-aff3-511c39b08ba4-maas-api-service-ca\") pod \"b9e92a50-71ae-4605-aff3-511c39b08ba4\" (UID: \"b9e92a50-71ae-4605-aff3-511c39b08ba4\") " Jul 13 18:01:40.012204 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.012182 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e92a50-71ae-4605-aff3-511c39b08ba4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9e92a50-71ae-4605-aff3-511c39b08ba4" (UID: "b9e92a50-71ae-4605-aff3-511c39b08ba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:01:40.113555 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.113527 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e92a50-71ae-4605-aff3-511c39b08ba4-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:01:40.771529 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.771501 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" Jul 13 18:01:40.771702 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.771500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9" event={"ID":"b9e92a50-71ae-4605-aff3-511c39b08ba4","Type":"ContainerDied","Data":"b1dcc04609da0fe5ad97f5ce9f665b10ba6abf52187f0b04bd6411fc20ca169e"} Jul 13 18:01:40.771702 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.771637 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1dcc04609da0fe5ad97f5ce9f665b10ba6abf52187f0b04bd6411fc20ca169e" Jul 13 18:01:40.799871 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.799845 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9"] Jul 13 18:01:40.805981 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:40.805957 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-tqng9"] Jul 13 18:01:41.638436 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:01:41.638407 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" path="/var/lib/kubelet/pods/b9e92a50-71ae-4605-aff3-511c39b08ba4/volumes" Jul 13 18:02:01.861654 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:01.861615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:11.624661 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.624629 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc"] Jul 13 18:02:11.625054 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625042 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625099 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625056 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625099 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625071 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625099 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625077 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625099 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625095 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625226 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625101 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625226 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625180 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625226 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625188 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.625226 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.625195 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e92a50-71ae-4605-aff3-511c39b08ba4" containerName="revoke-keys" Jul 13 18:02:11.627944 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.627927 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" Jul 13 18:02:11.629859 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.629837 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:02:11.651140 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.651111 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc"] Jul 13 18:02:11.710603 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.710574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097c49ab-73bd-4c8d-b052-2ced66183f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-338f4926-wwtmc\" (UID: \"097c49ab-73bd-4c8d-b052-2ced66183f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" Jul 13 18:02:11.811595 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.811563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097c49ab-73bd-4c8d-b052-2ced66183f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-338f4926-wwtmc\" (UID: \"097c49ab-73bd-4c8d-b052-2ced66183f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" Jul 13 18:02:11.812145 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.812127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097c49ab-73bd-4c8d-b052-2ced66183f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-338f4926-wwtmc\" (UID: \"097c49ab-73bd-4c8d-b052-2ced66183f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" Jul 13 18:02:11.950374 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:11.950305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" Jul 13 18:02:12.079228 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:12.079203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc"] Jul 13 18:02:12.081408 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:02:12.081363 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097c49ab_73bd_4c8d_b052_2ced66183f43.slice/crio-721ba0e5835c60326a754fd213a984fb9aee144d1d844409384ff6c8e009d904 WatchSource:0}: Error finding container 721ba0e5835c60326a754fd213a984fb9aee144d1d844409384ff6c8e009d904: Status 404 returned error can't find the container with id 721ba0e5835c60326a754fd213a984fb9aee144d1d844409384ff6c8e009d904 Jul 13 18:02:12.952897 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:12.952863 2564 generic.go:356] "Generic (PLEG): container finished" podID="097c49ab-73bd-4c8d-b052-2ced66183f43" containerID="9f9e51b335e313f484907a8da7cdb8299b3440123f2c7504ab7e5208905d7166" exitCode=0 Jul 13 18:02:12.953228 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:12.952953 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" event={"ID":"097c49ab-73bd-4c8d-b052-2ced66183f43","Type":"ContainerDied","Data":"9f9e51b335e313f484907a8da7cdb8299b3440123f2c7504ab7e5208905d7166"} Jul 13 18:02:12.953228 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:12.952996 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" event={"ID":"097c49ab-73bd-4c8d-b052-2ced66183f43","Type":"ContainerStarted","Data":"721ba0e5835c60326a754fd213a984fb9aee144d1d844409384ff6c8e009d904"} Jul 13 18:02:14.112036 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:14.112008 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" Jul 13 18:02:14.251591 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:14.251523 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097c49ab-73bd-4c8d-b052-2ced66183f43-maas-api-service-ca\") pod \"097c49ab-73bd-4c8d-b052-2ced66183f43\" (UID: \"097c49ab-73bd-4c8d-b052-2ced66183f43\") " Jul 13 18:02:14.251844 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:14.251822 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097c49ab-73bd-4c8d-b052-2ced66183f43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "097c49ab-73bd-4c8d-b052-2ced66183f43" (UID: "097c49ab-73bd-4c8d-b052-2ced66183f43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:02:14.352384 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:14.352357 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097c49ab-73bd-4c8d-b052-2ced66183f43-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:02:14.965556 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:14.965527 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" Jul 13 18:02:14.965700 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:14.965567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-338f4926-wwtmc" event={"ID":"097c49ab-73bd-4c8d-b052-2ced66183f43","Type":"ContainerDied","Data":"721ba0e5835c60326a754fd213a984fb9aee144d1d844409384ff6c8e009d904"} Jul 13 18:02:14.965700 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:14.965610 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721ba0e5835c60326a754fd213a984fb9aee144d1d844409384ff6c8e009d904" Jul 13 18:02:22.170787 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.170736 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw"] Jul 13 18:02:22.171287 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.171013 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" podUID="ee881572-0827-4494-9c1d-629acfeb36a4" containerName="maas-api" containerID="cri-o://90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3" gracePeriod=30 Jul 13 18:02:22.205731 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.205704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:22.425745 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.425685 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:02:22.532629 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.532594 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee881572-0827-4494-9c1d-629acfeb36a4-maas-api-tls\") pod \"ee881572-0827-4494-9c1d-629acfeb36a4\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " Jul 13 18:02:22.532821 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.532717 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qsgt\" (UniqueName: \"kubernetes.io/projected/ee881572-0827-4494-9c1d-629acfeb36a4-kube-api-access-9qsgt\") pod \"ee881572-0827-4494-9c1d-629acfeb36a4\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " Jul 13 18:02:22.532821 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.532753 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee881572-0827-4494-9c1d-629acfeb36a4-openshift-service-ca\") pod \"ee881572-0827-4494-9c1d-629acfeb36a4\" (UID: \"ee881572-0827-4494-9c1d-629acfeb36a4\") " Jul 13 18:02:22.533134 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.533110 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee881572-0827-4494-9c1d-629acfeb36a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee881572-0827-4494-9c1d-629acfeb36a4" (UID: "ee881572-0827-4494-9c1d-629acfeb36a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:02:22.534698 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.534672 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee881572-0827-4494-9c1d-629acfeb36a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee881572-0827-4494-9c1d-629acfeb36a4" (UID: "ee881572-0827-4494-9c1d-629acfeb36a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:02:22.534802 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.534703 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee881572-0827-4494-9c1d-629acfeb36a4-kube-api-access-9qsgt" (OuterVolumeSpecName: "kube-api-access-9qsgt") pod "ee881572-0827-4494-9c1d-629acfeb36a4" (UID: "ee881572-0827-4494-9c1d-629acfeb36a4"). InnerVolumeSpecName "kube-api-access-9qsgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:02:22.634583 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.634546 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee881572-0827-4494-9c1d-629acfeb36a4-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:02:22.634583 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.634578 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qsgt\" (UniqueName: \"kubernetes.io/projected/ee881572-0827-4494-9c1d-629acfeb36a4-kube-api-access-9qsgt\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:02:22.634583 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:22.634589 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee881572-0827-4494-9c1d-629acfeb36a4-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:02:23.010274 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.010224 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee881572-0827-4494-9c1d-629acfeb36a4" containerID="90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3" exitCode=0 Jul 13 18:02:23.010435 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.010287 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" Jul 13 18:02:23.010435 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.010270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" event={"ID":"ee881572-0827-4494-9c1d-629acfeb36a4","Type":"ContainerDied","Data":"90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3"} Jul 13 18:02:23.010435 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.010419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw" event={"ID":"ee881572-0827-4494-9c1d-629acfeb36a4","Type":"ContainerDied","Data":"6e69720ddb38fcdd6154c8bfd136ca332b29ac8de5f69cf7f87e44a22e765193"} Jul 13 18:02:23.010551 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.010449 2564 scope.go:117] "RemoveContainer" containerID="90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3" Jul 13 18:02:23.022709 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.022691 2564 scope.go:117] "RemoveContainer" containerID="90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3" Jul 13 18:02:23.022968 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:02:23.022942 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3\": container with ID starting with 90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3 not found: ID does not exist" containerID="90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3" Jul 13 18:02:23.023036 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.022977 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3"} err="failed to get container status \"90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3\": rpc error: code = NotFound desc = could not find container \"90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3\": container with ID starting with 90f3825415f4f8da76b92356bacc691e27c3c93bfdf1b5a68b86358e763749f3 not found: ID does not exist" Jul 13 18:02:23.032024 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.032000 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw"] Jul 13 18:02:23.034431 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.034408 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-338f4926-79cd9746f9-psvdw"] Jul 13 18:02:23.638104 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:23.638067 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee881572-0827-4494-9c1d-629acfeb36a4" path="/var/lib/kubelet/pods/ee881572-0827-4494-9c1d-629acfeb36a4/volumes" Jul 13 18:02:25.629810 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:25.629717 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:02:25.654167 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:25.640035 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:02:26.731428 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:26.731389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:32.616019 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:32.615986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:32.715664 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:32.715628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:36.922958 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:36.922917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:37.533513 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:37.533474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:37.624220 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:37.624182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:37.724974 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:37.724944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:39.241982 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:39.241941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:43.373174 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:43.373137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:45.133973 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:45.133934 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:45.240549 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:45.240514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:45.335874 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:45.335843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:45.434314 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:45.434195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:02:45.865907 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:02:45.865871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:11.857984 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:11.857928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:18.224993 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:18.224956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:18.331827 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:18.331784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:18.431761 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:18.431727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:26.102273 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:26.102215 2564 scope.go:117] "RemoveContainer" containerID="b2b84cb7bb41aaf31ad564ef600ae8f1a4b2dc449b7fd770be4b9cb0c454bd58" Jul 13 18:03:34.368124 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:34.368080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:40.271404 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:40.271357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:40.380056 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:40.380009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:40.469803 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:40.469768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:51.073736 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:51.073655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:51.954547 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:51.954510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:58.072122 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:58.072086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:58.171935 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:58.171899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:58.275027 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:58.274967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:58.374539 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:58.374411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:58.478122 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:58.478083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:03:59.346297 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:03:59.346254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:00.135613 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:00.135565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:00.197886 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:00.197853 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:01.256181 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:01.256131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:09.062591 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:09.062531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:09.121274 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:09.121197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:14.428983 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:14.428946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:14.529337 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:14.529285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:14.701223 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:14.701112 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:15.028398 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:15.028301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:15.416965 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:15.416929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:17.858624 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.858563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:17.943277 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.943206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:17.944706 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.944681 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c"] Jul 13 18:04:17.945158 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.945143 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee881572-0827-4494-9c1d-629acfeb36a4" containerName="maas-api" Jul 13 18:04:17.945158 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.945159 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee881572-0827-4494-9c1d-629acfeb36a4" containerName="maas-api" Jul 13 18:04:17.945296 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.945175 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097c49ab-73bd-4c8d-b052-2ced66183f43" containerName="revoke-keys" Jul 13 18:04:17.945296 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.945180 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="097c49ab-73bd-4c8d-b052-2ced66183f43" containerName="revoke-keys" Jul 13 18:04:17.945296 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.945268 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee881572-0827-4494-9c1d-629acfeb36a4" containerName="maas-api" Jul 13 18:04:17.945296 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.945279 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="097c49ab-73bd-4c8d-b052-2ced66183f43" containerName="revoke-keys" Jul 13 18:04:17.948398 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.948381 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:17.950632 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.950611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3bf93f7b-serving-cert\"" Jul 13 18:04:17.957386 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.956889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c"] Jul 13 18:04:17.963565 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.963539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwsp\" (UniqueName: \"kubernetes.io/projected/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-kube-api-access-6rwsp\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:17.963696 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.963610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:17.963696 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:17.963673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.026787 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.026757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:18.064198 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.064168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.064198 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.064201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.064422 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.064287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwsp\" (UniqueName: \"kubernetes.io/projected/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-kube-api-access-6rwsp\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.064422 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:04:18.064324 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-serving-cert: secret "maas-api-e2e-mt-3bf93f7b-serving-cert" not found Jul 13 18:04:18.064422 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:04:18.064409 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls podName:9a2ab4c2-6137-40ba-bca1-38e7aa32d81f nodeName:}" failed. No retries permitted until 2026-07-13 18:04:18.564388305 +0000 UTC m=+1913.571790551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls") pod "maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" (UID: "9a2ab4c2-6137-40ba-bca1-38e7aa32d81f") : secret "maas-api-e2e-mt-3bf93f7b-serving-cert" not found Jul 13 18:04:18.064997 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.064975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-openshift-service-ca\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.075993 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.075964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwsp\" (UniqueName: \"kubernetes.io/projected/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-kube-api-access-6rwsp\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.157506 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.157418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:18.524914 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.524827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:18.568861 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.568826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.571337 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.571311 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls\") pod \"maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.687969 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.687931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:18.861634 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.861587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:18.999443 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:18.999206 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c"] Jul 13 18:04:19.001840 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:04:19.001809 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2ab4c2_6137_40ba_bca1_38e7aa32d81f.slice/crio-5e48d9665d1774bee9058a830c176fbabb18577b7967429389c67a71f082b642 WatchSource:0}: Error finding container 5e48d9665d1774bee9058a830c176fbabb18577b7967429389c67a71f082b642: Status 404 returned error can't find the container with id 5e48d9665d1774bee9058a830c176fbabb18577b7967429389c67a71f082b642 Jul 13 18:04:19.003165 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:19.003148 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 18:04:19.654481 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:19.654446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" event={"ID":"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f","Type":"ContainerStarted","Data":"5e48d9665d1774bee9058a830c176fbabb18577b7967429389c67a71f082b642"} Jul 13 18:04:20.661766 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:20.661665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" event={"ID":"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f","Type":"ContainerStarted","Data":"6b94d054289bf7e5b2d4bcbd85d6b04e1e112066dfc2fbd2d0e0deff0a8ea39a"} Jul 13 18:04:20.661766 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:20.661756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:20.681225 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:20.681175 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" podStartSLOduration=2.339542598 podStartE2EDuration="3.681158814s" totalImagesPullingTime="1.341616216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:04:20.679041773 +0000 UTC m=+1915.686444018" watchObservedRunningTime="2026-07-13 18:04:20.681158814 +0000 UTC m=+1915.688561101" Jul 13 18:04:22.388286 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:22.388216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:23.049279 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:23.049219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:23.152968 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:23.152920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:23.518518 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:23.518427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:26.673309 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:26.673279 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:29.602206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:29.602173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:29.704419 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:29.704385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:29.975948 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:29.975874 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv"] Jul 13 18:04:29.980325 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:29.980302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" Jul 13 18:04:29.982568 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:29.982540 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:04:29.986108 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:29.986081 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv"] Jul 13 18:04:30.081068 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.081026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff04689-9cec-4783-b23a-f34944ccbd76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv\" (UID: \"2ff04689-9cec-4783-b23a-f34944ccbd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" Jul 13 18:04:30.182463 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.182424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff04689-9cec-4783-b23a-f34944ccbd76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv\" (UID: \"2ff04689-9cec-4783-b23a-f34944ccbd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" Jul 13 18:04:30.183076 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.183055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff04689-9cec-4783-b23a-f34944ccbd76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv\" (UID: \"2ff04689-9cec-4783-b23a-f34944ccbd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" Jul 13 18:04:30.293153 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.293116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" Jul 13 18:04:30.428514 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.428486 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv"] Jul 13 18:04:30.430028 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:04:30.430002 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff04689_9cec_4783_b23a_f34944ccbd76.slice/crio-93ed44c19b13d97ada32f3acbc9eed2806bb52f96bf95bdfe70e9b7d508da0e9 WatchSource:0}: Error finding container 93ed44c19b13d97ada32f3acbc9eed2806bb52f96bf95bdfe70e9b7d508da0e9: Status 404 returned error can't find the container with id 93ed44c19b13d97ada32f3acbc9eed2806bb52f96bf95bdfe70e9b7d508da0e9 Jul 13 18:04:30.721489 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.721450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" event={"ID":"2ff04689-9cec-4783-b23a-f34944ccbd76","Type":"ContainerStarted","Data":"d8f4c12757d868549b79c3bad1a65384c28ed154faba4e23bb6071d57ee14074"} Jul 13 18:04:30.721882 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.721515 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" event={"ID":"2ff04689-9cec-4783-b23a-f34944ccbd76","Type":"ContainerStarted","Data":"93ed44c19b13d97ada32f3acbc9eed2806bb52f96bf95bdfe70e9b7d508da0e9"} Jul 13 18:04:30.735689 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:30.735637 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" podStartSLOduration=1.7356163489999998 podStartE2EDuration="1.735616349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:04:30.734533308 +0000 UTC m=+1925.741935549" watchObservedRunningTime="2026-07-13 18:04:30.735616349 +0000 UTC m=+1925.743018582" Jul 13 18:04:31.728804 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:31.728766 2564 generic.go:356] "Generic (PLEG): container finished" podID="2ff04689-9cec-4783-b23a-f34944ccbd76" containerID="d8f4c12757d868549b79c3bad1a65384c28ed154faba4e23bb6071d57ee14074" exitCode=0 Jul 13 18:04:31.729202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:31.728810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" event={"ID":"2ff04689-9cec-4783-b23a-f34944ccbd76","Type":"ContainerDied","Data":"d8f4c12757d868549b79c3bad1a65384c28ed154faba4e23bb6071d57ee14074"} Jul 13 18:04:32.877697 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:32.877669 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" Jul 13 18:04:33.016875 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:33.016780 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff04689-9cec-4783-b23a-f34944ccbd76-maas-api-service-ca\") pod \"2ff04689-9cec-4783-b23a-f34944ccbd76\" (UID: \"2ff04689-9cec-4783-b23a-f34944ccbd76\") " Jul 13 18:04:33.017117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:33.017095 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff04689-9cec-4783-b23a-f34944ccbd76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ff04689-9cec-4783-b23a-f34944ccbd76" (UID: "2ff04689-9cec-4783-b23a-f34944ccbd76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:04:33.117962 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:33.117922 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff04689-9cec-4783-b23a-f34944ccbd76-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:04:33.740182 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:33.740149 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" Jul 13 18:04:33.740373 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:33.740182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv" event={"ID":"2ff04689-9cec-4783-b23a-f34944ccbd76","Type":"ContainerDied","Data":"93ed44c19b13d97ada32f3acbc9eed2806bb52f96bf95bdfe70e9b7d508da0e9"} Jul 13 18:04:33.740373 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:33.740228 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ed44c19b13d97ada32f3acbc9eed2806bb52f96bf95bdfe70e9b7d508da0e9" Jul 13 18:04:40.527132 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.527096 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c"] Jul 13 18:04:40.527582 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.527391 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" podUID="9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" containerName="maas-api" containerID="cri-o://6b94d054289bf7e5b2d4bcbd85d6b04e1e112066dfc2fbd2d0e0deff0a8ea39a" gracePeriod=30 Jul 13 18:04:40.583554 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.583517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:40.781909 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.781879 2564 generic.go:356] "Generic (PLEG): container finished" podID="9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" containerID="6b94d054289bf7e5b2d4bcbd85d6b04e1e112066dfc2fbd2d0e0deff0a8ea39a" exitCode=0 Jul 13 18:04:40.782044 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.781949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" event={"ID":"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f","Type":"ContainerDied","Data":"6b94d054289bf7e5b2d4bcbd85d6b04e1e112066dfc2fbd2d0e0deff0a8ea39a"} Jul 13 18:04:40.782044 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.782004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" event={"ID":"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f","Type":"ContainerDied","Data":"5e48d9665d1774bee9058a830c176fbabb18577b7967429389c67a71f082b642"} Jul 13 18:04:40.782044 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.782020 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e48d9665d1774bee9058a830c176fbabb18577b7967429389c67a71f082b642" Jul 13 18:04:40.794977 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.794947 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:40.898060 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.898031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls\") pod \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " Jul 13 18:04:40.898268 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.898105 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rwsp\" (UniqueName: \"kubernetes.io/projected/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-kube-api-access-6rwsp\") pod \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " Jul 13 18:04:40.898268 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.898125 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-openshift-service-ca\") pod \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\" (UID: \"9a2ab4c2-6137-40ba-bca1-38e7aa32d81f\") " Jul 13 18:04:40.898513 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.898475 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" (UID: "9a2ab4c2-6137-40ba-bca1-38e7aa32d81f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:04:40.900195 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.900168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-kube-api-access-6rwsp" (OuterVolumeSpecName: "kube-api-access-6rwsp") pod "9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" (UID: "9a2ab4c2-6137-40ba-bca1-38e7aa32d81f"). InnerVolumeSpecName "kube-api-access-6rwsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:04:40.900296 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.900228 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" (UID: "9a2ab4c2-6137-40ba-bca1-38e7aa32d81f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:04:40.999009 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.998967 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:04:40.999009 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.999003 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rwsp\" (UniqueName: \"kubernetes.io/projected/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-kube-api-access-6rwsp\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:04:40.999009 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:40.999014 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:04:41.787374 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:41.787340 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c" Jul 13 18:04:41.804832 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:41.804806 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c"] Jul 13 18:04:41.809452 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:41.809428 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3bf93f7b-7ccc9cf966-vj94c"] Jul 13 18:04:43.638038 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:43.638001 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" path="/var/lib/kubelet/pods/9a2ab4c2-6137-40ba-bca1-38e7aa32d81f/volumes" Jul 13 18:04:45.066051 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:45.066015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:51.162665 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:51.162624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:51.262489 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:51.262459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:55.742031 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:55.741998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:57.480168 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:57.480134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:04:57.534953 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:04:57.534908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:02.248848 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:02.248815 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:02.350582 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:02.350523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:02.452820 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:02.452780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:02.715836 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:02.715801 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:08.649540 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:08.649500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:08.749275 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:08.749216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:14.802530 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:14.802497 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:14.853040 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:14.853001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:15.552153 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:15.552119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:15.903472 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:15.903389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:16.089563 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:16.089531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:19.409701 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.409664 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q"] Jul 13 18:05:19.410336 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.410318 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff04689-9cec-4783-b23a-f34944ccbd76" containerName="revoke-keys" Jul 13 18:05:19.410446 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.410338 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff04689-9cec-4783-b23a-f34944ccbd76" containerName="revoke-keys" Jul 13 18:05:19.410446 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.410359 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" containerName="maas-api" Jul 13 18:05:19.410446 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.410367 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" containerName="maas-api" Jul 13 18:05:19.410599 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.410479 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff04689-9cec-4783-b23a-f34944ccbd76" containerName="revoke-keys" Jul 13 18:05:19.410599 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.410494 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a2ab4c2-6137-40ba-bca1-38e7aa32d81f" containerName="maas-api" Jul 13 18:05:19.412809 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.412788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.414938 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.414916 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ee8b02ac-serving-cert\"" Jul 13 18:05:19.423176 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.423158 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q"] Jul 13 18:05:19.575146 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.575115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kdl\" (UniqueName: \"kubernetes.io/projected/c48fb420-778a-49b1-b54e-060b8a7379f4-kube-api-access-z8kdl\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.575329 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.575161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48fb420-778a-49b1-b54e-060b8a7379f4-openshift-service-ca\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.575329 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.575276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.676097 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.676019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48fb420-778a-49b1-b54e-060b8a7379f4-openshift-service-ca\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.676097 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.676072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.676367 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.676138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kdl\" (UniqueName: \"kubernetes.io/projected/c48fb420-778a-49b1-b54e-060b8a7379f4-kube-api-access-z8kdl\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.676367 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:05:19.676255 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-serving-cert: secret "maas-api-e2e-mt-ee8b02ac-serving-cert" not found Jul 13 18:05:19.676367 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:05:19.676319 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls podName:c48fb420-778a-49b1-b54e-060b8a7379f4 nodeName:}" failed. No retries permitted until 2026-07-13 18:05:20.176303896 +0000 UTC m=+1975.183706121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls") pod "maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" (UID: "c48fb420-778a-49b1-b54e-060b8a7379f4") : secret "maas-api-e2e-mt-ee8b02ac-serving-cert" not found Jul 13 18:05:19.676801 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.676778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48fb420-778a-49b1-b54e-060b8a7379f4-openshift-service-ca\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:19.685811 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:19.685787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kdl\" (UniqueName: \"kubernetes.io/projected/c48fb420-778a-49b1-b54e-060b8a7379f4-kube-api-access-z8kdl\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:20.180344 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:20.180308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:20.182682 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:20.182650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls\") pod \"maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:20.325138 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:20.325105 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:20.460334 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:20.460304 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q"] Jul 13 18:05:20.461348 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:05:20.461312 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48fb420_778a_49b1_b54e_060b8a7379f4.slice/crio-e83392093cefc4b50d5526aa6c384233ea5c4d664a95c875d60a41149c29e993 WatchSource:0}: Error finding container e83392093cefc4b50d5526aa6c384233ea5c4d664a95c875d60a41149c29e993: Status 404 returned error can't find the container with id e83392093cefc4b50d5526aa6c384233ea5c4d664a95c875d60a41149c29e993 Jul 13 18:05:21.016609 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:21.016572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" event={"ID":"c48fb420-778a-49b1-b54e-060b8a7379f4","Type":"ContainerStarted","Data":"6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f"} Jul 13 18:05:21.016780 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:21.016626 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" event={"ID":"c48fb420-778a-49b1-b54e-060b8a7379f4","Type":"ContainerStarted","Data":"e83392093cefc4b50d5526aa6c384233ea5c4d664a95c875d60a41149c29e993"} Jul 13 18:05:21.016780 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:21.016646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:21.036270 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:21.036205 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" podStartSLOduration=1.6436410110000002 podStartE2EDuration="2.036189194s" totalImagesPullingTime="392.548183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:05:21.033783395 +0000 UTC m=+1976.041185638" watchObservedRunningTime="2026-07-13 18:05:21.036189194 +0000 UTC m=+1976.043591479" Jul 13 18:05:21.218087 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:21.217988 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:21.316787 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:21.316756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:23.605852 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:23.605806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:26.134881 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:26.134853 2564 scope.go:117] "RemoveContainer" containerID="7a022e4d3bcc21d8e1185ce3949463d6c88243479cd9ff0b72221acdf2b56dcb" Jul 13 18:05:27.028136 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:27.028107 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:05:32.549254 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:32.549185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:05:56.066221 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:05:56.066184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:06:07.145766 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.145731 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2"] Jul 13 18:06:07.149638 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.149617 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" Jul 13 18:06:07.151588 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.151570 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:06:07.156901 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.156877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2"] Jul 13 18:06:07.244103 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.244063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/befcded0-97ef-4977-9fbf-eaadfdbe7177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2\" (UID: \"befcded0-97ef-4977-9fbf-eaadfdbe7177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" Jul 13 18:06:07.345427 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.345393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/befcded0-97ef-4977-9fbf-eaadfdbe7177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2\" (UID: \"befcded0-97ef-4977-9fbf-eaadfdbe7177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" Jul 13 18:06:07.346024 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.346003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/befcded0-97ef-4977-9fbf-eaadfdbe7177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2\" (UID: \"befcded0-97ef-4977-9fbf-eaadfdbe7177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" Jul 13 18:06:07.461191 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.461110 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" Jul 13 18:06:07.603456 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:07.603407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2"] Jul 13 18:06:07.605771 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:06:07.605743 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefcded0_97ef_4977_9fbf_eaadfdbe7177.slice/crio-ca41b2e26394490868d0d8c51fd642e846cf2d2e1e8420ecb79c55d2960ec769 WatchSource:0}: Error finding container ca41b2e26394490868d0d8c51fd642e846cf2d2e1e8420ecb79c55d2960ec769: Status 404 returned error can't find the container with id ca41b2e26394490868d0d8c51fd642e846cf2d2e1e8420ecb79c55d2960ec769 Jul 13 18:06:08.292929 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:08.292896 2564 generic.go:356] "Generic (PLEG): container finished" podID="befcded0-97ef-4977-9fbf-eaadfdbe7177" containerID="9576da9b7b0771ab2fabdd7242593b0fd918a462a7a9217b6dfeefba03a310a1" exitCode=0 Jul 13 18:06:08.293364 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:08.292977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" event={"ID":"befcded0-97ef-4977-9fbf-eaadfdbe7177","Type":"ContainerDied","Data":"9576da9b7b0771ab2fabdd7242593b0fd918a462a7a9217b6dfeefba03a310a1"} Jul 13 18:06:08.293364 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:08.293047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" event={"ID":"befcded0-97ef-4977-9fbf-eaadfdbe7177","Type":"ContainerStarted","Data":"ca41b2e26394490868d0d8c51fd642e846cf2d2e1e8420ecb79c55d2960ec769"} Jul 13 18:06:09.439670 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:09.439642 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" Jul 13 18:06:09.469754 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:09.469720 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/befcded0-97ef-4977-9fbf-eaadfdbe7177-maas-api-service-ca\") pod \"befcded0-97ef-4977-9fbf-eaadfdbe7177\" (UID: \"befcded0-97ef-4977-9fbf-eaadfdbe7177\") " Jul 13 18:06:09.470169 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:09.470142 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befcded0-97ef-4977-9fbf-eaadfdbe7177-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "befcded0-97ef-4977-9fbf-eaadfdbe7177" (UID: "befcded0-97ef-4977-9fbf-eaadfdbe7177"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:06:09.570474 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:09.570384 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/befcded0-97ef-4977-9fbf-eaadfdbe7177-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:06:10.305781 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:10.305751 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" Jul 13 18:06:10.305957 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:10.305779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2" event={"ID":"befcded0-97ef-4977-9fbf-eaadfdbe7177","Type":"ContainerDied","Data":"ca41b2e26394490868d0d8c51fd642e846cf2d2e1e8420ecb79c55d2960ec769"} Jul 13 18:06:10.305957 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:10.305822 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca41b2e26394490868d0d8c51fd642e846cf2d2e1e8420ecb79c55d2960ec769" Jul 13 18:06:17.564681 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.564625 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q"] Jul 13 18:06:17.565660 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.564890 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" podUID="c48fb420-778a-49b1-b54e-060b8a7379f4" containerName="maas-api" containerID="cri-o://6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f" gracePeriod=30 Jul 13 18:06:17.601404 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.601366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:06:17.824854 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.824788 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:06:17.952994 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.952956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48fb420-778a-49b1-b54e-060b8a7379f4-openshift-service-ca\") pod \"c48fb420-778a-49b1-b54e-060b8a7379f4\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " Jul 13 18:06:17.953197 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.953024 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8kdl\" (UniqueName: \"kubernetes.io/projected/c48fb420-778a-49b1-b54e-060b8a7379f4-kube-api-access-z8kdl\") pod \"c48fb420-778a-49b1-b54e-060b8a7379f4\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " Jul 13 18:06:17.953197 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.953056 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls\") pod \"c48fb420-778a-49b1-b54e-060b8a7379f4\" (UID: \"c48fb420-778a-49b1-b54e-060b8a7379f4\") " Jul 13 18:06:17.953446 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.953421 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48fb420-778a-49b1-b54e-060b8a7379f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c48fb420-778a-49b1-b54e-060b8a7379f4" (UID: "c48fb420-778a-49b1-b54e-060b8a7379f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:06:17.955216 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.955182 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c48fb420-778a-49b1-b54e-060b8a7379f4" (UID: "c48fb420-778a-49b1-b54e-060b8a7379f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:06:17.955216 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:17.955192 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48fb420-778a-49b1-b54e-060b8a7379f4-kube-api-access-z8kdl" (OuterVolumeSpecName: "kube-api-access-z8kdl") pod "c48fb420-778a-49b1-b54e-060b8a7379f4" (UID: "c48fb420-778a-49b1-b54e-060b8a7379f4"). InnerVolumeSpecName "kube-api-access-z8kdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:06:18.054430 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.054392 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48fb420-778a-49b1-b54e-060b8a7379f4-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:06:18.054430 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.054423 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8kdl\" (UniqueName: \"kubernetes.io/projected/c48fb420-778a-49b1-b54e-060b8a7379f4-kube-api-access-z8kdl\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:06:18.054430 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.054433 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c48fb420-778a-49b1-b54e-060b8a7379f4-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:06:18.351278 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.351217 2564 generic.go:356] "Generic (PLEG): container finished" podID="c48fb420-778a-49b1-b54e-060b8a7379f4" containerID="6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f" exitCode=0 Jul 13 18:06:18.351452 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.351268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" event={"ID":"c48fb420-778a-49b1-b54e-060b8a7379f4","Type":"ContainerDied","Data":"6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f"} Jul 13 18:06:18.351452 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.351282 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" Jul 13 18:06:18.351452 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.351379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q" event={"ID":"c48fb420-778a-49b1-b54e-060b8a7379f4","Type":"ContainerDied","Data":"e83392093cefc4b50d5526aa6c384233ea5c4d664a95c875d60a41149c29e993"} Jul 13 18:06:18.351452 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.351412 2564 scope.go:117] "RemoveContainer" containerID="6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f" Jul 13 18:06:18.361541 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.361507 2564 scope.go:117] "RemoveContainer" containerID="6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f" Jul 13 18:06:18.361815 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:06:18.361797 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f\": container with ID starting with 6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f not found: ID does not exist" containerID="6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f" Jul 13 18:06:18.361869 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.361823 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f"} err="failed to get container status \"6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f\": rpc error: code = NotFound desc = could not find container \"6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f\": container with ID starting with 6f0bb50877207550aa0073e1845738fc5025387b4f8b99af3cca2594b21b776f not found: ID does not exist" Jul 13 18:06:18.373489 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.373461 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q"] Jul 13 18:06:18.377364 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:18.377342 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee8b02ac-976b9c744-tbx2q"] Jul 13 18:06:19.638207 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:19.638173 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c48fb420-778a-49b1-b54e-060b8a7379f4" path="/var/lib/kubelet/pods/c48fb420-778a-49b1-b54e-060b8a7379f4/volumes" Jul 13 18:06:21.780747 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:21.780714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:06:22.238209 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:22.238160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:06:26.159044 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:26.159011 2564 scope.go:117] "RemoveContainer" containerID="ae9a710238549b85148ddf562221be5e552584650dc0009e236d8dca95519a80" Jul 13 18:06:26.167643 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:26.167624 2564 scope.go:117] "RemoveContainer" containerID="75f3eb800c7168a8d893e4d781bb47d681ce9aaa1ff1ff6c9b0b9ca5c11de042" Jul 13 18:06:27.263957 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:06:27.263904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:04.184489 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:04.184392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:04.545907 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:04.545865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:06.426945 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:06.426904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:10.228450 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:10.228402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:25.670057 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:25.669956 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:07:25.678244 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:25.678211 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:07:27.933811 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:27.933777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:33.552207 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:33.552172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:33.653502 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:33.653448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:44.764787 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:44.764746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:45.007617 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:45.007587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:46.113483 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:46.113433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:46.223575 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:46.223519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:47.145851 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:47.145781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:47.247129 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:47.247094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:48.705886 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:48.705847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:54.763246 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:54.763169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:54.864737 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:54.864710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:07:55.576375 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:07:55.576319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:01.429463 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:01.429416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:01.536507 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:01.536473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:01.632439 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:01.632403 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:12.253590 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:12.253554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:12.670161 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:12.670126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:26.203601 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:26.203564 2564 scope.go:117] "RemoveContainer" containerID="492170cd5895f350920e31d9a6ecbf024a9633216ee860f0cbb063ad12f03404" Jul 13 18:08:33.798548 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:33.798511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:37.955663 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:37.955624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:39.927538 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:39.927507 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:40.028031 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:40.027994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:40.132075 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:40.132039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:40.510714 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:40.510659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:41.344287 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:41.344245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:43.181054 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.180998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:43.262989 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.262946 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp"] Jul 13 18:08:43.263459 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.263444 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48fb420-778a-49b1-b54e-060b8a7379f4" containerName="maas-api" Jul 13 18:08:43.263516 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.263461 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48fb420-778a-49b1-b54e-060b8a7379f4" containerName="maas-api" Jul 13 18:08:43.263516 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.263488 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befcded0-97ef-4977-9fbf-eaadfdbe7177" containerName="revoke-keys" Jul 13 18:08:43.263516 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.263497 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="befcded0-97ef-4977-9fbf-eaadfdbe7177" containerName="revoke-keys" Jul 13 18:08:43.263609 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.263573 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48fb420-778a-49b1-b54e-060b8a7379f4" containerName="maas-api" Jul 13 18:08:43.263609 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.263586 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="befcded0-97ef-4977-9fbf-eaadfdbe7177" containerName="revoke-keys" Jul 13 18:08:43.266900 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.266877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.267900 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.267874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:43.268932 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.268916 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d09cc8-serving-cert\"" Jul 13 18:08:43.273705 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.273680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp"] Jul 13 18:08:43.331606 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.331570 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:43.391100 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.391059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4459cbf-46ca-40cc-a447-09fc6f5e975a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.391305 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.391114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv87h\" (UniqueName: \"kubernetes.io/projected/d4459cbf-46ca-40cc-a447-09fc6f5e975a-kube-api-access-kv87h\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.391360 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.391305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4459cbf-46ca-40cc-a447-09fc6f5e975a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.492847 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.492744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv87h\" (UniqueName: \"kubernetes.io/projected/d4459cbf-46ca-40cc-a447-09fc6f5e975a-kube-api-access-kv87h\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.493040 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.492854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4459cbf-46ca-40cc-a447-09fc6f5e975a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.493040 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.492910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4459cbf-46ca-40cc-a447-09fc6f5e975a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.493643 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.493618 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4459cbf-46ca-40cc-a447-09fc6f5e975a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.493821 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.493793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:43.495664 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.495643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4459cbf-46ca-40cc-a447-09fc6f5e975a-maas-api-tls\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.501309 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.501290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv87h\" (UniqueName: \"kubernetes.io/projected/d4459cbf-46ca-40cc-a447-09fc6f5e975a-kube-api-access-kv87h\") pod \"maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.579226 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.579183 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:43.718102 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:43.718045 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp"] Jul 13 18:08:43.721123 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:08:43.721095 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4459cbf_46ca_40cc_a447_09fc6f5e975a.slice/crio-094d9456f1b9630f4a80561292707dfe1d4df8863e39ff9f55cfca75dcde1a92 WatchSource:0}: Error finding container 094d9456f1b9630f4a80561292707dfe1d4df8863e39ff9f55cfca75dcde1a92: Status 404 returned error can't find the container with id 094d9456f1b9630f4a80561292707dfe1d4df8863e39ff9f55cfca75dcde1a92 Jul 13 18:08:44.160498 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:44.160460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" event={"ID":"d4459cbf-46ca-40cc-a447-09fc6f5e975a","Type":"ContainerStarted","Data":"094d9456f1b9630f4a80561292707dfe1d4df8863e39ff9f55cfca75dcde1a92"} Jul 13 18:08:44.289592 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:44.289555 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:44.392550 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:44.392514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:45.167610 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:45.167577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" event={"ID":"d4459cbf-46ca-40cc-a447-09fc6f5e975a","Type":"ContainerStarted","Data":"9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed"} Jul 13 18:08:45.167793 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:45.167641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:45.184738 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:45.184685 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" podStartSLOduration=1.4657529550000001 podStartE2EDuration="2.184667086s" totalImagesPullingTime="718.914131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:08:45.183544872 +0000 UTC m=+2180.190947114" watchObservedRunningTime="2026-07-13 18:08:45.184667086 +0000 UTC m=+2180.192069328" Jul 13 18:08:47.810215 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:47.810181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:48.215658 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:48.215572 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:48.570200 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:48.570158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:48.670273 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:48.670216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:49.548485 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:49.548449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:51.177397 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.177367 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:08:51.355921 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.355886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:51.414159 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.414126 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9"] Jul 13 18:08:51.418411 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.418387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.420696 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.420674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dc368b-serving-cert\"" Jul 13 18:08:51.434713 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.434657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9"] Jul 13 18:08:51.469023 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.468993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:51.555211 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.555183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:51.573570 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.573537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21b2d3b-4a6f-4493-978a-562cc0d32a78-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.573741 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.573586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21b2d3b-4a6f-4493-978a-562cc0d32a78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.573741 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.573660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8k8\" (UniqueName: \"kubernetes.io/projected/a21b2d3b-4a6f-4493-978a-562cc0d32a78-kube-api-access-tr8k8\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.646713 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.646678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:51.674917 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.674881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8k8\" (UniqueName: \"kubernetes.io/projected/a21b2d3b-4a6f-4493-978a-562cc0d32a78-kube-api-access-tr8k8\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.675128 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.674987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21b2d3b-4a6f-4493-978a-562cc0d32a78-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.675128 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.675024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21b2d3b-4a6f-4493-978a-562cc0d32a78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.675739 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.675703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21b2d3b-4a6f-4493-978a-562cc0d32a78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.677548 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.677528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21b2d3b-4a6f-4493-978a-562cc0d32a78-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.683843 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.683818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8k8\" (UniqueName: \"kubernetes.io/projected/a21b2d3b-4a6f-4493-978a-562cc0d32a78-kube-api-access-tr8k8\") pod \"maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.730706 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.730624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:51.873497 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:51.873466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9"] Jul 13 18:08:51.874787 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:08:51.874763 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21b2d3b_4a6f_4493_978a_562cc0d32a78.slice/crio-0712ee585a76748eab3dac285ad0ecf4684915269e4544994f033ce401f9149a WatchSource:0}: Error finding container 0712ee585a76748eab3dac285ad0ecf4684915269e4544994f033ce401f9149a: Status 404 returned error can't find the container with id 0712ee585a76748eab3dac285ad0ecf4684915269e4544994f033ce401f9149a Jul 13 18:08:52.166943 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:52.166905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:52.212718 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:52.212682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" event={"ID":"a21b2d3b-4a6f-4493-978a-562cc0d32a78","Type":"ContainerStarted","Data":"0712ee585a76748eab3dac285ad0ecf4684915269e4544994f033ce401f9149a"} Jul 13 18:08:52.285703 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:52.285676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:53.220921 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:53.220884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" event={"ID":"a21b2d3b-4a6f-4493-978a-562cc0d32a78","Type":"ContainerStarted","Data":"80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e"} Jul 13 18:08:53.221336 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:53.221047 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:08:53.240628 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:53.240571 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" podStartSLOduration=1.869828405 podStartE2EDuration="2.240554085s" totalImagesPullingTime="370.72568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:08:53.240014754 +0000 UTC m=+2188.247416998" watchObservedRunningTime="2026-07-13 18:08:53.240554085 +0000 UTC m=+2188.247956329" Jul 13 18:08:55.937409 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:55.937376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:08:59.231867 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:08:59.231833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:09:05.759386 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:05.759337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:05.859316 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:05.859276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:11.235135 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:11.235093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:14.114795 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:14.114752 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:14.215162 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:14.215129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:26.546730 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:26.546698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:31.056819 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:31.056789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:44.447880 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:44.447842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:45.386529 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.386495 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:45.475064 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.475025 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4"] Jul 13 18:09:45.479000 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.478975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" Jul 13 18:09:45.480953 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.480929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:09:45.485288 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.485267 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4"] Jul 13 18:09:45.579172 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.579139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75476f-e1c1-4451-9eeb-e20f5b70f8f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4\" (UID: \"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" Jul 13 18:09:45.680028 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.679945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75476f-e1c1-4451-9eeb-e20f5b70f8f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4\" (UID: \"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" Jul 13 18:09:45.680639 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.680613 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75476f-e1c1-4451-9eeb-e20f5b70f8f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4\" (UID: \"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" Jul 13 18:09:45.792426 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.792382 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" Jul 13 18:09:45.926514 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:45.926490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4"] Jul 13 18:09:45.928214 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:09:45.928185 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b75476f_e1c1_4451_9eeb_e20f5b70f8f0.slice/crio-1ce1007e92b0cadcb03a289cf24ff259e1c4390734c29f13ad187859e8d34213 WatchSource:0}: Error finding container 1ce1007e92b0cadcb03a289cf24ff259e1c4390734c29f13ad187859e8d34213: Status 404 returned error can't find the container with id 1ce1007e92b0cadcb03a289cf24ff259e1c4390734c29f13ad187859e8d34213 Jul 13 18:09:46.066743 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:46.066715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:46.169021 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:46.168991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:46.513264 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:46.513206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" event={"ID":"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0","Type":"ContainerStarted","Data":"e6eb8bf50b06557de0b76b38e5033abc292ea8f011513545c3fa375459c0df68"} Jul 13 18:09:46.513675 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:46.513294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" event={"ID":"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0","Type":"ContainerStarted","Data":"1ce1007e92b0cadcb03a289cf24ff259e1c4390734c29f13ad187859e8d34213"} Jul 13 18:09:46.527676 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:46.527624 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" podStartSLOduration=1.527606203 podStartE2EDuration="1.527606203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:09:46.526877584 +0000 UTC m=+2241.534279847" watchObservedRunningTime="2026-07-13 18:09:46.527606203 +0000 UTC m=+2241.535008445" Jul 13 18:09:47.519885 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:47.519795 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b75476f-e1c1-4451-9eeb-e20f5b70f8f0" containerID="e6eb8bf50b06557de0b76b38e5033abc292ea8f011513545c3fa375459c0df68" exitCode=0 Jul 13 18:09:47.519885 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:47.519864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" event={"ID":"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0","Type":"ContainerDied","Data":"e6eb8bf50b06557de0b76b38e5033abc292ea8f011513545c3fa375459c0df68"} Jul 13 18:09:48.663449 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:48.663424 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" Jul 13 18:09:48.705898 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:48.705873 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75476f-e1c1-4451-9eeb-e20f5b70f8f0-maas-api-service-ca\") pod \"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0\" (UID: \"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0\") " Jul 13 18:09:48.706195 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:48.706169 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b75476f-e1c1-4451-9eeb-e20f5b70f8f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b75476f-e1c1-4451-9eeb-e20f5b70f8f0" (UID: "1b75476f-e1c1-4451-9eeb-e20f5b70f8f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:09:48.807117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:48.807095 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b75476f-e1c1-4451-9eeb-e20f5b70f8f0-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:09:49.532714 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:49.532686 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" Jul 13 18:09:49.532886 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:49.532723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4" event={"ID":"1b75476f-e1c1-4451-9eeb-e20f5b70f8f0","Type":"ContainerDied","Data":"1ce1007e92b0cadcb03a289cf24ff259e1c4390734c29f13ad187859e8d34213"} Jul 13 18:09:49.532886 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:49.532769 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ce1007e92b0cadcb03a289cf24ff259e1c4390734c29f13ad187859e8d34213" Jul 13 18:09:55.925697 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:55.925668 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp"] Jul 13 18:09:55.926464 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:55.925884 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" podUID="d4459cbf-46ca-40cc-a447-09fc6f5e975a" containerName="maas-api" containerID="cri-o://9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed" gracePeriod=30 Jul 13 18:09:55.948714 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:55.948690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:56.197838 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.197807 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:09:56.267641 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.267605 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv87h\" (UniqueName: \"kubernetes.io/projected/d4459cbf-46ca-40cc-a447-09fc6f5e975a-kube-api-access-kv87h\") pod \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " Jul 13 18:09:56.267820 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.267701 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4459cbf-46ca-40cc-a447-09fc6f5e975a-openshift-service-ca\") pod \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " Jul 13 18:09:56.267820 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.267784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4459cbf-46ca-40cc-a447-09fc6f5e975a-maas-api-tls\") pod \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\" (UID: \"d4459cbf-46ca-40cc-a447-09fc6f5e975a\") " Jul 13 18:09:56.268095 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.268069 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4459cbf-46ca-40cc-a447-09fc6f5e975a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4459cbf-46ca-40cc-a447-09fc6f5e975a" (UID: "d4459cbf-46ca-40cc-a447-09fc6f5e975a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:09:56.269603 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.269580 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4459cbf-46ca-40cc-a447-09fc6f5e975a-kube-api-access-kv87h" (OuterVolumeSpecName: "kube-api-access-kv87h") pod "d4459cbf-46ca-40cc-a447-09fc6f5e975a" (UID: "d4459cbf-46ca-40cc-a447-09fc6f5e975a"). InnerVolumeSpecName "kube-api-access-kv87h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:09:56.269812 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.269785 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4459cbf-46ca-40cc-a447-09fc6f5e975a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4459cbf-46ca-40cc-a447-09fc6f5e975a" (UID: "d4459cbf-46ca-40cc-a447-09fc6f5e975a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:09:56.369373 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.369332 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4459cbf-46ca-40cc-a447-09fc6f5e975a-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:09:56.369373 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.369364 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4459cbf-46ca-40cc-a447-09fc6f5e975a-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:09:56.369373 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.369378 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv87h\" (UniqueName: \"kubernetes.io/projected/d4459cbf-46ca-40cc-a447-09fc6f5e975a-kube-api-access-kv87h\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:09:56.580026 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.579984 2564 generic.go:356] "Generic (PLEG): container finished" podID="d4459cbf-46ca-40cc-a447-09fc6f5e975a" containerID="9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed" exitCode=0 Jul 13 18:09:56.580243 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.580046 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" Jul 13 18:09:56.580243 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.580063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" event={"ID":"d4459cbf-46ca-40cc-a447-09fc6f5e975a","Type":"ContainerDied","Data":"9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed"} Jul 13 18:09:56.580243 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.580141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" event={"ID":"d4459cbf-46ca-40cc-a447-09fc6f5e975a","Type":"ContainerDied","Data":"094d9456f1b9630f4a80561292707dfe1d4df8863e39ff9f55cfca75dcde1a92"} Jul 13 18:09:56.580243 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.580162 2564 scope.go:117] "RemoveContainer" containerID="9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed" Jul 13 18:09:56.589596 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.589579 2564 scope.go:117] "RemoveContainer" containerID="9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed" Jul 13 18:09:56.589854 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:09:56.589837 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed\": container with ID starting with 9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed not found: ID does not exist" containerID="9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed" Jul 13 18:09:56.589908 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.589861 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed"} err="failed to get container status \"9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed\": rpc error: code = NotFound desc = could not find container \"9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed\": container with ID starting with 9264d2c2372e61bca19ebbddd8233ea5805e691e69062b9887ff9ac5dff2f5ed not found: ID does not exist" Jul 13 18:09:56.601501 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.601477 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp"] Jul 13 18:09:56.605614 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.605595 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp"] Jul 13 18:09:56.812886 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.812857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:56.912090 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:56.911984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:09:57.638713 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:57.638678 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4459cbf-46ca-40cc-a447-09fc6f5e975a" path="/var/lib/kubelet/pods/d4459cbf-46ca-40cc-a447-09fc6f5e975a/volumes" Jul 13 18:09:59.172297 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:09:59.172263 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d09cc8-859f5c5fc-mrnpp" podUID="d4459cbf-46ca-40cc-a447-09fc6f5e975a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.66:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 13 18:10:00.583559 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:00.583521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:06.545130 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:06.545078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:06.644975 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:06.644926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:11.321200 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.321164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:11.565610 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.565579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:11.702453 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.702374 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x"] Jul 13 18:10:11.703066 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.703046 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4459cbf-46ca-40cc-a447-09fc6f5e975a" containerName="maas-api" Jul 13 18:10:11.703117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.703071 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4459cbf-46ca-40cc-a447-09fc6f5e975a" containerName="maas-api" Jul 13 18:10:11.703117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.703095 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b75476f-e1c1-4451-9eeb-e20f5b70f8f0" containerName="revoke-keys" Jul 13 18:10:11.703117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.703105 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b75476f-e1c1-4451-9eeb-e20f5b70f8f0" containerName="revoke-keys" Jul 13 18:10:11.703210 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.703204 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b75476f-e1c1-4451-9eeb-e20f5b70f8f0" containerName="revoke-keys" Jul 13 18:10:11.703279 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.703221 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4459cbf-46ca-40cc-a447-09fc6f5e975a" containerName="maas-api" Jul 13 18:10:11.706883 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.706868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" Jul 13 18:10:11.709818 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.709790 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:10:11.712292 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.712255 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x"] Jul 13 18:10:11.819622 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.819591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/307905eb-c6f3-4f38-8684-d9cc6e12f976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x\" (UID: \"307905eb-c6f3-4f38-8684-d9cc6e12f976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" Jul 13 18:10:11.920904 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.920870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/307905eb-c6f3-4f38-8684-d9cc6e12f976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x\" (UID: \"307905eb-c6f3-4f38-8684-d9cc6e12f976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" Jul 13 18:10:11.921532 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:11.921507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/307905eb-c6f3-4f38-8684-d9cc6e12f976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x\" (UID: \"307905eb-c6f3-4f38-8684-d9cc6e12f976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" Jul 13 18:10:12.019015 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:12.018950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" Jul 13 18:10:12.163508 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:12.163482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x"] Jul 13 18:10:12.165647 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:10:12.165613 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307905eb_c6f3_4f38_8684_d9cc6e12f976.slice/crio-7f2574fc022faa450937f3dc8cad3e20a613067b7f708aa4d731cfd614f9415e WatchSource:0}: Error finding container 7f2574fc022faa450937f3dc8cad3e20a613067b7f708aa4d731cfd614f9415e: Status 404 returned error can't find the container with id 7f2574fc022faa450937f3dc8cad3e20a613067b7f708aa4d731cfd614f9415e Jul 13 18:10:12.260925 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:12.260903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:12.362397 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:12.362368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:12.672028 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:12.671935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" event={"ID":"307905eb-c6f3-4f38-8684-d9cc6e12f976","Type":"ContainerStarted","Data":"c885558217876f0e4d10b2781fe4404d7fa2a3711e1a4f0917783b050b3fa425"} Jul 13 18:10:12.672028 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:12.672000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" event={"ID":"307905eb-c6f3-4f38-8684-d9cc6e12f976","Type":"ContainerStarted","Data":"7f2574fc022faa450937f3dc8cad3e20a613067b7f708aa4d731cfd614f9415e"} Jul 13 18:10:12.690277 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:12.690207 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" podStartSLOduration=1.690188897 podStartE2EDuration="1.690188897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:10:12.685844305 +0000 UTC m=+2267.693246549" watchObservedRunningTime="2026-07-13 18:10:12.690188897 +0000 UTC m=+2267.697591143" Jul 13 18:10:13.677795 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:13.677759 2564 generic.go:356] "Generic (PLEG): container finished" podID="307905eb-c6f3-4f38-8684-d9cc6e12f976" containerID="c885558217876f0e4d10b2781fe4404d7fa2a3711e1a4f0917783b050b3fa425" exitCode=0 Jul 13 18:10:13.678153 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:13.677815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" event={"ID":"307905eb-c6f3-4f38-8684-d9cc6e12f976","Type":"ContainerDied","Data":"c885558217876f0e4d10b2781fe4404d7fa2a3711e1a4f0917783b050b3fa425"} Jul 13 18:10:14.822706 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:14.822684 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" Jul 13 18:10:14.947188 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:14.947106 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/307905eb-c6f3-4f38-8684-d9cc6e12f976-maas-api-service-ca\") pod \"307905eb-c6f3-4f38-8684-d9cc6e12f976\" (UID: \"307905eb-c6f3-4f38-8684-d9cc6e12f976\") " Jul 13 18:10:14.947481 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:14.947459 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307905eb-c6f3-4f38-8684-d9cc6e12f976-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "307905eb-c6f3-4f38-8684-d9cc6e12f976" (UID: "307905eb-c6f3-4f38-8684-d9cc6e12f976"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:10:15.048466 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:15.048435 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/307905eb-c6f3-4f38-8684-d9cc6e12f976-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:10:15.689777 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:15.689749 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" Jul 13 18:10:15.689922 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:15.689793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x" event={"ID":"307905eb-c6f3-4f38-8684-d9cc6e12f976","Type":"ContainerDied","Data":"7f2574fc022faa450937f3dc8cad3e20a613067b7f708aa4d731cfd614f9415e"} Jul 13 18:10:15.689922 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:15.689836 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2574fc022faa450937f3dc8cad3e20a613067b7f708aa4d731cfd614f9415e" Jul 13 18:10:22.090152 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.090121 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9"] Jul 13 18:10:22.090628 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.090335 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" podUID="a21b2d3b-4a6f-4493-978a-562cc0d32a78" containerName="maas-api" containerID="cri-o://80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e" gracePeriod=30 Jul 13 18:10:22.134695 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.134661 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:22.536355 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.536332 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:10:22.620081 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.620020 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21b2d3b-4a6f-4493-978a-562cc0d32a78-maas-api-tls\") pod \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " Jul 13 18:10:22.620202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.620080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21b2d3b-4a6f-4493-978a-562cc0d32a78-openshift-service-ca\") pod \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " Jul 13 18:10:22.620202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.620108 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr8k8\" (UniqueName: \"kubernetes.io/projected/a21b2d3b-4a6f-4493-978a-562cc0d32a78-kube-api-access-tr8k8\") pod \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\" (UID: \"a21b2d3b-4a6f-4493-978a-562cc0d32a78\") " Jul 13 18:10:22.620479 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.620456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21b2d3b-4a6f-4493-978a-562cc0d32a78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a21b2d3b-4a6f-4493-978a-562cc0d32a78" (UID: "a21b2d3b-4a6f-4493-978a-562cc0d32a78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:10:22.622171 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.622147 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21b2d3b-4a6f-4493-978a-562cc0d32a78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a21b2d3b-4a6f-4493-978a-562cc0d32a78" (UID: "a21b2d3b-4a6f-4493-978a-562cc0d32a78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:10:22.622291 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.622187 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21b2d3b-4a6f-4493-978a-562cc0d32a78-kube-api-access-tr8k8" (OuterVolumeSpecName: "kube-api-access-tr8k8") pod "a21b2d3b-4a6f-4493-978a-562cc0d32a78" (UID: "a21b2d3b-4a6f-4493-978a-562cc0d32a78"). InnerVolumeSpecName "kube-api-access-tr8k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:10:22.720730 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.720705 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a21b2d3b-4a6f-4493-978a-562cc0d32a78-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:10:22.720730 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.720729 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a21b2d3b-4a6f-4493-978a-562cc0d32a78-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:10:22.720878 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.720742 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr8k8\" (UniqueName: \"kubernetes.io/projected/a21b2d3b-4a6f-4493-978a-562cc0d32a78-kube-api-access-tr8k8\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:10:22.731481 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.731457 2564 generic.go:356] "Generic (PLEG): container finished" podID="a21b2d3b-4a6f-4493-978a-562cc0d32a78" containerID="80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e" exitCode=0 Jul 13 18:10:22.731578 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.731501 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" Jul 13 18:10:22.731578 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.731531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" event={"ID":"a21b2d3b-4a6f-4493-978a-562cc0d32a78","Type":"ContainerDied","Data":"80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e"} Jul 13 18:10:22.731676 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.731568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9" event={"ID":"a21b2d3b-4a6f-4493-978a-562cc0d32a78","Type":"ContainerDied","Data":"0712ee585a76748eab3dac285ad0ecf4684915269e4544994f033ce401f9149a"} Jul 13 18:10:22.731676 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.731602 2564 scope.go:117] "RemoveContainer" containerID="80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e" Jul 13 18:10:22.743261 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.743219 2564 scope.go:117] "RemoveContainer" containerID="80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e" Jul 13 18:10:22.743517 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:10:22.743496 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e\": container with ID starting with 80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e not found: ID does not exist" containerID="80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e" Jul 13 18:10:22.743574 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.743527 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e"} err="failed to get container status \"80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e\": rpc error: code = NotFound desc = could not find container \"80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e\": container with ID starting with 80fb39dd5008aa5ffb4ffb224543aa70fb6e38aa01448f99ed78f01160449a3e not found: ID does not exist" Jul 13 18:10:22.756585 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.756565 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9"] Jul 13 18:10:22.765397 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:10:22.765375 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21b2d3b_4a6f_4493_978a_562cc0d32a78.slice\": RecentStats: unable to find data in memory cache]" Jul 13 18:10:22.766386 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:22.766365 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc368b-6cbb6496bc-ct8d9"] Jul 13 18:10:23.637990 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:23.637952 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a21b2d3b-4a6f-4493-978a-562cc0d32a78" path="/var/lib/kubelet/pods/a21b2d3b-4a6f-4493-978a-562cc0d32a78/volumes" Jul 13 18:10:26.235211 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:26.235180 2564 scope.go:117] "RemoveContainer" containerID="6b94d054289bf7e5b2d4bcbd85d6b04e1e112066dfc2fbd2d0e0deff0a8ea39a" Jul 13 18:10:26.790966 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:26.790929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:33.372166 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:33.372129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:33.444819 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:33.444783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:37.578127 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:37.578097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:37.859091 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:37.859018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:38.183305 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:38.183197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:38.516064 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:38.515955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:38.765051 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:38.765023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:41.825893 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.825860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:41.911504 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.911465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:41.912703 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.912681 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9"] Jul 13 18:10:41.913102 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.913091 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307905eb-c6f3-4f38-8684-d9cc6e12f976" containerName="revoke-keys" Jul 13 18:10:41.913142 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.913105 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="307905eb-c6f3-4f38-8684-d9cc6e12f976" containerName="revoke-keys" Jul 13 18:10:41.913142 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.913134 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21b2d3b-4a6f-4493-978a-562cc0d32a78" containerName="maas-api" Jul 13 18:10:41.913142 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.913140 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21b2d3b-4a6f-4493-978a-562cc0d32a78" containerName="maas-api" Jul 13 18:10:41.913257 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.913202 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="307905eb-c6f3-4f38-8684-d9cc6e12f976" containerName="revoke-keys" Jul 13 18:10:41.913257 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.913213 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21b2d3b-4a6f-4493-978a-562cc0d32a78" containerName="maas-api" Jul 13 18:10:41.916636 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.916615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:41.918999 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.918977 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a50684-serving-cert\"" Jul 13 18:10:41.928523 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.928498 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9"] Jul 13 18:10:41.985850 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.985812 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:41.991144 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.991111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9634ab8f-4c52-4d90-b07a-0e3b311a109e-openshift-service-ca\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:41.991272 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.991185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmnr\" (UniqueName: \"kubernetes.io/projected/9634ab8f-4c52-4d90-b07a-0e3b311a109e-kube-api-access-bbmnr\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:41.991272 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:41.991217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9634ab8f-4c52-4d90-b07a-0e3b311a109e-maas-api-tls\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.087984 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.087894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:42.092695 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.092673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9634ab8f-4c52-4d90-b07a-0e3b311a109e-openshift-service-ca\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.092767 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.092719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmnr\" (UniqueName: \"kubernetes.io/projected/9634ab8f-4c52-4d90-b07a-0e3b311a109e-kube-api-access-bbmnr\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.092767 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.092749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9634ab8f-4c52-4d90-b07a-0e3b311a109e-maas-api-tls\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.093276 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.093252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9634ab8f-4c52-4d90-b07a-0e3b311a109e-openshift-service-ca\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.096671 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.096644 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9634ab8f-4c52-4d90-b07a-0e3b311a109e-maas-api-tls\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.101202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.101174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmnr\" (UniqueName: \"kubernetes.io/projected/9634ab8f-4c52-4d90-b07a-0e3b311a109e-kube-api-access-bbmnr\") pod \"maas-api-e2e-api-a-a50684-f4594dbc9-m56w9\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.228538 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.228507 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:42.368519 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.368493 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9"] Jul 13 18:10:42.370222 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:10:42.370187 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9634ab8f_4c52_4d90_b07a_0e3b311a109e.slice/crio-f5bae63f5eef12ede5de99227581d6d255dd02c5a99762fba2fef99294b846e5 WatchSource:0}: Error finding container f5bae63f5eef12ede5de99227581d6d255dd02c5a99762fba2fef99294b846e5: Status 404 returned error can't find the container with id f5bae63f5eef12ede5de99227581d6d255dd02c5a99762fba2fef99294b846e5 Jul 13 18:10:42.371978 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.371961 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 18:10:42.532172 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.532137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:42.640550 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.640473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:42.849293 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:42.849260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" event={"ID":"9634ab8f-4c52-4d90-b07a-0e3b311a109e","Type":"ContainerStarted","Data":"f5bae63f5eef12ede5de99227581d6d255dd02c5a99762fba2fef99294b846e5"} Jul 13 18:10:43.863128 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:43.863077 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" event={"ID":"9634ab8f-4c52-4d90-b07a-0e3b311a109e","Type":"ContainerStarted","Data":"3bdbdc0648f4cf1c5791fc30cd73329ef32464d756c2d2cd9456b2a10b84b756"} Jul 13 18:10:43.863612 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:43.863182 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:43.881125 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:43.881068 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" podStartSLOduration=2.380793881 podStartE2EDuration="2.88105179s" totalImagesPullingTime="500.257909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:10:43.880511808 +0000 UTC m=+2298.887914094" watchObservedRunningTime="2026-07-13 18:10:43.88105179 +0000 UTC m=+2298.888454033" Jul 13 18:10:46.384018 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:46.383983 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:46.923065 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:46.923033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:47.281102 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:47.281010 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:47.382694 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:47.382662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:49.854938 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:49.854907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:49.876072 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:49.876046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:10:49.956279 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:49.956250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:50.031096 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.031069 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr"] Jul 13 18:10:50.034991 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.034974 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.038002 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.037975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b8e947-serving-cert\"" Jul 13 18:10:50.048894 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.048868 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr"] Jul 13 18:10:50.073053 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.073027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:50.163679 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.163607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a3d97d-5e7d-43f5-8fc4-6d356491e459-openshift-service-ca\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.163679 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.163648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.163859 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.163717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2sl\" (UniqueName: \"kubernetes.io/projected/45a3d97d-5e7d-43f5-8fc4-6d356491e459-kube-api-access-vl2sl\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.264929 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.264884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a3d97d-5e7d-43f5-8fc4-6d356491e459-openshift-service-ca\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.265127 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.264936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.265127 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.264986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2sl\" (UniqueName: \"kubernetes.io/projected/45a3d97d-5e7d-43f5-8fc4-6d356491e459-kube-api-access-vl2sl\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.265127 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:10:50.265101 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-serving-cert: secret "maas-api-e2e-api-b-b8e947-serving-cert" not found Jul 13 18:10:50.265333 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:10:50.265165 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls podName:45a3d97d-5e7d-43f5-8fc4-6d356491e459 nodeName:}" failed. No retries permitted until 2026-07-13 18:10:50.765148667 +0000 UTC m=+2305.772550888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls") pod "maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" (UID: "45a3d97d-5e7d-43f5-8fc4-6d356491e459") : secret "maas-api-e2e-api-b-b8e947-serving-cert" not found Jul 13 18:10:50.265609 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.265586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a3d97d-5e7d-43f5-8fc4-6d356491e459-openshift-service-ca\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.282526 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.282498 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2sl\" (UniqueName: \"kubernetes.io/projected/45a3d97d-5e7d-43f5-8fc4-6d356491e459-kube-api-access-vl2sl\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.648906 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.648868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:50.759512 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.759475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:50.769653 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.769633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.771986 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.771963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls\") pod \"maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:50.851758 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.851720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:50.946604 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:50.946536 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:51.085983 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:51.085954 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr"] Jul 13 18:10:51.087926 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:10:51.087898 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a3d97d_5e7d_43f5_8fc4_6d356491e459.slice/crio-664a90db22998638cb4636258b5779956b7f74f6d76d50918d58624935d55c6e WatchSource:0}: Error finding container 664a90db22998638cb4636258b5779956b7f74f6d76d50918d58624935d55c6e: Status 404 returned error can't find the container with id 664a90db22998638cb4636258b5779956b7f74f6d76d50918d58624935d55c6e Jul 13 18:10:51.909226 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:51.909191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" event={"ID":"45a3d97d-5e7d-43f5-8fc4-6d356491e459","Type":"ContainerStarted","Data":"07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c"} Jul 13 18:10:51.909438 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:51.909270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" event={"ID":"45a3d97d-5e7d-43f5-8fc4-6d356491e459","Type":"ContainerStarted","Data":"664a90db22998638cb4636258b5779956b7f74f6d76d50918d58624935d55c6e"} Jul 13 18:10:51.909438 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:51.909299 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:10:51.929402 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:51.929358 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" podStartSLOduration=1.493134586 podStartE2EDuration="1.929344068s" totalImagesPullingTime="436.209482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:10:51.927119093 +0000 UTC m=+2306.934521348" watchObservedRunningTime="2026-07-13 18:10:51.929344068 +0000 UTC m=+2306.936746326" Jul 13 18:10:54.524047 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:54.524020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:10:57.920151 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:10:57.920124 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:11:02.637190 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.637157 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs"] Jul 13 18:11:02.640988 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.640957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" Jul 13 18:11:02.643341 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.643319 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:11:02.648745 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.648718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs"] Jul 13 18:11:02.778322 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.778286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7099a5-90d1-4399-9023-c5a73bff6bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a50684-74qvs\" (UID: \"1d7099a5-90d1-4399-9023-c5a73bff6bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" Jul 13 18:11:02.879559 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.879528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7099a5-90d1-4399-9023-c5a73bff6bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a50684-74qvs\" (UID: \"1d7099a5-90d1-4399-9023-c5a73bff6bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" Jul 13 18:11:02.880149 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.880131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7099a5-90d1-4399-9023-c5a73bff6bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a50684-74qvs\" (UID: \"1d7099a5-90d1-4399-9023-c5a73bff6bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" Jul 13 18:11:02.954536 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:02.954467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" Jul 13 18:11:03.098733 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:03.098709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs"] Jul 13 18:11:03.100054 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:11:03.100027 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7099a5_90d1_4399_9023_c5a73bff6bd2.slice/crio-06ee70380f9ff0da49738ff24bf827a4b589d10c03bf71a7cf159f7713075c4b WatchSource:0}: Error finding container 06ee70380f9ff0da49738ff24bf827a4b589d10c03bf71a7cf159f7713075c4b: Status 404 returned error can't find the container with id 06ee70380f9ff0da49738ff24bf827a4b589d10c03bf71a7cf159f7713075c4b Jul 13 18:11:03.286760 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:03.286729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:03.385757 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:03.385734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:03.976855 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:03.976819 2564 generic.go:356] "Generic (PLEG): container finished" podID="1d7099a5-90d1-4399-9023-c5a73bff6bd2" containerID="1b0229658382836e17d7fef0e3865dd5f61c319a52af3b58eccb915e13258fc0" exitCode=0 Jul 13 18:11:03.977275 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:03.976885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" event={"ID":"1d7099a5-90d1-4399-9023-c5a73bff6bd2","Type":"ContainerDied","Data":"1b0229658382836e17d7fef0e3865dd5f61c319a52af3b58eccb915e13258fc0"} Jul 13 18:11:03.977275 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:03.976942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" event={"ID":"1d7099a5-90d1-4399-9023-c5a73bff6bd2","Type":"ContainerStarted","Data":"06ee70380f9ff0da49738ff24bf827a4b589d10c03bf71a7cf159f7713075c4b"} Jul 13 18:11:05.120854 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:05.120824 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" Jul 13 18:11:05.204280 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:05.204254 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7099a5-90d1-4399-9023-c5a73bff6bd2-maas-api-service-ca\") pod \"1d7099a5-90d1-4399-9023-c5a73bff6bd2\" (UID: \"1d7099a5-90d1-4399-9023-c5a73bff6bd2\") " Jul 13 18:11:05.204530 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:05.204509 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7099a5-90d1-4399-9023-c5a73bff6bd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d7099a5-90d1-4399-9023-c5a73bff6bd2" (UID: "1d7099a5-90d1-4399-9023-c5a73bff6bd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:11:05.305020 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:05.304994 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d7099a5-90d1-4399-9023-c5a73bff6bd2-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:05.989957 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:05.989881 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" Jul 13 18:11:05.990088 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:05.989885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a50684-74qvs" event={"ID":"1d7099a5-90d1-4399-9023-c5a73bff6bd2","Type":"ContainerDied","Data":"06ee70380f9ff0da49738ff24bf827a4b589d10c03bf71a7cf159f7713075c4b"} Jul 13 18:11:05.990088 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:05.990005 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ee70380f9ff0da49738ff24bf827a4b589d10c03bf71a7cf159f7713075c4b" Jul 13 18:11:12.562079 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:12.562041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:12.660647 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:12.660611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:12.821055 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:12.820965 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9"] Jul 13 18:11:12.821213 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:12.821182 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" podUID="9634ab8f-4c52-4d90-b07a-0e3b311a109e" containerName="maas-api" containerID="cri-o://3bdbdc0648f4cf1c5791fc30cd73329ef32464d756c2d2cd9456b2a10b84b756" gracePeriod=30 Jul 13 18:11:12.868424 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:12.868393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:13.034370 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.034340 2564 generic.go:356] "Generic (PLEG): container finished" podID="9634ab8f-4c52-4d90-b07a-0e3b311a109e" containerID="3bdbdc0648f4cf1c5791fc30cd73329ef32464d756c2d2cd9456b2a10b84b756" exitCode=0 Jul 13 18:11:13.034520 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.034411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" event={"ID":"9634ab8f-4c52-4d90-b07a-0e3b311a109e","Type":"ContainerDied","Data":"3bdbdc0648f4cf1c5791fc30cd73329ef32464d756c2d2cd9456b2a10b84b756"} Jul 13 18:11:13.081793 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.081744 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:11:13.175528 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.175503 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9634ab8f-4c52-4d90-b07a-0e3b311a109e-maas-api-tls\") pod \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " Jul 13 18:11:13.175709 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.175584 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9634ab8f-4c52-4d90-b07a-0e3b311a109e-openshift-service-ca\") pod \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " Jul 13 18:11:13.175709 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.175620 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbmnr\" (UniqueName: \"kubernetes.io/projected/9634ab8f-4c52-4d90-b07a-0e3b311a109e-kube-api-access-bbmnr\") pod \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\" (UID: \"9634ab8f-4c52-4d90-b07a-0e3b311a109e\") " Jul 13 18:11:13.175984 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.175959 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9634ab8f-4c52-4d90-b07a-0e3b311a109e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9634ab8f-4c52-4d90-b07a-0e3b311a109e" (UID: "9634ab8f-4c52-4d90-b07a-0e3b311a109e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:11:13.177645 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.177622 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9634ab8f-4c52-4d90-b07a-0e3b311a109e-kube-api-access-bbmnr" (OuterVolumeSpecName: "kube-api-access-bbmnr") pod "9634ab8f-4c52-4d90-b07a-0e3b311a109e" (UID: "9634ab8f-4c52-4d90-b07a-0e3b311a109e"). InnerVolumeSpecName "kube-api-access-bbmnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:11:13.177747 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.177697 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9634ab8f-4c52-4d90-b07a-0e3b311a109e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9634ab8f-4c52-4d90-b07a-0e3b311a109e" (UID: "9634ab8f-4c52-4d90-b07a-0e3b311a109e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:11:13.276659 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.276638 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9634ab8f-4c52-4d90-b07a-0e3b311a109e-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:13.276659 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.276660 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmnr\" (UniqueName: \"kubernetes.io/projected/9634ab8f-4c52-4d90-b07a-0e3b311a109e-kube-api-access-bbmnr\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:13.276808 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:13.276672 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9634ab8f-4c52-4d90-b07a-0e3b311a109e-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:14.041919 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:14.041884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" event={"ID":"9634ab8f-4c52-4d90-b07a-0e3b311a109e","Type":"ContainerDied","Data":"f5bae63f5eef12ede5de99227581d6d255dd02c5a99762fba2fef99294b846e5"} Jul 13 18:11:14.042403 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:14.041952 2564 scope.go:117] "RemoveContainer" containerID="3bdbdc0648f4cf1c5791fc30cd73329ef32464d756c2d2cd9456b2a10b84b756" Jul 13 18:11:14.042403 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:14.041902 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9" Jul 13 18:11:14.059572 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:14.059549 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9"] Jul 13 18:11:14.063791 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:14.063766 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a50684-f4594dbc9-m56w9"] Jul 13 18:11:15.638559 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:15.638525 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9634ab8f-4c52-4d90-b07a-0e3b311a109e" path="/var/lib/kubelet/pods/9634ab8f-4c52-4d90-b07a-0e3b311a109e/volumes" Jul 13 18:11:17.766458 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:17.766379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:23.759099 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:23.759049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:25.339614 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:25.339578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:28.501443 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.501411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:28.739087 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.739052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:28.871726 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.871698 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj"] Jul 13 18:11:28.872162 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.872150 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7099a5-90d1-4399-9023-c5a73bff6bd2" containerName="revoke-keys" Jul 13 18:11:28.872206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.872164 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7099a5-90d1-4399-9023-c5a73bff6bd2" containerName="revoke-keys" Jul 13 18:11:28.872206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.872189 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9634ab8f-4c52-4d90-b07a-0e3b311a109e" containerName="maas-api" Jul 13 18:11:28.872206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.872195 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9634ab8f-4c52-4d90-b07a-0e3b311a109e" containerName="maas-api" Jul 13 18:11:28.872327 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.872279 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7099a5-90d1-4399-9023-c5a73bff6bd2" containerName="revoke-keys" Jul 13 18:11:28.872327 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.872293 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9634ab8f-4c52-4d90-b07a-0e3b311a109e" containerName="maas-api" Jul 13 18:11:28.877042 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.877024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" Jul 13 18:11:28.879733 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.879625 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:11:28.882302 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:28.882277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj"] Jul 13 18:11:29.017747 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:29.017718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06472a-141c-4b56-acc0-aca6133091bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj\" (UID: \"4f06472a-141c-4b56-acc0-aca6133091bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" Jul 13 18:11:29.119156 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:29.119134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06472a-141c-4b56-acc0-aca6133091bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj\" (UID: \"4f06472a-141c-4b56-acc0-aca6133091bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" Jul 13 18:11:29.119771 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:29.119741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06472a-141c-4b56-acc0-aca6133091bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj\" (UID: \"4f06472a-141c-4b56-acc0-aca6133091bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" Jul 13 18:11:29.189211 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:29.189155 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" Jul 13 18:11:29.328101 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:29.328075 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj"] Jul 13 18:11:29.330021 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:11:29.329984 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f06472a_141c_4b56_acc0_aca6133091bd.slice/crio-292134fdde5d4f9ad9453f8963ed3117f96e1463958edb96d7bd419b739eec4b WatchSource:0}: Error finding container 292134fdde5d4f9ad9453f8963ed3117f96e1463958edb96d7bd419b739eec4b: Status 404 returned error can't find the container with id 292134fdde5d4f9ad9453f8963ed3117f96e1463958edb96d7bd419b739eec4b Jul 13 18:11:30.136548 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:30.136521 2564 generic.go:356] "Generic (PLEG): container finished" podID="4f06472a-141c-4b56-acc0-aca6133091bd" containerID="0548d7963fd5b033b9d8f3d9e4f174b40ab8f023b50dda064dff6bc89bfa1c86" exitCode=0 Jul 13 18:11:30.136880 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:30.136590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" event={"ID":"4f06472a-141c-4b56-acc0-aca6133091bd","Type":"ContainerDied","Data":"0548d7963fd5b033b9d8f3d9e4f174b40ab8f023b50dda064dff6bc89bfa1c86"} Jul 13 18:11:30.136880 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:30.136655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" event={"ID":"4f06472a-141c-4b56-acc0-aca6133091bd","Type":"ContainerStarted","Data":"292134fdde5d4f9ad9453f8963ed3117f96e1463958edb96d7bd419b739eec4b"} Jul 13 18:11:31.279416 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:31.279378 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" Jul 13 18:11:31.342890 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:31.342857 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06472a-141c-4b56-acc0-aca6133091bd-maas-api-service-ca\") pod \"4f06472a-141c-4b56-acc0-aca6133091bd\" (UID: \"4f06472a-141c-4b56-acc0-aca6133091bd\") " Jul 13 18:11:31.343271 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:31.343217 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f06472a-141c-4b56-acc0-aca6133091bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f06472a-141c-4b56-acc0-aca6133091bd" (UID: "4f06472a-141c-4b56-acc0-aca6133091bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:11:31.444659 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:31.444581 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f06472a-141c-4b56-acc0-aca6133091bd-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:32.149116 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:32.149087 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" Jul 13 18:11:32.149304 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:32.149127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj" event={"ID":"4f06472a-141c-4b56-acc0-aca6133091bd","Type":"ContainerDied","Data":"292134fdde5d4f9ad9453f8963ed3117f96e1463958edb96d7bd419b739eec4b"} Jul 13 18:11:32.149304 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:32.149175 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="292134fdde5d4f9ad9453f8963ed3117f96e1463958edb96d7bd419b739eec4b" Jul 13 18:11:39.311174 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.311140 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr"] Jul 13 18:11:39.311628 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.311411 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" podUID="45a3d97d-5e7d-43f5-8fc4-6d356491e459" containerName="maas-api" containerID="cri-o://07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c" gracePeriod=30 Jul 13 18:11:39.589106 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.589082 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:11:39.723468 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.723437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a3d97d-5e7d-43f5-8fc4-6d356491e459-openshift-service-ca\") pod \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " Jul 13 18:11:39.723630 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.723491 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl2sl\" (UniqueName: \"kubernetes.io/projected/45a3d97d-5e7d-43f5-8fc4-6d356491e459-kube-api-access-vl2sl\") pod \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " Jul 13 18:11:39.723630 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.723606 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls\") pod \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\" (UID: \"45a3d97d-5e7d-43f5-8fc4-6d356491e459\") " Jul 13 18:11:39.723856 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.723821 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a3d97d-5e7d-43f5-8fc4-6d356491e459-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45a3d97d-5e7d-43f5-8fc4-6d356491e459" (UID: "45a3d97d-5e7d-43f5-8fc4-6d356491e459"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:11:39.724046 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.724025 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a3d97d-5e7d-43f5-8fc4-6d356491e459-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:39.726271 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.726244 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45a3d97d-5e7d-43f5-8fc4-6d356491e459" (UID: "45a3d97d-5e7d-43f5-8fc4-6d356491e459"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:11:39.726361 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.726266 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a3d97d-5e7d-43f5-8fc4-6d356491e459-kube-api-access-vl2sl" (OuterVolumeSpecName: "kube-api-access-vl2sl") pod "45a3d97d-5e7d-43f5-8fc4-6d356491e459" (UID: "45a3d97d-5e7d-43f5-8fc4-6d356491e459"). InnerVolumeSpecName "kube-api-access-vl2sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:11:39.824460 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.824404 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2sl\" (UniqueName: \"kubernetes.io/projected/45a3d97d-5e7d-43f5-8fc4-6d356491e459-kube-api-access-vl2sl\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:39.824460 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:39.824427 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45a3d97d-5e7d-43f5-8fc4-6d356491e459-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:11:40.199030 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.198937 2564 generic.go:356] "Generic (PLEG): container finished" podID="45a3d97d-5e7d-43f5-8fc4-6d356491e459" containerID="07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c" exitCode=0 Jul 13 18:11:40.199030 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.198994 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" Jul 13 18:11:40.199206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.199021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" event={"ID":"45a3d97d-5e7d-43f5-8fc4-6d356491e459","Type":"ContainerDied","Data":"07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c"} Jul 13 18:11:40.199206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.199078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr" event={"ID":"45a3d97d-5e7d-43f5-8fc4-6d356491e459","Type":"ContainerDied","Data":"664a90db22998638cb4636258b5779956b7f74f6d76d50918d58624935d55c6e"} Jul 13 18:11:40.199206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.199102 2564 scope.go:117] "RemoveContainer" containerID="07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c" Jul 13 18:11:40.209081 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.209067 2564 scope.go:117] "RemoveContainer" containerID="07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c" Jul 13 18:11:40.209333 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:11:40.209316 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c\": container with ID starting with 07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c not found: ID does not exist" containerID="07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c" Jul 13 18:11:40.209389 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.209343 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c"} err="failed to get container status \"07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c\": rpc error: code = NotFound desc = could not find container \"07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c\": container with ID starting with 07b2ea4e05e61c63ede29c62bc1f565fe2bef737801c19f8fe7f6cd01f1db51c not found: ID does not exist" Jul 13 18:11:40.220939 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.220918 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr"] Jul 13 18:11:40.225268 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.225244 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b8e947-6cdf4dc4bf-cxsxr"] Jul 13 18:11:40.865674 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:40.865640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:41.637712 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:41.637676 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45a3d97d-5e7d-43f5-8fc4-6d356491e459" path="/var/lib/kubelet/pods/45a3d97d-5e7d-43f5-8fc4-6d356491e459/volumes" Jul 13 18:11:43.967836 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:43.967788 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:49.979961 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:49.979933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:50.080096 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:50.080062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:54.680402 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:54.680374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:54.925090 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:54.925055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:55.289641 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:55.289603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:55.631934 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:55.631840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:56.669558 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:56.669523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:58.261160 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.261127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:58.360785 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.360753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:58.529222 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.529128 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc"] Jul 13 18:11:58.529856 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.529829 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f06472a-141c-4b56-acc0-aca6133091bd" containerName="revoke-keys" Jul 13 18:11:58.530013 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.529854 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f06472a-141c-4b56-acc0-aca6133091bd" containerName="revoke-keys" Jul 13 18:11:58.530013 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.529908 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a3d97d-5e7d-43f5-8fc4-6d356491e459" containerName="maas-api" Jul 13 18:11:58.530013 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.529917 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a3d97d-5e7d-43f5-8fc4-6d356491e459" containerName="maas-api" Jul 13 18:11:58.530155 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.530016 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f06472a-141c-4b56-acc0-aca6133091bd" containerName="revoke-keys" Jul 13 18:11:58.530155 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.530029 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45a3d97d-5e7d-43f5-8fc4-6d356491e459" containerName="maas-api" Jul 13 18:11:58.534961 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.534933 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.536925 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.536904 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a74fcd-serving-cert\"" Jul 13 18:11:58.541222 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.541199 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc"] Jul 13 18:11:58.574751 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.574703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:58.600389 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.600364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb546b49-4afe-41ea-9169-607f65f9c6ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.600503 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.600400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb546b49-4afe-41ea-9169-607f65f9c6ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.600503 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.600493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25md\" (UniqueName: \"kubernetes.io/projected/cb546b49-4afe-41ea-9169-607f65f9c6ab-kube-api-access-b25md\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.673147 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.673123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:58.701633 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.701610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb546b49-4afe-41ea-9169-607f65f9c6ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.701731 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.701643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb546b49-4afe-41ea-9169-607f65f9c6ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.701731 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.701692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25md\" (UniqueName: \"kubernetes.io/projected/cb546b49-4afe-41ea-9169-607f65f9c6ab-kube-api-access-b25md\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.702225 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.702203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb546b49-4afe-41ea-9169-607f65f9c6ab-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.704059 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.704038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb546b49-4afe-41ea-9169-607f65f9c6ab-maas-api-tls\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.710148 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.710126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25md\" (UniqueName: \"kubernetes.io/projected/cb546b49-4afe-41ea-9169-607f65f9c6ab-kube-api-access-b25md\") pod \"maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.848121 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.848091 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:11:58.986104 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:58.986081 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc"] Jul 13 18:11:58.987979 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:11:58.987951 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb546b49_4afe_41ea_9169_607f65f9c6ab.slice/crio-0e1ef1d6c0a475d2241d6099bbc5e2ba3f73aede8f33874493539b51cbf41abf WatchSource:0}: Error finding container 0e1ef1d6c0a475d2241d6099bbc5e2ba3f73aede8f33874493539b51cbf41abf: Status 404 returned error can't find the container with id 0e1ef1d6c0a475d2241d6099bbc5e2ba3f73aede8f33874493539b51cbf41abf Jul 13 18:11:59.028802 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:59.028776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:59.129111 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:59.129050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:11:59.311737 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:11:59.311697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" event={"ID":"cb546b49-4afe-41ea-9169-607f65f9c6ab","Type":"ContainerStarted","Data":"0e1ef1d6c0a475d2241d6099bbc5e2ba3f73aede8f33874493539b51cbf41abf"} Jul 13 18:12:00.319288 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:00.319224 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" event={"ID":"cb546b49-4afe-41ea-9169-607f65f9c6ab","Type":"ContainerStarted","Data":"d9b3c5b161a241ef45885a9fa81eb590a78e5637bfbf6dcbbf29c263a41fec12"} Jul 13 18:12:00.319781 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:00.319387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:12:00.336999 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:00.336955 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" podStartSLOduration=1.4199921 podStartE2EDuration="2.336941696s" totalImagesPullingTime="916.949596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:12:00.335318698 +0000 UTC m=+2375.342720942" watchObservedRunningTime="2026-07-13 18:12:00.336941696 +0000 UTC m=+2375.344343939" Jul 13 18:12:02.801070 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:02.801028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:03.197116 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:03.197026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:03.523520 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:03.523427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:04.291076 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:04.291044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:06.331450 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.331420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:06.332000 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.331982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:12:06.427182 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.427147 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67"] Jul 13 18:12:06.429890 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.429855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:06.431624 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.431605 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.433874 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.433852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-762866-serving-cert\"" Jul 13 18:12:06.440407 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.440382 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67"] Jul 13 18:12:06.481696 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.481661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.481854 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.481740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6855\" (UniqueName: \"kubernetes.io/projected/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-kube-api-access-q6855\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.481854 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.481770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.505130 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.505097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:06.582351 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.582276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6855\" (UniqueName: \"kubernetes.io/projected/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-kube-api-access-q6855\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.582351 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.582331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.582533 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.582403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.582533 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:12:06.582510 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-serving-cert: secret "maas-api-e2e-shared-b-762866-serving-cert" not found Jul 13 18:12:06.582603 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:12:06.582573 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls podName:b40b3eb5-fa1c-4370-b638-4a8c75be7aa8 nodeName:}" failed. No retries permitted until 2026-07-13 18:12:07.082554201 +0000 UTC m=+2382.089956423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls") pod "maas-api-e2e-shared-b-762866-647fc95b8-ckd67" (UID: "b40b3eb5-fa1c-4370-b638-4a8c75be7aa8") : secret "maas-api-e2e-shared-b-762866-serving-cert" not found Jul 13 18:12:06.582979 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.582958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:06.591787 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:06.591765 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6855\" (UniqueName: \"kubernetes.io/projected/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-kube-api-access-q6855\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:07.088261 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:07.088205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:07.090671 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:07.090642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls\") pod \"maas-api-e2e-shared-b-762866-647fc95b8-ckd67\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:07.344565 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:07.344478 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:07.487482 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:07.487448 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67"] Jul 13 18:12:07.491150 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:12:07.491108 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40b3eb5_fa1c_4370_b638_4a8c75be7aa8.slice/crio-b020a37c2940c7d483d3968f466dd24840375fbafef486109600ed19d7f39885 WatchSource:0}: Error finding container b020a37c2940c7d483d3968f466dd24840375fbafef486109600ed19d7f39885: Status 404 returned error can't find the container with id b020a37c2940c7d483d3968f466dd24840375fbafef486109600ed19d7f39885 Jul 13 18:12:07.689631 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:07.689540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:07.789538 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:07.789480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:08.374927 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:08.374884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" event={"ID":"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8","Type":"ContainerStarted","Data":"bcfb1e91ecdf0f8937982a7e89d513e5c2e575e960ea049fab1815d9036b5ca4"} Jul 13 18:12:08.375402 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:08.375098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:08.375402 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:08.375146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" event={"ID":"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8","Type":"ContainerStarted","Data":"b020a37c2940c7d483d3968f466dd24840375fbafef486109600ed19d7f39885"} Jul 13 18:12:08.393118 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:08.393069 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" podStartSLOduration=2.027239575 podStartE2EDuration="2.393055784s" totalImagesPullingTime="365.816209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:12:08.391339987 +0000 UTC m=+2383.398742229" watchObservedRunningTime="2026-07-13 18:12:08.393055784 +0000 UTC m=+2383.400458026" Jul 13 18:12:10.850173 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:10.850121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:14.386508 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:14.386479 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:12:20.205634 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:20.205589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:20.833934 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:20.833898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:21.659487 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:21.659441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:25.706891 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:25.706784 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:12:25.716261 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:25.716224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:12:29.256547 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:29.256513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:29.308202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:29.308172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:41.802521 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:41.802488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:41.866800 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:41.866767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:12:49.433206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:12:49.433122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:13:37.412676 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:13:37.412636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:14:55.404935 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:14:55.404895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:14:56.367086 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:14:56.367051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:14:57.523257 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:14:57.523201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:14:57.624022 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:14:57.623987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:15:00.138073 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.138029 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2"] Jul 13 18:15:00.142605 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.142576 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69"] Jul 13 18:15:00.142750 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.142690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" Jul 13 18:15:00.145190 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.144992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:15:00.149450 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.149423 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw"] Jul 13 18:15:00.149909 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.149883 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" Jul 13 18:15:00.155804 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.155655 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" Jul 13 18:15:00.155991 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.155974 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2"] Jul 13 18:15:00.156125 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.156112 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69"] Jul 13 18:15:00.158148 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.157956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" Jul 13 18:15:00.158346 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.158324 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw"] Jul 13 18:15:00.172369 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.172344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" Jul 13 18:15:00.213616 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.213587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" Jul 13 18:15:00.342701 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.342658 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2"] Jul 13 18:15:00.344472 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:15:00.344428 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70f6e12_0326_43aa_affa_fdb73861d236.slice/crio-af59a378df5370cb3895fd06d21e4022361c1aae5228e067772a5b7b071db2a9 WatchSource:0}: Error finding container af59a378df5370cb3895fd06d21e4022361c1aae5228e067772a5b7b071db2a9: Status 404 returned error can't find the container with id af59a378df5370cb3895fd06d21e4022361c1aae5228e067772a5b7b071db2a9 Jul 13 18:15:00.371450 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.371417 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69"] Jul 13 18:15:00.371647 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:15:00.371496 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c8c8f5_9f94_44e9_8964_cf0c6832f70d.slice/crio-3a93f7c3171cc98311dc2173935301a1959439df5e38ca969148328d49633976 WatchSource:0}: Error finding container 3a93f7c3171cc98311dc2173935301a1959439df5e38ca969148328d49633976: Status 404 returned error can't find the container with id 3a93f7c3171cc98311dc2173935301a1959439df5e38ca969148328d49633976 Jul 13 18:15:00.392794 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:00.392765 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw"] Jul 13 18:15:00.393832 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:15:00.393802 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a49836_9d46_4f12_8f3c_e933d462b223.slice/crio-17121df2502c05143c16ffc34abb43ad484311929e0b191cf1440ca1b89bac9d WatchSource:0}: Error finding container 17121df2502c05143c16ffc34abb43ad484311929e0b191cf1440ca1b89bac9d: Status 404 returned error can't find the container with id 17121df2502c05143c16ffc34abb43ad484311929e0b191cf1440ca1b89bac9d Jul 13 18:15:01.350005 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.349966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" event={"ID":"65a49836-9d46-4f12-8f3c-e933d462b223","Type":"ContainerStarted","Data":"0588b19815126ae3ea15a1f81a7e17fb1b67146e7db96fd87e1e68f065bdaee4"} Jul 13 18:15:01.350466 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.350056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" event={"ID":"65a49836-9d46-4f12-8f3c-e933d462b223","Type":"ContainerStarted","Data":"17121df2502c05143c16ffc34abb43ad484311929e0b191cf1440ca1b89bac9d"} Jul 13 18:15:01.351580 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.351553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" event={"ID":"d9c8c8f5-9f94-44e9-8964-cf0c6832f70d","Type":"ContainerStarted","Data":"a6da2304bbbd8a954d16d83f20fbe68ad6f43beae1f7dde87efbbb68b144bf04"} Jul 13 18:15:01.351700 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.351599 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" event={"ID":"d9c8c8f5-9f94-44e9-8964-cf0c6832f70d","Type":"ContainerStarted","Data":"3a93f7c3171cc98311dc2173935301a1959439df5e38ca969148328d49633976"} Jul 13 18:15:01.352877 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.352853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" event={"ID":"a70f6e12-0326-43aa-affa-fdb73861d236","Type":"ContainerStarted","Data":"3dbfe82a9acd9ba96ba595ae6a838be47289c49258a23b1297c1f288eadc8b9e"} Jul 13 18:15:01.352966 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.352894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" event={"ID":"a70f6e12-0326-43aa-affa-fdb73861d236","Type":"ContainerStarted","Data":"af59a378df5370cb3895fd06d21e4022361c1aae5228e067772a5b7b071db2a9"} Jul 13 18:15:01.367672 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.367622 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" podStartSLOduration=1.367604938 podStartE2EDuration="1.367604938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:15:01.365166064 +0000 UTC m=+2556.372568308" watchObservedRunningTime="2026-07-13 18:15:01.367604938 +0000 UTC m=+2556.375007240" Jul 13 18:15:01.384574 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.384525 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" podStartSLOduration=1.38450922 podStartE2EDuration="1.38450922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:15:01.383680847 +0000 UTC m=+2556.391083092" watchObservedRunningTime="2026-07-13 18:15:01.38450922 +0000 UTC m=+2556.391911462" Jul 13 18:15:01.398407 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:01.398362 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" podStartSLOduration=1.398347921 podStartE2EDuration="1.398347921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:15:01.39701732 +0000 UTC m=+2556.404419563" watchObservedRunningTime="2026-07-13 18:15:01.398347921 +0000 UTC m=+2556.405750164" Jul 13 18:15:02.360691 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:02.360653 2564 generic.go:356] "Generic (PLEG): container finished" podID="a70f6e12-0326-43aa-affa-fdb73861d236" containerID="3dbfe82a9acd9ba96ba595ae6a838be47289c49258a23b1297c1f288eadc8b9e" exitCode=0 Jul 13 18:15:02.361125 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:02.360732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" event={"ID":"a70f6e12-0326-43aa-affa-fdb73861d236","Type":"ContainerDied","Data":"3dbfe82a9acd9ba96ba595ae6a838be47289c49258a23b1297c1f288eadc8b9e"} Jul 13 18:15:02.362391 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:02.362365 2564 generic.go:356] "Generic (PLEG): container finished" podID="65a49836-9d46-4f12-8f3c-e933d462b223" containerID="0588b19815126ae3ea15a1f81a7e17fb1b67146e7db96fd87e1e68f065bdaee4" exitCode=0 Jul 13 18:15:02.362548 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:02.362443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" event={"ID":"65a49836-9d46-4f12-8f3c-e933d462b223","Type":"ContainerDied","Data":"0588b19815126ae3ea15a1f81a7e17fb1b67146e7db96fd87e1e68f065bdaee4"} Jul 13 18:15:02.364110 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:02.364088 2564 generic.go:356] "Generic (PLEG): container finished" podID="d9c8c8f5-9f94-44e9-8964-cf0c6832f70d" containerID="a6da2304bbbd8a954d16d83f20fbe68ad6f43beae1f7dde87efbbb68b144bf04" exitCode=0 Jul 13 18:15:02.364244 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:02.364148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" event={"ID":"d9c8c8f5-9f94-44e9-8964-cf0c6832f70d","Type":"ContainerDied","Data":"a6da2304bbbd8a954d16d83f20fbe68ad6f43beae1f7dde87efbbb68b144bf04"} Jul 13 18:15:03.528311 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:03.528283 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" Jul 13 18:15:03.561955 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:03.561926 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" Jul 13 18:15:03.592710 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:03.592683 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" Jul 13 18:15:04.377384 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.377359 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" Jul 13 18:15:04.377563 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.377356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2" event={"ID":"a70f6e12-0326-43aa-affa-fdb73861d236","Type":"ContainerDied","Data":"af59a378df5370cb3895fd06d21e4022361c1aae5228e067772a5b7b071db2a9"} Jul 13 18:15:04.377563 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.377488 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af59a378df5370cb3895fd06d21e4022361c1aae5228e067772a5b7b071db2a9" Jul 13 18:15:04.378665 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.378646 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" Jul 13 18:15:04.378781 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.378686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732775-tkcjw" event={"ID":"65a49836-9d46-4f12-8f3c-e933d462b223","Type":"ContainerDied","Data":"17121df2502c05143c16ffc34abb43ad484311929e0b191cf1440ca1b89bac9d"} Jul 13 18:15:04.378781 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.378732 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17121df2502c05143c16ffc34abb43ad484311929e0b191cf1440ca1b89bac9d" Jul 13 18:15:04.380012 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.379993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" event={"ID":"d9c8c8f5-9f94-44e9-8964-cf0c6832f70d","Type":"ContainerDied","Data":"3a93f7c3171cc98311dc2173935301a1959439df5e38ca969148328d49633976"} Jul 13 18:15:04.380012 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.380005 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69" Jul 13 18:15:04.380153 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.380027 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a93f7c3171cc98311dc2173935301a1959439df5e38ca969148328d49633976" Jul 13 18:15:04.854866 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:04.854829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:15:19.661186 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:19.661146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:15:56.240052 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:15:56.239947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:16:14.090909 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:16:14.090874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:16:14.936061 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:16:14.936028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:16:24.545347 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:16:24.545313 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:16:31.758819 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:16:31.758779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:16:37.044374 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:16:37.044340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:05.452797 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:05.452755 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:22.245900 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:22.245861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:22.354890 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:22.354857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:25.750676 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:25.750568 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:17:25.761188 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:25.761167 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74zgv_f88cd6c6-7717-46de-8c19-a15c236db534/ovn-acl-logging/0.log" Jul 13 18:17:30.021535 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:30.021501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:30.123916 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:30.123886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:53.818112 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:53.818078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:54.720720 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:54.720686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:17:55.832826 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.832796 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7496-140575038006320"] Jul 13 18:17:55.833292 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833278 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c8c8f5-9f94-44e9-8964-cf0c6832f70d" containerName="cleanup" Jul 13 18:17:55.833335 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833293 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c8c8f5-9f94-44e9-8964-cf0c6832f70d" containerName="cleanup" Jul 13 18:17:55.833335 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833304 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70f6e12-0326-43aa-affa-fdb73861d236" containerName="cleanup" Jul 13 18:17:55.833335 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833311 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70f6e12-0326-43aa-affa-fdb73861d236" containerName="cleanup" Jul 13 18:17:55.833436 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833339 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a49836-9d46-4f12-8f3c-e933d462b223" containerName="cleanup" Jul 13 18:17:55.833436 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833347 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a49836-9d46-4f12-8f3c-e933d462b223" containerName="cleanup" Jul 13 18:17:55.833436 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833417 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c8c8f5-9f94-44e9-8964-cf0c6832f70d" containerName="cleanup" Jul 13 18:17:55.833436 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833429 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a49836-9d46-4f12-8f3c-e933d462b223" containerName="cleanup" Jul 13 18:17:55.833436 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.833435 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70f6e12-0326-43aa-affa-fdb73861d236" containerName="cleanup" Jul 13 18:17:55.836788 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.836773 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:55.839372 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.839336 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-dl4kl\"" Jul 13 18:17:55.843472 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.843448 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7496-140575038006320"] Jul 13 18:17:55.892934 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.892901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82ld\" (UniqueName: \"kubernetes.io/projected/bd8d957f-cb2d-40ef-9a06-fa1e00e13f59-kube-api-access-g82ld\") pod \"test-curl-7496-140575038006320\" (UID: \"bd8d957f-cb2d-40ef-9a06-fa1e00e13f59\") " pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:55.993926 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:55.993891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82ld\" (UniqueName: \"kubernetes.io/projected/bd8d957f-cb2d-40ef-9a06-fa1e00e13f59-kube-api-access-g82ld\") pod \"test-curl-7496-140575038006320\" (UID: \"bd8d957f-cb2d-40ef-9a06-fa1e00e13f59\") " pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:56.003856 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:56.003830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82ld\" (UniqueName: \"kubernetes.io/projected/bd8d957f-cb2d-40ef-9a06-fa1e00e13f59-kube-api-access-g82ld\") pod \"test-curl-7496-140575038006320\" (UID: \"bd8d957f-cb2d-40ef-9a06-fa1e00e13f59\") " pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:56.147155 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:56.147060 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:56.284156 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:56.284132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7496-140575038006320"] Jul 13 18:17:56.285693 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:17:56.285665 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8d957f_cb2d_40ef_9a06_fa1e00e13f59.slice/crio-3a2d34d02fc2075cae765f98e0352c1ce30c0e95fceef750dbf613ee962fc53d WatchSource:0}: Error finding container 3a2d34d02fc2075cae765f98e0352c1ce30c0e95fceef750dbf613ee962fc53d: Status 404 returned error can't find the container with id 3a2d34d02fc2075cae765f98e0352c1ce30c0e95fceef750dbf613ee962fc53d Jul 13 18:17:56.286962 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:56.286933 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 18:17:56.392207 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:56.392167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7496-140575038006320" event={"ID":"bd8d957f-cb2d-40ef-9a06-fa1e00e13f59","Type":"ContainerStarted","Data":"3a2d34d02fc2075cae765f98e0352c1ce30c0e95fceef750dbf613ee962fc53d"} Jul 13 18:17:57.398861 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:57.398822 2564 generic.go:356] "Generic (PLEG): container finished" podID="bd8d957f-cb2d-40ef-9a06-fa1e00e13f59" containerID="8bf64a0c8029d8b8c62a20381bd71de8ec99610ed7f83d7c02b0b8b56bbc0298" exitCode=0 Jul 13 18:17:57.399296 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:57.398905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7496-140575038006320" event={"ID":"bd8d957f-cb2d-40ef-9a06-fa1e00e13f59","Type":"ContainerDied","Data":"8bf64a0c8029d8b8c62a20381bd71de8ec99610ed7f83d7c02b0b8b56bbc0298"} Jul 13 18:17:58.551793 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:58.551772 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:58.572561 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:58.572531 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7496-140575038006320_bd8d957f-cb2d-40ef-9a06-fa1e00e13f59/test-curl-7496-140575038006320/0.log" Jul 13 18:17:58.599717 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:58.599676 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7496-140575038006320"] Jul 13 18:17:58.602190 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:58.602168 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7496-140575038006320"] Jul 13 18:17:58.621138 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:58.621118 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g82ld\" (UniqueName: \"kubernetes.io/projected/bd8d957f-cb2d-40ef-9a06-fa1e00e13f59-kube-api-access-g82ld\") pod \"bd8d957f-cb2d-40ef-9a06-fa1e00e13f59\" (UID: \"bd8d957f-cb2d-40ef-9a06-fa1e00e13f59\") " Jul 13 18:17:58.623202 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:58.623183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8d957f-cb2d-40ef-9a06-fa1e00e13f59-kube-api-access-g82ld" (OuterVolumeSpecName: "kube-api-access-g82ld") pod "bd8d957f-cb2d-40ef-9a06-fa1e00e13f59" (UID: "bd8d957f-cb2d-40ef-9a06-fa1e00e13f59"). InnerVolumeSpecName "kube-api-access-g82ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:17:58.722160 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:58.722083 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g82ld\" (UniqueName: \"kubernetes.io/projected/bd8d957f-cb2d-40ef-9a06-fa1e00e13f59-kube-api-access-g82ld\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:17:59.411753 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:59.411722 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a2d34d02fc2075cae765f98e0352c1ce30c0e95fceef750dbf613ee962fc53d" Jul 13 18:17:59.411753 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:59.411730 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:59.421327 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:17:59.421293 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-140575038006320\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="bd8d957f-cb2d-40ef-9a06-fa1e00e13f59" pod="openshift-ingress/test-curl-7496-140575038006320" Jul 13 18:17:59.638505 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:17:59.638476 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8d957f-cb2d-40ef-9a06-fa1e00e13f59" path="/var/lib/kubelet/pods/bd8d957f-cb2d-40ef-9a06-fa1e00e13f59/volumes" Jul 13 18:18:01.818706 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.818675 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7496-140575035473968"] Jul 13 18:18:01.819147 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.819132 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8d957f-cb2d-40ef-9a06-fa1e00e13f59" containerName="test-curl-7496-140575038006320" Jul 13 18:18:01.819191 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.819148 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8d957f-cb2d-40ef-9a06-fa1e00e13f59" containerName="test-curl-7496-140575038006320" Jul 13 18:18:01.819243 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.819214 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8d957f-cb2d-40ef-9a06-fa1e00e13f59" containerName="test-curl-7496-140575038006320" Jul 13 18:18:01.822566 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.822548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:01.824749 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.824728 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-dl4kl\"" Jul 13 18:18:01.828656 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.828631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7496-140575035473968"] Jul 13 18:18:01.853596 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.853575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hjz\" (UniqueName: \"kubernetes.io/projected/cec87c2f-0781-405a-89e9-ba6afeaa649f-kube-api-access-z6hjz\") pod \"test-curl-7496-140575035473968\" (UID: \"cec87c2f-0781-405a-89e9-ba6afeaa649f\") " pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:01.954689 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.954662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hjz\" (UniqueName: \"kubernetes.io/projected/cec87c2f-0781-405a-89e9-ba6afeaa649f-kube-api-access-z6hjz\") pod \"test-curl-7496-140575035473968\" (UID: \"cec87c2f-0781-405a-89e9-ba6afeaa649f\") " pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:01.962784 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:01.962763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hjz\" (UniqueName: \"kubernetes.io/projected/cec87c2f-0781-405a-89e9-ba6afeaa649f-kube-api-access-z6hjz\") pod \"test-curl-7496-140575035473968\" (UID: \"cec87c2f-0781-405a-89e9-ba6afeaa649f\") " pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:02.132973 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:02.132911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:02.263493 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:02.263469 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7496-140575035473968"] Jul 13 18:18:02.264734 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:18:02.264706 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcec87c2f_0781_405a_89e9_ba6afeaa649f.slice/crio-be349279aacc8d11aab2273dbe52638d99bc5dea4ec2c6daf381f5183abac04c WatchSource:0}: Error finding container be349279aacc8d11aab2273dbe52638d99bc5dea4ec2c6daf381f5183abac04c: Status 404 returned error can't find the container with id be349279aacc8d11aab2273dbe52638d99bc5dea4ec2c6daf381f5183abac04c Jul 13 18:18:02.429971 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:02.429897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7496-140575035473968" event={"ID":"cec87c2f-0781-405a-89e9-ba6afeaa649f","Type":"ContainerStarted","Data":"be349279aacc8d11aab2273dbe52638d99bc5dea4ec2c6daf381f5183abac04c"} Jul 13 18:18:03.437530 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:03.437490 2564 generic.go:356] "Generic (PLEG): container finished" podID="cec87c2f-0781-405a-89e9-ba6afeaa649f" containerID="306dee14487b4c245a6c8b5d4f47c660ab307a0026afd95fe0139aa5fda401ee" exitCode=0 Jul 13 18:18:03.437992 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:03.437718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7496-140575035473968" event={"ID":"cec87c2f-0781-405a-89e9-ba6afeaa649f","Type":"ContainerDied","Data":"306dee14487b4c245a6c8b5d4f47c660ab307a0026afd95fe0139aa5fda401ee"} Jul 13 18:18:04.594102 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:04.594078 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:04.614995 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:04.614939 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7496-140575035473968_cec87c2f-0781-405a-89e9-ba6afeaa649f/test-curl-7496-140575035473968/0.log" Jul 13 18:18:04.644842 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:04.644817 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7496-140575035473968"] Jul 13 18:18:04.647320 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:04.647296 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7496-140575035473968"] Jul 13 18:18:04.679423 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:04.679390 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6hjz\" (UniqueName: \"kubernetes.io/projected/cec87c2f-0781-405a-89e9-ba6afeaa649f-kube-api-access-z6hjz\") pod \"cec87c2f-0781-405a-89e9-ba6afeaa649f\" (UID: \"cec87c2f-0781-405a-89e9-ba6afeaa649f\") " Jul 13 18:18:04.681441 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:04.681417 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec87c2f-0781-405a-89e9-ba6afeaa649f-kube-api-access-z6hjz" (OuterVolumeSpecName: "kube-api-access-z6hjz") pod "cec87c2f-0781-405a-89e9-ba6afeaa649f" (UID: "cec87c2f-0781-405a-89e9-ba6afeaa649f"). InnerVolumeSpecName "kube-api-access-z6hjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:18:04.780646 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:04.780591 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hjz\" (UniqueName: \"kubernetes.io/projected/cec87c2f-0781-405a-89e9-ba6afeaa649f-kube-api-access-z6hjz\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:05.449470 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:05.449445 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:05.449621 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:05.449489 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be349279aacc8d11aab2273dbe52638d99bc5dea4ec2c6daf381f5183abac04c" Jul 13 18:18:05.458895 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:18:05.458870 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-140575035473968\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="cec87c2f-0781-405a-89e9-ba6afeaa649f" pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:05.636950 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:18:05.636883 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-140575035473968\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="cec87c2f-0781-405a-89e9-ba6afeaa649f" pod="openshift-ingress/test-curl-7496-140575035473968" Jul 13 18:18:05.639326 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:05.639305 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec87c2f-0781-405a-89e9-ba6afeaa649f" path="/var/lib/kubelet/pods/cec87c2f-0781-405a-89e9-ba6afeaa649f/volumes" Jul 13 18:18:07.775272 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.775243 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7496-140575035474976"] Jul 13 18:18:07.775709 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.775695 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec87c2f-0781-405a-89e9-ba6afeaa649f" containerName="test-curl-7496-140575035473968" Jul 13 18:18:07.775754 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.775711 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec87c2f-0781-405a-89e9-ba6afeaa649f" containerName="test-curl-7496-140575035473968" Jul 13 18:18:07.775794 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.775773 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec87c2f-0781-405a-89e9-ba6afeaa649f" containerName="test-curl-7496-140575035473968" Jul 13 18:18:07.778896 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.778880 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:07.780805 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.780787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-dl4kl\"" Jul 13 18:18:07.789840 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.789815 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7496-140575035474976"] Jul 13 18:18:07.909999 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:07.909971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt6bg\" (UniqueName: \"kubernetes.io/projected/5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447-kube-api-access-wt6bg\") pod \"test-curl-7496-140575035474976\" (UID: \"5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447\") " pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:08.010978 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:08.010936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt6bg\" (UniqueName: \"kubernetes.io/projected/5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447-kube-api-access-wt6bg\") pod \"test-curl-7496-140575035474976\" (UID: \"5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447\") " pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:08.019103 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:08.019079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt6bg\" (UniqueName: \"kubernetes.io/projected/5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447-kube-api-access-wt6bg\") pod \"test-curl-7496-140575035474976\" (UID: \"5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447\") " pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:08.098355 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:08.098327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:08.237623 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:08.237596 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7496-140575035474976"] Jul 13 18:18:08.238803 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:18:08.238769 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a00aeb2_3fb9_40a8_9c06_7c1aee1d0447.slice/crio-aed3b3d57564ffcd86212a07ef9bd27ee18ae400beae600ce5e181f72964b3b6 WatchSource:0}: Error finding container aed3b3d57564ffcd86212a07ef9bd27ee18ae400beae600ce5e181f72964b3b6: Status 404 returned error can't find the container with id aed3b3d57564ffcd86212a07ef9bd27ee18ae400beae600ce5e181f72964b3b6 Jul 13 18:18:08.469823 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:08.469731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7496-140575035474976" event={"ID":"5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447","Type":"ContainerStarted","Data":"aed3b3d57564ffcd86212a07ef9bd27ee18ae400beae600ce5e181f72964b3b6"} Jul 13 18:18:09.481100 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:09.481066 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447" containerID="bc54ed1213cea2a64221e24e163091f24dffc53c0e953262dcfd3c086353cd99" exitCode=0 Jul 13 18:18:09.481538 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:09.481139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7496-140575035474976" event={"ID":"5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447","Type":"ContainerDied","Data":"bc54ed1213cea2a64221e24e163091f24dffc53c0e953262dcfd3c086353cd99"} Jul 13 18:18:10.629910 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:10.629884 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:10.649206 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:10.649182 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7496-140575035474976_5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447/test-curl-7496-140575035474976/0.log" Jul 13 18:18:10.675398 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:10.675366 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7496-140575035474976"] Jul 13 18:18:10.677841 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:10.677817 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7496-140575035474976"] Jul 13 18:18:10.736493 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:10.736459 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt6bg\" (UniqueName: \"kubernetes.io/projected/5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447-kube-api-access-wt6bg\") pod \"5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447\" (UID: \"5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447\") " Jul 13 18:18:10.738635 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:10.738570 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447-kube-api-access-wt6bg" (OuterVolumeSpecName: "kube-api-access-wt6bg") pod "5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447" (UID: "5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447"). InnerVolumeSpecName "kube-api-access-wt6bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:18:10.837527 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:10.837493 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt6bg\" (UniqueName: \"kubernetes.io/projected/5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447-kube-api-access-wt6bg\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:11.493973 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:11.493944 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed3b3d57564ffcd86212a07ef9bd27ee18ae400beae600ce5e181f72964b3b6" Jul 13 18:18:11.493973 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:11.493978 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:11.503459 ip-10-0-139-16 kubenswrapper[2564]: E0713 18:18:11.503431 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-140575035474976\" is forbidden: User \"system:node:ip-10-0-139-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-16.ec2.internal' and this object" podUID="5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447" pod="openshift-ingress/test-curl-7496-140575035474976" Jul 13 18:18:11.638632 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:11.638600 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447" path="/var/lib/kubelet/pods/5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447/volumes" Jul 13 18:18:13.875407 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.875368 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h"] Jul 13 18:18:13.875890 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.875875 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447" containerName="test-curl-7496-140575035474976" Jul 13 18:18:13.875931 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.875892 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447" containerName="test-curl-7496-140575035474976" Jul 13 18:18:13.876010 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.875999 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a00aeb2-3fb9-40a8-9c06-7c1aee1d0447" containerName="test-curl-7496-140575035474976" Jul 13 18:18:13.879346 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.879325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" Jul 13 18:18:13.881544 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.881521 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:18:13.886252 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.886203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h"] Jul 13 18:18:13.970254 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:13.970202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ea988a-db2b-41ec-a328-56900374408f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h\" (UID: \"d8ea988a-db2b-41ec-a328-56900374408f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" Jul 13 18:18:14.071189 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:14.071155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ea988a-db2b-41ec-a328-56900374408f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h\" (UID: \"d8ea988a-db2b-41ec-a328-56900374408f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" Jul 13 18:18:14.071808 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:14.071788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ea988a-db2b-41ec-a328-56900374408f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h\" (UID: \"d8ea988a-db2b-41ec-a328-56900374408f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" Jul 13 18:18:14.192929 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:14.192836 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" Jul 13 18:18:14.343016 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:14.342992 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h"] Jul 13 18:18:14.345358 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:18:14.345330 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ea988a_db2b_41ec_a328_56900374408f.slice/crio-2c21a3e2de01d2504d303d82527ab696e211258fa76974f989629331bd46fa21 WatchSource:0}: Error finding container 2c21a3e2de01d2504d303d82527ab696e211258fa76974f989629331bd46fa21: Status 404 returned error can't find the container with id 2c21a3e2de01d2504d303d82527ab696e211258fa76974f989629331bd46fa21 Jul 13 18:18:14.462026 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:14.461938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:14.512797 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:14.512760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" event={"ID":"d8ea988a-db2b-41ec-a328-56900374408f","Type":"ContainerStarted","Data":"2c21a3e2de01d2504d303d82527ab696e211258fa76974f989629331bd46fa21"} Jul 13 18:18:14.562268 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:14.562216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:15.519579 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:15.519488 2564 generic.go:356] "Generic (PLEG): container finished" podID="d8ea988a-db2b-41ec-a328-56900374408f" containerID="473cfb99266f21991f035c19b054eccd411ebeb9e9c63cf5201ac739634ae329" exitCode=0 Jul 13 18:18:15.519579 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:15.519564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" event={"ID":"d8ea988a-db2b-41ec-a328-56900374408f","Type":"ContainerDied","Data":"473cfb99266f21991f035c19b054eccd411ebeb9e9c63cf5201ac739634ae329"} Jul 13 18:18:16.668726 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:16.668700 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" Jul 13 18:18:16.800252 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:16.800200 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ea988a-db2b-41ec-a328-56900374408f-maas-api-service-ca\") pod \"d8ea988a-db2b-41ec-a328-56900374408f\" (UID: \"d8ea988a-db2b-41ec-a328-56900374408f\") " Jul 13 18:18:16.800617 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:16.800589 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ea988a-db2b-41ec-a328-56900374408f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8ea988a-db2b-41ec-a328-56900374408f" (UID: "d8ea988a-db2b-41ec-a328-56900374408f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:18:16.901639 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:16.901597 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ea988a-db2b-41ec-a328-56900374408f-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:17.532759 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:17.532728 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" Jul 13 18:18:17.532759 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:17.532737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h" event={"ID":"d8ea988a-db2b-41ec-a328-56900374408f","Type":"ContainerDied","Data":"2c21a3e2de01d2504d303d82527ab696e211258fa76974f989629331bd46fa21"} Jul 13 18:18:17.532976 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:17.532783 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c21a3e2de01d2504d303d82527ab696e211258fa76974f989629331bd46fa21" Jul 13 18:18:24.342378 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.342348 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc"] Jul 13 18:18:24.342876 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.342537 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" podUID="cb546b49-4afe-41ea-9169-607f65f9c6ab" containerName="maas-api" containerID="cri-o://d9b3c5b161a241ef45885a9fa81eb590a78e5637bfbf6dcbbf29c263a41fec12" gracePeriod=30 Jul 13 18:18:24.370332 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.370305 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:24.379896 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.379871 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69"] Jul 13 18:18:24.384798 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.384772 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a74fcd-29732775-tmc69"] Jul 13 18:18:24.578308 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.578266 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb546b49-4afe-41ea-9169-607f65f9c6ab" containerID="d9b3c5b161a241ef45885a9fa81eb590a78e5637bfbf6dcbbf29c263a41fec12" exitCode=0 Jul 13 18:18:24.578485 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.578359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" event={"ID":"cb546b49-4afe-41ea-9169-607f65f9c6ab","Type":"ContainerDied","Data":"d9b3c5b161a241ef45885a9fa81eb590a78e5637bfbf6dcbbf29c263a41fec12"} Jul 13 18:18:24.605777 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.605704 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:18:24.778739 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.778699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b25md\" (UniqueName: \"kubernetes.io/projected/cb546b49-4afe-41ea-9169-607f65f9c6ab-kube-api-access-b25md\") pod \"cb546b49-4afe-41ea-9169-607f65f9c6ab\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " Jul 13 18:18:24.779010 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.778767 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb546b49-4afe-41ea-9169-607f65f9c6ab-openshift-service-ca\") pod \"cb546b49-4afe-41ea-9169-607f65f9c6ab\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " Jul 13 18:18:24.779010 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.778872 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb546b49-4afe-41ea-9169-607f65f9c6ab-maas-api-tls\") pod \"cb546b49-4afe-41ea-9169-607f65f9c6ab\" (UID: \"cb546b49-4afe-41ea-9169-607f65f9c6ab\") " Jul 13 18:18:24.779137 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.779099 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb546b49-4afe-41ea-9169-607f65f9c6ab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb546b49-4afe-41ea-9169-607f65f9c6ab" (UID: "cb546b49-4afe-41ea-9169-607f65f9c6ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:18:24.779452 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.779411 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb546b49-4afe-41ea-9169-607f65f9c6ab-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:24.780969 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.780948 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb546b49-4afe-41ea-9169-607f65f9c6ab-kube-api-access-b25md" (OuterVolumeSpecName: "kube-api-access-b25md") pod "cb546b49-4afe-41ea-9169-607f65f9c6ab" (UID: "cb546b49-4afe-41ea-9169-607f65f9c6ab"). InnerVolumeSpecName "kube-api-access-b25md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:18:24.781089 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.781064 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb546b49-4afe-41ea-9169-607f65f9c6ab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb546b49-4afe-41ea-9169-607f65f9c6ab" (UID: "cb546b49-4afe-41ea-9169-607f65f9c6ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:18:24.881048 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.880965 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25md\" (UniqueName: \"kubernetes.io/projected/cb546b49-4afe-41ea-9169-607f65f9c6ab-kube-api-access-b25md\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:24.881048 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:24.880996 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb546b49-4afe-41ea-9169-607f65f9c6ab-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:25.585650 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:25.585613 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" Jul 13 18:18:25.586140 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:25.585646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc" event={"ID":"cb546b49-4afe-41ea-9169-607f65f9c6ab","Type":"ContainerDied","Data":"0e1ef1d6c0a475d2241d6099bbc5e2ba3f73aede8f33874493539b51cbf41abf"} Jul 13 18:18:25.586140 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:25.585721 2564 scope.go:117] "RemoveContainer" containerID="d9b3c5b161a241ef45885a9fa81eb590a78e5637bfbf6dcbbf29c263a41fec12" Jul 13 18:18:25.613263 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:25.613217 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc"] Jul 13 18:18:25.615894 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:25.615871 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a74fcd-65fc9477f7-zsmlc"] Jul 13 18:18:25.639476 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:25.639448 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb546b49-4afe-41ea-9169-607f65f9c6ab" path="/var/lib/kubelet/pods/cb546b49-4afe-41ea-9169-607f65f9c6ab/volumes" Jul 13 18:18:25.639760 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:25.639747 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c8c8f5-9f94-44e9-8964-cf0c6832f70d" path="/var/lib/kubelet/pods/d9c8c8f5-9f94-44e9-8964-cf0c6832f70d/volumes" Jul 13 18:18:28.976600 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:28.976562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:35.097141 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:35.097105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:35.160332 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:35.160293 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:40.040607 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.040570 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc"] Jul 13 18:18:40.041041 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.041027 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb546b49-4afe-41ea-9169-607f65f9c6ab" containerName="maas-api" Jul 13 18:18:40.041131 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.041042 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb546b49-4afe-41ea-9169-607f65f9c6ab" containerName="maas-api" Jul 13 18:18:40.041131 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.041061 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ea988a-db2b-41ec-a328-56900374408f" containerName="revoke-keys" Jul 13 18:18:40.041131 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.041067 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ea988a-db2b-41ec-a328-56900374408f" containerName="revoke-keys" Jul 13 18:18:40.041131 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.041123 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb546b49-4afe-41ea-9169-607f65f9c6ab" containerName="maas-api" Jul 13 18:18:40.041279 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.041135 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ea988a-db2b-41ec-a328-56900374408f" containerName="revoke-keys" Jul 13 18:18:40.044403 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.044385 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" Jul 13 18:18:40.046439 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.046420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mkgg4\"" Jul 13 18:18:40.050722 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.050688 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc"] Jul 13 18:18:40.138797 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.138759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac33380c-dff9-4138-b30f-74f2e2c152d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-762866-kwkbc\" (UID: \"ac33380c-dff9-4138-b30f-74f2e2c152d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" Jul 13 18:18:40.239659 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.239619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac33380c-dff9-4138-b30f-74f2e2c152d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-762866-kwkbc\" (UID: \"ac33380c-dff9-4138-b30f-74f2e2c152d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" Jul 13 18:18:40.240257 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.240218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac33380c-dff9-4138-b30f-74f2e2c152d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-762866-kwkbc\" (UID: \"ac33380c-dff9-4138-b30f-74f2e2c152d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" Jul 13 18:18:40.260049 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.260016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:40.355952 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.355852 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" Jul 13 18:18:40.365419 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.365388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:40.508343 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.508314 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc"] Jul 13 18:18:40.511062 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:18:40.511026 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac33380c_dff9_4138_b30f_74f2e2c152d8.slice/crio-f0a380a708b9610f4523b5284f4101e8004c43a59d517e173ef2d27c38fef250 WatchSource:0}: Error finding container f0a380a708b9610f4523b5284f4101e8004c43a59d517e173ef2d27c38fef250: Status 404 returned error can't find the container with id f0a380a708b9610f4523b5284f4101e8004c43a59d517e173ef2d27c38fef250 Jul 13 18:18:40.615030 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.614890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:40.685550 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.685516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" event={"ID":"ac33380c-dff9-4138-b30f-74f2e2c152d8","Type":"ContainerStarted","Data":"f0a380a708b9610f4523b5284f4101e8004c43a59d517e173ef2d27c38fef250"} Jul 13 18:18:40.717965 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:40.717935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:41.693065 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:41.692963 2564 generic.go:356] "Generic (PLEG): container finished" podID="ac33380c-dff9-4138-b30f-74f2e2c152d8" containerID="632e610491fe618aab6c3bf13f3c7ceb6fc712268e97cc56bbb1ce415530a827" exitCode=0 Jul 13 18:18:41.693065 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:41.693049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" event={"ID":"ac33380c-dff9-4138-b30f-74f2e2c152d8","Type":"ContainerDied","Data":"632e610491fe618aab6c3bf13f3c7ceb6fc712268e97cc56bbb1ce415530a827"} Jul 13 18:18:42.834815 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:42.834791 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" Jul 13 18:18:42.971632 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:42.971549 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac33380c-dff9-4138-b30f-74f2e2c152d8-maas-api-service-ca\") pod \"ac33380c-dff9-4138-b30f-74f2e2c152d8\" (UID: \"ac33380c-dff9-4138-b30f-74f2e2c152d8\") " Jul 13 18:18:42.971986 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:42.971963 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac33380c-dff9-4138-b30f-74f2e2c152d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac33380c-dff9-4138-b30f-74f2e2c152d8" (UID: "ac33380c-dff9-4138-b30f-74f2e2c152d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:18:43.072801 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:43.072762 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac33380c-dff9-4138-b30f-74f2e2c152d8-maas-api-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:43.706503 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:43.706474 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" Jul 13 18:18:43.706679 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:43.706472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-762866-kwkbc" event={"ID":"ac33380c-dff9-4138-b30f-74f2e2c152d8","Type":"ContainerDied","Data":"f0a380a708b9610f4523b5284f4101e8004c43a59d517e173ef2d27c38fef250"} Jul 13 18:18:43.706679 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:43.706593 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a380a708b9610f4523b5284f4101e8004c43a59d517e173ef2d27c38fef250" Jul 13 18:18:50.488866 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.488780 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67"] Jul 13 18:18:50.489295 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.488995 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" podUID="b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" containerName="maas-api" containerID="cri-o://bcfb1e91ecdf0f8937982a7e89d513e5c2e575e960ea049fab1815d9036b5ca4" gracePeriod=30 Jul 13 18:18:50.517128 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.517097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:18:50.540932 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.540904 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2"] Jul 13 18:18:50.550915 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.550886 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-762866-29732775-5cqd2"] Jul 13 18:18:50.750356 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.750326 2564 generic.go:356] "Generic (PLEG): container finished" podID="b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" containerID="bcfb1e91ecdf0f8937982a7e89d513e5c2e575e960ea049fab1815d9036b5ca4" exitCode=0 Jul 13 18:18:50.750483 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.750394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" event={"ID":"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8","Type":"ContainerDied","Data":"bcfb1e91ecdf0f8937982a7e89d513e5c2e575e960ea049fab1815d9036b5ca4"} Jul 13 18:18:50.750483 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.750442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" event={"ID":"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8","Type":"ContainerDied","Data":"b020a37c2940c7d483d3968f466dd24840375fbafef486109600ed19d7f39885"} Jul 13 18:18:50.750483 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.750458 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b020a37c2940c7d483d3968f466dd24840375fbafef486109600ed19d7f39885" Jul 13 18:18:50.763312 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.763292 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:18:50.848826 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.848786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-openshift-service-ca\") pod \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " Jul 13 18:18:50.849023 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.848936 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls\") pod \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " Jul 13 18:18:50.849023 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.848976 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6855\" (UniqueName: \"kubernetes.io/projected/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-kube-api-access-q6855\") pod \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\" (UID: \"b40b3eb5-fa1c-4370-b638-4a8c75be7aa8\") " Jul 13 18:18:50.849193 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.849165 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" (UID: "b40b3eb5-fa1c-4370-b638-4a8c75be7aa8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 18:18:50.851019 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.850991 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" (UID: "b40b3eb5-fa1c-4370-b638-4a8c75be7aa8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 18:18:50.851137 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.851094 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-kube-api-access-q6855" (OuterVolumeSpecName: "kube-api-access-q6855") pod "b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" (UID: "b40b3eb5-fa1c-4370-b638-4a8c75be7aa8"). InnerVolumeSpecName "kube-api-access-q6855". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 18:18:50.950612 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.950572 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-maas-api-tls\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:50.950612 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.950604 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6855\" (UniqueName: \"kubernetes.io/projected/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-kube-api-access-q6855\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:50.950612 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:50.950617 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8-openshift-service-ca\") on node \"ip-10-0-139-16.ec2.internal\" DevicePath \"\"" Jul 13 18:18:51.638161 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:51.638125 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70f6e12-0326-43aa-affa-fdb73861d236" path="/var/lib/kubelet/pods/a70f6e12-0326-43aa-affa-fdb73861d236/volumes" Jul 13 18:18:51.755976 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:51.755943 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67" Jul 13 18:18:51.773463 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:51.773427 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67"] Jul 13 18:18:51.777042 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:51.777014 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-762866-647fc95b8-ckd67"] Jul 13 18:18:53.640701 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:53.640667 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" path="/var/lib/kubelet/pods/b40b3eb5-fa1c-4370-b638-4a8c75be7aa8/volumes" Jul 13 18:18:55.763966 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:18:55.763926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:01.459455 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:01.459420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:01.521200 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:01.521149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:05.886174 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:05.886135 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:06.130381 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:06.130348 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:11.813418 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:11.813384 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ccf2v_4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541/manager/0.log" Jul 13 18:19:11.946145 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:11.946112 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-spqdt_74ddef11-460c-445b-a5f5-59d0843387c8/manager/0.log" Jul 13 18:19:12.075212 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:12.075137 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cbcd4b6b8-hh2pd_b2b285b0-632a-48d2-b5a8-90015cc80bb8/manager/0.log" Jul 13 18:19:12.777974 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:12.777947 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59fbf78564-n4ckc_fdf44874-a2e4-4769-834c-948b0d47384c/copy-manifests/0.log" Jul 13 18:19:12.788996 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:12.788968 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59fbf78564-n4ckc_fdf44874-a2e4-4769-834c-948b0d47384c/manager/0.log" Jul 13 18:19:13.172352 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:13.172324 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78fc4dddbd-b5jqr_4dafd306-8edf-40b5-8bf1-421805ba4210/maas-api/0.log" Jul 13 18:19:13.302050 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:13.302025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732745-bbbxw_ac522ace-8fed-40a1-8c79-24927b111be0/cleanup/0.log" Jul 13 18:19:13.419400 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:13.419373 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732760-xm76r_41735e63-82a0-4687-91cd-f8b9ae4055c3/cleanup/0.log" Jul 13 18:19:13.542909 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:13.542884 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732775-tkcjw_65a49836-9d46-4f12-8f3c-e933d462b223/cleanup/0.log" Jul 13 18:19:13.662895 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:13.662865 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-a4635dfb-92g4j_36272dc3-c046-48cc-8889-36c110c773ef/revoke-keys/0.log" Jul 13 18:19:13.800272 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:13.800173 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-ee0971b4-cg484_73491f11-7890-4a1b-8e6d-65b4d34c4a6e/revoke-keys/0.log" Jul 13 18:19:13.926482 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:13.926455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-a50684-74qvs_1d7099a5-90d1-4399-9023-c5a73bff6bd2/revoke-keys/0.log" Jul 13 18:19:14.056864 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.056779 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-b8e947-9l4lj_4f06472a-141c-4b56-acc0-aca6133091bd/revoke-keys/0.log" Jul 13 18:19:14.171737 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.171698 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-338f4926-wwtmc_097c49ab-73bd-4c8d-b052-2ced66183f43/revoke-keys/0.log" Jul 13 18:19:14.285520 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.285487 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-d09cc8-gjvt4_1b75476f-e1c1-4451-9eeb-e20f5b70f8f0/revoke-keys/0.log" Jul 13 18:19:14.403844 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.403777 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-dc368b-cc77x_307905eb-c6f3-4f38-8684-d9cc6e12f976/revoke-keys/0.log" Jul 13 18:19:14.519069 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.519043 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-bcf3d6aa-8g84m_c5da0444-4c29-4540-a2ae-8bddb45bbf5e/revoke-keys/0.log" Jul 13 18:19:14.644658 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.644634 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-3bf93f7b-4vskv_2ff04689-9cec-4783-b23a-f34944ccbd76/revoke-keys/0.log" Jul 13 18:19:14.762194 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.762140 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ee8b02ac-qr8l2_befcded0-97ef-4977-9fbf-eaadfdbe7177/revoke-keys/0.log" Jul 13 18:19:14.885759 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:14.885734 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-a74fcd-kxx8h_d8ea988a-db2b-41ec-a328-56900374408f/revoke-keys/0.log" Jul 13 18:19:15.019998 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:15.019925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-762866-kwkbc_ac33380c-dff9-4138-b30f-74f2e2c152d8/revoke-keys/0.log" Jul 13 18:19:15.146157 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:15.146125 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lbrrn_41f09881-4c99-4a5f-a14c-ac0d7c5e43c1/postgres/0.log" Jul 13 18:19:15.583919 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:15.583887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:16.964296 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:16.964269 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dgq4p_20e6b5f0-fd39-4dd9-8ee0-598baa5f295f/kuadrant-console-plugin/0.log" Jul 13 18:19:17.905474 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:17.905442 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xtj26_4f9a8244-cb35-479c-928a-7a21b0dcf134/istio-proxy/0.log" Jul 13 18:19:18.397441 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:18.397411 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-87fqm_88b20046-7323-4547-b96c-0931ce17020a/istio-proxy/1.log" Jul 13 18:19:18.523729 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:18.523696 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-9gggq_ac4d8a62-5774-4a52-95c1-8ff6643c7598/payload-pre-processing/0.log" Jul 13 18:19:18.650710 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:18.650629 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pdp4t_ac8bec71-e814-400f-b48f-9932bfc4776d/payload-processing/0.log" Jul 13 18:19:19.122625 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:19.122595 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wsbb4_2b22d4ba-fb4a-4179-953d-3bf5322de980/main/0.log" Jul 13 18:19:19.612807 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:19.612777 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cmjvp_48465a2d-2a39-49db-a243-a1c25af2d3c5/main/0.log" Jul 13 18:19:24.458263 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.458223 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nbr9z/must-gather-9gwww"] Jul 13 18:19:24.458692 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.458643 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac33380c-dff9-4138-b30f-74f2e2c152d8" containerName="revoke-keys" Jul 13 18:19:24.458692 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.458654 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac33380c-dff9-4138-b30f-74f2e2c152d8" containerName="revoke-keys" Jul 13 18:19:24.458692 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.458666 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" containerName="maas-api" Jul 13 18:19:24.458692 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.458671 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" containerName="maas-api" Jul 13 18:19:24.458827 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.458746 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac33380c-dff9-4138-b30f-74f2e2c152d8" containerName="revoke-keys" Jul 13 18:19:24.458827 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.458756 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40b3eb5-fa1c-4370-b638-4a8c75be7aa8" containerName="maas-api" Jul 13 18:19:24.461635 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.461619 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.463630 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.463602 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nbr9z\"/\"default-dockercfg-svj4k\"" Jul 13 18:19:24.463630 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.463605 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nbr9z\"/\"kube-root-ca.crt\"" Jul 13 18:19:24.464421 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.464402 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nbr9z\"/\"openshift-service-ca.crt\"" Jul 13 18:19:24.469765 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.469730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:24.471505 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.471486 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbr9z/must-gather-9gwww"] Jul 13 18:19:24.528752 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.528720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" limit=1024 Jul 13 18:19:24.573197 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.573168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsjp6\" (UniqueName: \"kubernetes.io/projected/af3a649d-f283-427c-b7bc-ca4879ea238d-kube-api-access-bsjp6\") pod \"must-gather-9gwww\" (UID: \"af3a649d-f283-427c-b7bc-ca4879ea238d\") " pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.573367 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.573311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af3a649d-f283-427c-b7bc-ca4879ea238d-must-gather-output\") pod \"must-gather-9gwww\" (UID: \"af3a649d-f283-427c-b7bc-ca4879ea238d\") " pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.674014 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.673983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsjp6\" (UniqueName: \"kubernetes.io/projected/af3a649d-f283-427c-b7bc-ca4879ea238d-kube-api-access-bsjp6\") pod \"must-gather-9gwww\" (UID: \"af3a649d-f283-427c-b7bc-ca4879ea238d\") " pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.674169 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.674034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af3a649d-f283-427c-b7bc-ca4879ea238d-must-gather-output\") pod \"must-gather-9gwww\" (UID: \"af3a649d-f283-427c-b7bc-ca4879ea238d\") " pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.674349 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.674333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af3a649d-f283-427c-b7bc-ca4879ea238d-must-gather-output\") pod \"must-gather-9gwww\" (UID: \"af3a649d-f283-427c-b7bc-ca4879ea238d\") " pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.682042 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.682017 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsjp6\" (UniqueName: \"kubernetes.io/projected/af3a649d-f283-427c-b7bc-ca4879ea238d-kube-api-access-bsjp6\") pod \"must-gather-9gwww\" (UID: \"af3a649d-f283-427c-b7bc-ca4879ea238d\") " pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.772651 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.772570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbr9z/must-gather-9gwww" Jul 13 18:19:24.907869 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.907843 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbr9z/must-gather-9gwww"] Jul 13 18:19:24.908692 ip-10-0-139-16 kubenswrapper[2564]: W0713 18:19:24.908651 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf3a649d_f283_427c_b7bc_ca4879ea238d.slice/crio-eefe9b62f4591de63b6f7d98153335462b678e31430fd78fe7c0e341da9ca159 WatchSource:0}: Error finding container eefe9b62f4591de63b6f7d98153335462b678e31430fd78fe7c0e341da9ca159: Status 404 returned error can't find the container with id eefe9b62f4591de63b6f7d98153335462b678e31430fd78fe7c0e341da9ca159 Jul 13 18:19:24.953483 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:24.953448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbr9z/must-gather-9gwww" event={"ID":"af3a649d-f283-427c-b7bc-ca4879ea238d","Type":"ContainerStarted","Data":"eefe9b62f4591de63b6f7d98153335462b678e31430fd78fe7c0e341da9ca159"} Jul 13 18:19:25.962122 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:25.962018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbr9z/must-gather-9gwww" event={"ID":"af3a649d-f283-427c-b7bc-ca4879ea238d","Type":"ContainerStarted","Data":"69e011f9df3255fdaac8b5605199113364568f9e710c9e09116511dd744097e3"} Jul 13 18:19:25.962122 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:25.962070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbr9z/must-gather-9gwww" event={"ID":"af3a649d-f283-427c-b7bc-ca4879ea238d","Type":"ContainerStarted","Data":"3bb92a4f14e836d1562a795f66791b57bf6f0e014399304eef180bfb2ba79f33"} Jul 13 18:19:25.976893 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:25.976810 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nbr9z/must-gather-9gwww" podStartSLOduration=1.195885359 podStartE2EDuration="1.976788009s" totalImagesPullingTime="780.90265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:19:25.975954222 +0000 UTC m=+2820.983356489" watchObservedRunningTime="2026-07-13 18:19:25.976788009 +0000 UTC m=+2820.984190253" Jul 13 18:19:26.356722 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:26.356687 2564 scope.go:117] "RemoveContainer" containerID="bcfb1e91ecdf0f8937982a7e89d513e5c2e575e960ea049fab1815d9036b5ca4" Jul 13 18:19:27.293289 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:27.293251 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n8t6r_30053369-dbea-4ebb-8e82-4ee42689e422/global-pull-secret-syncer/0.log" Jul 13 18:19:27.382029 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:27.381998 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-27mpb_d12f0ad1-bc89-4309-9162-54e55c73e79f/konnectivity-agent/0.log" Jul 13 18:19:27.508560 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:27.508532 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-16.ec2.internal_8df8d47ff6ebf83b9a6270b9d8e65671/haproxy/0.log" Jul 13 18:19:32.694432 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:32.694399 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dgq4p_20e6b5f0-fd39-4dd9-8ee0-598baa5f295f/kuadrant-console-plugin/0.log" Jul 13 18:19:34.726706 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.726565 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebb5417-10a2-4288-beea-1a8ee2c30776/alertmanager/0.log" Jul 13 18:19:34.752336 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.752298 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebb5417-10a2-4288-beea-1a8ee2c30776/config-reloader/0.log" Jul 13 18:19:34.777523 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.777495 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebb5417-10a2-4288-beea-1a8ee2c30776/kube-rbac-proxy-web/0.log" Jul 13 18:19:34.804057 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.804023 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebb5417-10a2-4288-beea-1a8ee2c30776/kube-rbac-proxy/0.log" Jul 13 18:19:34.826409 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.826350 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebb5417-10a2-4288-beea-1a8ee2c30776/kube-rbac-proxy-metric/0.log" Jul 13 18:19:34.851246 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.851112 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebb5417-10a2-4288-beea-1a8ee2c30776/prom-label-proxy/0.log" Jul 13 18:19:34.876591 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.876527 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ebb5417-10a2-4288-beea-1a8ee2c30776/init-config-reloader/0.log" Jul 13 18:19:34.962868 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.962840 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nd4nm_49dfab4f-19d7-459a-9634-023750e51552/kube-state-metrics/0.log" Jul 13 18:19:34.993963 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:34.993880 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nd4nm_49dfab4f-19d7-459a-9634-023750e51552/kube-rbac-proxy-main/0.log" Jul 13 18:19:35.028880 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.028803 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nd4nm_49dfab4f-19d7-459a-9634-023750e51552/kube-rbac-proxy-self/0.log" Jul 13 18:19:35.062345 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.062308 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dcd9bfc4d-td6c8_b06cdc92-4954-4afa-bc32-6e2dfdac1faa/metrics-server/0.log" Jul 13 18:19:35.087679 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.087636 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-hjb27_51db2690-8634-4f9c-b6a9-69f00b05ba2e/monitoring-plugin/0.log" Jul 13 18:19:35.118922 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.118858 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9nr6w_15e4507a-4eb4-4e02-8afd-e6b9055e9af8/node-exporter/0.log" Jul 13 18:19:35.140960 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.140934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9nr6w_15e4507a-4eb4-4e02-8afd-e6b9055e9af8/kube-rbac-proxy/0.log" Jul 13 18:19:35.162271 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.162214 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9nr6w_15e4507a-4eb4-4e02-8afd-e6b9055e9af8/init-textfile/0.log" Jul 13 18:19:35.380088 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.380039 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-hg4v5_ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a/kube-rbac-proxy-main/0.log" Jul 13 18:19:35.407619 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.407591 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-hg4v5_ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a/kube-rbac-proxy-self/0.log" Jul 13 18:19:35.435072 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.434996 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-hg4v5_ae03d7c9-d123-4c13-a2fd-b6a8ea2b755a/openshift-state-metrics/0.log" Jul 13 18:19:35.482845 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.482821 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61/prometheus/0.log" Jul 13 18:19:35.509843 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.509740 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61/config-reloader/0.log" Jul 13 18:19:35.536014 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.535974 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61/thanos-sidecar/0.log" Jul 13 18:19:35.559421 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.559353 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61/kube-rbac-proxy-web/0.log" Jul 13 18:19:35.583181 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.583125 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61/kube-rbac-proxy/0.log" Jul 13 18:19:35.608973 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.608942 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61/kube-rbac-proxy-thanos/0.log" Jul 13 18:19:35.630644 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.630564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b87bfe57-0e9d-4ce4-96bb-3c3eed39cf61/init-config-reloader/0.log" Jul 13 18:19:35.663132 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.663099 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6w656_f9498398-0795-44e3-a8d2-4de79b211633/prometheus-operator/0.log" Jul 13 18:19:35.683993 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.683969 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6w656_f9498398-0795-44e3-a8d2-4de79b211633/kube-rbac-proxy/0.log" Jul 13 18:19:35.708841 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.708814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-pdl84_9eb98afd-67bb-4a40-bc76-166bede84957/prometheus-operator-admission-webhook/0.log" Jul 13 18:19:35.751086 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.751032 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c8fbf89f4-fk4kt_b59cfd5f-2222-44d3-bb81-3caaf98b632e/telemeter-client/0.log" Jul 13 18:19:35.773940 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.773842 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c8fbf89f4-fk4kt_b59cfd5f-2222-44d3-bb81-3caaf98b632e/reload/0.log" Jul 13 18:19:35.795445 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.795412 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c8fbf89f4-fk4kt_b59cfd5f-2222-44d3-bb81-3caaf98b632e/kube-rbac-proxy/0.log" Jul 13 18:19:35.825900 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.825866 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79f46758fc-2cnvm_c2b5ed53-33d0-4f7b-846f-f5603bba2992/thanos-query/0.log" Jul 13 18:19:35.848620 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.848552 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79f46758fc-2cnvm_c2b5ed53-33d0-4f7b-846f-f5603bba2992/kube-rbac-proxy-web/0.log" Jul 13 18:19:35.870428 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.870367 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79f46758fc-2cnvm_c2b5ed53-33d0-4f7b-846f-f5603bba2992/kube-rbac-proxy/0.log" Jul 13 18:19:35.895036 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.895010 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79f46758fc-2cnvm_c2b5ed53-33d0-4f7b-846f-f5603bba2992/prom-label-proxy/0.log" Jul 13 18:19:35.933917 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.933886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79f46758fc-2cnvm_c2b5ed53-33d0-4f7b-846f-f5603bba2992/kube-rbac-proxy-rules/0.log" Jul 13 18:19:35.955097 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:35.955073 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79f46758fc-2cnvm_c2b5ed53-33d0-4f7b-846f-f5603bba2992/kube-rbac-proxy-metrics/0.log" Jul 13 18:19:37.434667 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.434627 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7"] Jul 13 18:19:37.443300 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.443269 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.453940 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.453914 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7"] Jul 13 18:19:37.526832 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.526796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-lib-modules\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.527128 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.527108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-podres\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.527325 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.527306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjh56\" (UniqueName: \"kubernetes.io/projected/0a2ff17b-132b-4caf-90b6-1afc66947cc7-kube-api-access-xjh56\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.527472 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.527459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-proc\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.527580 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.527568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-sys\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.628820 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.628779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjh56\" (UniqueName: \"kubernetes.io/projected/0a2ff17b-132b-4caf-90b6-1afc66947cc7-kube-api-access-xjh56\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629009 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.628851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-proc\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629009 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.628880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-sys\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629009 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.628928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-lib-modules\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629009 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.628984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-podres\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629266 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.629184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-podres\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629266 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.629217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-sys\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629385 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.629298 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-proc\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.629385 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.629299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a2ff17b-132b-4caf-90b6-1afc66947cc7-lib-modules\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.638174 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.638148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjh56\" (UniqueName: \"kubernetes.io/projected/0a2ff17b-132b-4caf-90b6-1afc66947cc7-kube-api-access-xjh56\") pod \"perf-node-gather-daemonset-6j7j7\" (UID: \"0a2ff17b-132b-4caf-90b6-1afc66947cc7\") " pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.758777 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.758679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:37.963308 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:37.963278 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7"] Jul 13 18:19:38.080060 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:38.080017 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" event={"ID":"0a2ff17b-132b-4caf-90b6-1afc66947cc7","Type":"ContainerStarted","Data":"3b63793603b9f2611d0bed9d3de05796eadd47d39bcc31ece76d6520db2b46a0"} Jul 13 18:19:39.089851 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:39.089813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" event={"ID":"0a2ff17b-132b-4caf-90b6-1afc66947cc7","Type":"ContainerStarted","Data":"d3f253d960e9dc1298c98f1b8d00ce1ccd598723ce572917d75bc40fa00cb929"} Jul 13 18:19:39.090373 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:39.089885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:39.107508 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:39.107452 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" podStartSLOduration=2.107433085 podStartE2EDuration="2.107433085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:19:39.104420303 +0000 UTC m=+2834.111822560" watchObservedRunningTime="2026-07-13 18:19:39.107433085 +0000 UTC m=+2834.114835330" Jul 13 18:19:39.989160 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:39.989121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nvsmc_ff15fb73-676b-4e3a-9f03-e7a50131d82e/dns/0.log" Jul 13 18:19:40.015865 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:40.015836 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nvsmc_ff15fb73-676b-4e3a-9f03-e7a50131d82e/kube-rbac-proxy/0.log" Jul 13 18:19:40.162743 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:40.162711 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9tx5d_ca2937f1-bf59-4da2-a238-5d057c9bcec2/dns-node-resolver/0.log" Jul 13 18:19:40.781775 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:40.781744 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jfss5_52c44237-67a1-466a-a213-73d437bd48de/node-ca/0.log" Jul 13 18:19:41.754117 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:41.754071 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xtj26_4f9a8244-cb35-479c-928a-7a21b0dcf134/istio-proxy/0.log" Jul 13 18:19:42.036765 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:42.036732 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-87fqm_88b20046-7323-4547-b96c-0931ce17020a/istio-proxy/0.log" Jul 13 18:19:42.082463 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:42.082431 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-87fqm_88b20046-7323-4547-b96c-0931ce17020a/istio-proxy/1.log" Jul 13 18:19:42.117813 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:42.117783 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-9gggq_ac4d8a62-5774-4a52-95c1-8ff6643c7598/payload-pre-processing/0.log" Jul 13 18:19:42.150263 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:42.150217 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pdp4t_ac8bec71-e814-400f-b48f-9932bfc4776d/payload-processing/0.log" Jul 13 18:19:42.915931 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:42.915903 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hdllh_3636a3d0-240f-4f02-a5dd-19c8530fd533/serve-healthcheck-canary/0.log" Jul 13 18:19:43.521073 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:43.521034 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qr6m_ead76a25-f4a9-453c-b1bb-4449cff1edc0/kube-rbac-proxy/0.log" Jul 13 18:19:43.541646 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:43.541618 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qr6m_ead76a25-f4a9-453c-b1bb-4449cff1edc0/exporter/0.log" Jul 13 18:19:43.564053 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:43.564016 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qr6m_ead76a25-f4a9-453c-b1bb-4449cff1edc0/extractor/0.log" Jul 13 18:19:45.115717 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:45.114751 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nbr9z/perf-node-gather-daemonset-6j7j7" Jul 13 18:19:45.724227 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:45.724194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ccf2v_4a2a1d4b-bdf6-4ecc-9e92-42144bdd9541/manager/0.log" Jul 13 18:19:45.915561 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:45.915526 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-spqdt_74ddef11-460c-445b-a5f5-59d0843387c8/manager/0.log" Jul 13 18:19:46.057953 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:46.057850 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cbcd4b6b8-hh2pd_b2b285b0-632a-48d2-b5a8-90015cc80bb8/manager/0.log" Jul 13 18:19:46.411620 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:46.411517 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59fbf78564-n4ckc_fdf44874-a2e4-4769-834c-948b0d47384c/manager/0.log" Jul 13 18:19:46.433961 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:46.433933 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59fbf78564-n4ckc_fdf44874-a2e4-4769-834c-948b0d47384c/copy-manifests/0.log" Jul 13 18:19:47.299276 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:47.299226 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xtj26_4f9a8244-cb35-479c-928a-7a21b0dcf134/istio-proxy/0.log" Jul 13 18:19:47.549497 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:47.549465 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-87fqm_88b20046-7323-4547-b96c-0931ce17020a/istio-proxy/0.log" Jul 13 18:19:47.589022 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:47.588974 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-87fqm_88b20046-7323-4547-b96c-0931ce17020a/istio-proxy/1.log" Jul 13 18:19:47.612721 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:47.612693 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-9gggq_ac4d8a62-5774-4a52-95c1-8ff6643c7598/payload-pre-processing/0.log" Jul 13 18:19:47.635942 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:47.635916 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pdp4t_ac8bec71-e814-400f-b48f-9932bfc4776d/payload-processing/0.log" Jul 13 18:19:47.845072 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:47.844977 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-84567cbc5f-68grs_b32cc088-a455-4bf1-b259-5bec26ff5918/manager/0.log" Jul 13 18:19:48.870474 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:48.870442 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xtj26_4f9a8244-cb35-479c-928a-7a21b0dcf134/istio-proxy/0.log" Jul 13 18:19:49.111519 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:49.111493 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-87fqm_88b20046-7323-4547-b96c-0931ce17020a/istio-proxy/0.log" Jul 13 18:19:49.153332 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:49.153282 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-87fqm_88b20046-7323-4547-b96c-0931ce17020a/istio-proxy/1.log" Jul 13 18:19:49.180340 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:49.180310 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-9gggq_ac4d8a62-5774-4a52-95c1-8ff6643c7598/payload-pre-processing/0.log" Jul 13 18:19:49.205595 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:49.205566 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-pdp4t_ac8bec71-e814-400f-b48f-9932bfc4776d/payload-processing/0.log" Jul 13 18:19:54.182345 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:54.182320 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bcvkf_57acac65-9ea0-4727-90d8-ac9ade3c7055/kube-multus-additional-cni-plugins/0.log" Jul 13 18:19:54.206014 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:54.205992 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bcvkf_57acac65-9ea0-4727-90d8-ac9ade3c7055/egress-router-binary-copy/0.log" Jul 13 18:19:54.231748 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:54.231726 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bcvkf_57acac65-9ea0-4727-90d8-ac9ade3c7055/cni-plugins/0.log" Jul 13 18:19:54.255729 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:54.255711 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bcvkf_57acac65-9ea0-4727-90d8-ac9ade3c7055/bond-cni-plugin/0.log" Jul 13 18:19:54.277398 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:54.277348 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bcvkf_57acac65-9ea0-4727-90d8-ac9ade3c7055/routeoverride-cni/0.log" Jul 13 18:19:54.298844 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:54.298826 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bcvkf_57acac65-9ea0-4727-90d8-ac9ade3c7055/whereabouts-cni-bincopy/0.log" Jul 13 18:19:54.324083 ip-10-0-139-16 kubenswrapper[2564]: I0713 18:19:54.324063 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bcvkf_57acac65-9ea0-4727-90d8-ac9ade3c7055/whereabouts-cni/0.log"